shishengliang/lingniu-vehicle-ingest
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat(go): expose history batch pending metrics

Browse Source
This commit is contained in:
lingniu
2026-07-03 18:46:40 +08:00
parent 1cbe29c5fc
commit 71905aca3d
6 changed files with 79 additions and 7 deletions
Download Patch File Download Diff File Expand all files Collapse all files

4
docs/ops/100k-capacity-baseline.md
View File

@@ -68,6 +68,8 @@ systemd gateway 已配置 `LimitNOFILE=1048576`，需要持续保持。
| Gateway | parse/publish duration | p99 小于容量目标 |
| History writer | `vehicle_history_batch_flush_total{status}` | `error` 不增长 |
| History writer | `vehicle_history_batch_rows_total{status}` | batch 行数持续增长 |
| History writer | `vehicle_history_batch_pending_messages` | 稳态接近 0，burst 后下降 |
| History writer | `vehicle_history_batch_pending_rows` | 稳态接近 0，burst 后下降 |
| History writer | `vehicle_history_batch_flush_duration_ms{status}` | flush 延迟不持续上升 |
| NATS bridge | `vehicle_bridge_nats_consumer_ack_pending` | 稳态为 0 |
| NATS bridge | `vehicle_bridge_nats_consumer_pending` | burst 后下降 |
@@ -112,7 +114,7 @@ systemd gateway 已配置 `LimitNOFILE=1048576`，需要持续保持。
- gateway frame/publish counters
- NATS pending 和 ack pending
- Kafka lag
- Redis/MySQL/TDengine 写入耗时
- Redis/MySQL/TDengine 写入耗时和 pending depth
- CPU/load/memory/file descriptors
- 错误日志

3
docs/ops/go-service-observability.md
View File

@@ -43,6 +43,9 @@ curl -fsS http://127.0.0.1:20200/readyz
| `vehicle_bridge_nats_consumer_ack_pending` | JetStream messages delivered to bridge but not yet acked. |
| `vehicle_bridge_nats_consumer_waiting` | Pull requests waiting on the bridge durable consumer. |
| `vehicle_history_writes_total` | TDengine history writes. Labels: `topic`, `status`. |
| `vehicle_history_batch_pending_messages` | Messages fetched by history writer but not yet flushed and committed. |
| `vehicle_history_batch_pending_rows` | Parsed raw envelopes waiting in the current TDengine batch. |
| `vehicle_history_batch_flush_duration_ms` | Last TDengine batch flush duration. Labels: `status`. |
| `vehicle_stat_writes_total` | MySQL metric writes. Labels: `topic`, `status`. |
| `vehicle_realtime_updates_total` | Redis/MySQL realtime projector updates. Labels: `topic`, `status`. |
| `vehicle_history_kafka_lag` | Estimated Kafka lag for history writer by topic and partition. |

4
docs/ops/go-vehicle-ingest-memory.md
View File

@@ -420,10 +420,12 @@ vehicle_async_sink_queue_depth{sink}
```text
vehicle_history_batch_flush_total{status}
vehicle_history_batch_rows_total{status}
vehicle_history_batch_pending_messages
vehicle_history_batch_pending_rows
vehicle_history_batch_flush_duration_ms{status}
```
进入带帧率压测时，必须同时观察 `vehicle_history_kafka_lag` 和 `vehicle_history_batch_*`。
进入带帧率压测时，必须同时观察 `vehicle_history_kafka_lag`、`vehicle_history_batch_pending_*` 和 `vehicle_history_batch_*`。如果 pending 持续非 0 但 Kafka lag 尚未放大，说明 history-writer 已经在本地批写或提交阶段形成早期积压。
## 下次恢复上下文时先做

2
docs/ops/vehicle-ingest-runbook.md
View File

@@ -120,6 +120,8 @@ go run ./cmd/load-sim \
| `vehicle_async_sink_enqueue_total{status="timeout"}` | 任意增长 | Gateway publish 队列已满或 worker 长时间阻塞，入口可能开始丢实时性。 |
| `vehicle_async_sink_publish_total{status="error"}` | 连续增长 | NATS/Kafka publish 失败，需要先查中间件连接和日志。 |
| `vehicle_history_batch_flush_total{status="error"}` | 任意增长 | TDengine 批写失败；Kafka offset 不会提交，应先查 TDengine 和 SQL 错误。 |
| `vehicle_history_batch_pending_messages` | 持续非 0 或 burst 后不回落 | history-writer 已拉取但未完成写入/提交，可能卡在 TDengine 或 Kafka commit。 |
| `vehicle_history_batch_pending_rows` | 持续非 0 或 burst 后不回落 | TDengine 有批量写入积压，通常早于 Kafka lag 放大。 |
| `vehicle_history_batch_flush_duration_ms{status="ok"}` | 持续上升 | TDengine 写入延迟增加，可能需要降低 batch size 或扩容 TDengine。 |
| `vehicle_bridge_nats_consumer_ack_pending` | 连续 2 分钟 `> 0` | 消息已投递给 bridge，但 Kafka 写入后未完成 ack。 |
| `vehicle_bridge_nats_consumer_pending` | 持续增长且 `> 10000` | bridge 消费 NATS 的速度跟不上生产速度。 |