shishengliang/lingniu-vehicle-ingest
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat(go): expose bridge batch pressure metrics

Browse Source
This commit is contained in:
lingniu
2026-07-03 18:58:37 +08:00
parent 16ee09180c
commit 018015f9de
6 changed files with 97 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

2
docs/ops/100k-capacity-baseline.md
View File

@@ -73,6 +73,8 @@ systemd gateway 已配置 `LimitNOFILE=1048576`，需要持续保持。
| History writer | `vehicle_history_batch_flush_duration_ms{status}` | flush 延迟不持续上升 |
| NATS bridge | `vehicle_bridge_nats_consumer_ack_pending` | 稳态为 0 |
| NATS bridge | `vehicle_bridge_nats_consumer_pending` | burst 后下降 |
| NATS bridge | `vehicle_bridge_batch_pending_messages` | 稳态接近 0，burst 后下降 |
| NATS bridge | `vehicle_bridge_batch_duration_ms_histogram_bucket` | p99 不持续上升 |
| Kafka consumers | `vehicle_*_kafka_lag` | 稳态为 0，burst 后下降 |
| Realtime Redis | `vehicle_realtime_store_update_duration_ms{store="redis"}` | 毫秒级 |
| Realtime MySQL | async queue depth | 不持续增长 |

3
docs/ops/go-service-observability.md
View File

@@ -43,6 +43,9 @@ curl -fsS http://127.0.0.1:20200/readyz
| `vehicle_bridge_nats_consumer_pending` | JetStream messages pending for the bridge durable consumer. |
| `vehicle_bridge_nats_consumer_ack_pending` | JetStream messages delivered to bridge but not yet acked. |
| `vehicle_bridge_nats_consumer_waiting` | Pull requests waiting on the bridge durable consumer. |
| `vehicle_bridge_batch_pending_messages` | NATS messages fetched by the bridge but not yet written to Kafka and acked. |
| `vehicle_bridge_batch_pending_kafka_messages` | Kafka messages prepared for the current bridge batch. |
| `vehicle_bridge_batch_duration_ms_histogram` | Bridge batch duration histogram covering Kafka write and NATS ack. Labels: `status`. |
| `vehicle_history_writes_total` | TDengine history writes. Labels: `topic`, `status`. |
| `vehicle_history_batch_pending_messages` | Messages fetched by history writer but not yet flushed and committed. |
| `vehicle_history_batch_pending_rows` | Parsed raw envelopes waiting in the current TDengine batch. |

14
docs/ops/go-vehicle-ingest-memory.md
View File

@@ -406,6 +406,20 @@ vehicle_async_sink_queue_depth{sink}
这些指标是进入带帧率压测前的关键保护栏：如果 `queue_depth` 持续增长或 `enqueue_total{status="timeout"}` 增长，说明入口 publish 队列已成为瓶颈。
### NATS Kafka bridge batch 指标
2026-07-03 已为 NATS -> Kafka bridge 增加 batch pending 和 duration histogram：
```text
vehicle_bridge_batch_pending_messages
vehicle_bridge_batch_pending_kafka_messages
vehicle_bridge_batch_duration_ms_histogram_bucket{status,le}
vehicle_bridge_batch_duration_ms_histogram_count{status}
vehicle_bridge_batch_duration_ms_histogram_sum{status}
```
该耗时覆盖一批 NATS 消息路由到 Kafka、Kafka 同步写入、NATS ack 的完整过程。带帧率压测时，如果 pending 持续非 0 或 duration p99 持续上升，应优先排查 Kafka broker、bridge batch size、NATS ack-pending 和网络。
### Gateway frame duration histogram
2026-07-03 已为 Gateway 帧处理耗时增加 histogram：

2
docs/ops/vehicle-ingest-runbook.md
View File

@@ -126,6 +126,8 @@ go run ./cmd/load-sim \
| `vehicle_history_batch_flush_duration_ms{status="ok"}` | 持续上升 | TDengine 写入延迟增加，可能需要降低 batch size 或扩容 TDengine。 |
| `vehicle_bridge_nats_consumer_ack_pending` | 连续 2 分钟 `> 0` | 消息已投递给 bridge，但 Kafka 写入后未完成 ack。 |
| `vehicle_bridge_nats_consumer_pending` | 持续增长且 `> 10000` | bridge 消费 NATS 的速度跟不上生产速度。 |
| `vehicle_bridge_batch_pending_messages` | 持续非 0 或 burst 后不回落 | bridge 已拉取 NATS 消息但尚未完成 Kafka 写入和 NATS ack。 |
| `vehicle_bridge_batch_duration_ms_histogram_bucket` | p99 连续 5 分钟上升 | Kafka 写入或 NATS ack 开始变慢，通常会先于 ack-pending 扩大。 |
| Kafka lag | 连续 5 分钟增长或 `> 10000` | 下游 consumer 或存储存在瓶颈。 |
| Writer 成功计数 | 入口增长但 writer 不增长 | bridge、Kafka、consumer 或存储链路断开。 |