shishengliang/lingniu-vehicle-ingest
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat(go): expose fast writer stage latency histogram

Browse Source
This commit is contained in:
lingniu
2026-07-03 19:12:58 +08:00
parent 07d17e5adf
commit 2db05d100d
6 changed files with 79 additions and 7 deletions
Download Patch File Download Diff File Expand all files Collapse all files

1
docs/ops/100k-capacity-baseline.md
View File

@@ -76,6 +76,7 @@ systemd gateway 已配置 `LimitNOFILE=1048576`，需要持续保持。
| NATS bridge | `vehicle_bridge_nats_consumer_pending` | burst 后下降 |
| NATS bridge | `vehicle_bridge_batch_pending_messages` | 稳态接近 0，burst 后下降 |
| NATS bridge | `vehicle_bridge_batch_duration_ms_histogram_bucket` | p99 不持续上升 |
| NATS fast writer | `vehicle_fast_writer_stage_duration_ms_histogram_bucket` | TDengine/Redis/ack 各阶段 p99 不持续上升 |
| Kafka consumers | `vehicle_*_kafka_lag` | 稳态为 0，burst 后下降 |
| Realtime Redis | `vehicle_realtime_store_update_duration_ms_histogram_bucket{store="redis"}` | p99 毫秒级且不持续上升 |
| Realtime MySQL | `vehicle_realtime_store_update_duration_ms_histogram_bucket{store="mysql"}` | p99 不持续上升 |

1
docs/ops/go-service-observability.md
View File

@@ -47,6 +47,7 @@ curl -fsS http://127.0.0.1:20200/readyz
| `vehicle_bridge_batch_pending_messages` | NATS messages fetched by the bridge but not yet written to Kafka and acked. |
| `vehicle_bridge_batch_pending_kafka_messages` | Kafka messages prepared for the current bridge batch. |
| `vehicle_bridge_batch_duration_ms_histogram` | Bridge batch duration histogram covering Kafka write and NATS ack. Labels: `status`. |
| `vehicle_fast_writer_stage_duration_ms_histogram` | Fast writer stage duration histogram for TDengine, Redis, and NATS ack. Labels: `subject`, `stage`, `status`. |
| `vehicle_history_writes_total` | TDengine history writes. Labels: `topic`, `status`. |
| `vehicle_history_batch_pending_messages` | Messages fetched by history writer but not yet flushed and committed. |
| `vehicle_history_batch_pending_rows` | Parsed raw envelopes waiting in the current TDengine batch. |

12
docs/ops/go-vehicle-ingest-memory.md
View File

@@ -409,6 +409,18 @@ vehicle_async_sink_queue_depth{sink}
这些指标是进入带帧率压测前的关键保护栏：如果 `queue_depth` 持续增长、`enqueue_total{status="timeout"}` 增长，或 `vehicle_async_sink_publish_duration_ms_histogram_bucket` 的 p99 持续上升，说明入口 publish 队列或下游 NATS/Kafka 写入已成为瓶颈。
### NATS fast writer stage histogram
2026-07-03 已为 NATS fast writer 增加阶段耗时 histogram：
```text
vehicle_fast_writer_stage_duration_ms_histogram_bucket{subject,stage,status,le}
vehicle_fast_writer_stage_duration_ms_histogram_count{subject,stage,status}
vehicle_fast_writer_stage_duration_ms_histogram_sum{subject,stage,status}
```
`stage` 取值包括 `tdengine`、`redis`、`ack`，分别对应快速写 TDengine raw/location、Redis realtime 投影、NATS ack。带帧率压测时如果某个 stage 的 p99 持续上升，可以直接定位 fast path 是存储瓶颈还是 ack/网络瓶颈。
### NATS Kafka bridge batch 指标
2026-07-03 已为 NATS -> Kafka bridge 增加 batch pending 和 duration histogram：

1
docs/ops/vehicle-ingest-runbook.md
View File

@@ -129,6 +129,7 @@ go run ./cmd/load-sim \
| `vehicle_bridge_nats_consumer_pending` | 持续增长且 `> 10000` | bridge 消费 NATS 的速度跟不上生产速度。 |
| `vehicle_bridge_batch_pending_messages` | 持续非 0 或 burst 后不回落 | bridge 已拉取 NATS 消息但尚未完成 Kafka 写入和 NATS ack。 |
| `vehicle_bridge_batch_duration_ms_histogram_bucket` | p99 连续 5 分钟上升 | Kafka 写入或 NATS ack 开始变慢，通常会先于 ack-pending 扩大。 |
| `vehicle_fast_writer_stage_duration_ms_histogram_bucket` | 某个 stage 的 p99 连续 5 分钟上升 | NATS 快速写链路在 TDengine、Redis 或 NATS ack 某一阶段变慢。 |
| `vehicle_realtime_store_update_duration_ms_histogram_bucket{store="redis"}` | p99 连续 5 分钟上升 | Redis 实时投影变慢，会直接影响 realtime consumer 追平能力。 |
| `vehicle_realtime_store_update_duration_ms_histogram_bucket{store="mysql"}` | p99 连续 5 分钟上升 | MySQL 当前态/位置投影变慢，需结合 async queue depth 和 dropped 计数判断。 |
| `vehicle_stat_write_duration_ms_histogram_bucket` | p99 连续 5 分钟上升 | MySQL 每日里程统计写入变慢，可能导致 stat Kafka lag 增长。 |