shishengliang/lingniu-vehicle-ingest
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat(go): expose stat writer latency histogram

Browse Source
This commit is contained in:
lingniu
2026-07-03 19:08:49 +08:00
parent 3900144ea4
commit 07d17e5adf
6 changed files with 41 additions and 1 deletions
Download Patch File Download Diff File Expand all files Collapse all files

1
docs/ops/100k-capacity-baseline.md
View File

@@ -80,6 +80,7 @@ systemd gateway 已配置 `LimitNOFILE=1048576`，需要持续保持。
| Realtime Redis | `vehicle_realtime_store_update_duration_ms_histogram_bucket{store="redis"}` | p99 毫秒级且不持续上升 |
| Realtime MySQL | `vehicle_realtime_store_update_duration_ms_histogram_bucket{store="mysql"}` | p99 不持续上升 |
| Realtime MySQL | async queue depth | 不持续增长 |
| Stat writer | `vehicle_stat_write_duration_ms_histogram_bucket` | p99 不持续上升 |
## 分阶段压测

1
docs/ops/go-service-observability.md
View File

@@ -52,6 +52,7 @@ curl -fsS http://127.0.0.1:20200/readyz
| `vehicle_history_batch_pending_rows` | Parsed raw envelopes waiting in the current TDengine batch. |
| `vehicle_history_batch_flush_duration_ms` | Last TDengine batch flush duration. Labels: `status`. |
| `vehicle_stat_writes_total` | MySQL metric writes. Labels: `topic`, `status`. |
| `vehicle_stat_write_duration_ms_histogram` | MySQL stat writer append duration histogram. Labels: `topic`, `status`. |
| `vehicle_realtime_updates_total` | Redis/MySQL realtime projector updates. Labels: `topic`, `status`. |
| `vehicle_realtime_store_update_duration_ms_histogram` | Redis/MySQL store update duration histogram. Labels: `store`, `protocol`, `status`. |
| `vehicle_history_kafka_lag` | Estimated Kafka lag for history writer by topic and partition. |

12
docs/ops/go-vehicle-ingest-memory.md
View File

@@ -435,6 +435,18 @@ vehicle_realtime_store_update_duration_ms_histogram_sum{store,protocol,status}
该耗时覆盖 Redis 快速实时投影和 MySQL 异步当前态/位置投影的实际写入耗时。带帧率压测时，Redis p99 上升会直接影响 realtime consumer 追平；MySQL p99 上升需要结合 `vehicle_realtime_async_queue_depth` 和 `vehicle_realtime_async_queue_total{status="dropped"}` 判断是否开始丢弃低优先级当前态写入。
### Stat writer write histogram
2026-07-03 已为 stat-writer MySQL append 增加 duration histogram：
```text
vehicle_stat_write_duration_ms_histogram_bucket{topic,status,le}
vehicle_stat_write_duration_ms_histogram_count{topic,status}
vehicle_stat_write_duration_ms_histogram_sum{topic,status}
```
该耗时覆盖单条 Kafka 消息解析后写入 `vehicle_daily_mileage` 的 MySQL append 路径。带帧率压测时，如果该 p99 持续上升并伴随 `vehicle_stat_kafka_lag` 增长，优先排查 MySQL 写入、每日里程表索引和 stat-writer 消费能力。
### Gateway frame duration histogram
2026-07-03 已为 Gateway 帧处理耗时增加 histogram：

1
docs/ops/vehicle-ingest-runbook.md
View File

@@ -131,6 +131,7 @@ go run ./cmd/load-sim \
| `vehicle_bridge_batch_duration_ms_histogram_bucket` | p99 连续 5 分钟上升 | Kafka 写入或 NATS ack 开始变慢，通常会先于 ack-pending 扩大。 |
| `vehicle_realtime_store_update_duration_ms_histogram_bucket{store="redis"}` | p99 连续 5 分钟上升 | Redis 实时投影变慢，会直接影响 realtime consumer 追平能力。 |
| `vehicle_realtime_store_update_duration_ms_histogram_bucket{store="mysql"}` | p99 连续 5 分钟上升 | MySQL 当前态/位置投影变慢，需结合 async queue depth 和 dropped 计数判断。 |
| `vehicle_stat_write_duration_ms_histogram_bucket` | p99 连续 5 分钟上升 | MySQL 每日里程统计写入变慢，可能导致 stat Kafka lag 增长。 |
| Kafka lag | 连续 5 分钟增长或 `> 10000` | 下游 consumer 或存储存在瓶颈。 |
| Writer 成功计数 | 入口增长但 writer 不增长 | bridge、Kafka、consumer 或存储链路断开。 |