diff --git a/docs/ops/100k-capacity-baseline.md b/docs/ops/100k-capacity-baseline.md index 7949e09a..f709c3ef 100644 --- a/docs/ops/100k-capacity-baseline.md +++ b/docs/ops/100k-capacity-baseline.md @@ -77,7 +77,8 @@ systemd gateway 已配置 `LimitNOFILE=1048576`，需要持续保持。 | NATS bridge | `vehicle_bridge_batch_pending_messages` | 稳态接近 0，burst 后下降 | | NATS bridge | `vehicle_bridge_batch_duration_ms_histogram_bucket` | p99 不持续上升 | | Kafka consumers | `vehicle_*_kafka_lag` | 稳态为 0，burst 后下降 | -| Realtime Redis | `vehicle_realtime_store_update_duration_ms{store="redis"}` | 毫秒级 | +| Realtime Redis | `vehicle_realtime_store_update_duration_ms_histogram_bucket{store="redis"}` | p99 毫秒级且不持续上升 | +| Realtime MySQL | `vehicle_realtime_store_update_duration_ms_histogram_bucket{store="mysql"}` | p99 不持续上升 | | Realtime MySQL | async queue depth | 不持续增长 | ## 分阶段压测 diff --git a/docs/ops/go-service-observability.md b/docs/ops/go-service-observability.md index d8da6877..cd569c59 100644 --- a/docs/ops/go-service-observability.md +++ b/docs/ops/go-service-observability.md @@ -53,6 +53,7 @@ curl -fsS http://127.0.0.1:20200/readyz | `vehicle_history_batch_flush_duration_ms` | Last TDengine batch flush duration. Labels: `status`. | | `vehicle_stat_writes_total` | MySQL metric writes. Labels: `topic`, `status`. | | `vehicle_realtime_updates_total` | Redis/MySQL realtime projector updates. Labels: `topic`, `status`. | +| `vehicle_realtime_store_update_duration_ms_histogram` | Redis/MySQL store update duration histogram. Labels: `store`, `protocol`, `status`. | | `vehicle_history_kafka_lag` | Estimated Kafka lag for history writer by topic and partition. | | `vehicle_stat_kafka_lag` | Estimated Kafka lag for stat writer by topic and partition. | | `vehicle_realtime_kafka_lag` | Estimated Kafka lag for realtime projector by topic and partition. | diff --git a/docs/ops/go-vehicle-ingest-memory.md b/docs/ops/go-vehicle-ingest-memory.md index 17880f79..8e29f5d7 100644 --- a/docs/ops/go-vehicle-ingest-memory.md +++ b/docs/ops/go-vehicle-ingest-memory.md @@ -423,6 +423,18 @@ vehicle_bridge_batch_duration_ms_histogram_sum{status} 该耗时覆盖一批 NATS 消息路由到 Kafka、Kafka 同步写入、NATS ack 的完整过程。带帧率压测时，如果 pending 持续非 0 或 duration p99 持续上升，应优先排查 Kafka broker、bridge batch size、NATS ack-pending 和网络。 +### Realtime store update histogram + +2026-07-03 已为 realtime-api Redis/MySQL store update 增加 duration histogram： + +```text +vehicle_realtime_store_update_duration_ms_histogram_bucket{store,protocol,status,le} +vehicle_realtime_store_update_duration_ms_histogram_count{store,protocol,status} +vehicle_realtime_store_update_duration_ms_histogram_sum{store,protocol,status} +``` + +该耗时覆盖 Redis 快速实时投影和 MySQL 异步当前态/位置投影的实际写入耗时。带帧率压测时，Redis p99 上升会直接影响 realtime consumer 追平；MySQL p99 上升需要结合 `vehicle_realtime_async_queue_depth` 和 `vehicle_realtime_async_queue_total{status="dropped"}` 判断是否开始丢弃低优先级当前态写入。 + ### Gateway frame duration histogram 2026-07-03 已为 Gateway 帧处理耗时增加 histogram： diff --git a/docs/ops/vehicle-ingest-runbook.md b/docs/ops/vehicle-ingest-runbook.md index 7560169f..437f683e 100644 --- a/docs/ops/vehicle-ingest-runbook.md +++ b/docs/ops/vehicle-ingest-runbook.md @@ -129,6 +129,8 @@ go run ./cmd/load-sim \ | `vehicle_bridge_nats_consumer_pending` | 持续增长且 `> 10000` | bridge 消费 NATS 的速度跟不上生产速度。 | | `vehicle_bridge_batch_pending_messages` | 持续非 0 或 burst 后不回落 | bridge 已拉取 NATS 消息但尚未完成 Kafka 写入和 NATS ack。 | | `vehicle_bridge_batch_duration_ms_histogram_bucket` | p99 连续 5 分钟上升 | Kafka 写入或 NATS ack 开始变慢，通常会先于 ack-pending 扩大。 | +| `vehicle_realtime_store_update_duration_ms_histogram_bucket{store="redis"}` | p99 连续 5 分钟上升 | Redis 实时投影变慢，会直接影响 realtime consumer 追平能力。 | +| `vehicle_realtime_store_update_duration_ms_histogram_bucket{store="mysql"}` | p99 连续 5 分钟上升 | MySQL 当前态/位置投影变慢，需结合 async queue depth 和 dropped 计数判断。 | | Kafka lag | 连续 5 分钟增长或 `> 10000` | 下游 consumer 或存储存在瓶颈。 | | Writer 成功计数 | 入口增长但 writer 不增长 | bridge、Kafka、consumer 或存储链路断开。 | diff --git a/go/vehicle-gateway/cmd/realtime-api/main.go b/go/vehicle-gateway/cmd/realtime-api/main.go index 62d10c19..de619380 100644 --- a/go/vehicle-gateway/cmd/realtime-api/main.go +++ b/go/vehicle-gateway/cmd/realtime-api/main.go @@ -249,6 +249,8 @@ type storeMetricUpdater struct { registry *metrics.Registry } +var realtimeStoreUpdateDurationBucketsMS = []float64{1, 5, 10, 25, 50, 100, 250, 500, 1000, 5000} + func (u storeMetricUpdater) Update(ctx context.Context, env envelope.FrameEnvelope) error { if u.delegate == nil { return nil @@ -261,16 +263,15 @@ func (u storeMetricUpdater) Update(ctx context.Context, env envelope.FrameEnvelo status = "error" } if u.registry != nil { - u.registry.IncCounter("vehicle_realtime_store_updates_total", metrics.Labels{ + labels := metrics.Labels{ "store": u.store, "protocol": string(env.Protocol), "status": status, - }) - u.registry.SetGauge("vehicle_realtime_store_update_duration_ms", metrics.Labels{ - "store": u.store, - "protocol": string(env.Protocol), - "status": status, - }, float64(elapsed.Milliseconds())) + } + elapsedMS := float64(elapsed.Milliseconds()) + u.registry.IncCounter("vehicle_realtime_store_updates_total", labels) + u.registry.SetGauge("vehicle_realtime_store_update_duration_ms", labels, elapsedMS) + u.registry.ObserveHistogram("vehicle_realtime_store_update_duration_ms_histogram", labels, realtimeStoreUpdateDurationBucketsMS, elapsedMS) } return err } diff --git a/go/vehicle-gateway/cmd/realtime-api/main_test.go b/go/vehicle-gateway/cmd/realtime-api/main_test.go index 67c113cf..516e5f76 100644 --- a/go/vehicle-gateway/cmd/realtime-api/main_test.go +++ b/go/vehicle-gateway/cmd/realtime-api/main_test.go @@ -179,6 +179,9 @@ func TestStoreMetricUpdaterRecordsStoreUpdateResults(t *testing.T) { for _, want := range []string{ `vehicle_realtime_store_updates_total{protocol="JT808",status="ok",store="redis"} 1`, `vehicle_realtime_store_update_duration_ms{protocol="JT808",status="ok",store="redis"}`, + `vehicle_realtime_store_update_duration_ms_histogram_bucket{le="+Inf",protocol="JT808",status="ok",store="redis"} 1`, + `vehicle_realtime_store_update_duration_ms_histogram_count{protocol="JT808",status="ok",store="redis"} 1`, + `vehicle_realtime_store_update_duration_ms_histogram_sum{protocol="JT808",status="ok",store="redis"}`, } { if !strings.Contains(text, want) { t.Fatalf("store update metric missing %s:\n%s", want, text)