feat(go): expose fast writer nats consumer metrics
This commit is contained in:
@@ -76,6 +76,8 @@ systemd gateway 已配置 `LimitNOFILE=1048576`,需要持续保持。
|
||||
| NATS bridge | `vehicle_bridge_nats_consumer_pending` | burst 后下降 |
|
||||
| NATS bridge | `vehicle_bridge_batch_pending_messages` | 稳态接近 0,burst 后下降 |
|
||||
| NATS bridge | `vehicle_bridge_batch_duration_ms_histogram_bucket` | p99 不持续上升 |
|
||||
| NATS fast writer | `vehicle_fast_writer_nats_consumer_ack_pending` | 稳态为 0 |
|
||||
| NATS fast writer | `vehicle_fast_writer_nats_consumer_pending` | burst 后下降 |
|
||||
| NATS fast writer | `vehicle_fast_writer_batch_pending_messages` | 稳态接近 0,burst 后下降 |
|
||||
| NATS fast writer | `vehicle_fast_writer_batch_pending_envelopes` | 稳态接近 0,burst 后下降 |
|
||||
| NATS fast writer | `vehicle_fast_writer_stage_duration_ms_histogram_bucket` | TDengine/Redis/ack 各阶段 p99 不持续上升 |
|
||||
|
||||
@@ -47,6 +47,9 @@ curl -fsS http://127.0.0.1:20200/readyz
|
||||
| `vehicle_bridge_batch_pending_messages` | NATS messages fetched by the bridge but not yet written to Kafka and acked. |
|
||||
| `vehicle_bridge_batch_pending_kafka_messages` | Kafka messages prepared for the current bridge batch. |
|
||||
| `vehicle_bridge_batch_duration_ms_histogram` | Bridge batch duration histogram covering Kafka write and NATS ack. Labels: `status`. |
|
||||
| `vehicle_fast_writer_nats_consumer_pending` | JetStream messages pending for the fast writer durable consumer. |
|
||||
| `vehicle_fast_writer_nats_consumer_ack_pending` | JetStream messages delivered to fast writer but not yet acked. |
|
||||
| `vehicle_fast_writer_nats_consumer_waiting` | Pull requests waiting on the fast writer durable consumer. |
|
||||
| `vehicle_fast_writer_batch_pending_messages` | NATS fast writer messages fetched but not yet written to TDengine/Redis and acked. |
|
||||
| `vehicle_fast_writer_batch_pending_envelopes` | Valid parsed envelopes in the current NATS fast writer batch. |
|
||||
| `vehicle_fast_writer_stage_duration_ms_histogram` | Fast writer stage duration histogram for TDengine, Redis, and NATS ack. Labels: `subject`, `stage`, `status`. |
|
||||
|
||||
@@ -426,11 +426,14 @@ vehicle_fast_writer_stage_duration_ms_histogram_sum{subject,stage,status}
|
||||
同时新增 batch pending gauge:
|
||||
|
||||
```text
|
||||
vehicle_fast_writer_nats_consumer_pending{stream,consumer}
|
||||
vehicle_fast_writer_nats_consumer_ack_pending{stream,consumer}
|
||||
vehicle_fast_writer_nats_consumer_waiting{stream,consumer}
|
||||
vehicle_fast_writer_batch_pending_messages
|
||||
vehicle_fast_writer_batch_pending_envelopes
|
||||
```
|
||||
|
||||
这两个指标表示 fast-writer 已拉取但尚未完成 TDengine/Redis/ack 的批内压力。带帧率压测时它们应在稳态接近 0,burst 后能够回落;如果持续非 0,需要结合 `vehicle_fast_writer_stage_duration_ms_histogram` 判断卡在 TDengine、Redis 还是 ack。
|
||||
`vehicle_fast_writer_nats_consumer_*` 表示 JetStream durable consumer 层面的 backlog/ack-pending;`vehicle_fast_writer_batch_pending_*` 表示 fast-writer 已拉取但尚未完成 TDengine/Redis/ack 的批内压力。带帧率压测时它们应在稳态接近 0,burst 后能够回落;如果持续非 0,需要结合 `vehicle_fast_writer_stage_duration_ms_histogram` 判断卡在 NATS 拉取、TDengine、Redis 还是 ack。
|
||||
|
||||
### NATS Kafka bridge batch 指标
|
||||
|
||||
|
||||
@@ -129,6 +129,8 @@ go run ./cmd/load-sim \
|
||||
| `vehicle_bridge_nats_consumer_pending` | 持续增长且 `> 10000` | bridge 消费 NATS 的速度跟不上生产速度。 |
|
||||
| `vehicle_bridge_batch_pending_messages` | 持续非 0 或 burst 后不回落 | bridge 已拉取 NATS 消息但尚未完成 Kafka 写入和 NATS ack。 |
|
||||
| `vehicle_bridge_batch_duration_ms_histogram_bucket` | p99 连续 5 分钟上升 | Kafka 写入或 NATS ack 开始变慢,通常会先于 ack-pending 扩大。 |
|
||||
| `vehicle_fast_writer_nats_consumer_ack_pending` | 连续 2 分钟 `> 0` | 消息已投递给 fast-writer,但 TDengine/Redis 写入后未完成 ack。 |
|
||||
| `vehicle_fast_writer_nats_consumer_pending` | 持续增长且 `> 10000` | fast-writer 消费 NATS 的速度跟不上入口写入速度。 |
|
||||
| `vehicle_fast_writer_batch_pending_messages` | 持续非 0 或 burst 后不回落 | fast-writer 已拉取 NATS 消息但尚未完成 TDengine/Redis 写入和 ack。 |
|
||||
| `vehicle_fast_writer_batch_pending_envelopes` | 持续非 0 或 burst 后不回落 | fast-writer 当前批次已有有效 envelope 在等待落库或 ack。 |
|
||||
| `vehicle_fast_writer_stage_duration_ms_histogram_bucket` | 某个 stage 的 p99 连续 5 分钟上升 | NATS 快速写链路在 TDengine、Redis 或 NATS ack 某一阶段变慢。 |
|
||||
|
||||
Reference in New Issue
Block a user