feat(go): batch redis fast writer projections
This commit is contained in:
@@ -54,7 +54,7 @@ curl -fsS http://127.0.0.1:20200/readyz
|
||||
| `vehicle_fast_writer_nats_consumer_waiting` | Pull requests waiting on the fast writer durable consumer. |
|
||||
| `vehicle_fast_writer_batch_pending_messages` | NATS fast writer messages fetched but not yet written to TDengine/Redis and acked. |
|
||||
| `vehicle_fast_writer_batch_pending_envelopes` | Valid parsed envelopes in the current NATS fast writer batch. |
|
||||
| `vehicle_fast_writer_stage_duration_ms_histogram` | Fast writer stage duration histogram for TDengine, Redis, and NATS ack. Labels: `subject`, `stage`, `status`. |
|
||||
| `vehicle_fast_writer_stage_duration_ms_histogram` | Fast writer stage duration histogram for TDengine, Redis, and NATS ack. The Redis stage is one batch pipeline when the repository supports batch updates. Labels: `subject`, `stage`, `status`. |
|
||||
| `vehicle_history_writes_total` | TDengine history writes. Labels: `topic`, `status`. |
|
||||
| `vehicle_history_batch_pending_messages` | Messages fetched by history writer but not yet flushed and committed. |
|
||||
| `vehicle_history_batch_pending_rows` | Parsed raw envelopes waiting in the current TDengine batch. |
|
||||
|
||||
@@ -20,6 +20,8 @@ Go 版本车辆数据接入链路已经作为生产主链路运行在 ECS `115.2
|
||||
5. 解析字段只投影一次,避免 TDengine/Redis/MySQL 各自重复 flatten。
|
||||
6. 新长期目标:朝 10W 车辆生产稳定运行优化,容量基线见 `docs/ops/100k-capacity-baseline.md`。
|
||||
|
||||
2026-07-03 之后,`nats-fast-writer` 的快速路径是 TDengine 批写 + Redis 批量 pipeline:一个 NATS fetch batch 先批量写 TDengine,再用 `FastUpdateBatch` 把实时 KV、在线 TTL、last_seen 和协议索引合并到一次 Redis pipeline,最后逐条 ack NATS。
|
||||
|
||||
## 服务和端口
|
||||
|
||||
ECS:`115.29.187.205`
|
||||
@@ -431,7 +433,7 @@ vehicle_fast_writer_stage_duration_ms_histogram_sum{subject,stage,status}
|
||||
|
||||
`stage` 取值包括 `tdengine`、`redis`、`ack`,分别对应快速写 TDengine raw/location、Redis realtime 投影、NATS ack。带帧率压测时如果某个 stage 的 p99 持续上升,可以直接定位 fast path 是存储瓶颈还是 ack/网络瓶颈。
|
||||
|
||||
2026-07-03 后续优化:`nats-fast-writer` 已从“Fetch batch 后逐条写 TDengine”改为“Fetch batch 后先调用 `history.Writer.AppendAllBatch` 批量写 TDengine,再逐条更新 Redis 和 ack”。这样可以减少 TDengine round trip;Redis/ack 仍按单条执行,避免某条实时投影失败时误 ack。
|
||||
2026-07-03 后续优化:`nats-fast-writer` 已从“Fetch batch 后逐条写 TDengine/Redis”改为“Fetch batch 后先调用 `history.Writer.AppendAllBatch` 批量写 TDengine,再调用 `realtime.Repository.FastUpdateBatch` 用一次 Redis pipeline 写实时 KV/在线状态/索引,最后逐条 ack NATS”。这样可以同时减少 TDengine 和 Redis round trip;ack 仍按单条执行,避免存储失败时误 ack。
|
||||
|
||||
2026-07-03 后续优化:`nats-fast-writer` 的 TDengine SQL 连接池已从固定 `1` 改为可配置:
|
||||
|
||||
|
||||
@@ -136,7 +136,7 @@ go run ./cmd/load-sim \
|
||||
| `vehicle_fast_writer_nats_consumer_pending` | 持续增长且 `> 10000` | fast-writer 消费 NATS 的速度跟不上入口写入速度。 |
|
||||
| `vehicle_fast_writer_batch_pending_messages` | 持续非 0 或 burst 后不回落 | fast-writer 已拉取 NATS 消息但尚未完成 TDengine/Redis 写入和 ack。 |
|
||||
| `vehicle_fast_writer_batch_pending_envelopes` | 持续非 0 或 burst 后不回落 | fast-writer 当前批次已有有效 envelope 在等待落库或 ack。 |
|
||||
| `vehicle_fast_writer_stage_duration_ms_histogram_bucket` | 某个 stage 的 p99 连续 5 分钟上升 | NATS 快速写链路在 TDengine、Redis 或 NATS ack 某一阶段变慢;TDengine 阶段可小步调整 `FAST_WRITER_TDENGINE_MAX_OPEN_CONNS`,调整后必须观察是否出现选库错误和 ack-pending 增长。 |
|
||||
| `vehicle_fast_writer_stage_duration_ms_histogram_bucket` | 某个 stage 的 p99 连续 5 分钟上升 | NATS 快速写链路在 TDengine、Redis 或 NATS ack 某一阶段变慢;TDengine 阶段可小步调整 `FAST_WRITER_TDENGINE_MAX_OPEN_CONNS`,Redis 阶段是整批 pipeline 写入耗时,调整后必须观察是否出现选库错误和 ack-pending 增长。 |
|
||||
| `vehicle_realtime_store_update_duration_ms_histogram_bucket{store="redis"}` | p99 连续 5 分钟上升 | Redis 实时投影变慢,会直接影响 realtime consumer 追平能力。 |
|
||||
| `vehicle_realtime_store_update_duration_ms_histogram_bucket{store="mysql"}` | p99 连续 5 分钟上升 | MySQL 当前态/位置投影变慢,需结合 async queue depth 和 dropped 计数判断。 |
|
||||
| `vehicle_stat_write_duration_ms_histogram_bucket` | p99 连续 5 分钟上升 | MySQL 每日里程统计写入变慢,可能导致 stat Kafka lag 增长。 |
|
||||
|
||||
Reference in New Issue
Block a user