feat(go): batch tdengine history writes
This commit is contained in:
@@ -50,10 +50,9 @@
|
||||
|
||||
| 指标 | 含义 |
|
||||
| --- | --- |
|
||||
| `vehicle_history_batch_flush_total{table,status}` | 批写成功/失败次数 |
|
||||
| `vehicle_history_batch_rows_total{table,status}` | 批写行数 |
|
||||
| `vehicle_history_batch_flush_duration_ms{table,status}` | 批写耗时 |
|
||||
| `vehicle_history_batch_pending_rows{table}` | 当前待 flush 行数 |
|
||||
| `vehicle_history_batch_flush_total{status}` | 批写成功/失败次数 |
|
||||
| `vehicle_history_batch_rows_total{status}` | 批写行数 |
|
||||
| `vehicle_history_batch_flush_duration_ms{status}` | 批写耗时 |
|
||||
| `vehicle_history_kafka_lag` | 下游是否追得上 Kafka |
|
||||
|
||||
## Rollout
|
||||
@@ -63,3 +62,28 @@
|
||||
3. 在测试环境打开 batch writer。
|
||||
4. 生产先用小批次 `100/100ms`,观察 TDengine latency 和 Kafka lag。
|
||||
5. 逐步提升到 `200-500/100ms`。
|
||||
|
||||
## Implementation Status
|
||||
|
||||
2026-07-03 已完成第一阶段实现:
|
||||
|
||||
- `history.Writer.AppendAllBatch` 按 raw child table 和 location child table 生成多行 `INSERT ... VALUES (...),(...)`。
|
||||
- oversized payload chunks 仍写入 `raw_frame_payload_chunks`,同样按 child table 批量写。
|
||||
- `history-writer` Kafka 消费默认按 `HISTORY_BATCH_SIZE=200`、`HISTORY_BATCH_WAIT_MS=100` 收集消息。
|
||||
- TDengine batch 成功后才批量提交 Kafka messages;batch 失败不提交 offset,让 Kafka 保留可重放语义。
|
||||
- 保留 `processHistoryMessage` 和 `AppendAll` 单条路径,便于回退和测试。
|
||||
|
||||
新增指标:
|
||||
|
||||
```text
|
||||
vehicle_history_batch_flush_total{status}
|
||||
vehicle_history_batch_rows_total{status}
|
||||
vehicle_history_batch_flush_duration_ms{status}
|
||||
```
|
||||
|
||||
生产 rollout 建议:
|
||||
|
||||
1. 先部署默认 `200/100ms`。
|
||||
2. 观察 `vehicle_history_batch_*` 和 `vehicle_history_kafka_lag`。
|
||||
3. 如果 TDengine latency 上升或 Kafka lag 不下降,将 `HISTORY_BATCH_SIZE` 降到 `100`。
|
||||
4. 如果稳定且有 backlog,再逐步提升到 `500`。
|
||||
|
||||
@@ -66,6 +66,9 @@ systemd gateway 已配置 `LimitNOFILE=1048576`,需要持续保持。
|
||||
| Gateway async sink | `vehicle_async_sink_publish_total{sink,kind,status}` | `error` 不增长 |
|
||||
| Gateway | `vehicle_gateway_publish_total{status="ok"}` | 持续增长 |
|
||||
| Gateway | parse/publish duration | p99 小于容量目标 |
|
||||
| History writer | `vehicle_history_batch_flush_total{status}` | `error` 不增长 |
|
||||
| History writer | `vehicle_history_batch_rows_total{status}` | batch 行数持续增长 |
|
||||
| History writer | `vehicle_history_batch_flush_duration_ms{status}` | flush 延迟不持续上升 |
|
||||
| NATS bridge | `vehicle_bridge_nats_consumer_ack_pending` | 稳态为 0 |
|
||||
| NATS bridge | `vehicle_bridge_nats_consumer_pending` | burst 后下降 |
|
||||
| Kafka consumers | `vehicle_*_kafka_lag` | 稳态为 0,burst 后下降 |
|
||||
|
||||
@@ -406,6 +406,25 @@ vehicle_async_sink_queue_depth{sink}
|
||||
|
||||
这些指标是进入带帧率压测前的关键保护栏:如果 `queue_depth` 持续增长或 `enqueue_total{status="timeout"}` 增长,说明入口 publish 队列已成为瓶颈。
|
||||
|
||||
### History writer TDengine 批写
|
||||
|
||||
2026-07-03 已实现 history-writer 第一阶段批写:
|
||||
|
||||
- `history.Writer.AppendAllBatch` 按 TDengine child table 聚合 raw/location 多行 `INSERT`。
|
||||
- `cmd/history-writer` 默认 `HISTORY_BATCH_SIZE=200`、`HISTORY_BATCH_WAIT_MS=100`。
|
||||
- TDengine batch 成功后才批量提交 Kafka messages;失败不提交 offset。
|
||||
- 保留单条 `AppendAll` 路径作为回退。
|
||||
|
||||
新增指标:
|
||||
|
||||
```text
|
||||
vehicle_history_batch_flush_total{status}
|
||||
vehicle_history_batch_rows_total{status}
|
||||
vehicle_history_batch_flush_duration_ms{status}
|
||||
```
|
||||
|
||||
进入带帧率压测时,必须同时观察 `vehicle_history_kafka_lag` 和 `vehicle_history_batch_*`。
|
||||
|
||||
## 下次恢复上下文时先做
|
||||
|
||||
1. 读本文件。
|
||||
|
||||
@@ -73,6 +73,7 @@ curl -fsS http://127.0.0.1:20214/metrics \
|
||||
| grep -E 'vehicle_bridge_(nats_consumer|kafka_writes_total|nats_acks_total)'
|
||||
|
||||
curl -fsS http://127.0.0.1:20212/metrics | grep vehicle_history_kafka_lag
|
||||
curl -fsS http://127.0.0.1:20212/metrics | grep vehicle_history_batch
|
||||
curl -fsS http://127.0.0.1:20213/metrics | grep vehicle_stat_kafka_lag
|
||||
curl -fsS http://127.0.0.1:20200/metrics | grep vehicle_realtime_kafka_lag
|
||||
```
|
||||
@@ -118,6 +119,8 @@ go run ./cmd/load-sim \
|
||||
| `vehicle_async_sink_queue_depth{sink="nats"}` | 持续增长且不回落 | Gateway 到 NATS/Kafka 的异步 publish 队列开始积压。 |
|
||||
| `vehicle_async_sink_enqueue_total{status="timeout"}` | 任意增长 | Gateway publish 队列已满或 worker 长时间阻塞,入口可能开始丢实时性。 |
|
||||
| `vehicle_async_sink_publish_total{status="error"}` | 连续增长 | NATS/Kafka publish 失败,需要先查中间件连接和日志。 |
|
||||
| `vehicle_history_batch_flush_total{status="error"}` | 任意增长 | TDengine 批写失败;Kafka offset 不会提交,应先查 TDengine 和 SQL 错误。 |
|
||||
| `vehicle_history_batch_flush_duration_ms{status="ok"}` | 持续上升 | TDengine 写入延迟增加,可能需要降低 batch size 或扩容 TDengine。 |
|
||||
| `vehicle_bridge_nats_consumer_ack_pending` | 连续 2 分钟 `> 0` | 消息已投递给 bridge,但 Kafka 写入后未完成 ack。 |
|
||||
| `vehicle_bridge_nats_consumer_pending` | 持续增长且 `> 10000` | bridge 消费 NATS 的速度跟不上生产速度。 |
|
||||
| Kafka lag | 连续 5 分钟增长或 `> 10000` | 下游 consumer 或存储存在瓶颈。 |
|
||||
|
||||
Reference in New Issue
Block a user