556 lines
22 KiB
Markdown
556 lines
22 KiB
Markdown
# Go 车辆接入上下文记忆
|
||
|
||
更新时间：2026-07-03
|
||
|
||
这份文档用于在 Codex 上下文重载后快速恢复项目状态。更完整的运行排查步骤见：
|
||
|
||
- `docs/ops/vehicle-ingest-runbook.md`
|
||
- `docs/architecture/production-data-plane-inventory.md`
|
||
|
||
## 当前目标
|
||
|
||
Go 版本车辆数据接入链路已经作为生产主链路运行在 ECS `115.29.187.205`。
|
||
|
||
核心目标：
|
||
|
||
1. 高性能可靠接收 GB32960、JT808、宇通 MQTT。
|
||
2. 协议解析后先写 NATS，再桥接 Kafka。
|
||
3. Kafka raw 供 TDengine 历史、Redis 实时、MySQL 当前态/统计消费。
|
||
4. 尽量避免 Java 老链路、Xinda 相关链路和重复存储。
|
||
5. 解析字段只投影一次，避免 TDengine/Redis/MySQL 各自重复 flatten。
|
||
6. 新长期目标：朝 10W 车辆生产稳定运行优化，容量基线见 `docs/ops/100k-capacity-baseline.md`。
|
||
|
||
2026-07-03 之后，`nats-fast-writer` 的快速路径是 TDengine 批写 + Redis 批量 pipeline：一个 NATS fetch batch 先批量写 TDengine，再用 `FastUpdateBatch` 把实时 KV、在线 TTL、last_seen 和协议索引合并到一次 Redis pipeline，最后逐条 ack NATS。
|
||
|
||
TDengine writer 的 raw/location 子表创建有进程内单飞保护：同一子表 key 并发首次写入时，只有一个 goroutine 执行 `CREATE TABLE IF NOT EXISTS` 和 tag 更新，其他 goroutine 等待结果后继续 INSERT，避免 10W 车辆启动或回放时对 TDengine 形成重复 DDL 风暴。
|
||
|
||
容量健康摘要工具：`/opt/lingniu-go-native/current/capacity-check` 会抓本机各 Go 服务 `/metrics` 并输出 JSON。关键 backlog、Kafka lag、连接拒绝或 metrics 抓取失败时退出码为 `2`，可以接 cron/告警。
|
||
|
||
## 服务和端口
|
||
|
||
ECS：`115.29.187.205`
|
||
|
||
| 服务 | systemd | 端口 | 作用 |
|
||
| --- | --- | --- | --- |
|
||
| Gateway | `lingniu-go-gateway.service` | `808`、`32960`、`20211` | JT808、GB32960、宇通 MQTT 接入和解析 |
|
||
| NATS fast writer | `lingniu-go-nats-fast-writer.service` | 健康端口随 env | NATS 快速写 TDengine/Redis |
|
||
| NATS Kafka bridge | `lingniu-go-nats-kafka-bridge.service` | `20214` | NATS 到 Kafka |
|
||
| History writer | `lingniu-go-history-writer.service` | `20212` | Kafka raw 写 TDengine |
|
||
| Stat writer | `lingniu-go-stat-writer.service` | `20213` | Kafka raw 写每日里程 |
|
||
| Realtime API | `lingniu-go-realtime-api.service` | `20200` | Redis/MySQL 投影和 HTTP API |
|
||
|
||
业务端口：
|
||
|
||
- JT808：`0.0.0.0:808`
|
||
- GB32960：`0.0.0.0:32960`
|
||
- Swagger/API：`http://115.29.187.205:20200/swagger-ui/index.html`
|
||
|
||
## 部署路径
|
||
|
||
Go 服务使用 systemd 裸机部署，不使用 Docker。
|
||
|
||
```bash
|
||
/opt/lingniu-go-native/current
|
||
/opt/lingniu-go-native/releases/*
|
||
/opt/lingniu-go-native/env/*.env
|
||
```
|
||
|
||
最近 release：
|
||
|
||
```text
|
||
/opt/lingniu-go-native/releases/production-track-telemetry-20260714082451
|
||
/opt/lingniu-go-native/releases/100k-hardening-20260703175730
|
||
```
|
||
|
||
服务重启：
|
||
|
||
```bash
|
||
systemctl restart lingniu-go-gateway.service
|
||
systemctl restart lingniu-go-nats-fast-writer.service
|
||
systemctl restart lingniu-go-nats-kafka-bridge.service
|
||
systemctl restart lingniu-go-history-writer.service
|
||
systemctl restart lingniu-go-stat-writer.service
|
||
systemctl restart lingniu-go-realtime-api.service
|
||
```
|
||
|
||
健康检查：
|
||
|
||
```bash
|
||
for port in 20211 20212 20213 20214 20200; do
|
||
curl -fsS "http://127.0.0.1:${port}/readyz"
|
||
echo
|
||
done
|
||
```
|
||
|
||
## 中间件
|
||
|
||
Kafka / NATS ECS：
|
||
|
||
- 公网：`114.55.58.251`
|
||
- 内网：`172.17.111.56`
|
||
- NATS：`172.17.111.56:4222`
|
||
|
||
TDengine：
|
||
|
||
- 内网：`172.17.111.57`
|
||
- WS：`172.17.111.57:6041`
|
||
- 数据库：`lingniu_vehicle_ts`
|
||
|
||
MySQL RDS：
|
||
|
||
- 内网：`rm-bp179zbv481rnw3e2.mysql.rds.aliyuncs.com:3306`
|
||
- 数据库：`lingniu_vehicle_data`
|
||
|
||
Redis：
|
||
|
||
- 内网：`r-bp1u741kij7e51i481.redis.rds.aliyuncs.com:6379`
|
||
- DB：`50`
|
||
|
||
不要在最终回答中明文输出生产密码。
|
||
|
||
## Topic
|
||
|
||
当前 Go raw topic：
|
||
|
||
```text
|
||
vehicle.raw.go.gb32960.v1
|
||
vehicle.raw.go.jt808.v1
|
||
vehicle.raw.go.yutong-mqtt.v1
|
||
```
|
||
|
||
fields topic 已存在：
|
||
|
||
```text
|
||
vehicle.fields.go.gb32960.v1
|
||
vehicle.fields.go.jt808.v1
|
||
vehicle.fields.go.yutong-mqtt.v1
|
||
```
|
||
|
||
当前核心链路仍以 raw topic 作为事实输入；raw envelope 里已经包含一次性预计算的 `parsed_fields`。
|
||
|
||
## 代码结构
|
||
|
||
主要目录：
|
||
|
||
```text
|
||
go/vehicle-gateway/cmd/gateway
|
||
go/vehicle-gateway/cmd/history-writer
|
||
go/vehicle-gateway/cmd/realtime-api
|
||
go/vehicle-gateway/cmd/stat-writer
|
||
go/vehicle-gateway/cmd/nats-fast-writer
|
||
go/vehicle-gateway/cmd/nats-kafka-bridge
|
||
go/vehicle-gateway/internal/protocol/gb32960
|
||
go/vehicle-gateway/internal/protocol/jt808
|
||
go/vehicle-gateway/internal/protocol/yutongmqtt
|
||
go/vehicle-gateway/internal/realtime
|
||
go/vehicle-gateway/internal/history
|
||
go/vehicle-gateway/internal/identity
|
||
```
|
||
|
||
每次改代码后在本地跑：
|
||
|
||
```bash
|
||
cd go/vehicle-gateway
|
||
go test ./...
|
||
```
|
||
|
||
## 解析和存储原则
|
||
|
||
### 一次解析字段
|
||
|
||
2026-07-03 已完成优化：
|
||
|
||
- `FrameEnvelope` 增加：
|
||
- `parsed_fields`
|
||
- `parsed_field_types`
|
||
- Gateway 在发布 raw 前调用 `realtime.EnsureParsedFields(&env)`。
|
||
- TDengine writer、Redis realtime KV、MySQL snapshot 优先使用 `env.ParsedFields`。
|
||
- 旧消息没有 `parsed_fields` 时，保留 fallback，从 `Parsed` 展开一次。
|
||
|
||
相关文件：
|
||
|
||
```text
|
||
go/vehicle-gateway/internal/envelope/envelope.go
|
||
go/vehicle-gateway/internal/realtime/kv.go
|
||
go/vehicle-gateway/internal/gateway/tcp_server.go
|
||
go/vehicle-gateway/internal/gateway/mqtt_client.go
|
||
go/vehicle-gateway/internal/history/writer.go
|
||
go/vehicle-gateway/internal/realtime/repository.go
|
||
go/vehicle-gateway/internal/realtime/snapshot_writer.go
|
||
```
|
||
|
||
### GB32960 多帧
|
||
|
||
GB32960 有两类实际模式：
|
||
|
||
1. 单条 `0x02` 同时包含标准 data unit 和 vendor data unit。
|
||
2. 标准帧加 vendor 栈分片帧。
|
||
|
||
例子：
|
||
|
||
- `LB9A32A24R0LS1426`：单帧完整混合上报，单帧约 `774 bytes`，`gd_fc_stack.cell_count=108`。
|
||
- `LNXNEGRR7SR318212`：多帧上报，`gd_fc_stack.cell_count=432`，按 `200 + 200 + 32` 分片。
|
||
|
||
TDengine raw_frames 必须保持单帧事实，不跨帧合并。
|
||
|
||
Redis/MySQL 实时视图可以按 `protocol + vin` 增量合并，但不能让后来的 stack-only 帧覆盖标准车辆字段。
|
||
|
||
### JT808
|
||
|
||
JT808 主入口端口是 `808`，不再使用 `8089`。
|
||
|
||
解析重点：
|
||
|
||
- 包头手机号按 BCD 去前导 0。
|
||
- `0x0100` 注册帧写 `jt808_registration`。
|
||
- `0x0102` 鉴权帧尝试按 phone 查 registration/binding 关联 VIN。
|
||
- `0x0200` 位置帧即使没有注册/鉴权，也会按 phone 降级查 `vehicle_identity_binding`，建立 session。
|
||
- 位置附加信息 `0x01` 是 GPS 总里程，按差值用于每日里程。
|
||
|
||
## MySQL 身份表
|
||
|
||
`vehicle_identity_binding` 是外部维护事实表，服务只读，不允许运行时写入。
|
||
|
||
核心字段：
|
||
|
||
```text
|
||
vin
|
||
plate
|
||
phone
|
||
oem
|
||
updated_at
|
||
```
|
||
|
||
用途：
|
||
|
||
- JT808 用 phone 或 plate 找 VIN。
|
||
- Realtime snapshot/location 用 VIN 反查 plate。
|
||
- 后续导入 G7s、广安车联等来源时更新 phone/oem。
|
||
|
||
`jt808_registration` 是 808 注册鉴权运行状态表，以 phone 为主键，记录注册、鉴权、source_endpoint、关联 VIN 状态。
|
||
|
||
## 2026-07-03 数据维护记录
|
||
|
||
### G7s binding
|
||
|
||
源文件：
|
||
|
||
```text
|
||
/Users/lingniu/Library/Mobile Documents/com~apple~CloudDocs/rsync/2026_07_02_17_11_37转发配置列表-1(2).xlsx
|
||
```
|
||
|
||
处理规则：
|
||
|
||
1. 删除原有 G7/G7s 的 `phone/oem`。
|
||
2. 使用文件中的 `手机号` 列更新。
|
||
3. 冲突的不更新，其他更新。
|
||
|
||
执行结果：
|
||
|
||
- 清空旧 G7/G7s：`666` 行。
|
||
- 更新安全 G7s：`561` 行。
|
||
- 跳过：`72` 行。
|
||
- 缺 plate 或 VIN：`14`
|
||
- 与非 G7/G7s 绑定冲突：`58`
|
||
|
||
跳过清单：
|
||
|
||
```text
|
||
outputs/g7s_binding_skipped_apply.csv
|
||
```
|
||
|
||
### JT808 unknown VIN 导出
|
||
|
||
导出条件：
|
||
|
||
- `jt808_registration.vin` 为空、`unknown` 或 `unknow`
|
||
- 且 `phone` 不存在于 `vehicle_identity_binding.phone`
|
||
|
||
结果：
|
||
|
||
- `30` 个 phone
|
||
|
||
导出文件：
|
||
|
||
```text
|
||
outputs/jt808_unknown_vin_phone_not_in_binding.csv
|
||
```
|
||
|
||
## 常用查询
|
||
|
||
历史 raw frame：
|
||
|
||
```bash
|
||
curl -s 'http://115.29.187.205:20200/api/history/raw-frames?protocol=GB32960&vin=LB9A32A24R0LS1426&limit=1&includeFields=true'
|
||
```
|
||
|
||
只取指定 parsed fields：
|
||
|
||
```bash
|
||
curl -s 'http://115.29.187.205:20200/api/history/raw-frames?protocol=GB32960&vin=LB9A32A24R0LS1426&limit=1&includeFields=true&fields=gb32960.vehicle.speed_kmh&fields=gb32960.gd_fc_stack.stack_water_outlet_temp_c'
|
||
```
|
||
|
||
808 源 IP：
|
||
|
||
```bash
|
||
journalctl -u lingniu-go-gateway.service --since '10 minutes ago' --no-pager
|
||
ss -tn sport = :808
|
||
```
|
||
|
||
最近已观察到 808 源 IP：
|
||
|
||
```text
|
||
115.159.85.149
|
||
222.66.200.68
|
||
122.152.221.156
|
||
115.231.168.135
|
||
```
|
||
|
||
## 部署步骤
|
||
|
||
本地构建：
|
||
|
||
```bash
|
||
cd go/vehicle-gateway
|
||
GOOS=linux GOARCH=amd64 CGO_ENABLED=0 go build -trimpath -ldflags='-s -w' -o /tmp/lingniu-go-deploy/gateway ./cmd/gateway
|
||
GOOS=linux GOARCH=amd64 CGO_ENABLED=0 go build -trimpath -ldflags='-s -w' -o /tmp/lingniu-go-deploy/history-writer ./cmd/history-writer
|
||
GOOS=linux GOARCH=amd64 CGO_ENABLED=0 go build -trimpath -ldflags='-s -w' -o /tmp/lingniu-go-deploy/realtime-api ./cmd/realtime-api
|
||
GOOS=linux GOARCH=amd64 CGO_ENABLED=0 go build -trimpath -ldflags='-s -w' -o /tmp/lingniu-go-deploy/stat-writer ./cmd/stat-writer
|
||
GOOS=linux GOARCH=amd64 CGO_ENABLED=0 go build -trimpath -ldflags='-s -w' -o /tmp/lingniu-go-deploy/nats-fast-writer ./cmd/nats-fast-writer
|
||
GOOS=linux GOARCH=amd64 CGO_ENABLED=0 go build -trimpath -ldflags='-s -w' -o /tmp/lingniu-go-deploy/nats-kafka-bridge ./cmd/nats-kafka-bridge
|
||
GOOS=linux GOARCH=amd64 CGO_ENABLED=0 go build -trimpath -ldflags='-s -w' -o /tmp/lingniu-go-deploy/capacity-check ./cmd/capacity-check
|
||
```
|
||
|
||
上传部署：
|
||
|
||
```bash
|
||
release="name-$(date +%Y%m%d%H%M%S)"
|
||
ssh root@115.29.187.205 "mkdir -p /opt/lingniu-go-native/releases/$release"
|
||
scp /tmp/lingniu-go-deploy/* root@115.29.187.205:/opt/lingniu-go-native/releases/$release/
|
||
ssh root@115.29.187.205 "chmod 755 /opt/lingniu-go-native/releases/$release/* && ln -sfn /opt/lingniu-go-native/releases/$release /opt/lingniu-go-native/current"
|
||
```
|
||
|
||
部署后验证：
|
||
|
||
```bash
|
||
ssh root@115.29.187.205 '
|
||
systemctl is-active lingniu-go-gateway.service lingniu-go-nats-fast-writer.service lingniu-go-nats-kafka-bridge.service lingniu-go-history-writer.service lingniu-go-stat-writer.service lingniu-go-realtime-api.service
|
||
for port in 20211 20212 20213 20214 20200; do curl -fsS "http://127.0.0.1:${port}/readyz"; echo; done
|
||
journalctl -u lingniu-go-gateway.service --since "2 minutes ago" --no-pager | grep -Ei "error|failed|panic|fatal" || true
|
||
'
|
||
```
|
||
|
||
## 2026-07-03 10W 接入优化部署记录
|
||
|
||
Release：
|
||
|
||
```text
|
||
/opt/lingniu-go-native/releases/100k-hardening-20260703175730
|
||
```
|
||
|
||
包含：
|
||
|
||
- Gateway 默认 `TCP_MAX_CONNECTIONS=120000`。
|
||
- 新增 `vehicle_gateway_connection_rejections_total`。
|
||
- 新增 `/opt/lingniu-go-native/current/load-sim` 运维压测工具。
|
||
- Redis-first realtime projector 增加 MySQL async queue dropped 回归测试。
|
||
|
||
ECS 已落地内核参数：
|
||
|
||
```text
|
||
net.core.somaxconn = 65535
|
||
net.ipv4.tcp_max_syn_backlog = 65535
|
||
net.ipv4.ip_local_port_range = 10000 65000
|
||
net.ipv4.tcp_tw_reuse = 1
|
||
```
|
||
|
||
注意：远端 `/etc/sysctl.conf` 原本有 `net.ipv4.tcp_max_syn_backlog = 1024`，已改为 `65535`，同时保留 `/etc/sysctl.d/zz-lingniu-vehicle-ingest.conf`。
|
||
|
||
部署后验证：
|
||
|
||
- `current` 指向 `100k-hardening-20260703175730`。
|
||
- `20211/20212/20213/20214/20200` readyz 均为 ok。
|
||
- `808`、`32960`、`20200`、`20211-20214` 监听 backlog 均显示 `65535`。
|
||
- Gateway 重启后持续接收 GB32960、JT808、Yutong MQTT 帧。
|
||
- 重启完成后窗口未发现 gateway/realtime-api 持续 error/fatal 日志。
|
||
|
||
### 10W 阶段压测进展
|
||
|
||
2026-07-03 已完成 ECS 本机 JT808 hold-only 阶段压测，不发送业务帧：
|
||
|
||
| 阶段 | 结果 |
|
||
| --- | --- |
|
||
| 100 连接 / 60s | opened `100`，failed `0`，frames `0`，write_errors `0` |
|
||
| 1,000 连接 / 60s | opened `1000`，failed `0`，frames `0`，write_errors `0` |
|
||
| 10,000 连接 / 120s | opened `10000`，failed `0`，frames `0`，write_errors `0` |
|
||
| 50,000 连接 / 180s | opened `50000`，failed `0`，frames `0`，write_errors `0` |
|
||
| 100,000 总连接 / 180s | JT808 opened `50000`、GB32960 opened `50000`，failed `0`，frames `0`，write_errors `0` |
|
||
|
||
1W 阶段峰值约 `10240` JT808 active connections，gateway FD 约 `10256`，NATS ack pending 为 `0`，压测后连接回落到生产基线约 `240`，readyz 全部 OK。
|
||
|
||
5W 阶段峰值约 `50238` JT808 active connections，gateway FD 约 `50254`，available memory 约 `5590 MB`。NATS ack pending 中间短时到 `29`，`18:16:49 CST` 回到 `0`；history/stat/realtime Kafka lag 总和均为 `0`。压测后 JT808 active 回落到生产基线约 `238-240`，readyz 全部 OK。
|
||
|
||
10W 总连接阶段使用双端口 loopback：JT808 `50000` + GB32960 `50000`，均 `-send=false`。峰值约 `100238` active connections，gateway FD 约 `100252`，available memory 约 `4511 MB`。NATS ack pending 中间短时到 `10`，`18:23:02 CST` 回到 `0`；history/stat/realtime Kafka lag 总和均为 `0`。压测后 JT808 回落到约 `236`，GB32960 回落到约 `2`，readyz 全部 OK。
|
||
|
||
### Gateway async publish 指标
|
||
|
||
2026-07-03 已部署 gateway async sink 指标到 ECS 当前 release：
|
||
|
||
```text
|
||
vehicle_async_sink_enqueue_total{sink,kind,status}
|
||
vehicle_async_sink_publish_total{sink,kind,status}
|
||
vehicle_async_sink_publish_duration_ms{sink,kind,status}
|
||
vehicle_async_sink_publish_duration_ms_histogram_bucket{sink,kind,status,le}
|
||
vehicle_async_sink_publish_duration_ms_histogram_count{sink,kind,status}
|
||
vehicle_async_sink_publish_duration_ms_histogram_sum{sink,kind,status}
|
||
vehicle_async_sink_queue_depth{sink}
|
||
```
|
||
|
||
部署后验证：
|
||
|
||
- `vehicle_async_sink_queue_depth{sink="nats"} 0`
|
||
- `vehicle_async_sink_enqueue_total{kind="raw",sink="nats",status="queued"}` 持续增长
|
||
- `vehicle_async_sink_publish_total{kind="raw",sink="nats",status="ok"}` 持续增长
|
||
- NATS bridge ack pending 为 `0`
|
||
- Gateway 启动后窗口无 `error|failed|panic|fatal` 日志
|
||
|
||
这些指标是进入带帧率压测前的关键保护栏：如果 `queue_depth` 持续增长、`enqueue_total{status="timeout"}` 增长，或 `vehicle_async_sink_publish_duration_ms_histogram_bucket` 的 p99 持续上升，说明入口 publish 队列或下游 NATS/Kafka 写入已成为瓶颈。
|
||
|
||
### Gateway connection close reason
|
||
|
||
2026-07-03 已为 TCP gateway 增加连接关闭原因计数：
|
||
|
||
```text
|
||
vehicle_gateway_connection_closes_total{protocol,reason}
|
||
```
|
||
|
||
当前 `reason` 包括 `eof`、`read_timeout`、`read_error`、`extract_error`、`context_cancelled`、`max_connections`。10W 长连接和带帧率压测时，应重点关注 `read_error`、`extract_error`、`max_connections` 是否异常增长；`read_timeout` 需要结合协议上报周期和在线数判断。
|
||
|
||
### NATS fast writer stage histogram
|
||
|
||
2026-07-03 已为 NATS fast writer 增加阶段耗时 histogram：
|
||
|
||
```text
|
||
vehicle_fast_writer_stage_duration_ms_histogram_bucket{subject,stage,status,le}
|
||
vehicle_fast_writer_stage_duration_ms_histogram_count{subject,stage,status}
|
||
vehicle_fast_writer_stage_duration_ms_histogram_sum{subject,stage,status}
|
||
```
|
||
|
||
`stage` 取值包括 `tdengine`、`redis`、`ack`，分别对应快速写 TDengine raw/location、Redis realtime 投影、NATS ack。带帧率压测时如果某个 stage 的 p99 持续上升，可以直接定位 fast path 是存储瓶颈还是 ack/网络瓶颈。
|
||
|
||
2026-07-03 后续优化：`nats-fast-writer` 已从“Fetch batch 后逐条写 TDengine/Redis”改为“Fetch batch 后先调用 `history.Writer.AppendAllBatch` 批量写 TDengine，再调用 `realtime.Repository.FastUpdateBatch` 用一次 Redis pipeline 写实时 KV/在线状态/索引，最后逐条 ack NATS”。这样可以同时减少 TDengine 和 Redis round trip；ack 仍按单条执行，避免存储失败时误 ack。
|
||
|
||
2026-07-03 后续优化：`nats-fast-writer` 的 TDengine SQL 连接池已从固定 `1` 改为可配置：
|
||
|
||
```text
|
||
FAST_WRITER_TDENGINE_MAX_OPEN_CONNS
|
||
FAST_WRITER_TDENGINE_MAX_IDLE_CONNS
|
||
```
|
||
|
||
默认 `max_open=1`、`max_idle=1`，生产可以通过 env 小步放大。此前实测默认跟随 worker 放大到 8 会导致 `[0x200] db is not specified` / `[0x2616] Database not specified`，根因是 TDengine `USE <database>` 不会自动作用到 `database/sql` 新建连接。2026-07-03 已将 fast-writer 使用的 history writer 改为生成 `database.table` 形式的 database-qualified SQL，并在 ECS 上将 pool 调到 `2/2` 验证通过：未再出现选库错误，写入 ok 计数持续增长。
|
||
|
||
同时新增 batch pending gauge：
|
||
|
||
```text
|
||
vehicle_fast_writer_nats_consumer_pending{stream,consumer}
|
||
vehicle_fast_writer_nats_consumer_ack_pending{stream,consumer}
|
||
vehicle_fast_writer_nats_consumer_waiting{stream,consumer}
|
||
vehicle_fast_writer_batch_pending_messages
|
||
vehicle_fast_writer_batch_pending_envelopes
|
||
```
|
||
|
||
`vehicle_fast_writer_nats_consumer_*` 表示 JetStream durable consumer 层面的 backlog/ack-pending；`vehicle_fast_writer_batch_pending_*` 表示 fast-writer 已拉取但尚未完成 TDengine/Redis/ack 的批内压力。带帧率压测时它们应在稳态接近 0，burst 后能够回落；如果持续非 0，需要结合 `vehicle_fast_writer_stage_duration_ms_histogram` 判断卡在 NATS 拉取、TDengine、Redis 还是 ack。
|
||
|
||
### NATS Kafka bridge batch 指标
|
||
|
||
2026-07-03 已为 NATS -> Kafka bridge 增加 batch pending 和 duration histogram：
|
||
|
||
```text
|
||
vehicle_bridge_batch_pending_messages
|
||
vehicle_bridge_batch_pending_kafka_messages
|
||
vehicle_bridge_batch_duration_ms_histogram_bucket{status,le}
|
||
vehicle_bridge_batch_duration_ms_histogram_count{status}
|
||
vehicle_bridge_batch_duration_ms_histogram_sum{status}
|
||
```
|
||
|
||
该耗时覆盖一批 NATS 消息路由到 Kafka、Kafka 同步写入、NATS ack 的完整过程。带帧率压测时，如果 pending 持续非 0 或 duration p99 持续上升，应优先排查 Kafka broker、bridge batch size、NATS ack-pending 和网络。
|
||
|
||
### Realtime store update histogram
|
||
|
||
2026-07-03 已为 realtime-api Redis/MySQL store update 增加 duration histogram：
|
||
|
||
```text
|
||
vehicle_realtime_store_update_duration_ms_histogram_bucket{store,protocol,status,le}
|
||
vehicle_realtime_store_update_duration_ms_histogram_count{store,protocol,status}
|
||
vehicle_realtime_store_update_duration_ms_histogram_sum{store,protocol,status}
|
||
```
|
||
|
||
该耗时覆盖 Redis 快速实时投影和 MySQL 异步当前态/位置投影的实际写入耗时。带帧率压测时，Redis p99 上升会直接影响 realtime consumer 追平；MySQL p99 上升需要结合 `vehicle_realtime_async_queue_depth` 和 `vehicle_realtime_async_queue_total{status="dropped"}` 判断是否开始丢弃低优先级当前态写入。
|
||
|
||
### Stat writer write histogram
|
||
|
||
2026-07-03 已为 stat-writer MySQL append 增加 duration histogram：
|
||
|
||
```text
|
||
vehicle_stat_write_duration_ms_histogram_bucket{topic,status,le}
|
||
vehicle_stat_write_duration_ms_histogram_count{topic,status}
|
||
vehicle_stat_write_duration_ms_histogram_sum{topic,status}
|
||
```
|
||
|
||
该耗时覆盖单条 Kafka 消息解析后写入 `vehicle_daily_mileage` 的 MySQL append 路径。带帧率压测时，如果该 p99 持续上升并伴随 `vehicle_stat_kafka_lag` 增长，优先排查 MySQL 写入、每日里程表索引和 stat-writer 消费能力。
|
||
|
||
### Gateway frame duration histogram
|
||
|
||
2026-07-03 已为 Gateway 帧处理耗时增加 histogram：
|
||
|
||
```text
|
||
vehicle_gateway_frame_duration_ms_histogram_bucket{protocol,status,le}
|
||
vehicle_gateway_frame_duration_ms_histogram_count{protocol,status}
|
||
vehicle_gateway_frame_duration_ms_histogram_sum{protocol,status}
|
||
```
|
||
|
||
该耗时覆盖单帧从解析、identity resolve、parsed fields 生成、publish enqueue 到协议响应写入的整体入口处理路径。带帧率压测时用它按协议观察 p95/p99，不能只看 `vehicle_gateway_frame_duration_ms` 最后一条 gauge。
|
||
|
||
### Gateway identity resolve duration
|
||
|
||
2026-07-11 已为 Gateway 身份解析单独增加耗时 histogram 和超时分类：
|
||
|
||
```text
|
||
vehicle_gateway_identity_total{protocol,status}
|
||
vehicle_gateway_identity_duration_ms_histogram_bucket{protocol,status,le}
|
||
vehicle_gateway_identity_duration_ms_histogram_count{protocol,status}
|
||
vehicle_gateway_identity_duration_ms_histogram_sum{protocol,status}
|
||
```
|
||
|
||
生产默认使用周期身份快照，每帧只查本机内存；`status="timeout"` 主要用于兼容关闭 `IDENTITY_SNAPSHOT_ONLY_ENABLED` 后的回源查询模式。快照模式应重点观察 `vehicle_gateway_identity_snapshot_ready` 和最近成功刷新时间；RDS 故障时入口继续接收，上一版快照继续服务，未知 phone 暂时 unresolved。
|
||
|
||
### History writer TDengine 批写
|
||
|
||
2026-07-03 已实现 history-writer 第一阶段批写：
|
||
|
||
- `history.Writer.AppendAllBatch` 按 TDengine child table 聚合 raw/location 多行 `INSERT`。
|
||
- `cmd/history-writer` 默认 `HISTORY_WORKERS=3`、`HISTORY_BATCH_SIZE=200`、`HISTORY_BATCH_WAIT_MS=20`。
|
||
- TDengine batch 成功后才批量提交 Kafka messages；失败不提交 offset。
|
||
- 保留单条 `AppendAll` 路径作为回退。
|
||
- `production-track-telemetry-20260714082451` 起，`vehicle_locations` stable 增加 `soc_percent`。history-writer 在建表后执行幂等 `ALTER STABLE ... ADD COLUMN soc_percent DOUBLE`，只忽略 TDengine 明确的重复列错误；其他迁移错误会阻止 ready，禁止带病继续写入。单条和批量位置 INSERT 都必须携带 SOC。
|
||
|
||
新增指标：
|
||
|
||
```text
|
||
vehicle_history_batch_flush_total{status}
|
||
vehicle_history_batch_rows_total{status}
|
||
vehicle_history_batch_pending_messages
|
||
vehicle_history_batch_pending_rows
|
||
vehicle_history_batch_flush_duration_ms{status}
|
||
```
|
||
|
||
进入带帧率压测时，必须同时观察 `vehicle_history_kafka_lag`、`vehicle_history_batch_pending_*` 和 `vehicle_history_batch_*`。如果 pending 持续非 0 但 Kafka lag 尚未放大，说明 history-writer 已经在本地批写或提交阶段形成早期积压。
|
||
|
||
## 下次恢复上下文时先做
|
||
|
||
1. 读本文件。
|
||
2. 读 `docs/ops/vehicle-ingest-runbook.md`。
|
||
3. 读 `docs/architecture/production-data-plane-inventory.md`。
|
||
4. 执行 `git status --short`。
|
||
5. 如果涉及生产，先查 ECS 服务健康和最近日志。
|