diff --git a/docs/architecture/iot-data-platform-principles.md b/docs/architecture/iot-data-platform-principles.md index 24c87ba0..b70eec5a 100644 --- a/docs/architecture/iot-data-platform-principles.md +++ b/docs/architecture/iot-data-platform-principles.md @@ -41,7 +41,7 @@ flowchart LR ## Minimal Storage Contract -详细的落库边界见 [车辆数据最小落库合约](storage-minimal-contract.md)。 +详细的落库边界见 [车辆数据最小落库合约](storage-minimal-contract.md),当前生产数据面见 [生产数据面清单](production-data-plane-inventory.md)。 | Store | Table or key | Purpose | Keep | Avoid | | --- | --- | --- | --- | --- | @@ -52,7 +52,8 @@ flowchart LR | MySQL | `vehicle_daily_mileage` | Queryable daily mileage | vin or vehicle key, date, protocol, daily mileage, source mileage | generic metric key/value rows, per-frame raw details | | MySQL | `vehicle_identity_binding` | Manual identity mapping | vin, plate, phone, device id | registration history | | MySQL | `jt808_registration` | JT808 registration and auth trace | phone, device id, plate, auth code, vin match state, first/latest seen | GB32960 or MQTT records | -| Redis | `realtime-raw:{protocol}:{vin}` | Latest full realtime protocol state | merged latest protocol payload | historical data | +| Redis | `vehicle:latest:{vehicleKey}` | Latest merged realtime state | cross-protocol latest fields | historical data | +| Redis | `vehicle:realtime-raw:{protocol}:{vehicleKey}` | Latest full realtime protocol state | latest protocol parsed payload | historical data | ## Event Envelope Rules @@ -77,10 +78,11 @@ Every received frame should become one `FrameEnvelope`. ## Current Next Steps -1. Add health and readiness endpoints to each long-running Go service. -2. Add runtime metrics for protocol frame counts, parse failures, Kafka/NATS publish failures, and writer lag. -3. Review TDengine tables and remove unused tables only after export or explicit confirmation. -4. Keep new business tables narrow by default. Add a column only when a query or index proves it is needed. +1. Keep production inventory current whenever a service, topic, table, or Redis key family changes. +2. Add parser correctness tests before changing GB32960, JT808, or Yutong MQTT field extraction. +3. Add lag and write-failure alerts from the existing `/metrics` endpoints. +4. Isolate or delete legacy Kafka topics only after confirming no external consumers still depend on them. +5. Keep new business tables narrow by default. Add a column only when a query or index proves it is needed. ## Runtime Metrics Baseline diff --git a/docs/architecture/production-data-plane-inventory.md b/docs/architecture/production-data-plane-inventory.md new file mode 100644 index 00000000..67031106 --- /dev/null +++ b/docs/architecture/production-data-plane-inventory.md @@ -0,0 +1,99 @@ +# 生产数据面清单 + +审计时间:2026-07-02 + +本文记录当前 ECS 上 Go 版本车辆接入链路的真实数据面,用来约束后续重构:新增表、topic、key 之前先对照这里,避免把已经删除的历史设计重新带回来。 + +## 运行服务 + +生产接入 ECS:`115.29.187.205` + +| 服务 | systemd 单元 | 端口 | 职责 | +| --- | --- | --- | --- | +| Gateway | `lingniu-go-gateway.service` | `0.0.0.0:32960`、`0.0.0.0:808`、`127.0.0.1:20211` | GB32960、JT808、宇通 MQTT 接入和协议解析 | +| NATS Kafka bridge | `lingniu-go-nats-kafka-bridge.service` | `127.0.0.1:20214` | NATS JetStream 到 Kafka 的可靠桥接 | +| History writer | `lingniu-go-history-writer.service` | `127.0.0.1:20212` | Kafka raw 消费,写 TDengine | +| Stat writer | `lingniu-go-stat-writer.service` | `127.0.0.1:20213` | Kafka raw 消费,写每日里程 | +| Realtime API/projector | `lingniu-go-realtime-api.service` | `0.0.0.0:20200` | Kafka raw 消费,写 Redis/MySQL 实时表,并提供 API | + +## 总线 + +### NATS JetStream + +NATS 部署在 Kafka ECS 内网 `172.17.111.56:4222`。 + +| 项 | 当前值 | +| --- | --- | +| Stream | `VEHICLE_INGEST` | +| Subjects | `vehicle.raw.go.gb32960.v1`、`vehicle.raw.go.jt808.v1`、`vehicle.raw.go.yutong-mqtt.v1`、`vehicle.event.go.unified.v1` | +| Durable consumer | `vehicle-kafka-bridge` | +| 语义 | Gateway 先写 NATS;bridge 写 Kafka 成功后才 ACK NATS | + +### Kafka + +当前 Go 链路正在使用的 topic: + +| Topic | 分区 | 写入方 | 主要消费方 | +| --- | --- | --- | --- | +| `vehicle.raw.go.gb32960.v1` | 12 | NATS Kafka bridge | history、stat、realtime | +| `vehicle.raw.go.jt808.v1` | 12 | NATS Kafka bridge | history、stat、realtime | +| `vehicle.raw.go.yutong-mqtt.v1` | 12 | NATS Kafka bridge | history、stat、realtime | +| `vehicle.event.go.unified.v1` | 12 | NATS Kafka bridge | 预留给统一事件消费 | + +审计时 Kafka 上仍存在旧 Java/Xinda/telemetry topic,例如 `vehicle.raw.gb32960.v1`、`vehicle.raw.jt808.v1`、`vehicle.event.xinda.v1`、`vehicle.raw.telemetry-input.v1`。这些不属于当前 Go 最小链路,后续如果确认没有消费者依赖,应单独做 Kafka topic 下线清单,不在业务代码里继续引用。 + +## TDengine + +数据库:`lingniu_vehicle_ts` + +| Stable | 主列 | Tags | 职责 | +| --- | --- | --- | --- | +| `raw_frames` | `ts`、`frame_id`、`event_id`、`message_id`、`event_time`、`received_at`、`raw_size_bytes`、`raw_hex`、`raw_text`、`parsed_json`、`parse_status`、`parse_error`、`source_endpoint` | `protocol`、`vehicle_key`、`vin`、`phone`、`device_id` | RAW 证据层,保存完整解析 JSON | +| `raw_frame_payload_chunks` | 分片 payload 列 | 协议和车辆标识 tags | 保存超长 raw/parsed payload | +| `vehicle_locations` | `ts`、`event_id`、`frame_id`、`received_at`、`longitude`、`latitude`、`altitude_m`、`speed_kmh`、`direction_deg`、`alarm_flag`、`status_flag`、`total_mileage_km` | `protocol`、`vehicle_key`、`vin`、`phone`、`device_id` | 高频历史位置和总里程查询 | + +不应重新出现: + +- `vehicle_mileage_points` +- `raw_frames.fields_json` + +## MySQL + +数据库:`lingniu_vehicle_data` + +| 表 | 核心字段 | 写入方 | 职责 | +| --- | --- | --- | --- | +| `vehicle_identity_binding` | `vin`、`plate`、`phone`、`device_id` | 导入/人工维护,Gateway 读取 | 通过车牌、手机号、设备号反查 VIN | +| `jt808_registration` | `phone`、`device_id`、`plate`、`vin`、`manufacturer`、`auth_token`、`source_endpoint`、首次/最新注册鉴权时间 | Gateway | JT808 注册、鉴权和 VIN 匹配状态 | +| `vehicle_realtime_snapshot` | `protocol`、`vin`、`plate`、`event_time`、`received_at`、`event_id` | Realtime API/projector | 每协议每 VIN 最新事件轻量快照 | +| `vehicle_realtime_location` | `protocol`、`vin`、`plate`、经纬度、速度、总里程、SOC、事件时间 | Realtime API/projector | 每协议每 VIN 最新位置业务缓存 | +| `vehicle_daily_mileage` | `vehicle_key`、`vin`、`stat_date`、`protocol`、`daily_mileage_km`、首末总里程、样本数 | Stat writer | 基于总里程差值的每日里程 | + +不应重新出现: + +- `vehicle_daily_metric` +- `vehicle_identity_binding_registration` +- `vehicle_identity_bindings` + +## Redis + +Redis 使用 DB 50,定位为实时缓存,不作为历史事实来源。 + +| Key 族 | 用途 | +| --- | --- | +| `vehicle:latest:{vehicleKey}` | 跨协议合并后的最新实时快照 | +| `vehicle:latest:{vehicleKey}:{protocol}` | 单协议最新快照 | +| `vehicle:realtime-raw:{protocol}:{vehicleKey}` | 单协议最新完整 parsed 状态 | +| `vehicle:online:{vehicleKey}` | 在线状态和 TTL | +| `vehicle:protocols:{vehicleKey}` | 当前车辆最近出现过的协议集合 | +| `vehicle:last_seen` | 最近活跃车辆排序集合 | + +审计时活跃 key 族主要是 `vehicle:latest:*`、`vehicle:realtime-raw:*`、`vehicle:online:*`、`vehicle:protocols:*`、`vehicle:last_seen`。旧文档中的 `vehicle:realtime:*`、`vehicle:merged:*` 不是当前活跃 key 族。 + +## 后续优化约束 + +1. 接入层先写 NATS,Kafka 是持久回放层,下游消费者不能反向依赖 Gateway 内存状态。 +2. TDengine 只放高写入时序数据:RAW 证据和位置历史。 +3. MySQL 只放低基数业务状态:身份、实时轻量快照、每日里程。 +4. Redis 只放当前态,所有 key 都必须允许 TTL 过期后从 Kafka/TDengine/MySQL 重建。 +5. 协议新增字段默认进入 `raw_frames.parsed_json` 和 Redis realtime-raw;只有稳定查询需求出现后才提升为 TDengine/MySQL 列。 diff --git a/docs/go-version-data-flow.html b/docs/go-version-data-flow.html index f379fb34..de2b6598 100644 --- a/docs/go-version-data-flow.html +++ b/docs/go-version-data-flow.html @@ -559,7 +559,7 @@ TDengine raw_frames history-writer - 原始报文、完整 parsed JSON、fields JSON、parse status、source endpoint + 原始报文、完整 parsed JSON、parse status、source endpoint RAW 查询、追溯、排查解析问题 @@ -576,30 +576,23 @@ 经纬度、速度、方向、状态、报警、总里程等核心字段 高频历史位置分页查询 - - TDengine - vehicle_mileage_points - history-writer - 累计里程点、速度、经纬度 - 里程曲线、区间差值分析 - Redis - vehicle:realtime:{vehicle_key} + vehicle:latest:{vehicle_key} realtime-api Kafka consumer 跨协议合并后的实时字段快照 查 VIN 是否在线、查 VIN 实时数据 Redis - vehicle:realtime:{vehicle_key}:{protocol} + vehicle:latest:{vehicle_key}:{protocol} realtime-api Kafka consumer 单协议实时快照 区分 32960/808/MQTT 的实时状态 Redis - vehicle:realtime-raw:{vehicle_key}:{protocol} + vehicle:realtime-raw:{protocol}:{vehicle_key} realtime-api Kafka consumer 单协议最新 parsed 全量字段 实时 RAW 字段查看,不走 TDengine 历史扫描 @@ -627,9 +620,9 @@ MySQL - vehicle_daily_metric + vehicle_daily_mileage stat-writer - 每日里程等指标,按总里程差值计算 + 每日里程,按首末总里程差值计算,保留样本数 统计查询 @@ -657,8 +650,8 @@ RAW 帧查询TDengine raw_frames + chunks按协议、VIN/phone、时间、消息类型分页。 位置历史TDengine vehicle_locations高频位置分页查询,避免每次扫完整 JSON。 - 里程点TDengine vehicle_mileage_points区间里程、曲线、异常总里程排查。 - 每日指标MySQL vehicle_daily_metric按日期、协议、指标查询统计结果。 + 里程历史TDengine vehicle_locations从位置核心表读取总里程点,不再单独维护里程点表。 + 每日里程MySQL vehicle_daily_mileage按日期、协议查询首末总里程差值结果。 @@ -700,7 +693,7 @@ TDengine ECS TDengine 内网 172.17.111.57:6041 - 历史 RAW、位置、里程点时序存储。 + 历史 RAW、位置时序存储。 云服务 @@ -729,7 +722,7 @@
实时多协议合并
- Redis 同时保留 merged snapshot、protocol snapshot、realtime-raw,既能看统一实时,也能追单协议原始字段。 + Redis 同时保留 latest snapshot、protocol snapshot、realtime-raw,既能看统一实时,也能追单协议原始字段。
身份解析降级
diff --git a/docs/ops/vehicle-ingest-runbook.md b/docs/ops/vehicle-ingest-runbook.md index 9068c524..b9e26ae0 100644 --- a/docs/ops/vehicle-ingest-runbook.md +++ b/docs/ops/vehicle-ingest-runbook.md @@ -13,6 +13,8 @@ 运行手册的目标是按自上而下的顺序定位问题:入口、队列、桥接、消费、存储、查询 API。 +当前生产服务、topic、表和 Redis key 的清单见 [生产数据面清单](../architecture/production-data-plane-inventory.md)。 + ## 服务地图 | 层级 | 服务 | systemd 单元 | 本机端点 |