docs: add production data plane inventory
This commit is contained in:
lingniu
@@ -41,7 +41,7 @@ flowchart LR
|
||||
|
||||
## Minimal Storage Contract
|
||||
|
||||
详细的落库边界见 [车辆数据最小落库合约](storage-minimal-contract.md)。
|
||||
详细的落库边界见 [车辆数据最小落库合约](storage-minimal-contract.md)，当前生产数据面见 [生产数据面清单](production-data-plane-inventory.md)。
|
||||
|
||||
| Store | Table or key | Purpose | Keep | Avoid |
|
||||
| --- | --- | --- | --- | --- |
|
||||
@@ -52,7 +52,8 @@ flowchart LR
|
||||
| MySQL | `vehicle_daily_mileage` | Queryable daily mileage | vin or vehicle key, date, protocol, daily mileage, source mileage | generic metric key/value rows, per-frame raw details |
|
||||
| MySQL | `vehicle_identity_binding` | Manual identity mapping | vin, plate, phone, device id | registration history |
|
||||
| MySQL | `jt808_registration` | JT808 registration and auth trace | phone, device id, plate, auth code, vin match state, first/latest seen | GB32960 or MQTT records |
|
||||
| Redis | `realtime-raw:{protocol}:{vin}` | Latest full realtime protocol state | merged latest protocol payload | historical data |
|
||||
| Redis | `vehicle:latest:{vehicleKey}` | Latest merged realtime state | cross-protocol latest fields | historical data |
|
||||
| Redis | `vehicle:realtime-raw:{protocol}:{vehicleKey}` | Latest full realtime protocol state | latest protocol parsed payload | historical data |
|
||||
|
||||
## Event Envelope Rules
|
||||
|
||||
@@ -77,10 +78,11 @@ Every received frame should become one `FrameEnvelope`.
|
||||
|
||||
## Current Next Steps
|
||||
|
||||
1. Add health and readiness endpoints to each long-running Go service.
|
||||
2. Add runtime metrics for protocol frame counts, parse failures, Kafka/NATS publish failures, and writer lag.
|
||||
3. Review TDengine tables and remove unused tables only after export or explicit confirmation.
|
||||
4. Keep new business tables narrow by default. Add a column only when a query or index proves it is needed.
|
||||
1. Keep production inventory current whenever a service, topic, table, or Redis key family changes.
|
||||
2. Add parser correctness tests before changing GB32960, JT808, or Yutong MQTT field extraction.
|
||||
3. Add lag and write-failure alerts from the existing `/metrics` endpoints.
|
||||
4. Isolate or delete legacy Kafka topics only after confirming no external consumers still depend on them.
|
||||
5. Keep new business tables narrow by default. Add a column only when a query or index proves it is needed.
|
||||
|
||||
## Runtime Metrics Baseline
|
||||
|
||||
|
||||
99
docs/architecture/production-data-plane-inventory.md
Normal file
99
docs/architecture/production-data-plane-inventory.md
Normal file
@@ -0,0 +1,99 @@
|
||||
# 生产数据面清单
|
||||
|
||||
审计时间：2026-07-02
|
||||
|
||||
本文记录当前 ECS 上 Go 版本车辆接入链路的真实数据面，用来约束后续重构：新增表、topic、key 之前先对照这里，避免把已经删除的历史设计重新带回来。
|
||||
|
||||
## 运行服务
|
||||
|
||||
生产接入 ECS：`115.29.187.205`
|
||||
|
||||
| 服务 | systemd 单元 | 端口 | 职责 |
|
||||
| --- | --- | --- | --- |
|
||||
| Gateway | `lingniu-go-gateway.service` | `0.0.0.0:32960`、`0.0.0.0:808`、`127.0.0.1:20211` | GB32960、JT808、宇通 MQTT 接入和协议解析 |
|
||||
| NATS Kafka bridge | `lingniu-go-nats-kafka-bridge.service` | `127.0.0.1:20214` | NATS JetStream 到 Kafka 的可靠桥接 |
|
||||
| History writer | `lingniu-go-history-writer.service` | `127.0.0.1:20212` | Kafka raw 消费，写 TDengine |
|
||||
| Stat writer | `lingniu-go-stat-writer.service` | `127.0.0.1:20213` | Kafka raw 消费，写每日里程 |
|
||||
| Realtime API/projector | `lingniu-go-realtime-api.service` | `0.0.0.0:20200` | Kafka raw 消费，写 Redis/MySQL 实时表，并提供 API |
|
||||
|
||||
## 总线
|
||||
|
||||
### NATS JetStream
|
||||
|
||||
NATS 部署在 Kafka ECS 内网 `172.17.111.56:4222`。
|
||||
|
||||
| 项 | 当前值 |
|
||||
| --- | --- |
|
||||
| Stream | `VEHICLE_INGEST` |
|
||||
| Subjects | `vehicle.raw.go.gb32960.v1`、`vehicle.raw.go.jt808.v1`、`vehicle.raw.go.yutong-mqtt.v1`、`vehicle.event.go.unified.v1` |
|
||||
| Durable consumer | `vehicle-kafka-bridge` |
|
||||
| 语义 | Gateway 先写 NATS；bridge 写 Kafka 成功后才 ACK NATS |
|
||||
|
||||
### Kafka
|
||||
|
||||
当前 Go 链路正在使用的 topic：
|
||||
|
||||
| Topic | 分区 | 写入方 | 主要消费方 |
|
||||
| --- | --- | --- | --- |
|
||||
| `vehicle.raw.go.gb32960.v1` | 12 | NATS Kafka bridge | history、stat、realtime |
|
||||
| `vehicle.raw.go.jt808.v1` | 12 | NATS Kafka bridge | history、stat、realtime |
|
||||
| `vehicle.raw.go.yutong-mqtt.v1` | 12 | NATS Kafka bridge | history、stat、realtime |
|
||||
| `vehicle.event.go.unified.v1` | 12 | NATS Kafka bridge | 预留给统一事件消费 |
|
||||
|
||||
审计时 Kafka 上仍存在旧 Java/Xinda/telemetry topic，例如 `vehicle.raw.gb32960.v1`、`vehicle.raw.jt808.v1`、`vehicle.event.xinda.v1`、`vehicle.raw.telemetry-input.v1`。这些不属于当前 Go 最小链路，后续如果确认没有消费者依赖，应单独做 Kafka topic 下线清单，不在业务代码里继续引用。
|
||||
|
||||
## TDengine
|
||||
|
||||
数据库：`lingniu_vehicle_ts`
|
||||
|
||||
| Stable | 主列 | Tags | 职责 |
|
||||
| --- | --- | --- | --- |
|
||||
| `raw_frames` | `ts`、`frame_id`、`event_id`、`message_id`、`event_time`、`received_at`、`raw_size_bytes`、`raw_hex`、`raw_text`、`parsed_json`、`parse_status`、`parse_error`、`source_endpoint` | `protocol`、`vehicle_key`、`vin`、`phone`、`device_id` | RAW 证据层，保存完整解析 JSON |
|
||||
| `raw_frame_payload_chunks` | 分片 payload 列 | 协议和车辆标识 tags | 保存超长 raw/parsed payload |
|
||||
| `vehicle_locations` | `ts`、`event_id`、`frame_id`、`received_at`、`longitude`、`latitude`、`altitude_m`、`speed_kmh`、`direction_deg`、`alarm_flag`、`status_flag`、`total_mileage_km` | `protocol`、`vehicle_key`、`vin`、`phone`、`device_id` | 高频历史位置和总里程查询 |
|
||||
|
||||
不应重新出现：
|
||||
|
||||
- `vehicle_mileage_points`
|
||||
- `raw_frames.fields_json`
|
||||
|
||||
## MySQL
|
||||
|
||||
数据库：`lingniu_vehicle_data`
|
||||
|
||||
| 表 | 核心字段 | 写入方 | 职责 |
|
||||
| --- | --- | --- | --- |
|
||||
| `vehicle_identity_binding` | `vin`、`plate`、`phone`、`device_id` | 导入/人工维护，Gateway 读取 | 通过车牌、手机号、设备号反查 VIN |
|
||||
| `jt808_registration` | `phone`、`device_id`、`plate`、`vin`、`manufacturer`、`auth_token`、`source_endpoint`、首次/最新注册鉴权时间 | Gateway | JT808 注册、鉴权和 VIN 匹配状态 |
|
||||
| `vehicle_realtime_snapshot` | `protocol`、`vin`、`plate`、`event_time`、`received_at`、`event_id` | Realtime API/projector | 每协议每 VIN 最新事件轻量快照 |
|
||||
| `vehicle_realtime_location` | `protocol`、`vin`、`plate`、经纬度、速度、总里程、SOC、事件时间 | Realtime API/projector | 每协议每 VIN 最新位置业务缓存 |
|
||||
| `vehicle_daily_mileage` | `vehicle_key`、`vin`、`stat_date`、`protocol`、`daily_mileage_km`、首末总里程、样本数 | Stat writer | 基于总里程差值的每日里程 |
|
||||
|
||||
不应重新出现：
|
||||
|
||||
- `vehicle_daily_metric`
|
||||
- `vehicle_identity_binding_registration`
|
||||
- `vehicle_identity_bindings`
|
||||
|
||||
## Redis
|
||||
|
||||
Redis 使用 DB 50，定位为实时缓存，不作为历史事实来源。
|
||||
|
||||
| Key 族 | 用途 |
|
||||
| --- | --- |
|
||||
| `vehicle:latest:{vehicleKey}` | 跨协议合并后的最新实时快照 |
|
||||
| `vehicle:latest:{vehicleKey}:{protocol}` | 单协议最新快照 |
|
||||
| `vehicle:realtime-raw:{protocol}:{vehicleKey}` | 单协议最新完整 parsed 状态 |
|
||||
| `vehicle:online:{vehicleKey}` | 在线状态和 TTL |
|
||||
| `vehicle:protocols:{vehicleKey}` | 当前车辆最近出现过的协议集合 |
|
||||
| `vehicle:last_seen` | 最近活跃车辆排序集合 |
|
||||
|
||||
审计时活跃 key 族主要是 `vehicle:latest:*`、`vehicle:realtime-raw:*`、`vehicle:online:*`、`vehicle:protocols:*`、`vehicle:last_seen`。旧文档中的 `vehicle:realtime:*`、`vehicle:merged:*` 不是当前活跃 key 族。
|
||||
|
||||
## 后续优化约束
|
||||
|
||||
1. 接入层先写 NATS，Kafka 是持久回放层，下游消费者不能反向依赖 Gateway 内存状态。
|
||||
2. TDengine 只放高写入时序数据：RAW 证据和位置历史。
|
||||
3. MySQL 只放低基数业务状态：身份、实时轻量快照、每日里程。
|
||||
4. Redis 只放当前态，所有 key 都必须允许 TTL 过期后从 Kafka/TDengine/MySQL 重建。
|
||||
5. 协议新增字段默认进入 `raw_frames.parsed_json` 和 Redis realtime-raw；只有稳定查询需求出现后才提升为 TDengine/MySQL 列。
|
||||
Reference in New Issue
Block a user