docs: add production data plane inventory

2026-07-02 20:14:56 +08:00
parent 3c9678491a
commit 3248336bea
4 changed files with 119 additions and 23 deletions
--- a/docs/architecture/iot-data-platform-principles.md
+++ b/docs/architecture/iot-data-platform-principles.md
@@ -41,7 +41,7 @@ flowchart LR

 ## Minimal Storage Contract

-详细的落库边界见 [车辆数据最小落库合约](storage-minimal-contract.md)。
+详细的落库边界见 [车辆数据最小落库合约](storage-minimal-contract.md)，当前生产数据面见 [生产数据面清单](production-data-plane-inventory.md)。

 | Store | Table or key | Purpose | Keep | Avoid |
 | --- | --- | --- | --- | --- |
@@ -52,7 +52,8 @@ flowchart LR
 | MySQL | `vehicle_daily_mileage` | Queryable daily mileage | vin or vehicle key, date, protocol, daily mileage, source mileage | generic metric key/value rows, per-frame raw details |
 | MySQL | `vehicle_identity_binding` | Manual identity mapping | vin, plate, phone, device id | registration history |
 | MySQL | `jt808_registration` | JT808 registration and auth trace | phone, device id, plate, auth code, vin match state, first/latest seen | GB32960 or MQTT records |
-| Redis | `realtime-raw:{protocol}:{vin}` | Latest full realtime protocol state | merged latest protocol payload | historical data |
+| Redis | `vehicle:latest:{vehicleKey}` | Latest merged realtime state | cross-protocol latest fields | historical data |
+| Redis | `vehicle:realtime-raw:{protocol}:{vehicleKey}` | Latest full realtime protocol state | latest protocol parsed payload | historical data |

 ## Event Envelope Rules

@@ -77,10 +78,11 @@ Every received frame should become one `FrameEnvelope`.

 ## Current Next Steps

-1. Add health and readiness endpoints to each long-running Go service.
-2. Add runtime metrics for protocol frame counts, parse failures, Kafka/NATS publish failures, and writer lag.
-3. Review TDengine tables and remove unused tables only after export or explicit confirmation.
-4. Keep new business tables narrow by default. Add a column only when a query or index proves it is needed.
+1. Keep production inventory current whenever a service, topic, table, or Redis key family changes.
+2. Add parser correctness tests before changing GB32960, JT808, or Yutong MQTT field extraction.
+3. Add lag and write-failure alerts from the existing `/metrics` endpoints.
+4. Isolate or delete legacy Kafka topics only after confirming no external consumers still depend on them.
+5. Keep new business tables narrow by default. Add a column only when a query or index proves it is needed.

 ## Runtime Metrics Baseline

--- a/docs/architecture/production-data-plane-inventory.md
+++ b/docs/architecture/production-data-plane-inventory.md
@@ -0,0 +1,99 @@
+# 生产数据面清单
+
+审计时间：2026-07-02
+
+本文记录当前 ECS 上 Go 版本车辆接入链路的真实数据面，用来约束后续重构：新增表、topic、key 之前先对照这里，避免把已经删除的历史设计重新带回来。
+
+## 运行服务
+
+生产接入 ECS：`115.29.187.205`
+
+| 服务 | systemd 单元 | 端口 | 职责 |
+| --- | --- | --- | --- |
+| Gateway | `lingniu-go-gateway.service` | `0.0.0.0:32960`、`0.0.0.0:808`、`127.0.0.1:20211` | GB32960、JT808、宇通 MQTT 接入和协议解析 |
+| NATS Kafka bridge | `lingniu-go-nats-kafka-bridge.service` | `127.0.0.1:20214` | NATS JetStream 到 Kafka 的可靠桥接 |
+| History writer | `lingniu-go-history-writer.service` | `127.0.0.1:20212` | Kafka raw 消费，写 TDengine |
+| Stat writer | `lingniu-go-stat-writer.service` | `127.0.0.1:20213` | Kafka raw 消费，写每日里程 |
+| Realtime API/projector | `lingniu-go-realtime-api.service` | `0.0.0.0:20200` | Kafka raw 消费，写 Redis/MySQL 实时表，并提供 API |
+
+## 总线
+
+### NATS JetStream
+
+NATS 部署在 Kafka ECS 内网 `172.17.111.56:4222`。
+
+| 项 | 当前值 |
+| --- | --- |
+| Stream | `VEHICLE_INGEST` |
+| Subjects | `vehicle.raw.go.gb32960.v1`、`vehicle.raw.go.jt808.v1`、`vehicle.raw.go.yutong-mqtt.v1`、`vehicle.event.go.unified.v1` |
+| Durable consumer | `vehicle-kafka-bridge` |
+| 语义 | Gateway 先写 NATS；bridge 写 Kafka 成功后才 ACK NATS |
+
+### Kafka
+
+当前 Go 链路正在使用的 topic：
+
+| Topic | 分区 | 写入方 | 主要消费方 |
+| --- | --- | --- | --- |
+| `vehicle.raw.go.gb32960.v1` | 12 | NATS Kafka bridge | history、stat、realtime |
+| `vehicle.raw.go.jt808.v1` | 12 | NATS Kafka bridge | history、stat、realtime |
+| `vehicle.raw.go.yutong-mqtt.v1` | 12 | NATS Kafka bridge | history、stat、realtime |
+| `vehicle.event.go.unified.v1` | 12 | NATS Kafka bridge | 预留给统一事件消费 |
+
+审计时 Kafka 上仍存在旧 Java/Xinda/telemetry topic，例如 `vehicle.raw.gb32960.v1`、`vehicle.raw.jt808.v1`、`vehicle.event.xinda.v1`、`vehicle.raw.telemetry-input.v1`。这些不属于当前 Go 最小链路，后续如果确认没有消费者依赖，应单独做 Kafka topic 下线清单，不在业务代码里继续引用。
+
+## TDengine
+
+数据库：`lingniu_vehicle_ts`
+
+| Stable | 主列 | Tags | 职责 |
+| --- | --- | --- | --- |
+| `raw_frames` | `ts`、`frame_id`、`event_id`、`message_id`、`event_time`、`received_at`、`raw_size_bytes`、`raw_hex`、`raw_text`、`parsed_json`、`parse_status`、`parse_error`、`source_endpoint` | `protocol`、`vehicle_key`、`vin`、`phone`、`device_id` | RAW 证据层，保存完整解析 JSON |
+| `raw_frame_payload_chunks` | 分片 payload 列 | 协议和车辆标识 tags | 保存超长 raw/parsed payload |
+| `vehicle_locations` | `ts`、`event_id`、`frame_id`、`received_at`、`longitude`、`latitude`、`altitude_m`、`speed_kmh`、`direction_deg`、`alarm_flag`、`status_flag`、`total_mileage_km` | `protocol`、`vehicle_key`、`vin`、`phone`、`device_id` | 高频历史位置和总里程查询 |
+
+不应重新出现：
+
+- `vehicle_mileage_points`
+- `raw_frames.fields_json`
+
+## MySQL
+
+数据库：`lingniu_vehicle_data`
+
+| 表 | 核心字段 | 写入方 | 职责 |
+| --- | --- | --- | --- |
+| `vehicle_identity_binding` | `vin`、`plate`、`phone`、`device_id` | 导入/人工维护，Gateway 读取 | 通过车牌、手机号、设备号反查 VIN |
+| `jt808_registration` | `phone`、`device_id`、`plate`、`vin`、`manufacturer`、`auth_token`、`source_endpoint`、首次/最新注册鉴权时间 | Gateway | JT808 注册、鉴权和 VIN 匹配状态 |
+| `vehicle_realtime_snapshot` | `protocol`、`vin`、`plate`、`event_time`、`received_at`、`event_id` | Realtime API/projector | 每协议每 VIN 最新事件轻量快照 |
+| `vehicle_realtime_location` | `protocol`、`vin`、`plate`、经纬度、速度、总里程、SOC、事件时间 | Realtime API/projector | 每协议每 VIN 最新位置业务缓存 |
+| `vehicle_daily_mileage` | `vehicle_key`、`vin`、`stat_date`、`protocol`、`daily_mileage_km`、首末总里程、样本数 | Stat writer | 基于总里程差值的每日里程 |
+
+不应重新出现：
+
+- `vehicle_daily_metric`
+- `vehicle_identity_binding_registration`
+- `vehicle_identity_bindings`
+
+## Redis
+
+Redis 使用 DB 50，定位为实时缓存，不作为历史事实来源。
+
+| Key 族 | 用途 |
+| --- | --- |
+| `vehicle:latest:{vehicleKey}` | 跨协议合并后的最新实时快照 |
+| `vehicle:latest:{vehicleKey}:{protocol}` | 单协议最新快照 |
+| `vehicle:realtime-raw:{protocol}:{vehicleKey}` | 单协议最新完整 parsed 状态 |
+| `vehicle:online:{vehicleKey}` | 在线状态和 TTL |
+| `vehicle:protocols:{vehicleKey}` | 当前车辆最近出现过的协议集合 |
+| `vehicle:last_seen` | 最近活跃车辆排序集合 |
+
+审计时活跃 key 族主要是 `vehicle:latest:*`、`vehicle:realtime-raw:*`、`vehicle:online:*`、`vehicle:protocols:*`、`vehicle:last_seen`。旧文档中的 `vehicle:realtime:*`、`vehicle:merged:*` 不是当前活跃 key 族。
+
+## 后续优化约束
+
+1. 接入层先写 NATS，Kafka 是持久回放层，下游消费者不能反向依赖 Gateway 内存状态。
+2. TDengine 只放高写入时序数据：RAW 证据和位置历史。
+3. MySQL 只放低基数业务状态：身份、实时轻量快照、每日里程。
+4. Redis 只放当前态，所有 key 都必须允许 TTL 过期后从 Kafka/TDengine/MySQL 重建。
+5. 协议新增字段默认进入 `raw_frames.parsed_json` 和 Redis realtime-raw；只有稳定查询需求出现后才提升为 TDengine/MySQL 列。
--- a/docs/go-version-data-flow.html
+++ b/docs/go-version-data-flow.html
@@ -559,7 +559,7 @@
            <td><strong>TDengine</strong></td>
            <td><code>raw_frames</code></td>
            <td>history-writer</td>
-            <td>原始报文、完整 parsed JSON、fields JSON、parse status、source endpoint</td>
+            <td>原始报文、完整 parsed JSON、parse status、source endpoint</td>
            <td>RAW 查询、追溯、排查解析问题</td>
          </tr>
          <tr>
@@ -576,30 +576,23 @@
            <td>经纬度、速度、方向、状态、报警、总里程等核心字段</td>
            <td>高频历史位置分页查询</td>
          </tr>
-          <tr>
-            <td><strong>TDengine</strong></td>
-            <td><code>vehicle_mileage_points</code></td>
-            <td>history-writer</td>
-            <td>累计里程点、速度、经纬度</td>
-            <td>里程曲线、区间差值分析</td>
-          </tr>
          <tr>
            <td><strong>Redis</strong></td>
-            <td><code>vehicle:realtime:{vehicle_key}</code></td>
+            <td><code>vehicle:latest:{vehicle_key}</code></td>
            <td>realtime-api Kafka consumer</td>
            <td>跨协议合并后的实时字段快照</td>
            <td>查 VIN 是否在线、查 VIN 实时数据</td>
          </tr>
          <tr>
            <td><strong>Redis</strong></td>
-            <td><code>vehicle:realtime:{vehicle_key}:{protocol}</code></td>
+            <td><code>vehicle:latest:{vehicle_key}:{protocol}</code></td>
            <td>realtime-api Kafka consumer</td>
            <td>单协议实时快照</td>
            <td>区分 32960/808/MQTT 的实时状态</td>
          </tr>
          <tr>
            <td><strong>Redis</strong></td>
-            <td><code>vehicle:realtime-raw:{vehicle_key}:{protocol}</code></td>
+            <td><code>vehicle:realtime-raw:{protocol}:{vehicle_key}</code></td>
            <td>realtime-api Kafka consumer</td>
            <td>单协议最新 parsed 全量字段</td>
            <td>实时 RAW 字段查看，不走 TDengine 历史扫描</td>
@@ -627,9 +620,9 @@
          </tr>
          <tr>
            <td><strong>MySQL</strong></td>
-            <td><code>vehicle_daily_metric</code></td>
+            <td><code>vehicle_daily_mileage</code></td>
            <td>stat-writer</td>
-            <td>每日里程等指标，按总里程差值计算</td>
+            <td>每日里程，按首末总里程差值计算，保留样本数</td>
            <td>统计查询</td>
          </tr>
        </tbody>
@@ -657,8 +650,8 @@
            <tbody>
              <tr><td>RAW 帧查询</td><td>TDengine raw_frames + chunks</td><td>按协议、VIN/phone、时间、消息类型分页。</td></tr>
              <tr><td>位置历史</td><td>TDengine vehicle_locations</td><td>高频位置分页查询，避免每次扫完整 JSON。</td></tr>
-              <tr><td>里程点</td><td>TDengine vehicle_mileage_points</td><td>区间里程、曲线、异常总里程排查。</td></tr>
-              <tr><td>每日指标</td><td>MySQL vehicle_daily_metric</td><td>按日期、协议、指标查询统计结果。</td></tr>
+              <tr><td>里程历史</td><td>TDengine vehicle_locations</td><td>从位置核心表读取总里程点，不再单独维护里程点表。</td></tr>
+              <tr><td>每日里程</td><td>MySQL vehicle_daily_mileage</td><td>按日期、协议查询首末总里程差值结果。</td></tr>
            </tbody>
          </table>
        </div>
@@ -700,7 +693,7 @@
            <td>TDengine ECS</td>
            <td>TDengine</td>
            <td>内网 <code>172.17.111.57:6041</code></td>
-            <td>历史 RAW、位置、里程点时序存储。</td>
+            <td>历史 RAW、位置时序存储。</td>
          </tr>
          <tr>
            <td>云服务</td>
@@ -729,7 +722,7 @@
        </div>
        <div class="ok">
          <strong>实时多协议合并</strong><br>
-          Redis 同时保留 merged snapshot、protocol snapshot、realtime-raw，既能看统一实时，也能追单协议原始字段。
+          Redis 同时保留 latest snapshot、protocol snapshot、realtime-raw，既能看统一实时，也能追单协议原始字段。
        </div>
        <div class="ok">
          <strong>身份解析降级</strong><br>
--- a/docs/ops/vehicle-ingest-runbook.md
+++ b/docs/ops/vehicle-ingest-runbook.md
@@ -13,6 +13,8 @@

 运行手册的目标是按自上而下的顺序定位问题：入口、队列、桥接、消费、存储、查询 API。

+当前生产服务、topic、表和 Redis key 的清单见 [生产数据面清单](../architecture/production-data-plane-inventory.md)。
+
 ## 服务地图

 | 层级 | 服务 | systemd 单元 | 本机端点 |