lingniu-vehicle-ingest/docs/architecture/iot-data-platform-principles.md

Vehicle IoT Data Platform Principles

Goal

Build a production vehicle data platform that can receive GB32960, JT808, and Yutong MQTT data reliably, keep raw evidence traceable, and expose only simple business tables for realtime, history, identity, and metrics.

First Principles

1. Raw data is the source of truth.
2. Kafka is the durable replay log.
3. Redis is realtime cache, not permanent storage.
4. MySQL stores identity, light realtime business snapshots, and low-cardinality metrics.
5. TDengine stores time-series raw and location history.
6. Protocol-specific fields stay in raw parsed JSON unless they become a stable query or metric requirement.
7. Upper business tables must not duplicate full parsed payloads.

Data Flow

flowchart LR
    GB["GB32960 TCP"] --> GW["Go Gateway"]
    JT["JT808 TCP"] --> GW
    YM["Yutong MQTT"] --> GW

    GW --> NATS["NATS JetStream ingress"]
    NATS --> BR["NATS Kafka Bridge"]
    BR --> KRAW["Kafka raw topics"]
    GW -. fallback .-> KRAW

    KRAW --> HW["history-writer"]
    KRAW --> SW["stat-writer"]
    KRAW --> RA["realtime-api projector"]

    HW --> TDRAW["TDengine raw_frames"]
    HW --> TDLOC["TDengine vehicle_locations"]
    SW --> MYMET["MySQL vehicle_daily_mileage"]
    RA --> REDIS["Redis realtime-raw"]
    RA --> MYSNAP["MySQL vehicle_realtime_snapshot"]
    RA --> MYLOC["MySQL vehicle_realtime_location"]

Minimal Storage Contract

详细的落库边界见 车辆数据最小落库合约，当前生产数据面见 生产数据面清单。

Store	Table or key	Purpose	Keep	Avoid
TDengine	raw_frames	Replay and audit evidence	raw hex/text, full parsed JSON, parse status, protocol tags	business-only duplicated columns
TDengine	vehicle_locations	High-volume location history	time, vin, protocol, longitude, latitude, speed, direction, mileage	full parsed JSON
MySQL	vehicle_realtime_snapshot	Latest per-protocol vehicle heartbeat	protocol, vin, plate, event time, received time	phone, device, message sequence, parsed JSON
MySQL	vehicle_realtime_location	Latest business location cache	vin, plate, location, speed, mileage, SOC, event time	full raw payload and protocol internals
MySQL	vehicle_daily_mileage	Queryable daily mileage	vin, date, protocol, daily mileage, source mileage	temporary vehicle keys, generic metric key/value rows, per-frame raw details
MySQL	vehicle_identity_binding	Manual identity mapping	vin, plate, phone, device id	registration history
MySQL	jt808_registration	JT808 registration and auth trace	phone, device id, plate, auth code, vin match state, first/latest seen	GB32960 or MQTT records
Redis	vehicle:latest:{vehicleKey}	Latest merged realtime state	cross-protocol latest fields only	full parsed payloads and historical data
Redis	vehicle:realtime-raw:{protocol}:{vehicleKey}	Latest full realtime protocol state	latest protocol parsed payload	historical data

Event Envelope Rules

Every received frame should become one FrameEnvelope.

• protocol, message_id, event_time_ms, received_at_ms, and parse_status are mandatory.
• vin is preferred as the vehicle identity.
• JT808 can use phone as a temporary key before VIN binding is resolved.
• raw_hex or raw_text must be retained for replay and parser correction.
• parsed stores protocol-specific fields.
• fields stores only stable cross-protocol core fields: speed, total mileage, longitude, latitude, SOC.

Optimization Order

1. Stabilize ingress: connection lifecycle, protocol parser correctness, bounded backpressure, durable publish.
2. Stabilize event log: NATS to Kafka bridge, topic names, partition keys, retry and replay.
3. Simplify storage: keep only the minimal tables above, remove duplicated payloads from business tables.
4. Stabilize realtime: Redis full latest raw state, MySQL lightweight snapshot and location.
5. Stabilize history: raw and location query pagination, clear time zone behavior.
6. Stabilize metrics: idempotent daily metrics from mileage differences and raw replay.
7. Add operations: health, readiness, metrics, lag, connection counts, deploy notes.

Current Next Steps

1. Keep production inventory current whenever a service, topic, table, or Redis key family changes.
2. Add parser correctness tests before changing GB32960, JT808, or Yutong MQTT field extraction.
3. Add lag and write-failure alerts from the existing /metrics endpoints.
4. Isolate or delete legacy Kafka topics only after confirming no external consumers still depend on them.
5. Keep new business tables narrow by default. Add a column only when a query or index proves it is needed.

Runtime Metrics Baseline

Runtime metrics are exposed as Prometheus text from /metrics. They are operational signals and must not create new business tables by default.

Core counters:

• vehicle_gateway_frames_total: received protocol frames by protocol and parse status.
• vehicle_gateway_publish_total: raw and unified publish results by protocol.
• vehicle_realtime_kafka_messages_total: realtime consumer messages by topic and status.
• vehicle_realtime_updates_total: Redis/MySQL realtime projector updates by topic and status.
• vehicle_history_writes_total: TDengine history writes by topic and status.
• vehicle_stat_writes_total: MySQL metric writes by topic and status.
• vehicle_bridge_kafka_writes_total: NATS to Kafka bridge writes by Kafka topic and status.

If a metric becomes a product requirement, derive a narrow metric table from Kafka replay instead of widening raw or realtime tables.