# Vehicle IoT Data Platform Principles

## Goal

Build a production vehicle data platform that can receive GB32960, JT808, and Yutong MQTT data reliably, keep raw evidence traceable, and expose only simple business tables for realtime, history, identity, and metrics.

## First Principles

1. Raw data is the source of truth.
2. Kafka is the durable replay log.
3. Redis is realtime cache, not permanent storage.
4. MySQL stores identity, light realtime business snapshots, and low-cardinality metrics.
5. TDengine stores time-series raw and location history.
6. Protocol-specific fields stay in raw parsed JSON unless they become a stable query or metric requirement.
7. Upper business tables must not duplicate full parsed payloads.

## Data Flow

```mermaid
flowchart LR
    GB["GB32960 TCP"] --> GW["Go Gateway"]
    JT["JT808 TCP"] --> GW
    YM["Yutong MQTT"] --> GW

    GW --> NATS["NATS JetStream ingress"]
    NATS --> BR["NATS Kafka Bridge"]
    BR --> KRAW["Kafka raw topics"]
    GW -. fallback .-> KRAW

    KRAW --> HW["history-writer"]
    KRAW --> SW["stat-writer"]
    KRAW --> RA["realtime-api projector"]

    HW --> TDRAW["TDengine raw_frames"]
    HW --> TDLOC["TDengine vehicle_locations"]
    SW --> MYMET["MySQL vehicle_daily_mileage"]
    RA --> REDIS["Redis realtime-raw"]
    RA --> MYSNAP["MySQL vehicle_realtime_snapshot"]
    RA --> MYLOC["MySQL vehicle_realtime_location"]
```

## Minimal Storage Contract

详细的落库边界见 [车辆数据最小落库合约](storage-minimal-contract.md)，当前生产数据面见 [生产数据面清单](production-data-plane-inventory.md)。

| Store | Table or key | Purpose | Keep | Avoid |
| --- | --- | --- | --- | --- |
| TDengine | `raw_frames` | Replay and audit evidence | raw hex/text, full parsed JSON, parse status, protocol tags | business-only duplicated columns |
| TDengine | `vehicle_locations` | High-volume location history | time, vin, protocol, longitude, latitude, speed, direction, mileage | full parsed JSON |
| MySQL | `vehicle_realtime_snapshot` | Latest per-protocol vehicle heartbeat | protocol, vin, plate, event time, received time | phone, device, message sequence, parsed JSON |
| MySQL | `vehicle_realtime_location` | Latest business location cache | vin, plate, location, speed, mileage, SOC, event time | full raw payload and protocol internals |
| MySQL | `vehicle_daily_mileage` | Queryable daily mileage | vin, date, protocol, daily mileage, source mileage | temporary vehicle keys, generic metric key/value rows, per-frame raw details |
| MySQL | `vehicle_identity_binding` | Manual identity mapping | vin, plate, phone, oem | registration history, device id |
| MySQL | `jt808_registration` | JT808 registration and auth trace | phone, device id, plate, auth code, vin match state, first/latest seen | GB32960 or MQTT records |
| Redis | `vehicle:latest:{vin}` | Latest merged realtime state | cross-protocol latest fields only for VIN-bound vehicles | full parsed payloads, temporary identities, and historical data |
| Redis | `vehicle:realtime-raw:{protocol}:{vin}` | Latest full realtime protocol state | latest protocol parsed payload for VIN-bound vehicles | historical data |

## Event Envelope Rules

Every received frame should become one `FrameEnvelope`.

- `protocol`, `message_id`, `event_time_ms`, `received_at_ms`, and `parse_status` are mandatory.
- `vin` is preferred as the vehicle identity.
- JT808 can use `phone` as a temporary key before VIN binding is resolved.
- `raw_hex` or `raw_text` must be retained for replay and parser correction.
- `parsed` stores protocol-specific fields.
- `fields` stores only stable cross-protocol core fields: speed, total mileage, longitude, latitude, SOC.

## Optimization Order

1. Stabilize ingress: connection lifecycle, protocol parser correctness, bounded backpressure, durable publish.
2. Stabilize event log: NATS to Kafka bridge, topic names, partition keys, retry and replay.
3. Simplify storage: keep only the minimal tables above, remove duplicated payloads from business tables.
4. Stabilize realtime: Redis full latest raw state, MySQL lightweight snapshot and location. Realtime consumers start from latest when no committed offset exists; historical rebuilds must be explicit replay jobs, not accidental backlog scans.
5. Stabilize history: raw and location query pagination, clear time zone behavior.
6. Stabilize metrics: idempotent daily metrics from mileage differences and raw replay.
7. Add operations: health, readiness, metrics, lag, connection counts, deploy notes.

## Current Next Steps

1. Keep production inventory current whenever a service, topic, table, or Redis key family changes.
2. Add parser correctness tests before changing GB32960, JT808, or Yutong MQTT field extraction.
3. Add lag and write-failure alerts from the existing `/metrics` endpoints.
4. Isolate or delete legacy Kafka topics only after confirming no external consumers still depend on them.
5. Keep new business tables narrow by default. Add a column only when a query or index proves it is needed.

## Runtime Metrics Baseline

Runtime metrics are exposed as Prometheus text from `/metrics`. They are operational signals and must not create new business tables by default.

Core counters:

- `vehicle_gateway_frames_total`: received protocol frames by protocol and parse status.
- `vehicle_gateway_publish_total`: raw publish results by protocol. Unified publish metrics appear only when the explicit compatibility stream is enabled.
- `vehicle_realtime_kafka_messages_total`: realtime consumer messages by topic and status.
- `vehicle_realtime_updates_total`: Redis/MySQL realtime projector updates by topic and status.
- `vehicle_history_writes_total`: TDengine history writes by topic and status.
- `vehicle_stat_writes_total`: MySQL metric writes by topic and status.
- `vehicle_bridge_kafka_writes_total`: NATS to Kafka bridge writes by Kafka topic and status.

If a metric becomes a product requirement, derive a narrow metric table from Kafka replay instead of widening raw or realtime tables.
