98 lines
5.2 KiB
Markdown
98 lines
5.2 KiB
Markdown
# Vehicle IoT Data Platform Principles
|
|
|
|
## Goal
|
|
|
|
Build a production vehicle data platform that can receive GB32960, JT808, and Yutong MQTT data reliably, keep raw evidence traceable, and expose only simple business tables for realtime, history, identity, and metrics.
|
|
|
|
## First Principles
|
|
|
|
1. Raw data is the source of truth.
|
|
2. Kafka is the durable replay log.
|
|
3. Redis is realtime cache, not permanent storage.
|
|
4. MySQL stores identity, light realtime business snapshots, and low-cardinality metrics.
|
|
5. TDengine stores time-series raw and location history.
|
|
6. Protocol-specific fields stay in raw parsed JSON unless they become a stable query or metric requirement.
|
|
7. Upper business tables must not duplicate full parsed payloads.
|
|
|
|
## Data Flow
|
|
|
|
```mermaid
|
|
flowchart LR
|
|
GB["GB32960 TCP"] --> GW["Go Gateway"]
|
|
JT["JT808 TCP"] --> GW
|
|
YM["Yutong MQTT"] --> GW
|
|
|
|
GW --> NATS["NATS JetStream ingress"]
|
|
NATS --> BR["NATS Kafka Bridge"]
|
|
BR --> KRAW["Kafka raw topics"]
|
|
GW -. fallback .-> KRAW
|
|
|
|
KRAW --> HW["history-writer"]
|
|
KRAW --> SW["stat-writer"]
|
|
KRAW --> RA["realtime-api projector"]
|
|
|
|
HW --> TDRAW["TDengine raw_frames"]
|
|
HW --> TDLOC["TDengine vehicle_locations"]
|
|
SW --> MYMET["MySQL vehicle_daily_metric"]
|
|
RA --> REDIS["Redis realtime-raw"]
|
|
RA --> MYSNAP["MySQL vehicle_realtime_snapshot"]
|
|
RA --> MYLOC["MySQL vehicle_realtime_location"]
|
|
```
|
|
|
|
## Minimal Storage Contract
|
|
|
|
| Store | Table or key | Purpose | Keep | Avoid |
|
|
| --- | --- | --- | --- | --- |
|
|
| TDengine | `raw_frames` | Replay and audit evidence | raw hex/text, full parsed JSON, parse status, protocol tags | business-only duplicated columns |
|
|
| TDengine | `vehicle_locations` | High-volume location history | time, vin, protocol, longitude, latitude, speed, direction, mileage | full parsed JSON |
|
|
| MySQL | `vehicle_realtime_snapshot` | Latest per-protocol vehicle heartbeat | protocol, vin, plate, event time, received time | phone, device, message sequence, parsed JSON |
|
|
| MySQL | `vehicle_realtime_location` | Latest business location cache | vin, plate, location, speed, mileage, SOC, event time | full raw payload and protocol internals |
|
|
| MySQL | `vehicle_daily_metric` | Queryable daily metrics | vin or vehicle key, date, metric key, value, method, source mileage | per-frame raw details |
|
|
| MySQL | `vehicle_identity_binding` | Manual identity mapping | vin, plate, phone, device id | registration history |
|
|
| MySQL | `jt808_registration` | JT808 registration and auth trace | phone, device id, plate, auth code, vin match state, first/latest seen | GB32960 or MQTT records |
|
|
| Redis | `realtime-raw:{protocol}:{vin}` | Latest full realtime protocol state | merged latest protocol payload | historical data |
|
|
|
|
## Event Envelope Rules
|
|
|
|
Every received frame should become one `FrameEnvelope`.
|
|
|
|
- `protocol`, `message_id`, `event_time_ms`, `received_at_ms`, and `parse_status` are mandatory.
|
|
- `vin` is preferred as the vehicle identity.
|
|
- JT808 can use `phone` as a temporary key before VIN binding is resolved.
|
|
- `raw_hex` or `raw_text` must be retained for replay and parser correction.
|
|
- `parsed` stores protocol-specific fields.
|
|
- `fields` stores only stable cross-protocol core fields: speed, total mileage, longitude, latitude, SOC.
|
|
|
|
## Optimization Order
|
|
|
|
1. Stabilize ingress: connection lifecycle, protocol parser correctness, bounded backpressure, durable publish.
|
|
2. Stabilize event log: NATS to Kafka bridge, topic names, partition keys, retry and replay.
|
|
3. Simplify storage: keep only the minimal tables above, remove duplicated payloads from business tables.
|
|
4. Stabilize realtime: Redis full latest raw state, MySQL lightweight snapshot and location.
|
|
5. Stabilize history: raw and location query pagination, clear time zone behavior.
|
|
6. Stabilize metrics: idempotent daily metrics from mileage differences and raw replay.
|
|
7. Add operations: health, readiness, metrics, lag, connection counts, deploy notes.
|
|
|
|
## Current Next Steps
|
|
|
|
1. Add health and readiness endpoints to each long-running Go service.
|
|
2. Add runtime metrics for protocol frame counts, parse failures, Kafka/NATS publish failures, and writer lag.
|
|
3. Review TDengine tables and remove unused tables only after export or explicit confirmation.
|
|
4. Keep new business tables narrow by default. Add a column only when a query or index proves it is needed.
|
|
|
|
## Runtime Metrics Baseline
|
|
|
|
Runtime metrics are exposed as Prometheus text from `/metrics`. They are operational signals and must not create new business tables by default.
|
|
|
|
Core counters:
|
|
|
|
- `vehicle_gateway_frames_total`: received protocol frames by protocol and parse status.
|
|
- `vehicle_gateway_publish_total`: raw and unified publish results by protocol.
|
|
- `vehicle_realtime_kafka_messages_total`: realtime consumer messages by topic and status.
|
|
- `vehicle_realtime_updates_total`: Redis/MySQL realtime projector updates by topic and status.
|
|
- `vehicle_history_writes_total`: TDengine history writes by topic and status.
|
|
- `vehicle_stat_writes_total`: MySQL metric writes by topic and status.
|
|
- `vehicle_bridge_kafka_writes_total`: NATS to Kafka bridge writes by Kafka topic and status.
|
|
|
|
If a metric becomes a product requirement, derive a narrow metric table from Kafka replay instead of widening raw or realtime tables.
|