feat(go): add platform principles and health probes
This commit is contained in:
81
docs/architecture/iot-data-platform-principles.md
Normal file
81
docs/architecture/iot-data-platform-principles.md
Normal file
@@ -0,0 +1,81 @@
|
||||
# Vehicle IoT Data Platform Principles
|
||||
|
||||
## Goal
|
||||
|
||||
Build a production vehicle data platform that can receive GB32960, JT808, and Yutong MQTT data reliably, keep raw evidence traceable, and expose only simple business tables for realtime, history, identity, and metrics.
|
||||
|
||||
## First Principles
|
||||
|
||||
1. Raw data is the source of truth.
|
||||
2. Kafka is the durable replay log.
|
||||
3. Redis is realtime cache, not permanent storage.
|
||||
4. MySQL stores identity, light realtime business snapshots, and low-cardinality metrics.
|
||||
5. TDengine stores time-series raw and location history.
|
||||
6. Protocol-specific fields stay in raw parsed JSON unless they become a stable query or metric requirement.
|
||||
7. Upper business tables must not duplicate full parsed payloads.
|
||||
|
||||
## Data Flow
|
||||
|
||||
```mermaid
|
||||
flowchart LR
|
||||
GB["GB32960 TCP"] --> GW["Go Gateway"]
|
||||
JT["JT808 TCP"] --> GW
|
||||
YM["Yutong MQTT"] --> GW
|
||||
|
||||
GW --> NATS["NATS JetStream ingress"]
|
||||
NATS --> BR["NATS Kafka Bridge"]
|
||||
BR --> KRAW["Kafka raw topics"]
|
||||
GW -. fallback .-> KRAW
|
||||
|
||||
KRAW --> HW["history-writer"]
|
||||
KRAW --> SW["stat-writer"]
|
||||
KRAW --> RA["realtime-api projector"]
|
||||
|
||||
HW --> TDRAW["TDengine raw_frames"]
|
||||
HW --> TDLOC["TDengine vehicle_locations"]
|
||||
SW --> MYMET["MySQL vehicle_daily_metric"]
|
||||
RA --> REDIS["Redis realtime-raw"]
|
||||
RA --> MYSNAP["MySQL vehicle_realtime_snapshot"]
|
||||
RA --> MYLOC["MySQL vehicle_realtime_location"]
|
||||
```
|
||||
|
||||
## Minimal Storage Contract
|
||||
|
||||
| Store | Table or key | Purpose | Keep | Avoid |
|
||||
| --- | --- | --- | --- | --- |
|
||||
| TDengine | `raw_frames` | Replay and audit evidence | raw hex/text, full parsed JSON, parse status, protocol tags | business-only duplicated columns |
|
||||
| TDengine | `vehicle_locations` | High-volume location history | time, vin, protocol, longitude, latitude, speed, direction, mileage | full parsed JSON |
|
||||
| MySQL | `vehicle_realtime_snapshot` | Latest per-protocol vehicle heartbeat | protocol, vin, plate, event time, received time | phone, device, message sequence, parsed JSON |
|
||||
| MySQL | `vehicle_realtime_location` | Latest business location cache | vin, plate, location, speed, mileage, SOC, event time | full raw payload and protocol internals |
|
||||
| MySQL | `vehicle_daily_metric` | Queryable daily metrics | vin or vehicle key, date, metric key, value, method, source mileage | per-frame raw details |
|
||||
| MySQL | `vehicle_identity_binding` | Manual identity mapping | vin, plate, phone, device id | registration history |
|
||||
| MySQL | `jt808_registration` | JT808 registration and auth trace | phone, device id, plate, auth code, vin match state, first/latest seen | GB32960 or MQTT records |
|
||||
| Redis | `realtime-raw:{protocol}:{vin}` | Latest full realtime protocol state | merged latest protocol payload | historical data |
|
||||
|
||||
## Event Envelope Rules
|
||||
|
||||
Every received frame should become one `FrameEnvelope`.
|
||||
|
||||
- `protocol`, `message_id`, `event_time_ms`, `received_at_ms`, and `parse_status` are mandatory.
|
||||
- `vin` is preferred as the vehicle identity.
|
||||
- JT808 can use `phone` as a temporary key before VIN binding is resolved.
|
||||
- `raw_hex` or `raw_text` must be retained for replay and parser correction.
|
||||
- `parsed` stores protocol-specific fields.
|
||||
- `fields` stores only stable cross-protocol core fields: speed, total mileage, longitude, latitude, SOC.
|
||||
|
||||
## Optimization Order
|
||||
|
||||
1. Stabilize ingress: connection lifecycle, protocol parser correctness, bounded backpressure, durable publish.
|
||||
2. Stabilize event log: NATS to Kafka bridge, topic names, partition keys, retry and replay.
|
||||
3. Simplify storage: keep only the minimal tables above, remove duplicated payloads from business tables.
|
||||
4. Stabilize realtime: Redis full latest raw state, MySQL lightweight snapshot and location.
|
||||
5. Stabilize history: raw and location query pagination, clear time zone behavior.
|
||||
6. Stabilize metrics: idempotent daily metrics from mileage differences and raw replay.
|
||||
7. Add operations: health, readiness, metrics, lag, connection counts, deploy notes.
|
||||
|
||||
## Current Next Steps
|
||||
|
||||
1. Add health and readiness endpoints to each long-running Go service.
|
||||
2. Add runtime metrics for protocol frame counts, parse failures, Kafka/NATS publish failures, and writer lag.
|
||||
3. Review TDengine tables and remove unused tables only after export or explicit confirmation.
|
||||
4. Keep new business tables narrow by default. Add a column only when a query or index proves it is needed.
|
||||
Reference in New Issue
Block a user