docs: design gb32960 production readiness
This commit is contained in:
lingniu
@@ -0,0 +1,487 @@
|
||||
# GB32960 Production Readiness Design
|
||||
|
||||
## Goal
|
||||
|
||||
Make the GB32960 ingestion, history, and analytics services production-ready:
|
||||
|
||||
- receive and parse GB32960 traffic quickly and accurately;
|
||||
- maximize VIN-to-platform/vendor-profile association through platform login and local mappings;
|
||||
- store complete RAW frames and queryable telemetry history with DuckDB;
|
||||
- support full frame replay from persisted RAW bytes;
|
||||
- compute real-time daily vehicle metrics and alarm timelines into MySQL;
|
||||
- reduce service coupling and improve cohesion inside each module.
|
||||
|
||||
## Non-Goals
|
||||
|
||||
- Do not put database credentials in code, YAML files, tests, docs, or commits.
|
||||
- Do not make MySQL the source for full telemetry history. MySQL is for derived daily metrics and alarm timelines.
|
||||
- Do not require every private GB32960 extension to be perfect before storing data. RAW bytes must always be kept when a valid frame is received.
|
||||
|
||||
## Production Architecture
|
||||
|
||||
```mermaid
|
||||
flowchart LR
|
||||
tcp["GB32960 TCP:32960"] --> ingest["gb32960-ingest-app"]
|
||||
mapping["local vin-platform-profile.jsonl"] --> ingest
|
||||
ingest --> kafkaEvent["Kafka vehicle.event.gb32960.v1"]
|
||||
ingest --> kafkaRaw["Kafka vehicle.raw.gb32960.v1"]
|
||||
kafkaEvent --> history["vehicle-history-app"]
|
||||
kafkaRaw --> history
|
||||
history --> duckdb["DuckDB hot history"]
|
||||
history --> raw["RAW frame archive"]
|
||||
kafkaEvent --> analytics["vehicle-analytics-app"]
|
||||
analytics --> mysql["MySQL daily stats + alarms"]
|
||||
```
|
||||
|
||||
## Service Boundaries
|
||||
|
||||
### 1. `gb32960-ingest-app`
|
||||
|
||||
Responsibilities:
|
||||
|
||||
- Netty TCP accept, frame splitting, GB32960 decode, auth, ACK, and Kafka production.
|
||||
- Platform login handling and connection-local `platformAccount`.
|
||||
- VIN-to-platform/vendor-profile resolution.
|
||||
- Produce structured telemetry, session, alarm, diagnostics, and RAW envelope messages.
|
||||
|
||||
It must not:
|
||||
|
||||
- write DuckDB, MySQL, or local archive files;
|
||||
- run historical queries;
|
||||
- run daily statistics.
|
||||
|
||||
ACK policy:
|
||||
|
||||
- login/report-like GB32960 commands use durable ACK: ACK only after Kafka dispatch succeeds;
|
||||
- malformed bytes that cannot form a valid frame are dropped with diagnostics;
|
||||
- valid frames with partial private-block parse failures still go to Kafka with `parseStatus=PARTIAL`.
|
||||
|
||||
VIN/profile resolution priority:
|
||||
|
||||
1. active platform login on the current TCP channel;
|
||||
2. exact VIN entry from local mapping file;
|
||||
3. VIN prefix entry from local mapping file;
|
||||
4. configured platform extension rule;
|
||||
5. standard GB32960 profile.
|
||||
|
||||
Local mapping file format:
|
||||
|
||||
```jsonl
|
||||
{"vin":"LNXNEGRR2SR321390","platformAccount":"Hyundai","vendorProfile":"guangdong-fc","enabled":true}
|
||||
{"vinPrefix":"LNXNEGRR","platformAccount":"Hyundai","vendorProfile":"guangdong-fc","enabled":true}
|
||||
```
|
||||
|
||||
The mapping file is loaded at startup and can be reloaded by an actuator endpoint or file watcher in a later phase.
|
||||
|
||||
### 2. `vehicle-history-app`
|
||||
|
||||
Responsibilities:
|
||||
|
||||
- Consume Kafka event and RAW topics.
|
||||
- Persist RAW bytes to an append-only archive.
|
||||
- Persist query indexes and compact telemetry points to DuckDB.
|
||||
- Serve high-performance query APIs:
|
||||
- full frame list by `vin + eventTime`;
|
||||
- telemetry fields by `vin + eventTime`;
|
||||
- one-frame replay by `rawArchiveUri`;
|
||||
- parse diagnostics and missing-profile troubleshooting.
|
||||
|
||||
It must not:
|
||||
|
||||
- listen on GB32960 TCP;
|
||||
- run MySQL statistics;
|
||||
- mutate daily business metrics.
|
||||
|
||||
### 3. `vehicle-analytics-app`
|
||||
|
||||
Responsibilities:
|
||||
|
||||
- Consume Kafka event topic.
|
||||
- Maintain real-time daily stats in MySQL.
|
||||
- Maintain full alarm timelines in MySQL.
|
||||
- Expose metrics query APIs and operational health.
|
||||
|
||||
It must not:
|
||||
|
||||
- parse RAW frames for normal operation;
|
||||
- write DuckDB history;
|
||||
- own GB32960 TCP connectivity.
|
||||
|
||||
## Kafka Contract
|
||||
|
||||
Keep Protobuf `VehicleEnvelope`, but make the producer contract stricter:
|
||||
|
||||
- `eventId`: stable idempotency key, deterministic from protocol, VIN, command, event time, raw checksum, and sequence when available.
|
||||
- `vin`: required for vehicle frames; platform-only commands use `_platform` or empty plus `platformAccount`.
|
||||
- `eventTimeMs`: GB32960 data collection time when available.
|
||||
- `ingestTimeMs`: service receive time.
|
||||
- `metadata.platformAccount`: resolved platform account.
|
||||
- `metadata.vendorProfile`: selected parser profile.
|
||||
- `metadata.parseStatus`: `OK`, `PARTIAL`, or `FAILED`.
|
||||
- `metadata.parseErrorCode`: stable code for failures.
|
||||
- `raw_archive.data`: present on RAW topic only; event topic keeps `rawArchiveUri` pointer.
|
||||
- `telemetry_snapshot`: standard field projection for downstream services.
|
||||
|
||||
Consumer rule:
|
||||
|
||||
- History consumes both event and RAW topics.
|
||||
- Analytics consumes only event topic.
|
||||
- Consumer offset is committed only after the target store commit succeeds.
|
||||
- Reprocessing must be safe through deterministic ids and database upsert/ignore semantics.
|
||||
|
||||
## DuckDB History Design
|
||||
|
||||
Use DuckDB as the hot historical database, not a Parquet-rewrite sidecar.
|
||||
|
||||
DuckDB official guidance says row-by-row insert loops are inefficient for bulk loading, and Appender/batched writes are the intended high-throughput path. Therefore history writes use a single writer worker with bounded batches and one DuckDB connection.
|
||||
|
||||
### Tables
|
||||
|
||||
`gb32960_frame_index`
|
||||
|
||||
```sql
|
||||
CREATE TABLE IF NOT EXISTS gb32960_frame_index (
|
||||
event_id VARCHAR PRIMARY KEY,
|
||||
vin VARCHAR NOT NULL,
|
||||
platform_account VARCHAR,
|
||||
vendor_profile VARCHAR,
|
||||
command VARCHAR NOT NULL,
|
||||
command_code INTEGER NOT NULL,
|
||||
event_time TIMESTAMP NOT NULL,
|
||||
event_time_ms BIGINT NOT NULL,
|
||||
ingest_time TIMESTAMP NOT NULL,
|
||||
ingest_time_ms BIGINT NOT NULL,
|
||||
partition_date DATE NOT NULL,
|
||||
raw_archive_uri VARCHAR NOT NULL,
|
||||
raw_checksum VARCHAR,
|
||||
raw_size_bytes BIGINT NOT NULL,
|
||||
parse_status VARCHAR NOT NULL,
|
||||
parse_error_code VARCHAR,
|
||||
block_count INTEGER NOT NULL,
|
||||
metadata_json VARCHAR NOT NULL
|
||||
);
|
||||
|
||||
CREATE INDEX IF NOT EXISTS gb32960_frame_vin_time_idx
|
||||
ON gb32960_frame_index(vin, event_time_ms);
|
||||
|
||||
CREATE INDEX IF NOT EXISTS gb32960_frame_raw_uri_idx
|
||||
ON gb32960_frame_index(raw_archive_uri);
|
||||
|
||||
CREATE INDEX IF NOT EXISTS gb32960_frame_partition_time_idx
|
||||
ON gb32960_frame_index(partition_date, event_time_ms);
|
||||
```
|
||||
|
||||
`gb32960_telemetry_point`
|
||||
|
||||
```sql
|
||||
CREATE TABLE IF NOT EXISTS gb32960_telemetry_point (
|
||||
event_id VARCHAR NOT NULL,
|
||||
vin VARCHAR NOT NULL,
|
||||
event_time TIMESTAMP NOT NULL,
|
||||
event_time_ms BIGINT NOT NULL,
|
||||
ingest_time TIMESTAMP NOT NULL,
|
||||
platform_account VARCHAR,
|
||||
vendor_profile VARCHAR,
|
||||
field_key VARCHAR NOT NULL,
|
||||
value_type VARCHAR NOT NULL,
|
||||
value_text VARCHAR NOT NULL,
|
||||
value_num DOUBLE,
|
||||
unit VARCHAR,
|
||||
quality VARCHAR NOT NULL,
|
||||
raw_archive_uri VARCHAR NOT NULL,
|
||||
PRIMARY KEY(event_id, field_key)
|
||||
);
|
||||
|
||||
CREATE INDEX IF NOT EXISTS gb32960_point_vin_field_time_idx
|
||||
ON gb32960_telemetry_point(vin, field_key, event_time_ms);
|
||||
```
|
||||
|
||||
`gb32960_alarm_frame`
|
||||
|
||||
```sql
|
||||
CREATE TABLE IF NOT EXISTS gb32960_alarm_frame (
|
||||
event_id VARCHAR PRIMARY KEY,
|
||||
vin VARCHAR NOT NULL,
|
||||
event_time TIMESTAMP NOT NULL,
|
||||
event_time_ms BIGINT NOT NULL,
|
||||
level VARCHAR NOT NULL,
|
||||
active_bits_json VARCHAR NOT NULL,
|
||||
fault_codes_json VARCHAR NOT NULL,
|
||||
raw_archive_uri VARCHAR NOT NULL
|
||||
);
|
||||
|
||||
CREATE INDEX IF NOT EXISTS gb32960_alarm_vin_time_idx
|
||||
ON gb32960_alarm_frame(vin, event_time_ms);
|
||||
```
|
||||
|
||||
### RAW Archive
|
||||
|
||||
RAW archive remains file/object based:
|
||||
|
||||
```text
|
||||
archive://yyyy/MM/dd/GB32960/{vin}/{event_id}.bin
|
||||
```
|
||||
|
||||
The archive writer verifies:
|
||||
|
||||
- stored byte length equals envelope `raw_archive.size_bytes`;
|
||||
- checksum matches;
|
||||
- URI is unique for the deterministic event id.
|
||||
|
||||
Frame replay path:
|
||||
|
||||
1. query `gb32960_frame_index` by `rawArchiveUri`;
|
||||
2. read RAW bytes from archive;
|
||||
3. resolve `platformAccount/vendorProfile` from index metadata;
|
||||
4. decode using current parser;
|
||||
5. return decoded frame plus original indexed parse diagnostics.
|
||||
|
||||
## Query APIs
|
||||
|
||||
Keep existing endpoints but back them by the new DuckDB tables:
|
||||
|
||||
- `GET /api/event-history/gb32960/frames`
|
||||
- full frame query by VIN/time.
|
||||
- `GET /api/event-history/gb32960/frame`
|
||||
- one RAW frame replay by `rawArchiveUri`.
|
||||
- `GET /api/event-history/gb32960/telemetry-snapshots`
|
||||
- compact telemetry snapshot by VIN/time.
|
||||
- `GET /api/event-history/gb32960/telemetry-fields`
|
||||
- field projection optimized by `gb32960_telemetry_point`.
|
||||
- `GET /api/event-history/gb32960/diagnostics`
|
||||
- parse status and missing-profile investigation by VIN/time.
|
||||
|
||||
Time semantics:
|
||||
|
||||
- `eventTime` is business query time.
|
||||
- `ingestTime` is kept for operational replay, late-arrival detection, and Kafka lag investigation.
|
||||
- Dates without timezone are interpreted in `Asia/Shanghai`.
|
||||
|
||||
## MySQL Analytics Design
|
||||
|
||||
Credentials are supplied only by environment variables:
|
||||
|
||||
```text
|
||||
MYSQL_HOST
|
||||
MYSQL_PORT
|
||||
MYSQL_DATABASE
|
||||
MYSQL_USERNAME
|
||||
MYSQL_PASSWORD
|
||||
```
|
||||
|
||||
`vehicle_daily_stat`
|
||||
|
||||
```sql
|
||||
CREATE TABLE IF NOT EXISTS vehicle_daily_stat (
|
||||
vin VARCHAR(32) NOT NULL,
|
||||
stat_date DATE NOT NULL,
|
||||
platform_account VARCHAR(64),
|
||||
total_mileage_km DECIMAL(14,3),
|
||||
daily_mileage_km DECIMAL(14,3),
|
||||
daily_power_kwh DECIMAL(14,3),
|
||||
daily_hydrogen_kg DECIMAL(14,4),
|
||||
daily_hydrogen_kg_per_100km DECIMAL(14,4),
|
||||
first_event_time DATETIME(3),
|
||||
last_event_time DATETIME(3),
|
||||
first_total_mileage_km DECIMAL(14,3),
|
||||
last_total_mileage_km DECIMAL(14,3),
|
||||
last_soc_percent DECIMAL(8,3),
|
||||
sample_count BIGINT NOT NULL DEFAULT 0,
|
||||
late_sample_count BIGINT NOT NULL DEFAULT 0,
|
||||
calculation_quality VARCHAR(32) NOT NULL,
|
||||
updated_at DATETIME(3) NOT NULL,
|
||||
PRIMARY KEY (vin, stat_date)
|
||||
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4;
|
||||
```
|
||||
|
||||
`vehicle_alarm_timeline`
|
||||
|
||||
```sql
|
||||
CREATE TABLE IF NOT EXISTS vehicle_alarm_timeline (
|
||||
alarm_id VARCHAR(128) NOT NULL,
|
||||
vin VARCHAR(32) NOT NULL,
|
||||
alarm_key VARCHAR(128) NOT NULL,
|
||||
level VARCHAR(32) NOT NULL,
|
||||
start_time DATETIME(3) NOT NULL,
|
||||
end_time DATETIME(3),
|
||||
last_seen_time DATETIME(3) NOT NULL,
|
||||
start_raw_archive_uri VARCHAR(512),
|
||||
last_raw_archive_uri VARCHAR(512),
|
||||
status VARCHAR(16) NOT NULL,
|
||||
details_json JSON,
|
||||
updated_at DATETIME(3) NOT NULL,
|
||||
PRIMARY KEY (alarm_id),
|
||||
KEY vehicle_alarm_vin_time_idx (vin, start_time),
|
||||
KEY vehicle_alarm_open_idx (vin, status, alarm_key)
|
||||
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4;
|
||||
```
|
||||
|
||||
`vehicle_stat_event_dedup`
|
||||
|
||||
```sql
|
||||
CREATE TABLE IF NOT EXISTS vehicle_stat_event_dedup (
|
||||
event_id VARCHAR(128) NOT NULL,
|
||||
consumer_group VARCHAR(128) NOT NULL,
|
||||
processed_at DATETIME(3) NOT NULL,
|
||||
PRIMARY KEY (event_id, consumer_group)
|
||||
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4;
|
||||
```
|
||||
|
||||
Writes use `INSERT ... ON DUPLICATE KEY UPDATE` so replayed Kafka messages update the same daily row instead of creating duplicates.
|
||||
|
||||
## Daily Metric Algorithms
|
||||
|
||||
All daily statistics are keyed by `eventTime` in `Asia/Shanghai`.
|
||||
|
||||
### Total Mileage
|
||||
|
||||
Use the latest valid `totalMileageKm` seen for the day.
|
||||
|
||||
Validation:
|
||||
|
||||
- reject negative values;
|
||||
- reject impossible jumps based on configurable max speed and elapsed time;
|
||||
- keep the sample as diagnostic if rejected.
|
||||
|
||||
### Daily Mileage
|
||||
|
||||
Preferred:
|
||||
|
||||
```text
|
||||
dailyMileage = today.lastTotalMileage - previousDay.lastTotalMileage
|
||||
```
|
||||
|
||||
Fallback:
|
||||
|
||||
```text
|
||||
dailyMileage = today.lastTotalMileage - today.firstTotalMileage
|
||||
```
|
||||
|
||||
Quality:
|
||||
|
||||
- `PREVIOUS_DAY_BASELINE` when previous day baseline exists;
|
||||
- `INTRADAY_DELTA` when only same-day baseline exists;
|
||||
- `INSUFFICIENT_DATA` when neither is reliable.
|
||||
|
||||
### Daily Power Consumption
|
||||
|
||||
Preferred:
|
||||
|
||||
- vendor cumulative power-consumption field if available.
|
||||
|
||||
Fallback:
|
||||
|
||||
```text
|
||||
deltaKwh = sum(max(0, totalVoltageV * totalCurrentA) * deltaSeconds / 3600000)
|
||||
```
|
||||
|
||||
The sign convention is configurable per vendor profile because GB32960 vehicle current may be positive or negative depending on charging/discharging representation.
|
||||
|
||||
### Daily Hydrogen Consumption
|
||||
|
||||
Use a tiered market-practical algorithm instead of a single fixed formula:
|
||||
|
||||
1. `DIRECT_COUNTER`: vendor cumulative hydrogen consumption or remaining hydrogen mass delta.
|
||||
2. `PVT_MASS_DELTA`: pressure/temperature/tank-volume mass estimate when storage data is available.
|
||||
3. `FUEL_CELL_ENERGY_MODEL`: integrate fuel-cell electric output and divide by calibrated efficiency.
|
||||
4. `UNKNOWN`: keep null when required signals are missing.
|
||||
|
||||
The PVT approach follows the industry pattern behind compressed hydrogen consumption tests, where pressure, temperature, and tank volume can be used to estimate hydrogen mass. The implementation must expose calibration constants per vehicle model/profile:
|
||||
|
||||
```json
|
||||
{
|
||||
"vinPrefix": "LNXNEGRR",
|
||||
"tankVolumeLiter": 140,
|
||||
"hydrogenModel": "PVT_MASS_DELTA",
|
||||
"fuelCellEfficiency": 0.52,
|
||||
"currentDischargeSign": "POSITIVE"
|
||||
}
|
||||
```
|
||||
|
||||
### Daily Hydrogen Consumption Rate
|
||||
|
||||
```text
|
||||
dailyHydrogenKgPer100km = dailyHydrogenKg / dailyMileageKm * 100
|
||||
```
|
||||
|
||||
If mileage is missing or less than the configured minimum distance, keep the rate null and set quality to `INSUFFICIENT_DISTANCE`.
|
||||
|
||||
### Alarm Timeline
|
||||
|
||||
For each telemetry frame:
|
||||
|
||||
- derive active alarm keys from alarm bits and fault code arrays;
|
||||
- open a timeline row when a key first appears;
|
||||
- update `last_seen_time` while it remains active;
|
||||
- close open alarms when the key disappears after a configurable grace window;
|
||||
- keep start/end raw archive URIs for replay.
|
||||
|
||||
## Reliability and Observability
|
||||
|
||||
Required metrics:
|
||||
|
||||
- TCP active connections;
|
||||
- frames received per command;
|
||||
- decode failures by code;
|
||||
- missing platform login rejections;
|
||||
- vendor profile resolution source;
|
||||
- Kafka send latency and failures;
|
||||
- history write batch size, latency, failure count;
|
||||
- DuckDB query latency by endpoint;
|
||||
- MySQL upsert latency and failure count;
|
||||
- consumer lag by group/topic/partition;
|
||||
- late sample count by VIN/date.
|
||||
|
||||
Required logs:
|
||||
|
||||
- one structured line for every dropped or rejected valid frame;
|
||||
- no per-frame INFO spam for successful reports after first frame per VIN/profile unless debug is enabled;
|
||||
- include `vin`, `platformAccount`, `command`, `eventId`, `reasonCode`, and `peer`.
|
||||
|
||||
## Migration Plan
|
||||
|
||||
Phase 1: History foundation
|
||||
|
||||
- Replace Parquet rewrite store with DuckDB hot tables and batch writer.
|
||||
- Keep RAW bytes archive.
|
||||
- Move existing history endpoints onto the new store.
|
||||
- Add diagnostic endpoint for missing VIN/profile cases.
|
||||
|
||||
Phase 2: Ingest association and diagnostics
|
||||
|
||||
- Add local `vin-platform-profile.jsonl` loader.
|
||||
- Use resolved profile before private block parsing.
|
||||
- Add structured parse/reject diagnostics to Kafka metadata and logs.
|
||||
- Add tests for platform login, mapping fallback, and partial private parsing.
|
||||
|
||||
Phase 3: MySQL analytics
|
||||
|
||||
- Add MySQL repository and migrations.
|
||||
- Implement daily stat accumulator with idempotent upsert.
|
||||
- Implement alarm timeline open/update/close logic.
|
||||
- Add integration tests using Testcontainers or a local MySQL profile.
|
||||
|
||||
Phase 4: Production hardening
|
||||
|
||||
- Add backfill/replay commands from DuckDB RAW archive to analytics.
|
||||
- Add retention/export from DuckDB to Parquet for cold storage.
|
||||
- Add dashboards and operational runbook.
|
||||
|
||||
## Acceptance Criteria
|
||||
|
||||
- A valid GB32960 frame received on port 32960 appears in Kafka with RAW bytes or RAW URI.
|
||||
- A valid frame is queryable by `vin + eventTime` from history.
|
||||
- A returned frame can be replayed from `rawArchiveUri` and decoded with the stored profile.
|
||||
- A VIN with only local mapping, and no current platform login, can still use the correct private parser profile.
|
||||
- A private block parse failure does not drop the frame.
|
||||
- Daily MySQL rows update in real time for mileage, total mileage, power consumption, hydrogen consumption, hydrogen rate, and sample metadata.
|
||||
- Alarm timeline rows preserve full alarm start, update, and close history.
|
||||
- Replaying the same Kafka event does not duplicate daily stats or alarm rows.
|
||||
- No secret appears in committed files.
|
||||
|
||||
## References
|
||||
|
||||
- DuckDB Appender: https://duckdb.org/docs/current/data/appender
|
||||
- DuckDB INSERT performance guidance: https://duckdb.org/docs/current/data/insert
|
||||
- MySQL `INSERT ... ON DUPLICATE KEY UPDATE`: https://dev.mysql.com/doc/refman/9.7/en/insert-on-duplicate.html
|
||||
- SAE J2572 overview: https://h2tools.org/fuel-cell-codes-and-standards/sae-j2572-measuring-exhaust-emissions-energy-consumption-and-range
|
||||
Reference in New Issue
Block a user