shishengliang/lingniu-vehicle-ingest
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
4eaaa88e9c2ea7143dc6b7684d9896187e18a434
lingniu-vehicle-ingest/docs/superpowers/specs/2026-06-23-gb32960-production-readiness-design.md
lingniu 4eaaa88e9c chore: remove xinda ingestion modules
2026-07-01 18:22:51 +08:00

16 KiB
Raw Blame History

Superseded: This 2026-06-23 DuckDB-based design is historical context. Use docs/target-architecture.md and docs/superpowers/specs/2026-06-29-vehicle-ingest-redesign.md for the current production architecture: TDengine is the default hot history store, event-file-store and Xinda Push have been removed from the production codebase.

GB32960 Production Readiness Design

Goal

Make the GB32960 ingestion, history, and analytics services production-ready:

  • receive and parse GB32960 traffic quickly and accurately;
  • maximize VIN-to-platform/vendor-profile association through platform login and local mappings;
  • store complete RAW frames and queryable telemetry history with DuckDB;
  • support full frame replay from persisted RAW bytes;
  • compute real-time daily vehicle metrics and alarm timelines into MySQL;
  • reduce service coupling and improve cohesion inside each module.

Non-Goals

  • Do not put database credentials in code, YAML files, tests, docs, or commits.
  • Do not make MySQL the source for full telemetry history. MySQL is for derived daily metrics and alarm timelines.
  • Do not require every private GB32960 extension to be perfect before storing data. RAW bytes must always be kept when a valid frame is received.

Production Architecture

flowchart LR
  tcp["GB32960 TCP:32960"] --> ingest["gb32960-ingest-app"]
  mapping["local vin-platform-profile.jsonl"] --> ingest
  ingest --> kafkaEvent["Kafka vehicle.event.gb32960.v1"]
  ingest --> kafkaRaw["Kafka vehicle.raw.gb32960.v1"]
  kafkaEvent --> history["vehicle-history-app"]
  kafkaRaw --> history
  history --> duckdb["DuckDB hot history"]
  history --> raw["RAW frame archive"]
  kafkaEvent --> analytics["vehicle-analytics-app"]
  analytics --> mysql["MySQL daily stats + alarms"]

Service Boundaries

1. gb32960-ingest-app

Responsibilities:

  • Netty TCP accept, frame splitting, GB32960 decode, auth, ACK, and Kafka production.
  • Platform login handling and connection-local platformAccount.
  • VIN-to-platform/vendor-profile resolution.
  • Produce structured telemetry, session, alarm, diagnostics, and RAW envelope messages.

It must not:

  • write DuckDB, MySQL, or local archive files;
  • run historical queries;
  • run daily statistics.

ACK policy:

  • login/report-like GB32960 commands use durable ACK: ACK only after Kafka dispatch succeeds;
  • malformed bytes that cannot form a valid frame are dropped with diagnostics;
  • valid frames with partial private-block parse failures still go to Kafka with parseStatus=PARTIAL.

VIN/profile resolution priority:

  1. active platform login on the current TCP channel;
  2. exact VIN entry from local mapping file;
  3. VIN prefix entry from local mapping file;
  4. configured platform extension rule;
  5. standard GB32960 profile.

Local mapping file format:

{"vin":"LNXNEGRR2SR321390","platformAccount":"Hyundai","vendorProfile":"guangdong-fc","enabled":true}
{"vinPrefix":"LNXNEGRR","platformAccount":"Hyundai","vendorProfile":"guangdong-fc","enabled":true}

The mapping file is loaded at startup and can be reloaded by an actuator endpoint or file watcher in a later phase.

2. vehicle-history-app

Responsibilities:

  • Consume Kafka event and RAW topics.
  • Persist RAW bytes to an append-only archive.
  • Persist query indexes and compact telemetry points to DuckDB.
  • Serve high-performance query APIs:
    • full frame list by vin + eventTime;
    • telemetry fields by vin + eventTime;
    • one-frame replay by rawArchiveUri;
    • parse diagnostics and missing-profile troubleshooting.

It must not:

  • listen on GB32960 TCP;
  • run MySQL statistics;
  • mutate daily business metrics.

3. vehicle-analytics-app

Responsibilities:

  • Consume Kafka event topic.
  • Maintain real-time daily stats in MySQL.
  • Maintain full alarm timelines in MySQL.
  • Expose metrics query APIs and operational health.

It must not:

  • parse RAW frames for normal operation;
  • write DuckDB history;
  • own GB32960 TCP connectivity.

Kafka Contract

Keep Protobuf VehicleEnvelope, but make the producer contract stricter:

  • eventId: stable idempotency key, deterministic from protocol, VIN, command, event time, raw checksum, and sequence when available.
  • vin: required for vehicle frames; platform-only commands use _platform or empty plus platformAccount.
  • eventTimeMs: GB32960 data collection time when available.
  • ingestTimeMs: service receive time.
  • metadata.platformAccount: resolved platform account.
  • metadata.vendorProfile: selected parser profile.
  • metadata.parseStatus: OK, PARTIAL, or FAILED.
  • metadata.parseErrorCode: stable code for failures.
  • raw_archive.data: present on RAW topic only; event topic keeps rawArchiveUri pointer.
  • telemetry_snapshot: standard field projection for downstream services.

Consumer rule:

  • History consumes both event and RAW topics.
  • Analytics consumes only event topic.
  • Consumer offset is committed only after the target store commit succeeds.
  • Reprocessing must be safe through deterministic ids and database upsert/ignore semantics.

DuckDB History Design

Use DuckDB as the hot historical database, not a Parquet-rewrite sidecar.

DuckDB official guidance says row-by-row insert loops are inefficient for bulk loading, and Appender/batched writes are the intended high-throughput path. Therefore history writes use a single writer worker with bounded batches and one DuckDB connection.

Tables

gb32960_frame_index

CREATE TABLE IF NOT EXISTS gb32960_frame_index (
  event_id VARCHAR PRIMARY KEY,
  vin VARCHAR NOT NULL,
  platform_account VARCHAR,
  vendor_profile VARCHAR,
  command VARCHAR NOT NULL,
  command_code INTEGER NOT NULL,
  event_time TIMESTAMP NOT NULL,
  event_time_ms BIGINT NOT NULL,
  ingest_time TIMESTAMP NOT NULL,
  ingest_time_ms BIGINT NOT NULL,
  partition_date DATE NOT NULL,
  raw_archive_uri VARCHAR NOT NULL,
  raw_checksum VARCHAR,
  raw_size_bytes BIGINT NOT NULL,
  parse_status VARCHAR NOT NULL,
  parse_error_code VARCHAR,
  block_count INTEGER NOT NULL,
  metadata_json VARCHAR NOT NULL
);

CREATE INDEX IF NOT EXISTS gb32960_frame_vin_time_idx
  ON gb32960_frame_index(vin, event_time_ms);

CREATE INDEX IF NOT EXISTS gb32960_frame_raw_uri_idx
  ON gb32960_frame_index(raw_archive_uri);

CREATE INDEX IF NOT EXISTS gb32960_frame_partition_time_idx
  ON gb32960_frame_index(partition_date, event_time_ms);

gb32960_telemetry_point

CREATE TABLE IF NOT EXISTS gb32960_telemetry_point (
  event_id VARCHAR NOT NULL,
  vin VARCHAR NOT NULL,
  event_time TIMESTAMP NOT NULL,
  event_time_ms BIGINT NOT NULL,
  ingest_time TIMESTAMP NOT NULL,
  platform_account VARCHAR,
  vendor_profile VARCHAR,
  field_key VARCHAR NOT NULL,
  value_type VARCHAR NOT NULL,
  value_text VARCHAR NOT NULL,
  value_num DOUBLE,
  unit VARCHAR,
  quality VARCHAR NOT NULL,
  raw_archive_uri VARCHAR NOT NULL,
  PRIMARY KEY(event_id, field_key)
);

CREATE INDEX IF NOT EXISTS gb32960_point_vin_field_time_idx
  ON gb32960_telemetry_point(vin, field_key, event_time_ms);

gb32960_alarm_frame

CREATE TABLE IF NOT EXISTS gb32960_alarm_frame (
  event_id VARCHAR PRIMARY KEY,
  vin VARCHAR NOT NULL,
  event_time TIMESTAMP NOT NULL,
  event_time_ms BIGINT NOT NULL,
  level VARCHAR NOT NULL,
  active_bits_json VARCHAR NOT NULL,
  fault_codes_json VARCHAR NOT NULL,
  raw_archive_uri VARCHAR NOT NULL
);

CREATE INDEX IF NOT EXISTS gb32960_alarm_vin_time_idx
  ON gb32960_alarm_frame(vin, event_time_ms);

RAW Archive

RAW archive remains file/object based:

archive://yyyy/MM/dd/GB32960/{vin}/{event_id}.bin

The archive writer verifies:

  • stored byte length equals envelope raw_archive.size_bytes;
  • checksum matches;
  • URI is unique for the deterministic event id.

Frame replay path:

  1. query gb32960_frame_index by rawArchiveUri;
  2. read RAW bytes from archive;
  3. resolve platformAccount/vendorProfile from index metadata;
  4. decode using current parser;
  5. return decoded frame plus original indexed parse diagnostics.

Query APIs

Keep existing endpoints but back them by the new DuckDB tables:

  • GET /api/event-history/gb32960/frames
    • full frame query by VIN/time.
  • GET /api/event-history/gb32960/frame
    • one RAW frame replay by rawArchiveUri.
  • GET /api/event-history/gb32960/telemetry-snapshots
    • compact telemetry snapshot by VIN/time.
  • GET /api/event-history/gb32960/telemetry-fields
    • field projection optimized by gb32960_telemetry_point.
  • GET /api/event-history/gb32960/diagnostics
    • parse status and missing-profile investigation by VIN/time.

Time semantics:

  • eventTime is business query time.
  • ingestTime is kept for operational replay, late-arrival detection, and Kafka lag investigation.
  • Dates without timezone are interpreted in Asia/Shanghai.

MySQL Analytics Design

Credentials are supplied only by environment variables:

MYSQL_HOST
MYSQL_PORT
MYSQL_DATABASE
MYSQL_USERNAME
MYSQL_PASSWORD

Daily derived metrics are stored in the common metric table, not in a protocol-specific daily-mileage table.

CREATE TABLE IF NOT EXISTS vehicle_stat_metric (
  vin VARCHAR(64) NOT NULL,
  stat_date DATE NOT NULL,
  metric_key VARCHAR(64) NOT NULL,
  metric_value DECIMAL(18,6) NULL,
  metric_unit VARCHAR(16) NOT NULL,
  calculation_method VARCHAR(64) NOT NULL,
  created_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
  updated_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
  PRIMARY KEY (vin, stat_date, metric_key)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4;

vehicle_alarm_timeline

CREATE TABLE IF NOT EXISTS vehicle_alarm_timeline (
  alarm_id VARCHAR(128) NOT NULL,
  vin VARCHAR(32) NOT NULL,
  alarm_key VARCHAR(128) NOT NULL,
  level VARCHAR(32) NOT NULL,
  start_time DATETIME(3) NOT NULL,
  end_time DATETIME(3),
  last_seen_time DATETIME(3) NOT NULL,
  start_raw_archive_uri VARCHAR(512),
  last_raw_archive_uri VARCHAR(512),
  status VARCHAR(16) NOT NULL,
  details_json JSON,
  updated_at DATETIME(3) NOT NULL,
  PRIMARY KEY (alarm_id),
  KEY vehicle_alarm_vin_time_idx (vin, start_time),
  KEY vehicle_alarm_open_idx (vin, status, alarm_key)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4;

vehicle_stat_event_dedup

CREATE TABLE IF NOT EXISTS vehicle_stat_event_dedup (
  event_id VARCHAR(128) NOT NULL,
  consumer_group VARCHAR(128) NOT NULL,
  processed_at DATETIME(3) NOT NULL,
  PRIMARY KEY (event_id, consumer_group)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4;

Writes use INSERT ... ON DUPLICATE KEY UPDATE so replayed Kafka messages update the same daily row instead of creating duplicates.

Daily Metric Algorithms

All daily statistics are keyed by eventTime in Asia/Shanghai.

Total Mileage

Use the latest valid totalMileageKm seen for the day.

Validation:

  • reject negative values;
  • reject impossible jumps based on configurable max speed and elapsed time;
  • keep the sample as diagnostic if rejected.

Daily Mileage

JT808 daily mileage uses the GPS total mileage reported in location additional information. The local-day minimum and maximum GPS total mileage values rewrite the same metric row, so ordered streaming and historical replay use one calculation path.

dailyMileage = today.maxTotalMileage - today.minTotalMileage
calculationMethod = JT808_TOTAL_MILEAGE_DIFF

The result is stored in vehicle_stat_metric with metric_key=daily_mileage_km. There is no JT808-specific daily mileage table, no Redis mileage state, and no previous-day baseline calculation.

Daily Power Consumption

Preferred:

  • vendor cumulative power-consumption field if available.

Fallback:

deltaKwh = sum(max(0, totalVoltageV * totalCurrentA) * deltaSeconds / 3600000)

The sign convention is configurable per vendor profile because GB32960 vehicle current may be positive or negative depending on charging/discharging representation.

Daily Hydrogen Consumption

Use a tiered market-practical algorithm instead of a single fixed formula:

  1. DIRECT_COUNTER: vendor cumulative hydrogen consumption or remaining hydrogen mass delta.
  2. PVT_MASS_DELTA: pressure/temperature/tank-volume mass estimate when storage data is available.
  3. FUEL_CELL_ENERGY_MODEL: integrate fuel-cell electric output and divide by calibrated efficiency.
  4. UNKNOWN: keep null when required signals are missing.

The PVT approach follows the industry pattern behind compressed hydrogen consumption tests, where pressure, temperature, and tank volume can be used to estimate hydrogen mass. The implementation must expose calibration constants per vehicle model/profile:

{
  "vinPrefix": "LNXNEGRR",
  "tankVolumeLiter": 140,
  "hydrogenModel": "PVT_MASS_DELTA",
  "fuelCellEfficiency": 0.52,
  "currentDischargeSign": "POSITIVE"
}

Daily Hydrogen Consumption Rate

dailyHydrogenKgPer100km = dailyHydrogenKg / dailyMileageKm * 100

If mileage is missing or less than the configured minimum distance, keep the rate null and set quality to INSUFFICIENT_DISTANCE.

Alarm Timeline

For each telemetry frame:

  • derive active alarm keys from alarm bits and fault code arrays;
  • open a timeline row when a key first appears;
  • update last_seen_time while it remains active;
  • close open alarms when the key disappears after a configurable grace window;
  • keep start/end raw archive URIs for replay.

Reliability and Observability

Required metrics:

  • TCP active connections;
  • frames received per command;
  • decode failures by code;
  • missing platform login rejections;
  • vendor profile resolution source;
  • Kafka send latency and failures;
  • history write batch size, latency, failure count;
  • DuckDB query latency by endpoint;
  • MySQL upsert latency and failure count;
  • consumer lag by group/topic/partition;
  • late sample count by VIN/date.

Required logs:

  • one structured line for every dropped or rejected valid frame;
  • no per-frame INFO spam for successful reports after first frame per VIN/profile unless debug is enabled;
  • include vin, platformAccount, command, eventId, reasonCode, and peer.

Migration Plan

Phase 1: History foundation

  • Replace Parquet rewrite store with DuckDB hot tables and batch writer.
  • Keep RAW bytes archive.
  • Move existing history endpoints onto the new store.
  • Add diagnostic endpoint for missing VIN/profile cases.

Phase 2: Ingest association and diagnostics

  • Add local vin-platform-profile.jsonl loader.
  • Use resolved profile before private block parsing.
  • Add structured parse/reject diagnostics to Kafka metadata and logs.
  • Add tests for platform login, mapping fallback, and partial private parsing.

Phase 3: MySQL analytics

  • Add MySQL repository and migrations.
  • Implement daily stat accumulator with idempotent upsert.
  • Implement alarm timeline open/update/close logic.
  • Add integration tests using Testcontainers or a local MySQL profile.

Phase 4: Production hardening

  • Add backfill/replay commands from DuckDB RAW archive to analytics.
  • Add retention/export from DuckDB to Parquet for cold storage.
  • Add dashboards and operational runbook.

Acceptance Criteria

  • A valid GB32960 frame received on port 32960 appears in Kafka with RAW bytes or RAW URI.
  • A valid frame is queryable by vin + eventTime from history.
  • A returned frame can be replayed from rawArchiveUri and decoded with the stored profile.
  • A VIN with only local mapping, and no current platform login, can still use the correct private parser profile.
  • A private block parse failure does not drop the frame.
  • Daily MySQL rows update in real time for mileage, total mileage, power consumption, hydrogen consumption, hydrogen rate, and sample metadata.
  • Alarm timeline rows preserve full alarm start, update, and close history.
  • Replaying the same Kafka event does not duplicate daily stats or alarm rows.
  • No secret appears in committed files.

References

Reference in New Issue View Git Blame Copy Permalink