lingniu-vehicle-ingest/docs/target-architecture.md

Target Architecture

This document records the production target for lingniu-vehicle-ingest. The project should stay small at runtime: protocol apps ingest and publish, history consumes and indexes, analytics derives metrics, and business systems read through explicit APIs or Kafka.

Active Scope

Active production protocols:

Protocol	Runtime app	Event topic	Raw topic
GB/T 32960	gb32960-ingest-app	vehicle.event.gb32960.v1	vehicle.raw.gb32960.v1
JT/T 808	jt808-ingest-app	vehicle.event.jt808.v1	vehicle.raw.jt808.v1
Yutong MQTT	yutong-mqtt-app	vehicle.event.mqtt-yutong.v1	vehicle.raw.mqtt-yutong.v1

Xinda Push is legacy and is not part of the default production surface. It may remain in source for reference, but new optimization work should target the active protocols above.

Goals

• Keep protocol IO, raw archive, history, latest-state, and statistics as separate responsibilities.
• Publish normalized Kafka envelopes with VIN as the partition key whenever a VIN is known.
• Store complete raw payload metadata and parsed JSON in TDengine raw_frames.
• Store query-friendly location rows separately from raw payload JSON.
• Keep derived statistics in metric tables, not protocol-specific daily tables.
• Use Redis only for hot state or streaming calculation state that can be rebuilt from Kafka.
• Keep optional compatibility modules outside the production hot path.

Non-Goals

• Ingest apps do not write business tables.
• Ingest apps do not serve history query APIs.
• Redis is not the long-term historical store.
• event-file-store Parquet/DuckDB is not the default history store.
• telemetry_fields parsing is not handled by this project when a separate field parsing service owns that job.

Runtime Responsibilities

Protocol Apps

Apps:

• gb32960-ingest-app
• jt808-ingest-app
• yutong-mqtt-app

Responsibilities:

• Accept protocol traffic.
• Decode, authenticate, acknowledge, and maintain sessions where the protocol requires it.
• Archive raw bytes through sink-archive when bytes are available.
• Publish parsed event envelopes to Kafka.
• Publish raw envelopes to Kafka.
• Resolve vehicle identity through the shared MySQL identity binding table.

Protocol apps must stay independent from TDengine history readers, Redis state repositories, and statistic calculators.

Event Contract

sink-mq owns the Kafka protobuf envelope and consumer plumbing. Consumers should use EnvelopeConsumerProcessor so invalid protobuf, skipped envelopes, and downstream failures are converted to structured results and DLQ records instead of blocking a partition.

Core identity fields:

Field	Meaning
event_id	Parsed event idempotency key
trace_id	Cross-service trace id
vin	Vehicle id and Kafka partition key
source	Source protocol, for example GB32960, JT808, YUTONG_MQTT
event_time_ms	Device event time
ingest_time_ms	Platform receive time
raw_uri	archive://... reference to raw bytes when available

Full-field telemetry uses TelemetrySnapshot fields with stable keys such as total_mileage_km, speed_kmh, longitude, and latitude.

History App

App: vehicle-history-app

Responsibilities:

• Consume active protocol event topics.
• Consume active protocol raw topics.
• Write raw frame rows into TDengine raw_frames.
• Write compact location rows into TDengine location tables.
• Keep payloadJson.parsed on raw rows for full raw inspection.
• Expose paged history APIs for raw frames and locations.
• Keep specialized APIs off by default unless explicitly enabled.

Production default:

• TDENGINE_HISTORY_ENABLED=true in deployment.
• EVENT_FILE_STORE_ENABLED=false.
• TDENGINE_TELEMETRY_FIELDS_ENABLED=false.

event-file-store remains an optional compatibility path for old low-level record APIs or local investigations. It should not be treated as the primary history backend.

Analytics App

App: vehicle-analytics-app

Responsibilities:

• Consume JT808 event envelopes.
• Calculate 808 daily mileage from the reported GPS total mileage only:

daily_mileage_km = last_total_mileage_km - first_total_mileage_km
calculation_method = JT808_TOTAL_MILEAGE_DIFF

• Store the metric in vehicle_stat_metric with metric_key = daily_mileage_km; the first and latest GPS total mileage used for the subtraction stay on the same metric row as calculation source columns.

There is no JT808-specific daily mileage table and no in-memory production state-store mode. Restart recovery reads the same metric row, so it does not require a separate Redis mileage state.

Latest State

Module: vehicle-state-service

Responsibilities:

• Consume normalized Kafka envelopes.
• Maintain latest state, location, and safety snapshots in Redis.
• Serve latest-state APIs for operational screens.

Redis state must be rebuildable from Kafka replay and should not be used as the source of truth for historical queries.

Data Flow

flowchart LR
  vehicle["Vehicles / Platforms"] --> ingest["Protocol apps"]
  ingest --> archive["sink-archive<br/>raw bytes"]
  ingest --> kafka["Kafka<br/>event + raw envelopes"]
  kafka --> history["vehicle-history-app"]
  kafka --> analytics["vehicle-analytics-app"]
  kafka --> state["vehicle-state-service"]
  history --> tdengine["TDengine<br/>raw_frames + locations"]
  analytics --> mysql["MySQL<br/>vehicle_stat_metric"]
  state --> redisState["Redis<br/>latest state"]

Failure Handling

• Protocol apps continue accepting traffic when history, analytics, or Redis consumers are unavailable.
• Kafka producer failures are handled by sink retry/circuit-breaker behavior and DLQ topics where configured.
• History consumers should commit offsets only after TDengine writes succeed.
• Analytics daily mileage state can recover from vehicle_stat_metric or Kafka replay.
• Raw archive failures must be observable, but event publication and history indexing remain separate concerns.

Acceptance Criteria

• Active production deployment contains GB32960, JT808, Yutong MQTT, vehicle-history, and vehicle-analytics apps.
• Xinda Push is absent from default deployment and history consumer bindings.
• History APIs read from TDengine by default; Parquet/DuckDB is optional compatibility only.
• Raw frame queries can return complete parsed JSON through payloadJson.parsed.
• Location queries page over compact rows and reference raw records instead of duplicating full raw JSON.
• JT808 daily mileage is stored only in vehicle_stat_metric.
• Runtime state needed for 808 mileage is also in vehicle_stat_metric, with no Redis or production memory state-store mode.
• Build validation covers the active modules and their composition tests.