lingniu-vehicle-ingest/docs/target-architecture.md

# Target Architecture

This document records the production target for `lingniu-vehicle-ingest`.
The project should stay small at runtime: protocol apps ingest and publish,
history consumes and indexes, analytics derives metrics, and business systems
read through explicit APIs or Kafka.

## Active Scope

Active production protocols:

| Protocol | Runtime app | Event topic | Raw topic |
|---|---|---|---|
| GB/T 32960 | `gb32960-ingest-app` | `vehicle.event.gb32960.v1` | `vehicle.raw.gb32960.v1` |
| JT/T 808 | `jt808-ingest-app` | `vehicle.event.jt808.v1` | `vehicle.raw.jt808.v1` |
| Yutong MQTT | `yutong-mqtt-app` | `vehicle.event.mqtt-yutong.v1` | `vehicle.raw.mqtt-yutong.v1` |

Xinda Push is legacy and is not part of the default production surface.
It may remain in source for reference, but new optimization work should target
the active protocols above.

JSATL12 attachment upload is also outside the default production reactor. Keep
it available through the `optional-attachments` profile until an attachment app
is explicitly deployed.

## Goals

- Keep protocol IO, raw archive, history, latest-state, and statistics as
  separate responsibilities.
- Publish normalized Kafka envelopes with VIN as the partition key whenever a
  VIN is known.
- Store complete raw payload metadata and parsed JSON in TDengine `raw_frames`.
- Store query-friendly location rows separately from raw payload JSON.
- Keep derived statistics in metric tables, not protocol-specific daily tables.
- Use Redis only for hot state or streaming calculation state that can be
  rebuilt from Kafka.
- Keep optional compatibility modules outside the production hot path.

## Non-Goals

- Ingest apps do not write business tables.
- Ingest apps do not serve history query APIs.
- Redis is not the long-term historical store.
- `event-file-store` Parquet/DuckDB is not the default history store.
- `telemetry_fields` parsing is not handled by this project when a separate
  field parsing service owns that job; the `telemetry_fields` query API is
  also disabled by default unless `TDENGINE_TELEMETRY_FIELDS_ENABLED=true`.

## Runtime Responsibilities

### Protocol Apps

Apps:

- `gb32960-ingest-app`
- `jt808-ingest-app`
- `yutong-mqtt-app`

Responsibilities:

- Accept protocol traffic.
- Decode, authenticate, acknowledge, and maintain sessions where the protocol
  requires it.
- Archive raw bytes through `sink-archive` when bytes are available.
- Publish parsed event envelopes to Kafka.
- Publish raw envelopes to Kafka.
- Resolve vehicle identity through the shared MySQL identity binding table.

Protocol apps must stay independent from TDengine history readers, Redis state
repositories, and statistic calculators.

### Event Contract

`sink-mq` owns the Kafka protobuf envelope and consumer plumbing.
Consumers should use `EnvelopeConsumerProcessor` so invalid protobuf, skipped
envelopes, and downstream failures are converted to structured results and DLQ
records instead of blocking a partition.

Core identity fields:

| Field | Meaning |
|---|---|
| `event_id` | Parsed event idempotency key |
| `trace_id` | Cross-service trace id |
| `vin` | Vehicle id and Kafka partition key |
| `source` | Source protocol, for example `GB32960`, `JT808`, `YUTONG_MQTT` |
| `event_time_ms` | Device event time |
| `ingest_time_ms` | Platform receive time |
| `raw_uri` | `archive://...` reference to raw bytes when available |

Full-field telemetry uses `TelemetrySnapshot` fields with stable keys such as
`total_mileage_km`, `speed_kmh`, `longitude`, and `latitude`.

### History App

App: `vehicle-history-app`

Responsibilities:

- Consume active protocol event topics.
- Consume active protocol raw topics.
- Write raw frame rows into TDengine `raw_frames`.
- Write compact location rows into TDengine location tables.
- Keep `payloadJson.parsed` on raw rows for full raw inspection.
- Expose paged history APIs for raw frames and locations.
- Keep specialized APIs off by default unless explicitly enabled.

Production default:

- `TDENGINE_HISTORY_ENABLED=true` in deployment.
- `EVENT_FILE_STORE_ENABLED=false`.
- `TDENGINE_TELEMETRY_FIELDS_ENABLED=false`.

`event-file-store` remains an optional compatibility path for old low-level
record APIs or local investigations. It should not be treated as the primary
history backend.

### Analytics App

App: `vehicle-analytics-app`

Responsibilities:

- Consume JT808 event envelopes.
- Calculate 808 daily mileage from the reported GPS total mileage only:

```text
daily_mileage_km = max_total_mileage_km - min_total_mileage_km
calculation_method = JT808_TOTAL_MILEAGE_DIFF
```

- Store the metric in `vehicle_stat_metric` with
  `metric_key = daily_mileage_km`; the local-day minimum and maximum GPS total
  mileage used for the subtraction stay on the same metric row as calculation
  source columns.

There is no JT808-specific daily mileage table and no in-memory production
state-store mode. Restart recovery reads the same metric row, so it does not
require a separate Redis mileage state.

### Latest State

Module: `vehicle-state-service`

This module is outside the default production reactor. Build it with
`-Poptional-latest-state` when Redis latest-state APIs are explicitly needed.

Responsibilities:

- Consume normalized Kafka envelopes.
- Maintain latest state, location, and safety snapshots in Redis.
- Serve latest-state APIs for operational screens.

Redis state must be rebuildable from Kafka replay and should not be used as the
source of truth for historical queries.

## Data Flow

```mermaid
flowchart LR
  vehicle["Vehicles / Platforms"] --> ingest["Protocol apps"]
  ingest --> archive["sink-archive<br/>raw bytes"]
  ingest --> kafka["Kafka<br/>event + raw envelopes"]
  kafka --> history["vehicle-history-app"]
  kafka --> analytics["vehicle-analytics-app"]
  kafka --> state["vehicle-state-service"]
  history --> tdengine["TDengine<br/>raw_frames + locations"]
  analytics --> mysql["MySQL<br/>vehicle_stat_metric"]
  state --> redisState["Redis<br/>latest state"]
```

## Failure Handling

- Protocol apps continue accepting traffic when history, analytics, or Redis
  consumers are unavailable.
- Kafka producer failures are handled by sink retry/circuit-breaker behavior and
  DLQ topics where configured.
- History consumers should commit offsets only after TDengine writes succeed.
- Analytics daily mileage state can recover from `vehicle_stat_metric` or Kafka
  replay.
- Raw archive failures must be observable, but event publication and history
  indexing remain separate concerns.

## Acceptance Criteria

- Active production deployment contains GB32960, JT808, Yutong MQTT,
  vehicle-history, and vehicle-analytics apps.
- Xinda Push is absent from default deployment and history consumer bindings.
- JSATL12 is absent from the default Maven reactor and available through
  `optional-attachments`.
- vehicle-state-service is absent from the default Maven reactor and available
  through `optional-latest-state`.
- raw-archive-store is absent from the default Maven reactor and available
  through `optional-raw-archive-store`.
- event-file-store is absent from the default Maven reactor and available
  through `optional-event-file-store`.
- History APIs read from TDengine by default; Parquet/DuckDB is optional
  compatibility only.
- Raw frame queries can return complete parsed JSON through `payloadJson.parsed`.
- Location queries page over compact rows and reference raw records instead of
  duplicating full raw JSON.
- JT808 daily mileage is stored only in `vehicle_stat_metric`.
- Runtime state needed for 808 mileage is also in `vehicle_stat_metric`, with no
  Redis or production memory state-store mode.
- Build validation covers the active modules and their composition tests.