189 lines
6.7 KiB
Markdown
189 lines
6.7 KiB
Markdown
# Target Architecture
|
|
|
|
This document records the production target for `lingniu-vehicle-ingest`.
|
|
The project should stay small at runtime: protocol apps ingest and publish,
|
|
history consumes and indexes, analytics derives metrics, and business systems
|
|
read through explicit APIs or Kafka.
|
|
|
|
## Active Scope
|
|
|
|
Active production protocols:
|
|
|
|
| Protocol | Runtime app | Event topic | Raw topic |
|
|
|---|---|---|---|
|
|
| GB/T 32960 | `gb32960-ingest-app` | `vehicle.event.gb32960.v1` | `vehicle.raw.gb32960.v1` |
|
|
| JT/T 808 | `jt808-ingest-app` | `vehicle.event.jt808.v1` | `vehicle.raw.jt808.v1` |
|
|
| Yutong MQTT | `yutong-mqtt-app` | `vehicle.event.mqtt-yutong.v1` | `vehicle.raw.mqtt-yutong.v1` |
|
|
|
|
Xinda Push is legacy and is not part of the default production surface.
|
|
It may remain in source for reference, but new optimization work should target
|
|
the active protocols above.
|
|
|
|
## Goals
|
|
|
|
- Keep protocol IO, raw archive, history, latest-state, and statistics as
|
|
separate responsibilities.
|
|
- Publish normalized Kafka envelopes with VIN as the partition key whenever a
|
|
VIN is known.
|
|
- Store complete raw payload metadata and parsed JSON in TDengine `raw_frames`.
|
|
- Store query-friendly location rows separately from raw payload JSON.
|
|
- Keep derived statistics in metric tables, not protocol-specific daily tables.
|
|
- Use Redis only for hot state or streaming calculation state that can be
|
|
rebuilt from Kafka.
|
|
- Keep optional compatibility modules outside the production hot path.
|
|
|
|
## Non-Goals
|
|
|
|
- Ingest apps do not write business tables.
|
|
- Ingest apps do not serve history query APIs.
|
|
- Redis is not the long-term historical store.
|
|
- `event-file-store` Parquet/DuckDB is not the default history store.
|
|
- `telemetry_fields` parsing is not handled by this project when a separate
|
|
field parsing service owns that job.
|
|
|
|
## Runtime Responsibilities
|
|
|
|
### Protocol Apps
|
|
|
|
Apps:
|
|
|
|
- `gb32960-ingest-app`
|
|
- `jt808-ingest-app`
|
|
- `yutong-mqtt-app`
|
|
|
|
Responsibilities:
|
|
|
|
- Accept protocol traffic.
|
|
- Decode, authenticate, acknowledge, and maintain sessions where the protocol
|
|
requires it.
|
|
- Archive raw bytes through `sink-archive` when bytes are available.
|
|
- Publish parsed event envelopes to Kafka.
|
|
- Publish raw envelopes to Kafka.
|
|
- Resolve vehicle identity through the shared MySQL identity binding table.
|
|
|
|
Protocol apps must stay independent from TDengine history readers, Redis state
|
|
repositories, and statistic calculators.
|
|
|
|
### Event Contract
|
|
|
|
`sink-mq` owns the Kafka protobuf envelope and consumer plumbing.
|
|
Consumers should use `EnvelopeConsumerProcessor` so invalid protobuf, skipped
|
|
envelopes, and downstream failures are converted to structured results and DLQ
|
|
records instead of blocking a partition.
|
|
|
|
Core identity fields:
|
|
|
|
| Field | Meaning |
|
|
|---|---|
|
|
| `event_id` | Parsed event idempotency key |
|
|
| `trace_id` | Cross-service trace id |
|
|
| `vin` | Vehicle id and Kafka partition key |
|
|
| `source` | Source protocol, for example `GB32960`, `JT808`, `YUTONG_MQTT` |
|
|
| `event_time_ms` | Device event time |
|
|
| `ingest_time_ms` | Platform receive time |
|
|
| `raw_uri` | `archive://...` reference to raw bytes when available |
|
|
|
|
Full-field telemetry uses `TelemetrySnapshot` fields with stable keys such as
|
|
`total_mileage_km`, `speed_kmh`, `longitude`, and `latitude`.
|
|
|
|
### History App
|
|
|
|
App: `vehicle-history-app`
|
|
|
|
Responsibilities:
|
|
|
|
- Consume active protocol event topics.
|
|
- Consume active protocol raw topics.
|
|
- Write raw frame rows into TDengine `raw_frames`.
|
|
- Write compact location rows into TDengine location tables.
|
|
- Keep `payloadJson.parsed` on raw rows for full raw inspection.
|
|
- Expose paged history APIs for raw frames and locations.
|
|
- Keep specialized APIs off by default unless explicitly enabled.
|
|
|
|
Production default:
|
|
|
|
- `TDENGINE_HISTORY_ENABLED=true` in deployment.
|
|
- `EVENT_FILE_STORE_ENABLED=false`.
|
|
- `TDENGINE_TELEMETRY_FIELDS_ENABLED=false`.
|
|
|
|
`event-file-store` remains an optional compatibility path for old low-level
|
|
record APIs or local investigations. It should not be treated as the primary
|
|
history backend.
|
|
|
|
### Analytics App
|
|
|
|
App: `vehicle-analytics-app`
|
|
|
|
Responsibilities:
|
|
|
|
- Consume JT808 event envelopes.
|
|
- Calculate 808 daily mileage from the reported GPS total mileage only:
|
|
|
|
```text
|
|
daily_mileage_km = last_total_mileage_km - first_total_mileage_km
|
|
calculation_method = JT808_TOTAL_MILEAGE_DIFF
|
|
```
|
|
|
|
- Store the metric in `vehicle_stat_metric` with
|
|
`metric_key = daily_mileage_km`; the first and latest GPS total mileage used
|
|
for the subtraction stay on the same metric row as calculation source columns.
|
|
|
|
There is no JT808-specific daily mileage table and no in-memory production
|
|
state-store mode. Restart recovery reads the same metric row, so it does not
|
|
require a separate Redis mileage state.
|
|
|
|
### Latest State
|
|
|
|
Module: `vehicle-state-service`
|
|
|
|
Responsibilities:
|
|
|
|
- Consume normalized Kafka envelopes.
|
|
- Maintain latest state, location, and safety snapshots in Redis.
|
|
- Serve latest-state APIs for operational screens.
|
|
|
|
Redis state must be rebuildable from Kafka replay and should not be used as the
|
|
source of truth for historical queries.
|
|
|
|
## Data Flow
|
|
|
|
```mermaid
|
|
flowchart LR
|
|
vehicle["Vehicles / Platforms"] --> ingest["Protocol apps"]
|
|
ingest --> archive["sink-archive<br/>raw bytes"]
|
|
ingest --> kafka["Kafka<br/>event + raw envelopes"]
|
|
kafka --> history["vehicle-history-app"]
|
|
kafka --> analytics["vehicle-analytics-app"]
|
|
kafka --> state["vehicle-state-service"]
|
|
history --> tdengine["TDengine<br/>raw_frames + locations"]
|
|
analytics --> mysql["MySQL<br/>vehicle_stat_metric"]
|
|
state --> redisState["Redis<br/>latest state"]
|
|
```
|
|
|
|
## Failure Handling
|
|
|
|
- Protocol apps continue accepting traffic when history, analytics, or Redis
|
|
consumers are unavailable.
|
|
- Kafka producer failures are handled by sink retry/circuit-breaker behavior and
|
|
DLQ topics where configured.
|
|
- History consumers should commit offsets only after TDengine writes succeed.
|
|
- Analytics daily mileage state can recover from `vehicle_stat_metric` or Kafka
|
|
replay.
|
|
- Raw archive failures must be observable, but event publication and history
|
|
indexing remain separate concerns.
|
|
|
|
## Acceptance Criteria
|
|
|
|
- Active production deployment contains GB32960, JT808, Yutong MQTT,
|
|
vehicle-history, and vehicle-analytics apps.
|
|
- Xinda Push is absent from default deployment and history consumer bindings.
|
|
- History APIs read from TDengine by default; Parquet/DuckDB is optional
|
|
compatibility only.
|
|
- Raw frame queries can return complete parsed JSON through `payloadJson.parsed`.
|
|
- Location queries page over compact rows and reference raw records instead of
|
|
duplicating full raw JSON.
|
|
- JT808 daily mileage is stored only in `vehicle_stat_metric`.
|
|
- Runtime state needed for 808 mileage is also in `vehicle_stat_metric`, with no
|
|
Redis or production memory state-store mode.
|
|
- Build validation covers the active modules and their composition tests.
|