shishengliang/lingniu-vehicle-ingest
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs: align target architecture with production flow

Browse Source
This commit is contained in:
lingniu
2026-07-01 02:44:57 +08:00
parent e3dfa9d1c0
commit ee549f5ba2
1 changed files with 133 additions and 217 deletions
Download Patch File Download Diff File Expand all files Collapse all files

350
docs/target-architecture.md
View File

@@ -1,271 +1,187 @@
# Target Architecture # Target Architecture
This document defines the target architecture for Lingniu vehicle ingest. The This document records the production target for `lingniu-vehicle-ingest`.
main goal is to keep protocol ingest, raw archive, historical detail storage, The project should stay small at runtime: protocol apps ingest and publish,
realtime state, and statistics as separate responsibilities connected by stable history consumes and indexes, analytics derives metrics, and business systems
event contracts. read through explicit APIs or Kafka.
## Active Scope
Active production protocols:
| Protocol | Runtime app | Event topic | Raw topic |
|---|---|---|---|
| GB/T 32960 | `gb32960-ingest-app` | `vehicle.event.gb32960.v1` | `vehicle.raw.gb32960.v1` |
| JT/T 808 | `jt808-ingest-app` | `vehicle.event.jt808.v1` | `vehicle.raw.jt808.v1` |
| Yutong MQTT | `yutong-mqtt-app` | `vehicle.event.mqtt-yutong.v1` | `vehicle.raw.mqtt-yutong.v1` |
Xinda Push is legacy and is not part of the default production surface.
It may remain in source for reference, but new optimization work should target
the active protocols above.
## Goals ## Goals
- The ingest service receives protocol messages, archives raw frames, and - Keep protocol IO, raw archive, history, latest-state, and statistics as
publishes normalized full-field events to Kafka. separate responsibilities.
- Historical detail storage consumes Kafka events and stores raw parsed event - Publish normalized Kafka envelopes with VIN as the partition key whenever a
details in Parquet, queried through DuckDB. VIN is known.
- Realtime vehicle state consumes Kafka events and writes hot state to Redis. - Store complete raw payload metadata and parsed JSON in TDengine `raw_frames`.
- Statistics consumes Kafka events and calculates daily operational metrics by - Store query-friendly location rows separately from raw payload JSON.
configurable rules. - Keep derived statistics in metric tables, not protocol-specific daily tables.
- Kafka consumers expose non-throwing ingest results for bad protobuf payloads, - Use Redis only for hot state or streaming calculation state that can be
unsupported envelopes, and downstream failures so one bad record does not rebuilt from Kafka.
block a vehicle partition. - Keep optional compatibility modules outside the production hot path.
- Consumer services expose `EnvelopeConsumerProcessor` beans. Kafka workers or
framework listeners should pass topic, partition, offset, key, and value into
the processor; `SKIPPED`, `INVALID_ENVELOPE`, and `FAILED` results are routed
through `EnvelopeDeadLetterSink`.
- `sink-mq` owns Kafka-specific consumer plumbing: `KafkaEnvelopeConsumerFactory`
creates one independent Kafka consumer per service processor binding,
`KafkaEnvelopeConsumerRunner` manages the long-running poll loop, and
`KafkaEnvelopeDeadLetterSink` publishes failed records to the configured DLQ
topic with diagnostic headers.
- Business consumers depend on internal fields, not protocol-specific field
names.
## Non-Goals ## Non-Goals
- The ingest service must not write business tables. - Ingest apps do not write business tables.
- The ingest service must not serve historical detail queries. - Ingest apps do not serve history query APIs.
- Parquet history storage must not store derived daily statistics. - Redis is not the long-term historical store.
- Redis hot state must not become the long-term historical store. - `event-file-store` Parquet/DuckDB is not the default history store.
- Statistics services must not parse GB/T 32960 frames directly. - `telemetry_fields` parsing is not handled by this project when a separate
field parsing service owns that job.
## Service Boundaries ## Runtime Responsibilities
### Ingest Service ### Protocol Apps
The ingest service owns protocol IO and event publication. Apps:
- `gb32960-ingest-app`
- `jt808-ingest-app`
- `yutong-mqtt-app`
Responsibilities: Responsibilities:
- Accept GB/T 32960, JT/T 808, MQTT, Xinda Push, and future inbound protocols. - Accept protocol traffic.
- Decode frames and acknowledge protocol messages where required. - Decode, authenticate, acknowledge, and maintain sessions where the protocol
- Emit `RawArchive` before deduplication and rate limiting when raw bytes are requires it.
available. - Archive raw bytes through `sink-archive` when bytes are available.
- Write raw bytes through `sink-archive`. - Publish parsed event envelopes to Kafka.
- Map protocol messages into internal telemetry events. - Publish raw envelopes to Kafka.
- Publish internal events to Kafka with VIN as the partition key. - Resolve vehicle identity through the shared MySQL identity binding table.
The ingest service must not depend on Redis, DuckDB query endpoints, or Protocol apps must stay independent from TDengine history readers, Redis state
statistics rule repositories. repositories, and statistic calculators.
### Event Contract ### Event Contract
The event contract is the only dependency shared by ingest, history, realtime `sink-mq` owns the Kafka protobuf envelope and consumer plumbing.
state, and statistics. Consumers should use `EnvelopeConsumerProcessor` so invalid protobuf, skipped
envelopes, and downstream failures are converted to structured results and DLQ
records instead of blocking a partition.
Core event identity: Core identity fields:
| Field | Meaning | | Field | Meaning |
|---|---| |---|---|
| `event_id` | Idempotency key for the parsed event | | `event_id` | Parsed event idempotency key |
| `trace_id` | Cross-service trace id | | `trace_id` | Cross-service trace id |
| `vin` | Vehicle VIN and Kafka partition key | | `vin` | Vehicle id and Kafka partition key |
| `source_protocol` | Protocol source such as `GB32960`; consumers may map unknown future source names to `UNKNOWN` and preserve the original source in metadata | | `source` | Source protocol, for example `GB32960`, `JT808`, `YUTONG_MQTT` |
| `protocol_version` | Source protocol version | | `event_time_ms` | Device event time |
| `event_type` | Business event type, for example `REALTIME`, `LOCATION`, `ALARM` | | `ingest_time_ms` | Platform receive time |
| `event_time` | Device collection time | | `raw_uri` | `archive://...` reference to raw bytes when available |
| `ingest_time` | Platform receive time |
| `raw_archive_uri` | Optional reference to archived raw bytes |
Full-field telemetry is represented as internal field values: Full-field telemetry uses `TelemetrySnapshot` fields with stable keys such as
`total_mileage_km`, `speed_kmh`, `longitude`, and `latitude`.
| Field | Meaning | ### History App
|---|---|
| `field_key` | Stable internal field key such as `total_mileage_km` |
| `value_type` | `STRING`, `DOUBLE`, `LONG`, `BOOLEAN`, `INSTANT`, `JSON` |
| `value` | String representation used for durable serialization |
| `unit` | Stable unit when applicable |
| `quality` | `GOOD`, `ESTIMATED`, `INVALID`, `MISSING` |
| `source_path` | Protocol source path, for traceability |
The internal field list is defined in App: `vehicle-history-app`
`docs/vehicle-telemetry-internal-fields.md`. New protocol-specific fields start
as extension fields and are promoted only after a business meaning and unit are
stable.
### Historical Detail Service
The historical detail service owns raw parsed event detail storage and exports.
Responsibilities: Responsibilities:
- Consume normalized telemetry events from Kafka. - Consume active protocol event topics.
- Store events in Parquet partitioned by protocol and event date. - Consume active protocol raw topics.
- Query Parquet through DuckDB. - Write raw frame rows into TDengine `raw_frames`.
- Provide HTTP APIs for date-range query and export. - Write compact location rows into TDengine location tables.
- Support ascending and descending time order. - Keep `payloadJson.parsed` on raw rows for full raw inspection.
- Support multi-day concatenated export. - Expose paged history APIs for raw frames and locations.
- Keep specialized APIs off by default unless explicitly enabled.
Storage partition: Production default:
- `TDENGINE_HISTORY_ENABLED=true` in deployment.
- `EVENT_FILE_STORE_ENABLED=false`.
- `TDENGINE_TELEMETRY_FIELDS_ENABLED=false`.
`event-file-store` remains an optional compatibility path for old low-level
record APIs or local investigations. It should not be treated as the primary
history backend.
### Analytics App
App: `vehicle-analytics-app`
Responsibilities:
- Consume JT808 event envelopes.
- Calculate 808 daily mileage from the reported GPS total mileage only:
```text ```text
<root>/ daily_mileage_km = last_total_mileage_km - first_total_mileage_km
protocol=GB32960/ calculation_method = JT808_TOTAL_MILEAGE_DIFF
date=2026-06-22/
part-*.parquet
``` ```
The service stores detail events and field snapshots only. It does not calculate - Store the metric in `vehicle_stat_metric` with
or store daily mileage, daily hydrogen usage, or other derived metrics. `metric_key = daily_mileage_km`.
- Keep rolling daily calculation state in Redis.
### Realtime State Service There is no JT808-specific daily mileage table and no in-memory production
state-store mode.
The realtime state service owns hot vehicle state. ### Latest State
Module: `vehicle-state-service`
Responsibilities: Responsibilities:
- Consume normalized telemetry events from Kafka. - Consume normalized Kafka envelopes.
- Keep latest vehicle state by VIN in Redis. - Maintain latest state, location, and safety snapshots in Redis.
- Keep latest location by VIN in Redis. - Serve latest-state APIs for operational screens.
- Keep latest safety state and hydrogen leak state by VIN in Redis.
- Optionally keep short recent windows for operations screens.
Suggested Redis keys: Redis state must be rebuildable from Kafka replay and should not be used as the
source of truth for historical queries.
| Key | Value |
|---|---|
| `vehicle:state:{vin}` | Latest full state snapshot |
| `vehicle:location:{vin}` | Latest location snapshot |
| `vehicle:safety:{vin}` | Latest safety snapshot |
| `vehicle:event:last:{vin}` | Latest event identity and timestamps |
Redis is for millisecond-level reads and recent state. It is not the historical
source of truth.
Current module: `vehicle-state-service`.
Implemented boundaries:
- `VehicleStateEnvelopeIngestor` parses Kafka envelope bytes.
- `VehicleStateUpdater` maps full-field telemetry snapshots to Redis JSON
payloads.
- `VehicleStateRepository` isolates storage from event parsing.
- `VehicleStateController` exposes read-only latest state APIs.
### Statistics Service
The statistics service owns derived operational metrics.
Responsibilities:
- Consume normalized telemetry events from Kafka.
- Maintain day-level vehicle point state needed for statistics.
- Run scheduled and event-triggered calculations.
- Store derived results in the common metric table.
Current JT808 daily mileage strategy:
| Strategy | Formula |
|---|---|
| `JT808_TOTAL_MILEAGE_DIFF` | Current local day last valid JT808 GPS total mileage minus first valid JT808 GPS total mileage |
The derived value is stored as `metric_key = daily_mileage_km` in
`vehicle_stat_metric`; there is no protocol-specific JT808 daily-mileage table.
Hydrogen and electricity statistics use internal fields from the same event
contract. Refueling and charging resets must be handled in statistics logic, not
in protocol mappers.
Current module: `vehicle-stat-service`.
Implemented boundaries:
- `VehicleStatEnvelopeIngestor` parses Kafka envelope bytes.
- `VehicleStatEventProcessor` extracts statistic source points from internal
fields such as `total_mileage_km`.
- `VehicleStatRepository` isolates point/result persistence from calculation.
- `VehicleStatRuleRepository` isolates per-vehicle calculation rules.
- `DailyVehicleStatService` calculates and saves day-level results.
## Data Flow ## Data Flow
```mermaid ```mermaid
flowchart LR flowchart LR
vehicle["Vehicle / Platform"] --> ingest["ingest service"] vehicle["Vehicles / Platforms"] --> ingest["Protocol apps"]
ingest --> archive["sink-archive<br/>raw bytes"] ingest --> archive["sink-archive<br/>raw bytes"]
ingest --> kafka["Kafka<br/>full-field telemetry events"] ingest --> kafka["Kafka<br/>event + raw envelopes"]
kafka --> history["event-history-service"] kafka --> history["vehicle-history-app"]
kafka --> analytics["vehicle-analytics-app"]
kafka --> state["vehicle-state-service"] kafka --> state["vehicle-state-service"]
kafka --> stat["vehicle-stat-service"] history --> tdengine["TDengine<br/>raw_frames + locations"]
history --> parquet["Parquet files"] analytics --> mysql["MySQL<br/>vehicle_stat_metric"]
history --> duckdb["DuckDB query"] analytics --> redisCalc["Redis<br/>mileage state"]
state --> redis["Redis hot state"] state --> redisState["Redis<br/>latest state"]
stat --> statdb["Statistics DB"]
``` ```
## Kafka Topic Boundaries
Suggested topics:
| Topic | Producer | Consumer |
|---|---|---|
| `vehicle.realtime` | protocol ingress | history, realtime state, statistics |
| `vehicle.location` | protocol ingress | history, realtime state, statistics |
| `vehicle.alarm` | protocol ingress | history, realtime state |
| `vehicle.session` | protocol ingress | history |
| `vehicle.media.meta` | protocol ingress | history, media tooling |
| `vehicle.raw.archive` | optional archive publisher | replay or audit services |
| `vehicle.dlq` | any consumer | operations and replay tooling |
All telemetry event topics use VIN as the partition key to preserve per-vehicle
ordering.
## Failure Handling ## Failure Handling
- Ingest must keep receiving traffic when history, Redis, or statistics services - Protocol apps continue accepting traffic when history, analytics, or Redis
are down. consumers are unavailable.
- Kafka publish failures go through the existing sink circuit breaker and DLQ - Kafka producer failures are handled by sink retry/circuit-breaker behavior and
strategy. DLQ topics where configured.
- History consumer commits Kafka offsets only after Parquet append succeeds. - History consumers should commit offsets only after TDengine writes succeed.
- Redis state consumer can replay from Kafka to rebuild hot state. - Analytics state can recover from Redis state or Kafka replay.
- Statistics consumer stores checkpoints and can recompute a day from historical - Raw archive failures must be observable, but event publication and history
detail events when rules change. indexing remain separate concerns.
- Raw archive failures are logged and alerted, but parsed event publication
remains independent.
## Implementation Phases
### Phase 1: Full-Field Contract And Historical Detail
- Add a full-field telemetry contract to `ingest-api`.
- Map GB/T 32960 parsed blocks into internal field snapshots.
- Extend Kafka envelope schema for full-field events.
- Make `event-file-store` store full-field telemetry records.
- Add query and export APIs for date-range historical detail.
### Phase 2: Redis Hot State
- Add a Redis-backed realtime state module.
- Consume full-field telemetry events.
- Store latest state, location, and safety snapshots by VIN.
- Provide query APIs for operations screens.
### Phase 3: Statistics Service
- Add a statistics module with rule configuration.
- Consume full-field telemetry events.
- Implement daily mileage strategies.
- Add result persistence boundary and calculation tests.
## Acceptance Criteria ## Acceptance Criteria
- Ingest has no compile-time dependency on history query controllers, Redis - Active production deployment contains GB32960, JT808, Yutong MQTT,
state repositories, or statistics calculators. vehicle-history, and vehicle-analytics apps.
- Every service boundary communicates through Kafka events or explicit public - Xinda Push is absent from default deployment and history consumer bindings.
APIs, not shared mutable state. - History APIs read from TDengine by default; Parquet/DuckDB is optional
- GB/T 32960 detail fields are traceable from internal field key back to compatibility only.
protocol source path. - Raw frame queries can return complete parsed JSON through `payloadJson.parsed`.
- Historical query works across multiple days with ascending and descending - Location queries page over compact rows and reference raw records instead of
order. duplicating full raw JSON.
- Redis state can be rebuilt from Kafka replay. - JT808 daily mileage is stored only in `vehicle_stat_metric`.
- Daily mileage can be calculated by at least the two initial strategies. - Runtime state needed for 808 mileage uses Redis, with no production memory
- Tests cover mapper output, Kafka serialization, Parquet query order, Redis mode.
state update, and mileage strategy calculation. - Build validation covers the active modules and their composition tests.