lingniu-vehicle-ingest/docs/superpowers/specs/2026-06-23-gb32960-service-split-design.md

Superseded: This 2026-06-23 split design is historical context. Use docs/target-architecture.md and docs/superpowers/specs/2026-06-29-vehicle-ingest-redesign.md for the current production architecture: TDengine is the default hot history store, sink-archive owns raw bytes, and event-file-store is optional compatibility only.

GB32960 Service Split Design

Date: 2026-06-23

Goal

Split the current all-in-one vehicle ingest runtime into lower-coupled, higher-cohesion services. The first production slice focuses on GB/T 32960 because it is the active high-pressure path: persistent TCP connections, protocol ACKs, raw frame volume, history queries, and downstream analytics are currently assembled in one bootstrap-all process.

The target outcome is:

• Keep GB32960 ingest reliable and fast.
• Move disk writes and history queries out of the protocol process.
• Move statistics and analysis out of the protocol process.
• Use Kafka as the durable boundary between ingest, history, and analytics.
• Preserve bootstrap-all during migration as a local development and rollback entry point.

Decision

Use ACK semantics A:

A GB32960 frame is ACKed successfully after the protocol service parses/accepts it and writes the required Kafka message(s) successfully.

The protocol service must not wait for raw archive, DuckDB/Parquet writes, snapshot generation, or statistics computation before sending the GB32960 response frame.

Service Boundaries

gb32960-ingest-app

Purpose: own GB32960 connection handling and Kafka production.

Responsibilities:

• Listen on the GB32960 TCP port.
• Handle platform login, VIN authorization, TLS if enabled, and session state required for protocol responses.
• Decode GB32960 frames and apply vendor extension parsing.
• Produce raw frame records and normalized event records to Kafka.
• ACK successful reports only after Kafka production succeeds.
• Publish malformed or rejected frames to a DLQ when possible.
• Expose operational health and metrics for connections, decode rate, Kafka latency, ACK latency, and DLQ count.

Non-responsibilities:

• No local raw archive writes.
• No DuckDB or Parquet history writes.
• No snapshot query API.
• No vehicle statistics or analysis.
• No long-running analytical calculations.

Initial module dependencies:

• ingest-api
• ingest-core
• ingest-codec-common
• session-core
• protocol-gb32960
• sink-kafka
• observability

vehicle-history-app

Purpose: own raw storage, historical index storage, and history query APIs.

Responsibilities:

• Consume GB32960 raw records and normalized event records from Kafka.
• Write raw .bin payloads through sink-archive.
• Write queryable event indexes through event-file-store.
• Provide APIs for raw frame lookup, decoded frame lookup, history query, and snapshot/source-frame lookup.
• Rebuild or backfill the DuckDB sidecar index from stored Parquet files when needed.
• Track consumer lag, write latency, archive failures, index failures, and query latency.

Non-responsibilities:

• No TCP protocol listener.
• No GB32960 ACK decisions.
• No online statistics computation.
• No direct coupling to gb32960-ingest-app internals.

Initial module dependencies:

• ingest-api
• sink-kafka
• sink-archive
• event-file-store
• event-history-service
• protocol-gb32960 only for raw-frame decode/query support
• observability

vehicle-analytics-app

Purpose: own vehicle state, daily statistics, alarms, and later analytics.

Responsibilities:

• Consume normalized vehicle event records from Kafka.
• Maintain latest vehicle state if enabled.
• Compute daily vehicle statistics and alarm-derived metrics.
• Persist analytical outputs to the selected backend.
• Support reprocessing from Kafka offsets when analytical logic changes.
• Track consumer lag, per-VIN processing latency, aggregation failures, and output write latency.

Non-responsibilities:

• No protocol listener.
• No GB32960 ACK decisions.
• No raw archive writes.
• No normal-path dependency on raw archive or history APIs.

Initial module dependencies:

• ingest-api
• sink-kafka
• vehicle-state-service
• vehicle-stat-service
• observability

Kafka Contract

Kafka is the service boundary. Messages must be stable enough that vehicle-history-app and vehicle-analytics-app do not depend on protocol service internals.

Topics

Use versioned topics:

• vehicle.raw.gb32960.v1
• vehicle.event.gb32960.v1
• vehicle.dlq.gb32960.v1

The current topic names under lingniu.ingest.sink.kafka.topics can remain temporarily for compatibility, but the split apps should converge on versioned topic names.

Keys

Use VIN as the Kafka key whenever a VIN is available.

Benefits:

• Preserves per-vehicle ordering within a partition.
• Lets history and analytics scale by partition.
• Keeps stateful analytics simpler.

When VIN is unavailable, use a stable fallback key such as platform account plus peer address plus receive time bucket.

Raw Record

Topic: vehicle.raw.gb32960.v1

Minimum fields:

• schemaVersion
• protocol = GB32960
• eventId
• receiveTime
• eventTime if decoded
• peer
• platformAccount
• vin if decoded
• command
• protocolVersion
• rawBytes
• checksumStatus
• decodeStatus
• metadata

Consumers:

• vehicle-history-app writes archive .bin files and raw-frame query indexes.
• Future replay tools can regenerate normalized events from raw records.

Normalized Event Record

Topic: vehicle.event.gb32960.v1

Minimum fields:

• schemaVersion
• protocol = GB32960
• eventId
• sourceRawEventId
• receiveTime
• eventTime
• platformAccount
• vin
• command
• eventType
• normalizedFields
• vendorFields
• alarmFields
• metadata

Consumers:

• vehicle-history-app writes queryable history records.
• vehicle-analytics-app updates state and statistics.

DLQ Record

Topic: vehicle.dlq.gb32960.v1

Minimum fields:

• schemaVersion
• stage
• errorCode
• errorMessage
• receiveTime
• peer
• platformAccount if known
• vin if known
• rawBytes when available
• metadata

Typical stages:

• FRAME_DECODE
• BODY_PARSE
• AUTH
• KAFKA_PRODUCE
• ACK_WRITE

ACK and Failure Semantics

Success Path

1. Receive TCP bytes.
2. Decode a complete GB32960 frame.
3. Validate checksum/auth/session rules.
4. Build raw and normalized Kafka records.
5. Produce required Kafka records successfully.
6. Send GB32960 success ACK.

Kafka Produce Failure

If required Kafka production fails, the service must not send a success ACK.

Allowed behavior:

• Retry within a bounded timeout.
• Return a GB32960 failure response when the protocol allows it.
• Close the channel after repeated produce failures.
• Emit local error logs and metrics.

Do not silently ACK frames that have not crossed the Kafka durability boundary.

History or Analytics Failure

Failures in vehicle-history-app or vehicle-analytics-app must not affect GB32960 ACKs. They are handled by Kafka offset retry, DLQ, operational alerting, and backfill/replay.

Current Coupling to Remove

bootstrap-all currently assembles protocol modules, Kafka producer/consumer, archive sink, event file store, history API, vehicle state, vehicle statistics, command gateway, and multiple inbound protocols in one application.

The split should remove these production couplings:

• Protocol runtime directly containing archive and event-file-store writers.
• Protocol runtime directly containing statistics processors.
• History query API sharing the same process as TCP ingest.
• Kafka consumer workers living in the same all-in-one app as protocol listeners.
• Operational scaling tied to one JVM for ingest, storage, queries, and analytics.

Migration Plan

Phase 1: Add Split App Entrypoints

Add new app modules:

• modules/apps/gb32960-ingest-app
• modules/apps/vehicle-history-app
• modules/apps/vehicle-analytics-app

Keep modules/apps/bootstrap-all unchanged as the fallback runtime.

Phase 2: GB32960 Ingest App

Create a production profile for gb32960-ingest-app:

• Enable protocol-gb32960.
• Enable Kafka producer.
• Disable local archive.
• Disable event-file-store.
• Disable history API.
• Disable vehicle state/stat services.
• Disable unrelated protocols.

Acceptance:

• GB32960 TCP port starts.
• Platform login works.
• Realtime report frames are parsed.
• Kafka records are produced with VIN keys.
• ACK is sent only after Kafka success.

Phase 3: History App

Create vehicle-history-app:

• Enable Kafka consumer.
• Enable archive sink.
• Enable event-file-store.
• Enable event-history HTTP API.
• Disable protocol listeners.
• Disable analytics processors.

Acceptance:

• Consumes vehicle.raw.gb32960.v1.
• Writes raw .bin files.
• Consumes vehicle.event.gb32960.v1.
• Writes Parquet/DuckDB indexes.
• Queries can locate raw frames and decoded snapshots.

Phase 4: Analytics App

Create vehicle-analytics-app:

• Enable Kafka consumer.
• Enable vehicle state if a state backend is configured.
• Enable vehicle statistics.
• Disable protocol listeners.
• Disable archive and event-file-store.

Acceptance:

• Consumes vehicle.event.gb32960.v1.
• Maintains expected per-VIN state/stat outputs.
• Can be restarted and resume from Kafka offsets.
• Does not affect ingest ACK latency.

Phase 5: Parallel Run and Cutover

Run old and new paths in parallel for a bounded validation window.

Compare:

• Received frame count.
• Kafka produced record count.
• Raw archive file count.
• Unique VIN count.
• History query count and sample query correctness.
• Analytics output count and sample values.
• GB32960 ACK latency before and after split.

Cut over only after counts and sample queries match within the agreed tolerance.

Operational Guidance

Scale services independently:

• Scale gb32960-ingest-app by connection count and Kafka produce latency.
• Scale vehicle-history-app by disk throughput, consumer lag, and query latency.
• Scale vehicle-analytics-app by consumer lag and computation latency.

Monitor:

• Kafka producer error rate and p99 produce latency.
• ACK p99 latency.
• TCP active connections.
• Decode failures and DLQ counts.
• Consumer lag per group.
• Archive write latency and failure rate.
• Event-file-store flush latency and failure rate.
• Analytics processing latency.

Risks and Mitigations

Risk: Kafka outage blocks GB32960 ACKs.

Mitigation: bounded producer retry, clear failure ACK/close behavior, Kafka cluster monitoring, and optional local emergency spool only as a later explicit design.

Risk: Two Kafka topics for raw and normalized events can diverge.

Mitigation: include eventId and sourceRawEventId, produce records transactionally if required, or publish a single envelope containing both raw and normalized sections in the first implementation if transaction support is not ready.

Risk: History app reprocessing duplicates archive/index records.

Mitigation: deterministic archive keys and idempotent index writes keyed by eventId or rawArchiveUri.

Risk: Analytics logic changes require backfill.

Mitigation: keep raw and normalized topics with sufficient retention; allow analytics consumers to reset offsets or run a backfill group.

Risk: Query APIs accidentally depend on ingest internals.

Mitigation: history APIs should depend on event-file-store, sink-archive, and decode libraries only, not gb32960-ingest-app.

Open Decisions

These are intentionally left for implementation planning:

• Whether Kafka production uses two independent sends or a transaction for raw + normalized records.
• Exact protobuf/JSON schema shape for vehicle.raw.gb32960.v1 and vehicle.event.gb32960.v1.
• Whether latest snapshot state belongs only to vehicle-analytics-app or is also materialized by vehicle-history-app for query convenience.
• Initial Kafka partition count and retention duration.
• Whether command downlink stays with command-gateway or gets a separate command service later.

Completion Criteria

The split is complete when:

• gb32960-ingest-app, vehicle-history-app, and vehicle-analytics-app are separate runnable app modules.
• gb32960-ingest-app can receive live GB32960 data and ACK after Kafka success.
• vehicle-history-app can consume Kafka and provide raw/history/snapshot query APIs.
• vehicle-analytics-app can consume Kafka and compute state/stat outputs.
• bootstrap-all remains available for development and rollback.
• Verification covers build, app startup, Kafka production/consumption, raw archive writes, history queries, analytics outputs, and ACK latency.