lingniu-vehicle-ingest/docs/superpowers/specs/2026-06-23-gb32960-service-split-design.md

> **Superseded:** This 2026-06-23 split design is historical context.
> Use `docs/target-architecture.md` and
> `docs/superpowers/specs/2026-06-29-vehicle-ingest-redesign.md` for the
> current production architecture: TDengine is the default hot history store,
> `sink-archive` owns raw bytes, and `event-file-store` is optional
> compatibility only.

# GB32960 Service Split Design

Date: 2026-06-23

## Goal

Split the current all-in-one vehicle ingest runtime into lower-coupled, higher-cohesion services. The first production slice focuses on GB/T 32960 because it is the active high-pressure path: persistent TCP connections, protocol ACKs, raw frame volume, history queries, and downstream analytics are currently assembled in one `bootstrap-all` process.

The target outcome is:

- Keep GB32960 ingest reliable and fast.
- Move disk writes and history queries out of the protocol process.
- Move statistics and analysis out of the protocol process.
- Use Kafka as the durable boundary between ingest, history, and analytics.
- Preserve `bootstrap-all` during migration as a local development and rollback entry point.

## Decision

Use ACK semantics A:

> A GB32960 frame is ACKed successfully after the protocol service parses/accepts it and writes the required Kafka message(s) successfully.

The protocol service must not wait for raw archive, DuckDB/Parquet writes, snapshot generation, or statistics computation before sending the GB32960 response frame.

## Service Boundaries

### `gb32960-ingest-app`

Purpose: own GB32960 connection handling and Kafka production.

Responsibilities:

- Listen on the GB32960 TCP port.
- Handle platform login, VIN authorization, TLS if enabled, and session state required for protocol responses.
- Decode GB32960 frames and apply vendor extension parsing.
- Produce raw frame records and normalized event records to Kafka.
- ACK successful reports only after Kafka production succeeds.
- Publish malformed or rejected frames to a DLQ when possible.
- Expose operational health and metrics for connections, decode rate, Kafka latency, ACK latency, and DLQ count.

Non-responsibilities:

- No local raw archive writes.
- No DuckDB or Parquet history writes.
- No snapshot query API.
- No vehicle statistics or analysis.
- No long-running analytical calculations.

Initial module dependencies:

- `ingest-api`
- `ingest-core`
- `ingest-codec-common`
- `session-core`
- `protocol-gb32960`
- `sink-kafka`
- `observability`

### `vehicle-history-app`

Purpose: own raw storage, historical index storage, and history query APIs.

Responsibilities:

- Consume GB32960 raw records and normalized event records from Kafka.
- Write raw `.bin` payloads through `sink-archive`.
- Write queryable event indexes through `event-file-store`.
- Provide APIs for raw frame lookup, decoded frame lookup, history query, and snapshot/source-frame lookup.
- Rebuild or backfill the DuckDB sidecar index from stored Parquet files when needed.
- Track consumer lag, write latency, archive failures, index failures, and query latency.

Non-responsibilities:

- No TCP protocol listener.
- No GB32960 ACK decisions.
- No online statistics computation.
- No direct coupling to `gb32960-ingest-app` internals.

Initial module dependencies:

- `ingest-api`
- `sink-kafka`
- `sink-archive`
- `event-file-store`
- `event-history-service`
- `protocol-gb32960` only for raw-frame decode/query support
- `observability`

### `vehicle-analytics-app`

Purpose: own vehicle state, daily statistics, alarms, and later analytics.

Responsibilities:

- Consume normalized vehicle event records from Kafka.
- Maintain latest vehicle state if enabled.
- Compute daily vehicle statistics and alarm-derived metrics.
- Persist analytical outputs to the selected backend.
- Support reprocessing from Kafka offsets when analytical logic changes.
- Track consumer lag, per-VIN processing latency, aggregation failures, and output write latency.

Non-responsibilities:

- No protocol listener.
- No GB32960 ACK decisions.
- No raw archive writes.
- No normal-path dependency on raw archive or history APIs.

Initial module dependencies:

- `ingest-api`
- `sink-kafka`
- `vehicle-state-service`
- `vehicle-stat-service`
- `observability`

## Kafka Contract

Kafka is the service boundary. Messages must be stable enough that `vehicle-history-app` and `vehicle-analytics-app` do not depend on protocol service internals.

### Topics

Use versioned topics:

- `vehicle.raw.gb32960.v1`
- `vehicle.event.gb32960.v1`
- `vehicle.dlq.gb32960.v1`

The current topic names under `lingniu.ingest.sink.kafka.topics` can remain temporarily for compatibility, but the split apps should converge on versioned topic names.

### Keys

Use VIN as the Kafka key whenever a VIN is available.

Benefits:

- Preserves per-vehicle ordering within a partition.
- Lets history and analytics scale by partition.
- Keeps stateful analytics simpler.

When VIN is unavailable, use a stable fallback key such as platform account plus peer address plus receive time bucket.

### Raw Record

Topic: `vehicle.raw.gb32960.v1`

Minimum fields:

- `schemaVersion`
- `protocol = GB32960`
- `eventId`
- `receiveTime`
- `eventTime` if decoded
- `peer`
- `platformAccount`
- `vin` if decoded
- `command`
- `protocolVersion`
- `rawBytes`
- `checksumStatus`
- `decodeStatus`
- `metadata`

Consumers:

- `vehicle-history-app` writes archive `.bin` files and raw-frame query indexes.
- Future replay tools can regenerate normalized events from raw records.

### Normalized Event Record

Topic: `vehicle.event.gb32960.v1`

Minimum fields:

- `schemaVersion`
- `protocol = GB32960`
- `eventId`
- `sourceRawEventId`
- `receiveTime`
- `eventTime`
- `platformAccount`
- `vin`
- `command`
- `eventType`
- `normalizedFields`
- `vendorFields`
- `alarmFields`
- `metadata`

Consumers:

- `vehicle-history-app` writes queryable history records.
- `vehicle-analytics-app` updates state and statistics.

### DLQ Record

Topic: `vehicle.dlq.gb32960.v1`

Minimum fields:

- `schemaVersion`
- `stage`
- `errorCode`
- `errorMessage`
- `receiveTime`
- `peer`
- `platformAccount` if known
- `vin` if known
- `rawBytes` when available
- `metadata`

Typical stages:

- `FRAME_DECODE`
- `BODY_PARSE`
- `AUTH`
- `KAFKA_PRODUCE`
- `ACK_WRITE`

## ACK and Failure Semantics

### Success Path

1. Receive TCP bytes.
2. Decode a complete GB32960 frame.
3. Validate checksum/auth/session rules.
4. Build raw and normalized Kafka records.
5. Produce required Kafka records successfully.
6. Send GB32960 success ACK.

### Kafka Produce Failure

If required Kafka production fails, the service must not send a success ACK.

Allowed behavior:

- Retry within a bounded timeout.
- Return a GB32960 failure response when the protocol allows it.
- Close the channel after repeated produce failures.
- Emit local error logs and metrics.

Do not silently ACK frames that have not crossed the Kafka durability boundary.

### History or Analytics Failure

Failures in `vehicle-history-app` or `vehicle-analytics-app` must not affect GB32960 ACKs. They are handled by Kafka offset retry, DLQ, operational alerting, and backfill/replay.

## Current Coupling to Remove

`bootstrap-all` currently assembles protocol modules, Kafka producer/consumer, archive sink, event file store, history API, vehicle state, vehicle statistics, command gateway, and multiple inbound protocols in one application.

The split should remove these production couplings:

- Protocol runtime directly containing archive and event-file-store writers.
- Protocol runtime directly containing statistics processors.
- History query API sharing the same process as TCP ingest.
- Kafka consumer workers living in the same all-in-one app as protocol listeners.
- Operational scaling tied to one JVM for ingest, storage, queries, and analytics.

## Migration Plan

### Phase 1: Add Split App Entrypoints

Add new app modules:

- `modules/apps/gb32960-ingest-app`
- `modules/apps/vehicle-history-app`
- `modules/apps/vehicle-analytics-app`

Keep `modules/apps/bootstrap-all` unchanged as the fallback runtime.

### Phase 2: GB32960 Ingest App

Create a production profile for `gb32960-ingest-app`:

- Enable `protocol-gb32960`.
- Enable Kafka producer.
- Disable local archive.
- Disable event-file-store.
- Disable history API.
- Disable vehicle state/stat services.
- Disable unrelated protocols.

Acceptance:

- GB32960 TCP port starts.
- Platform login works.
- Realtime report frames are parsed.
- Kafka records are produced with VIN keys.
- ACK is sent only after Kafka success.

### Phase 3: History App

Create `vehicle-history-app`:

- Enable Kafka consumer.
- Enable archive sink.
- Enable event-file-store.
- Enable event-history HTTP API.
- Disable protocol listeners.
- Disable analytics processors.

Acceptance:

- Consumes `vehicle.raw.gb32960.v1`.
- Writes raw `.bin` files.
- Consumes `vehicle.event.gb32960.v1`.
- Writes Parquet/DuckDB indexes.
- Queries can locate raw frames and decoded snapshots.

### Phase 4: Analytics App

Create `vehicle-analytics-app`:

- Enable Kafka consumer.
- Enable vehicle state if a state backend is configured.
- Enable vehicle statistics.
- Disable protocol listeners.
- Disable archive and event-file-store.

Acceptance:

- Consumes `vehicle.event.gb32960.v1`.
- Maintains expected per-VIN state/stat outputs.
- Can be restarted and resume from Kafka offsets.
- Does not affect ingest ACK latency.

### Phase 5: Parallel Run and Cutover

Run old and new paths in parallel for a bounded validation window.

Compare:

- Received frame count.
- Kafka produced record count.
- Raw archive file count.
- Unique VIN count.
- History query count and sample query correctness.
- Analytics output count and sample values.
- GB32960 ACK latency before and after split.

Cut over only after counts and sample queries match within the agreed tolerance.

## Operational Guidance

Scale services independently:

- Scale `gb32960-ingest-app` by connection count and Kafka produce latency.
- Scale `vehicle-history-app` by disk throughput, consumer lag, and query latency.
- Scale `vehicle-analytics-app` by consumer lag and computation latency.

Monitor:

- Kafka producer error rate and p99 produce latency.
- ACK p99 latency.
- TCP active connections.
- Decode failures and DLQ counts.
- Consumer lag per group.
- Archive write latency and failure rate.
- Event-file-store flush latency and failure rate.
- Analytics processing latency.

## Risks and Mitigations

Risk: Kafka outage blocks GB32960 ACKs.

Mitigation: bounded producer retry, clear failure ACK/close behavior, Kafka cluster monitoring, and optional local emergency spool only as a later explicit design.

Risk: Two Kafka topics for raw and normalized events can diverge.

Mitigation: include `eventId` and `sourceRawEventId`, produce records transactionally if required, or publish a single envelope containing both raw and normalized sections in the first implementation if transaction support is not ready.

Risk: History app reprocessing duplicates archive/index records.

Mitigation: deterministic archive keys and idempotent index writes keyed by `eventId` or `rawArchiveUri`.

Risk: Analytics logic changes require backfill.

Mitigation: keep raw and normalized topics with sufficient retention; allow analytics consumers to reset offsets or run a backfill group.

Risk: Query APIs accidentally depend on ingest internals.

Mitigation: history APIs should depend on `event-file-store`, `sink-archive`, and decode libraries only, not `gb32960-ingest-app`.

## Open Decisions

These are intentionally left for implementation planning:

- Whether Kafka production uses two independent sends or a transaction for raw + normalized records.
- Exact protobuf/JSON schema shape for `vehicle.raw.gb32960.v1` and `vehicle.event.gb32960.v1`.
- Whether latest snapshot state belongs only to `vehicle-analytics-app` or is also materialized by `vehicle-history-app` for query convenience.
- Initial Kafka partition count and retention duration.
- Whether command downlink stays with `command-gateway` or gets a separate command service later.

## Completion Criteria

The split is complete when:

- `gb32960-ingest-app`, `vehicle-history-app`, and `vehicle-analytics-app` are separate runnable app modules.
- `gb32960-ingest-app` can receive live GB32960 data and ACK after Kafka success.
- `vehicle-history-app` can consume Kafka and provide raw/history/snapshot query APIs.
- `vehicle-analytics-app` can consume Kafka and compute state/stat outputs.
- `bootstrap-all` remains available for development and rollback.
- Verification covers build, app startup, Kafka production/consumption, raw archive writes, history queries, analytics outputs, and ACK latency.