> **Superseded:** This 2026-06-23 split design is historical context. > Use `docs/target-architecture.md` and > `docs/superpowers/specs/2026-06-29-vehicle-ingest-redesign.md` for the > current production architecture: TDengine is the default hot history store, > `sink-archive` owns raw bytes, and `event-file-store` is optional > compatibility only. # GB32960 Service Split Design Date: 2026-06-23 ## Goal Split the current all-in-one vehicle ingest runtime into lower-coupled, higher-cohesion services. The first production slice focuses on GB/T 32960 because it is the active high-pressure path: persistent TCP connections, protocol ACKs, raw frame volume, history queries, and downstream analytics are currently assembled in one `bootstrap-all` process. The target outcome is: - Keep GB32960 ingest reliable and fast. - Move disk writes and history queries out of the protocol process. - Move statistics and analysis out of the protocol process. - Use Kafka as the durable boundary between ingest, history, and analytics. - Preserve `bootstrap-all` during migration as a local development and rollback entry point. ## Decision Use ACK semantics A: > A GB32960 frame is ACKed successfully after the protocol service parses/accepts it and writes the required Kafka message(s) successfully. The protocol service must not wait for raw archive, DuckDB/Parquet writes, snapshot generation, or statistics computation before sending the GB32960 response frame. ## Service Boundaries ### `gb32960-ingest-app` Purpose: own GB32960 connection handling and Kafka production. Responsibilities: - Listen on the GB32960 TCP port. - Handle platform login, VIN authorization, TLS if enabled, and session state required for protocol responses. - Decode GB32960 frames and apply vendor extension parsing. - Produce raw frame records and normalized event records to Kafka. - ACK successful reports only after Kafka production succeeds. - Publish malformed or rejected frames to a DLQ when possible. - Expose operational health and metrics for connections, decode rate, Kafka latency, ACK latency, and DLQ count. Non-responsibilities: - No local raw archive writes. - No DuckDB or Parquet history writes. - No snapshot query API. - No vehicle statistics or analysis. - No long-running analytical calculations. Initial module dependencies: - `ingest-api` - `ingest-core` - `ingest-codec-common` - `session-core` - `protocol-gb32960` - `sink-kafka` - `observability` ### `vehicle-history-app` Purpose: own raw storage, historical index storage, and history query APIs. Responsibilities: - Consume GB32960 raw records and normalized event records from Kafka. - Write raw `.bin` payloads through `sink-archive`. - Write queryable event indexes through `event-file-store`. - Provide APIs for raw frame lookup, decoded frame lookup, history query, and snapshot/source-frame lookup. - Rebuild or backfill the DuckDB sidecar index from stored Parquet files when needed. - Track consumer lag, write latency, archive failures, index failures, and query latency. Non-responsibilities: - No TCP protocol listener. - No GB32960 ACK decisions. - No online statistics computation. - No direct coupling to `gb32960-ingest-app` internals. Initial module dependencies: - `ingest-api` - `sink-kafka` - `sink-archive` - `event-file-store` - `event-history-service` - `protocol-gb32960` only for raw-frame decode/query support - `observability` ### `vehicle-analytics-app` Purpose: own vehicle state, daily statistics, alarms, and later analytics. Responsibilities: - Consume normalized vehicle event records from Kafka. - Maintain latest vehicle state if enabled. - Compute daily vehicle statistics and alarm-derived metrics. - Persist analytical outputs to the selected backend. - Support reprocessing from Kafka offsets when analytical logic changes. - Track consumer lag, per-VIN processing latency, aggregation failures, and output write latency. Non-responsibilities: - No protocol listener. - No GB32960 ACK decisions. - No raw archive writes. - No normal-path dependency on raw archive or history APIs. Initial module dependencies: - `ingest-api` - `sink-kafka` - `vehicle-state-service` - `vehicle-stat-service` - `observability` ## Kafka Contract Kafka is the service boundary. Messages must be stable enough that `vehicle-history-app` and `vehicle-analytics-app` do not depend on protocol service internals. ### Topics Use versioned topics: - `vehicle.raw.gb32960.v1` - `vehicle.event.gb32960.v1` - `vehicle.dlq.gb32960.v1` The current topic names under `lingniu.ingest.sink.kafka.topics` can remain temporarily for compatibility, but the split apps should converge on versioned topic names. ### Keys Use VIN as the Kafka key whenever a VIN is available. Benefits: - Preserves per-vehicle ordering within a partition. - Lets history and analytics scale by partition. - Keeps stateful analytics simpler. When VIN is unavailable, use a stable fallback key such as platform account plus peer address plus receive time bucket. ### Raw Record Topic: `vehicle.raw.gb32960.v1` Minimum fields: - `schemaVersion` - `protocol = GB32960` - `eventId` - `receiveTime` - `eventTime` if decoded - `peer` - `platformAccount` - `vin` if decoded - `command` - `protocolVersion` - `rawBytes` - `checksumStatus` - `decodeStatus` - `metadata` Consumers: - `vehicle-history-app` writes archive `.bin` files and raw-frame query indexes. - Future replay tools can regenerate normalized events from raw records. ### Normalized Event Record Topic: `vehicle.event.gb32960.v1` Minimum fields: - `schemaVersion` - `protocol = GB32960` - `eventId` - `sourceRawEventId` - `receiveTime` - `eventTime` - `platformAccount` - `vin` - `command` - `eventType` - `normalizedFields` - `vendorFields` - `alarmFields` - `metadata` Consumers: - `vehicle-history-app` writes queryable history records. - `vehicle-analytics-app` updates state and statistics. ### DLQ Record Topic: `vehicle.dlq.gb32960.v1` Minimum fields: - `schemaVersion` - `stage` - `errorCode` - `errorMessage` - `receiveTime` - `peer` - `platformAccount` if known - `vin` if known - `rawBytes` when available - `metadata` Typical stages: - `FRAME_DECODE` - `BODY_PARSE` - `AUTH` - `KAFKA_PRODUCE` - `ACK_WRITE` ## ACK and Failure Semantics ### Success Path 1. Receive TCP bytes. 2. Decode a complete GB32960 frame. 3. Validate checksum/auth/session rules. 4. Build raw and normalized Kafka records. 5. Produce required Kafka records successfully. 6. Send GB32960 success ACK. ### Kafka Produce Failure If required Kafka production fails, the service must not send a success ACK. Allowed behavior: - Retry within a bounded timeout. - Return a GB32960 failure response when the protocol allows it. - Close the channel after repeated produce failures. - Emit local error logs and metrics. Do not silently ACK frames that have not crossed the Kafka durability boundary. ### History or Analytics Failure Failures in `vehicle-history-app` or `vehicle-analytics-app` must not affect GB32960 ACKs. They are handled by Kafka offset retry, DLQ, operational alerting, and backfill/replay. ## Current Coupling to Remove `bootstrap-all` currently assembles protocol modules, Kafka producer/consumer, archive sink, event file store, history API, vehicle state, vehicle statistics, command gateway, and multiple inbound protocols in one application. The split should remove these production couplings: - Protocol runtime directly containing archive and event-file-store writers. - Protocol runtime directly containing statistics processors. - History query API sharing the same process as TCP ingest. - Kafka consumer workers living in the same all-in-one app as protocol listeners. - Operational scaling tied to one JVM for ingest, storage, queries, and analytics. ## Migration Plan ### Phase 1: Add Split App Entrypoints Add new app modules: - `modules/apps/gb32960-ingest-app` - `modules/apps/vehicle-history-app` - `modules/apps/vehicle-analytics-app` Keep `modules/apps/bootstrap-all` unchanged as the fallback runtime. ### Phase 2: GB32960 Ingest App Create a production profile for `gb32960-ingest-app`: - Enable `protocol-gb32960`. - Enable Kafka producer. - Disable local archive. - Disable event-file-store. - Disable history API. - Disable vehicle state/stat services. - Disable unrelated protocols. Acceptance: - GB32960 TCP port starts. - Platform login works. - Realtime report frames are parsed. - Kafka records are produced with VIN keys. - ACK is sent only after Kafka success. ### Phase 3: History App Create `vehicle-history-app`: - Enable Kafka consumer. - Enable archive sink. - Enable event-file-store. - Enable event-history HTTP API. - Disable protocol listeners. - Disable analytics processors. Acceptance: - Consumes `vehicle.raw.gb32960.v1`. - Writes raw `.bin` files. - Consumes `vehicle.event.gb32960.v1`. - Writes Parquet/DuckDB indexes. - Queries can locate raw frames and decoded snapshots. ### Phase 4: Analytics App Create `vehicle-analytics-app`: - Enable Kafka consumer. - Enable vehicle state if a state backend is configured. - Enable vehicle statistics. - Disable protocol listeners. - Disable archive and event-file-store. Acceptance: - Consumes `vehicle.event.gb32960.v1`. - Maintains expected per-VIN state/stat outputs. - Can be restarted and resume from Kafka offsets. - Does not affect ingest ACK latency. ### Phase 5: Parallel Run and Cutover Run old and new paths in parallel for a bounded validation window. Compare: - Received frame count. - Kafka produced record count. - Raw archive file count. - Unique VIN count. - History query count and sample query correctness. - Analytics output count and sample values. - GB32960 ACK latency before and after split. Cut over only after counts and sample queries match within the agreed tolerance. ## Operational Guidance Scale services independently: - Scale `gb32960-ingest-app` by connection count and Kafka produce latency. - Scale `vehicle-history-app` by disk throughput, consumer lag, and query latency. - Scale `vehicle-analytics-app` by consumer lag and computation latency. Monitor: - Kafka producer error rate and p99 produce latency. - ACK p99 latency. - TCP active connections. - Decode failures and DLQ counts. - Consumer lag per group. - Archive write latency and failure rate. - Event-file-store flush latency and failure rate. - Analytics processing latency. ## Risks and Mitigations Risk: Kafka outage blocks GB32960 ACKs. Mitigation: bounded producer retry, clear failure ACK/close behavior, Kafka cluster monitoring, and optional local emergency spool only as a later explicit design. Risk: Two Kafka topics for raw and normalized events can diverge. Mitigation: include `eventId` and `sourceRawEventId`, produce records transactionally if required, or publish a single envelope containing both raw and normalized sections in the first implementation if transaction support is not ready. Risk: History app reprocessing duplicates archive/index records. Mitigation: deterministic archive keys and idempotent index writes keyed by `eventId` or `rawArchiveUri`. Risk: Analytics logic changes require backfill. Mitigation: keep raw and normalized topics with sufficient retention; allow analytics consumers to reset offsets or run a backfill group. Risk: Query APIs accidentally depend on ingest internals. Mitigation: history APIs should depend on `event-file-store`, `sink-archive`, and decode libraries only, not `gb32960-ingest-app`. ## Open Decisions These are intentionally left for implementation planning: - Whether Kafka production uses two independent sends or a transaction for raw + normalized records. - Exact protobuf/JSON schema shape for `vehicle.raw.gb32960.v1` and `vehicle.event.gb32960.v1`. - Whether latest snapshot state belongs only to `vehicle-analytics-app` or is also materialized by `vehicle-history-app` for query convenience. - Initial Kafka partition count and retention duration. - Whether command downlink stays with `command-gateway` or gets a separate command service later. ## Completion Criteria The split is complete when: - `gb32960-ingest-app`, `vehicle-history-app`, and `vehicle-analytics-app` are separate runnable app modules. - `gb32960-ingest-app` can receive live GB32960 data and ACK after Kafka success. - `vehicle-history-app` can consume Kafka and provide raw/history/snapshot query APIs. - `vehicle-analytics-app` can consume Kafka and compute state/stat outputs. - `bootstrap-all` remains available for development and rollback. - Verification covers build, app startup, Kafka production/consumption, raw archive writes, history queries, analytics outputs, and ACK latency.