lingniu-vehicle-ingest/docs/operations/gb32960-service-split-runbook.md

GB32960 Split Service Runbook

This runbook covers the default production runtimes: GB32960 ingest, JT808 ingest, Yutong MQTT ingest, history, and analytics. Kafka is the durability boundary between protocol ingest services and downstream business services.

Service Roles

Service	Module	Role	Local ports
GB32960 ingest	:gb32960-ingest-app	Accept GB32960 TCP connections, decode/authenticate frames, produce raw and normalized records to Kafka, and send protocol ACKs after required Kafka production succeeds.	TCP 32960, HTTP 20100
JT808 ingest	:jt808-ingest-app	Accept JT/T 808 TCP connections on the production receive port, parse raw frames, archive frame bytes, and produce raw/event envelopes to Kafka.	TCP 808, HTTP 20400
Yutong MQTT ingest	:yutong-mqtt-app	Connect to Yutong MQTT, parse subscribed messages, archive payloads, and produce raw/event envelopes to Kafka.	HTTP 20500
Vehicle history	:vehicle-history-app	Consume GB32960/JT808/Yutong MQTT raw/event Kafka records, store compact history rows plus TDengine raw_frames parsedJson/metadataJson/rawUri, and expose TDengine-backed history APIs.	HTTP 20200
Vehicle analytics	:vehicle-analytics-app	Consume JT808 event Kafka records, calculate daily mileage from reported GPS total mileage, and write metrics to vehicle_stat_metric.	HTTP 20300

Kafka Contract

Topic	Producer	Consumers	Purpose
vehicle.raw.gb32960.v1	gb32960-ingest-app	vehicle-history-app	Raw frame records. History writes the parsed JSON, metadata JSON, and raw URI to TDengine raw_frames.
vehicle.event.gb32960.v1	gb32960-ingest-app	vehicle-history-app, vehicle-analytics-app	Normalized vehicle events keyed by VIN when available.
vehicle.dlq.gb32960.v1	Kafka producer/consumer error paths	Operators/replay tooling	Dead-letter records for failed production or consumer processing.
vehicle.raw.jt808.v1	jt808-ingest-app	vehicle-history-app	JT808 raw frame records for TDengine raw_frames and raw-frame troubleshooting APIs.
vehicle.event.jt808.v1	jt808-ingest-app	vehicle-history-app	JT808 parsed location/session/alarm events keyed by phone or mapped VIN.
vehicle.dlq.jt808.v1	Kafka producer/consumer error paths	Operators/replay tooling	Dead-letter records for failed JT808 production or consumer processing.
vehicle.raw.mqtt-yutong.v1	yutong-mqtt-app	vehicle-history-app	Yutong MQTT raw payload records for TDengine raw_frames and troubleshooting APIs.
vehicle.event.mqtt-yutong.v1	yutong-mqtt-app	vehicle-history-app	Yutong MQTT parsed telemetry events keyed by mapped VIN or device identity.
vehicle.dlq.mqtt-yutong.v1	Kafka producer/consumer error paths	Operators/replay tooling	Dead-letter records for failed Yutong MQTT production or consumer processing.

Default local consumer groups:

• History: vehicle-history
• Analytics statistics: vehicle-stat

ACK Semantics

GB32960 success ACKs are sent only after the ingest service completes the required Kafka production boundary for the accepted frame. If required Kafka production fails, the ingest service must not claim a successful ACK for that frame.

History and analytics failures do not block GB32960 ACKs. They are downstream Kafka consumer failures and should be handled through retry, DLQ inspection, offset replay, and service-specific recovery.

Authorization failures are rejected before the Kafka durability boundary and can be ACKed/rejected immediately according to protocol handling.

Local Prerequisites

• Java 25 and Maven available on PATH.
• Local Kafka reachable at 127.0.0.1:9092, plus Kafka CLI tools such as kafka-topics and optionally kafka-console-consumer.
• Session state requires Redis. GB32960/JT808 ingest services keep live Netty channels in-process, but session indexes and TTL metadata use Redis only.
• No repository-local Kafka bootstrap script or compose file was found during Task 8 verification on 2026-06-23.

If your shell needs an explicit JDK:

export JAVA_HOME=/opt/homebrew/opt/openjdk@25/libexec/openjdk.jdk/Contents/Home
export PATH="$JAVA_HOME/bin:/opt/homebrew/bin:$PATH"

Build

From the repository root:

mvn -pl :gb32960-ingest-app,:jt808-ingest-app,:yutong-mqtt-app,:vehicle-history-app,:vehicle-analytics-app -am package -Dmaven.test.skip=true

Expected result: BUILD SUCCESS and these runnable jars:

• modules/apps/gb32960-ingest-app/target/gb32960-ingest-app.jar
• modules/apps/jt808-ingest-app/target/jt808-ingest-app.jar
• modules/apps/yutong-mqtt-app/target/yutong-mqtt-app.jar
• modules/apps/vehicle-history-app/target/vehicle-history-app.jar
• modules/apps/vehicle-analytics-app/target/vehicle-analytics-app.jar

Create Topics

Run these only after Kafka is reachable at 127.0.0.1:9092:

kafka-topics --bootstrap-server 127.0.0.1:9092 --create --if-not-exists --topic vehicle.raw.gb32960.v1 --partitions 12 --replication-factor 1
kafka-topics --bootstrap-server 127.0.0.1:9092 --create --if-not-exists --topic vehicle.event.gb32960.v1 --partitions 12 --replication-factor 1
kafka-topics --bootstrap-server 127.0.0.1:9092 --create --if-not-exists --topic vehicle.dlq.gb32960.v1 --partitions 3 --replication-factor 1
kafka-topics --bootstrap-server 127.0.0.1:9092 --create --if-not-exists --topic vehicle.raw.jt808.v1 --partitions 12 --replication-factor 1
kafka-topics --bootstrap-server 127.0.0.1:9092 --create --if-not-exists --topic vehicle.event.jt808.v1 --partitions 12 --replication-factor 1
kafka-topics --bootstrap-server 127.0.0.1:9092 --create --if-not-exists --topic vehicle.dlq.jt808.v1 --partitions 3 --replication-factor 1
kafka-topics --bootstrap-server 127.0.0.1:9092 --create --if-not-exists --topic vehicle.raw.mqtt-yutong.v1 --partitions 12 --replication-factor 1
kafka-topics --bootstrap-server 127.0.0.1:9092 --create --if-not-exists --topic vehicle.event.mqtt-yutong.v1 --partitions 12 --replication-factor 1
kafka-topics --bootstrap-server 127.0.0.1:9092 --create --if-not-exists --topic vehicle.dlq.mqtt-yutong.v1 --partitions 3 --replication-factor 1

Optional sanity check:

kafka-topics --bootstrap-server 127.0.0.1:9092 --list | grep -E 'vehicle\.(raw|event|dlq)\.(gb32960|jt808|mqtt-yutong)\.v1'

Start Split Services Locally

Run each command in a separate terminal from the repository root.

GB32960 ingest:

KAFKA_BROKERS=127.0.0.1:9092 \
REDIS_HOST=127.0.0.1 \
REDIS_PORT=6379 \
REDIS_DATABASE=50 \
GB32960_PORT=32960 \
HTTP_PORT=20100 \
java --sun-misc-unsafe-memory-access=allow \
  -jar modules/apps/gb32960-ingest-app/target/gb32960-ingest-app.jar

JT808 ingest:

KAFKA_BROKERS=127.0.0.1:9092 \
REDIS_HOST=127.0.0.1 \
REDIS_PORT=6379 \
REDIS_DATABASE=50 \
JT808_PORT=808 \
HTTP_PORT=20400 \
java --sun-misc-unsafe-memory-access=allow \
  -jar modules/apps/jt808-ingest-app/target/jt808-ingest-app.jar

Yutong MQTT ingest:

KAFKA_BROKERS=127.0.0.1:9092 \
YUTONG_MQTT_ENABLED=true \
YUTONG_MQTT_URI='<mqtt-uri>' \
YUTONG_MQTT_USERNAME='<mqtt-user>' \
YUTONG_MQTT_PASSWORD='<mqtt-password>' \
HTTP_PORT=20500 \
java --sun-misc-unsafe-memory-access=allow \
  -jar modules/apps/yutong-mqtt-app/target/yutong-mqtt-app.jar

Vehicle history:

KAFKA_BROKERS=127.0.0.1:9092 \
HTTP_PORT=20200 \
TDENGINE_HISTORY_ENABLED=true \
TDENGINE_HISTORY_DATABASE=vehicle_ts \
TDENGINE_JDBC_URL='jdbc:TAOS-WS://<tdengine-host>:6041/vehicle_ts' \
TDENGINE_USERNAME=root \
TDENGINE_PASSWORD='<tdengine-password>' \
TDENGINE_TELEMETRY_FIELDS_ENABLED=false \
java --sun-misc-unsafe-memory-access=allow \
  -jar modules/apps/vehicle-history-app/target/vehicle-history-app.jar

SINK_ARCHIVE_PATH is intentionally omitted from the history JVM. The history hot path is Kafka to TDengine: raw envelopes are stored in TDengine raw_frames with parsedJson, metadataJson, and rawUri. Ingest services may still write original .bin files as cold backup, but history API queries must not depend on a shared local archive mount.

Vehicle analytics:

KAFKA_BROKERS=127.0.0.1:9092 \
HTTP_PORT=20300 \
java --sun-misc-unsafe-memory-access=allow \
  -jar modules/apps/vehicle-analytics-app/target/vehicle-analytics-app.jar

vehicle-analytics-app is intentionally a metric runtime only. Latest-state APIs belong to vehicle-state-service, which stays outside the default reactor and should be built with -Poptional-latest-state and deployed as a separate consumer if that surface is needed.

Health Verification

curl -sS http://127.0.0.1:20100/actuator/health
curl -sS http://127.0.0.1:20400/actuator/health
curl -sS http://127.0.0.1:20500/actuator/health
curl -sS http://127.0.0.1:20200/actuator/health
curl -sS http://127.0.0.1:20300/actuator/health

Expected: each endpoint returns {"status":"UP"} or an equivalent Spring Boot health JSON with top-level status UP.

End-to-End Verification

Use a known-valid GB32960 fixture, for example one of the hex samples under modules/protocols/protocol-gb32960/src/test/resources/samples/gb32960/.

Example send command:

xxd -r -p modules/protocols/protocol-gb32960/src/test/resources/samples/gb32960/realtime_001.hex | nc 127.0.0.1 32960

To capture and inspect the binary ACK, write the response to a file:

rm -f /tmp/gb32960-ack.bin
xxd -r -p modules/protocols/protocol-gb32960/src/test/resources/samples/gb32960/realtime_001.hex \
  | nc -w 3 127.0.0.1 32960 > /tmp/gb32960-ack.bin
xxd -p /tmp/gb32960-ack.bin

Expected: the ACK file is non-empty and the decoded bytes begin with the GB32960 frame marker 2323. Treat the ACK as verified only after checking the response bytes from your run.

Verify only what was actually run in your environment:

kafka-console-consumer --bootstrap-server 127.0.0.1:9092 --topic vehicle.raw.gb32960.v1 --from-beginning --max-messages 1 --timeout-ms 10000
kafka-console-consumer --bootstrap-server 127.0.0.1:9092 --topic vehicle.event.gb32960.v1 --from-beginning --max-messages 1 --timeout-ms 10000
kafka-console-consumer --bootstrap-server 127.0.0.1:9092 --topic vehicle.raw.mqtt-yutong.v1 --from-beginning --max-messages 1 --timeout-ms 10000
kafka-console-consumer --bootstrap-server 127.0.0.1:9092 --topic vehicle.event.mqtt-yutong.v1 --from-beginning --max-messages 1 --timeout-ms 10000

Use TDengine CLI or a JDBC client to verify history writes:

USE vehicle_ts;
SELECT COUNT(*) FROM raw_frames WHERE protocol = 'GB32960';
SELECT COUNT(*) FROM vehicle_locations WHERE protocol = 'GB32960';
SELECT COUNT(*) FROM raw_frames WHERE protocol = 'JT808';
SELECT COUNT(*) FROM jt808_locations WHERE protocol = 'JT808';
SELECT COUNT(*) FROM raw_frames WHERE protocol = 'MQTT_YUTONG';
SELECT COUNT(*) FROM vehicle_locations WHERE protocol = 'MQTT_YUTONG';

Useful HTTP query checks after events are consumed. For realtime_001.hex, the fixture VIN is LTEST000000000001; replace <date-from>, <date-to>, and <stat-date> with values that cover the consumed record's event time:

curl -sS 'http://127.0.0.1:20200/api/event-history/locations?protocol=GB32960&dateFrom=<date-from>&dateTo=<date-to>&vin=LTEST000000000001&limit=10'
curl -sS 'http://127.0.0.1:20200/api/event-history/raw-frames?protocol=GB32960&dateFrom=<date-from>&dateTo=<date-to>&vin=LTEST000000000001&limit=10'
curl -sS 'http://127.0.0.1:20200/api/event-history/gb32960/dictionary'
curl -sS 'http://127.0.0.1:20300/api/vehicle-stat/LTEST000000000001/daily?date=<stat-date>'

Expected E2E result when Kafka and all services are running:

• A GB32960 client receives a binary success ACK only after required Kafka production succeeds, and the captured ACK bytes are inspected.
• vehicle.raw.gb32960.v1 receives a raw frame envelope.
• vehicle.event.gb32960.v1 receives one or more normalized records.
• TDengine raw_frames receives GB32960/JT808 RAW rows with parsed JSON and metadata.
• TDengine raw_frames receives Yutong MQTT RAW rows with parsed JSON and metadata when MQTT payloads are consumed.
• TDengine location tables receive compact GB32960/JT808/Yutong MQTT location rows when applicable telemetry is present.
• JT808 analytics writes daily mileage rows to MySQL vehicle_stat_metric after consuming applicable vehicle.event.jt808.v1 records.

Do not expect vehicle-history-app to create raw .bin archive files from Kafka raw records in the current implementation. RawArchiveEventSink can write archive files only when it receives VehicleEvent.RawArchive.rawBytes() inside the same JVM; the Kafka envelope carries only RawArchiveRef metadata.

Do not mark any of these as verified unless the matching command was run and the output was inspected.

Local launchctl Deployment

Use the templates under deploy/local/launchctl/ for the local TCP-focused production-verification machine. They currently cover GB32960, JT808, and history; run Yutong MQTT separately with the command above or deploy it through Portainer.

• GB32960 ingest: com.lingniu.gb32960, TCP 32960, HTTP 20100
• JT808 ingest: com.lingniu.jt808, TCP 808, HTTP 20400
• History: com.lingniu.vehicle-history, HTTP 20200

The launchctl history template does not inject SINK_ARCHIVE_PATH. Use TDengine raw_frames as the authoritative hot-query store; keep any ingest-side raw archive path as an optional cold backup for operator inspection.

For Portainer deployments, mirror this same shape: no history archive volume, history consumes raw/event Kafka topics and writes TDengine history rows.

Rollback Guidance

If the split deployment is unhealthy:

1. Stop gb32960-ingest-app first so new GB32960 ACKs are not issued against an unhealthy Kafka boundary.
2. Keep Kafka topics intact for replay unless storage or privacy policy requires deletion.
3. Restart or roll back vehicle-history-app and vehicle-analytics-app independently; their failures do not require protocol ACK rollback because they consume from Kafka offsets.
4. If Kafka itself is unavailable, stop gb32960-ingest-app or route GB32960 traffic to a previously verified release that preserves the Kafka durability boundary.
5. After rollback, compare Kafka consumer group lag and DLQ contents before resuming the split services.

Rollback commands depend on the deployment environment. For local testing, stop the active Java processes and restart the previous verified app jars or images.

Task 8 Verification Notes

Observed on 2026-06-23 in worktree .worktrees/gb32960-service-split:

• Package build should use the current active-app command shown in the Build section.
• Repository-local Kafka setup inspection found no Kafka script, no Docker Compose file, and no compose YAML within the searched repository paths.
• nc -z -w 2 127.0.0.1 9092 exited 1, so no local Kafka broker was reachable at 127.0.0.1:9092.
• kafka-topics, kafka-topics.sh, docker, and docker-compose were not found on PATH.
• Kafka topic creation, service startup, health checks, Kafka record checks, archive checks, TDengine checks, stat output checks, and ACK observation were not run because the local Kafka prerequisite was absent.

Latest Local Verification

Observed on 2026-06-23 in worktree .worktrees/gb32960-service-split:

• Targeted module tests: mvn -pl :sink-kafka,:ingest-core,:protocol-gb32960,:event-history-service,:vehicle-state-service,:vehicle-stat-service test ended with BUILD SUCCESS; Maven reported 55 protocol/sink/core tests, 30 event-history tests, 14 vehicle-state tests, and 19 vehicle-stat tests with 0 failures/errors/skips.
• Split app tests should cover the current active apps listed in the Service Roles section.
• Package verification should use the active-app package command from the Build section.
• Split app startup: not run in this verification pass because local Kafka was still absent.
• Kafka raw records: not verified; no local Kafka broker or Kafka CLI tools were available.
• Kafka event records: not verified; no local Kafka broker or Kafka CLI tools were available.
• TDengine history rows: not verified; downstream services were not started without Kafka.
• Analytics output: not verified; downstream services were not started without Kafka.
• ACK behavior: not verified against a live broker in this pass. Unit tests cover the GB32960 ACK boundary and ordering, but an operator should still run the ACK capture command above in a Kafka-backed local or staging environment.

Remote Kafka Live Verification

Observed on 2026-06-23 with Kafka 114.55.58.251:9092:

• Replaced the local bootstrap-all listener on TCP 32960 with gb32960-ingest-app on TCP 32960 and HTTP 20100.
• Ingest app health check curl -sS http://127.0.0.1:20100/actuator/health returned {"status":"UP"}.
• Kafka AdminClient confirmed topics vehicle.raw.gb32960.v1 and vehicle.event.gb32960.v1 existed, and created vehicle.dlq.gb32960.v1.
• The app accepted the live Hyundai platform connection from 8.134.95.166, authenticated platform login, and received real GB32960 realtime/report/login/logout frames.
• Probe consumer with a temporary group read records from vehicle.event.gb32960.v1, including VIN keys such as LB9A32A2XR0LS1575 and LNXNEGRR3SR319485.
• Probe consumer with a temporary group read records from vehicle.raw.gb32960.v1, including VIN keys such as LNXNEGRR0SR319444, LNXNEGRR2SR318201, and LB9A32A23R0LS1532.
• Runtime logs showed successful GB32960 ACK flushes for platform login, vehicle login, and vehicle logout after the Kafka-backed split ingest service was running.

This verifies the live path GB32960 TCP 32960 -> gb32960-ingest-app -> Kafka raw/event topics against the remote Kafka broker. Downstream vehicle-history-app and vehicle-analytics-app were not started in this pass.

2026-06-29 Local Production-Path Verification

Observed on 2026-06-29 in worktree .worktrees/vehicle-ingest-redesign-phase1:

• jt808-ingest-app was running through launchctl on TCP 808 and HTTP 20400; curl -sS http://127.0.0.1:20400/actuator/health returned top-level status UP.
• gb32960-ingest-app was running through launchctl on TCP 32960 and HTTP 20100; curl -sS http://127.0.0.1:20100/actuator/health returned top-level status UP.
• vehicle-history-app was running through launchctl on HTTP 20200; curl -sS http://127.0.0.1:20200/actuator/health returned top-level status UP.
• External JT808 traffic on TCP 808 was parsed and written through Kafka into TDengine. Direct TDengine checks showed more than 6000 JT808 raw_frames, more than 1900 JT808 vehicle_locations, and more than 15000 JT808 telemetry_fields.
• A GB32960 synthetic realtime frame sent to TCP 32960 received a binary ACK beginning with 2323. The same VIN LTEST202606290001 was queryable from TDengine-backed history APIs with speed, mileage, longitude, and latitude fields.
• History raw and event Kafka consumption used separate bindings so raw-frame writes remained current while location/field event ingestion continued.

This verifies the local path GB32960/JT808 TCP -> protocol app -> Kafka raw/event topics -> vehicle-history-app -> TDengine raw_frames/location tables -> Swagger/API query.