shishengliang/lingniu-vehicle-ingest

Fork 0

Files

lingniu 22468e78b4

ci/woodpecker/manual/woodpecker Pipeline was canceled

Details

chore: wire split apps deployment config

2026-06-24 14:38:59 +08:00

13 KiB

Raw Blame History

GB32960 Split Service Runbook

This runbook covers the split GB32960 ingest, history, and analytics runtimes. Kafka is the durability boundary between the protocol service and downstream business services.

Service Roles

Service	Module	Role	Local ports
GB32960 ingest	`:gb32960-ingest-app`	Accept GB32960 TCP connections, decode/authenticate frames, produce raw and normalized records to Kafka, and send protocol ACKs after required Kafka production succeeds.	TCP `32960`, HTTP `20100`
Vehicle history	`:vehicle-history-app`	Consume GB32960 raw/event Kafka records, store event-history records and raw archive references, and expose event/history/GB32960 query APIs. Kafka raw envelopes carry archive URI/size metadata, not raw frame bytes.	HTTP `20200`
Vehicle analytics	`:vehicle-analytics-app`	Consume GB32960 event Kafka records and update analytics outputs such as daily vehicle statistics. Vehicle state can also be enabled from this runtime.	HTTP `20300`

Kafka Contract

Topic	Producer	Consumers	Purpose
`vehicle.raw.gb32960.v1`	`gb32960-ingest-app`	`vehicle-history-app`	Raw archive reference records. The envelope contains URI/size metadata, not the raw `.bin` payload.
`vehicle.event.gb32960.v1`	`gb32960-ingest-app`	`vehicle-history-app`, `vehicle-analytics-app`	Normalized vehicle events keyed by VIN when available.
`vehicle.dlq.gb32960.v1`	Kafka producer/consumer error paths	Operators/replay tooling	Dead-letter records for failed production or consumer processing.

Default local consumer groups:

History: vehicle-history
Analytics state: vehicle-state when VEHICLE_STATE_ENABLED=true
Analytics statistics: vehicle-stat

ACK Semantics

GB32960 success ACKs are sent only after the ingest service completes the required Kafka production boundary for the accepted frame. If required Kafka production fails, the ingest service must not claim a successful ACK for that frame.

History and analytics failures do not block GB32960 ACKs. They are downstream Kafka consumer failures and should be handled through retry, DLQ inspection, offset replay, and service-specific recovery.

Authorization failures are rejected before the Kafka durability boundary and can be ACKed/rejected immediately according to protocol handling.

Local Prerequisites

Java 25 and Maven available on PATH.
Local Kafka reachable at 127.0.0.1:9092, plus Kafka CLI tools such as kafka-topics and optionally kafka-console-consumer.
No repository-local Kafka bootstrap script or compose file was found during Task 8 verification on 2026-06-23.

If your shell needs an explicit JDK:

export JAVA_HOME=/opt/homebrew/opt/openjdk@25/libexec/openjdk.jdk/Contents/Home
export PATH="$JAVA_HOME/bin:/opt/homebrew/bin:$PATH"

Build

From the repository root:

mvn -pl :gb32960-ingest-app,:vehicle-history-app,:vehicle-analytics-app -am package -Dmaven.test.skip=true

Expected result: BUILD SUCCESS and these runnable jars:

modules/apps/gb32960-ingest-app/target/gb32960-ingest-app.jar
modules/apps/vehicle-history-app/target/vehicle-history-app.jar
modules/apps/vehicle-analytics-app/target/vehicle-analytics-app.jar

Create Topics

Run these only after Kafka is reachable at 127.0.0.1:9092:

kafka-topics --bootstrap-server 127.0.0.1:9092 --create --if-not-exists --topic vehicle.raw.gb32960.v1 --partitions 12 --replication-factor 1
kafka-topics --bootstrap-server 127.0.0.1:9092 --create --if-not-exists --topic vehicle.event.gb32960.v1 --partitions 12 --replication-factor 1
kafka-topics --bootstrap-server 127.0.0.1:9092 --create --if-not-exists --topic vehicle.dlq.gb32960.v1 --partitions 3 --replication-factor 1

Optional sanity check:

kafka-topics --bootstrap-server 127.0.0.1:9092 --list | grep -E 'vehicle\.(raw|event|dlq)\.gb32960\.v1'

Start Split Services Locally

Run each command in a separate terminal from the repository root.

GB32960 ingest:

KAFKA_BROKERS=127.0.0.1:9092 \
GB32960_PORT=32960 \
HTTP_PORT=20100 \
java --sun-misc-unsafe-memory-access=allow \
  -jar modules/apps/gb32960-ingest-app/target/gb32960-ingest-app.jar

Vehicle history:

KAFKA_BROKERS=127.0.0.1:9092 \
HTTP_PORT=20200 \
EVENT_FILE_STORE_PATH=./target/split-event-store \
java --sun-misc-unsafe-memory-access=allow \
  -jar modules/apps/vehicle-history-app/target/vehicle-history-app.jar

SINK_ARCHIVE_PATH is intentionally omitted in the split-service Kafka verification path. Current Kafka raw envelopes contain RawArchiveRef URI/size metadata only; they do not contain raw bytes that RawArchiveEventSink could materialize into .bin files in the history JVM.

Vehicle analytics:

KAFKA_BROKERS=127.0.0.1:9092 \
HTTP_PORT=20300 \
VEHICLE_STAT_FILE_PATH=./target/split-vehicle-stat \
java --sun-misc-unsafe-memory-access=allow \
  -jar modules/apps/vehicle-analytics-app/target/vehicle-analytics-app.jar

To enable the latest-state API in analytics, also set VEHICLE_STATE_ENABLED=true and configure the state repository requirements used by vehicle-state-service.

Health Verification

curl -sS http://127.0.0.1:20100/actuator/health
curl -sS http://127.0.0.1:20200/actuator/health
curl -sS http://127.0.0.1:20300/actuator/health

Expected: each endpoint returns {"status":"UP"} or an equivalent Spring Boot health JSON with top-level status UP.

End-to-End Verification

Use a known-valid GB32960 fixture, for example one of the hex samples under modules/protocols/protocol-gb32960/src/test/resources/samples/gb32960/.

Example send command:

xxd -r -p modules/protocols/protocol-gb32960/src/test/resources/samples/gb32960/realtime_001.hex | nc 127.0.0.1 32960

To capture and inspect the binary ACK, write the response to a file:

rm -f /tmp/gb32960-ack.bin
xxd -r -p modules/protocols/protocol-gb32960/src/test/resources/samples/gb32960/realtime_001.hex \
  | nc -w 3 127.0.0.1 32960 > /tmp/gb32960-ack.bin
xxd -p /tmp/gb32960-ack.bin

Expected: the ACK file is non-empty and the decoded bytes begin with the GB32960 frame marker 2323. Treat the ACK as verified only after checking the response bytes from your run.

Verify only what was actually run in your environment:

kafka-console-consumer --bootstrap-server 127.0.0.1:9092 --topic vehicle.raw.gb32960.v1 --from-beginning --max-messages 1 --timeout-ms 10000
kafka-console-consumer --bootstrap-server 127.0.0.1:9092 --topic vehicle.event.gb32960.v1 --from-beginning --max-messages 1 --timeout-ms 10000
find target/split-event-store -type f | head
find target/split-vehicle-stat -type f | head

Useful HTTP query checks after events are consumed. For realtime_001.hex, the fixture VIN is LTEST000000000001; replace <date-from>, <date-to>, and <stat-date> with values that cover the consumed record's event time:

curl -sS 'http://127.0.0.1:20200/api/event-history/records?protocol=GB32960&dateFrom=<date-from>&dateTo=<date-to>&vin=LTEST000000000001&limit=10'
curl -sS 'http://127.0.0.1:20200/api/event-history/gb32960/dictionary'
curl -sS 'http://127.0.0.1:20300/api/vehicle-stat/LTEST000000000001/daily?date=<stat-date>'

Expected E2E result when Kafka and all services are running:

A GB32960 client receives a binary success ACK only after required Kafka production succeeds, and the captured ACK bytes are inspected.
vehicle.raw.gb32960.v1 receives a raw archive reference envelope.
vehicle.event.gb32960.v1 receives one or more normalized records.
target/split-event-store receives event-store files after the history service flushes consumed Kafka records, including raw archive reference records.
target/split-vehicle-stat receives stat output after analytics consumes applicable event records.

Do not expect vehicle-history-app to create raw .bin archive files from Kafka raw records in the current implementation. RawArchiveEventSink can write archive files only when it receives VehicleEvent.RawArchive.rawBytes() inside the same JVM; the Kafka envelope carries only RawArchiveRef metadata.

Do not mark any of these as verified unless the matching command was run and the output was inspected.

Rollback Guidance

If the split deployment is unhealthy:

Stop gb32960-ingest-app first so new GB32960 ACKs are not issued against an unhealthy Kafka boundary.
Keep Kafka topics intact for replay unless storage or privacy policy requires deletion.
Restart or roll back vehicle-history-app and vehicle-analytics-app independently; their failures do not require protocol ACK rollback because they consume from Kafka offsets.
If Kafka itself is unavailable, stop gb32960-ingest-app or route GB32960 traffic to a previously verified release that preserves the Kafka durability boundary.
After rollback, compare Kafka consumer group lag and DLQ contents before resuming the split services.

Rollback commands depend on the deployment environment. For local testing, stop the three Java processes and restart the previous verified app jars or images.

Task 8 Verification Notes

Observed on 2026-06-23 in worktree .worktrees/gb32960-service-split:

Package build ran with mvn -pl :gb32960-ingest-app,:vehicle-history-app,:vehicle-analytics-app -am package -Dmaven.test.skip=true and ended with BUILD SUCCESS.
Repository-local Kafka setup inspection found no Kafka script, no Docker Compose file, and no compose YAML within the searched repository paths.
nc -z -w 2 127.0.0.1 9092 exited 1, so no local Kafka broker was reachable at 127.0.0.1:9092.
kafka-topics, kafka-topics.sh, docker, and docker-compose were not found on PATH.
Kafka topic creation, service startup, health checks, Kafka record checks, archive checks, event-store checks, stat output checks, and ACK observation were not run because the local Kafka prerequisite was absent.

Latest Local Verification