254 lines
15 KiB
Markdown
254 lines
15 KiB
Markdown
# GB32960 Split Service Runbook
|
|
|
|
This runbook covers the split GB32960 ingest, history, and analytics runtimes. Kafka is the durability boundary between the protocol service and downstream business services.
|
|
|
|
## Service Roles
|
|
|
|
| Service | Module | Role | Local ports |
|
|
| --- | --- | --- | --- |
|
|
| GB32960 ingest | `:gb32960-ingest-app` | Accept GB32960 TCP connections, decode/authenticate frames, produce raw and normalized records to Kafka, and send protocol ACKs after required Kafka production succeeds. | TCP `32960`, HTTP `20100` |
|
|
| JT808 ingest | `:jt808-ingest-app` | Accept JT/T 808 TCP connections on the production receive port, parse raw frames, archive frame bytes, and produce raw/event envelopes to Kafka. | TCP `808`, HTTP `20482` |
|
|
| Vehicle history | `:vehicle-history-app` | Consume GB32960 raw/event Kafka records, store event-history records and raw archive references, and expose event/history/GB32960 query APIs. Kafka raw envelopes carry archive URI/size metadata, not raw frame bytes. | HTTP `20200` |
|
|
| Vehicle analytics | `:vehicle-analytics-app` | Consume GB32960 event Kafka records and update analytics outputs such as daily vehicle statistics. Vehicle state can also be enabled from this runtime. | HTTP `20300` |
|
|
|
|
## Kafka Contract
|
|
|
|
| Topic | Producer | Consumers | Purpose |
|
|
| --- | --- | --- | --- |
|
|
| `vehicle.raw.gb32960.v1` | `gb32960-ingest-app` | `vehicle-history-app` | Raw archive reference records. The envelope contains URI/size metadata, not the raw `.bin` payload. |
|
|
| `vehicle.event.gb32960.v1` | `gb32960-ingest-app` | `vehicle-history-app`, `vehicle-analytics-app` | Normalized vehicle events keyed by VIN when available. |
|
|
| `vehicle.dlq.gb32960.v1` | Kafka producer/consumer error paths | Operators/replay tooling | Dead-letter records for failed production or consumer processing. |
|
|
| `vehicle.raw.jt808.v1` | `jt808-ingest-app` | `vehicle-history-app` | JT808 raw archive reference records. |
|
|
| `vehicle.event.jt808.v1` | `jt808-ingest-app` | `vehicle-history-app` | JT808 parsed location/session/alarm events keyed by phone or mapped VIN. |
|
|
| `vehicle.dlq.jt808.v1` | Kafka producer/consumer error paths | Operators/replay tooling | Dead-letter records for failed JT808 production or consumer processing. |
|
|
|
|
Default local consumer groups:
|
|
|
|
- History: `vehicle-history`
|
|
- Analytics state: `vehicle-state` when `VEHICLE_STATE_ENABLED=true`
|
|
- Analytics statistics: `vehicle-stat`
|
|
|
|
## ACK Semantics
|
|
|
|
GB32960 success ACKs are sent only after the ingest service completes the required Kafka production boundary for the accepted frame. If required Kafka production fails, the ingest service must not claim a successful ACK for that frame.
|
|
|
|
History and analytics failures do not block GB32960 ACKs. They are downstream Kafka consumer failures and should be handled through retry, DLQ inspection, offset replay, and service-specific recovery.
|
|
|
|
Authorization failures are rejected before the Kafka durability boundary and can be ACKed/rejected immediately according to protocol handling.
|
|
|
|
## Local Prerequisites
|
|
|
|
- Java 25 and Maven available on PATH.
|
|
- Local Kafka reachable at `127.0.0.1:9092`, plus Kafka CLI tools such as `kafka-topics` and optionally `kafka-console-consumer`.
|
|
- No repository-local Kafka bootstrap script or compose file was found during Task 8 verification on 2026-06-23.
|
|
|
|
If your shell needs an explicit JDK:
|
|
|
|
```bash
|
|
export JAVA_HOME=/opt/homebrew/opt/openjdk@25/libexec/openjdk.jdk/Contents/Home
|
|
export PATH="$JAVA_HOME/bin:/opt/homebrew/bin:$PATH"
|
|
```
|
|
|
|
## Build
|
|
|
|
From the repository root:
|
|
|
|
```bash
|
|
mvn -pl :gb32960-ingest-app,:vehicle-history-app,:vehicle-analytics-app -am package -Dmaven.test.skip=true
|
|
```
|
|
|
|
Expected result: `BUILD SUCCESS` and these runnable jars:
|
|
|
|
- `modules/apps/gb32960-ingest-app/target/gb32960-ingest-app.jar`
|
|
- `modules/apps/vehicle-history-app/target/vehicle-history-app.jar`
|
|
- `modules/apps/vehicle-analytics-app/target/vehicle-analytics-app.jar`
|
|
|
|
## Create Topics
|
|
|
|
Run these only after Kafka is reachable at `127.0.0.1:9092`:
|
|
|
|
```bash
|
|
kafka-topics --bootstrap-server 127.0.0.1:9092 --create --if-not-exists --topic vehicle.raw.gb32960.v1 --partitions 12 --replication-factor 1
|
|
kafka-topics --bootstrap-server 127.0.0.1:9092 --create --if-not-exists --topic vehicle.event.gb32960.v1 --partitions 12 --replication-factor 1
|
|
kafka-topics --bootstrap-server 127.0.0.1:9092 --create --if-not-exists --topic vehicle.dlq.gb32960.v1 --partitions 3 --replication-factor 1
|
|
```
|
|
|
|
Optional sanity check:
|
|
|
|
```bash
|
|
kafka-topics --bootstrap-server 127.0.0.1:9092 --list | grep -E 'vehicle\.(raw|event|dlq)\.gb32960\.v1'
|
|
```
|
|
|
|
## Start Split Services Locally
|
|
|
|
Run each command in a separate terminal from the repository root.
|
|
|
|
GB32960 ingest:
|
|
|
|
```bash
|
|
KAFKA_BROKERS=127.0.0.1:9092 \
|
|
GB32960_PORT=32960 \
|
|
HTTP_PORT=20100 \
|
|
java --sun-misc-unsafe-memory-access=allow \
|
|
-jar modules/apps/gb32960-ingest-app/target/gb32960-ingest-app.jar
|
|
```
|
|
|
|
Vehicle history:
|
|
|
|
```bash
|
|
KAFKA_BROKERS=127.0.0.1:9092 \
|
|
HTTP_PORT=20200 \
|
|
EVENT_FILE_STORE_PATH=./target/split-event-store \
|
|
java --sun-misc-unsafe-memory-access=allow \
|
|
-jar modules/apps/vehicle-history-app/target/vehicle-history-app.jar
|
|
```
|
|
|
|
`SINK_ARCHIVE_PATH` is intentionally omitted in the split-service Kafka verification path. Current Kafka raw envelopes contain `RawArchiveRef` URI/size metadata only; they do not contain raw bytes that `RawArchiveEventSink` could materialize into `.bin` files in the history JVM.
|
|
|
|
Vehicle analytics:
|
|
|
|
```bash
|
|
KAFKA_BROKERS=127.0.0.1:9092 \
|
|
HTTP_PORT=20300 \
|
|
VEHICLE_STAT_FILE_PATH=./target/split-vehicle-stat \
|
|
java --sun-misc-unsafe-memory-access=allow \
|
|
-jar modules/apps/vehicle-analytics-app/target/vehicle-analytics-app.jar
|
|
```
|
|
|
|
To enable the latest-state API in analytics, also set `VEHICLE_STATE_ENABLED=true` and configure the state repository requirements used by `vehicle-state-service`.
|
|
|
|
## Health Verification
|
|
|
|
```bash
|
|
curl -sS http://127.0.0.1:20100/actuator/health
|
|
curl -sS http://127.0.0.1:20200/actuator/health
|
|
curl -sS http://127.0.0.1:20300/actuator/health
|
|
```
|
|
|
|
Expected: each endpoint returns `{"status":"UP"}` or an equivalent Spring Boot health JSON with top-level status `UP`.
|
|
|
|
## End-to-End Verification
|
|
|
|
Use a known-valid GB32960 fixture, for example one of the hex samples under `modules/protocols/protocol-gb32960/src/test/resources/samples/gb32960/`.
|
|
|
|
Example send command:
|
|
|
|
```bash
|
|
xxd -r -p modules/protocols/protocol-gb32960/src/test/resources/samples/gb32960/realtime_001.hex | nc 127.0.0.1 32960
|
|
```
|
|
|
|
To capture and inspect the binary ACK, write the response to a file:
|
|
|
|
```bash
|
|
rm -f /tmp/gb32960-ack.bin
|
|
xxd -r -p modules/protocols/protocol-gb32960/src/test/resources/samples/gb32960/realtime_001.hex \
|
|
| nc -w 3 127.0.0.1 32960 > /tmp/gb32960-ack.bin
|
|
xxd -p /tmp/gb32960-ack.bin
|
|
```
|
|
|
|
Expected: the ACK file is non-empty and the decoded bytes begin with the GB32960 frame marker `2323`. Treat the ACK as verified only after checking the response bytes from your run.
|
|
|
|
Verify only what was actually run in your environment:
|
|
|
|
```bash
|
|
kafka-console-consumer --bootstrap-server 127.0.0.1:9092 --topic vehicle.raw.gb32960.v1 --from-beginning --max-messages 1 --timeout-ms 10000
|
|
kafka-console-consumer --bootstrap-server 127.0.0.1:9092 --topic vehicle.event.gb32960.v1 --from-beginning --max-messages 1 --timeout-ms 10000
|
|
find target/split-event-store -type f | head
|
|
find target/split-vehicle-stat -type f | head
|
|
```
|
|
|
|
Useful HTTP query checks after events are consumed. For `realtime_001.hex`, the fixture VIN is `LTEST000000000001`; replace `<date-from>`, `<date-to>`, and `<stat-date>` with values that cover the consumed record's event time:
|
|
|
|
```bash
|
|
curl -sS 'http://127.0.0.1:20200/api/event-history/records?protocol=GB32960&dateFrom=<date-from>&dateTo=<date-to>&vin=LTEST000000000001&limit=10'
|
|
curl -sS 'http://127.0.0.1:20200/api/event-history/gb32960/dictionary'
|
|
curl -sS 'http://127.0.0.1:20300/api/vehicle-stat/LTEST000000000001/daily?date=<stat-date>'
|
|
```
|
|
|
|
Expected E2E result when Kafka and all services are running:
|
|
|
|
- A GB32960 client receives a binary success ACK only after required Kafka production succeeds, and the captured ACK bytes are inspected.
|
|
- `vehicle.raw.gb32960.v1` receives a raw archive reference envelope.
|
|
- `vehicle.event.gb32960.v1` receives one or more normalized records.
|
|
- `target/split-event-store` receives event-store files after the history service flushes consumed Kafka records, including raw archive reference records.
|
|
- `target/split-vehicle-stat` receives stat output after analytics consumes applicable event records.
|
|
|
|
Do not expect `vehicle-history-app` to create raw `.bin` archive files from Kafka raw records in the current implementation. `RawArchiveEventSink` can write archive files only when it receives `VehicleEvent.RawArchive.rawBytes()` inside the same JVM; the Kafka envelope carries only `RawArchiveRef` metadata.
|
|
|
|
Do not mark any of these as verified unless the matching command was run and the output was inspected.
|
|
|
|
## Local launchctl Deployment
|
|
|
|
Use the templates under `deploy/local/launchctl/` for the local production-verification machine. They codify the current three-service deployment and keep the runtime knobs in one place:
|
|
|
|
- GB32960 ingest: `com.lingniu.gb32960`, TCP `32960`, HTTP `20100`
|
|
- JT808 ingest: `com.lingniu.jt808`, TCP `808`, HTTP `20482`
|
|
- History: `com.lingniu.vehicle-history`, HTTP `20200`
|
|
|
|
The two ingest services and history service must share the same `SINK_ARCHIVE_PATH`. Kafka raw messages contain `archive://...` references, so history can only read raw frames when its archive root points to the same directory or shared volume that ingest used to write the `.bin` files.
|
|
|
|
For Portainer deployments, mirror this same rule with a shared volume mounted to all three services, for example `/archive` in `deploy/portainer/docker-compose.yml`.
|
|
|
|
## Rollback Guidance
|
|
|
|
If the split deployment is unhealthy:
|
|
|
|
1. Stop `gb32960-ingest-app` first so new GB32960 ACKs are not issued against an unhealthy Kafka boundary.
|
|
2. Keep Kafka topics intact for replay unless storage or privacy policy requires deletion.
|
|
3. Restart or roll back `vehicle-history-app` and `vehicle-analytics-app` independently; their failures do not require protocol ACK rollback because they consume from Kafka offsets.
|
|
4. If Kafka itself is unavailable, stop `gb32960-ingest-app` or route GB32960 traffic to a previously verified release that preserves the Kafka durability boundary.
|
|
5. After rollback, compare Kafka consumer group lag and DLQ contents before resuming the split services.
|
|
|
|
Rollback commands depend on the deployment environment. For local testing, stop the three Java processes and restart the previous verified app jars or images.
|
|
|
|
## Task 8 Verification Notes
|
|
|
|
Observed on 2026-06-23 in worktree `.worktrees/gb32960-service-split`:
|
|
|
|
- Package build ran with `mvn -pl :gb32960-ingest-app,:vehicle-history-app,:vehicle-analytics-app -am package -Dmaven.test.skip=true` and ended with `BUILD SUCCESS`.
|
|
- Repository-local Kafka setup inspection found no Kafka script, no Docker Compose file, and no compose YAML within the searched repository paths.
|
|
- `nc -z -w 2 127.0.0.1 9092` exited `1`, so no local Kafka broker was reachable at `127.0.0.1:9092`.
|
|
- `kafka-topics`, `kafka-topics.sh`, `docker`, and `docker-compose` were not found on PATH.
|
|
- Kafka topic creation, service startup, health checks, Kafka record checks, archive checks, event-store checks, stat output checks, and ACK observation were not run because the local Kafka prerequisite was absent.
|
|
|
|
## Latest Local Verification
|
|
|
|
Observed on 2026-06-23 in worktree `.worktrees/gb32960-service-split`:
|
|
|
|
- Targeted module tests: `mvn -pl :sink-mq,:ingest-core,:protocol-gb32960,:event-history-service,:vehicle-state-service,:vehicle-stat-service test` ended with `BUILD SUCCESS`; Maven reported 55 protocol/sink/core tests, 30 event-history tests, 14 vehicle-state tests, and 19 vehicle-stat tests with 0 failures/errors/skips.
|
|
- Split app tests: `mvn -pl :gb32960-ingest-app,:vehicle-history-app,:vehicle-analytics-app test` ended with `BUILD SUCCESS`; Maven reported 6 app composition/default tests with 0 failures/errors/skips.
|
|
- Package verification: `mvn -pl :gb32960-ingest-app,:vehicle-history-app,:vehicle-analytics-app -am package -Dmaven.test.skip=true` ended with `BUILD SUCCESS` and repackaged all three app jars.
|
|
- Split app startup: not run in this verification pass because local Kafka was still absent.
|
|
- Kafka raw records: not verified; no local Kafka broker or Kafka CLI tools were available.
|
|
- Kafka event records: not verified; no local Kafka broker or Kafka CLI tools were available.
|
|
- Event-store files: not verified; downstream services were not started without Kafka.
|
|
- Analytics output: not verified; downstream services were not started without Kafka.
|
|
- ACK behavior: not verified against a live broker in this pass. Unit tests cover the GB32960 ACK boundary and ordering, but an operator should still run the ACK capture command above in a Kafka-backed local or staging environment.
|
|
|
|
## Remote Kafka Live Verification
|
|
|
|
Observed on 2026-06-23 with Kafka `114.55.58.251:9092`:
|
|
|
|
- Replaced the local `bootstrap-all` listener on TCP `32960` with `gb32960-ingest-app` on TCP `32960` and HTTP `20100`.
|
|
- Ingest app health check `curl -sS http://127.0.0.1:20100/actuator/health` returned `{"status":"UP"}`.
|
|
- Kafka AdminClient confirmed topics `vehicle.raw.gb32960.v1` and `vehicle.event.gb32960.v1` existed, and created `vehicle.dlq.gb32960.v1`.
|
|
- The app accepted the live Hyundai platform connection from `8.134.95.166`, authenticated platform login, and received real GB32960 realtime/report/login/logout frames.
|
|
- Probe consumer with a temporary group read records from `vehicle.event.gb32960.v1`, including VIN keys such as `LB9A32A2XR0LS1575` and `LNXNEGRR3SR319485`.
|
|
- Probe consumer with a temporary group read records from `vehicle.raw.gb32960.v1`, including VIN keys such as `LNXNEGRR0SR319444`, `LNXNEGRR2SR318201`, and `LB9A32A23R0LS1532`.
|
|
- Runtime logs showed successful GB32960 ACK flushes for platform login, vehicle login, and vehicle logout after the Kafka-backed split ingest service was running.
|
|
|
|
This verifies the live path `GB32960 TCP 32960 -> gb32960-ingest-app -> Kafka raw/event topics` against the remote Kafka broker. Downstream `vehicle-history-app` and `vehicle-analytics-app` were not started in this pass.
|
|
|
|
## 2026-06-29 Local Production-Path Verification
|
|
|
|
Observed on 2026-06-29 in worktree `.worktrees/vehicle-ingest-redesign-phase1`:
|
|
|
|
- `jt808-ingest-app` was running through launchctl on TCP `808` and HTTP `20482`; `curl -sS http://127.0.0.1:20482/actuator/health` returned top-level status `UP`.
|
|
- `gb32960-ingest-app` was running through launchctl on TCP `32960` and HTTP `20100`; `curl -sS http://127.0.0.1:20100/actuator/health` returned top-level status `UP`.
|
|
- `vehicle-history-app` was running through launchctl on HTTP `20200`; `curl -sS http://127.0.0.1:20200/actuator/health` returned top-level status `UP`.
|
|
- External JT808 traffic on TCP `808` was parsed and written through Kafka into TDengine. Direct TDengine checks showed more than `6000` JT808 `raw_frames`, more than `1900` JT808 `vehicle_locations`, and more than `15000` JT808 `telemetry_fields`.
|
|
- A GB32960 synthetic realtime frame sent to TCP `32960` received a binary ACK beginning with `2323`. The same VIN `LTEST202606290001` was queryable from TDengine-backed history APIs with speed, mileage, longitude, and latitude fields.
|
|
- History raw and event Kafka consumption used separate bindings so raw-frame writes remained current while location/field event ingestion continued.
|
|
|
|
This verifies the local path `GB32960/JT808 TCP -> protocol app -> shared raw archive -> Kafka raw/event topics -> vehicle-history-app -> TDengine -> Swagger/API query`.
|