274 lines
19 KiB
Markdown
274 lines
19 KiB
Markdown
# GB32960 Split Service Runbook
|
|
|
|
This runbook covers the default production runtimes: GB32960 ingest, JT808 ingest, Yutong MQTT ingest, history, and analytics. Kafka is the durability boundary between protocol ingest services and downstream business services.
|
|
|
|
## Service Roles
|
|
|
|
| Service | Module | Role | Production ports |
|
|
| --- | --- | --- | --- |
|
|
| GB32960 ingest | `:gb32960-ingest-app` | Accept GB32960 TCP connections, decode/authenticate frames, produce raw and normalized records to Kafka, and send protocol ACKs after required Kafka production succeeds. | TCP `32960`, HTTP `20100` |
|
|
| JT808 ingest | `:jt808-ingest-app` | Accept JT/T 808 TCP connections on the production receive port, parse raw frames, archive frame bytes, and produce raw/event envelopes to Kafka. | TCP `808`, HTTP `20400` |
|
|
| Yutong MQTT ingest | `:yutong-mqtt-app` | Connect to Yutong MQTT, parse subscribed messages, archive payloads, and produce raw/event envelopes to Kafka. | HTTP `20500` |
|
|
| Vehicle history | `:vehicle-history-app` | Consume GB32960/JT808/Yutong MQTT raw/event Kafka records, store compact history rows plus TDengine `raw_frames` `parsedJson`/`metadataJson`/`rawUri`, and expose TDengine-backed history APIs. | HTTP `20200` |
|
|
| Vehicle analytics | `:vehicle-analytics-app` | Consume JT808 event Kafka records, calculate daily mileage from reported GPS total mileage, and write metrics to `vehicle_stat_metric`. | HTTP `20300` |
|
|
|
|
## Kafka Contract
|
|
|
|
| Topic | Producer | Consumers | Purpose |
|
|
| --- | --- | --- | --- |
|
|
| `vehicle.raw.gb32960.v1` | `gb32960-ingest-app` | `vehicle-history-app` | Raw frame records. History writes the parsed JSON, metadata JSON, and raw URI to TDengine `raw_frames`. |
|
|
| `vehicle.event.gb32960.v1` | `gb32960-ingest-app` | `vehicle-history-app` | Normalized vehicle events keyed by VIN when available. |
|
|
| `vehicle.dlq.gb32960.v1` | Kafka producer/consumer error paths | Operators/replay tooling | Dead-letter records for failed production or consumer processing. |
|
|
| `vehicle.raw.jt808.v1` | `jt808-ingest-app` | `vehicle-history-app` | JT808 raw frame records for TDengine `raw_frames` and raw-frame troubleshooting APIs. |
|
|
| `vehicle.event.jt808.v1` | `jt808-ingest-app` | `vehicle-history-app`, `vehicle-analytics-app` | JT808 parsed location/session/alarm events keyed by phone or mapped VIN; analytics uses location total mileage for daily mileage metrics. |
|
|
| `vehicle.dlq.jt808.v1` | Kafka producer/consumer error paths | Operators/replay tooling | Dead-letter records for failed JT808 production or consumer processing. |
|
|
| `vehicle.raw.mqtt-yutong.v1` | `yutong-mqtt-app` | `vehicle-history-app` | Yutong MQTT raw payload records for TDengine `raw_frames` and troubleshooting APIs. |
|
|
| `vehicle.event.mqtt-yutong.v1` | `yutong-mqtt-app` | `vehicle-history-app` | Yutong MQTT parsed telemetry events keyed by mapped VIN or device identity. |
|
|
| `vehicle.dlq.mqtt-yutong.v1` | Kafka producer/consumer error paths | Operators/replay tooling | Dead-letter records for failed Yutong MQTT production or consumer processing. |
|
|
| `vehicle.dlq.history.v1` | History app consumer error paths | Operators/replay tooling | Dead-letter records for failed history storage processing. |
|
|
|
|
Default consumer groups:
|
|
|
|
- History: `vehicle-history`
|
|
- Analytics statistics: `vehicle-stat`
|
|
|
|
## ACK Semantics
|
|
|
|
GB32960 success ACKs are sent only after the ingest service completes the required Kafka production boundary for the accepted frame. If required Kafka production fails, the ingest service must not claim a successful ACK for that frame.
|
|
|
|
History and analytics failures do not block GB32960 ACKs. They are downstream Kafka consumer failures and should be handled through retry, DLQ inspection, offset replay, and service-specific recovery.
|
|
|
|
Authorization failures are rejected before the Kafka durability boundary and can be ACKed/rejected immediately according to protocol handling.
|
|
|
|
## Build Prerequisites
|
|
|
|
- Java 25 and Maven available on PATH.
|
|
- Kafka CLI tools such as `kafka-topics` and optionally `kafka-console-consumer` available on the ECS/operator host.
|
|
- Session state requires Redis. GB32960/JT808 ingest services keep live Netty
|
|
channels in-process, but session indexes and TTL metadata use Redis only.
|
|
- Redis is configured through Portainer environment variables such as `REDIS_HOST` and `REDIS_DATABASE=50`; there is no memory session-store fallback.
|
|
|
|
If your shell needs an explicit JDK:
|
|
|
|
```bash
|
|
export JAVA_HOME=/opt/homebrew/opt/openjdk@25/libexec/openjdk.jdk/Contents/Home
|
|
export PATH="$JAVA_HOME/bin:/opt/homebrew/bin:$PATH"
|
|
```
|
|
|
|
## Build
|
|
|
|
From the repository root:
|
|
|
|
```bash
|
|
mvn -pl :gb32960-ingest-app,:jt808-ingest-app,:yutong-mqtt-app,:vehicle-history-app,:vehicle-analytics-app -am package -Dmaven.test.skip=true
|
|
```
|
|
|
|
Expected result: `BUILD SUCCESS` and these runnable jars:
|
|
|
|
- `modules/apps/gb32960-ingest-app/target/gb32960-ingest-app.jar`
|
|
- `modules/apps/jt808-ingest-app/target/jt808-ingest-app.jar`
|
|
- `modules/apps/yutong-mqtt-app/target/yutong-mqtt-app.jar`
|
|
- `modules/apps/vehicle-history-app/target/vehicle-history-app.jar`
|
|
- `modules/apps/vehicle-analytics-app/target/vehicle-analytics-app.jar`
|
|
|
|
## Create Topics
|
|
|
|
Run these only on the ECS/operator host after Kafka is reachable. The default bootstrap address is the ECS private Kafka address used by the production stack:
|
|
|
|
```bash
|
|
KAFKA_BOOTSTRAP="${KAFKA_BOOTSTRAP:-172.17.111.56:9092}"
|
|
kafka-topics --bootstrap-server "$KAFKA_BOOTSTRAP" --create --if-not-exists --topic vehicle.raw.gb32960.v1 --partitions 12 --replication-factor 1
|
|
kafka-topics --bootstrap-server "$KAFKA_BOOTSTRAP" --create --if-not-exists --topic vehicle.event.gb32960.v1 --partitions 12 --replication-factor 1
|
|
kafka-topics --bootstrap-server "$KAFKA_BOOTSTRAP" --create --if-not-exists --topic vehicle.dlq.gb32960.v1 --partitions 3 --replication-factor 1
|
|
kafka-topics --bootstrap-server "$KAFKA_BOOTSTRAP" --create --if-not-exists --topic vehicle.raw.jt808.v1 --partitions 12 --replication-factor 1
|
|
kafka-topics --bootstrap-server "$KAFKA_BOOTSTRAP" --create --if-not-exists --topic vehicle.event.jt808.v1 --partitions 12 --replication-factor 1
|
|
kafka-topics --bootstrap-server "$KAFKA_BOOTSTRAP" --create --if-not-exists --topic vehicle.dlq.jt808.v1 --partitions 3 --replication-factor 1
|
|
kafka-topics --bootstrap-server "$KAFKA_BOOTSTRAP" --create --if-not-exists --topic vehicle.raw.mqtt-yutong.v1 --partitions 12 --replication-factor 1
|
|
kafka-topics --bootstrap-server "$KAFKA_BOOTSTRAP" --create --if-not-exists --topic vehicle.event.mqtt-yutong.v1 --partitions 12 --replication-factor 1
|
|
kafka-topics --bootstrap-server "$KAFKA_BOOTSTRAP" --create --if-not-exists --topic vehicle.dlq.mqtt-yutong.v1 --partitions 3 --replication-factor 1
|
|
kafka-topics --bootstrap-server "$KAFKA_BOOTSTRAP" --create --if-not-exists --topic vehicle.dlq.history.v1 --partitions 3 --replication-factor 1
|
|
```
|
|
|
|
Optional sanity check:
|
|
|
|
```bash
|
|
kafka-topics --bootstrap-server "$KAFKA_BOOTSTRAP" --list | grep -E 'vehicle\.(raw|event|dlq)\.(gb32960|jt808|mqtt-yutong|history)\.v1'
|
|
```
|
|
|
|
## Production Runtime
|
|
|
|
Production services run on ECS through Portainer/Docker. Do not start GB32960, JT808, Yutong MQTT, history, or analytics services on local developer machines; local machines are for code build, unit tests, and source inspection only.
|
|
|
|
Use `deploy/portainer/docker-compose.yml` as the production runtime manifest. The default stack is:
|
|
|
|
- `gb32960-ingest-app`
|
|
- `jt808-ingest-app`
|
|
- `yutong-mqtt-app`
|
|
- `vehicle-history-app`
|
|
- `vehicle-analytics-app`
|
|
|
|
`vehicle-history-app` intentionally has no raw archive volume. The history hot path is Kafka to TDengine: raw envelopes are stored in TDengine `raw_frames` with `parsedJson`, `metadataJson`, and `rawUri`. Ingest services may still write original `.bin` files as cold backup, but history API queries must not depend on a shared local archive mount.
|
|
|
|
Keep `TDENGINE_HISTORY_ENABLED=true` in the `vehicle-history-app` environment so history reads and writes use TDengine in production.
|
|
|
|
`vehicle-analytics-app` is intentionally a metric runtime only. Latest-state APIs belong to `vehicle-state-service`, which stays outside the default reactor and should be built with `-Poptional-latest-state` and deployed as a separate consumer if that surface is needed.
|
|
|
|
## Health Verification
|
|
|
|
Run these from the ECS host or inside the matching container:
|
|
|
|
```bash
|
|
curl -sS http://127.0.0.1:20100/actuator/health
|
|
curl -sS http://127.0.0.1:20100/actuator/health/liveness
|
|
curl -sS http://127.0.0.1:20100/actuator/health/readiness
|
|
curl -sS http://127.0.0.1:20400/actuator/health
|
|
curl -sS http://127.0.0.1:20400/actuator/health/liveness
|
|
curl -sS http://127.0.0.1:20400/actuator/health/readiness
|
|
curl -sS http://127.0.0.1:20500/actuator/health
|
|
curl -sS http://127.0.0.1:20500/actuator/health/liveness
|
|
curl -sS http://127.0.0.1:20500/actuator/health/readiness
|
|
curl -sS http://127.0.0.1:20200/actuator/health
|
|
curl -sS http://127.0.0.1:20200/actuator/health/liveness
|
|
curl -sS http://127.0.0.1:20200/actuator/health/readiness
|
|
curl -sS http://127.0.0.1:20300/actuator/health
|
|
curl -sS http://127.0.0.1:20300/actuator/health/liveness
|
|
curl -sS http://127.0.0.1:20300/actuator/health/readiness
|
|
```
|
|
|
|
Expected: each aggregate, liveness, and readiness endpoint returns `{"status":"UP"}` or an equivalent Spring Boot health JSON with top-level status `UP`.
|
|
|
|
## End-to-End Verification
|
|
|
|
Use a known-valid GB32960 fixture, for example one of the hex samples under `modules/protocols/protocol-gb32960/src/test/resources/samples/gb32960/`.
|
|
|
|
Example send command:
|
|
|
|
```bash
|
|
xxd -r -p modules/protocols/protocol-gb32960/src/test/resources/samples/gb32960/realtime_001.hex | nc 127.0.0.1 32960
|
|
```
|
|
|
|
To capture and inspect the binary ACK, write the response to a file:
|
|
|
|
```bash
|
|
rm -f /tmp/gb32960-ack.bin
|
|
xxd -r -p modules/protocols/protocol-gb32960/src/test/resources/samples/gb32960/realtime_001.hex \
|
|
| nc -w 3 127.0.0.1 32960 > /tmp/gb32960-ack.bin
|
|
xxd -p /tmp/gb32960-ack.bin
|
|
```
|
|
|
|
Expected: the ACK file is non-empty and the decoded bytes begin with the GB32960 frame marker `2323`. Treat the ACK as verified only after checking the response bytes from your run.
|
|
|
|
Verify only what was actually run in your environment:
|
|
|
|
```bash
|
|
kafka-console-consumer --bootstrap-server "$KAFKA_BOOTSTRAP" --topic vehicle.raw.gb32960.v1 --from-beginning --max-messages 1 --timeout-ms 10000
|
|
kafka-console-consumer --bootstrap-server "$KAFKA_BOOTSTRAP" --topic vehicle.event.gb32960.v1 --from-beginning --max-messages 1 --timeout-ms 10000
|
|
kafka-console-consumer --bootstrap-server "$KAFKA_BOOTSTRAP" --topic vehicle.raw.mqtt-yutong.v1 --from-beginning --max-messages 1 --timeout-ms 10000
|
|
kafka-console-consumer --bootstrap-server "$KAFKA_BOOTSTRAP" --topic vehicle.event.mqtt-yutong.v1 --from-beginning --max-messages 1 --timeout-ms 10000
|
|
```
|
|
|
|
Use TDengine CLI or a JDBC client to verify history writes:
|
|
|
|
```sql
|
|
USE vehicle_ts;
|
|
SELECT COUNT(*) FROM raw_frames WHERE protocol = 'GB32960';
|
|
SELECT COUNT(*) FROM vehicle_locations WHERE protocol = 'GB32960';
|
|
SELECT COUNT(*) FROM raw_frames WHERE protocol = 'JT808';
|
|
SELECT COUNT(*) FROM jt808_locations WHERE protocol = 'JT808';
|
|
SELECT COUNT(*) FROM raw_frames WHERE protocol = 'MQTT_YUTONG';
|
|
SELECT COUNT(*) FROM vehicle_locations WHERE protocol = 'MQTT_YUTONG';
|
|
```
|
|
|
|
Useful HTTP query checks after events are consumed. For `realtime_001.hex`, the fixture VIN is `LTEST000000000001`; replace `<date-from>`, `<date-to>`, and `<stat-date>` with values that cover the consumed record's event time:
|
|
|
|
```bash
|
|
curl -sS 'http://127.0.0.1:20200/api/event-history/locations?protocol=GB32960&dateFrom=<date-from>&dateTo=<date-to>&vin=LTEST000000000001&limit=10'
|
|
curl -sS 'http://127.0.0.1:20200/api/event-history/raw-frames?protocol=GB32960&dateFrom=<date-from>&dateTo=<date-to>&vin=LTEST000000000001&limit=10'
|
|
curl -sS 'http://127.0.0.1:20200/api/event-history/gb32960/dictionary'
|
|
curl -sS 'http://127.0.0.1:20300/api/vehicle-stat/LTEST000000000001/daily?date=<stat-date>'
|
|
```
|
|
|
|
Expected E2E result when Kafka and all services are running:
|
|
|
|
- A GB32960 client receives a binary success ACK only after required Kafka production succeeds, and the captured ACK bytes are inspected.
|
|
- `vehicle.raw.gb32960.v1` receives a raw frame envelope.
|
|
- `vehicle.event.gb32960.v1` receives one or more normalized records.
|
|
- TDengine `raw_frames` receives GB32960/JT808 RAW rows with parsed JSON and metadata.
|
|
- TDengine `raw_frames` receives Yutong MQTT RAW rows with parsed JSON and metadata when MQTT payloads are consumed.
|
|
- TDengine location tables receive compact GB32960/JT808/Yutong MQTT location rows when applicable telemetry is present.
|
|
- JT808 analytics writes daily mileage rows to MySQL `vehicle_stat_metric` after consuming applicable `vehicle.event.jt808.v1` records.
|
|
|
|
Do not expect `vehicle-history-app` to create raw `.bin` archive files from Kafka raw records in the current implementation. `RawArchiveEventSink` can write archive files only when it receives `VehicleEvent.RawArchive.rawBytes()` inside the same JVM; the Kafka envelope carries only `RawArchiveRef` metadata.
|
|
|
|
Do not mark any of these as verified unless the matching command was run and the output was inspected.
|
|
|
|
## Runtime Location Policy
|
|
|
|
Production services run on ECS through Portainer/Docker.
|
|
|
|
- Do not create local service runners for GB32960, JT808, Yutong MQTT, history, or analytics.
|
|
- Do not run the service jars directly on developer machines for production verification.
|
|
- Keep local work limited to source edits, builds, and automated tests.
|
|
|
|
For Portainer deployments, keep history without an archive volume: history consumes raw/event Kafka topics and writes TDengine history rows.
|
|
|
|
## Rollback Guidance
|
|
|
|
If the split deployment is unhealthy:
|
|
|
|
1. Stop `gb32960-ingest-app` first so new GB32960 ACKs are not issued against an unhealthy Kafka boundary.
|
|
2. Keep Kafka topics intact for replay unless storage or privacy policy requires deletion.
|
|
3. Restart or roll back `vehicle-history-app` and `vehicle-analytics-app` independently; their failures do not require protocol ACK rollback because they consume from Kafka offsets.
|
|
4. If Kafka itself is unavailable, stop `gb32960-ingest-app` or route GB32960 traffic to a previously verified release that preserves the Kafka durability boundary.
|
|
5. After rollback, compare Kafka consumer group lag and DLQ contents before resuming the split services.
|
|
|
|
Rollback commands depend on the deployment environment. For the default production stack, roll back the Portainer stack image version and let Docker restart the affected containers.
|
|
|
|
## Task 8 Verification Notes
|
|
|
|
Observed on 2026-06-23 in worktree `.worktrees/gb32960-service-split`:
|
|
|
|
- Package build should use the current active-app command shown in the Build section.
|
|
- Repository-local Kafka setup inspection found no Kafka script, no Docker Compose file, and no compose YAML within the searched repository paths.
|
|
- `nc -z -w 2 127.0.0.1 9092` exited `1`, so no local Kafka broker was reachable at `127.0.0.1:9092`.
|
|
- `kafka-topics`, `kafka-topics.sh`, `docker`, and `docker-compose` were not found on PATH.
|
|
- Kafka topic creation, service startup, health checks, Kafka record checks, archive checks, TDengine checks, stat output checks, and ACK observation were not run because the local Kafka prerequisite was absent.
|
|
|
|
## Latest Build Verification
|
|
|
|
Observed on 2026-06-23 in worktree `.worktrees/gb32960-service-split`:
|
|
|
|
- Targeted module tests: `mvn -pl :sink-kafka,:ingest-core,:protocol-gb32960,:event-history-service,:vehicle-state-service,:vehicle-stat-service test` ended with `BUILD SUCCESS`; Maven reported 55 protocol/sink/core tests, 30 event-history tests, 14 vehicle-state tests, and 19 vehicle-stat tests with 0 failures/errors/skips.
|
|
- Split app tests should cover the current active apps listed in the Service Roles section.
|
|
- Package verification should use the active-app package command from the Build section.
|
|
- Split app startup: not run in this verification pass because an ECS/Portainer runtime was outside that build-only check.
|
|
- Kafka raw records: not verified in that build-only check.
|
|
- Kafka event records: not verified in that build-only check.
|
|
- TDengine history rows: not verified; downstream services were not started without Kafka.
|
|
- Analytics output: not verified; downstream services were not started without Kafka.
|
|
- ACK behavior: not verified against a live broker in this pass. Unit tests cover the GB32960 ACK boundary and ordering, but an operator should still run the ACK capture command above on ECS or staging.
|
|
|
|
## Remote Kafka Live Verification
|
|
|
|
Observed on 2026-06-23 with Kafka `114.55.58.251:9092`:
|
|
|
|
- `gb32960-ingest-app` handled TCP `32960` and HTTP `20100`.
|
|
- Ingest app health check `curl -sS http://127.0.0.1:20100/actuator/health` returned `{"status":"UP"}`.
|
|
- Kafka AdminClient confirmed topics `vehicle.raw.gb32960.v1` and `vehicle.event.gb32960.v1` existed, and created `vehicle.dlq.gb32960.v1`.
|
|
- The app accepted the live Hyundai platform connection from `8.134.95.166`, authenticated platform login, and received real GB32960 realtime/report/login/logout frames.
|
|
- Probe consumer with a temporary group read records from `vehicle.event.gb32960.v1`, including VIN keys such as `LB9A32A2XR0LS1575` and `LNXNEGRR3SR319485`.
|
|
- Probe consumer with a temporary group read records from `vehicle.raw.gb32960.v1`, including VIN keys such as `LNXNEGRR0SR319444`, `LNXNEGRR2SR318201`, and `LB9A32A23R0LS1532`.
|
|
- Runtime logs showed successful GB32960 ACK flushes for platform login, vehicle login, and vehicle logout after the Kafka-backed split ingest service was running.
|
|
|
|
This verifies the live path `GB32960 TCP 32960 -> gb32960-ingest-app -> Kafka raw/event topics` against the remote Kafka broker. Downstream `vehicle-history-app` and `vehicle-analytics-app` were not started in this pass.
|
|
|
|
## 2026-06-29 Superseded Local Verification Record
|
|
|
|
Observed on 2026-06-29 in worktree `.worktrees/vehicle-ingest-redesign-phase1`. This record is retained only as historical evidence of parser/storage behavior; do not repeat it as a deployment procedure because production services now run on ECS through Portainer/Docker.
|
|
|
|
- External JT808 traffic on TCP `808` was parsed and written through Kafka into TDengine. Direct TDengine checks showed more than `6000` JT808 `raw_frames` and more than `1900` JT808 `vehicle_locations`; current production history does not maintain a `telemetry_fields` table.
|
|
- A GB32960 synthetic realtime frame sent to TCP `32960` received a binary ACK beginning with `2323`. The same VIN `LTEST202606290001` was queryable from TDengine-backed history APIs with speed, mileage, longitude, and latitude fields.
|
|
- History raw and event Kafka consumption used separate bindings so raw-frame writes remained current while location/field event ingestion continued.
|
|
|
|
The verified behavior was `GB32960/JT808 TCP -> protocol app -> Kafka raw/event topics -> vehicle-history-app -> TDengine raw_frames/location tables -> Swagger/API query`; the deployment method from that historical run is superseded.
|