shishengliang/lingniu-vehicle-ingest
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs: use tdengine checks in split runbook

Browse Source
This commit is contained in:
lingniu
2026-07-01 06:25:20 +08:00
parent 6bc36eb5a8
commit 3d4676adf3
2 changed files with 38 additions and 8 deletions
Download Patch File Download Diff File Expand all files Collapse all files

31
docs/operations/gb32960-service-split-runbook.md
View File

@@ -115,7 +115,12 @@ Vehicle history:
```bash ```bash
KAFKA_BROKERS=127.0.0.1:9092 \ KAFKA_BROKERS=127.0.0.1:9092 \
HTTP_PORT=20200 \ HTTP_PORT=20200 \
EVENT_FILE_STORE_PATH=./target/split-event-store \ TDENGINE_HISTORY_ENABLED=true \
TDENGINE_HISTORY_DATABASE=vehicle_ts \
TDENGINE_JDBC_URL='jdbc:TAOS-WS://<tdengine-host>:6041/vehicle_ts' \
TDENGINE_USERNAME=root \
TDENGINE_PASSWORD='<tdengine-password>' \
TDENGINE_TELEMETRY_FIELDS_ENABLED=false \
java --sun-misc-unsafe-memory-access=allow \ java --sun-misc-unsafe-memory-access=allow \
-jar modules/apps/vehicle-history-app/target/vehicle-history-app.jar -jar modules/apps/vehicle-history-app/target/vehicle-history-app.jar
``` ```
@@ -170,14 +175,23 @@ Verify only what was actually run in your environment:
```bash ```bash
kafka-console-consumer --bootstrap-server 127.0.0.1:9092 --topic vehicle.raw.gb32960.v1 --from-beginning --max-messages 1 --timeout-ms 10000 kafka-console-consumer --bootstrap-server 127.0.0.1:9092 --topic vehicle.raw.gb32960.v1 --from-beginning --max-messages 1 --timeout-ms 10000
kafka-console-consumer --bootstrap-server 127.0.0.1:9092 --topic vehicle.event.gb32960.v1 --from-beginning --max-messages 1 --timeout-ms 10000 kafka-console-consumer --bootstrap-server 127.0.0.1:9092 --topic vehicle.event.gb32960.v1 --from-beginning --max-messages 1 --timeout-ms 10000
find target/split-event-store -type f | head ```
find target/split-vehicle-stat -type f | head
Use TDengine CLI or a JDBC client to verify history writes:
```sql
USE vehicle_ts;
SELECT COUNT(*) FROM raw_frames WHERE protocol = 'GB32960';
SELECT COUNT(*) FROM vehicle_locations WHERE protocol = 'GB32960';
SELECT COUNT(*) FROM raw_frames WHERE protocol = 'JT808';
SELECT COUNT(*) FROM jt808_locations WHERE protocol = 'JT808';
``` ```
Useful HTTP query checks after events are consumed. For `realtime_001.hex`, the fixture VIN is `LTEST000000000001`; replace `<date-from>`, `<date-to>`, and `<stat-date>` with values that cover the consumed record's event time: Useful HTTP query checks after events are consumed. For `realtime_001.hex`, the fixture VIN is `LTEST000000000001`; replace `<date-from>`, `<date-to>`, and `<stat-date>` with values that cover the consumed record's event time:
```bash ```bash
curl -sS 'http://127.0.0.1:20200/api/event-history/records?protocol=GB32960&dateFrom=<date-from>&dateTo=<date-to>&vin=LTEST000000000001&limit=10' curl -sS 'http://127.0.0.1:20200/api/event-history/locations?protocol=GB32960&dateFrom=<date-from>&dateTo=<date-to>&vin=LTEST000000000001&limit=10'
curl -sS 'http://127.0.0.1:20200/api/event-history/raw-frames?protocol=GB32960&dateFrom=<date-from>&dateTo=<date-to>&vin=LTEST000000000001&limit=10'
curl -sS 'http://127.0.0.1:20200/api/event-history/gb32960/dictionary' curl -sS 'http://127.0.0.1:20200/api/event-history/gb32960/dictionary'
curl -sS 'http://127.0.0.1:20300/api/vehicle-stat/LTEST000000000001/daily?date=<stat-date>' curl -sS 'http://127.0.0.1:20300/api/vehicle-stat/LTEST000000000001/daily?date=<stat-date>'
``` ```
@@ -185,9 +199,10 @@ curl -sS 'http://127.0.0.1:20300/api/vehicle-stat/LTEST000000000001/daily?date=<
Expected E2E result when Kafka and all services are running: Expected E2E result when Kafka and all services are running:
- A GB32960 client receives a binary success ACK only after required Kafka production succeeds, and the captured ACK bytes are inspected. - A GB32960 client receives a binary success ACK only after required Kafka production succeeds, and the captured ACK bytes are inspected.
- `vehicle.raw.gb32960.v1` receives a raw archive reference envelope. - `vehicle.raw.gb32960.v1` receives a raw frame envelope.
- `vehicle.event.gb32960.v1` receives one or more normalized records. - `vehicle.event.gb32960.v1` receives one or more normalized records.
- `target/split-event-store` receives event-store files after the history service flushes consumed Kafka records, including raw archive reference records. - TDengine `raw_frames` receives GB32960/JT808 RAW rows with parsed JSON and metadata.
- TDengine location tables receive compact GB32960/JT808 location rows.
- JT808 analytics writes daily mileage rows to MySQL `vehicle_stat_metric` after consuming applicable `vehicle.event.jt808.v1` records. - JT808 analytics writes daily mileage rows to MySQL `vehicle_stat_metric` after consuming applicable `vehicle.event.jt808.v1` records.
Do not expect `vehicle-history-app` to create raw `.bin` archive files from Kafka raw records in the current implementation. `RawArchiveEventSink` can write archive files only when it receives `VehicleEvent.RawArchive.rawBytes()` inside the same JVM; the Kafka envelope carries only `RawArchiveRef` metadata. Do not expect `vehicle-history-app` to create raw `.bin` archive files from Kafka raw records in the current implementation. `RawArchiveEventSink` can write archive files only when it receives `VehicleEvent.RawArchive.rawBytes()` inside the same JVM; the Kafka envelope carries only `RawArchiveRef` metadata.
@@ -226,7 +241,7 @@ Observed on 2026-06-23 in worktree `.worktrees/gb32960-service-split`:
- Repository-local Kafka setup inspection found no Kafka script, no Docker Compose file, and no compose YAML within the searched repository paths. - Repository-local Kafka setup inspection found no Kafka script, no Docker Compose file, and no compose YAML within the searched repository paths.
- `nc -z -w 2 127.0.0.1 9092` exited `1`, so no local Kafka broker was reachable at `127.0.0.1:9092`. - `nc -z -w 2 127.0.0.1 9092` exited `1`, so no local Kafka broker was reachable at `127.0.0.1:9092`.
- `kafka-topics`, `kafka-topics.sh`, `docker`, and `docker-compose` were not found on PATH. - `kafka-topics`, `kafka-topics.sh`, `docker`, and `docker-compose` were not found on PATH.
- Kafka topic creation, service startup, health checks, Kafka record checks, archive checks, event-store checks, stat output checks, and ACK observation were not run because the local Kafka prerequisite was absent. - Kafka topic creation, service startup, health checks, Kafka record checks, archive checks, TDengine checks, stat output checks, and ACK observation were not run because the local Kafka prerequisite was absent.
## Latest Local Verification ## Latest Local Verification
@@ -238,7 +253,7 @@ Observed on 2026-06-23 in worktree `.worktrees/gb32960-service-split`:
- Split app startup: not run in this verification pass because local Kafka was still absent. - Split app startup: not run in this verification pass because local Kafka was still absent.
- Kafka raw records: not verified; no local Kafka broker or Kafka CLI tools were available. - Kafka raw records: not verified; no local Kafka broker or Kafka CLI tools were available.
- Kafka event records: not verified; no local Kafka broker or Kafka CLI tools were available. - Kafka event records: not verified; no local Kafka broker or Kafka CLI tools were available.
- Event-store files: not verified; downstream services were not started without Kafka. - TDengine history rows: not verified; downstream services were not started without Kafka.
- Analytics output: not verified; downstream services were not started without Kafka. - Analytics output: not verified; downstream services were not started without Kafka.
- ACK behavior: not verified against a live broker in this pass. Unit tests cover the GB32960 ACK boundary and ordering, but an operator should still run the ACK capture command above in a Kafka-backed local or staging environment. - ACK behavior: not verified against a live broker in this pass. Unit tests cover the GB32960 ACK boundary and ordering, but an operator should still run the ACK capture command above in a Kafka-backed local or staging environment.

15
modules/apps/vehicle-history-app/src/test/java/com/lingniu/ingest/historyapp/PortainerComposeResourceLimitsTest.java
View File

@@ -107,6 +107,21 @@ class PortainerComposeResourceLimitsTest {
.doesNotContain("Vehicle state can also be enabled from this runtime"); .doesNotContain("Vehicle state can also be enabled from this runtime");
} }
@Test
void splitRunbookUsesTdengineHistoryVerificationInsteadOfEventFileStore() throws IOException {
String splitRunbook = Files.readString(repositoryRoot()
.resolve("docs/operations/gb32960-service-split-runbook.md"));
assertThat(splitRunbook)
.contains("TDENGINE_HISTORY_ENABLED=true")
.contains("SELECT COUNT(*) FROM raw_frames")
.contains("SELECT COUNT(*) FROM vehicle_locations")
.doesNotContain("EVENT_FILE_STORE_PATH")
.doesNotContain("target/split-event-store")
.doesNotContain("event-store files")
.doesNotContain("/api/event-history/records?");
}
@Test @Test
void vehicleAnalyticsComposeOnlyExposesStatConsumerEnvironment() throws IOException { void vehicleAnalyticsComposeOnlyExposesStatConsumerEnvironment() throws IOException {
String compose = Files.readString(repositoryRoot().resolve("deploy/portainer/docker-compose.yml")); String compose = Files.readString(repositoryRoot().resolve("deploy/portainer/docker-compose.yml"));