shishengliang/lingniu-vehicle-ingest
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs: align runbooks with tdengine raw frames

Browse Source
This commit is contained in:
lingniu
2026-07-01 06:11:10 +08:00
parent 56149c884e
commit f91fa21463
3 changed files with 33 additions and 15 deletions
Download Patch File Download Diff File Expand all files Collapse all files

14
docs/operations/gb32960-service-split-runbook.md
View File

@@ -8,17 +8,17 @@ This runbook covers the split GB32960 ingest, history, and analytics runtimes. K
| --- | --- | --- | --- |
| GB32960 ingest | `:gb32960-ingest-app` | Accept GB32960 TCP connections, decode/authenticate frames, produce raw and normalized records to Kafka, and send protocol ACKs after required Kafka production succeeds. | TCP `32960`, HTTP `20100` |
| JT808 ingest | `:jt808-ingest-app` | Accept JT/T 808 TCP connections on the production receive port, parse raw frames, archive frame bytes, and produce raw/event envelopes to Kafka. | TCP `808`, HTTP `20482` |
| Vehicle history | `:vehicle-history-app` | Consume GB32960 raw/event Kafka records, store event-history records and raw archive references, and expose event/history/GB32960 query APIs. Kafka raw envelopes carry archive URI/size metadata, not raw frame bytes. | HTTP `20200` |
| Vehicle history | `:vehicle-history-app` | Consume GB32960/JT808 raw/event Kafka records, store compact history rows plus TDengine `raw_frames` `parsedJson`/`metadataJson`/`rawUri`, and expose TDengine-backed history APIs. | HTTP `20200` |
| Vehicle analytics | `:vehicle-analytics-app` | Consume GB32960 event Kafka records and update analytics outputs such as daily vehicle statistics. Vehicle state can also be enabled from this runtime. | HTTP `20300` |
## Kafka Contract
| Topic | Producer | Consumers | Purpose |
| --- | --- | --- | --- |
| `vehicle.raw.gb32960.v1` | `gb32960-ingest-app` | `vehicle-history-app` | Raw archive reference records. The envelope contains URI/size metadata, not the raw `.bin` payload. |
| `vehicle.raw.gb32960.v1` | `gb32960-ingest-app` | `vehicle-history-app` | Raw frame records. History writes the parsed JSON, metadata JSON, and raw URI to TDengine `raw_frames`. |
| `vehicle.event.gb32960.v1` | `gb32960-ingest-app` | `vehicle-history-app`, `vehicle-analytics-app` | Normalized vehicle events keyed by VIN when available. |
| `vehicle.dlq.gb32960.v1` | Kafka producer/consumer error paths | Operators/replay tooling | Dead-letter records for failed production or consumer processing. |
| `vehicle.raw.jt808.v1` | `jt808-ingest-app` | `vehicle-history-app` | JT808 raw archive reference records. |
| `vehicle.raw.jt808.v1` | `jt808-ingest-app` | `vehicle-history-app` | JT808 raw frame records for TDengine `raw_frames` and raw-frame troubleshooting APIs. |
| `vehicle.event.jt808.v1` | `jt808-ingest-app` | `vehicle-history-app` | JT808 parsed location/session/alarm events keyed by phone or mapped VIN. |
| `vehicle.dlq.jt808.v1` | Kafka producer/consumer error paths | Operators/replay tooling | Dead-letter records for failed JT808 production or consumer processing. |
@@ -121,7 +121,7 @@ java --sun-misc-unsafe-memory-access=allow \
-jar modules/apps/vehicle-history-app/target/vehicle-history-app.jar
```
`SINK_ARCHIVE_PATH` is intentionally omitted in the split-service Kafka verification path. Current Kafka raw envelopes contain `RawArchiveRef` URI/size metadata only; they do not contain raw bytes that `RawArchiveEventSink` could materialize into `.bin` files in the history JVM.
`SINK_ARCHIVE_PATH` is intentionally omitted from the history JVM. The history hot path is Kafka to TDengine: raw envelopes are stored in TDengine `raw_frames` with `parsedJson`, `metadataJson`, and `rawUri`. Ingest services may still write original `.bin` files as cold backup, but history API queries must not depend on a shared local archive mount.
Vehicle analytics:
@@ -203,9 +203,9 @@ Use the templates under `deploy/local/launchctl/` for the local production-verif
- JT808 ingest: `com.lingniu.jt808`, TCP `808`, HTTP `20482`
- History: `com.lingniu.vehicle-history`, HTTP `20200`
The two ingest services and history service must share the same `SINK_ARCHIVE_PATH`. Kafka raw messages contain `archive://...` references, so history can only read raw frames when its archive root points to the same directory or shared volume that ingest used to write the `.bin` files.
The launchctl history template does not inject `SINK_ARCHIVE_PATH`. Use TDengine `raw_frames` as the authoritative hot-query store; keep any ingest-side raw archive path as an optional cold backup for operator inspection.
For Portainer deployments, mirror this same rule with a shared volume mounted to all three services, for example `/archive` in `deploy/portainer/docker-compose.yml`.
For Portainer deployments, mirror this same shape: no history archive volume, history consumes raw/event Kafka topics and writes TDengine history rows.
## Rollback Guidance
@@ -268,4 +268,4 @@ Observed on 2026-06-29 in worktree `.worktrees/vehicle-ingest-redesign-phase1`:
- A GB32960 synthetic realtime frame sent to TCP `32960` received a binary ACK beginning with `2323`. The same VIN `LTEST202606290001` was queryable from TDengine-backed history APIs with speed, mileage, longitude, and latitude fields.
- History raw and event Kafka consumption used separate bindings so raw-frame writes remained current while location/field event ingestion continued.
This verifies the local path `GB32960/JT808 TCP -> protocol app -> shared raw archive -> Kafka raw/event topics -> vehicle-history-app -> TDengine -> Swagger/API query`.
This verifies the local path `GB32960/JT808 TCP -> protocol app -> Kafka raw/event topics -> vehicle-history-app -> TDengine raw_frames/location tables -> Swagger/API query`.

16
docs/operations/vehicle-ingest-tdengine-verification.md
View File

@@ -30,7 +30,7 @@ export TDENGINE_TELEMETRY_FIELDS_ENABLED=false
`TDENGINE_MIN_IDLE=0` 用于减少冷启动时 TDengine 连接重试日志。TDengine 地址稳定后，再按生产并发调大连接池。
高吞吐生产验收默认关闭 `EVENT_FILE_STORE_ENABLED` 和 `TDENGINE_TELEMETRY_FIELDS_ENABLED`，验证 `raw_frames`、位置表、raw archive 文件和分页查询闭环。GB32960 snapshot/fields 接口会通过 TDengine `raw_frames` 索引找到 `rawUri`，再回读共享 archive 的原始 `.bin` 即时解析；需要兼容旧 `/api/event-history/records` 低层记录接口或逐字段趋势宽表时，再显式开启对应开关，并单独压测写入放大。
高吞吐生产验收默认关闭 `EVENT_FILE_STORE_ENABLED` 和 `TDENGINE_TELEMETRY_FIELDS_ENABLED`，验证 TDengine `raw_frames`、位置表和分页查询闭环。RAW 帧的 `parsedJson`、`metadataJson`、`rawUri` 直接进入 TDengine，GB32960 snapshot/fields 接口优先基于 `raw_frames` 中的结构化 JSON 返回结果；接入服务写出的原始 `.bin` 只作为冷备复核材料。需要兼容旧 `/api/event-history/records` 低层记录接口或逐字段趋势宽表时，再显式开启对应开关，并单独压测写入放大。
## 构建
@@ -86,7 +86,6 @@ KAFKA_CONSUMER_ENABLED=true \
EVENT_FILE_STORE_ENABLED=false \
EVENT_FILE_STORE_PATH=./target/tdengine-verification/event-store \
TDENGINE_TELEMETRY_FIELDS_ENABLED=false \
SINK_ARCHIVE_PATH=./target/tdengine-verification/archive \
java -jar modules/apps/vehicle-history-app/target/vehicle-history-app.jar
```
@@ -113,7 +112,8 @@ deploy/local/launchctl/
- `jt808-ingest-app` 监听 TCP `808`，HTTP `20482`。
- `gb32960-ingest-app` 监听 TCP `32960`，HTTP `20100`。
- `vehicle-history-app` 监听 HTTP `20200`。
- 三个服务必须共用同一个 `SINK_ARCHIVE_PATH`，否则 TDengine 中的 `raw_uri` 能入库，但 history API 无法按 `archive://...` 读取原始帧。
- history 热查询只依赖 Kafka 和 TDengine `raw_frames`；本地模板不会给 history 注入 `SINK_ARCHIVE_PATH`。
- GB32960/JT808 接入服务可以继续使用 `SINK_ARCHIVE_PATH` 保存原始 `.bin` 冷备，但这不是 history API 的实时查询前置条件。
- 替换 jar 前先 `launchctl bootout`，再复制 jar 和 `launchctl bootstrap`，避免 KeepAlive 在 jar 拷贝中途重启。
JT808 注册身份绑定：
@@ -176,7 +176,7 @@ curl -sS 'http://127.0.0.1:20200/api/event-history/jt808/locations?phone=<phone>
## GB32960 Snapshot / 字段投影验收
默认高吞吐模式下，GB32960 全字段查询不依赖 telemetry_fields 宽表。history 先查 TDengine `raw_frames`，再按 `rawUri` 读取共享 archive 中的原始 `.bin` 并即时解析：
默认高吞吐模式下，GB32960 全字段查询不依赖 telemetry_fields 宽表。history 先查 TDengine `raw_frames`，直接使用 RAW 行中的 `parsedJson`/`parsedFields` 生成 snapshot 和字段投影：
```bash
curl -sS 'http://127.0.0.1:20200/api/event-history/gb32960/snapshots?vin=<vin>&dateFrom=2026-06-29T00:00:00%2B08:00&dateTo=2026-06-29T23:59:59%2B08:00&limit=10'
@@ -184,7 +184,7 @@ curl -sS 'http://127.0.0.1:20200/api/event-history/gb32960/snapshots?vin=<vin>&d
curl -sS 'http://127.0.0.1:20200/api/event-history/gb32960/snapshots/fields?vin=<vin>&fields=VEHICLE.speedKmh,VEHICLE.totalMileageKm,POSITION_V2016.longitude,POSITION_V2016.latitude&dateFrom=2026-06-29T00:00:00%2B08:00&dateTo=2026-06-29T23:59:59%2B08:00&limit=10'
```
预期：snapshot 返回 `sourceFrames.rawArchiveUri` 和解析后的 `blocks`；字段投影返回所选字段。新增协议字段后，只要解析器和字典补齐，就可以基于历史 raw 重新查询。
预期：snapshot 返回 `sourceFrames.rawArchiveUri` 和解析后的 `blocks`；字段投影返回所选字段。新增协议字段后，先通过 replay 或专用解析任务补写 `raw_frames.parsedJson`，再基于历史 RAW 结构化结果查询。
## Telemetry Field 查询验收
@@ -321,7 +321,7 @@ curl -sS 'http://127.0.0.1:20200/api/event-history/jt808/raw-frames?phone=130799
GB32960 链路已用正式平台登录和合成实时帧验证：
- 正式平台 `115.29.187.205` 已连接 TCP `32960`，服务收到 `0x05 PLATFORM_LOGIN` 并返回 ACK；日志显示账号 `Hyundai`、策略 `ALLOW`。
- 2026-06-29 正式平台登录 raw 已有 `3` 条，均落入 TDengine `raw_frames` 和共享 archive，可通过 `/api/event-history/gb32960/frame?rawArchiveUri=...` 回读解析为 `PLATFORM_LOGIN`。
- 2026-06-29 正式平台登录 raw 已有 `3` 条，均落入 TDengine `raw_frames`，可通过 RAW 查询看到 `parsedJson` 中的 `PLATFORM_LOGIN` 解析结果。
- 当前正式平台尚未发送车辆实时上报 `0x02`；非测试 VIN 的 `message_id=2` 计数为 `0`，因此不能把平台登录误判为车辆数据在线。
- TCP `32960` 返回 GB32960 `2323` ACK。
- `raw_frames` 中测试 VIN `LTEST202606290001` 已有 `2` 条 raw 记录。
@@ -352,7 +352,7 @@ SELECT COUNT(*) FROM raw_frames
WHERE protocol = 'GB32960' AND message_id = 2 AND vin <> '' AND vin NOT LIKE 'LTEST%';
```
也可以直接运行仓库 smoke 工具验证 TCP `32960`、GB32960 snapshot/fields、TDengine `raw_frames` 和 raw archive：
也可以直接运行仓库 smoke 工具验证 TCP `32960`、GB32960 snapshot/fields、TDengine `raw_frames`。如果需要复核冷备 `.bin`，再传入接入服务 archive 根目录：
```bash
export TDENGINE_REST_URL='http://<tdengine-host>:6041/rest/sql/vehicle_ts'
@@ -366,7 +366,7 @@ python3 tools/gb32960_e2e_smoke.py \
--tdengine-password "$TDENGINE_PASSWORD"
```
预期输出包含 `records >= 1`、`fieldCounts` 中关键字段均大于 0、`tdengineRawFrames` 等于本次可见唯一 `rawUri` 数量、`archiveChecked >= 1`。这里的 `records` 字段表示 snapshot 数量，用于兼容早期脚本输出名。
预期输出包含 `records >= 1`、`fieldCounts` 中关键字段均大于 0、`tdengineRawFrames` 等于本次可见唯一 `rawUri` 数量。传入 `--archive-root` 时，`archiveChecked >= 1` 说明冷备文件也能复核；这里的 `records` 字段表示 snapshot 数量，用于兼容早期脚本输出名。
复查 API：

18
modules/apps/vehicle-history-app/src/test/java/com/lingniu/ingest/historyapp/PortainerComposeResourceLimitsTest.java
View File

@@ -75,6 +75,24 @@ class PortainerComposeResourceLimitsTest {
.doesNotContain("history 服务按同一个 archive root 读取原始帧");
}
@Test
void operationRunbooksDescribeTdengineRawJsonHotPath() throws IOException {
Path operations = repositoryRoot().resolve("docs/operations");
String tdengineVerification = Files.readString(operations.resolve("vehicle-ingest-tdengine-verification.md"));
String splitRunbook = Files.readString(operations.resolve("gb32960-service-split-runbook.md"));
assertThat(tdengineVerification)
.contains("TDengine `raw_frames`")
.doesNotContain("三个服务必须共用同一个 `SINK_ARCHIVE_PATH`")
.doesNotContain("再回读共享 archive")
.doesNotContain("读取共享 archive 中的原始 `.bin`");
assertThat(splitRunbook)
.contains("TDengine `raw_frames`")
.doesNotContain("must share the same `SINK_ARCHIVE_PATH`")
.doesNotContain("history can only read raw frames")
.doesNotContain("shared volume mounted to all three services");
}
@Test
void vehicleAnalyticsComposeOnlyExposesStatConsumerEnvironment() throws IOException {
String compose = Files.readString(repositoryRoot().resolve("deploy/portainer/docker-compose.yml"));