diff --git a/docs/operations/gb32960-service-split-runbook.md b/docs/operations/gb32960-service-split-runbook.md index 89d3b7c0..b71ac8c2 100644 --- a/docs/operations/gb32960-service-split-runbook.md +++ b/docs/operations/gb32960-service-split-runbook.md @@ -8,17 +8,17 @@ This runbook covers the split GB32960 ingest, history, and analytics runtimes. K | --- | --- | --- | --- | | GB32960 ingest | `:gb32960-ingest-app` | Accept GB32960 TCP connections, decode/authenticate frames, produce raw and normalized records to Kafka, and send protocol ACKs after required Kafka production succeeds. | TCP `32960`, HTTP `20100` | | JT808 ingest | `:jt808-ingest-app` | Accept JT/T 808 TCP connections on the production receive port, parse raw frames, archive frame bytes, and produce raw/event envelopes to Kafka. | TCP `808`, HTTP `20482` | -| Vehicle history | `:vehicle-history-app` | Consume GB32960 raw/event Kafka records, store event-history records and raw archive references, and expose event/history/GB32960 query APIs. Kafka raw envelopes carry archive URI/size metadata, not raw frame bytes. | HTTP `20200` | +| Vehicle history | `:vehicle-history-app` | Consume GB32960/JT808 raw/event Kafka records, store compact history rows plus TDengine `raw_frames` `parsedJson`/`metadataJson`/`rawUri`, and expose TDengine-backed history APIs. | HTTP `20200` | | Vehicle analytics | `:vehicle-analytics-app` | Consume GB32960 event Kafka records and update analytics outputs such as daily vehicle statistics. Vehicle state can also be enabled from this runtime. | HTTP `20300` | ## Kafka Contract | Topic | Producer | Consumers | Purpose | | --- | --- | --- | --- | -| `vehicle.raw.gb32960.v1` | `gb32960-ingest-app` | `vehicle-history-app` | Raw archive reference records. The envelope contains URI/size metadata, not the raw `.bin` payload. | +| `vehicle.raw.gb32960.v1` | `gb32960-ingest-app` | `vehicle-history-app` | Raw frame records. History writes the parsed JSON, metadata JSON, and raw URI to TDengine `raw_frames`. | | `vehicle.event.gb32960.v1` | `gb32960-ingest-app` | `vehicle-history-app`, `vehicle-analytics-app` | Normalized vehicle events keyed by VIN when available. | | `vehicle.dlq.gb32960.v1` | Kafka producer/consumer error paths | Operators/replay tooling | Dead-letter records for failed production or consumer processing. | -| `vehicle.raw.jt808.v1` | `jt808-ingest-app` | `vehicle-history-app` | JT808 raw archive reference records. | +| `vehicle.raw.jt808.v1` | `jt808-ingest-app` | `vehicle-history-app` | JT808 raw frame records for TDengine `raw_frames` and raw-frame troubleshooting APIs. | | `vehicle.event.jt808.v1` | `jt808-ingest-app` | `vehicle-history-app` | JT808 parsed location/session/alarm events keyed by phone or mapped VIN. | | `vehicle.dlq.jt808.v1` | Kafka producer/consumer error paths | Operators/replay tooling | Dead-letter records for failed JT808 production or consumer processing. | @@ -121,7 +121,7 @@ java --sun-misc-unsafe-memory-access=allow \ -jar modules/apps/vehicle-history-app/target/vehicle-history-app.jar ``` -`SINK_ARCHIVE_PATH` is intentionally omitted in the split-service Kafka verification path. Current Kafka raw envelopes contain `RawArchiveRef` URI/size metadata only; they do not contain raw bytes that `RawArchiveEventSink` could materialize into `.bin` files in the history JVM. +`SINK_ARCHIVE_PATH` is intentionally omitted from the history JVM. The history hot path is Kafka to TDengine: raw envelopes are stored in TDengine `raw_frames` with `parsedJson`, `metadataJson`, and `rawUri`. Ingest services may still write original `.bin` files as cold backup, but history API queries must not depend on a shared local archive mount. Vehicle analytics: @@ -203,9 +203,9 @@ Use the templates under `deploy/local/launchctl/` for the local production-verif - JT808 ingest: `com.lingniu.jt808`, TCP `808`, HTTP `20482` - History: `com.lingniu.vehicle-history`, HTTP `20200` -The two ingest services and history service must share the same `SINK_ARCHIVE_PATH`. Kafka raw messages contain `archive://...` references, so history can only read raw frames when its archive root points to the same directory or shared volume that ingest used to write the `.bin` files. +The launchctl history template does not inject `SINK_ARCHIVE_PATH`. Use TDengine `raw_frames` as the authoritative hot-query store; keep any ingest-side raw archive path as an optional cold backup for operator inspection. -For Portainer deployments, mirror this same rule with a shared volume mounted to all three services, for example `/archive` in `deploy/portainer/docker-compose.yml`. +For Portainer deployments, mirror this same shape: no history archive volume, history consumes raw/event Kafka topics and writes TDengine history rows. ## Rollback Guidance @@ -268,4 +268,4 @@ Observed on 2026-06-29 in worktree `.worktrees/vehicle-ingest-redesign-phase1`: - A GB32960 synthetic realtime frame sent to TCP `32960` received a binary ACK beginning with `2323`. The same VIN `LTEST202606290001` was queryable from TDengine-backed history APIs with speed, mileage, longitude, and latitude fields. - History raw and event Kafka consumption used separate bindings so raw-frame writes remained current while location/field event ingestion continued. -This verifies the local path `GB32960/JT808 TCP -> protocol app -> shared raw archive -> Kafka raw/event topics -> vehicle-history-app -> TDengine -> Swagger/API query`. +This verifies the local path `GB32960/JT808 TCP -> protocol app -> Kafka raw/event topics -> vehicle-history-app -> TDengine raw_frames/location tables -> Swagger/API query`. diff --git a/docs/operations/vehicle-ingest-tdengine-verification.md b/docs/operations/vehicle-ingest-tdengine-verification.md index 73369a39..0a7485b3 100644 --- a/docs/operations/vehicle-ingest-tdengine-verification.md +++ b/docs/operations/vehicle-ingest-tdengine-verification.md @@ -30,7 +30,7 @@ export TDENGINE_TELEMETRY_FIELDS_ENABLED=false `TDENGINE_MIN_IDLE=0` 用于减少冷启动时 TDengine 连接重试日志。TDengine 地址稳定后，再按生产并发调大连接池。 -高吞吐生产验收默认关闭 `EVENT_FILE_STORE_ENABLED` 和 `TDENGINE_TELEMETRY_FIELDS_ENABLED`，验证 `raw_frames`、位置表、raw archive 文件和分页查询闭环。GB32960 snapshot/fields 接口会通过 TDengine `raw_frames` 索引找到 `rawUri`，再回读共享 archive 的原始 `.bin` 即时解析；需要兼容旧 `/api/event-history/records` 低层记录接口或逐字段趋势宽表时，再显式开启对应开关，并单独压测写入放大。 +高吞吐生产验收默认关闭 `EVENT_FILE_STORE_ENABLED` 和 `TDENGINE_TELEMETRY_FIELDS_ENABLED`，验证 TDengine `raw_frames`、位置表和分页查询闭环。RAW 帧的 `parsedJson`、`metadataJson`、`rawUri` 直接进入 TDengine，GB32960 snapshot/fields 接口优先基于 `raw_frames` 中的结构化 JSON 返回结果；接入服务写出的原始 `.bin` 只作为冷备复核材料。需要兼容旧 `/api/event-history/records` 低层记录接口或逐字段趋势宽表时，再显式开启对应开关，并单独压测写入放大。 ## 构建 @@ -86,7 +86,6 @@ KAFKA_CONSUMER_ENABLED=true \ EVENT_FILE_STORE_ENABLED=false \ EVENT_FILE_STORE_PATH=./target/tdengine-verification/event-store \ TDENGINE_TELEMETRY_FIELDS_ENABLED=false \ -SINK_ARCHIVE_PATH=./target/tdengine-verification/archive \ java -jar modules/apps/vehicle-history-app/target/vehicle-history-app.jar ``` @@ -113,7 +112,8 @@ deploy/local/launchctl/ - `jt808-ingest-app` 监听 TCP `808`，HTTP `20482`。 - `gb32960-ingest-app` 监听 TCP `32960`，HTTP `20100`。 - `vehicle-history-app` 监听 HTTP `20200`。 -- 三个服务必须共用同一个 `SINK_ARCHIVE_PATH`，否则 TDengine 中的 `raw_uri` 能入库，但 history API 无法按 `archive://...` 读取原始帧。 +- history 热查询只依赖 Kafka 和 TDengine `raw_frames`；本地模板不会给 history 注入 `SINK_ARCHIVE_PATH`。 +- GB32960/JT808 接入服务可以继续使用 `SINK_ARCHIVE_PATH` 保存原始 `.bin` 冷备，但这不是 history API 的实时查询前置条件。 - 替换 jar 前先 `launchctl bootout`，再复制 jar 和 `launchctl bootstrap`，避免 KeepAlive 在 jar 拷贝中途重启。 JT808 注册身份绑定： @@ -176,7 +176,7 @@ curl -sS 'http://127.0.0.1:20200/api/event-history/jt808/locations?phone= ## GB32960 Snapshot / 字段投影验收 -默认高吞吐模式下，GB32960 全字段查询不依赖 telemetry_fields 宽表。history 先查 TDengine `raw_frames`，再按 `rawUri` 读取共享 archive 中的原始 `.bin` 并即时解析： +默认高吞吐模式下，GB32960 全字段查询不依赖 telemetry_fields 宽表。history 先查 TDengine `raw_frames`，直接使用 RAW 行中的 `parsedJson`/`parsedFields` 生成 snapshot 和字段投影： ```bash curl -sS 'http://127.0.0.1:20200/api/event-history/gb32960/snapshots?vin=&dateFrom=2026-06-29T00:00:00%2B08:00&dateTo=2026-06-29T23:59:59%2B08:00&limit=10' @@ -184,7 +184,7 @@ curl -sS 'http://127.0.0.1:20200/api/event-history/gb32960/snapshots?vin=&d curl -sS 'http://127.0.0.1:20200/api/event-history/gb32960/snapshots/fields?vin=&fields=VEHICLE.speedKmh,VEHICLE.totalMileageKm,POSITION_V2016.longitude,POSITION_V2016.latitude&dateFrom=2026-06-29T00:00:00%2B08:00&dateTo=2026-06-29T23:59:59%2B08:00&limit=10' ``` -预期：snapshot 返回 `sourceFrames.rawArchiveUri` 和解析后的 `blocks`；字段投影返回所选字段。新增协议字段后，只要解析器和字典补齐，就可以基于历史 raw 重新查询。 +预期：snapshot 返回 `sourceFrames.rawArchiveUri` 和解析后的 `blocks`；字段投影返回所选字段。新增协议字段后，先通过 replay 或专用解析任务补写 `raw_frames.parsedJson`，再基于历史 RAW 结构化结果查询。 ## Telemetry Field 查询验收 @@ -321,7 +321,7 @@ curl -sS 'http://127.0.0.1:20200/api/event-history/jt808/raw-frames?phone=130799 GB32960 链路已用正式平台登录和合成实时帧验证： - 正式平台 `115.29.187.205` 已连接 TCP `32960`，服务收到 `0x05 PLATFORM_LOGIN` 并返回 ACK；日志显示账号 `Hyundai`、策略 `ALLOW`。 -- 2026-06-29 正式平台登录 raw 已有 `3` 条，均落入 TDengine `raw_frames` 和共享 archive，可通过 `/api/event-history/gb32960/frame?rawArchiveUri=...` 回读解析为 `PLATFORM_LOGIN`。 +- 2026-06-29 正式平台登录 raw 已有 `3` 条，均落入 TDengine `raw_frames`，可通过 RAW 查询看到 `parsedJson` 中的 `PLATFORM_LOGIN` 解析结果。 - 当前正式平台尚未发送车辆实时上报 `0x02`；非测试 VIN 的 `message_id=2` 计数为 `0`，因此不能把平台登录误判为车辆数据在线。 - TCP `32960` 返回 GB32960 `2323` ACK。 - `raw_frames` 中测试 VIN `LTEST202606290001` 已有 `2` 条 raw 记录。 @@ -352,7 +352,7 @@ SELECT COUNT(*) FROM raw_frames WHERE protocol = 'GB32960' AND message_id = 2 AND vin <> '' AND vin NOT LIKE 'LTEST%'; ``` -也可以直接运行仓库 smoke 工具验证 TCP `32960`、GB32960 snapshot/fields、TDengine `raw_frames` 和 raw archive： +也可以直接运行仓库 smoke 工具验证 TCP `32960`、GB32960 snapshot/fields、TDengine `raw_frames`。如果需要复核冷备 `.bin`，再传入接入服务 archive 根目录： ```bash export TDENGINE_REST_URL='http://:6041/rest/sql/vehicle_ts' @@ -366,7 +366,7 @@ python3 tools/gb32960_e2e_smoke.py \ --tdengine-password "$TDENGINE_PASSWORD" ``` -预期输出包含 `records >= 1`、`fieldCounts` 中关键字段均大于 0、`tdengineRawFrames` 等于本次可见唯一 `rawUri` 数量、`archiveChecked >= 1`。这里的 `records` 字段表示 snapshot 数量，用于兼容早期脚本输出名。 +预期输出包含 `records >= 1`、`fieldCounts` 中关键字段均大于 0、`tdengineRawFrames` 等于本次可见唯一 `rawUri` 数量。传入 `--archive-root` 时，`archiveChecked >= 1` 说明冷备文件也能复核；这里的 `records` 字段表示 snapshot 数量，用于兼容早期脚本输出名。 复查 API： diff --git a/modules/apps/vehicle-history-app/src/test/java/com/lingniu/ingest/historyapp/PortainerComposeResourceLimitsTest.java b/modules/apps/vehicle-history-app/src/test/java/com/lingniu/ingest/historyapp/PortainerComposeResourceLimitsTest.java index 52ddb5b5..de50985c 100644 --- a/modules/apps/vehicle-history-app/src/test/java/com/lingniu/ingest/historyapp/PortainerComposeResourceLimitsTest.java +++ b/modules/apps/vehicle-history-app/src/test/java/com/lingniu/ingest/historyapp/PortainerComposeResourceLimitsTest.java @@ -75,6 +75,24 @@ class PortainerComposeResourceLimitsTest { .doesNotContain("history 服务按同一个 archive root 读取原始帧"); } + @Test + void operationRunbooksDescribeTdengineRawJsonHotPath() throws IOException { + Path operations = repositoryRoot().resolve("docs/operations"); + String tdengineVerification = Files.readString(operations.resolve("vehicle-ingest-tdengine-verification.md")); + String splitRunbook = Files.readString(operations.resolve("gb32960-service-split-runbook.md")); + + assertThat(tdengineVerification) + .contains("TDengine `raw_frames`") + .doesNotContain("三个服务必须共用同一个 `SINK_ARCHIVE_PATH`") + .doesNotContain("再回读共享 archive") + .doesNotContain("读取共享 archive 中的原始 `.bin`"); + assertThat(splitRunbook) + .contains("TDengine `raw_frames`") + .doesNotContain("must share the same `SINK_ARCHIVE_PATH`") + .doesNotContain("history can only read raw frames") + .doesNotContain("shared volume mounted to all three services"); + } + @Test void vehicleAnalyticsComposeOnlyExposesStatConsumerEnvironment() throws IOException { String compose = Files.readString(repositoryRoot().resolve("deploy/portainer/docker-compose.yml"));