docs: document local ingest deployment verification
All checks were successful
ci/woodpecker/push/woodpecker Pipeline was successful
All checks were successful
ci/woodpecker/push/woodpecker Pipeline was successful
This commit is contained in:
lingniu
@@ -7,6 +7,7 @@ This runbook covers the split GB32960 ingest, history, and analytics runtimes. K
|
||||
| Service | Module | Role | Local ports |
|
||||
| --- | --- | --- | --- |
|
||||
| GB32960 ingest | `:gb32960-ingest-app` | Accept GB32960 TCP connections, decode/authenticate frames, produce raw and normalized records to Kafka, and send protocol ACKs after required Kafka production succeeds. | TCP `32960`, HTTP `20100` |
|
||||
| JT808 ingest | `:jt808-ingest-app` | Accept JT/T 808 TCP connections on the production receive port, parse raw frames, archive frame bytes, and produce raw/event envelopes to Kafka. | TCP `808`, HTTP `20482` |
|
||||
| Vehicle history | `:vehicle-history-app` | Consume GB32960 raw/event Kafka records, store event-history records and raw archive references, and expose event/history/GB32960 query APIs. Kafka raw envelopes carry archive URI/size metadata, not raw frame bytes. | HTTP `20200` |
|
||||
| Vehicle analytics | `:vehicle-analytics-app` | Consume GB32960 event Kafka records and update analytics outputs such as daily vehicle statistics. Vehicle state can also be enabled from this runtime. | HTTP `20300` |
|
||||
|
||||
@@ -17,6 +18,9 @@ This runbook covers the split GB32960 ingest, history, and analytics runtimes. K
|
||||
| `vehicle.raw.gb32960.v1` | `gb32960-ingest-app` | `vehicle-history-app` | Raw archive reference records. The envelope contains URI/size metadata, not the raw `.bin` payload. |
|
||||
| `vehicle.event.gb32960.v1` | `gb32960-ingest-app` | `vehicle-history-app`, `vehicle-analytics-app` | Normalized vehicle events keyed by VIN when available. |
|
||||
| `vehicle.dlq.gb32960.v1` | Kafka producer/consumer error paths | Operators/replay tooling | Dead-letter records for failed production or consumer processing. |
|
||||
| `vehicle.raw.jt808.v1` | `jt808-ingest-app` | `vehicle-history-app` | JT808 raw archive reference records. |
|
||||
| `vehicle.event.jt808.v1` | `jt808-ingest-app` | `vehicle-history-app` | JT808 parsed location/session/alarm events keyed by phone or mapped VIN. |
|
||||
| `vehicle.dlq.jt808.v1` | Kafka producer/consumer error paths | Operators/replay tooling | Dead-letter records for failed JT808 production or consumer processing. |
|
||||
|
||||
Default local consumer groups:
|
||||
|
||||
@@ -173,6 +177,18 @@ Do not expect `vehicle-history-app` to create raw `.bin` archive files from Kafk
|
||||
|
||||
Do not mark any of these as verified unless the matching command was run and the output was inspected.
|
||||
|
||||
## Local launchctl Deployment
|
||||
|
||||
Use the templates under `deploy/local/launchctl/` for the local production-verification machine. They codify the current three-service deployment and keep the runtime knobs in one place:
|
||||
|
||||
- GB32960 ingest: `com.lingniu.gb32960`, TCP `32960`, HTTP `20100`
|
||||
- JT808 ingest: `com.lingniu.jt808`, TCP `808`, HTTP `20482`
|
||||
- History: `com.lingniu.vehicle-history`, HTTP `20200`
|
||||
|
||||
The two ingest services and history service must share the same `SINK_ARCHIVE_PATH`. Kafka raw messages contain `archive://...` references, so history can only read raw frames when its archive root points to the same directory or shared volume that ingest used to write the `.bin` files.
|
||||
|
||||
For Portainer deployments, mirror this same rule with a shared volume mounted to all three services, for example `/archive` in `deploy/portainer/docker-compose.yml`.
|
||||
|
||||
## Rollback Guidance
|
||||
|
||||
If the split deployment is unhealthy:
|
||||
@@ -222,3 +238,16 @@ Observed on 2026-06-23 with Kafka `114.55.58.251:9092`:
|
||||
- Runtime logs showed successful GB32960 ACK flushes for platform login, vehicle login, and vehicle logout after the Kafka-backed split ingest service was running.
|
||||
|
||||
This verifies the live path `GB32960 TCP 32960 -> gb32960-ingest-app -> Kafka raw/event topics` against the remote Kafka broker. Downstream `vehicle-history-app` and `vehicle-analytics-app` were not started in this pass.
|
||||
|
||||
## 2026-06-29 Local Production-Path Verification
|
||||
|
||||
Observed on 2026-06-29 in worktree `.worktrees/vehicle-ingest-redesign-phase1`:
|
||||
|
||||
- `jt808-ingest-app` was running through launchctl on TCP `808` and HTTP `20482`; `curl -sS http://127.0.0.1:20482/actuator/health` returned top-level status `UP`.
|
||||
- `gb32960-ingest-app` was running through launchctl on TCP `32960` and HTTP `20100`; `curl -sS http://127.0.0.1:20100/actuator/health` returned top-level status `UP`.
|
||||
- `vehicle-history-app` was running through launchctl on HTTP `20200`; `curl -sS http://127.0.0.1:20200/actuator/health` returned top-level status `UP`.
|
||||
- External JT808 traffic on TCP `808` was parsed and written through Kafka into TDengine. Direct TDengine checks showed more than `6000` JT808 `raw_frames`, more than `1900` JT808 `vehicle_locations`, and more than `15000` JT808 `telemetry_fields`.
|
||||
- A GB32960 synthetic realtime frame sent to TCP `32960` received a binary ACK beginning with `2323`. The same VIN `LTEST202606290001` was queryable from TDengine-backed history APIs with speed, mileage, longitude, and latitude fields.
|
||||
- History raw and event Kafka consumption used separate bindings so raw-frame writes remained current while location/field event ingestion continued.
|
||||
|
||||
This verifies the local path `GB32960/JT808 TCP -> protocol app -> shared raw archive -> Kafka raw/event topics -> vehicle-history-app -> TDengine -> Swagger/API query`.
|
||||
|
||||
@@ -93,6 +93,36 @@ curl -sS http://127.0.0.1:20200/v3/api-docs \
|
||||
|
||||
预期：健康检查为 `UP`，OpenAPI 中包含这两个 TDengine 查询接口。
|
||||
|
||||
## 本机 launchctl 部署
|
||||
|
||||
当前本机生产验证建议使用仓库模板生成 plist：
|
||||
|
||||
```bash
|
||||
deploy/local/launchctl/
|
||||
```
|
||||
|
||||
关键约束：
|
||||
|
||||
- `jt808-ingest-app` 监听 TCP `808`，HTTP `20482`。
|
||||
- `gb32960-ingest-app` 监听 TCP `32960`，HTTP `20100`。
|
||||
- `vehicle-history-app` 监听 HTTP `20200`。
|
||||
- 三个服务必须共用同一个 `SINK_ARCHIVE_PATH`，否则 TDengine 中的 `raw_uri` 能入库，但 history API 无法按 `archive://...` 读取原始帧。
|
||||
- 替换 jar 前先 `launchctl bootout`，再复制 jar 和 `launchctl bootstrap`，避免 KeepAlive 在 jar 拷贝中途重启。
|
||||
|
||||
健康检查：
|
||||
|
||||
```bash
|
||||
curl -sS http://127.0.0.1:20482/actuator/health
|
||||
curl -sS http://127.0.0.1:20100/actuator/health
|
||||
curl -sS http://127.0.0.1:20200/actuator/health
|
||||
```
|
||||
|
||||
Swagger：
|
||||
|
||||
- JT808 ingest: `http://127.0.0.1:20482/swagger-ui/index.html`
|
||||
- GB32960 ingest: `http://127.0.0.1:20100/swagger-ui/index.html`
|
||||
- History query: `http://127.0.0.1:20200/swagger-ui/index.html`
|
||||
|
||||
## Kafka Topic 验证
|
||||
|
||||
历史服务必须消费事件 topic 和 raw topic：
|
||||
@@ -161,6 +191,78 @@ SELECT COUNT(*) FROM telemetry_fields;
|
||||
- TDengine 未启用时，TDengine 专属接口不应该出现在 OpenAPI 中。
|
||||
- 路径不存在时，API 层应该返回 HTTP `404`，而不是泛化的存储失败。
|
||||
|
||||
## 当前本地缺口
|
||||
## 2026-06-29 本机实时链路验证结果
|
||||
|
||||
本次实现验证环境中，Kafka `114.55.58.251:9092` 可连通，但本机 `127.0.0.1:6041` 不可连通。完整 TDengine 写入和查询验收需要提供真实可用的 `TDENGINE_JDBC_URL`。
|
||||
运行服务：
|
||||
|
||||
- `com.lingniu.jt808.808`: `jt808-ingest-app`，TCP `808`，HTTP `20482`
|
||||
- `com.lingniu.gb32960.32960`: `gb32960-ingest-app`，TCP `32960`，HTTP `20100`
|
||||
- `com.lingniu.vehicle-history.jt808`: `vehicle-history-app`，HTTP `20200`
|
||||
|
||||
健康检查已确认三个服务均返回 `{"status":"UP"}`。
|
||||
|
||||
JT808 真实转发链路已验证：
|
||||
|
||||
- 本机 TCP `808` 有外部平台持续发送 JT808 报文。
|
||||
- `vehicle.raw.jt808.v1` 和 `vehicle.event.jt808.v1` 被 history 消费。
|
||||
- `raw_frames` 中 `protocol='JT808'` 的真实行数已超过 `6000`。
|
||||
- `vehicle_locations` 中 `protocol='JT808'` 的位置行数已超过 `1900`。
|
||||
- `telemetry_fields` 中 `protocol='JT808'` 的字段行数已超过 `15000`。
|
||||
- 最新样例包含终端号 `13079963310`、消息 ID `512`、`archive://2026/06/29/JT808/...` 原始帧引用。
|
||||
|
||||
复查 SQL：
|
||||
|
||||
```sql
|
||||
USE vehicle_ts;
|
||||
SELECT COUNT(*) FROM raw_frames WHERE protocol = 'JT808';
|
||||
SELECT COUNT(*) FROM vehicle_locations WHERE protocol = 'JT808';
|
||||
SELECT COUNT(*) FROM telemetry_fields WHERE protocol = 'JT808';
|
||||
SELECT event_time, vehicle_key, phone, message_id, raw_uri
|
||||
FROM raw_frames
|
||||
WHERE protocol = 'JT808'
|
||||
ORDER BY event_time DESC
|
||||
LIMIT 5;
|
||||
```
|
||||
|
||||
复查 API：
|
||||
|
||||
```bash
|
||||
curl -sS 'http://127.0.0.1:20200/api/event-history/jt808/locations?phone=13079963296&dateFrom=2026-06-29T00:00:00%2B08:00&dateTo=2026-06-29T23:59:59%2B08:00&limit=10'
|
||||
```
|
||||
|
||||
GB32960 链路已用合成实时帧验证：
|
||||
|
||||
- TCP `32960` 返回 GB32960 `2323` ACK。
|
||||
- `raw_frames` 中测试 VIN `LTEST202606290001` 已有 `2` 条 raw 记录。
|
||||
- `telemetry_fields` 中测试 VIN `LTEST202606290001` 已有 `34` 条字段记录。
|
||||
- latest raw 包含 `archive://2026/06/29/GB32960/LTEST202606290001/...`。
|
||||
- snapshots API 可查到 `speedKmh=62.4`、`totalMileageKm=223456.7`、`longitude=116.397128`、`latitude=39.916527`。
|
||||
|
||||
复查 SQL：
|
||||
|
||||
```sql
|
||||
USE vehicle_ts;
|
||||
SELECT COUNT(*) FROM raw_frames
|
||||
WHERE protocol = 'GB32960' AND vin = 'LTEST202606290001';
|
||||
SELECT COUNT(*) FROM telemetry_fields
|
||||
WHERE protocol = 'GB32960' AND vin = 'LTEST202606290001';
|
||||
SELECT event_time, vin, raw_uri, raw_id
|
||||
FROM raw_frames
|
||||
WHERE protocol = 'GB32960' AND vin = 'LTEST202606290001'
|
||||
ORDER BY event_time DESC
|
||||
LIMIT 5;
|
||||
```
|
||||
|
||||
复查 API：
|
||||
|
||||
```bash
|
||||
curl -sS 'http://127.0.0.1:20200/api/event-history/gb32960/snapshots?vin=LTEST202606290001&dateFrom=2026-06-29T00:00:00%2B08:00&dateTo=2026-06-29T23:59:59%2B08:00&limit=10'
|
||||
curl -sS 'http://127.0.0.1:20200/api/event-history/gb32960/snapshots/fields?vin=LTEST202606290001&fields=VEHICLE.speedKmh,VEHICLE.totalMileageKm,POSITION_V2016.longitude,POSITION_V2016.latitude&dateFrom=2026-06-29T00:00:00%2B08:00&dateTo=2026-06-29T23:59:59%2B08:00&limit=10'
|
||||
```
|
||||
|
||||
## 当前注意事项
|
||||
|
||||
- 当前本机使用 TDengine WebSocket JDBC，例如 `jdbc:TAOS-WS://<tdengine-host>:6041/vehicle_ts`。
|
||||
- `KAFKA_CONSUMER_AUTO_OFFSET_RESET=latest` 适合生产接入新流量；如果要回放历史 Kafka 数据，需要切换 consumer group 或重置 offset。
|
||||
- `vehicle-history-app` raw 和 event 使用不同 consumer binding；raw consumer 必须开启，否则 `raw_frames` 不会持续增长。
|
||||
- JT808 设备没有 VIN 映射时，`vehicle_key` 使用 `jt808:<phone>`，后续维护注册/VIN 映射后再反写 VIN。
|
||||
|
||||
Reference in New Issue
Block a user