Compare commits

...

34 Commits

Author SHA1 Message Date
lingniu
ef9ac327fe docs: document local ingest deployment verification
All checks were successful
ci/woodpecker/push/woodpecker Pipeline was successful
2026-06-29 16:18:52 +08:00
lingniu
d2bc4a0d0a fix: align portainer archive and jt808 port 2026-06-29 16:09:02 +08:00
lingniu
98b37e71e9 fix: keep jt808 raw history consumer current 2026-06-29 15:56:41 +08:00
lingniu
57ef82f563 fix: expose jt808 location raw archive uri 2026-06-29 15:27:43 +08:00
lingniu
6646b9b1e1 fix: preserve jt808 location raw archive uri 2026-06-29 15:23:34 +08:00
lingniu
b244984fcb fix: stabilize jt808 tdengine history writes 2026-06-29 15:17:31 +08:00
lingniu
8fe0adcac9 fix: create history consumer processor in app runtime 2026-06-29 14:58:19 +08:00
lingniu
00eee9a769 fix: start history kafka consumer after processor setup 2026-06-29 14:55:44 +08:00
lingniu
06423db8d4 fix: always register kafka consumer runner when enabled 2026-06-29 14:52:18 +08:00
lingniu
7abac30ea4 fix: drop history export and tolerate empty tdengine tables 2026-06-29 14:47:54 +08:00
lingniu
b631d5d998 docs: add tdengine verification runbook 2026-06-29 14:35:20 +08:00
lingniu
a66cfe1532 fix: report history storage availability clearly 2026-06-29 14:35:13 +08:00
lingniu
2273d72e95 fix: initialize tdengine schema before first write 2026-06-29 14:28:53 +08:00
lingniu
57ba7dca90 feat: configure tdengine history datasource 2026-06-29 14:21:56 +08:00
lingniu
192f115176 feat: add telemetry field history api 2026-06-29 13:59:06 +08:00
lingniu
90342fe5bf feat: query telemetry fields from tdengine 2026-06-29 13:55:25 +08:00
lingniu
84de9b2dfa test: stabilize gb32960 durable ack boundary 2026-06-29 13:55:18 +08:00
lingniu
3904dcf869 feat: persist telemetry fields to tdengine 2026-06-29 13:48:04 +08:00
lingniu
ab9c669e76 feat: add jt808 tdengine location query api 2026-06-29 13:42:37 +08:00
lingniu
8466d622f6 feat: add tdengine history keyset queries 2026-06-29 13:38:39 +08:00
lingniu
e708091f2d feat: write history envelopes to tdengine 2026-06-29 13:31:33 +08:00
lingniu
4de723d0b4 feat: add tdengine jdbc history writer 2026-06-29 13:28:36 +08:00
lingniu
f58fe51cf5 feat: add tdengine history store foundation 2026-06-29 13:25:31 +08:00
lingniu
5a60b8c43c fix: consume jt808 history topics by default 2026-06-29 13:11:25 +08:00
lingniu
feb9552234 feat: persist raw frames and publish raw facts 2026-06-29 13:10:14 +08:00
lingniu
9c2779f82b feat: add fact payloads to kafka envelope 2026-06-29 12:58:27 +08:00
lingniu
0ec29a732c feat: add raw archive store contract 2026-06-29 12:56:50 +08:00
lingniu
969a245454 feat: add decoded fact model 2026-06-29 12:55:20 +08:00
lingniu
b5453ff888 feat: add raw frame fact model 2026-06-29 12:54:16 +08:00
lingniu
aed023043c feat: add vehicle key derivation 2026-06-29 12:53:08 +08:00
lingniu
4feb1ec829 build: add ingest facts module 2026-06-29 12:52:17 +08:00
lingniu
310ef94a6e docs: plan vehicle ingest redesign phase 1 2026-06-29 12:50:41 +08:00
lingniu
8d69296b09 docs: add chinese vehicle ingest redesign spec 2026-06-29 12:35:31 +08:00
lingniu
cd953e7cd7 docs: add vehicle ingest redesign spec 2026-06-29 12:32:05 +08:00
101 changed files with 8558 additions and 304 deletions

View File

@@ -0,0 +1,73 @@
# macOS launchctl 本机部署模板
本目录用于把本机 32960/JT808/history 三个服务的启动方式仓库化。模板面向当前生产验证链路:
- `gb32960-ingest-app`: TCP `32960`HTTP `20100`
- `jt808-ingest-app`: TCP `808`HTTP `20482`
- `vehicle-history-app`: HTTP `20200`
三个服务必须使用同一个 `SINK_ARCHIVE_PATH`。协议接入服务负责落原始 `.bin` 文件Kafka raw envelope 只携带 `archive://...` 引用history 服务按同一个 archive root 读取原始帧并提供查询。
## 生成 plist
在仓库根目录执行:
```bash
export PROJECT_ROOT="$(pwd)"
export JAVA_BIN="$(/usr/libexec/java_home 2>/dev/null)/bin/java"
export SHARED_ARCHIVE_PATH="$PROJECT_ROOT/data/archive-live"
export EVENT_STORE_PATH="$PROJECT_ROOT/data/event-store-live"
export TDENGINE_JDBC_URL='jdbc:TAOS-WS://<tdengine-host>:6041/vehicle_ts'
export TDENGINE_USERNAME='root'
export TDENGINE_PASSWORD='<tdengine-password>'
mkdir -p "$HOME/Library/LaunchAgents" "$PROJECT_ROOT/data"
mkdir -p /tmp/lingniu-gb32960-live /tmp/lingniu-jt808-live /tmp/lingniu-history-live
for service in gb32960 jt808 vehicle-history; do
sed \
-e "s#__PROJECT_ROOT__#$PROJECT_ROOT#g" \
-e "s#__JAVA_BIN__#$JAVA_BIN#g" \
-e "s#__SHARED_ARCHIVE_PATH__#$SHARED_ARCHIVE_PATH#g" \
-e "s#__EVENT_STORE_PATH__#$EVENT_STORE_PATH#g" \
-e "s#__TDENGINE_JDBC_URL__#$TDENGINE_JDBC_URL#g" \
-e "s#__TDENGINE_USERNAME__#$TDENGINE_USERNAME#g" \
-e "s#__TDENGINE_PASSWORD__#$TDENGINE_PASSWORD#g" \
"deploy/local/launchctl/com.lingniu.${service}.plist.template" \
> "$HOME/Library/LaunchAgents/com.lingniu.${service}.plist"
done
```
如果生产配置由 Nacos 下发,可以把模板里的 `NACOS_CONFIG_ENABLED` 改成 `true`,并补充 `NACOS_SERVER_ADDR``NACOS_NAMESPACE``NACOS_GROUP``NACOS_USERNAME``NACOS_PASSWORD`。端口、Kafka topic、archive root、TDengine 连接建议仍保留为启动环境变量,便于 Portainer、launchctl 和临时压测保持一致。
## 启停
```bash
launchctl bootstrap "gui/$(id -u)" "$HOME/Library/LaunchAgents/com.lingniu.gb32960.plist"
launchctl bootstrap "gui/$(id -u)" "$HOME/Library/LaunchAgents/com.lingniu.jt808.plist"
launchctl bootstrap "gui/$(id -u)" "$HOME/Library/LaunchAgents/com.lingniu.vehicle-history.plist"
launchctl list | rg 'com.lingniu.(gb32960|jt808|vehicle-history)'
```
替换 jar 前先停止服务,避免 KeepAlive 在 jar 拷贝中途拉起半包:
```bash
launchctl bootout "gui/$(id -u)" "$HOME/Library/LaunchAgents/com.lingniu.gb32960.plist" || true
launchctl bootout "gui/$(id -u)" "$HOME/Library/LaunchAgents/com.lingniu.jt808.plist" || true
launchctl bootout "gui/$(id -u)" "$HOME/Library/LaunchAgents/com.lingniu.vehicle-history.plist" || true
```
## 验证地址
```bash
curl -sS http://127.0.0.1:20100/actuator/health
curl -sS http://127.0.0.1:20482/actuator/health
curl -sS http://127.0.0.1:20200/actuator/health
```
Swagger
- GB32960 ingest: `http://127.0.0.1:20100/swagger-ui/index.html`
- JT808 ingest: `http://127.0.0.1:20482/swagger-ui/index.html`
- History query: `http://127.0.0.1:20200/swagger-ui/index.html`

View File

@@ -0,0 +1,55 @@
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
<key>Label</key>
<string>com.lingniu.gb32960</string>
<key>WorkingDirectory</key>
<string>__PROJECT_ROOT__</string>
<key>ProgramArguments</key>
<array>
<string>__JAVA_BIN__</string>
<string>-jar</string>
<string>__PROJECT_ROOT__/modules/apps/gb32960-ingest-app/target/gb32960-ingest-app.jar</string>
</array>
<key>EnvironmentVariables</key>
<dict>
<key>HTTP_PORT</key>
<string>20100</string>
<key>GB32960_PORT</key>
<string>32960</string>
<key>GB32960_AUTH_ENABLED</key>
<string>false</string>
<key>KAFKA_ENABLED</key>
<string>true</string>
<key>KAFKA_BROKERS</key>
<string>114.55.58.251:9092</string>
<key>KAFKA_NODE_ID</key>
<string>gb32960-live-32960</string>
<key>KAFKA_TOPIC_GB32960_EVENT</key>
<string>vehicle.event.gb32960.v1</string>
<key>KAFKA_TOPIC_GB32960_RAW</key>
<string>vehicle.raw.gb32960.v1</string>
<key>KAFKA_TOPIC_GB32960_DLQ</key>
<string>vehicle.dlq.gb32960.v1</string>
<key>SINK_ARCHIVE_PATH</key>
<string>__SHARED_ARCHIVE_PATH__</string>
<key>VEHICLE_IDENTITY_STORE</key>
<string>file</string>
<key>VEHICLE_IDENTITY_FILE</key>
<string>__PROJECT_ROOT__/data/vehicle-identity-gb32960.jsonl</string>
<key>NACOS_CONFIG_ENABLED</key>
<string>false</string>
<key>MANAGEMENT_HEALTH_REDIS_ENABLED</key>
<string>false</string>
</dict>
<key>StandardOutPath</key>
<string>/tmp/lingniu-gb32960-live/launchctl-gb32960.log</string>
<key>StandardErrorPath</key>
<string>/tmp/lingniu-gb32960-live/launchctl-gb32960.err.log</string>
<key>RunAtLoad</key>
<true/>
<key>KeepAlive</key>
<true/>
</dict>
</plist>

View File

@@ -0,0 +1,51 @@
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
<key>Label</key>
<string>com.lingniu.jt808</string>
<key>WorkingDirectory</key>
<string>__PROJECT_ROOT__</string>
<key>ProgramArguments</key>
<array>
<string>__JAVA_BIN__</string>
<string>-jar</string>
<string>__PROJECT_ROOT__/modules/apps/jt808-ingest-app/target/jt808-ingest-app.jar</string>
</array>
<key>EnvironmentVariables</key>
<dict>
<key>HTTP_PORT</key>
<string>20482</string>
<key>JT808_PORT</key>
<string>808</string>
<key>KAFKA_ENABLED</key>
<string>true</string>
<key>KAFKA_BROKERS</key>
<string>114.55.58.251:9092</string>
<key>KAFKA_NODE_ID</key>
<string>jt808-live-808</string>
<key>KAFKA_TOPIC_JT808_EVENT</key>
<string>vehicle.event.jt808.v1</string>
<key>KAFKA_TOPIC_JT808_RAW</key>
<string>vehicle.raw.jt808.v1</string>
<key>KAFKA_TOPIC_JT808_DLQ</key>
<string>vehicle.dlq.jt808.v1</string>
<key>SINK_ARCHIVE_PATH</key>
<string>__SHARED_ARCHIVE_PATH__</string>
<key>VEHICLE_IDENTITY_STORE</key>
<string>memory</string>
<key>NACOS_CONFIG_ENABLED</key>
<string>false</string>
<key>MANAGEMENT_HEALTH_REDIS_ENABLED</key>
<string>false</string>
</dict>
<key>StandardOutPath</key>
<string>/tmp/lingniu-jt808-live/launchctl-jt808.log</string>
<key>StandardErrorPath</key>
<string>/tmp/lingniu-jt808-live/launchctl-jt808.err.log</string>
<key>RunAtLoad</key>
<true/>
<key>KeepAlive</key>
<true/>
</dict>
</plist>

View File

@@ -0,0 +1,79 @@
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
<key>Label</key>
<string>com.lingniu.vehicle-history</string>
<key>WorkingDirectory</key>
<string>__PROJECT_ROOT__</string>
<key>ProgramArguments</key>
<array>
<string>__JAVA_BIN__</string>
<string>-jar</string>
<string>__PROJECT_ROOT__/modules/apps/vehicle-history-app/target/vehicle-history-app.jar</string>
</array>
<key>EnvironmentVariables</key>
<dict>
<key>HTTP_PORT</key>
<string>20200</string>
<key>KAFKA_ENABLED</key>
<string>true</string>
<key>KAFKA_CONSUMER_ENABLED</key>
<string>true</string>
<key>KAFKA_CONSUMER_AUTO_OFFSET_RESET</key>
<string>latest</string>
<key>KAFKA_CONSUMER_CLIENT_ID_PREFIX</key>
<string>vehicle-history-live</string>
<key>KAFKA_GROUP_HISTORY</key>
<string>vehicle-history-live</string>
<key>KAFKA_BROKERS</key>
<string>114.55.58.251:9092</string>
<key>KAFKA_TOPIC_GB32960_EVENT</key>
<string>vehicle.event.gb32960.v1</string>
<key>KAFKA_TOPIC_GB32960_RAW</key>
<string>vehicle.raw.gb32960.v1</string>
<key>KAFKA_TOPIC_JT808_EVENT</key>
<string>vehicle.event.jt808.v1</string>
<key>KAFKA_TOPIC_JT808_RAW</key>
<string>vehicle.raw.jt808.v1</string>
<key>EVENT_FILE_STORE_ENABLED</key>
<string>true</string>
<key>EVENT_FILE_STORE_PATH</key>
<string>__EVENT_STORE_PATH__</string>
<key>EVENT_FILE_STORE_ZONE_ID</key>
<string>Asia/Shanghai</string>
<key>SINK_ARCHIVE_PATH</key>
<string>__SHARED_ARCHIVE_PATH__</string>
<key>TDENGINE_HISTORY_ENABLED</key>
<string>true</string>
<key>TDENGINE_HISTORY_DATABASE</key>
<string>vehicle_ts</string>
<key>TDENGINE_JDBC_URL</key>
<string>__TDENGINE_JDBC_URL__</string>
<key>TDENGINE_DRIVER_CLASS_NAME</key>
<string>com.taosdata.jdbc.ws.WebSocketDriver</string>
<key>TDENGINE_USERNAME</key>
<string>__TDENGINE_USERNAME__</string>
<key>TDENGINE_PASSWORD</key>
<string>__TDENGINE_PASSWORD__</string>
<key>TDENGINE_MIN_IDLE</key>
<string>0</string>
<key>TDENGINE_MAX_POOL_SIZE</key>
<string>16</string>
<key>TDENGINE_INITIALIZE_SCHEMA</key>
<string>true</string>
<key>NACOS_CONFIG_ENABLED</key>
<string>false</string>
<key>MANAGEMENT_HEALTH_REDIS_ENABLED</key>
<string>false</string>
</dict>
<key>StandardOutPath</key>
<string>/tmp/lingniu-history-live/vehicle-history.log</string>
<key>StandardErrorPath</key>
<string>/tmp/lingniu-history-live/vehicle-history.err.log</string>
<key>RunAtLoad</key>
<true/>
<key>KeepAlive</key>
<true/>
</dict>
</plist>

View File

@@ -25,7 +25,7 @@ x-common-env: &common-env
services:
gb32960-ingest-app:
image: crpi-85r4m0ackrm3qpje-vpc.cn-shanghai.personal.cr.aliyuncs.com/oneos/gb32960-ingest-app:${LINGNIU_IMAGE_VERSION:?set LINGNIU_IMAGE_VERSION}
image: crpi-85r4m0ackrm3qpje.cn-shanghai.personal.cr.aliyuncs.com/oneos/gb32960-ingest-app:${LINGNIU_IMAGE_VERSION:?set LINGNIU_IMAGE_VERSION}
container_name: gb32960-ingest-app
restart: unless-stopped
environment:
@@ -39,36 +39,40 @@ services:
GB32960_PLATFORM_PWD_HYUNDAI: ${GB32960_PLATFORM_PWD_HYUNDAI:-}
GB32960_PLATFORM_IP_HYUNDAI: ${GB32960_PLATFORM_IP_HYUNDAI:-}
GB32960_TLS_ENABLED: ${GB32960_TLS_ENABLED:-false}
SINK_ARCHIVE_PATH: /archive/
VEHICLE_IDENTITY_FILE: /data/vehicle-identity.jsonl
ports:
- "${GB32960_HTTP_PORT:-20100}:20100"
- "${GB32960_TCP_PORT:-32960}:32960"
volumes:
- vehicle-history-archive:/archive
- gb32960-ingest-data:/data
networks:
- vehicle-ingest
jt808-ingest-app:
image: crpi-85r4m0ackrm3qpje-vpc.cn-shanghai.personal.cr.aliyuncs.com/oneos/jt808-ingest-app:${LINGNIU_IMAGE_VERSION:?set LINGNIU_IMAGE_VERSION}
image: crpi-85r4m0ackrm3qpje.cn-shanghai.personal.cr.aliyuncs.com/oneos/jt808-ingest-app:${LINGNIU_IMAGE_VERSION:?set LINGNIU_IMAGE_VERSION}
container_name: jt808-ingest-app
restart: unless-stopped
environment:
<<: *common-env
HTTP_PORT: 20400
JT808_PORT: 10808
JT808_PORT: 808
KAFKA_CONSUMER_ENABLED: "false"
KAFKA_NODE_ID: ${KAFKA_NODE_ID_JT808:-jt808-ingest-portainer}
SINK_ARCHIVE_PATH: /archive/
VEHICLE_IDENTITY_FILE: /data/vehicle-identity.jsonl
ports:
- "${JT808_HTTP_PORT:-20400}:20400"
- "${JT808_TCP_PORT:-10808}:10808"
- "${JT808_TCP_PORT:-808}:808"
volumes:
- vehicle-history-archive:/archive
- jt808-ingest-data:/data
networks:
- vehicle-ingest
vehicle-history-app:
image: crpi-85r4m0ackrm3qpje-vpc.cn-shanghai.personal.cr.aliyuncs.com/oneos/vehicle-history-app:${LINGNIU_IMAGE_VERSION:?set LINGNIU_IMAGE_VERSION}
image: crpi-85r4m0ackrm3qpje.cn-shanghai.personal.cr.aliyuncs.com/oneos/vehicle-history-app:${LINGNIU_IMAGE_VERSION:?set LINGNIU_IMAGE_VERSION}
container_name: vehicle-history-app
restart: unless-stopped
depends_on:
@@ -84,6 +88,15 @@ services:
EVENT_FILE_STORE_ENABLED: ${EVENT_FILE_STORE_ENABLED:-true}
EVENT_FILE_STORE_PATH: /event-store/
EVENT_FILE_STORE_ZONE_ID: ${EVENT_FILE_STORE_ZONE_ID:-Asia/Shanghai}
TDENGINE_HISTORY_ENABLED: ${TDENGINE_HISTORY_ENABLED:-true}
TDENGINE_HISTORY_DATABASE: ${TDENGINE_HISTORY_DATABASE:-vehicle_history}
TDENGINE_JDBC_URL: ${TDENGINE_JDBC_URL:-jdbc:TAOS-RS://tdengine:6041/vehicle_history}
TDENGINE_USERNAME: ${TDENGINE_USERNAME:-root}
TDENGINE_PASSWORD: ${TDENGINE_PASSWORD:-taosdata}
TDENGINE_MAX_POOL_SIZE: ${TDENGINE_MAX_POOL_SIZE:-32}
TDENGINE_MIN_IDLE: ${TDENGINE_MIN_IDLE:-0}
TDENGINE_CONNECTION_TIMEOUT_MILLIS: ${TDENGINE_CONNECTION_TIMEOUT_MILLIS:-5000}
TDENGINE_INITIALIZATION_FAIL_TIMEOUT_MILLIS: ${TDENGINE_INITIALIZATION_FAIL_TIMEOUT_MILLIS:--1}
ports:
- "${VEHICLE_HISTORY_HTTP_PORT:-20200}:20200"
volumes:
@@ -93,7 +106,7 @@ services:
- vehicle-ingest
vehicle-analytics-app:
image: crpi-85r4m0ackrm3qpje-vpc.cn-shanghai.personal.cr.aliyuncs.com/oneos/vehicle-analytics-app:${LINGNIU_IMAGE_VERSION:?set LINGNIU_IMAGE_VERSION}
image: crpi-85r4m0ackrm3qpje.cn-shanghai.personal.cr.aliyuncs.com/oneos/vehicle-analytics-app:${LINGNIU_IMAGE_VERSION:?set LINGNIU_IMAGE_VERSION}
container_name: vehicle-analytics-app
restart: unless-stopped
depends_on:

View File

@@ -7,6 +7,7 @@ This runbook covers the split GB32960 ingest, history, and analytics runtimes. K
| Service | Module | Role | Local ports |
| --- | --- | --- | --- |
| GB32960 ingest | `:gb32960-ingest-app` | Accept GB32960 TCP connections, decode/authenticate frames, produce raw and normalized records to Kafka, and send protocol ACKs after required Kafka production succeeds. | TCP `32960`, HTTP `20100` |
| JT808 ingest | `:jt808-ingest-app` | Accept JT/T 808 TCP connections on the production receive port, parse raw frames, archive frame bytes, and produce raw/event envelopes to Kafka. | TCP `808`, HTTP `20482` |
| Vehicle history | `:vehicle-history-app` | Consume GB32960 raw/event Kafka records, store event-history records and raw archive references, and expose event/history/GB32960 query APIs. Kafka raw envelopes carry archive URI/size metadata, not raw frame bytes. | HTTP `20200` |
| Vehicle analytics | `:vehicle-analytics-app` | Consume GB32960 event Kafka records and update analytics outputs such as daily vehicle statistics. Vehicle state can also be enabled from this runtime. | HTTP `20300` |
@@ -17,6 +18,9 @@ This runbook covers the split GB32960 ingest, history, and analytics runtimes. K
| `vehicle.raw.gb32960.v1` | `gb32960-ingest-app` | `vehicle-history-app` | Raw archive reference records. The envelope contains URI/size metadata, not the raw `.bin` payload. |
| `vehicle.event.gb32960.v1` | `gb32960-ingest-app` | `vehicle-history-app`, `vehicle-analytics-app` | Normalized vehicle events keyed by VIN when available. |
| `vehicle.dlq.gb32960.v1` | Kafka producer/consumer error paths | Operators/replay tooling | Dead-letter records for failed production or consumer processing. |
| `vehicle.raw.jt808.v1` | `jt808-ingest-app` | `vehicle-history-app` | JT808 raw archive reference records. |
| `vehicle.event.jt808.v1` | `jt808-ingest-app` | `vehicle-history-app` | JT808 parsed location/session/alarm events keyed by phone or mapped VIN. |
| `vehicle.dlq.jt808.v1` | Kafka producer/consumer error paths | Operators/replay tooling | Dead-letter records for failed JT808 production or consumer processing. |
Default local consumer groups:
@@ -173,6 +177,18 @@ Do not expect `vehicle-history-app` to create raw `.bin` archive files from Kafk
Do not mark any of these as verified unless the matching command was run and the output was inspected.
## Local launchctl Deployment
Use the templates under `deploy/local/launchctl/` for the local production-verification machine. They codify the current three-service deployment and keep the runtime knobs in one place:
- GB32960 ingest: `com.lingniu.gb32960`, TCP `32960`, HTTP `20100`
- JT808 ingest: `com.lingniu.jt808`, TCP `808`, HTTP `20482`
- History: `com.lingniu.vehicle-history`, HTTP `20200`
The two ingest services and history service must share the same `SINK_ARCHIVE_PATH`. Kafka raw messages contain `archive://...` references, so history can only read raw frames when its archive root points to the same directory or shared volume that ingest used to write the `.bin` files.
For Portainer deployments, mirror this same rule with a shared volume mounted to all three services, for example `/archive` in `deploy/portainer/docker-compose.yml`.
## Rollback Guidance
If the split deployment is unhealthy:
@@ -222,3 +238,16 @@ Observed on 2026-06-23 with Kafka `114.55.58.251:9092`:
- Runtime logs showed successful GB32960 ACK flushes for platform login, vehicle login, and vehicle logout after the Kafka-backed split ingest service was running.
This verifies the live path `GB32960 TCP 32960 -> gb32960-ingest-app -> Kafka raw/event topics` against the remote Kafka broker. Downstream `vehicle-history-app` and `vehicle-analytics-app` were not started in this pass.
## 2026-06-29 Local Production-Path Verification
Observed on 2026-06-29 in worktree `.worktrees/vehicle-ingest-redesign-phase1`:
- `jt808-ingest-app` was running through launchctl on TCP `808` and HTTP `20482`; `curl -sS http://127.0.0.1:20482/actuator/health` returned top-level status `UP`.
- `gb32960-ingest-app` was running through launchctl on TCP `32960` and HTTP `20100`; `curl -sS http://127.0.0.1:20100/actuator/health` returned top-level status `UP`.
- `vehicle-history-app` was running through launchctl on HTTP `20200`; `curl -sS http://127.0.0.1:20200/actuator/health` returned top-level status `UP`.
- External JT808 traffic on TCP `808` was parsed and written through Kafka into TDengine. Direct TDengine checks showed more than `6000` JT808 `raw_frames`, more than `1900` JT808 `vehicle_locations`, and more than `15000` JT808 `telemetry_fields`.
- A GB32960 synthetic realtime frame sent to TCP `32960` received a binary ACK beginning with `2323`. The same VIN `LTEST202606290001` was queryable from TDengine-backed history APIs with speed, mileage, longitude, and latitude fields.
- History raw and event Kafka consumption used separate bindings so raw-frame writes remained current while location/field event ingestion continued.
This verifies the local path `GB32960/JT808 TCP -> protocol app -> shared raw archive -> Kafka raw/event topics -> vehicle-history-app -> TDengine -> Swagger/API query`.

View File

@@ -0,0 +1,268 @@
# Vehicle Ingest TDengine 链路验收手册
这份手册用于验收当前 32960/JT808 接入重构链路:
1. 协议 App 接收 TCP 报文。
2. 协议 App 归档原始帧元数据,并把事件信封发布到 Kafka。
3. `vehicle-history-app` 消费 Kafka。
4. history 写入 raw、location、telemetry field 到 TDengine。
5. Swagger/API 通过 TDengine 做历史分页查询。
每一步都必须执行命令并检查输出后,才能标记为已验证。
## 必需运行参数
启动服务前先设置:
```bash
export KAFKA_BROKERS=114.55.58.251:9092
export TDENGINE_HISTORY_ENABLED=true
export TDENGINE_JDBC_URL='jdbc:TAOS-RS://<tdengine-host>:6041/vehicle_history'
export TDENGINE_USERNAME=root
export TDENGINE_PASSWORD='<tdengine-password>'
export TDENGINE_MIN_IDLE=0
export TDENGINE_MAX_POOL_SIZE=32
```
`TDENGINE_MIN_IDLE=0` 用于减少冷启动时 TDengine 连接重试日志。TDengine 地址稳定后,再按生产并发调大连接池。
## 构建
```bash
mvn -pl modules/apps/jt808-ingest-app,modules/apps/gb32960-ingest-app,modules/apps/vehicle-history-app -am package -DskipTests
```
预期产物:
- `modules/apps/jt808-ingest-app/target/jt808-ingest-app.jar`
- `modules/apps/gb32960-ingest-app/target/gb32960-ingest-app.jar`
- `modules/apps/vehicle-history-app/target/vehicle-history-app.jar`
## 启动 JT808 接入服务,监听 808 端口
```bash
HTTP_PORT=20400 \
JT808_PORT=808 \
KAFKA_CONSUMER_ENABLED=false \
java -jar modules/apps/jt808-ingest-app/target/jt808-ingest-app.jar
```
验证:
```bash
curl -sS http://127.0.0.1:20400/actuator/health
lsof -nP -iTCP:808 -sTCP:LISTEN
```
预期:健康检查为 `UP`,并且 Java 进程监听 TCP `808`
## 启动 GB32960 接入服务
```bash
HTTP_PORT=20100 \
GB32960_PORT=32960 \
KAFKA_CONSUMER_ENABLED=false \
java -jar modules/apps/gb32960-ingest-app/target/gb32960-ingest-app.jar
```
验证:
```bash
curl -sS http://127.0.0.1:20100/actuator/health
lsof -nP -iTCP:32960 -sTCP:LISTEN
```
## 启动 TDengine 版历史服务
```bash
HTTP_PORT=20200 \
KAFKA_CONSUMER_ENABLED=true \
EVENT_FILE_STORE_ENABLED=true \
EVENT_FILE_STORE_PATH=./target/tdengine-verification/event-store \
SINK_ARCHIVE_PATH=./target/tdengine-verification/archive \
java -jar modules/apps/vehicle-history-app/target/vehicle-history-app.jar
```
验证:
```bash
curl -sS http://127.0.0.1:20200/actuator/health
curl -sS http://127.0.0.1:20200/v3/api-docs \
| grep -E '/api/event-history/jt808/locations|/api/event-history/telemetry/fields'
```
预期:健康检查为 `UP`OpenAPI 中包含这两个 TDengine 查询接口。
## 本机 launchctl 部署
当前本机生产验证建议使用仓库模板生成 plist
```bash
deploy/local/launchctl/
```
关键约束:
- `jt808-ingest-app` 监听 TCP `808`HTTP `20482`
- `gb32960-ingest-app` 监听 TCP `32960`HTTP `20100`
- `vehicle-history-app` 监听 HTTP `20200`
- 三个服务必须共用同一个 `SINK_ARCHIVE_PATH`,否则 TDengine 中的 `raw_uri` 能入库,但 history API 无法按 `archive://...` 读取原始帧。
- 替换 jar 前先 `launchctl bootout`,再复制 jar 和 `launchctl bootstrap`,避免 KeepAlive 在 jar 拷贝中途重启。
健康检查:
```bash
curl -sS http://127.0.0.1:20482/actuator/health
curl -sS http://127.0.0.1:20100/actuator/health
curl -sS http://127.0.0.1:20200/actuator/health
```
Swagger
- JT808 ingest: `http://127.0.0.1:20482/swagger-ui/index.html`
- GB32960 ingest: `http://127.0.0.1:20100/swagger-ui/index.html`
- History query: `http://127.0.0.1:20200/swagger-ui/index.html`
## Kafka Topic 验证
历史服务必须消费事件 topic 和 raw topic
- `vehicle.event.gb32960.v1`
- `vehicle.raw.gb32960.v1`
- `vehicle.event.jt808.v1`
- `vehicle.raw.jt808.v1`
使用 Kafka CLI 或管理工具检查 topic 是否存在,并确认 `vehicle-history` consumer group 在测试流量后没有持续堆积。
## JT808 实时转发验收
如果外部平台已经把 JT808 报文转发到本机 TCP `808`,至少观察 60 秒日志和 Kafka 数据。
必须拿到以下证据:
- `jt808-ingest-app` 日志显示接入连接或解析到上游消息 ID。
- Kafka 的 `vehicle.event.jt808.v1``vehicle.raw.jt808.v1` 有新增记录。
- `vehicle-history-app` 日志没有持续出现 consumer 或 TDengine 写入失败。
消费完成后,使用已知终端手机号查询:
```bash
curl -sS 'http://127.0.0.1:20200/api/event-history/jt808/locations?phone=<phone>&dateFrom=2026-06-29T00:00:00%2B08:00&dateTo=2026-06-29T23:59:59%2B08:00&limit=10'
```
预期:响应包含 `items`,并且每条记录包含 `eventTime``phone``longitude``latitude``speedKmh``rawUri``metadataJson`
## Telemetry Field 查询验收
查询 GB32960 字段历史:
```bash
curl -sS 'http://127.0.0.1:20200/api/event-history/telemetry/fields?protocol=GB32960&vin=<vin>&fieldKey=<field-key>&dateFrom=2026-06-29T00:00:00%2B08:00&dateTo=2026-06-29T23:59:59%2B08:00&limit=10'
```
查询 JT808 字段历史。没有 VIN 映射时,优先使用 `phone`
```bash
curl -sS 'http://127.0.0.1:20200/api/event-history/telemetry/fields?protocol=JT808&phone=<phone>&fieldKey=location.speed_kmh&dateFrom=2026-06-29T00:00:00%2B08:00&dateTo=2026-06-29T23:59:59%2B08:00&limit=10'
```
预期:响应包含 `items`,可能包含 `nextCursor`。下一页使用 `nextCursor` 里的 `cursorTs``cursorId` 查询。
## TDengine 直接检查
使用 TDengine CLI 或 JDBC 客户端检查超级表和子表是否创建:
```sql
USE vehicle_history;
SHOW STABLES;
SELECT COUNT(*) FROM raw_frames;
SELECT COUNT(*) FROM vehicle_locations;
SELECT COUNT(*) FROM telemetry_fields;
```
预期:
- `raw_frames``vehicle_locations``telemetry_fields` 存在。
- 有实时流量后,计数持续增加。
## 失败语义
- TDengine 不可达时TDengine 查询接口应该返回 HTTP `503`,消息类似 `tdengine history query failed`
- TDengine 未启用时TDengine 专属接口不应该出现在 OpenAPI 中。
- 路径不存在时API 层应该返回 HTTP `404`,而不是泛化的存储失败。
## 2026-06-29 本机实时链路验证结果
运行服务:
- `com.lingniu.jt808.808`: `jt808-ingest-app`TCP `808`HTTP `20482`
- `com.lingniu.gb32960.32960`: `gb32960-ingest-app`TCP `32960`HTTP `20100`
- `com.lingniu.vehicle-history.jt808`: `vehicle-history-app`HTTP `20200`
健康检查已确认三个服务均返回 `{"status":"UP"}`
JT808 真实转发链路已验证:
- 本机 TCP `808` 有外部平台持续发送 JT808 报文。
- `vehicle.raw.jt808.v1``vehicle.event.jt808.v1` 被 history 消费。
- `raw_frames``protocol='JT808'` 的真实行数已超过 `6000`
- `vehicle_locations``protocol='JT808'` 的位置行数已超过 `1900`
- `telemetry_fields``protocol='JT808'` 的字段行数已超过 `15000`
- 最新样例包含终端号 `13079963310`、消息 ID `512``archive://2026/06/29/JT808/...` 原始帧引用。
复查 SQL
```sql
USE vehicle_ts;
SELECT COUNT(*) FROM raw_frames WHERE protocol = 'JT808';
SELECT COUNT(*) FROM vehicle_locations WHERE protocol = 'JT808';
SELECT COUNT(*) FROM telemetry_fields WHERE protocol = 'JT808';
SELECT event_time, vehicle_key, phone, message_id, raw_uri
FROM raw_frames
WHERE protocol = 'JT808'
ORDER BY event_time DESC
LIMIT 5;
```
复查 API
```bash
curl -sS 'http://127.0.0.1:20200/api/event-history/jt808/locations?phone=13079963296&dateFrom=2026-06-29T00:00:00%2B08:00&dateTo=2026-06-29T23:59:59%2B08:00&limit=10'
```
GB32960 链路已用合成实时帧验证:
- TCP `32960` 返回 GB32960 `2323` ACK。
- `raw_frames` 中测试 VIN `LTEST202606290001` 已有 `2` 条 raw 记录。
- `telemetry_fields` 中测试 VIN `LTEST202606290001` 已有 `34` 条字段记录。
- latest raw 包含 `archive://2026/06/29/GB32960/LTEST202606290001/...`
- snapshots API 可查到 `speedKmh=62.4``totalMileageKm=223456.7``longitude=116.397128``latitude=39.916527`
复查 SQL
```sql
USE vehicle_ts;
SELECT COUNT(*) FROM raw_frames
WHERE protocol = 'GB32960' AND vin = 'LTEST202606290001';
SELECT COUNT(*) FROM telemetry_fields
WHERE protocol = 'GB32960' AND vin = 'LTEST202606290001';
SELECT event_time, vin, raw_uri, raw_id
FROM raw_frames
WHERE protocol = 'GB32960' AND vin = 'LTEST202606290001'
ORDER BY event_time DESC
LIMIT 5;
```
复查 API
```bash
curl -sS 'http://127.0.0.1:20200/api/event-history/gb32960/snapshots?vin=LTEST202606290001&dateFrom=2026-06-29T00:00:00%2B08:00&dateTo=2026-06-29T23:59:59%2B08:00&limit=10'
curl -sS 'http://127.0.0.1:20200/api/event-history/gb32960/snapshots/fields?vin=LTEST202606290001&fields=VEHICLE.speedKmh,VEHICLE.totalMileageKm,POSITION_V2016.longitude,POSITION_V2016.latitude&dateFrom=2026-06-29T00:00:00%2B08:00&dateTo=2026-06-29T23:59:59%2B08:00&limit=10'
```
## 当前注意事项
- 当前本机使用 TDengine WebSocket JDBC例如 `jdbc:TAOS-WS://<tdengine-host>:6041/vehicle_ts`
- `KAFKA_CONSUMER_AUTO_OFFSET_RESET=latest` 适合生产接入新流量;如果要回放历史 Kafka 数据,需要切换 consumer group 或重置 offset。
- `vehicle-history-app` raw 和 event 使用不同 consumer bindingraw consumer 必须开启,否则 `raw_frames` 不会持续增长。
- JT808 设备没有 VIN 映射时,`vehicle_key` 使用 `jt808:<phone>`,后续维护注册/VIN 映射后再反写 VIN。

File diff suppressed because it is too large Load Diff

View File

@@ -0,0 +1,621 @@
# Vehicle Ingest Redesign Design
## Background
Current GB32960 and JT808 ingestion can receive, decode, publish, and query data, but the responsibilities are split across several partially overlapping paths:
- Netty handlers produce `RawFrame` and the dispatcher emits both raw archive events and normalized vehicle events.
- Kafka envelopes carry raw archive references, while raw bytes are stored separately by the archive sink.
- GB32960 historical queries use raw archive records plus file replay decoding.
- JT808 location history is materialized separately into TDengine.
- DuckDB/Parquet, local raw archive, Kafka, and TDengine each own part of the story, so query and durability semantics are hard to reason about.
The new design intentionally keeps useful protocol parsers and app deployment knowledge, but replaces the ingestion, durability, storage, and query boundaries with a first-principles model.
## Goals
1. Store every original inbound protocol frame, including malformed frames when bytes are available.
2. Query historical uploaded data for GB32960 and JT808 by vehicle, protocol, message type, and time range.
3. Extend to new fields, protocol messages, and vendor profiles without rewriting the core pipeline.
4. Sustain production load for 10,000 vehicles and 1,000 concurrent historical query requests per second.
5. Make ACK and durability rules explicit enough that an acknowledged frame is not silently lost.
## Non-Goals
- Do not make Kafka the long-term historical query database.
- Do not store large raw byte arrays directly in Kafka topics.
- Do not keep DuckDB/Parquet as the production hot query store for GB32960 and JT808.
- Do not build one generic API that hides protocol-specific semantics. Shared internals are preferred; external APIs may remain protocol-aware.
## Recommended Architecture
Use `Raw Archive + Kafka + TDengine` as the core architecture.
```mermaid
flowchart LR
GB["GB32960 TCP"] --> Edge["Protocol Edge"]
JT["JT808 TCP"] --> Edge
Edge --> RawStore["Raw Archive"]
RawStore --> RawTopic["Kafka vehicle.raw-frame.v1"]
Edge --> Decode["Protocol Decoder"]
Decode --> FactTopic["Kafka vehicle.decoded-fact.v1"]
RawTopic --> Writer["History Writer"]
FactTopic --> Writer
Writer --> TD["TDengine"]
Writer --> Redis["Realtime State"]
API["History API"] --> TD
API --> RawStore
RealtimeAPI["Realtime API"] --> Redis
```
The architecture has three facts:
- `RawFrameFact`: one row per inbound frame, whether parsing succeeds or fails.
- `DecodedFact`: zero or more normalized facts derived from a raw frame.
- `RawArchiveObject`: immutable raw bytes addressed by `raw_uri`.
The raw archive is the forensic source of truth. TDengine is the query source of truth. Kafka is the decoupling and replay pipe.
## Module Boundaries
### Protocol Edge Apps
Apps:
- `gb32960-ingest-app`
- `jt808-ingest-app`
Responsibilities:
- Accept TCP connections.
- Split frames.
- Perform protocol-level validation, authentication, session binding, and ACK.
- Create `RawFrameFact`.
- Persist raw bytes through `RawArchiveWriter`.
- Publish raw frame fact to Kafka.
- Decode frames and publish decoded facts.
They do not query history, export data, or write business tables directly.
### Ingest Core
New module:
- `modules/core/ingest-facts`
Core types:
```java
public record RawFrameFact(
String frameId,
ProtocolId protocol,
String vehicleKey,
String vin,
String phone,
int messageId,
int subType,
Instant eventTime,
Instant receivedAt,
String peer,
String rawUri,
String checksum,
long rawSizeBytes,
ParseStatus parseStatus,
String parseError,
Map<String, String> metadata
) {}
```
```java
public sealed interface DecodedFact permits
DecodedFact.Location,
DecodedFact.Realtime,
DecodedFact.Alarm,
DecodedFact.Session,
DecodedFact.Register,
DecodedFact.Extension {
String factId();
String frameId();
ProtocolId protocol();
String vehicleKey();
String vin();
Instant eventTime();
Instant receivedAt();
String rawUri();
Map<String, String> metadata();
}
```
`vehicleKey` is the stable partition key:
- GB32960: VIN.
- JT808: resolved VIN when known, otherwise `jt808:<phone>`.
- Unknown/malformed: `unknown:<protocol>:<hash>`.
### Raw Archive
New or refactored module:
- `modules/sinks/raw-archive-store`
Interface:
```java
public interface RawArchiveWriter {
RawArchiveReceipt write(RawArchiveWriteRequest request) throws IOException;
}
```
Receipt:
```java
public record RawArchiveReceipt(
String rawUri,
String checksum,
long sizeBytes
) {}
```
Rules:
- Raw bytes are immutable once written.
- Write path is content-addressed or deterministic by `protocol/date/vehicleKey/frameId.bin`.
- Write must be fsync-capable for local storage and pluggable for object storage.
- `rawUri` must be resolvable by history API for single-frame replay.
- Malformed frames are written with `parseStatus=FAILED` when bytes are available.
Deployment constraints:
- Raw archive, TDengine, and the history service may run on different ECS instances, but should use private-network access in the same VPC and availability zone.
- High-frequency historical queries only read TDengine and must not read raw archive.
- Raw archive is only used by single-frame detail, troubleshooting, replay, and asynchronous export paths.
- `rawUri` must be globally resolvable by services and must not be a private local absolute path on one ECS instance.
- The first local raw archive implementation must stay behind a replaceable interface so it can later move to OSS, MinIO, or another object store.
### Kafka Topics
Use stable versioned topics:
- `vehicle.raw-frame.v1`
- `vehicle.decoded-fact.v1`
- `vehicle.dlq.v1`
Kafka key:
```text
<protocol>:<vehicleKey>
```
`vehicle.raw-frame.v1` payload contains `RawFrameFact` without raw bytes. It carries `rawUri`, checksum, byte size, message id, parse status, and metadata.
`vehicle.decoded-fact.v1` payload contains one `DecodedFact`. It always references `frameId` and `rawUri`.
`vehicle.dlq.v1` carries failed store, publish, decode, and write attempts with enough metadata to replay from raw archive when possible.
### History Writer
New app or refactored app:
- `vehicle-history-app`
Responsibilities:
- Consume `vehicle.raw-frame.v1`.
- Consume `vehicle.decoded-fact.v1`.
- Batch write TDengine.
- Update realtime state store for latest vehicle state.
- Write DLQ on invalid payloads or storage failures.
The writer is idempotent by `(frame_id)` for raw facts and `(fact_id)` for decoded facts.
## TDengine Model
TDengine is the production historical query store.
### Raw Frames Super Table
```sql
CREATE STABLE IF NOT EXISTS raw_frames (
ts TIMESTAMP,
frame_id NCHAR(64),
received_at TIMESTAMP,
message_id INT,
sub_type INT,
event_time TIMESTAMP,
raw_uri NCHAR(512),
checksum NCHAR(128),
raw_size_bytes BIGINT,
parse_status NCHAR(16),
parse_error NCHAR(512),
peer NCHAR(128),
metadata_json NCHAR(4096)
) TAGS (
protocol NCHAR(16),
vehicle_key NCHAR(128),
vin NCHAR(64),
phone NCHAR(32)
);
```
Child table:
```text
raw_<protocol>_<hash(vehicleKey)>
```
Primary query patterns:
- Raw frames by vehicle and time.
- Raw frames by protocol, message id, parse status, and time.
- Single raw frame by `frame_id` or `raw_uri`.
### Location Super Table
```sql
CREATE STABLE IF NOT EXISTS vehicle_locations (
ts TIMESTAMP,
fact_id NCHAR(64),
frame_id NCHAR(64),
received_at TIMESTAMP,
longitude DOUBLE,
latitude DOUBLE,
altitude_m DOUBLE,
speed_kmh DOUBLE,
direction_deg DOUBLE,
alarm_flag BIGINT,
status_flag BIGINT,
total_mileage_km DOUBLE,
raw_uri NCHAR(512),
metadata_json NCHAR(4096)
) TAGS (
protocol NCHAR(16),
vehicle_key NCHAR(128),
vin NCHAR(64),
phone NCHAR(32)
);
```
### Realtime Super Table
```sql
CREATE STABLE IF NOT EXISTS vehicle_realtime (
ts TIMESTAMP,
fact_id NCHAR(64),
frame_id NCHAR(64),
received_at TIMESTAMP,
speed_kmh DOUBLE,
total_mileage_km DOUBLE,
battery_soc DOUBLE,
total_voltage_v DOUBLE,
total_current_a DOUBLE,
running_mode NCHAR(64),
vehicle_state NCHAR(64),
charging_state NCHAR(64),
raw_uri NCHAR(512),
fields_json NCHAR(8192),
metadata_json NCHAR(4096)
) TAGS (
protocol NCHAR(16),
vehicle_key NCHAR(128),
vin NCHAR(64),
phone NCHAR(32)
);
```
`fields_json` stores sparse full-field values for low-frequency inspection. High-frequency queried fields should be promoted to explicit columns or a dedicated extension table.
### Alarm Super Table
```sql
CREATE STABLE IF NOT EXISTS vehicle_alarms (
ts TIMESTAMP,
fact_id NCHAR(64),
frame_id NCHAR(64),
received_at TIMESTAMP,
alarm_level NCHAR(32),
alarm_code BIGINT,
alarm_name NCHAR(128),
longitude DOUBLE,
latitude DOUBLE,
raw_uri NCHAR(512),
fields_json NCHAR(8192),
metadata_json NCHAR(4096)
) TAGS (
protocol NCHAR(16),
vehicle_key NCHAR(128),
vin NCHAR(64),
phone NCHAR(32)
);
```
### Session and Registration Tables
```sql
CREATE STABLE IF NOT EXISTS vehicle_sessions (
ts TIMESTAMP,
fact_id NCHAR(64),
frame_id NCHAR(64),
received_at TIMESTAMP,
session_type NCHAR(32),
result_code INT,
raw_uri NCHAR(512),
fields_json NCHAR(8192),
metadata_json NCHAR(4096)
) TAGS (
protocol NCHAR(16),
vehicle_key NCHAR(128),
vin NCHAR(64),
phone NCHAR(32)
);
```
JT808 registration details go into `vehicle_sessions` first and may later be projected into MySQL identity bindings for durable identity lookup.
## ACK and Durability Rules
For frames that require a protocol ACK:
1. Frame bytes are accepted from Netty.
2. Raw bytes are written to raw archive.
3. `RawFrameFact` is published to Kafka.
4. Protocol-specific ACK is sent.
5. Decoding and `DecodedFact` publication may run before or after ACK, but failures must emit a DLQ event and update raw frame parse status.
For GB32960 realtime/report frames, this prevents acknowledging a frame whose original bytes cannot be recovered.
For JT808 register/auth/heartbeat, ACK follows the same raw archive and raw fact boundary. Register ACK can still include token generation before raw fact publish, but must not be flushed to the channel until the durability boundary succeeds.
If raw archive write fails, close or keep the channel according to protocol tolerance, but do not ACK success.
If Kafka raw fact publish fails, do not ACK success for durable-ACK frames. Non-durable frames go to local DLQ when configured.
## Protocol Decoding and Extension
Use a plugin contract:
```java
public interface ProtocolFrameDecoder {
ProtocolId protocol();
DecodeResult decode(RawFrameFact rawFact, byte[] rawBytes);
}
```
```java
public record DecodeResult(
ParseStatus status,
List<DecodedFact> facts,
String errorMessage,
Map<String, String> metadata
) {}
```
GB32960:
- Frame command maps to session, realtime, location, alarm, and extension facts.
- Vendor extensions are selected by platform account, VIN, or configured profile.
- Full raw replay remains possible by reading `rawUri` and running the current decoder.
JT808:
- Message id maps to register, auth, heartbeat, location, batch location, media metadata, and extension facts.
- Identity resolution updates `vehicleKey`.
- Registration facts carry province, city, maker, device id, plate, plate color, and auth token metadata.
New protocol messages do not require TDengine schema changes unless they become high-frequency query dimensions. Unknown fields are stored as extension facts with `fields_json`.
## Query APIs
Keep API surfaces explicit.
Raw frame APIs:
- `GET /api/history/raw-frames`
- Filters: `protocol`, `vin`, `phone`, `vehicleKey`, `messageId`, `parseStatus`, `dateFrom`, `dateTo`, `pageSize`, `cursor`.
- Returns frame metadata, raw URI, checksum, parse status, and compact decoded summary when available.
Single raw frame API:
- `GET /api/history/raw-frames/{frameId}`
- Returns raw metadata and decoded detail.
- Supports `includeRawHex=true` with a strict size limit.
Location APIs:
- `GET /api/history/locations`
- Filters: `protocol`, `vin`, `phone`, `vehicleKey`, time range, cursor.
Realtime APIs:
- `GET /api/history/realtime`
- Historical time-series query.
Realtime state APIs:
- `GET /api/realtime/vehicles/{vehicleKey}`
- Reads Redis or in-memory latest state, not TDengine.
Export:
- Export is asynchronous.
- Request creates an export job.
- Worker streams TDengine query results to object storage or local export files.
- API returns job status and download URI.
## Pagination
Use cursor pagination, not deep offset, for high-QPS history queries.
Cursor fields:
```text
ts, received_at, fact_id
```
Descending query predicate:
```sql
WHERE (
ts < :cursorTs
OR (ts = :cursorTs AND received_at < :cursorReceivedAt)
OR (ts = :cursorTs AND received_at = :cursorReceivedAt AND fact_id < :cursorFactId)
)
```
Page size defaults to 100 and is capped at 1000.
## Capacity Design
Assumptions:
- 10,000 vehicles.
- Normal report interval: 5 to 10 seconds.
- Expected writes: 1,000 to 2,000 frames per second.
- Peak burst factor: 3x.
- Historical query target: 1,000 requests per second.
Write path:
- Netty worker threads stay CPU-light.
- Raw archive writes use bounded async workers.
- Kafka producer uses compression, batching, idempotence, and vehicle-key partitioning.
- History writer batches TDengine inserts by super table and child table.
- TDengine child tables are partitioned by vehicle key to make single-vehicle queries cheap.
Read path:
- Most high-QPS queries require vehicle key and time range.
- Unbounded cross-vehicle queries are admin-only and capped.
- Realtime latest state is served from Redis/in-memory state.
- Raw detail replay reads one raw object and decodes on demand.
- Common dictionary and metadata responses are cached in memory.
Backpressure:
- Raw archive worker queue has a fixed bound.
- Kafka publish futures are tracked.
- If durability boundary cannot complete within timeout, durable ACK fails and the channel is closed or throttled.
- History writer lag is observable by Kafka consumer lag and TDengine batch latency.
## Migration Plan
Phase 1: Build new fact model beside existing code.
- Add `ingest-facts`.
- Add raw archive receipt contract.
- Add serializers for raw frame and decoded fact Kafka messages.
- Add tests for stable ids, vehicle key, and raw URI generation.
Phase 2: Harden raw durability boundary.
- Refactor GB32960 and JT808 handlers so ACK waits for raw archive and raw fact publish.
- Keep existing parsers.
- Emit decoded facts to the new Kafka topic.
Phase 3: Build TDengine writer.
- Create TDengine schema manager.
- Write raw frame facts.
- Write location, realtime, alarm, session facts.
- Add idempotent insert behavior by stable ids.
Phase 4: Build query APIs.
- Raw frame query.
- Single frame replay.
- Location history query.
- Realtime history query.
- Latest realtime state query.
Phase 5: Cut over and retire old hot paths.
- Stop using DuckDB/Parquet as production hot history for GB32960/JT808.
- Keep raw replay compatibility while historical data migrates.
- Remove protocol-specific ad hoc history storage once TDengine coverage is verified.
## Testing Strategy
Unit tests:
- `RawFrameFact` validation and vehicle key derivation.
- Raw archive path generation and checksum.
- GB32960 decoder facts from golden frames.
- JT808 decoder facts from sample frames.
- ACK boundary success and failure cases.
- TDengine SQL generation.
Integration tests:
- Start app with embedded or mocked Kafka producer and fake raw archive.
- Send GB32960 sample frame and verify raw fact, decoded fact, raw archive receipt.
- Send JT808 register/auth/location frames and verify registration/session/location facts.
- Simulate raw archive failure and verify no success ACK.
- Simulate Kafka publish failure and verify durable ACK failure.
Storage tests:
- TDengine schema initialization.
- Batch insert raw frames and decoded facts.
- Query by vehicle key and time range.
- Cursor pagination correctness.
- Single raw frame replay by `rawUri`.
Load tests:
- 2,000 frames/s sustained write for at least 30 minutes.
- 6,000 frames/s burst for 5 minutes.
- 1,000 QPS location/realtime history queries with bounded p95 latency.
- Verify no raw frame count gap between raw archive receipts, Kafka raw facts, and TDengine raw frame rows.
## Operational Requirements
Metrics:
- TCP connections.
- Frame receive rate.
- Raw archive write latency and failure count.
- Kafka publish latency and failure count.
- Decoder success/failure count by protocol and message id.
- TDengine batch size, latency, and failure count.
- Query QPS and latency by endpoint.
- Kafka consumer lag.
Logs:
- One structured log per failed frame with `frameId`, protocol, vehicle key, message id, raw URI, and error.
- No full raw bytes in normal logs.
- Raw hex only in explicit debug tools with size limits.
Alerts:
- Raw archive write failure.
- Durable ACK timeout.
- Kafka producer failure.
- History writer lag.
- TDengine write failure.
- Raw archive count and TDengine raw frame row count divergence.
## Acceptance Criteria
The redesign is complete only when all of the following are true:
1. Every accepted GB32960 and JT808 frame creates a raw archive object and a TDengine `raw_frames` row.
2. Malformed frames with bytes are queryable from raw history with parse status and error reason.
3. GB32960 and JT808 location history can be queried by vehicle and time range with cursor pagination.
4. GB32960 realtime history can be queried by vehicle and time range.
5. JT808 registration/session data is queryable and can update identity binding.
6. Single-frame detail can reload raw bytes by `rawUri` and decode with the current decoder.
7. ACK-required frames do not receive success ACK before raw archive and raw fact publication succeed.
8. New protocol fields can be added by implementing a decoder/projector and, only when needed, a TDengine table migration.
9. Load tests demonstrate sustained 2,000 frames/s writes and 1,000 QPS bounded historical queries.
10. Old DuckDB/Parquet hot-history paths are no longer required for GB32960/JT808 production queries.
## Open Decisions
The following choices are intentionally fixed for the first implementation:
- Use local raw archive first, with the interface shaped for object storage later.
- Use TDengine as the only production hot historical query store for GB32960/JT808.
- Use explicit protocol-aware query APIs instead of a single generic query API.
- Use cursor pagination for all high-QPS historical queries.
- Keep Kafka payloads byte-light by sending raw references instead of raw bytes.

View File

@@ -0,0 +1,624 @@
# 车辆接入重设计方案
## 背景
当前 GB32960 和 JT808 链路已经能够完成接收、解析、发布和查询,但职责边界被拆散在多条路径里:
- Netty Handler 生成 `RawFrame`Dispatcher 同时发出原始归档事件和标准化车辆事件。
- Kafka Envelope 只携带 raw archive 引用,原始 bytes 由 archive sink 另行存储。
- GB32960 历史查询依赖 raw archive 记录,再回读文件重新解析。
- JT808 位置历史又单独物化到 TDengine。
- DuckDB/Parquet、本地 raw archive、Kafka、TDengine 分别承担一部分职责,导致查询语义和持久化边界不好推理。
新设计保留已有协议解析器和部署经验,但重新定义接入、持久化、存储和查询边界。
## 目标
1. 保存每一条入站协议原始帧,包括可拿到 bytes 的异常帧。
2. 支持按车辆、协议、消息类型、时间范围查询 GB32960 和 JT808 历史上报数据。
3. 新增字段、协议消息、厂商扩展时,不需要重写核心管线。
4. 支撑 10,000 辆车生产写入,以及 1,000 QPS 历史查询。
5. 明确 ACK 和持久化规则,避免“已经 ACK 但数据悄悄丢失”。
## 非目标
- 不把 Kafka 作为长期历史查询数据库。
- 不把大块原始 bytes 直接塞进 Kafka topic。
- 不继续把 DuckDB/Parquet 作为 GB32960/JT808 的生产热查询库。
- 不做一个掩盖协议差异的万能接口。内部可以统一,外部 API 保持协议语义清晰。
## 推荐架构
采用 `Raw Archive + Kafka + TDengine` 作为核心架构。
```mermaid
flowchart LR
GB["GB32960 TCP"] --> Edge["协议接入边界"]
JT["JT808 TCP"] --> Edge
Edge --> RawStore["Raw Archive"]
RawStore --> RawTopic["Kafka vehicle.raw-frame.v1"]
Edge --> Decode["协议解析器"]
Decode --> FactTopic["Kafka vehicle.decoded-fact.v1"]
RawTopic --> Writer["History Writer"]
FactTopic --> Writer
Writer --> TD["TDengine"]
Writer --> Redis["实时状态"]
API["历史 API"] --> TD
API --> RawStore
RealtimeAPI["实时 API"] --> Redis
```
架构里只有三类事实:
- `RawFrameFact`:每一条入站帧一条记录,不管解析成功还是失败。
- `DecodedFact`:从 raw frame 派生出来的零条或多条标准事实。
- `RawArchiveObject`:不可变原始 bytes通过 `raw_uri` 寻址。
raw archive 是排障和回放的事实源。TDengine 是查询事实源。Kafka 是解耦和重放管道。
## 模块边界
### 协议接入 APP
APP
- `gb32960-ingest-app`
- `jt808-ingest-app`
职责:
- 接收 TCP 连接。
- 完成帧切分。
- 执行协议层校验、鉴权、会话绑定和 ACK。
- 创建 `RawFrameFact`
- 通过 `RawArchiveWriter` 持久化原始 bytes。
- 发布 raw frame fact 到 Kafka。
- 解析协议帧并发布 decoded facts。
接入 APP 不负责历史查询、导出,也不直接写业务查询表。
### Ingest Core
新增模块:
- `modules/core/ingest-facts`
核心类型:
```java
public record RawFrameFact(
String frameId,
ProtocolId protocol,
String vehicleKey,
String vin,
String phone,
int messageId,
int subType,
Instant eventTime,
Instant receivedAt,
String peer,
String rawUri,
String checksum,
long rawSizeBytes,
ParseStatus parseStatus,
String parseError,
Map<String, String> metadata
) {}
```
```java
public sealed interface DecodedFact permits
DecodedFact.Location,
DecodedFact.Realtime,
DecodedFact.Alarm,
DecodedFact.Session,
DecodedFact.Register,
DecodedFact.Extension {
String factId();
String frameId();
ProtocolId protocol();
String vehicleKey();
String vin();
Instant eventTime();
Instant receivedAt();
String rawUri();
Map<String, String> metadata();
}
```
`vehicleKey` 是稳定分区键:
- GB32960VIN。
- JT808已解析到 VIN 时使用 VIN否则使用 `jt808:<phone>`
- 未知或异常帧:使用 `unknown:<protocol>:<hash>`
### Raw Archive
新增或重构模块:
- `modules/sinks/raw-archive-store`
接口:
```java
public interface RawArchiveWriter {
RawArchiveReceipt write(RawArchiveWriteRequest request) throws IOException;
}
```
写入回执:
```java
public record RawArchiveReceipt(
String rawUri,
String checksum,
long sizeBytes
) {}
```
规则:
- 原始 bytes 一旦写入,不允许原地修改。
- 写入路径可以按内容寻址,也可以按 `protocol/date/vehicleKey/frameId.bin` 生成。
- 本地存储需要支持 fsync 能力,接口要预留对象存储实现。
- `rawUri` 必须能被历史 API 解析,用于单帧回放。
- 异常帧只要有 bytes也必须写入并记录 `parseStatus=FAILED`
部署约束:
- Raw archive、TDengine、history 服务可以不在同一台 ECS但应在同一 VPC 和同一可用区内通过内网访问。
- 高频历史查询只访问 TDengine不访问 raw archive。
- Raw archive 只进入单帧详情、问题排查、回放和异步导出路径。
- `rawUri` 必须是全局可解析地址,不能是某台 ECS 私有的本地绝对路径。
- 第一版本地 raw archive 需要通过可替换接口封装,后续可以切换到 OSS、MinIO 或其它对象存储。
### Kafka Topic
使用稳定的版本化 topic
- `vehicle.raw-frame.v1`
- `vehicle.decoded-fact.v1`
- `vehicle.dlq.v1`
Kafka key
```text
<protocol>:<vehicleKey>
```
`vehicle.raw-frame.v1` 的 payload 是不含 raw bytes 的 `RawFrameFact`,携带 `rawUri`、checksum、byte size、message id、parse status 和 metadata。
`vehicle.decoded-fact.v1` 的 payload 是单条 `DecodedFact`,必须引用 `frameId``rawUri`
`vehicle.dlq.v1` 保存存储失败、发布失败、解析失败和写入失败事件,并携带足够元数据,尽量支持从 raw archive 重放。
### History Writer
新增或重构 APP
- `vehicle-history-app`
职责:
- 消费 `vehicle.raw-frame.v1`
- 消费 `vehicle.decoded-fact.v1`
- 批量写 TDengine。
- 更新实时状态存储。
- 对非法 payload 或存储失败写 DLQ。
Writer 写入需要幂等:
- raw fact 以 `(frame_id)` 幂等。
- decoded fact 以 `(fact_id)` 幂等。
## TDengine 模型
TDengine 是生产历史查询主库。
### 原始帧超级表
```sql
CREATE STABLE IF NOT EXISTS raw_frames (
ts TIMESTAMP,
frame_id NCHAR(64),
received_at TIMESTAMP,
message_id INT,
sub_type INT,
event_time TIMESTAMP,
raw_uri NCHAR(512),
checksum NCHAR(128),
raw_size_bytes BIGINT,
parse_status NCHAR(16),
parse_error NCHAR(512),
peer NCHAR(128),
metadata_json NCHAR(4096)
) TAGS (
protocol NCHAR(16),
vehicle_key NCHAR(128),
vin NCHAR(64),
phone NCHAR(32)
);
```
子表命名:
```text
raw_<protocol>_<hash(vehicleKey)>
```
主要查询模式:
- 按车辆和时间查询原始帧。
- 按协议、消息 ID、解析状态和时间查询原始帧。
-`frame_id``raw_uri` 查询单帧。
### 位置超级表
```sql
CREATE STABLE IF NOT EXISTS vehicle_locations (
ts TIMESTAMP,
fact_id NCHAR(64),
frame_id NCHAR(64),
received_at TIMESTAMP,
longitude DOUBLE,
latitude DOUBLE,
altitude_m DOUBLE,
speed_kmh DOUBLE,
direction_deg DOUBLE,
alarm_flag BIGINT,
status_flag BIGINT,
total_mileage_km DOUBLE,
raw_uri NCHAR(512),
metadata_json NCHAR(4096)
) TAGS (
protocol NCHAR(16),
vehicle_key NCHAR(128),
vin NCHAR(64),
phone NCHAR(32)
);
```
### 实时数据超级表
```sql
CREATE STABLE IF NOT EXISTS vehicle_realtime (
ts TIMESTAMP,
fact_id NCHAR(64),
frame_id NCHAR(64),
received_at TIMESTAMP,
speed_kmh DOUBLE,
total_mileage_km DOUBLE,
battery_soc DOUBLE,
total_voltage_v DOUBLE,
total_current_a DOUBLE,
running_mode NCHAR(64),
vehicle_state NCHAR(64),
charging_state NCHAR(64),
raw_uri NCHAR(512),
fields_json NCHAR(8192),
metadata_json NCHAR(4096)
) TAGS (
protocol NCHAR(16),
vehicle_key NCHAR(128),
vin NCHAR(64),
phone NCHAR(32)
);
```
`fields_json` 存放稀疏全字段值,用于低频排查。高频查询字段应提升为明确列,或放入专用扩展表。
### 报警超级表
```sql
CREATE STABLE IF NOT EXISTS vehicle_alarms (
ts TIMESTAMP,
fact_id NCHAR(64),
frame_id NCHAR(64),
received_at TIMESTAMP,
alarm_level NCHAR(32),
alarm_code BIGINT,
alarm_name NCHAR(128),
longitude DOUBLE,
latitude DOUBLE,
raw_uri NCHAR(512),
fields_json NCHAR(8192),
metadata_json NCHAR(4096)
) TAGS (
protocol NCHAR(16),
vehicle_key NCHAR(128),
vin NCHAR(64),
phone NCHAR(32)
);
```
### 会话和注册表
```sql
CREATE STABLE IF NOT EXISTS vehicle_sessions (
ts TIMESTAMP,
fact_id NCHAR(64),
frame_id NCHAR(64),
received_at TIMESTAMP,
session_type NCHAR(32),
result_code INT,
raw_uri NCHAR(512),
fields_json NCHAR(8192),
metadata_json NCHAR(4096)
) TAGS (
protocol NCHAR(16),
vehicle_key NCHAR(128),
vin NCHAR(64),
phone NCHAR(32)
);
```
JT808 注册详情先进入 `vehicle_sessions`,后续可以投影到 MySQL 车辆身份绑定表,用于持久身份解析。
## ACK 和持久化规则
对需要协议 ACK 的帧:
1. Netty 接收到 frame bytes。
2. 原始 bytes 写入 raw archive。
3. `RawFrameFact` 发布到 Kafka。
4. 发送协议 ACK。
5. 解码和 `DecodedFact` 发布可以在 ACK 前或 ACK 后执行,但失败必须发 DLQ并更新 raw frame 解析状态。
对 GB32960 实时上报和补发帧,这条规则保证不会 ACK 一条无法恢复原始 bytes 的帧。
对 JT808 注册、鉴权、心跳,也遵循相同 raw archive 和 raw fact 边界。注册 ACK 可以先生成 token但在 durability boundary 成功前不能 flush 到 channel。
如果 raw archive 写入失败,不返回成功 ACK。具体是关闭连接还是保留连接由协议容忍度决定。
如果 Kafka raw fact 发布失败durable ACK 帧不返回成功 ACK。非 durable 帧可以写本地 DLQ。
## 协议解析和扩展
使用插件契约:
```java
public interface ProtocolFrameDecoder {
ProtocolId protocol();
DecodeResult decode(RawFrameFact rawFact, byte[] rawBytes);
}
```
```java
public record DecodeResult(
ParseStatus status,
List<DecodedFact> facts,
String errorMessage,
Map<String, String> metadata
) {}
```
GB32960
- 命令帧映射为会话、实时、位置、报警和扩展事实。
- 厂商扩展按平台账号、VIN 或配置 profile 选择。
- 全量 raw replay 仍通过读取 `rawUri` 并运行当前解析器完成。
JT808
- 消息 ID 映射为注册、鉴权、心跳、位置、批量位置、多媒体元数据和扩展事实。
- 身份解析会更新 `vehicleKey`
- 注册事实携带省、市、厂商、设备 ID、车牌、车牌颜色和 auth token metadata。
新增协议消息通常不需要改 TDengine schema除非它成为高频查询维度。未知字段先作为 extension fact 存入 `fields_json`
## 查询 API
API 保持明确,不做一个含糊的统一接口。
原始帧查询:
- `GET /api/history/raw-frames`
- 过滤条件:`protocol``vin``phone``vehicleKey``messageId``parseStatus``dateFrom``dateTo``pageSize``cursor`
- 返回 frame metadata、raw URI、checksum、parse status以及可选的紧凑解析摘要。
单帧详情:
- `GET /api/history/raw-frames/{frameId}`
- 返回 raw metadata 和解码详情。
- 支持 `includeRawHex=true`,但必须有严格大小限制。
位置历史:
- `GET /api/history/locations`
- 过滤条件:`protocol``vin``phone``vehicleKey`、时间范围、cursor。
实时历史:
- `GET /api/history/realtime`
- 查询历史时序数据。
实时状态:
- `GET /api/realtime/vehicles/{vehicleKey}`
- 从 Redis 或内存 latest state 读取,不扫 TDengine。
导出:
- 导出必须异步。
- 请求创建 export job。
- Worker 将 TDengine 查询结果流式写入对象存储或本地导出文件。
- API 返回 job 状态和下载 URI。
## 分页
高 QPS 历史查询使用 cursor pagination不使用深 offset。
Cursor 字段:
```text
ts, received_at, fact_id
```
倒序查询谓词:
```sql
WHERE (
ts < :cursorTs
OR (ts = :cursorTs AND received_at < :cursorReceivedAt)
OR (ts = :cursorTs AND received_at = :cursorReceivedAt AND fact_id < :cursorFactId)
)
```
默认 page size 为 100上限为 1000。
## 容量设计
假设:
- 10,000 辆车。
- 常规上报间隔 5 到 10 秒。
- 预计写入 1,000 到 2,000 frames/s。
- 峰值突发系数 3 倍。
- 历史查询目标 1,000 requests/s。
写路径:
- Netty worker 保持 CPU-light。
- Raw archive 写入使用有界异步 worker。
- Kafka producer 开启压缩、批量、幂等,并按 vehicle key 分区。
- History writer 按超级表和子表批量写 TDengine。
- TDengine 子表按 vehicle key 分区,使单车查询足够便宜。
读路径:
- 高频查询必须带 vehicle key 和时间范围。
- 无边界跨车查询仅限管理接口,并强制限流和限制返回量。
- 实时 latest state 从 Redis 或内存状态读取。
- Raw 详情只读取单个 raw object 并按需解析。
- 字典和元数据接口使用内存缓存。
背压:
- Raw archive worker queue 固定上限。
- Kafka publish future 必须被跟踪。
- Durability boundary 超时时durable ACK 失败channel 关闭或被限流。
- History writer lag 通过 Kafka consumer lag 和 TDengine batch latency 观察。
## 迁移计划
阶段 1在现有代码旁边建立新的 fact model。
- 新增 `ingest-facts`
- 新增 raw archive receipt contract。
- 新增 raw frame 和 decoded fact 的 Kafka 序列化。
- 为 stable id、vehicle key、raw URI 生成规则加测试。
阶段 2强化 raw durability boundary。
- 重构 GB32960 和 JT808 handler让 ACK 等待 raw archive 和 raw fact publish。
- 保留现有解析器。
- 向新 Kafka topic 发 decoded facts。
阶段 3构建 TDengine writer。
- 创建 TDengine schema manager。
- 写入 raw frame facts。
- 写入 location、realtime、alarm、session facts。
- 按 stable id 实现幂等写入。
阶段 4构建查询 API。
- Raw frame 查询。
- 单帧回放。
- 位置历史查询。
- 实时历史查询。
- 最新实时状态查询。
阶段 5切换并下线旧热路径。
- GB32960/JT808 生产热历史不再依赖 DuckDB/Parquet。
- 历史迁移期间保留 raw replay 兼容。
- TDengine 覆盖验证完成后,移除协议特定的临时历史存储。
## 测试策略
单元测试:
- `RawFrameFact` 校验和 vehicle key 派生。
- Raw archive 路径生成和 checksum。
- GB32960 golden frame 到 decoded facts。
- JT808 sample frame 到 decoded facts。
- ACK boundary 成功和失败场景。
- TDengine SQL 生成。
集成测试:
- 使用 mocked Kafka producer 和 fake raw archive 启动 APP。
- 发送 GB32960 sample frame验证 raw fact、decoded fact 和 raw archive receipt。
- 发送 JT808 注册、鉴权、位置帧,验证 registration、session、location facts。
- 模拟 raw archive 失败,验证不发送成功 ACK。
- 模拟 Kafka publish 失败,验证 durable ACK 失败。
存储测试:
- TDengine schema 初始化。
- 批量写入 raw frames 和 decoded facts。
- 按 vehicle key 和时间范围查询。
- Cursor 分页正确性。
-`rawUri` 单帧回放。
压测:
- 持续 30 分钟 2,000 frames/s 写入。
- 持续 5 分钟 6,000 frames/s 突发写入。
- 1,000 QPS location/realtime 历史查询p95 延迟受控。
- 校验 raw archive receipt、Kafka raw fact、TDengine raw frame row 三者数量无缺口。
## 运维要求
指标:
- TCP 连接数。
- Frame 接收速率。
- Raw archive 写入延迟和失败数。
- Kafka 发布延迟和失败数。
- 按协议和 message id 统计 decoder 成功/失败数。
- TDengine batch size、延迟和失败数。
- 各查询 endpoint 的 QPS 和延迟。
- Kafka consumer lag。
日志:
- 每个失败帧输出一条结构化日志,包含 `frameId`、protocol、vehicle key、message id、raw URI 和 error。
- 正常日志不输出完整 raw bytes。
- Raw hex 只允许在显式 debug 工具里输出,并限制大小。
告警:
- Raw archive 写入失败。
- Durable ACK 超时。
- Kafka producer 失败。
- History writer lag。
- TDengine 写入失败。
- Raw archive 数量和 TDengine raw frame row 数量不一致。
## 验收标准
重设计只有满足以下条件才算完成:
1. 每一条被接收的 GB32960 和 JT808 帧都会创建 raw archive object 和 TDengine `raw_frames` 行。
2. 有 bytes 的异常帧可以从 raw history 查询到,包含 parse status 和错误原因。
3. GB32960 和 JT808 位置历史可以按车辆和时间范围 cursor 分页查询。
4. GB32960 实时历史可以按车辆和时间范围查询。
5. JT808 注册和会话数据可查询,并且可以更新 identity binding。
6. 单帧详情可以通过 `rawUri` 重新读取 raw bytes并使用当前解析器解码。
7. 需要 ACK 的帧,在 raw archive 和 raw fact publish 成功前,不会发送成功 ACK。
8. 新协议字段可以通过新增 decoder/projector 扩展;只有成为高频查询字段时才需要 TDengine 表迁移。
9. 压测证明可以持续 2,000 frames/s 写入,并支持 1,000 QPS 有边界历史查询。
10. GB32960/JT808 生产查询不再依赖旧 DuckDB/Parquet 热历史路径。
## 已固定决策
第一版实现固定以下选择:
- 先使用本地 raw archive实现接口时预留对象存储。
- TDengine 是 GB32960/JT808 唯一生产热历史查询库。
- 使用协议明确的查询 API不做一个泛化万能查询 API。
- 高频历史查询统一使用 cursor pagination。
- Kafka payload 保持轻量,只传 raw 引用,不传 raw bytes。

View File

@@ -33,6 +33,10 @@
<groupId>com.lingniu.ingest</groupId>
<artifactId>sink-mq</artifactId>
</dependency>
<dependency>
<groupId>com.lingniu.ingest</groupId>
<artifactId>sink-archive</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-web</artifactId>

View File

@@ -102,6 +102,10 @@ lingniu:
dlq: ${KAFKA_TOPIC_GB32960_DLQ:vehicle.dlq.gb32960.v1}
consumer:
enabled: false
archive:
enabled: ${SINK_ARCHIVE_ENABLED:true}
type: local
path: ${SINK_ARCHIVE_PATH:./archive/}
event-file-store:
enabled: false
event-history:

View File

@@ -5,6 +5,9 @@ import com.lingniu.ingest.protocol.gb32960.codec.Gb32960MessageDecoder;
import com.lingniu.ingest.protocol.gb32960.config.Gb32960AutoConfiguration;
import com.lingniu.ingest.protocol.gb32960.inbound.Gb32960NettyServer;
import com.lingniu.ingest.session.config.SessionCoreAutoConfiguration;
import com.lingniu.ingest.sink.archive.ArchiveStore;
import com.lingniu.ingest.sink.archive.RawArchiveEventSink;
import com.lingniu.ingest.sink.archive.config.SinkArchiveAutoConfiguration;
import com.lingniu.ingest.sink.mq.KafkaEventSink;
import com.lingniu.ingest.sink.mq.KafkaEnvelopeDeadLetterSink;
import com.lingniu.ingest.sink.mq.SinkMqAutoConfiguration;
@@ -25,6 +28,7 @@ class Gb32960IngestAppCompositionTest {
IngestCoreAutoConfiguration.class,
SessionCoreAutoConfiguration.class,
SinkMqAutoConfiguration.class,
SinkArchiveAutoConfiguration.class,
Gb32960AutoConfiguration.class))
.withAllowBeanDefinitionOverriding(true)
.withBean("kafkaProducer", KafkaProducer.class, Gb32960IngestAppCompositionTest::kafkaProducer)
@@ -37,6 +41,8 @@ class Gb32960IngestAppCompositionTest {
"lingniu.ingest.sink.mq.type=kafka",
"lingniu.ingest.sink.mq.bootstrap-servers=localhost:9092",
"lingniu.ingest.sink.mq.consumer.enabled=false",
"lingniu.ingest.sink.archive.enabled=true",
"lingniu.ingest.sink.archive.path=target/test-archive-gb32960",
"lingniu.ingest.event-file-store.enabled=false",
"lingniu.ingest.event-history.enabled=false",
"lingniu.ingest.vehicle-state.enabled=false",
@@ -49,8 +55,9 @@ class Gb32960IngestAppCompositionTest {
assertThat(context).hasSingleBean(Gb32960NettyServer.class);
assertThat(context).hasSingleBean(KafkaEventSink.class);
assertThat(context).hasSingleBean(KafkaEnvelopeDeadLetterSink.class);
assertThat(context).hasSingleBean(ArchiveStore.class);
assertThat(context).hasSingleBean(RawArchiveEventSink.class);
assertTypeNotPresent(context, "com.lingniu.ingest.sink.archive.ArchiveStore");
assertTypeNotPresent(context, "com.lingniu.ingest.eventfilestore.EventFileStore");
});
}

View File

@@ -25,6 +25,8 @@ class Gb32960IngestAppDefaultsTest {
.containsEntry("lingniu.ingest.gb32960.port", "${GB32960_PORT:32960}")
.containsEntry("lingniu.ingest.sink.mq.enabled", "${KAFKA_ENABLED:true}")
.containsEntry("lingniu.ingest.sink.mq.consumer.enabled", false)
.containsEntry("lingniu.ingest.sink.archive.enabled", "${SINK_ARCHIVE_ENABLED:true}")
.containsEntry("lingniu.ingest.sink.archive.path", "${SINK_ARCHIVE_PATH:./archive/}")
.containsEntry("lingniu.ingest.event-file-store.enabled", false)
.containsEntry("lingniu.ingest.event-history.enabled", false)
.containsEntry("lingniu.ingest.vehicle-state.enabled", false)

View File

@@ -37,6 +37,10 @@
<groupId>com.lingniu.ingest</groupId>
<artifactId>sink-mq</artifactId>
</dependency>
<dependency>
<groupId>com.lingniu.ingest</groupId>
<artifactId>sink-archive</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-web</artifactId>

View File

@@ -31,7 +31,7 @@ lingniu:
ingest:
jt808:
enabled: true
port: ${JT808_PORT:10808}
port: ${JT808_PORT:808}
worker-threads: ${JT808_WORKER_THREADS:0}
pipeline:
disruptor:
@@ -72,6 +72,10 @@ lingniu:
dlq: ${KAFKA_TOPIC_JT808_DLQ:vehicle.dlq.jt808.v1}
consumer:
enabled: false
archive:
enabled: ${SINK_ARCHIVE_ENABLED:true}
type: local
path: ${SINK_ARCHIVE_PATH:./archive/}
event-file-store:
enabled: false
event-history:

View File

@@ -6,6 +6,9 @@ import com.lingniu.ingest.protocol.jt808.codec.Jt808MessageDecoder;
import com.lingniu.ingest.protocol.jt808.config.Jt808AutoConfiguration;
import com.lingniu.ingest.protocol.jt808.inbound.Jt808NettyServer;
import com.lingniu.ingest.session.config.SessionCoreAutoConfiguration;
import com.lingniu.ingest.sink.archive.ArchiveStore;
import com.lingniu.ingest.sink.archive.RawArchiveEventSink;
import com.lingniu.ingest.sink.archive.config.SinkArchiveAutoConfiguration;
import com.lingniu.ingest.sink.mq.KafkaEnvelopeDeadLetterSink;
import com.lingniu.ingest.sink.mq.KafkaEventSink;
import com.lingniu.ingest.sink.mq.SinkMqAutoConfiguration;
@@ -27,6 +30,7 @@ class Jt808IngestAppCompositionTest {
SessionCoreAutoConfiguration.class,
VehicleIdentityAutoConfiguration.class,
SinkMqAutoConfiguration.class,
SinkArchiveAutoConfiguration.class,
Jt808AutoConfiguration.class))
.withAllowBeanDefinitionOverriding(true)
.withBean("kafkaProducer", KafkaProducer.class, Jt808IngestAppCompositionTest::kafkaProducer)
@@ -38,6 +42,8 @@ class Jt808IngestAppCompositionTest {
"lingniu.ingest.sink.mq.type=kafka",
"lingniu.ingest.sink.mq.bootstrap-servers=localhost:9092",
"lingniu.ingest.sink.mq.consumer.enabled=false",
"lingniu.ingest.sink.archive.enabled=true",
"lingniu.ingest.sink.archive.path=target/test-archive-jt808",
"lingniu.ingest.event-file-store.enabled=false",
"lingniu.ingest.event-history.enabled=false",
"lingniu.ingest.vehicle-state.enabled=false",
@@ -50,8 +56,9 @@ class Jt808IngestAppCompositionTest {
assertThat(context).hasSingleBean(Jt808NettyServer.class);
assertThat(context).hasSingleBean(KafkaEventSink.class);
assertThat(context).hasSingleBean(KafkaEnvelopeDeadLetterSink.class);
assertThat(context).hasSingleBean(ArchiveStore.class);
assertThat(context).hasSingleBean(RawArchiveEventSink.class);
assertTypeNotPresent(context, "com.lingniu.ingest.sink.archive.ArchiveStore");
assertTypeNotPresent(context, "com.lingniu.ingest.eventfilestore.EventFileStore");
});
}

View File

@@ -22,12 +22,14 @@ class Jt808IngestAppDefaultsTest {
.containsEntry("spring.cloud.nacos.config.server-addr", "${NACOS_SERVER_ADDR:127.0.0.1:8848}")
.containsEntry("server.port", "${HTTP_PORT:20400}")
.containsEntry("lingniu.ingest.jt808.enabled", true)
.containsEntry("lingniu.ingest.jt808.port", "${JT808_PORT:10808}")
.containsEntry("lingniu.ingest.jt808.port", "${JT808_PORT:808}")
.containsEntry("lingniu.ingest.sink.mq.enabled", "${KAFKA_ENABLED:true}")
.containsEntry("lingniu.ingest.sink.mq.consumer.enabled", false)
.containsEntry("lingniu.ingest.sink.mq.topics.realtime", "${KAFKA_TOPIC_JT808_EVENT:vehicle.event.jt808.v1}")
.containsEntry("lingniu.ingest.sink.mq.topics.raw-archive", "${KAFKA_TOPIC_JT808_RAW:vehicle.raw.jt808.v1}")
.containsEntry("lingniu.ingest.sink.mq.topics.dlq", "${KAFKA_TOPIC_JT808_DLQ:vehicle.dlq.jt808.v1}")
.containsEntry("lingniu.ingest.sink.archive.enabled", "${SINK_ARCHIVE_ENABLED:true}")
.containsEntry("lingniu.ingest.sink.archive.path", "${SINK_ARCHIVE_PATH:./archive/}")
.containsEntry("lingniu.ingest.event-file-store.enabled", false)
.containsEntry("lingniu.ingest.event-history.enabled", false)
.containsEntry("lingniu.ingest.vehicle-state.enabled", false)

View File

@@ -33,6 +33,10 @@
<groupId>com.lingniu.ingest</groupId>
<artifactId>event-file-store</artifactId>
</dependency>
<dependency>
<groupId>com.lingniu.ingest</groupId>
<artifactId>tdengine-history-store</artifactId>
</dependency>
<dependency>
<groupId>com.lingniu.ingest</groupId>
<artifactId>event-history-service</artifactId>

View File

@@ -0,0 +1,76 @@
package com.lingniu.ingest.historyapp;
import com.lingniu.ingest.api.consumer.EnvelopeConsumerProcessor;
import com.lingniu.ingest.api.consumer.EnvelopeDeadLetterSink;
import com.lingniu.ingest.eventfilestore.EventFileStore;
import com.lingniu.ingest.eventhistory.EventHistoryEnvelopeIngestor;
import com.lingniu.ingest.eventhistory.TelemetryEnvelopeRecordMapper;
import com.lingniu.ingest.sink.mq.KafkaEnvelopeConsumerFactory;
import com.lingniu.ingest.sink.mq.KafkaEnvelopeConsumerRunner;
import com.lingniu.ingest.sink.mq.KafkaEnvelopeConsumerWorker;
import com.lingniu.ingest.sink.mq.SinkMqProperties;
import com.lingniu.ingest.tdenginehistory.TdengineHistoryWriter;
import org.springframework.beans.factory.annotation.Qualifier;
import org.springframework.beans.factory.ObjectProvider;
import org.springframework.boot.autoconfigure.condition.ConditionalOnMissingBean;
import org.springframework.boot.autoconfigure.condition.ConditionalOnProperty;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import java.time.Duration;
import java.util.List;
import java.util.Map;
@Configuration(proxyBeanMethods = false)
public class VehicleHistoryKafkaConsumerConfiguration {
@Bean
@ConditionalOnMissingBean
public TelemetryEnvelopeRecordMapper telemetryEnvelopeRecordMapper() {
return new TelemetryEnvelopeRecordMapper();
}
@Bean
@ConditionalOnMissingBean
public EventHistoryEnvelopeIngestor eventHistoryEnvelopeIngestor(EventFileStore store,
TelemetryEnvelopeRecordMapper mapper,
ObjectProvider<TdengineHistoryWriter> writer) {
return new EventHistoryEnvelopeIngestor(store, mapper, writer.getIfAvailable());
}
@Bean
@ConditionalOnMissingBean(name = "eventHistoryEnvelopeConsumerProcessor")
public EnvelopeConsumerProcessor eventHistoryEnvelopeConsumerProcessor(EventHistoryEnvelopeIngestor ingestor,
EnvelopeDeadLetterSink deadLetterSink) {
return new EnvelopeConsumerProcessor("event-history", ingestor, deadLetterSink);
}
@Bean
@ConditionalOnMissingBean(name = "eventHistoryRawEnvelopeConsumerProcessor")
public EnvelopeConsumerProcessor eventHistoryRawEnvelopeConsumerProcessor(EventHistoryEnvelopeIngestor ingestor,
EnvelopeDeadLetterSink deadLetterSink) {
return new EnvelopeConsumerProcessor("event-history-raw", ingestor, deadLetterSink);
}
@Bean
@ConditionalOnMissingBean
@ConditionalOnProperty(prefix = "lingniu.ingest.sink.mq.consumer", name = "enabled", havingValue = "true")
public KafkaEnvelopeConsumerRunner vehicleHistoryKafkaEnvelopeConsumerRunner(
@Qualifier("eventHistoryEnvelopeConsumerProcessor") EnvelopeConsumerProcessor processor,
@Qualifier("eventHistoryRawEnvelopeConsumerProcessor") EnvelopeConsumerProcessor rawProcessor,
SinkMqProperties props) {
List<KafkaEnvelopeConsumerWorker> workers = new KafkaEnvelopeConsumerFactory().createWorkers(
Map.of(
"eventHistoryEnvelopeConsumerProcessor", processor,
"eventHistoryRawEnvelopeConsumerProcessor", rawProcessor),
props);
if (workers.isEmpty()) {
throw new IllegalStateException("no vehicle history kafka consumer workers created; check consumer bindings");
}
return new KafkaEnvelopeConsumerRunner(
workers,
Duration.ofMillis(props.getConsumer().getPollTimeoutMillis()),
Duration.ofMillis(props.getConsumer().getLoopBackoffMillis()),
props.getConsumer().isAutoStartup());
}
}

View File

@@ -46,14 +46,21 @@ lingniu:
enabled: ${KAFKA_CONSUMER_ENABLED:true}
client-id-prefix: ${KAFKA_CONSUMER_CLIENT_ID_PREFIX:vehicle-history}
auto-offset-reset: ${KAFKA_CONSUMER_AUTO_OFFSET_RESET:earliest}
max-poll-records: ${KAFKA_CONSUMER_MAX_POLL_RECORDS:500}
max-poll-records: ${KAFKA_CONSUMER_MAX_POLL_RECORDS:100}
max-poll-interval-millis: ${KAFKA_CONSUMER_MAX_POLL_INTERVAL_MS:1800000}
bindings:
eventHistoryEnvelopeConsumerProcessor:
enabled: true
group-id: ${KAFKA_GROUP_HISTORY:vehicle-history}
group-id: ${KAFKA_GROUP_HISTORY_EVENT:${KAFKA_GROUP_HISTORY:vehicle-history}-event}
topics:
- ${KAFKA_TOPIC_GB32960_EVENT:vehicle.event.gb32960.v1}
- ${KAFKA_TOPIC_JT808_EVENT:vehicle.event.jt808.v1}
eventHistoryRawEnvelopeConsumerProcessor:
enabled: true
group-id: ${KAFKA_GROUP_HISTORY_RAW:${KAFKA_GROUP_HISTORY:vehicle-history}}
topics:
- ${KAFKA_TOPIC_GB32960_RAW:vehicle.raw.gb32960.v1}
- ${KAFKA_TOPIC_JT808_RAW:vehicle.raw.jt808.v1}
archive:
enabled: ${SINK_ARCHIVE_ENABLED:true}
type: local
@@ -64,6 +71,17 @@ lingniu:
zone-id: ${EVENT_FILE_STORE_ZONE_ID:Asia/Shanghai}
batch-size: ${EVENT_FILE_STORE_BATCH_SIZE:500}
flush-interval-millis: ${EVENT_FILE_STORE_FLUSH_INTERVAL_MILLIS:1000}
tdengine-history:
enabled: ${TDENGINE_HISTORY_ENABLED:false}
database: ${TDENGINE_HISTORY_DATABASE:vehicle_history}
jdbc-url: ${TDENGINE_JDBC_URL:jdbc:TAOS-RS://${TDENGINE_HOST:127.0.0.1}:${TDENGINE_PORT:6041}/${TDENGINE_HISTORY_DATABASE:vehicle_history}}
username: ${TDENGINE_USERNAME:root}
password: ${TDENGINE_PASSWORD:taosdata}
driver-class-name: ${TDENGINE_DRIVER_CLASS_NAME:com.taosdata.jdbc.rs.RestfulDriver}
maximum-pool-size: ${TDENGINE_MAX_POOL_SIZE:16}
minimum-idle: ${TDENGINE_MIN_IDLE:0}
connection-timeout-millis: ${TDENGINE_CONNECTION_TIMEOUT_MILLIS:5000}
initialization-fail-timeout-millis: ${TDENGINE_INITIALIZATION_FAIL_TIMEOUT_MILLIS:-1}
event-history:
enabled: true
vehicle-state:

View File

@@ -8,6 +8,8 @@ import com.lingniu.ingest.eventhistory.EventHistoryEnvelopeIngestor;
import com.lingniu.ingest.eventhistory.Gb32960DecodedFrameService;
import com.lingniu.ingest.eventhistory.Gb32960FrameController;
import com.lingniu.ingest.eventhistory.config.EventHistoryAutoConfiguration;
import com.lingniu.ingest.eventhistory.Jt808LocationHistoryController;
import com.lingniu.ingest.eventhistory.TelemetryFieldHistoryController;
import com.lingniu.ingest.protocol.gb32960.codec.Gb32960MessageDecoder;
import com.lingniu.ingest.protocol.gb32960.config.Gb32960AutoConfiguration;
import com.lingniu.ingest.protocol.gb32960.inbound.Gb32960NettyServer;
@@ -17,6 +19,8 @@ import com.lingniu.ingest.sink.archive.config.SinkArchiveAutoConfiguration;
import com.lingniu.ingest.sink.mq.KafkaEnvelopeDeadLetterSink;
import com.lingniu.ingest.sink.mq.KafkaEventSink;
import com.lingniu.ingest.sink.mq.SinkMqAutoConfiguration;
import com.lingniu.ingest.tdenginehistory.TdengineHistorySchema;
import com.lingniu.ingest.tdenginehistory.config.TdengineHistoryAutoConfiguration;
import org.apache.kafka.clients.producer.KafkaProducer;
import org.junit.jupiter.api.Test;
import org.junit.jupiter.api.io.TempDir;
@@ -39,6 +43,7 @@ class VehicleHistoryAppCompositionTest {
.withConfiguration(AutoConfigurations.of(
SinkArchiveAutoConfiguration.class,
EventFileStoreAutoConfiguration.class,
TdengineHistoryAutoConfiguration.class,
SinkMqAutoConfiguration.class,
Gb32960AutoConfiguration.class,
EventHistoryAutoConfiguration.class))
@@ -50,6 +55,8 @@ class VehicleHistoryAppCompositionTest {
"lingniu.ingest.sink.archive.path=" + tempDir.resolve("archive"),
"lingniu.ingest.event-file-store.enabled=true",
"lingniu.ingest.event-file-store.path=" + tempDir.resolve("event-store"),
"lingniu.ingest.tdengine-history.enabled=true",
"lingniu.ingest.tdengine-history.database=vehicle_history_test",
"lingniu.ingest.event-history.enabled=true",
"lingniu.ingest.gb32960.enabled=true",
"lingniu.ingest.gb32960.server.enabled=false",
@@ -64,6 +71,7 @@ class VehicleHistoryAppCompositionTest {
assertThat(context).hasSingleBean(RawArchiveEventSink.class);
assertThat(context).hasSingleBean(EventFileStore.class);
assertThat(context).hasSingleBean(EventFileStoreSink.class);
assertThat(context).hasSingleBean(TdengineHistorySchema.class);
assertThat(context).hasSingleBean(KafkaEventSink.class);
assertThat(context).hasSingleBean(KafkaEnvelopeDeadLetterSink.class);
@@ -72,6 +80,8 @@ class VehicleHistoryAppCompositionTest {
assertThat(context).hasSingleBean(Gb32960MessageDecoder.class);
assertThat(context).hasSingleBean(Gb32960DecodedFrameService.class);
assertThat(context).hasSingleBean(Gb32960FrameController.class);
assertThat(context).hasSingleBean(Jt808LocationHistoryController.class);
assertThat(context).hasSingleBean(TelemetryFieldHistoryController.class);
assertThat(context).doesNotHaveBean(Gb32960NettyServer.class);
});
}

View File

@@ -24,22 +24,50 @@ class VehicleHistoryAppDefaultsTest {
.containsEntry("lingniu.ingest.gb32960.server.enabled", false)
.containsEntry("lingniu.ingest.sink.archive.enabled", "${SINK_ARCHIVE_ENABLED:true}")
.containsEntry("lingniu.ingest.event-file-store.enabled", "${EVENT_FILE_STORE_ENABLED:true}")
.containsEntry("lingniu.ingest.tdengine-history.enabled", "${TDENGINE_HISTORY_ENABLED:false}")
.containsEntry("lingniu.ingest.tdengine-history.database", "${TDENGINE_HISTORY_DATABASE:vehicle_history}")
.containsEntry(
"lingniu.ingest.tdengine-history.jdbc-url",
"${TDENGINE_JDBC_URL:jdbc:TAOS-RS://${TDENGINE_HOST:127.0.0.1}:${TDENGINE_PORT:6041}/${TDENGINE_HISTORY_DATABASE:vehicle_history}}")
.containsEntry("lingniu.ingest.tdengine-history.username", "${TDENGINE_USERNAME:root}")
.containsEntry("lingniu.ingest.tdengine-history.password", "${TDENGINE_PASSWORD:taosdata}")
.containsEntry(
"lingniu.ingest.tdengine-history.driver-class-name",
"${TDENGINE_DRIVER_CLASS_NAME:com.taosdata.jdbc.rs.RestfulDriver}")
.containsEntry("lingniu.ingest.tdengine-history.maximum-pool-size", "${TDENGINE_MAX_POOL_SIZE:16}")
.containsEntry("lingniu.ingest.tdengine-history.minimum-idle", "${TDENGINE_MIN_IDLE:0}")
.containsEntry("lingniu.ingest.event-history.enabled", true)
.containsEntry("lingniu.ingest.vehicle-state.enabled", false)
.containsEntry("lingniu.ingest.vehicle-stat.enabled", false)
.containsEntry("lingniu.ingest.sink.mq.consumer.enabled", "${KAFKA_CONSUMER_ENABLED:true}")
.containsEntry("lingniu.ingest.sink.mq.consumer.max-poll-records", "${KAFKA_CONSUMER_MAX_POLL_RECORDS:100}")
.containsEntry(
"lingniu.ingest.sink.mq.consumer.max-poll-interval-millis",
"${KAFKA_CONSUMER_MAX_POLL_INTERVAL_MS:1800000}")
.containsEntry(
"lingniu.ingest.sink.mq.consumer.bindings.eventHistoryEnvelopeConsumerProcessor.enabled",
true)
.containsEntry(
"lingniu.ingest.sink.mq.consumer.bindings.eventHistoryEnvelopeConsumerProcessor.group-id",
"${KAFKA_GROUP_HISTORY:vehicle-history}")
"${KAFKA_GROUP_HISTORY_EVENT:${KAFKA_GROUP_HISTORY:vehicle-history}-event}")
.containsEntry(
"lingniu.ingest.sink.mq.consumer.bindings.eventHistoryEnvelopeConsumerProcessor.topics[0]",
"${KAFKA_TOPIC_GB32960_EVENT:vehicle.event.gb32960.v1}")
.containsEntry(
"lingniu.ingest.sink.mq.consumer.bindings.eventHistoryEnvelopeConsumerProcessor.topics[1]",
"${KAFKA_TOPIC_GB32960_RAW:vehicle.raw.gb32960.v1}");
"${KAFKA_TOPIC_JT808_EVENT:vehicle.event.jt808.v1}")
.containsEntry(
"lingniu.ingest.sink.mq.consumer.bindings.eventHistoryRawEnvelopeConsumerProcessor.enabled",
true)
.containsEntry(
"lingniu.ingest.sink.mq.consumer.bindings.eventHistoryRawEnvelopeConsumerProcessor.group-id",
"${KAFKA_GROUP_HISTORY_RAW:${KAFKA_GROUP_HISTORY:vehicle-history}}")
.containsEntry(
"lingniu.ingest.sink.mq.consumer.bindings.eventHistoryRawEnvelopeConsumerProcessor.topics[0]",
"${KAFKA_TOPIC_GB32960_RAW:vehicle.raw.gb32960.v1}")
.containsEntry(
"lingniu.ingest.sink.mq.consumer.bindings.eventHistoryRawEnvelopeConsumerProcessor.topics[1]",
"${KAFKA_TOPIC_JT808_RAW:vehicle.raw.jt808.v1}");
}
private static Properties applicationProperties() {

View File

@@ -1,7 +1,9 @@
package com.lingniu.ingest.api.consumer;
import java.time.Instant;
import java.util.ArrayList;
import java.util.EnumSet;
import java.util.List;
import java.util.Set;
public final class EnvelopeConsumerProcessor {
@@ -32,13 +34,40 @@ public final class EnvelopeConsumerProcessor {
if (record == null) {
throw new IllegalArgumentException("record must not be null");
}
EnvelopeIngestResult result = ingestor.tryIngest(record.payload());
EnvelopeIngestResult result = processAll(List.of(record)).getFirst();
return result;
}
public List<EnvelopeIngestResult> processAll(List<EnvelopeConsumerRecord> records) {
if (records == null) {
throw new IllegalArgumentException("records must not be null");
}
if (records.isEmpty()) {
return List.of();
}
List<byte[]> payloads = new ArrayList<>(records.size());
for (EnvelopeConsumerRecord record : records) {
if (record == null) {
throw new IllegalArgumentException("record must not be null");
}
payloads.add(record.payload());
}
List<EnvelopeIngestResult> results = ingestor.tryIngestAll(payloads);
if (results.size() != records.size()) {
throw new IllegalStateException("ingestor result count does not match record count");
}
for (int i = 0; i < records.size(); i++) {
publishDeadLetterIfNeeded(records.get(i), results.get(i));
}
return results;
}
private void publishDeadLetterIfNeeded(EnvelopeConsumerRecord record, EnvelopeIngestResult result) {
// 派生消费者不在这里抛出业务异常给 Kafka worker坏消息统一进入 DLQ
// worker 看到 process 正常返回后才能提交 offset避免同一坏消息无限阻塞消费组。
if (DEAD_LETTER_STATUSES.contains(result.status())) {
deadLetterSink.publish(toDeadLetter(record, result));
}
return result;
}
private EnvelopeDeadLetterRecord toDeadLetter(EnvelopeConsumerRecord record, EnvelopeIngestResult result) {

View File

@@ -1,6 +1,20 @@
package com.lingniu.ingest.api.consumer;
import java.util.ArrayList;
import java.util.List;
@FunctionalInterface
public interface EnvelopeIngestor {
EnvelopeIngestResult tryIngest(byte[] kafkaValue);
default List<EnvelopeIngestResult> tryIngestAll(List<byte[]> kafkaValues) {
if (kafkaValues == null || kafkaValues.isEmpty()) {
return List.of();
}
List<EnvelopeIngestResult> results = new ArrayList<>(kafkaValues.size());
for (byte[] kafkaValue : kafkaValues) {
results.add(tryIngest(kafkaValue));
}
return results;
}
}

View File

@@ -0,0 +1,30 @@
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0">
<modelVersion>4.0.0</modelVersion>
<parent>
<groupId>com.lingniu.ingest</groupId>
<artifactId>lingniu-vehicle-ingest</artifactId>
<version>0.1.0-SNAPSHOT</version>
<relativePath>../../../pom.xml</relativePath>
</parent>
<artifactId>ingest-facts</artifactId>
<name>ingest-facts</name>
<description>Protocol-neutral raw frame and decoded fact model for vehicle ingestion.</description>
<dependencies>
<dependency>
<groupId>com.lingniu.ingest</groupId>
<artifactId>ingest-api</artifactId>
</dependency>
<dependency>
<groupId>org.junit.jupiter</groupId>
<artifactId>junit-jupiter</artifactId>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.assertj</groupId>
<artifactId>assertj-core</artifactId>
<scope>test</scope>
</dependency>
</dependencies>
</project>

View File

@@ -0,0 +1,215 @@
package com.lingniu.ingest.facts;
import com.lingniu.ingest.api.ProtocolId;
import java.time.Instant;
import java.util.Map;
public sealed interface DecodedFact
permits DecodedFact.Location,
DecodedFact.Realtime,
DecodedFact.Alarm,
DecodedFact.Session,
DecodedFact.Register,
DecodedFact.Extension {
String factId();
String frameId();
ProtocolId protocol();
String vehicleKey();
String vin();
String phone();
Instant eventTime();
Instant receivedAt();
String rawUri();
Map<String, String> metadata();
record Location(
String factId,
String frameId,
ProtocolId protocol,
String vehicleKey,
String vin,
String phone,
Instant eventTime,
Instant receivedAt,
String rawUri,
Map<String, String> metadata,
Double longitude,
Double latitude,
Double altitudeM,
Double speedKmh,
Double directionDeg,
Long alarmFlag,
Long statusFlag,
Double totalMileageKm
) implements DecodedFact {
public Location {
BaseFields.validate(factId, frameId, protocol, vehicleKey, eventTime, receivedAt, rawUri);
vin = BaseFields.clean(vin);
phone = BaseFields.clean(phone);
metadata = BaseFields.metadata(metadata);
}
}
record Realtime(
String factId,
String frameId,
ProtocolId protocol,
String vehicleKey,
String vin,
String phone,
Instant eventTime,
Instant receivedAt,
String rawUri,
Map<String, String> metadata,
Map<String, String> fields
) implements DecodedFact {
public Realtime {
BaseFields.validate(factId, frameId, protocol, vehicleKey, eventTime, receivedAt, rawUri);
vin = BaseFields.clean(vin);
phone = BaseFields.clean(phone);
metadata = BaseFields.metadata(metadata);
fields = fields == null ? Map.of() : Map.copyOf(fields);
}
}
record Alarm(
String factId,
String frameId,
ProtocolId protocol,
String vehicleKey,
String vin,
String phone,
Instant eventTime,
Instant receivedAt,
String rawUri,
Map<String, String> metadata,
String alarmLevel,
Long alarmCode,
String alarmName,
Map<String, String> fields
) implements DecodedFact {
public Alarm {
BaseFields.validate(factId, frameId, protocol, vehicleKey, eventTime, receivedAt, rawUri);
vin = BaseFields.clean(vin);
phone = BaseFields.clean(phone);
alarmLevel = BaseFields.clean(alarmLevel);
alarmName = BaseFields.clean(alarmName);
metadata = BaseFields.metadata(metadata);
fields = fields == null ? Map.of() : Map.copyOf(fields);
}
}
record Session(
String factId,
String frameId,
ProtocolId protocol,
String vehicleKey,
String vin,
String phone,
Instant eventTime,
Instant receivedAt,
String rawUri,
Map<String, String> metadata,
String sessionType,
Integer resultCode,
Map<String, String> fields
) implements DecodedFact {
public Session {
BaseFields.validate(factId, frameId, protocol, vehicleKey, eventTime, receivedAt, rawUri);
vin = BaseFields.clean(vin);
phone = BaseFields.clean(phone);
sessionType = BaseFields.clean(sessionType);
metadata = BaseFields.metadata(metadata);
fields = fields == null ? Map.of() : Map.copyOf(fields);
}
}
record Register(
String factId,
String frameId,
ProtocolId protocol,
String vehicleKey,
String vin,
String phone,
Instant eventTime,
Instant receivedAt,
String rawUri,
Map<String, String> metadata,
Map<String, String> registrationFields
) implements DecodedFact {
public Register {
BaseFields.validate(factId, frameId, protocol, vehicleKey, eventTime, receivedAt, rawUri);
vin = BaseFields.clean(vin);
phone = BaseFields.clean(phone);
metadata = BaseFields.metadata(metadata);
registrationFields = registrationFields == null ? Map.of() : Map.copyOf(registrationFields);
}
}
record Extension(
String factId,
String frameId,
ProtocolId protocol,
String vehicleKey,
String vin,
String phone,
Instant eventTime,
Instant receivedAt,
String rawUri,
Map<String, String> metadata,
String extensionType,
Map<String, String> fields
) implements DecodedFact {
public Extension {
BaseFields.validate(factId, frameId, protocol, vehicleKey, eventTime, receivedAt, rawUri);
vin = BaseFields.clean(vin);
phone = BaseFields.clean(phone);
extensionType = BaseFields.clean(extensionType);
metadata = BaseFields.metadata(metadata);
fields = fields == null ? Map.of() : Map.copyOf(fields);
}
}
final class BaseFields {
private BaseFields() {
}
static void validate(String factId,
String frameId,
ProtocolId protocol,
String vehicleKey,
Instant eventTime,
Instant receivedAt,
String rawUri) {
required(factId, "factId");
required(frameId, "frameId");
if (protocol == null) {
throw new IllegalArgumentException("protocol must not be null");
}
required(vehicleKey, "vehicleKey");
if (eventTime == null) {
throw new IllegalArgumentException("eventTime must not be null");
}
if (receivedAt == null) {
throw new IllegalArgumentException("receivedAt must not be null");
}
required(rawUri, "rawUri");
}
static String clean(String value) {
return value == null ? "" : value.trim();
}
static Map<String, String> metadata(Map<String, String> metadata) {
return metadata == null ? Map.of() : Map.copyOf(metadata);
}
private static void required(String value, String field) {
if (clean(value).isBlank()) {
throw new IllegalArgumentException(field + " must not be blank");
}
}
}
}

View File

@@ -0,0 +1,50 @@
package com.lingniu.ingest.facts;
import com.lingniu.ingest.api.ProtocolId;
import java.nio.charset.StandardCharsets;
import java.security.MessageDigest;
import java.security.NoSuchAlgorithmException;
import java.time.Instant;
import java.util.HexFormat;
public final class FactIds {
private FactIds() {
}
public static String rawFrameId(ProtocolId protocol,
String vehicleKey,
int messageId,
int subType,
Instant receivedAt,
String checksum) {
if (protocol == null) {
throw new IllegalArgumentException("protocol must not be null");
}
String input = protocol.name() + "|"
+ clean(vehicleKey) + "|"
+ messageId + "|"
+ subType + "|"
+ (receivedAt == null ? "" : receivedAt.toString()) + "|"
+ clean(checksum);
return "rf_" + sha256(input).substring(0, 32);
}
public static String decodedFactId(String frameId, String kind, int ordinal) {
return "df_" + clean(frameId) + "_" + clean(kind) + "_" + ordinal;
}
private static String clean(String value) {
return value == null ? "" : value.trim();
}
private static String sha256(String value) {
try {
MessageDigest digest = MessageDigest.getInstance("SHA-256");
return HexFormat.of().formatHex(digest.digest(value.getBytes(StandardCharsets.UTF_8)));
} catch (NoSuchAlgorithmException e) {
throw new IllegalStateException("SHA-256 is unavailable", e);
}
}
}

View File

@@ -0,0 +1,7 @@
package com.lingniu.ingest.facts;
public enum ParseStatus {
NOT_PARSED,
SUCCEEDED,
FAILED
}

View File

@@ -0,0 +1,65 @@
package com.lingniu.ingest.facts;
import com.lingniu.ingest.api.ProtocolId;
import java.time.Instant;
import java.util.Map;
public record RawFrameFact(
String frameId,
ProtocolId protocol,
String vehicleKey,
String vin,
String phone,
int messageId,
int subType,
Instant eventTime,
Instant receivedAt,
String peer,
String rawUri,
String checksum,
long rawSizeBytes,
ParseStatus parseStatus,
String parseError,
Map<String, String> metadata
) {
public RawFrameFact {
frameId = required(frameId, "frameId");
if (protocol == null) {
throw new IllegalArgumentException("protocol must not be null");
}
vehicleKey = required(vehicleKey, "vehicleKey");
receivedAt = required(receivedAt, "receivedAt");
eventTime = eventTime == null ? receivedAt : eventTime;
rawUri = required(rawUri, "rawUri");
checksum = required(checksum, "checksum");
if (rawSizeBytes < 0) {
throw new IllegalArgumentException("rawSizeBytes must not be negative");
}
parseStatus = parseStatus == null ? ParseStatus.NOT_PARSED : parseStatus;
vin = clean(vin);
phone = clean(phone);
peer = clean(peer);
parseError = clean(parseError);
metadata = metadata == null ? Map.of() : Map.copyOf(metadata);
}
private static String required(String value, String field) {
String cleaned = clean(value);
if (cleaned.isBlank()) {
throw new IllegalArgumentException(field + " must not be blank");
}
return cleaned;
}
private static Instant required(Instant value, String field) {
if (value == null) {
throw new IllegalArgumentException(field + " must not be null");
}
return value;
}
private static String clean(String value) {
return value == null ? "" : value.trim();
}
}

View File

@@ -0,0 +1,46 @@
package com.lingniu.ingest.facts;
import com.lingniu.ingest.api.ProtocolId;
import java.nio.charset.StandardCharsets;
import java.security.MessageDigest;
import java.security.NoSuchAlgorithmException;
import java.util.HexFormat;
public final class VehicleKey {
private VehicleKey() {
}
public static String derive(ProtocolId protocol, String vin, String phone, String rawIdentitySeed) {
if (protocol == null) {
throw new IllegalArgumentException("protocol must not be null");
}
String normalizedVin = clean(vin);
if (!normalizedVin.isBlank()) {
return normalizedVin;
}
String normalizedPhone = clean(phone);
if (protocol == ProtocolId.JT808 && !normalizedPhone.isBlank()) {
return "jt808:" + normalizedPhone;
}
String seed = clean(rawIdentitySeed);
if (seed.isBlank()) {
seed = protocol.name();
}
return "unknown:" + protocol.name() + ":" + sha256(seed).substring(0, 16);
}
private static String clean(String value) {
return value == null ? "" : value.trim();
}
private static String sha256(String value) {
try {
MessageDigest digest = MessageDigest.getInstance("SHA-256");
return HexFormat.of().formatHex(digest.digest(value.getBytes(StandardCharsets.UTF_8)));
} catch (NoSuchAlgorithmException e) {
throw new IllegalStateException("SHA-256 is unavailable", e);
}
}
}

View File

@@ -0,0 +1,39 @@
package com.lingniu.ingest.facts;
import com.lingniu.ingest.api.ProtocolId;
import org.junit.jupiter.api.Test;
import java.time.Instant;
import java.util.Map;
import static org.assertj.core.api.Assertions.assertThat;
class DecodedFactTest {
@Test
void locationFactCarriesRawReferenceAndMetadata() {
DecodedFact.Location location = new DecodedFact.Location(
"fact-1",
"frame-1",
ProtocolId.JT808,
"jt808:013912345678",
"",
"013912345678",
Instant.parse("2026-06-29T00:00:00Z"),
Instant.parse("2026-06-29T00:00:01Z"),
"archive://jt808/frame-1.bin",
Map.of("messageId", "0x0200"),
120.123456,
30.123456,
10.0,
42.0,
180.0,
0L,
2L,
1000.5);
assertThat(location.factId()).isEqualTo("fact-1");
assertThat(location.rawUri()).isEqualTo("archive://jt808/frame-1.bin");
assertThat(location.metadata()).containsEntry("messageId", "0x0200");
}
}

View File

@@ -0,0 +1,38 @@
package com.lingniu.ingest.facts;
import com.lingniu.ingest.api.ProtocolId;
import org.junit.jupiter.api.Test;
import java.time.Instant;
import static org.assertj.core.api.Assertions.assertThat;
class FactIdsTest {
@Test
void rawFrameIdIsStableForSameInputs() {
String first = FactIds.rawFrameId(
ProtocolId.GB32960,
"VIN001",
2,
7,
Instant.parse("2026-06-29T00:00:00Z"),
"sha256:abc");
String second = FactIds.rawFrameId(
ProtocolId.GB32960,
"VIN001",
2,
7,
Instant.parse("2026-06-29T00:00:00Z"),
"sha256:abc");
assertThat(first).isEqualTo(second).startsWith("rf_");
}
@Test
void decodedFactIdIncludesKind() {
String id = FactIds.decodedFactId("rf_123", "location", 0);
assertThat(id).isEqualTo("df_rf_123_location_0");
}
}

View File

@@ -0,0 +1,63 @@
package com.lingniu.ingest.facts;
import com.lingniu.ingest.api.ProtocolId;
import org.junit.jupiter.api.Test;
import java.time.Instant;
import java.util.Map;
import static org.assertj.core.api.Assertions.assertThat;
import static org.assertj.core.api.Assertions.assertThatThrownBy;
class RawFrameFactTest {
@Test
void normalizesOptionalStringsAndMetadata() {
RawFrameFact fact = new RawFrameFact(
"frame-1",
ProtocolId.JT808,
"jt808:013912345678",
null,
"013912345678",
0x0200,
0,
null,
Instant.parse("2026-06-29T00:00:00Z"),
null,
"archive://2026/06/29/JT808/jt808-013912345678/frame-1.bin",
"sha256:abc",
128,
ParseStatus.SUCCEEDED,
null,
Map.of("messageName", "LOCATION"));
assertThat(fact.vin()).isEmpty();
assertThat(fact.eventTime()).isEqualTo(fact.receivedAt());
assertThat(fact.peer()).isEmpty();
assertThat(fact.parseError()).isEmpty();
assertThat(fact.metadata()).containsEntry("messageName", "LOCATION");
}
@Test
void rejectsMissingRequiredFields() {
assertThatThrownBy(() -> new RawFrameFact(
"",
ProtocolId.GB32960,
"VIN001",
"VIN001",
"",
2,
0,
Instant.parse("2026-06-29T00:00:00Z"),
Instant.parse("2026-06-29T00:00:00Z"),
"",
"archive://x",
"sha256:abc",
1,
ParseStatus.SUCCEEDED,
"",
Map.of()))
.isInstanceOf(IllegalArgumentException.class)
.hasMessageContaining("frameId");
}
}

View File

@@ -0,0 +1,42 @@
package com.lingniu.ingest.facts;
import com.lingniu.ingest.api.ProtocolId;
import org.junit.jupiter.api.Test;
import static org.assertj.core.api.Assertions.assertThat;
import static org.assertj.core.api.Assertions.assertThatThrownBy;
class VehicleKeyTest {
@Test
void usesVinForGb32960() {
assertThat(VehicleKey.derive(ProtocolId.GB32960, " LB9A32A20P0LS1257 ", "", "abc"))
.isEqualTo("LB9A32A20P0LS1257");
}
@Test
void usesResolvedVinForJt808WhenPresent() {
assertThat(VehicleKey.derive(ProtocolId.JT808, "VIN001", "013912345678", "abc"))
.isEqualTo("VIN001");
}
@Test
void usesPhoneForJt808WhenVinIsMissing() {
assertThat(VehicleKey.derive(ProtocolId.JT808, "", "013912345678", "abc"))
.isEqualTo("jt808:013912345678");
}
@Test
void usesStableUnknownHashWhenVehicleIdentifiersAreMissing() {
assertThat(VehicleKey.derive(ProtocolId.JT808, "", "", "abc"))
.isEqualTo(VehicleKey.derive(ProtocolId.JT808, null, null, "abc"))
.startsWith("unknown:JT808:");
}
@Test
void rejectsMissingProtocol() {
assertThatThrownBy(() -> VehicleKey.derive(null, "VIN001", "", "abc"))
.isInstanceOf(IllegalArgumentException.class)
.hasMessageContaining("protocol");
}
}

View File

@@ -42,9 +42,8 @@ class Gb32960ChannelHandlerAckBoundaryTest {
dispatch.awaitPublishCount(1);
dispatch.completeDurability();
channel.runPendingTasks();
ByteBuf ack = channel.readOutbound();
ByteBuf ack = readOutboundAfterRunningTasks(channel);
assertThat(ack).isNotNull();
assertThat(ack.getUnsignedByte(2)).isEqualTo((short) 0x02);
assertThat(ack.getUnsignedByte(3)).isEqualTo((short) 0x01);
@@ -85,10 +84,9 @@ class Gb32960ChannelHandlerAckBoundaryTest {
assertThat((Object) channel.readOutbound()).isNull();
dispatch.completeDurability(0);
channel.runPendingTasks();
ByteBuf firstAck = channel.readOutbound();
ByteBuf secondAck = channel.readOutbound();
ByteBuf firstAck = readOutboundAfterRunningTasks(channel);
ByteBuf secondAck = readOutboundAfterRunningTasks(channel);
assertThat(firstAck).isNotNull();
assertThat(secondAck).isNotNull();
assertThat(firstAck.getUnsignedByte(2)).isEqualTo((short) 0x02);
@@ -99,6 +97,19 @@ class Gb32960ChannelHandlerAckBoundaryTest {
}
}
private static ByteBuf readOutboundAfterRunningTasks(EmbeddedChannel channel) {
long deadline = System.currentTimeMillis() + 3000;
while (System.currentTimeMillis() < deadline) {
channel.runPendingTasks();
ByteBuf outbound = channel.readOutbound();
if (outbound != null) {
return outbound;
}
Thread.onSpinWait();
}
return null;
}
private static Gb32960ChannelHandler handlerWith(Dispatcher dispatcher) {
return new Gb32960ChannelHandler(
decoder(),

View File

@@ -16,6 +16,10 @@
<groupId>com.lingniu.ingest</groupId>
<artifactId>event-file-store</artifactId>
</dependency>
<dependency>
<groupId>com.lingniu.ingest</groupId>
<artifactId>tdengine-history-store</artifactId>
</dependency>
<dependency>
<groupId>com.lingniu.ingest</groupId>
<artifactId>sink-mq</artifactId>

View File

@@ -8,7 +8,9 @@ import org.springframework.web.bind.annotation.ExceptionHandler;
import org.springframework.web.bind.annotation.RestControllerAdvice;
import org.springframework.web.method.annotation.MethodArgumentTypeMismatchException;
import org.springframework.web.server.ResponseStatusException;
import org.springframework.web.servlet.resource.NoResourceFoundException;
import java.io.IOException;
import java.time.OffsetDateTime;
import java.time.ZoneId;
import java.time.format.DateTimeParseException;
@@ -69,6 +71,18 @@ public final class ApiExceptionHandler {
return error(HttpStatus.BAD_REQUEST, "invalid date/time parameter: " + ex.getParsedString(), request);
}
@ExceptionHandler(NoResourceFoundException.class)
public ResponseEntity<ApiError> noResource(NoResourceFoundException ex,
HttpServletRequest request) {
return error(HttpStatus.NOT_FOUND, "resource not found: " + request.getRequestURI(), request);
}
@ExceptionHandler(IOException.class)
public ResponseEntity<ApiError> io(IOException ex,
HttpServletRequest request) {
return error(HttpStatus.SERVICE_UNAVAILABLE, safeMessage(ex, "history storage unavailable"), request);
}
@ExceptionHandler(Exception.class)
public ResponseEntity<ApiError> generic(Exception ex,
HttpServletRequest request) {

View File

@@ -1,7 +1,5 @@
package com.lingniu.ingest.eventhistory;
import com.fasterxml.jackson.databind.JsonNode;
import com.fasterxml.jackson.databind.ObjectMapper;
import com.lingniu.ingest.api.ProtocolId;
import com.lingniu.ingest.eventfilestore.EventFileQuery;
import com.lingniu.ingest.eventfilestore.EventFileRecord;
@@ -14,8 +12,6 @@ import org.springframework.web.bind.annotation.RequestParam;
import org.springframework.web.bind.annotation.RestController;
import java.io.IOException;
import java.util.ArrayList;
import java.util.LinkedHashMap;
import java.util.List;
import java.util.Map;
@@ -31,8 +27,6 @@ import java.util.Map;
@RequestMapping("/api/event-history")
public class EventHistoryController {
private static final ObjectMapper OBJECT_MAPPER = new ObjectMapper();
private final EventFileStore store;
public EventHistoryController(EventFileStore store) {
@@ -52,51 +46,6 @@ public class EventHistoryController {
.toList();
}
@GetMapping(value = "/export.csv", produces = "text/csv;charset=UTF-8")
public String exportCsv(
@RequestParam ProtocolId protocol,
@RequestParam String dateFrom,
@RequestParam String dateTo,
@RequestParam(defaultValue = "ASC") EventFileQuery.Order order,
@RequestParam(defaultValue = "100000") int limit,
@RequestParam(required = false) String vin,
@RequestParam(required = false) String fields) throws IOException {
List<EventFileRecord> records = queryRecords(protocol, dateFrom, dateTo, order, limit, vin);
List<String> selectedFields = parseFields(fields);
StringBuilder csv = new StringBuilder(256 + records.size() * 128);
csv.append("event_id,protocol,event_type,vin,event_time,ingest_time,raw_archive_uri");
for (String field : selectedFields) {
csv.append(',').append(csv(field));
}
csv.append(",payload_json\n");
for (EventFileRecord record : records) {
Map<String, String> telemetryFields = selectedFields.isEmpty()
? Map.of()
: telemetryFields(record.payloadJson());
csv.append(csv(record.eventId())).append(',')
.append(csv(record.protocol().name())).append(',')
.append(csv(record.eventType())).append(',')
.append(csv(record.vin())).append(',')
.append(csv(record.eventTime().toString())).append(',')
.append(csv(record.ingestTime().toString())).append(',')
.append(csv(record.rawArchiveUri()));
for (String field : selectedFields) {
csv.append(',').append(csv(fieldValue(record, telemetryFields, field)));
}
csv.append(',')
.append(csv(record.payloadJson())).append('\n');
}
return csv.toString();
}
public String exportCsv(ProtocolId protocol,
String dateFrom,
String dateTo,
EventFileQuery.Order order,
int limit) throws IOException {
return exportCsv(protocol, dateFrom, dateTo, order, limit, null, null);
}
private List<EventFileRecord> queryRecords(ProtocolId protocol,
String dateFrom,
String dateTo,
@@ -140,64 +89,4 @@ public class EventHistoryController {
record.payloadJson());
}
}
private static List<String> parseFields(String fields) {
if (fields == null || fields.isBlank()) {
return List.of();
}
List<String> out = new ArrayList<>();
for (String field : fields.split(",")) {
String normalized = field.trim();
if (!normalized.isBlank()) {
out.add(normalized);
}
}
return List.copyOf(out);
}
private static Map<String, String> telemetryFields(String payloadJson) throws IOException {
if (payloadJson == null || payloadJson.isBlank()) {
return Map.of();
}
JsonNode root = OBJECT_MAPPER.readTree(payloadJson);
JsonNode fields = root.path("fields");
if (!fields.isArray()) {
return Map.of();
}
// 通用导出只识别 telemetry snapshot 的扁平 fields复杂 GB32960 block 字段用专用 CSV 接口。
Map<String, String> out = new LinkedHashMap<>();
for (JsonNode field : fields) {
String key = field.path("key").asText("");
if (!key.isBlank()) {
out.put(key, field.path("value").asText(""));
}
}
return out;
}
private static String fieldValue(EventFileRecord record,
Map<String, String> telemetryFields,
String field) {
if (field == null || field.isBlank()) {
return "";
}
if (field.startsWith("metadata.")) {
return record.metadata().get(field.substring("metadata.".length()));
}
return telemetryFields.get(field);
}
private static String csv(String value) {
if (value == null) {
return "";
}
boolean quote = value.indexOf(',') >= 0
|| value.indexOf('"') >= 0
|| value.indexOf('\n') >= 0
|| value.indexOf('\r') >= 0;
if (!quote) {
return value;
}
return "\"" + value.replace("\"", "\"\"") + "\"";
}
}

View File

@@ -6,8 +6,12 @@ import com.lingniu.ingest.api.consumer.EnvelopeIngestor;
import com.lingniu.ingest.eventfilestore.EventFileRecord;
import com.lingniu.ingest.eventfilestore.EventFileStore;
import com.lingniu.ingest.sink.mq.proto.VehicleEnvelope;
import com.lingniu.ingest.tdenginehistory.TdengineEnvelopeRows;
import com.lingniu.ingest.tdenginehistory.TdengineHistoryWriter;
import java.io.IOException;
import java.util.ArrayList;
import java.util.List;
/**
* Consumer-side entry point that stores one Kafka envelope value into the
@@ -20,8 +24,15 @@ public final class EventHistoryEnvelopeIngestor implements EnvelopeIngestor {
private final EventFileStore store;
private final TelemetryEnvelopeRecordMapper mapper;
private final TdengineHistoryWriter tdengineWriter;
public EventHistoryEnvelopeIngestor(EventFileStore store, TelemetryEnvelopeRecordMapper mapper) {
this(store, mapper, null);
}
public EventHistoryEnvelopeIngestor(EventFileStore store,
TelemetryEnvelopeRecordMapper mapper,
TdengineHistoryWriter tdengineWriter) {
if (store == null) {
throw new IllegalArgumentException("store must not be null");
}
@@ -30,34 +41,64 @@ public final class EventHistoryEnvelopeIngestor implements EnvelopeIngestor {
}
this.store = store;
this.mapper = mapper;
this.tdengineWriter = tdengineWriter;
}
public void ingest(byte[] kafkaValue) throws IOException {
VehicleEnvelope envelope = parse(kafkaValue);
EventFileRecord record = mapper.toRecord(envelope);
store.append(record);
writeTdengineFacts(List.of(envelope));
}
@Override
public EnvelopeIngestResult tryIngest(byte[] kafkaValue) {
VehicleEnvelope envelope = null;
try {
envelope = parse(kafkaValue);
EventFileRecord record = mapper.toRecord(envelope);
store.append(record);
return EnvelopeIngestResult.stored(record.eventId(), record.vin());
} catch (IllegalArgumentException ex) {
// 协议不可解析或缺少必要字段属于不可重试问题,避免 Kafka consumer 无限重放。
return envelope == null
? EnvelopeIngestResult.invalid(ex.getMessage())
: EnvelopeIngestResult.skipped(envelope.getEventId(), envelope.getVin(), ex.getMessage());
} catch (IOException ex) {
// 文件库写入失败可能是临时 I/O 问题,交给上层 worker 按失败处理。
return EnvelopeIngestResult.failed(
envelope == null ? "" : envelope.getEventId(),
envelope == null ? "" : envelope.getVin(),
ex.getMessage());
return tryIngestAll(List.of(kafkaValue)).getFirst();
}
@Override
public List<EnvelopeIngestResult> tryIngestAll(List<byte[]> kafkaValues) {
if (kafkaValues == null || kafkaValues.isEmpty()) {
return List.of();
}
List<EnvelopeIngestResult> results = new ArrayList<>(kafkaValues.size());
List<BatchEntry> valid = new ArrayList<>(kafkaValues.size());
for (byte[] kafkaValue : kafkaValues) {
VehicleEnvelope envelope = null;
try {
envelope = parse(kafkaValue);
EventFileRecord record = mapper.toRecord(envelope);
results.add(null);
valid.add(new BatchEntry(results.size() - 1, envelope, record));
} catch (IllegalArgumentException ex) {
results.add(envelope == null
? EnvelopeIngestResult.invalid(ex.getMessage())
: EnvelopeIngestResult.skipped(envelope.getEventId(), envelope.getVin(), ex.getMessage()));
}
}
if (valid.isEmpty()) {
return List.copyOf(results);
}
try {
store.appendAll(valid.stream().map(BatchEntry::record).toList());
writeTdengineFacts(valid.stream().map(BatchEntry::envelope).toList());
for (BatchEntry entry : valid) {
results.set(entry.index(), EnvelopeIngestResult.stored(
entry.record().eventId(),
entry.record().vin()));
}
} catch (IOException ex) {
for (BatchEntry entry : valid) {
results.set(entry.index(), EnvelopeIngestResult.failed(
entry.envelope().getEventId(),
entry.envelope().getVin(),
ex.getMessage()));
}
}
return List.copyOf(results);
}
private record BatchEntry(int index, VehicleEnvelope envelope, EventFileRecord record) {
}
private static VehicleEnvelope parse(byte[] kafkaValue) {
@@ -70,4 +111,28 @@ public final class EventHistoryEnvelopeIngestor implements EnvelopeIngestor {
throw new IllegalArgumentException("failed to parse VehicleEnvelope", e);
}
}
private void writeTdengineFacts(List<VehicleEnvelope> envelopes) throws IOException {
if (tdengineWriter == null) {
return;
}
var rawFrames = envelopes.stream()
.flatMap(envelope -> TdengineEnvelopeRows.rawFrame(envelope).stream())
.toList();
if (!rawFrames.isEmpty()) {
tdengineWriter.appendRawFrames(rawFrames);
}
var locations = envelopes.stream()
.flatMap(envelope -> TdengineEnvelopeRows.location(envelope).stream())
.toList();
if (!locations.isEmpty()) {
tdengineWriter.appendLocations(locations);
}
var telemetryFields = envelopes.stream()
.flatMap(envelope -> TdengineEnvelopeRows.telemetryFields(envelope).stream())
.toList();
if (!telemetryFields.isEmpty()) {
tdengineWriter.appendTelemetryFields(telemetryFields);
}
}
}

View File

@@ -0,0 +1,168 @@
package com.lingniu.ingest.eventhistory;
import com.fasterxml.jackson.core.JsonProcessingException;
import com.fasterxml.jackson.databind.ObjectMapper;
import com.lingniu.ingest.tdenginehistory.TdengineHistoryReader;
import com.lingniu.ingest.tdenginehistory.TdengineLocationQuery;
import com.lingniu.ingest.tdenginehistory.TdengineLocationRow;
import com.lingniu.ingest.tdenginehistory.TdenginePage;
import com.lingniu.ingest.tdenginehistory.TdenginePageCursor;
import com.lingniu.ingest.tdenginehistory.TdengineQueryOrder;
import io.swagger.v3.oas.annotations.Operation;
import io.swagger.v3.oas.annotations.Parameter;
import io.swagger.v3.oas.annotations.tags.Tag;
import org.springframework.boot.autoconfigure.condition.ConditionalOnBean;
import org.springframework.boot.autoconfigure.condition.ConditionalOnProperty;
import org.springframework.http.HttpStatus;
import org.springframework.web.bind.annotation.GetMapping;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.RequestParam;
import org.springframework.web.bind.annotation.RestController;
import org.springframework.web.server.ResponseStatusException;
import java.io.IOException;
import java.time.Instant;
import java.util.List;
@RestController
@ConditionalOnProperty(prefix = "lingniu.ingest.event-history", name = "enabled", havingValue = "true")
@ConditionalOnBean(TdengineHistoryReader.class)
@RequestMapping("/api/event-history/jt808")
@Tag(name = "jt-808-location-controller", description = "JT808 位置历史分页查询接口。")
public final class Jt808LocationHistoryController {
private static final ObjectMapper OBJECT_MAPPER = new ObjectMapper();
private final TdengineHistoryReader reader;
public Jt808LocationHistoryController(TdengineHistoryReader reader) {
if (reader == null) {
throw new IllegalArgumentException("reader must not be null");
}
this.reader = reader;
}
@GetMapping("/locations")
@Operation(
summary = "查询 JT808 位置历史",
description = "按终端手机号和时间范围查询 TDengine 中的 JT808 位置点。分页使用 cursorTs + cursorId不使用 offset适合高并发历史轨迹查询。")
public LocationPageResponse locations(
@Parameter(description = "JT808 终端手机号或终端标识;内部会映射到 jt808:<phone> 车辆键。", required = true, example = "g7gps")
@RequestParam String phone,
@Parameter(description = "开始时间,支持日期或精确到秒的时间;无时区时按东八区解析。", example = "2026-06-29T13:00:00")
@RequestParam String dateFrom,
@Parameter(description = "结束时间,支持日期或精确到秒的时间;无时区时按东八区解析。", example = "2026-06-29T13:10:00")
@RequestParam String dateTo,
@Parameter(description = "排序方向。", example = "DESC")
@RequestParam(defaultValue = "DESC") TdengineQueryOrder order,
@Parameter(description = "返回位置点数量上限,最大 1000。", example = "100")
@RequestParam(defaultValue = "100") int limit,
@Parameter(description = "上一页返回的 nextCursor.ts。", example = "2026-06-29T05:00:01Z")
@RequestParam(required = false) String cursorTs,
@Parameter(description = "上一页返回的 nextCursor.id。", example = "jt808-location-xxx")
@RequestParam(required = false) String cursorId) throws IOException {
if (phone == null || phone.isBlank()) {
throw new ResponseStatusException(HttpStatus.BAD_REQUEST, "phone is required for jt808 location query");
}
QueryTimeRange range = QueryTimeRange.parse(dateFrom, dateTo);
TdenginePage<TdengineLocationRow> page = reader.queryLocations(new TdengineLocationQuery(
"JT808",
vehicleKey(phone),
range.eventTimeFrom(),
range.eventTimeTo().plusMillis(1),
order,
limit,
cursor(cursorTs, cursorId)));
return LocationPageResponse.from(page);
}
private static String vehicleKey(String phone) {
String value = phone.trim();
return value.startsWith("jt808:") ? value : "jt808:" + value;
}
private static TdenginePageCursor cursor(String cursorTs, String cursorId) {
boolean hasTs = cursorTs != null && !cursorTs.isBlank();
boolean hasId = cursorId != null && !cursorId.isBlank();
if (!hasTs && !hasId) {
return null;
}
if (!hasTs || !hasId) {
throw new ResponseStatusException(HttpStatus.BAD_REQUEST, "cursorTs and cursorId must be provided together");
}
return new TdenginePageCursor(Instant.parse(cursorTs.trim()), cursorId.trim());
}
public record LocationPageResponse(
List<LocationResponse> items,
CursorResponse nextCursor) {
private static LocationPageResponse from(TdenginePage<TdengineLocationRow> page) {
CursorResponse next = page.nextCursor()
.map(cursor -> new CursorResponse(cursor.ts().toString(), cursor.tieBreaker()))
.orElse(null);
return new LocationPageResponse(
page.items().stream().map(LocationResponse::from).toList(),
next);
}
}
public record CursorResponse(String ts, String id) {
}
public record LocationResponse(
String eventTime,
String receivedAt,
String factId,
String frameId,
String phone,
String vin,
String vehicleKey,
double longitude,
double latitude,
double altitudeM,
double speedKmh,
double directionDeg,
long alarmFlag,
long statusFlag,
Double totalMileageKm,
String rawUri,
String metadataJson) {
private static LocationResponse from(TdengineLocationRow row) {
return new LocationResponse(
row.ts().toString(),
row.receivedAt().toString(),
row.factId(),
row.frameId(),
row.phone(),
row.vin(),
row.vehicleKey(),
row.longitude(),
row.latitude(),
row.altitudeM(),
row.speedKmh(),
row.directionDeg(),
row.alarmFlag(),
row.statusFlag(),
row.totalMileageKm(),
rawUri(row),
row.metadataJson());
}
private static String rawUri(TdengineLocationRow row) {
if (row.rawUri() != null && !row.rawUri().isBlank()) {
return row.rawUri();
}
String metadataJson = row.metadataJson();
if (metadataJson == null || metadataJson.isBlank()) {
return row.rawUri();
}
try {
return OBJECT_MAPPER.readTree(metadataJson).path("rawArchiveUri").asText(row.rawUri());
} catch (JsonProcessingException ignored) {
return row.rawUri();
}
}
}
}

View File

@@ -0,0 +1,189 @@
package com.lingniu.ingest.eventhistory;
import com.lingniu.ingest.tdenginehistory.TdengineHistoryReader;
import com.lingniu.ingest.tdenginehistory.TdenginePage;
import com.lingniu.ingest.tdenginehistory.TdenginePageCursor;
import com.lingniu.ingest.tdenginehistory.TdengineQueryOrder;
import com.lingniu.ingest.tdenginehistory.TdengineTelemetryFieldQuery;
import com.lingniu.ingest.tdenginehistory.TdengineTelemetryFieldRow;
import io.swagger.v3.oas.annotations.Operation;
import io.swagger.v3.oas.annotations.Parameter;
import io.swagger.v3.oas.annotations.tags.Tag;
import org.springframework.boot.autoconfigure.condition.ConditionalOnBean;
import org.springframework.boot.autoconfigure.condition.ConditionalOnProperty;
import org.springframework.http.HttpStatus;
import org.springframework.web.bind.annotation.GetMapping;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.RequestParam;
import org.springframework.web.bind.annotation.RestController;
import org.springframework.web.server.ResponseStatusException;
import java.io.IOException;
import java.time.Instant;
import java.util.List;
import java.util.Locale;
@RestController
@ConditionalOnProperty(prefix = "lingniu.ingest.event-history", name = "enabled", havingValue = "true")
@ConditionalOnBean(TdengineHistoryReader.class)
@RequestMapping("/api/event-history/telemetry")
@Tag(name = "telemetry-field-history-controller", description = "遥测字段历史分页查询接口。")
public final class TelemetryFieldHistoryController {
private final TdengineHistoryReader reader;
public TelemetryFieldHistoryController(TdengineHistoryReader reader) {
if (reader == null) {
throw new IllegalArgumentException("reader must not be null");
}
this.reader = reader;
}
@GetMapping("/fields")
@Operation(
summary = "查询单车单字段历史值",
description = "按协议、车辆标识、字段 key 和时间范围查询 TDengine telemetry_fields。分页使用 cursorTs + cursorId不使用 offset。")
public TelemetryFieldPageResponse fields(
@Parameter(description = "协议名,例如 GB32960 或 JT808。", required = true, example = "GB32960")
@RequestParam String protocol,
@Parameter(description = "字段 key例如 FUEL_CELL_V2016.hydrogenComsuxxx。", required = true)
@RequestParam String fieldKey,
@Parameter(description = "内部车辆键。传入后优先使用。", example = "LNVFC000000000001")
@RequestParam(required = false) String vehicleKey,
@Parameter(description = "VIN。GB32960 常用该字段作为车辆键。", example = "LNVFC000000000001")
@RequestParam(required = false) String vin,
@Parameter(description = "JT808 终端手机号或终端标识JT808 会自动映射为 jt808:<phone>。", example = "g7gps")
@RequestParam(required = false) String phone,
@Parameter(description = "开始时间,支持日期或精确到秒的时间;无时区时按东八区解析。", example = "2026-06-29T13:00:00")
@RequestParam String dateFrom,
@Parameter(description = "结束时间,支持日期或精确到秒的时间;无时区时按东八区解析。", example = "2026-06-29T13:10:00")
@RequestParam String dateTo,
@Parameter(description = "排序方向。", example = "DESC")
@RequestParam(defaultValue = "DESC") TdengineQueryOrder order,
@Parameter(description = "返回值数量上限,最大 1000。", example = "100")
@RequestParam(defaultValue = "100") int limit,
@Parameter(description = "上一页返回的 nextCursor.ts。", example = "2026-06-29T05:00:01Z")
@RequestParam(required = false) String cursorTs,
@Parameter(description = "上一页返回的 nextCursor.id。", example = "event-xxx#0")
@RequestParam(required = false) String cursorId) throws IOException {
String normalizedProtocol = require(protocol, "protocol is required for telemetry field query").toUpperCase(Locale.ROOT);
String normalizedFieldKey = require(fieldKey, "fieldKey is required for telemetry field query");
QueryTimeRange range = QueryTimeRange.parse(dateFrom, dateTo);
TdenginePage<TdengineTelemetryFieldRow> page = reader.queryTelemetryFields(new TdengineTelemetryFieldQuery(
normalizedProtocol,
resolveVehicleKey(normalizedProtocol, vehicleKey, vin, phone),
normalizedFieldKey,
range.eventTimeFrom(),
range.eventTimeTo().plusMillis(1),
order,
limit,
cursor(cursorTs, cursorId)));
return TelemetryFieldPageResponse.from(page);
}
private static String resolveVehicleKey(String protocol, String vehicleKey, String vin, String phone) {
String explicitVehicleKey = trimToNull(vehicleKey);
if (explicitVehicleKey != null) {
return explicitVehicleKey;
}
String vinValue = trimToNull(vin);
if (vinValue != null) {
return vinValue;
}
String phoneValue = trimToNull(phone);
if (phoneValue != null) {
if ("JT808".equals(protocol) && !phoneValue.startsWith("jt808:")) {
return "jt808:" + phoneValue;
}
return phoneValue;
}
throw new ResponseStatusException(HttpStatus.BAD_REQUEST, "vehicleKey, vin or phone is required");
}
private static String require(String value, String message) {
String trimmed = trimToNull(value);
if (trimmed == null) {
throw new ResponseStatusException(HttpStatus.BAD_REQUEST, message);
}
return trimmed;
}
private static String trimToNull(String value) {
if (value == null || value.isBlank()) {
return null;
}
return value.trim();
}
private static TdenginePageCursor cursor(String cursorTs, String cursorId) {
boolean hasTs = cursorTs != null && !cursorTs.isBlank();
boolean hasId = cursorId != null && !cursorId.isBlank();
if (!hasTs && !hasId) {
return null;
}
if (!hasTs || !hasId) {
throw new ResponseStatusException(HttpStatus.BAD_REQUEST, "cursorTs and cursorId must be provided together");
}
return new TdenginePageCursor(Instant.parse(cursorTs.trim()), cursorId.trim());
}
public record TelemetryFieldPageResponse(
List<TelemetryFieldResponse> items,
CursorResponse nextCursor) {
private static TelemetryFieldPageResponse from(TdenginePage<TdengineTelemetryFieldRow> page) {
CursorResponse next = page.nextCursor()
.map(cursor -> new CursorResponse(cursor.ts().toString(), cursor.tieBreaker()))
.orElse(null);
return new TelemetryFieldPageResponse(
page.items().stream().map(TelemetryFieldResponse::from).toList(),
next);
}
}
public record CursorResponse(String ts, String id) {
}
public record TelemetryFieldResponse(
String eventTime,
String receivedAt,
String factId,
String frameId,
String fieldKey,
String valueType,
String valueText,
Double valueDouble,
Long valueLong,
String unit,
String quality,
String sourcePath,
String rawUri,
String metadataJson,
String protocol,
String vehicleKey,
String vin,
String phone) {
private static TelemetryFieldResponse from(TdengineTelemetryFieldRow row) {
return new TelemetryFieldResponse(
row.ts().toString(),
row.receivedAt().toString(),
row.factId(),
row.frameId(),
row.fieldKey(),
row.valueType(),
row.valueText(),
row.valueDouble(),
row.valueLong(),
row.unit(),
row.quality(),
row.sourcePath(),
row.rawUri(),
row.metadataJson(),
row.protocol(),
row.vehicleKey(),
row.vin(),
row.phone());
}
}
}

View File

@@ -7,17 +7,23 @@ import com.lingniu.ingest.eventhistory.EventHistoryController;
import com.lingniu.ingest.eventhistory.EventHistoryEnvelopeIngestor;
import com.lingniu.ingest.eventhistory.Gb32960DecodedFrameService;
import com.lingniu.ingest.eventhistory.Gb32960FrameController;
import com.lingniu.ingest.eventhistory.Jt808LocationHistoryController;
import com.lingniu.ingest.eventhistory.TelemetryFieldHistoryController;
import com.lingniu.ingest.eventhistory.TelemetryEnvelopeRecordMapper;
import com.lingniu.ingest.protocol.gb32960.codec.Gb32960MessageDecoder;
import com.lingniu.ingest.protocol.gb32960.config.Gb32960AutoConfiguration;
import com.lingniu.ingest.sink.archive.config.SinkArchiveAutoConfiguration;
import com.lingniu.ingest.sink.archive.config.SinkArchiveProperties;
import com.lingniu.ingest.tdenginehistory.config.TdengineHistoryAutoConfiguration;
import com.lingniu.ingest.tdenginehistory.TdengineHistoryReader;
import com.lingniu.ingest.tdenginehistory.TdengineHistoryWriter;
import org.springframework.boot.autoconfigure.AutoConfiguration;
import org.springframework.boot.autoconfigure.AutoConfigureAfter;
import org.springframework.boot.autoconfigure.condition.ConditionalOnBean;
import org.springframework.boot.autoconfigure.condition.ConditionalOnMissingBean;
import org.springframework.boot.autoconfigure.condition.ConditionalOnProperty;
import org.springframework.context.annotation.Bean;
import org.springframework.beans.factory.ObjectProvider;
import java.net.URI;
import java.nio.file.Path;
@@ -35,7 +41,11 @@ import java.nio.file.Path;
* Kafka consumer 是否启动还取决于 {@code lingniu.ingest.sink.mq.consumer.enabled=true}。
*/
@AutoConfiguration
@AutoConfigureAfter({Gb32960AutoConfiguration.class, SinkArchiveAutoConfiguration.class})
@AutoConfigureAfter({
Gb32960AutoConfiguration.class,
SinkArchiveAutoConfiguration.class,
TdengineHistoryAutoConfiguration.class
})
@ConditionalOnProperty(prefix = "lingniu.ingest.event-history", name = "enabled", havingValue = "true")
public class EventHistoryAutoConfiguration {
@@ -49,8 +59,9 @@ public class EventHistoryAutoConfiguration {
@ConditionalOnBean(EventFileStore.class)
@ConditionalOnMissingBean
public EventHistoryEnvelopeIngestor eventHistoryEnvelopeIngestor(EventFileStore store,
TelemetryEnvelopeRecordMapper mapper) {
return new EventHistoryEnvelopeIngestor(store, mapper);
TelemetryEnvelopeRecordMapper mapper,
ObjectProvider<TdengineHistoryWriter> writer) {
return new EventHistoryEnvelopeIngestor(store, mapper, writer.getIfAvailable());
}
@Bean
@@ -86,6 +97,20 @@ public class EventHistoryAutoConfiguration {
return new Gb32960FrameController(service);
}
@Bean
@ConditionalOnBean(TdengineHistoryReader.class)
@ConditionalOnMissingBean
public Jt808LocationHistoryController jt808LocationHistoryController(TdengineHistoryReader reader) {
return new Jt808LocationHistoryController(reader);
}
@Bean
@ConditionalOnBean(TdengineHistoryReader.class)
@ConditionalOnMissingBean
public TelemetryFieldHistoryController telemetryFieldHistoryController(TdengineHistoryReader reader) {
return new TelemetryFieldHistoryController(reader);
}
static Path archiveRoot(String value) {
if (value == null || value.isBlank()) {
return Path.of(System.getProperty("java.io.tmpdir"), "lingniu-archive");

View File

@@ -3,14 +3,22 @@ package com.lingniu.ingest.eventhistory;
import com.lingniu.ingest.eventfilestore.EventFileStore;
import com.lingniu.ingest.eventfilestore.EventFileRecord;
import com.lingniu.ingest.protocol.gb32960.codec.Gb32960MessageDecoder;
import jakarta.servlet.http.HttpServletRequest;
import org.junit.jupiter.api.Test;
import org.junit.jupiter.api.io.TempDir;
import org.springframework.http.HttpMethod;
import org.springframework.test.web.servlet.MockMvc;
import org.springframework.web.bind.annotation.GetMapping;
import org.springframework.web.bind.annotation.RestController;
import org.springframework.web.servlet.resource.NoResourceFoundException;
import java.io.IOException;
import java.nio.file.Path;
import java.util.List;
import static org.mockito.Mockito.mock;
import static org.mockito.Mockito.when;
import static org.assertj.core.api.Assertions.assertThat;
import static org.springframework.test.web.servlet.request.MockMvcRequestBuilders.get;
import static org.springframework.test.web.servlet.result.MockMvcResultMatchers.jsonPath;
import static org.springframework.test.web.servlet.result.MockMvcResultMatchers.status;
@@ -52,6 +60,32 @@ class ApiExceptionHandlerTest {
.andExpect(jsonPath("$.message").value("vin is required for gb32960 snapshots"));
}
@Test
void ioExceptionReturnsServiceUnavailable() throws Exception {
MockMvc mvc = standaloneSetup(new IoFailureController())
.setControllerAdvice(new ApiExceptionHandler())
.build();
mvc.perform(get("/io"))
.andExpect(status().isServiceUnavailable())
.andExpect(jsonPath("$.message").value("tdengine history query failed"))
.andExpect(jsonPath("$.path").value("/io"));
}
@Test
void noResourceReturnsNotFound() {
HttpServletRequest request = mock(HttpServletRequest.class);
when(request.getRequestURI()).thenReturn("/missing");
var response = new ApiExceptionHandler().noResource(
new NoResourceFoundException(HttpMethod.GET, "/missing"),
request);
assertThat(response.getStatusCode().value()).isEqualTo(404);
assertThat(response.getBody()).isNotNull();
assertThat(response.getBody().message()).isEqualTo("resource not found: /missing");
}
private Gb32960DecodedFrameService service() {
EventFileStore store = new EventFileStore() {
@Override
@@ -69,4 +103,12 @@ class ApiExceptionHandlerTest {
archiveRoot,
null);
}
@RestController
private static final class IoFailureController {
@GetMapping("/io")
String io() throws IOException {
throw new IOException("tdengine history query failed");
}
}
}

View File

@@ -39,84 +39,6 @@ class EventHistoryControllerTest {
assertThat(store.lastQuery.vin()).isEqualTo("VIN001");
}
@Test
void exportsRecordsAsCsv() throws Exception {
EventHistoryController controller = new EventHistoryController(new CapturingStore(List.of(record("event-1"))));
String csv = controller.exportCsv(
ProtocolId.GB32960,
"2026-06-22",
"2026-06-22",
EventFileQuery.Order.ASC,
100);
assertThat(csv).startsWith("event_id,protocol,event_type,vin,event_time,ingest_time,raw_archive_uri,payload_json\n");
assertThat(csv).contains("event-1,GB32960,REALTIME,VIN001");
assertThat(csv).contains("\"{\"\"fields\"\":[]}\"");
}
@Test
void exportsConfiguredTelemetryFieldsAsCsvColumns() throws Exception {
EventHistoryController controller = new EventHistoryController(new CapturingStore(List.of(
new EventFileRecord(
"media-1",
ProtocolId.JT1078,
"MEDIA_META",
"VIN001",
Instant.parse("2026-06-22T08:00:00Z"),
Instant.parse("2026-06-22T08:00:01Z"),
"file:///archive/jt1078/media-1.rtp",
Map.of(),
"""
{"fields":[
{"key":"media_archive_ref","value":"file:///archive/jt1078/media-1.rtp"},
{"key":"passthrough_size_bytes","value":"128"}
]}
"""))));
String csv = controller.exportCsv(
ProtocolId.JT1078,
"2026-06-22",
"2026-06-22",
EventFileQuery.Order.ASC,
100,
null,
"media_archive_ref,passthrough_size_bytes");
assertThat(csv).startsWith("event_id,protocol,event_type,vin,event_time,ingest_time,raw_archive_uri,media_archive_ref,passthrough_size_bytes,payload_json\n");
assertThat(csv).contains("media-1,JT1078,MEDIA_META,VIN001,2026-06-22T08:00:00Z,2026-06-22T08:00:01Z,file:///archive/jt1078/media-1.rtp,file:///archive/jt1078/media-1.rtp,128,");
}
@Test
void exportsConfiguredMetadataFieldsAsCsvColumns() throws Exception {
EventHistoryController controller = new EventHistoryController(new CapturingStore(List.of(
new EventFileRecord(
"media-2",
ProtocolId.JT1078,
"MEDIA_META",
"VIN001",
Instant.parse("2026-06-22T08:00:00Z"),
Instant.parse("2026-06-22T08:00:01Z"),
"file:///archive/jt1078/media-2.rtp",
Map.of(
"channelId", "3",
"segmentSizeBytes", "512",
"archiveKey", "jt1078/2026/06/22/VIN001/ch-3/dt-0/1782115200.media"),
"{\"fields\":[]}"))));
String csv = controller.exportCsv(
ProtocolId.JT1078,
"2026-06-22",
"2026-06-22",
EventFileQuery.Order.ASC,
100,
null,
"metadata.channelId,metadata.segmentSizeBytes,metadata.archiveKey");
assertThat(csv).startsWith("event_id,protocol,event_type,vin,event_time,ingest_time,raw_archive_uri,metadata.channelId,metadata.segmentSizeBytes,metadata.archiveKey,payload_json\n");
assertThat(csv).contains("media-2,JT1078,MEDIA_META,VIN001,2026-06-22T08:00:00Z,2026-06-22T08:00:01Z,file:///archive/jt1078/media-2.rtp,3,512,jt1078/2026/06/22/VIN001/ch-3/dt-0/1782115200.media,");
}
private static EventFileRecord record(String eventId) {
return new EventFileRecord(
eventId,

View File

@@ -10,9 +10,16 @@ import com.lingniu.ingest.eventfilestore.EventFileQuery;
import com.lingniu.ingest.eventfilestore.EventFileRecord;
import com.lingniu.ingest.eventfilestore.EventFileStore;
import com.lingniu.ingest.sink.mq.proto.RawArchiveRef;
import com.lingniu.ingest.sink.mq.proto.RawFrameFactPayload;
import com.lingniu.ingest.sink.mq.proto.TelemetryField;
import com.lingniu.ingest.sink.mq.proto.TelemetrySnapshot;
import com.lingniu.ingest.sink.mq.proto.VehicleEnvelope;
import com.lingniu.ingest.sink.mq.proto.LocationPayload;
import com.lingniu.ingest.sink.mq.proto.ParseStatusProto;
import com.lingniu.ingest.tdenginehistory.TdengineHistoryWriter;
import com.lingniu.ingest.tdenginehistory.TdengineLocationRow;
import com.lingniu.ingest.tdenginehistory.TdengineRawFrameRow;
import com.lingniu.ingest.tdenginehistory.TdengineTelemetryFieldRow;
import org.junit.jupiter.api.Test;
import java.io.IOException;
@@ -118,6 +125,99 @@ class EventHistoryEnvelopeIngestorTest {
assertThat(payload.toString()).doesNotContain("rawBytes");
}
@Test
void tryIngestWritesRawLocationAndTelemetryFactsToTdengineWhenWriterExists() throws Exception {
CapturingStore store = new CapturingStore();
CapturingTdengineWriter tdengineWriter = new CapturingTdengineWriter();
EventHistoryEnvelopeIngestor ingestor = new EventHistoryEnvelopeIngestor(
store, new TelemetryEnvelopeRecordMapper(), tdengineWriter);
VehicleEnvelope envelope = VehicleEnvelope.newBuilder()
.setSchemaVersion("1.0")
.setEventId("jt808-location-1")
.setVin("VINJT808001")
.setSource("JT808")
.setEventTimeMs(1_782_112_400_000L)
.setIngestTimeMs(1_782_112_401_000L)
.putMetadata("vehicle_key", "jt808:g7gps")
.putMetadata("frame_id", "frame-jt808-1")
.setRawArchive(RawArchiveRef.newBuilder()
.setUri("archive://jt808/2026/06/29/frame-jt808-1.bin")
.setSizeBytes(68))
.setRawFrameFact(RawFrameFactPayload.newBuilder()
.setFrameId("frame-jt808-1")
.setVehicleKey("jt808:g7gps")
.setVin("VINJT808001")
.setPhone("013800000000")
.setMessageId(0x0200)
.setRawUri("archive://jt808/2026/06/29/frame-jt808-1.bin")
.setRawSizeBytes(68)
.setParseStatus(ParseStatusProto.PARSE_STATUS_SUCCEEDED))
.setLocation(LocationPayload.newBuilder()
.setLongitude(113.12)
.setLatitude(23.45)
.setSpeedKmh(42.5))
.setTelemetrySnapshot(TelemetrySnapshot.newBuilder()
.setEventType("LOCATION")
.setRawArchiveUri("archive://jt808/2026/06/29/frame-jt808-1.bin")
.addFields(TelemetryField.newBuilder()
.setKey("location.speedKmh")
.setValueType("DOUBLE")
.setValue("42.5")
.setUnit("km/h")
.setQuality("GOOD")
.setSourcePath("JT808.0x0200.speed")))
.build();
EnvelopeIngestResult result = ingestor.tryIngest(envelope.toByteArray());
assertThat(result.status()).isEqualTo(EnvelopeIngestResult.Status.STORED);
assertThat(store.records).hasSize(1);
assertThat(tdengineWriter.rawFrames)
.extracting(TdengineRawFrameRow::frameId)
.containsExactly("frame-jt808-1");
assertThat(tdengineWriter.locations)
.extracting(TdengineLocationRow::factId)
.containsExactly("jt808-location-1");
assertThat(tdengineWriter.locations.getFirst().longitude()).isEqualTo(113.12);
assertThat(tdengineWriter.telemetryFields)
.extracting(TdengineTelemetryFieldRow::fieldKey)
.containsExactly("location.speedKmh");
assertThat(tdengineWriter.telemetryFields.getFirst().valueDouble()).isEqualTo(42.5);
}
@Test
void tryIngestAllBatchesFileStoreAndTdengineWrites() throws Exception {
CapturingStore store = new CapturingStore();
CapturingTdengineWriter tdengineWriter = new CapturingTdengineWriter();
EventHistoryEnvelopeIngestor ingestor = new EventHistoryEnvelopeIngestor(
store, new TelemetryEnvelopeRecordMapper(), tdengineWriter);
VehicleEnvelope first = jt808LocationEnvelope("jt808-location-1", "frame-jt808-1", "013800000001");
VehicleEnvelope second = jt808LocationEnvelope("jt808-location-2", "frame-jt808-2", "013800000002");
List<EnvelopeIngestResult> results = ingestor.tryIngestAll(List.of(
first.toByteArray(),
new byte[]{0x01, 0x02},
second.toByteArray()));
assertThat(results).extracting(EnvelopeIngestResult::status)
.containsExactly(
EnvelopeIngestResult.Status.STORED,
EnvelopeIngestResult.Status.INVALID_ENVELOPE,
EnvelopeIngestResult.Status.STORED);
assertThat(store.appendAllCalls).isEqualTo(1);
assertThat(store.records).extracting(EventFileRecord::eventId)
.containsExactly("jt808-location-1", "jt808-location-2");
assertThat(tdengineWriter.rawFrames).extracting(TdengineRawFrameRow::frameId)
.containsExactly("frame-jt808-1", "frame-jt808-2");
assertThat(tdengineWriter.locations).extracting(TdengineLocationRow::factId)
.containsExactly("jt808-location-1", "jt808-location-2");
assertThat(tdengineWriter.telemetryFields)
.extracting(TdengineTelemetryFieldRow::fieldKey)
.containsExactly("location.speedKmh", "location.speedKmh");
}
private static VehicleEnvelope envelope(String eventId) {
return VehicleEnvelope.newBuilder()
.setEventId(eventId)
@@ -136,11 +236,52 @@ class EventHistoryEnvelopeIngestorTest {
.build();
}
private static VehicleEnvelope jt808LocationEnvelope(String eventId, String frameId, String phone) {
return VehicleEnvelope.newBuilder()
.setSchemaVersion("1.0")
.setEventId(eventId)
.setVin("VINJT808001")
.setSource("JT808")
.setEventTimeMs(1_782_112_400_000L)
.setIngestTimeMs(1_782_112_401_000L)
.putMetadata("vehicle_key", "jt808:" + phone)
.putMetadata("frame_id", frameId)
.setRawArchive(RawArchiveRef.newBuilder()
.setUri("archive://jt808/2026/06/29/" + frameId + ".bin")
.setSizeBytes(68))
.setRawFrameFact(RawFrameFactPayload.newBuilder()
.setFrameId(frameId)
.setVehicleKey("jt808:" + phone)
.setVin("VINJT808001")
.setPhone(phone)
.setMessageId(0x0200)
.setRawUri("archive://jt808/2026/06/29/" + frameId + ".bin")
.setRawSizeBytes(68)
.setParseStatus(ParseStatusProto.PARSE_STATUS_SUCCEEDED))
.setLocation(LocationPayload.newBuilder()
.setLongitude(113.12)
.setLatitude(23.45)
.setSpeedKmh(42.5))
.setTelemetrySnapshot(TelemetrySnapshot.newBuilder()
.setEventType("LOCATION")
.setRawArchiveUri("archive://jt808/2026/06/29/" + frameId + ".bin")
.addFields(TelemetryField.newBuilder()
.setKey("location.speedKmh")
.setValueType("DOUBLE")
.setValue("42.5")
.setUnit("km/h")
.setQuality("GOOD")
.setSourcePath("JT808.0x0200.speed")))
.build();
}
private static final class CapturingStore implements EventFileStore {
private final List<EventFileRecord> records = new ArrayList<>();
private int appendAllCalls;
@Override
public void appendAll(List<EventFileRecord> records) {
appendAllCalls++;
this.records.addAll(records);
}
@@ -149,4 +290,25 @@ class EventHistoryEnvelopeIngestorTest {
return List.copyOf(records);
}
}
private static final class CapturingTdengineWriter implements TdengineHistoryWriter {
private final List<TdengineRawFrameRow> rawFrames = new ArrayList<>();
private final List<TdengineLocationRow> locations = new ArrayList<>();
private final List<TdengineTelemetryFieldRow> telemetryFields = new ArrayList<>();
@Override
public void appendRawFrames(List<TdengineRawFrameRow> rows) {
rawFrames.addAll(rows);
}
@Override
public void appendLocations(List<TdengineLocationRow> rows) {
locations.addAll(rows);
}
@Override
public void appendTelemetryFields(List<TdengineTelemetryFieldRow> rows) {
telemetryFields.addAll(rows);
}
}
}

View File

@@ -0,0 +1,147 @@
package com.lingniu.ingest.eventhistory;
import com.lingniu.ingest.tdenginehistory.TdengineHistoryReader;
import com.lingniu.ingest.tdenginehistory.TdengineLocationQuery;
import com.lingniu.ingest.tdenginehistory.TdengineLocationRow;
import com.lingniu.ingest.tdenginehistory.TdenginePage;
import com.lingniu.ingest.tdenginehistory.TdenginePageCursor;
import com.lingniu.ingest.tdenginehistory.TdengineQueryOrder;
import org.junit.jupiter.api.Test;
import org.springframework.http.HttpStatus;
import org.springframework.web.server.ResponseStatusException;
import java.io.IOException;
import java.time.Instant;
import java.util.List;
import java.util.Optional;
import static org.assertj.core.api.Assertions.assertThat;
import static org.assertj.core.api.Assertions.assertThatThrownBy;
class Jt808LocationHistoryControllerTest {
@Test
void queriesLocationsByPhoneWithKeysetCursor() throws Exception {
CapturingReader reader = new CapturingReader(new TdenginePage<>(
List.of(row("fact-1", "2026-06-29T05:00:01Z")),
Optional.of(new TdenginePageCursor(Instant.parse("2026-06-29T05:00:01Z"), "fact-1"))));
Jt808LocationHistoryController controller = new Jt808LocationHistoryController(reader);
Jt808LocationHistoryController.LocationPageResponse response = controller.locations(
"g7gps",
"2026-06-29T13:00:00+08:00",
"2026-06-29T13:10:00+08:00",
TdengineQueryOrder.DESC,
50,
"2026-06-29T05:00:00Z",
"fact-0");
assertThat(reader.query.protocol()).isEqualTo("JT808");
assertThat(reader.query.vehicleKey()).isEqualTo("jt808:g7gps");
assertThat(reader.query.order()).isEqualTo(TdengineQueryOrder.DESC);
assertThat(reader.query.limit()).isEqualTo(50);
assertThat(reader.query.cursor()).isEqualTo(new TdenginePageCursor(
Instant.parse("2026-06-29T05:00:00Z"), "fact-0"));
assertThat(response.items())
.extracting(Jt808LocationHistoryController.LocationResponse::factId)
.containsExactly("fact-1");
assertThat(response.items().getFirst().longitude()).isEqualTo(113.12);
assertThat(response.nextCursor()).isEqualTo(new Jt808LocationHistoryController.CursorResponse(
"2026-06-29T05:00:01Z", "fact-1"));
}
@Test
void fallsBackToRawArchiveUriInMetadataWhenLocationRawUriIsBlank() throws Exception {
CapturingReader reader = new CapturingReader(new TdenginePage<>(
List.of(row(
"fact-blank-raw",
"2026-06-29T05:00:01Z",
"",
"{\"rawArchiveUri\":\"archive://jt808/metadata-frame.bin\"}")),
Optional.empty()));
Jt808LocationHistoryController controller = new Jt808LocationHistoryController(reader);
Jt808LocationHistoryController.LocationPageResponse response = controller.locations(
"g7gps",
"2026-06-29T13:00:00+08:00",
"2026-06-29T13:10:00+08:00",
TdengineQueryOrder.DESC,
50,
null,
null);
assertThat(response.items().getFirst().rawUri())
.isEqualTo("archive://jt808/metadata-frame.bin");
}
@Test
void rejectsBlankPhone() {
Jt808LocationHistoryController controller = new Jt808LocationHistoryController(
new CapturingReader(new TdenginePage<>(List.of(), Optional.empty())));
assertThatThrownBy(() -> controller.locations(
" ",
"2026-06-29T13:00:00+08:00",
"2026-06-29T13:10:00+08:00",
TdengineQueryOrder.DESC,
50,
null,
null))
.isInstanceOfSatisfying(ResponseStatusException.class, ex ->
assertThat(ex.getStatusCode()).isEqualTo(HttpStatus.BAD_REQUEST));
}
private static TdengineLocationRow row(String factId, String ts) {
return row(factId, ts, "archive://jt808/frame-1.bin", "{\"messageId\":\"512\"}");
}
private static TdengineLocationRow row(String factId, String ts, String rawUri, String metadataJson) {
Instant instant = Instant.parse(ts);
return new TdengineLocationRow(
instant,
factId,
"frame-1",
instant.plusMillis(500),
113.12,
23.45,
8.0,
42.5,
90.0,
1L,
3L,
null,
rawUri,
metadataJson,
"JT808",
"jt808:g7gps",
"",
"g7gps");
}
private static final class CapturingReader implements TdengineHistoryReader {
private final TdenginePage<TdengineLocationRow> response;
private TdengineLocationQuery query;
private CapturingReader(TdenginePage<TdengineLocationRow> response) {
this.response = response;
}
@Override
public TdenginePage<com.lingniu.ingest.tdenginehistory.TdengineRawFrameRow> queryRawFrames(
com.lingniu.ingest.tdenginehistory.TdengineRawFrameQuery query) {
throw new UnsupportedOperationException();
}
@Override
public TdenginePage<TdengineLocationRow> queryLocations(TdengineLocationQuery query) throws IOException {
this.query = query;
return response;
}
@Override
public TdenginePage<com.lingniu.ingest.tdenginehistory.TdengineTelemetryFieldRow> queryTelemetryFields(
com.lingniu.ingest.tdenginehistory.TdengineTelemetryFieldQuery query) {
throw new UnsupportedOperationException();
}
}
}

View File

@@ -0,0 +1,152 @@
package com.lingniu.ingest.eventhistory;
import com.lingniu.ingest.tdenginehistory.TdengineHistoryReader;
import com.lingniu.ingest.tdenginehistory.TdengineLocationQuery;
import com.lingniu.ingest.tdenginehistory.TdengineLocationRow;
import com.lingniu.ingest.tdenginehistory.TdenginePage;
import com.lingniu.ingest.tdenginehistory.TdenginePageCursor;
import com.lingniu.ingest.tdenginehistory.TdengineQueryOrder;
import com.lingniu.ingest.tdenginehistory.TdengineRawFrameQuery;
import com.lingniu.ingest.tdenginehistory.TdengineRawFrameRow;
import com.lingniu.ingest.tdenginehistory.TdengineTelemetryFieldQuery;
import com.lingniu.ingest.tdenginehistory.TdengineTelemetryFieldRow;
import org.junit.jupiter.api.Test;
import org.springframework.http.HttpStatus;
import org.springframework.web.server.ResponseStatusException;
import java.io.IOException;
import java.time.Instant;
import java.util.List;
import java.util.Optional;
import static org.assertj.core.api.Assertions.assertThat;
import static org.assertj.core.api.Assertions.assertThatThrownBy;
class TelemetryFieldHistoryControllerTest {
@Test
void queriesGb32960FieldByVinWithKeysetCursor() throws Exception {
CapturingReader reader = new CapturingReader(new TdenginePage<>(
List.of(row("field-1", "2026-06-29T05:00:01Z")),
Optional.of(new TdenginePageCursor(Instant.parse("2026-06-29T05:00:01Z"), "field-1"))));
TelemetryFieldHistoryController controller = new TelemetryFieldHistoryController(reader);
TelemetryFieldHistoryController.TelemetryFieldPageResponse response = controller.fields(
"GB32960",
"FUEL_CELL_V2016.hydrogenComsuxxx",
null,
"LNVFC000000000001",
null,
"2026-06-29T13:00:00+08:00",
"2026-06-29T13:10:00+08:00",
TdengineQueryOrder.ASC,
100,
"2026-06-29T05:00:00Z",
"field-0");
assertThat(reader.query.protocol()).isEqualTo("GB32960");
assertThat(reader.query.vehicleKey()).isEqualTo("LNVFC000000000001");
assertThat(reader.query.fieldKey()).isEqualTo("FUEL_CELL_V2016.hydrogenComsuxxx");
assertThat(reader.query.order()).isEqualTo(TdengineQueryOrder.ASC);
assertThat(reader.query.limit()).isEqualTo(100);
assertThat(reader.query.cursor()).isEqualTo(new TdenginePageCursor(
Instant.parse("2026-06-29T05:00:00Z"), "field-0"));
assertThat(response.items())
.extracting(TelemetryFieldHistoryController.TelemetryFieldResponse::factId)
.containsExactly("field-1");
assertThat(response.items().getFirst().valueDouble()).isEqualTo(3.14);
assertThat(response.nextCursor()).isEqualTo(new TelemetryFieldHistoryController.CursorResponse(
"2026-06-29T05:00:01Z", "field-1"));
}
@Test
void mapsJt808PhoneToVehicleKeyWhenVehicleKeyIsAbsent() throws Exception {
CapturingReader reader = new CapturingReader(new TdenginePage<>(List.of(), Optional.empty()));
TelemetryFieldHistoryController controller = new TelemetryFieldHistoryController(reader);
controller.fields(
"JT808",
"location.speed_kmh",
null,
null,
"g7gps",
"2026-06-29",
"2026-06-29",
TdengineQueryOrder.DESC,
50,
null,
null);
assertThat(reader.query.vehicleKey()).isEqualTo("jt808:g7gps");
}
@Test
void rejectsMissingVehicleIdentity() {
TelemetryFieldHistoryController controller = new TelemetryFieldHistoryController(
new CapturingReader(new TdenginePage<>(List.of(), Optional.empty())));
assertThatThrownBy(() -> controller.fields(
"GB32960",
"vehicle.speed",
null,
null,
null,
"2026-06-29",
"2026-06-29",
TdengineQueryOrder.DESC,
50,
null,
null))
.isInstanceOfSatisfying(ResponseStatusException.class, ex ->
assertThat(ex.getStatusCode()).isEqualTo(HttpStatus.BAD_REQUEST));
}
private static TdengineTelemetryFieldRow row(String factId, String ts) {
Instant instant = Instant.parse(ts);
return new TdengineTelemetryFieldRow(
instant,
factId,
"frame-1",
instant.plusMillis(500),
"FUEL_CELL_V2016.hydrogenComsuxxx",
"DOUBLE",
"3.14",
3.14,
null,
"kg/100km",
"GOOD",
"FUEL_CELL_V2016.hydrogenComsuxxx",
"archive://gb32960/frame-1.bin",
"{\"cmd\":\"REALTIME_REPORT\"}",
"GB32960",
"LNVFC000000000001",
"LNVFC000000000001",
"");
}
private static final class CapturingReader implements TdengineHistoryReader {
private final TdenginePage<TdengineTelemetryFieldRow> response;
private TdengineTelemetryFieldQuery query;
private CapturingReader(TdenginePage<TdengineTelemetryFieldRow> response) {
this.response = response;
}
@Override
public TdenginePage<TdengineRawFrameRow> queryRawFrames(TdengineRawFrameQuery query) {
throw new UnsupportedOperationException();
}
@Override
public TdenginePage<TdengineLocationRow> queryLocations(TdengineLocationQuery query) {
throw new UnsupportedOperationException();
}
@Override
public TdenginePage<TdengineTelemetryFieldRow> queryTelemetryFields(TdengineTelemetryFieldQuery query)
throws IOException {
this.query = query;
return response;
}
}
}

View File

@@ -7,8 +7,11 @@ import com.lingniu.ingest.eventhistory.EventHistoryController;
import com.lingniu.ingest.eventhistory.EventHistoryEnvelopeIngestor;
import com.lingniu.ingest.eventhistory.Gb32960DecodedFrameService;
import com.lingniu.ingest.eventhistory.Gb32960FrameController;
import com.lingniu.ingest.eventhistory.Jt808LocationHistoryController;
import com.lingniu.ingest.eventhistory.TelemetryFieldHistoryController;
import com.lingniu.ingest.eventhistory.TelemetryEnvelopeRecordMapper;
import com.lingniu.ingest.protocol.gb32960.codec.Gb32960MessageDecoder;
import com.lingniu.ingest.tdenginehistory.TdengineHistoryReader;
import com.lingniu.ingest.sink.archive.config.SinkArchiveProperties;
import org.junit.jupiter.api.Test;
import org.springframework.boot.autoconfigure.AutoConfigurations;
@@ -63,6 +66,22 @@ class EventHistoryAutoConfigurationTest {
});
}
@Test
void createsJt808LocationHistoryControllerWhenTdengineReaderExists() {
contextRunner
.withPropertyValues("lingniu.ingest.event-history.enabled=true")
.withBean(TdengineHistoryReader.class, () -> mock(TdengineHistoryReader.class))
.run(context -> assertThat(context).hasSingleBean(Jt808LocationHistoryController.class));
}
@Test
void createsTelemetryFieldHistoryControllerWhenTdengineReaderExists() {
contextRunner
.withPropertyValues("lingniu.ingest.event-history.enabled=true")
.withBean(TdengineHistoryReader.class, () -> mock(TdengineHistoryReader.class))
.run(context -> assertThat(context).hasSingleBean(TelemetryFieldHistoryController.class));
}
@Test
void parsesFileUriArchiveRoot() {
assertThat(EventHistoryAutoConfiguration.archiveRoot("file:///tmp/lingniu-test-archive"))

View File

@@ -0,0 +1,30 @@
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0">
<modelVersion>4.0.0</modelVersion>
<parent>
<groupId>com.lingniu.ingest</groupId>
<artifactId>lingniu-vehicle-ingest</artifactId>
<version>0.1.0-SNAPSHOT</version>
<relativePath>../../../pom.xml</relativePath>
</parent>
<artifactId>raw-archive-store</artifactId>
<name>raw-archive-store</name>
<description>Raw frame archive writer and reader contracts.</description>
<dependencies>
<dependency>
<groupId>com.lingniu.ingest</groupId>
<artifactId>ingest-api</artifactId>
</dependency>
<dependency>
<groupId>org.junit.jupiter</groupId>
<artifactId>junit-jupiter</artifactId>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.assertj</groupId>
<artifactId>assertj-core</artifactId>
<scope>test</scope>
</dependency>
</dependencies>
</project>

View File

@@ -0,0 +1,110 @@
package com.lingniu.ingest.rawarchive;
import java.io.IOException;
import java.nio.charset.StandardCharsets;
import java.nio.file.Files;
import java.nio.file.LinkOption;
import java.nio.file.Path;
import java.nio.file.StandardOpenOption;
import java.security.MessageDigest;
import java.security.NoSuchAlgorithmException;
import java.time.ZoneId;
import java.util.HexFormat;
public final class LocalRawArchiveStore implements RawArchiveWriter, RawArchiveReader {
private static final ZoneId PARTITION_ZONE = ZoneId.of("Asia/Shanghai");
private final Path root;
private final String uriPrefix;
public LocalRawArchiveStore(Path root, String uriPrefix) {
if (root == null) {
throw new IllegalArgumentException("root must not be null");
}
this.root = root.toAbsolutePath().normalize();
String prefix = uriPrefix == null || uriPrefix.isBlank() ? "archive://raw" : uriPrefix.trim();
this.uriPrefix = prefix.endsWith("/") ? prefix.substring(0, prefix.length() - 1) : prefix;
}
@Override
public RawArchiveReceipt write(RawArchiveWriteRequest request) throws IOException {
String key = key(request);
Path target = resolveKey(key);
Files.createDirectories(target.getParent());
rejectSymlinkPath(target);
byte[] bytes = request.rawBytes();
Files.write(target, bytes, StandardOpenOption.CREATE_NEW, StandardOpenOption.WRITE);
return new RawArchiveReceipt(uriPrefix + "/" + key, checksum(bytes), bytes.length);
}
@Override
public byte[] read(String rawUri) throws IOException {
String key = keyFromUri(rawUri);
Path target = resolveKey(key);
rejectSymlinkPath(target);
return Files.readAllBytes(target);
}
private String key(RawArchiveWriteRequest request) {
var date = request.receivedAt().atZone(PARTITION_ZONE).toLocalDate();
return "%04d/%02d/%02d/%s/%s/%s.bin".formatted(
date.getYear(),
date.getMonthValue(),
date.getDayOfMonth(),
request.protocol().name(),
safeSegment(request.vehicleKey()),
safeSegment(request.frameId()));
}
private String keyFromUri(String rawUri) {
if (rawUri == null || rawUri.isBlank()) {
throw new IllegalArgumentException("rawUri must not be blank");
}
String normalized = rawUri.trim();
String expected = uriPrefix + "/";
if (!normalized.startsWith(expected)) {
throw new IllegalArgumentException("rawUri does not use expected prefix: " + rawUri);
}
return normalized.substring(expected.length());
}
private Path resolveKey(String key) {
Path target = root.resolve(key).normalize();
if (!target.startsWith(root)) {
throw new IllegalArgumentException("raw archive key escapes root: " + key);
}
return target;
}
private void rejectSymlinkPath(Path target) throws IOException {
Path current = root;
Path relative = root.relativize(target);
for (Path segment : relative) {
current = current.resolve(segment);
if (Files.isSymbolicLink(current)) {
throw new IOException("raw archive path traverses symlink: " + current);
}
if (Files.notExists(current, LinkOption.NOFOLLOW_LINKS)) {
return;
}
}
}
private static String safeSegment(String value) {
String cleaned = value == null ? "" : value.trim();
if (cleaned.isBlank()) {
return "unknown";
}
return cleaned.replaceAll("[^A-Za-z0-9._-]", "_");
}
private static String checksum(byte[] bytes) {
try {
MessageDigest digest = MessageDigest.getInstance("SHA-256");
return "sha256:" + HexFormat.of().formatHex(digest.digest(bytes));
} catch (NoSuchAlgorithmException e) {
throw new IllegalStateException("SHA-256 is unavailable", e);
}
}
}

View File

@@ -0,0 +1,7 @@
package com.lingniu.ingest.rawarchive;
import java.io.IOException;
public interface RawArchiveReader {
byte[] read(String rawUri) throws IOException;
}

View File

@@ -0,0 +1,19 @@
package com.lingniu.ingest.rawarchive;
public record RawArchiveReceipt(String rawUri, String checksum, long sizeBytes) {
public RawArchiveReceipt {
rawUri = required(rawUri, "rawUri");
checksum = required(checksum, "checksum");
if (sizeBytes < 0) {
throw new IllegalArgumentException("sizeBytes must not be negative");
}
}
private static String required(String value, String field) {
String cleaned = value == null ? "" : value.trim();
if (cleaned.isBlank()) {
throw new IllegalArgumentException(field + " must not be blank");
}
return cleaned;
}
}

View File

@@ -0,0 +1,41 @@
package com.lingniu.ingest.rawarchive;
import com.lingniu.ingest.api.ProtocolId;
import java.time.Instant;
public record RawArchiveWriteRequest(
ProtocolId protocol,
String vehicleKey,
String frameId,
Instant receivedAt,
byte[] rawBytes
) {
public RawArchiveWriteRequest {
if (protocol == null) {
throw new IllegalArgumentException("protocol must not be null");
}
vehicleKey = required(vehicleKey, "vehicleKey");
frameId = required(frameId, "frameId");
if (receivedAt == null) {
throw new IllegalArgumentException("receivedAt must not be null");
}
if (rawBytes == null || rawBytes.length == 0) {
throw new IllegalArgumentException("rawBytes must not be empty");
}
rawBytes = rawBytes.clone();
}
@Override
public byte[] rawBytes() {
return rawBytes.clone();
}
private static String required(String value, String field) {
String cleaned = value == null ? "" : value.trim();
if (cleaned.isBlank()) {
throw new IllegalArgumentException(field + " must not be blank");
}
return cleaned;
}
}

View File

@@ -0,0 +1,7 @@
package com.lingniu.ingest.rawarchive;
import java.io.IOException;
public interface RawArchiveWriter {
RawArchiveReceipt write(RawArchiveWriteRequest request) throws IOException;
}

View File

@@ -0,0 +1,50 @@
package com.lingniu.ingest.rawarchive;
import com.lingniu.ingest.api.ProtocolId;
import org.junit.jupiter.api.Test;
import org.junit.jupiter.api.io.TempDir;
import java.nio.file.Path;
import java.time.Instant;
import java.util.HexFormat;
import static org.assertj.core.api.Assertions.assertThat;
import static org.assertj.core.api.Assertions.assertThatThrownBy;
class LocalRawArchiveStoreTest {
@TempDir
Path tempDir;
@Test
void writesReadableRawObjectWithArchiveUriAndChecksum() throws Exception {
LocalRawArchiveStore store = new LocalRawArchiveStore(tempDir, "archive://raw");
byte[] bytes = HexFormat.of().parseHex("7e020000007e");
RawArchiveReceipt receipt = store.write(new RawArchiveWriteRequest(
ProtocolId.JT808,
"jt808:013912345678",
"frame-1",
Instant.parse("2026-06-29T00:00:00Z"),
bytes));
assertThat(receipt.rawUri()).isEqualTo("archive://raw/2026/06/29/JT808/jt808_013912345678/frame-1.bin");
assertThat(receipt.checksum()).startsWith("sha256:");
assertThat(receipt.sizeBytes()).isEqualTo(bytes.length);
assertThat(store.read(receipt.rawUri())).containsExactly(bytes);
}
@Test
void rejectsBlankFrameId() {
LocalRawArchiveStore store = new LocalRawArchiveStore(tempDir, "archive://raw");
assertThatThrownBy(() -> store.write(new RawArchiveWriteRequest(
ProtocolId.GB32960,
"VIN001",
"",
Instant.parse("2026-06-29T00:00:00Z"),
new byte[]{1})))
.isInstanceOf(IllegalArgumentException.class)
.hasMessageContaining("frameId");
}
}

View File

@@ -4,15 +4,31 @@ import com.lingniu.ingest.api.event.RawArchiveKeys;
import com.lingniu.ingest.api.event.VehicleEvent;
import com.lingniu.ingest.api.sink.EventSink;
import java.io.ByteArrayInputStream;
import java.io.IOException;
import java.nio.file.FileAlreadyExistsException;
import java.nio.file.Files;
import java.nio.file.LinkOption;
import java.nio.file.Path;
import java.nio.file.StandardOpenOption;
import java.security.MessageDigest;
import java.security.NoSuchAlgorithmException;
import java.util.Arrays;
import java.util.HexFormat;
import java.util.concurrent.CompletableFuture;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;
public final class RawArchiveEventSink implements EventSink {
public final class RawArchiveEventSink implements EventSink, AutoCloseable {
private final ArchiveStore store;
private final Path root;
private final ExecutorService executor;
public RawArchiveEventSink(ArchiveStore store) {
this.store = store;
public RawArchiveEventSink(Path root) {
if (root == null) {
throw new IllegalArgumentException("root must not be null");
}
this.root = root.toAbsolutePath().normalize();
this.executor = Executors.newVirtualThreadPerTaskExecutor();
}
@Override
@@ -20,25 +36,115 @@ public final class RawArchiveEventSink implements EventSink {
return "raw-archive";
}
@Override
public boolean accepts(VehicleEvent event) {
return event instanceof VehicleEvent.RawArchive;
}
@Override
public CompletableFuture<Void> publish(VehicleEvent event) {
if (!(event instanceof VehicleEvent.RawArchive raw)) {
return CompletableFuture.completedFuture(null);
}
try {
byte[] bytes = raw.rawBytes();
if (bytes == null) {
throw new IllegalArgumentException("raw archive bytes must not be null");
}
store.put(RawArchiveKeys.key(raw), new ByteArrayInputStream(bytes), bytes.length);
return CompletableFuture.completedFuture(null);
} catch (Exception e) {
return CompletableFuture.failedFuture(e);
}
return CompletableFuture.runAsync(() -> write(raw), executor);
}
@Override
public boolean accepts(VehicleEvent event) {
return event instanceof VehicleEvent.RawArchive;
public void close() {
executor.shutdown();
}
public Path root() {
return root;
}
private void write(VehicleEvent.RawArchive raw) {
byte[] bytes = raw.rawBytes() == null ? new byte[0] : raw.rawBytes();
if (bytes.length == 0) {
return;
}
String key = RawArchiveKeys.key(raw);
Path target = resolveKey(key);
try {
createDirectoriesInsideRoot(target.getParent());
rejectSymlinkPath(target);
try {
Files.write(target, bytes, StandardOpenOption.CREATE_NEW, StandardOpenOption.WRITE);
} catch (FileAlreadyExistsException duplicate) {
byte[] existing = Files.readAllBytes(target);
if (!Arrays.equals(sha256(existing), sha256(bytes))) {
throw new IOException("raw archive duplicate key has different content: " + key, duplicate);
}
}
} catch (IOException e) {
throw new RawArchiveWriteException("failed to write raw archive " + RawArchiveKeys.logicalUri(key), e);
}
}
private Path resolveKey(String key) {
Path target = root.resolve(key == null ? "" : key).normalize();
if (!target.startsWith(root)) {
throw new IllegalArgumentException("raw archive key escapes root: " + key);
}
return target;
}
private void createDirectoriesInsideRoot(Path directory) throws IOException {
rejectEscapedPath(directory);
if (Files.notExists(root, LinkOption.NOFOLLOW_LINKS)) {
Files.createDirectories(root);
}
Path current = root;
Path relative = root.relativize(directory);
for (Path segment : relative) {
current = current.resolve(segment);
if (Files.isSymbolicLink(current)) {
throw new IOException("raw archive path traverses symlink: " + current);
}
if (Files.notExists(current, LinkOption.NOFOLLOW_LINKS)) {
Files.createDirectory(current);
} else if (!Files.isDirectory(current, LinkOption.NOFOLLOW_LINKS)) {
throw new IOException("raw archive path segment is not a directory: " + current);
}
}
}
private void rejectSymlinkPath(Path target) throws IOException {
rejectEscapedPath(target);
Path current = root;
Path relative = root.relativize(target);
for (Path segment : relative) {
current = current.resolve(segment);
if (Files.isSymbolicLink(current)) {
throw new IOException("raw archive path traverses symlink: " + current);
}
if (Files.notExists(current, LinkOption.NOFOLLOW_LINKS)) {
return;
}
}
}
private void rejectEscapedPath(Path path) {
if (!path.normalize().startsWith(root)) {
throw new IllegalArgumentException("raw archive path escapes root: " + path);
}
}
private static byte[] sha256(byte[] bytes) {
try {
return MessageDigest.getInstance("SHA-256").digest(bytes);
} catch (NoSuchAlgorithmException e) {
throw new IllegalStateException("SHA-256 is unavailable", e);
}
}
static String checksum(byte[] bytes) {
return "sha256:" + HexFormat.of().formatHex(sha256(bytes == null ? new byte[0] : bytes));
}
public static final class RawArchiveWriteException extends RuntimeException {
public RawArchiveWriteException(String message, Throwable cause) {
super(message, cause);
}
}
}

View File

@@ -5,31 +5,28 @@ import com.lingniu.ingest.sink.archive.LocalArchiveStore;
import com.lingniu.ingest.sink.archive.RawArchiveEventSink;
import org.springframework.boot.autoconfigure.AutoConfiguration;
import org.springframework.boot.autoconfigure.condition.ConditionalOnMissingBean;
import org.springframework.boot.autoconfigure.condition.ConditionalOnBean;
import org.springframework.boot.autoconfigure.condition.ConditionalOnProperty;
import org.springframework.boot.context.properties.EnableConfigurationProperties;
import org.springframework.context.annotation.Bean;
import java.nio.file.Path;
@AutoConfiguration
@EnableConfigurationProperties(SinkArchiveProperties.class)
@ConditionalOnProperty(
prefix = "lingniu.ingest.sink.archive",
name = "enabled",
havingValue = "true",
matchIfMissing = true)
@ConditionalOnProperty(prefix = "lingniu.ingest.sink.archive", name = "enabled", havingValue = "true", matchIfMissing = true)
public class SinkArchiveAutoConfiguration {
@Bean
@ConditionalOnMissingBean
@ConditionalOnProperty(prefix = "lingniu.ingest.sink.archive", name = "type", havingValue = "local", matchIfMissing = true)
public ArchiveStore archiveStore(SinkArchiveProperties properties) {
return new LocalArchiveStore(properties.getPath());
return new LocalArchiveStore(Path.of(properties.getPath()));
}
@Bean
@ConditionalOnMissingBean
@ConditionalOnBean(ArchiveStore.class)
public RawArchiveEventSink rawArchiveEventSink(ArchiveStore store) {
return new RawArchiveEventSink(store);
@ConditionalOnProperty(prefix = "lingniu.ingest.sink.archive", name = "type", havingValue = "local", matchIfMissing = true)
public RawArchiveEventSink rawArchiveEventSink(SinkArchiveProperties properties) {
return new RawArchiveEventSink(Path.of(properties.getPath()));
}
}

View File

@@ -7,7 +7,7 @@ public class SinkArchiveProperties {
private boolean enabled = true;
private String type = "local";
private String path = System.getProperty("java.io.tmpdir") + "/lingniu-archive";
private String path = "./archive/";
public boolean isEnabled() {
return enabled;

View File

@@ -1,63 +1,77 @@
package com.lingniu.ingest.sink.archive;
import com.lingniu.ingest.api.ProtocolId;
import com.lingniu.ingest.api.event.RawArchiveKeys;
import com.lingniu.ingest.api.event.VehicleEvent;
import org.junit.jupiter.api.Test;
import org.junit.jupiter.api.io.TempDir;
import java.io.InputStream;
import java.nio.file.Files;
import java.nio.file.Path;
import java.time.Instant;
import java.util.Map;
import java.util.concurrent.CompletionException;
import static org.assertj.core.api.Assertions.assertThat;
import static org.assertj.core.api.Assertions.assertThatThrownBy;
class RawArchiveEventSinkTest {
@TempDir
Path tempDir;
@Test
void failsFastWhenRawBytesAreNull() {
RawArchiveEventSink sink = new RawArchiveEventSink(new CapturingArchiveStore());
VehicleEvent.RawArchive raw = new VehicleEvent.RawArchive(
"event-1",
"VIN123",
ProtocolId.GB32960,
Instant.parse("2026-06-23T00:00:00Z"),
Instant.parse("2026-06-23T00:00:01Z"),
"trace-1",
Map.of(),
2,
1,
null);
void writesRawBytesUsingSharedArchiveKey() throws Exception {
RawArchiveEventSink sink = new RawArchiveEventSink(tempDir);
VehicleEvent.RawArchive raw = rawArchive(Map.of(), new byte[]{0x23, 0x23, 0x01});
sink.publish(raw).join();
String key = RawArchiveKeys.key(raw);
assertThat(Files.readAllBytes(tempDir.resolve(key))).containsExactly(0x23, 0x23, 0x01);
assertThat(RawArchiveKeys.logicalUri(key)).isEqualTo("archive://" + key);
}
@Test
void rejectsArchiveKeysThatEscapeRoot() {
RawArchiveEventSink sink = new RawArchiveEventSink(tempDir);
VehicleEvent.RawArchive raw = rawArchive(
Map.of(RawArchiveKeys.META_KEY, "../escape.bin"),
new byte[]{0x01});
assertThatThrownBy(() -> sink.publish(raw).join())
.isInstanceOf(CompletionException.class)
.hasCauseInstanceOf(IllegalArgumentException.class);
.hasCauseInstanceOf(IllegalArgumentException.class)
.hasMessageContaining("raw archive key escapes root");
}
private static final class CapturingArchiveStore implements ArchiveStore {
@Test
void rejectsSymlinkTraversalBeforeCreatingDirectories() throws Exception {
Path root = tempDir.resolve("archive");
Path outside = tempDir.resolve("outside");
Files.createDirectories(root);
Files.createDirectories(outside);
Files.createSymbolicLink(root.resolve("link"), outside);
@Override
public String put(String key, InputStream data, long length) {
return "archive://" + key;
}
RawArchiveEventSink sink = new RawArchiveEventSink(root);
VehicleEvent.RawArchive raw = rawArchive(
Map.of(RawArchiveKeys.META_KEY, "link/nested/raw.bin"),
new byte[]{0x01});
@Override
public String append(String key, byte[] chunk) {
return "archive://" + key;
}
assertThatThrownBy(() -> sink.publish(raw).join())
.hasRootCauseMessage("raw archive path traverses symlink: " + root.resolve("link"));
assertThat(outside.resolve("nested")).doesNotExist();
}
@Override
public InputStream get(String key) {
throw new UnsupportedOperationException();
}
@Override
public boolean exists(String key) {
return false;
}
@Override
public long size(String key) {
return 0;
}
private static VehicleEvent.RawArchive rawArchive(Map<String, String> metadata, byte[] rawBytes) {
return new VehicleEvent.RawArchive(
"raw-1",
"VIN001",
ProtocolId.GB32960,
Instant.parse("2026-06-22T08:00:00Z"),
Instant.parse("2026-06-22T08:00:01Z"),
"trace-1",
metadata,
0x02,
0,
rawBytes);
}
}

View File

@@ -16,6 +16,10 @@
<groupId>com.lingniu.ingest</groupId>
<artifactId>ingest-api</artifactId>
</dependency>
<dependency>
<groupId>com.lingniu.ingest</groupId>
<artifactId>ingest-facts</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter</artifactId>

View File

@@ -8,10 +8,17 @@ import com.lingniu.ingest.api.event.TelemetryFieldValue;
import com.lingniu.ingest.api.event.TelemetrySnapshot;
import com.lingniu.ingest.api.event.VehicleEvent;
import com.lingniu.ingest.api.event.VehicleEventTelemetrySnapshotMapper;
import com.lingniu.ingest.facts.FactIds;
import com.lingniu.ingest.facts.VehicleKey;
import com.lingniu.ingest.sink.mq.proto.ParseStatusProto;
import com.lingniu.ingest.sink.mq.proto.TelemetryField;
import com.lingniu.ingest.sink.mq.proto.TelemetrySnapshot.Builder;
import com.lingniu.ingest.sink.mq.proto.VehicleEnvelope;
import java.security.MessageDigest;
import java.security.NoSuchAlgorithmException;
import java.util.HexFormat;
/**
* 领域事件 → Protobuf Envelope 的纯函数映射。通过 switch 模式匹配穷尽处理,
* 新增 {@link VehicleEvent} 子类型时编译期就会提示补齐分支。
@@ -68,17 +75,39 @@ public final class EnvelopeMapper {
p.data() == null ? new byte[0] : p.data()))
.build());
case VehicleEvent.RawArchive ra -> {
int size = ra.rawBytes() == null ? 0 : ra.rawBytes().length;
byte[] rawBytes = ra.rawBytes() == null ? new byte[0] : ra.rawBytes();
int size = rawBytes.length;
String key = rawArchiveKey(ra);
String uri = rawArchiveUri(ra, key);
String phone = metadataValue(ra, "phone");
String vehicleKey = VehicleKey.derive(ra.source(), ra.vin(), phone, ra.eventId());
String checksum = sha256(rawBytes);
String frameId = FactIds.rawFrameId(
ra.source(), vehicleKey, ra.command(), ra.infoType(), ra.ingestTime(), checksum);
// RAW envelope 只携带 URI/size 引用信息,不把完整原始字节塞进 Kafka避免 topic 膨胀。
b.putMetadata(RawArchiveKeys.META_KEY, key);
b.putMetadata(RawArchiveKeys.META_URI, uri);
b.putMetadata(RawArchiveKeys.META_EVENT_ID, ra.eventId());
b.setRawArchive(com.lingniu.ingest.sink.mq.proto.RawArchiveRef.newBuilder()
.setUri(uri)
.setChecksum(checksum)
.setSizeBytes(size)
.build());
b.setRawFrameFact(com.lingniu.ingest.sink.mq.proto.RawFrameFactPayload.newBuilder()
.setFrameId(frameId)
.setVehicleKey(vehicleKey)
.setVin(nullToEmpty(ra.vin()))
.setPhone(phone)
.setMessageId(ra.command())
.setSubType(ra.infoType())
.setRawUri(uri)
.setChecksum(checksum)
.setRawSizeBytes(size)
.setParseStatus(parseStatus(ra))
.setParseError(parseError(ra))
.setPeer(metadataValue(ra, "peer"))
.putAllMetadata(ra.metadata() == null ? java.util.Map.of() : ra.metadata())
.build());
}
}
return b.build();
@@ -166,4 +195,45 @@ public final class EnvelopeMapper {
String uri = raw.metadata() == null ? "" : raw.metadata().getOrDefault(RawArchiveKeys.META_URI, "");
return uri == null || uri.isBlank() ? RawArchiveKeys.logicalUri(key) : uri;
}
private static String metadataValue(VehicleEvent event, String key) {
if (event.metadata() == null) {
return "";
}
return nullToEmpty(event.metadata().get(key));
}
private static ParseStatusProto parseStatus(VehicleEvent.RawArchive raw) {
if (hasTruthyMetadata(raw, "frameError")
|| hasTruthyMetadata(raw, "parseError")
|| hasTruthyMetadata(raw, "processingError")) {
return ParseStatusProto.PARSE_STATUS_FAILED;
}
return ParseStatusProto.PARSE_STATUS_NOT_PARSED;
}
private static String parseError(VehicleEvent.RawArchive raw) {
String frameError = metadataValue(raw, "frameErrorMessage");
if (!frameError.isBlank()) {
return frameError;
}
String parseError = metadataValue(raw, "parseErrorMessage");
if (!parseError.isBlank()) {
return parseError;
}
return metadataValue(raw, "processingErrorMessage");
}
private static boolean hasTruthyMetadata(VehicleEvent event, String key) {
return Boolean.parseBoolean(metadataValue(event, key));
}
private static String sha256(byte[] bytes) {
try {
MessageDigest digest = MessageDigest.getInstance("SHA-256");
return "sha256:" + HexFormat.of().formatHex(digest.digest(bytes == null ? new byte[0] : bytes));
} catch (NoSuchAlgorithmException e) {
throw new IllegalStateException("SHA-256 is unavailable", e);
}
}
}

View File

@@ -117,6 +117,7 @@ public final class KafkaEnvelopeConsumerFactory {
p.put(ConsumerConfig.ENABLE_AUTO_COMMIT_CONFIG, false);
p.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, consumer.getAutoOffsetReset());
p.put(ConsumerConfig.MAX_POLL_RECORDS_CONFIG, consumer.getMaxPollRecords());
p.put(ConsumerConfig.MAX_POLL_INTERVAL_MS_CONFIG, consumer.getMaxPollIntervalMillis());
return p;
}

View File

@@ -7,7 +7,10 @@ import org.apache.kafka.clients.consumer.ConsumerRecord;
import org.apache.kafka.clients.consumer.ConsumerRecords;
import java.time.Duration;
import java.util.ArrayList;
import java.util.Collection;
import java.util.LinkedHashMap;
import java.util.List;
import java.util.Map;
public final class KafkaEnvelopeConsumerWorker implements AutoCloseable {
@@ -33,22 +36,26 @@ public final class KafkaEnvelopeConsumerWorker implements AutoCloseable {
public int pollOnce(Duration timeout) {
ConsumerRecords<String, byte[]> records = consumer.poll(timeout == null ? Duration.ZERO : timeout);
int processed = 0;
Map<EnvelopeConsumerProcessor, List<EnvelopeConsumerRecord>> batches = new LinkedHashMap<>();
for (ConsumerRecord<String, byte[]> record : records) {
EnvelopeConsumerProcessor processor = processorsByTopic.get(record.topic());
if (processor == null) {
// worker 可能订阅多个 topic没有显式绑定处理器的 topic 不参与提交语义。
continue;
}
// EnvelopeConsumerProcessor 内部会把解析或业务错误转成 DLQ 记录,
// 这里保持 Kafka worker 的职责单一:轮询、分发、成功后提交 offset。
processor.process(new EnvelopeConsumerRecord(
batches.computeIfAbsent(processor, ignored -> new ArrayList<>()).add(new EnvelopeConsumerRecord(
record.topic(),
record.partition(),
record.offset(),
record.key(),
record.value()));
processed++;
}
int processed = 0;
for (Map.Entry<EnvelopeConsumerProcessor, List<EnvelopeConsumerRecord>> entry : batches.entrySet()) {
// EnvelopeConsumerProcessor 内部会把解析或业务错误转成 DLQ 记录,
// 这里保持 Kafka worker 的职责单一:轮询、批量分发、成功后提交 offset。
entry.getKey().processAll(entry.getValue());
processed += entry.getValue().size();
}
if (processed > 0) {
// commitSync 放在批次末尾,保证同一个 poll 批次内的消息按 Kafka offset 一起确认。

View File

@@ -93,6 +93,7 @@ public class SinkMqProperties {
private int loopBackoffMillis = 1000;
private String autoOffsetReset = "earliest";
private int maxPollRecords = 500;
private int maxPollIntervalMillis = 1800000;
/**
* Processor bean name -> Kafka binding。
*
@@ -115,6 +116,8 @@ public class SinkMqProperties {
public void setAutoOffsetReset(String autoOffsetReset) { this.autoOffsetReset = autoOffsetReset; }
public int getMaxPollRecords() { return maxPollRecords; }
public void setMaxPollRecords(int maxPollRecords) { this.maxPollRecords = maxPollRecords; }
public int getMaxPollIntervalMillis() { return maxPollIntervalMillis; }
public void setMaxPollIntervalMillis(int maxPollIntervalMillis) { this.maxPollIntervalMillis = maxPollIntervalMillis; }
public Map<String, Binding> getBindings() { return bindings; }
public void setBindings(Map<String, Binding> bindings) { this.bindings = bindings; }
}

View File

@@ -35,6 +35,10 @@ message VehicleEnvelope {
// 全字段内部遥测快照。新下游消费者优先读取这里,避免依赖协议字段名。
TelemetrySnapshot telemetry_snapshot = 50;
// 新一代事实模型RAW 帧索引与解析事实,均不携带完整 raw bytes。
RawFrameFactPayload raw_frame_fact = 60;
DecodedFactPayload decoded_fact = 61;
}
message RawArchiveRef {
@@ -58,6 +62,41 @@ message TelemetryField {
string source_path = 6;
}
enum ParseStatusProto {
PARSE_STATUS_UNSPECIFIED = 0;
PARSE_STATUS_NOT_PARSED = 1;
PARSE_STATUS_SUCCEEDED = 2;
PARSE_STATUS_FAILED = 3;
}
message RawFrameFactPayload {
string frame_id = 1;
string vehicle_key = 2;
string vin = 3;
string phone = 4;
int32 message_id = 5;
int32 sub_type = 6;
string raw_uri = 7;
string checksum = 8;
int64 raw_size_bytes = 9;
ParseStatusProto parse_status = 10;
string parse_error = 11;
string peer = 12;
map<string, string> metadata = 13;
}
message DecodedFactPayload {
string fact_id = 1;
string frame_id = 2;
string fact_type = 3;
string vehicle_key = 4;
string vin = 5;
string phone = 6;
string raw_uri = 7;
map<string, string> fields = 8;
map<string, string> metadata = 9;
}
message RealtimePayload {
optional double speed_kmh = 1;
optional double total_mileage_km = 2;

View File

@@ -90,5 +90,13 @@ class EnvelopeMapperTelemetrySnapshotTest {
assertThat(envelope.getRawArchive().getUri()).isEqualTo(RawArchiveKeys.logicalUri(key));
assertThat(envelope.getRawArchive().getSizeBytes()).isEqualTo(4);
assertThat(envelope.getMetadataMap()).containsEntry(RawArchiveKeys.META_URI, RawArchiveKeys.logicalUri(key));
assertThat(envelope.hasRawFrameFact()).isTrue();
assertThat(envelope.getRawFrameFact().getFrameId()).startsWith("rf_");
assertThat(envelope.getRawFrameFact().getVehicleKey()).isEqualTo("VIN-JT808-001");
assertThat(envelope.getRawFrameFact().getVin()).isEqualTo("VIN-JT808-001");
assertThat(envelope.getRawFrameFact().getMessageId()).isEqualTo(0x0200);
assertThat(envelope.getRawFrameFact().getRawUri()).isEqualTo(RawArchiveKeys.logicalUri(key));
assertThat(envelope.getRawFrameFact().getRawSizeBytes()).isEqualTo(4);
assertThat(envelope.getRawFrameFact().getChecksum()).startsWith("sha256:");
}
}

View File

@@ -40,6 +40,7 @@ class KafkaEnvelopeConsumerFactoryTest {
.allSatisfy(p -> {
assertThat(p.getProperty(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG)).isEqualTo("kafka-1:9092");
assertThat(p.get(ConsumerConfig.ENABLE_AUTO_COMMIT_CONFIG)).isEqualTo(false);
assertThat(p.get(ConsumerConfig.MAX_POLL_INTERVAL_MS_CONFIG)).isEqualTo(1800000);
});
}

View File

@@ -0,0 +1,65 @@
package com.lingniu.ingest.sink.mq;
import com.lingniu.ingest.sink.mq.proto.DecodedFactPayload;
import com.lingniu.ingest.sink.mq.proto.ParseStatusProto;
import com.lingniu.ingest.sink.mq.proto.RawFrameFactPayload;
import com.lingniu.ingest.sink.mq.proto.VehicleEnvelope;
import org.junit.jupiter.api.Test;
import static org.assertj.core.api.Assertions.assertThat;
class VehicleEnvelopeProtoCompatibilityTest {
@Test
void rawFrameFactPayloadRoundTripsThroughEnvelope() throws Exception {
VehicleEnvelope envelope = VehicleEnvelope.newBuilder()
.setSchemaVersion("2.0")
.setEventId("frame-1")
.setVin("VIN001")
.setSource("GB32960")
.setEventTimeMs(1782662400000L)
.setIngestTimeMs(1782662400001L)
.setRawFrameFact(RawFrameFactPayload.newBuilder()
.setFrameId("frame-1")
.setVehicleKey("VIN001")
.setMessageId(2)
.setRawUri("archive://raw/2026/06/29/GB32960/VIN001/frame-1.bin")
.setChecksum("sha256:abc")
.setRawSizeBytes(128)
.setParseStatus(ParseStatusProto.PARSE_STATUS_SUCCEEDED)
.build())
.build();
VehicleEnvelope parsed = VehicleEnvelope.parseFrom(envelope.toByteArray());
assertThat(parsed.hasRawFrameFact()).isTrue();
assertThat(parsed.getRawFrameFact().getFrameId()).isEqualTo("frame-1");
assertThat(parsed.getRawFrameFact().getParseStatus()).isEqualTo(ParseStatusProto.PARSE_STATUS_SUCCEEDED);
}
@Test
void decodedFactPayloadRoundTripsThroughEnvelope() throws Exception {
VehicleEnvelope envelope = VehicleEnvelope.newBuilder()
.setSchemaVersion("2.0")
.setEventId("fact-1")
.setVin("VIN001")
.setSource("JT808")
.setEventTimeMs(1782662400000L)
.setIngestTimeMs(1782662400001L)
.setDecodedFact(DecodedFactPayload.newBuilder()
.setFactId("fact-1")
.setFrameId("frame-1")
.setFactType("location")
.setVehicleKey("jt808:013912345678")
.setRawUri("archive://raw/2026/06/29/JT808/jt808_013912345678/frame-1.bin")
.putFields("longitude", "120.1")
.putFields("latitude", "30.1")
.build())
.build();
VehicleEnvelope parsed = VehicleEnvelope.parseFrom(envelope.toByteArray());
assertThat(parsed.hasDecodedFact()).isTrue();
assertThat(parsed.getDecodedFact().getFieldsMap()).containsEntry("longitude", "120.1");
}
}

View File

@@ -0,0 +1,60 @@
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0">
<modelVersion>4.0.0</modelVersion>
<parent>
<groupId>com.lingniu.ingest</groupId>
<artifactId>lingniu-vehicle-ingest</artifactId>
<version>0.1.0-SNAPSHOT</version>
<relativePath>../../../pom.xml</relativePath>
</parent>
<artifactId>tdengine-history-store</artifactId>
<name>tdengine-history-store</name>
<description>TDengine-backed hot history store schema and Kafka envelope mapping.</description>
<dependencies>
<dependency>
<groupId>com.lingniu.ingest</groupId>
<artifactId>sink-mq</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-autoconfigure</artifactId>
</dependency>
<dependency>
<groupId>com.taosdata.jdbc</groupId>
<artifactId>taos-jdbcdriver</artifactId>
</dependency>
<dependency>
<groupId>com.zaxxer</groupId>
<artifactId>HikariCP</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-configuration-processor</artifactId>
<optional>true</optional>
</dependency>
<dependency>
<groupId>com.fasterxml.jackson.core</groupId>
<artifactId>jackson-databind</artifactId>
</dependency>
<dependency>
<groupId>org.junit.jupiter</groupId>
<artifactId>junit-jupiter</artifactId>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.assertj</groupId>
<artifactId>assertj-core</artifactId>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-test</artifactId>
<scope>test</scope>
</dependency>
</dependencies>
</project>

View File

@@ -0,0 +1,15 @@
package com.lingniu.ingest.tdenginehistory;
import java.util.List;
import java.util.ArrayList;
import java.util.Collections;
public record TdengineBatchStatement(
String createChildTableSql,
String insertSql,
List<Object> values
) {
public TdengineBatchStatement {
values = values == null ? List.of() : Collections.unmodifiableList(new ArrayList<>(values));
}
}

View File

@@ -0,0 +1,233 @@
package com.lingniu.ingest.tdenginehistory;
import com.fasterxml.jackson.core.JsonProcessingException;
import com.fasterxml.jackson.databind.ObjectMapper;
import com.lingniu.ingest.sink.mq.proto.LocationPayload;
import com.lingniu.ingest.sink.mq.proto.RawFrameFactPayload;
import com.lingniu.ingest.sink.mq.proto.TelemetryField;
import com.lingniu.ingest.sink.mq.proto.VehicleEnvelope;
import java.time.Instant;
import java.util.ArrayList;
import java.util.LinkedHashMap;
import java.util.List;
import java.util.Map;
import java.util.Optional;
public final class TdengineEnvelopeRows {
private static final ObjectMapper OBJECT_MAPPER = new ObjectMapper();
private TdengineEnvelopeRows() {
}
public static Optional<TdengineRawFrameRow> rawFrame(VehicleEnvelope envelope) {
if (envelope == null || !envelope.hasRawFrameFact()) {
return Optional.empty();
}
RawFrameFactPayload raw = envelope.getRawFrameFact();
Map<String, String> metadata = new LinkedHashMap<>();
metadata.putAll(envelope.getMetadataMap());
metadata.putAll(raw.getMetadataMap());
return Optional.of(new TdengineRawFrameRow(
instant(envelope.getEventTimeMs()),
raw.getFrameId(),
instant(envelope.getIngestTimeMs()),
raw.getMessageId(),
raw.getSubType(),
instant(envelope.getEventTimeMs()),
raw.getRawUri(),
raw.getChecksum(),
raw.getRawSizeBytes(),
raw.getParseStatus().name().replace("PARSE_STATUS_", ""),
raw.getParseError(),
raw.getPeer(),
json(metadata),
protocol(envelope),
vehicleKey(envelope, raw.getVehicleKey(), raw.getPhone()),
firstNonBlank(raw.getVin(), envelope.getVin()),
raw.getPhone()
));
}
public static Optional<TdengineLocationRow> location(VehicleEnvelope envelope) {
if (envelope == null || !envelope.hasLocation()) {
return Optional.empty();
}
LocationPayload location = envelope.getLocation();
String rawUri = firstNonBlank(
envelope.hasRawArchive() ? envelope.getRawArchive().getUri() : "",
envelope.getMetadataOrDefault("rawArchiveUri", ""));
String frameId = envelope.getMetadataOrDefault("frame_id", envelope.getEventId());
return Optional.of(new TdengineLocationRow(
instant(envelope.getEventTimeMs()),
envelope.getEventId(),
frameId,
instant(envelope.getIngestTimeMs()),
location.getLongitude(),
location.getLatitude(),
location.getAltitudeM(),
location.getSpeedKmh(),
location.getDirectionDeg(),
location.getAlarmFlag(),
location.getStatusFlag(),
totalMileage(envelope),
rawUri,
json(envelope.getMetadataMap()),
protocol(envelope),
vehicleKey(envelope, envelope.getMetadataOrDefault("vehicle_key", ""),
envelope.getMetadataOrDefault("phone", "")),
envelope.getVin(),
envelope.getMetadataOrDefault("phone", "")
));
}
public static List<TdengineTelemetryFieldRow> telemetryFields(VehicleEnvelope envelope) {
if (envelope == null || !envelope.hasTelemetrySnapshot()) {
return List.of();
}
var snapshot = envelope.getTelemetrySnapshot();
if (snapshot.getFieldsCount() == 0) {
return List.of();
}
List<TdengineTelemetryFieldRow> rows = new ArrayList<>(snapshot.getFieldsCount());
String protocol = protocol(envelope);
String vin = envelope.getVin();
String phone = firstNonBlank(
envelope.getMetadataOrDefault("phone", ""),
envelope.hasRawFrameFact() ? envelope.getRawFrameFact().getPhone() : "");
String vehicleKey = vehicleKey(envelope,
firstNonBlank(
envelope.getMetadataOrDefault("vehicle_key", ""),
envelope.hasRawFrameFact() ? envelope.getRawFrameFact().getVehicleKey() : ""),
phone);
String frameId = firstNonBlank(
envelope.getMetadataOrDefault("frame_id", ""),
envelope.hasRawFrameFact() ? envelope.getRawFrameFact().getFrameId() : "",
envelope.getEventId());
String rawUri = firstNonBlank(
snapshot.getRawArchiveUri(),
envelope.hasRawArchive() ? envelope.getRawArchive().getUri() : "");
String metadataJson = json(envelope.getMetadataMap());
for (int i = 0; i < snapshot.getFieldsCount(); i++) {
TelemetryField field = snapshot.getFields(i);
if (field.getKey().isBlank()) {
continue;
}
rows.add(new TdengineTelemetryFieldRow(
instant(envelope.getEventTimeMs()),
envelope.getEventId() + "#" + i,
frameId,
instant(envelope.getIngestTimeMs()),
field.getKey(),
field.getValueType(),
field.getValue(),
valueDouble(field),
valueLong(field),
field.getUnit(),
field.getQuality(),
field.getSourcePath(),
rawUri,
metadataJson,
protocol,
vehicleKey,
vin,
phone
));
}
return List.copyOf(rows);
}
private static Instant instant(long epochMillis) {
return Instant.ofEpochMilli(epochMillis);
}
private static String protocol(VehicleEnvelope envelope) {
return firstNonBlank(envelope.getSource(), "UNKNOWN");
}
private static String vehicleKey(VehicleEnvelope envelope, String explicitVehicleKey, String phone) {
String key = firstKnownNonBlank(explicitVehicleKey, envelope.getMetadataOrDefault("vehicle_key", ""),
envelope.getVin());
if (!key.isBlank()) {
return key;
}
if ("JT808".equalsIgnoreCase(protocol(envelope))) {
String phoneValue = firstKnownNonBlank(phone, envelope.getMetadataOrDefault("phone", ""));
if (!phoneValue.isBlank()) {
return phoneValue.startsWith("jt808:") ? phoneValue : "jt808:" + phoneValue;
}
}
return firstNonBlank(explicitVehicleKey, envelope.getMetadataOrDefault("vehicle_key", ""),
envelope.getVin(), phone);
}
private static Double totalMileage(VehicleEnvelope envelope) {
String value = envelope.getMetadataOrDefault("total_mileage_km", "");
if (value.isBlank()) {
return null;
}
return Double.parseDouble(value);
}
private static Double valueDouble(TelemetryField field) {
String valueType = field.getValueType();
if (!"DOUBLE".equals(valueType) && !"FLOAT".equals(valueType)) {
return null;
}
return parseDouble(field.getValue());
}
private static Long valueLong(TelemetryField field) {
String valueType = field.getValueType();
if (!"LONG".equals(valueType) && !"INT".equals(valueType) && !"INTEGER".equals(valueType)) {
return null;
}
String value = field.getValue();
if (value == null || value.isBlank()) {
return null;
}
try {
return Long.parseLong(value);
} catch (NumberFormatException ignored) {
return null;
}
}
private static Double parseDouble(String value) {
if (value == null || value.isBlank()) {
return null;
}
try {
return Double.parseDouble(value);
} catch (NumberFormatException ignored) {
return null;
}
}
private static String json(Map<String, String> metadata) {
try {
return OBJECT_MAPPER.writeValueAsString(metadata == null ? Map.of() : metadata);
} catch (JsonProcessingException e) {
throw new IllegalArgumentException("metadata cannot be serialized as json", e);
}
}
private static String firstNonBlank(String... values) {
for (String value : values) {
if (value != null && !value.isBlank()) {
return value;
}
}
return "";
}
private static String firstKnownNonBlank(String... values) {
for (String value : values) {
if (value != null && !value.isBlank() && !"unknown".equalsIgnoreCase(value.trim())) {
return value;
}
}
return "";
}
}

View File

@@ -0,0 +1,74 @@
package com.lingniu.ingest.tdenginehistory;
import java.util.ArrayList;
import java.util.List;
public final class TdengineHistoryQueries {
private static final String RAW_COLUMNS = "ts, frame_id, received_at, message_id, sub_type, event_time, "
+ "raw_uri, checksum, raw_size_bytes, parse_status, parse_error, peer, metadata_json, "
+ "protocol, vehicle_key, vin, phone";
private static final String LOCATION_COLUMNS = "ts, fact_id, frame_id, received_at, longitude, latitude, "
+ "altitude_m, speed_kmh, direction_deg, alarm_flag, status_flag, total_mileage_km, raw_uri, "
+ "metadata_json, protocol, vehicle_key, vin, phone";
private static final String TELEMETRY_FIELD_COLUMNS = "ts, fact_id, frame_id, received_at, field_key, "
+ "value_type, value_text, value_double, value_long, unit, quality, source_path, raw_uri, "
+ "metadata_json, protocol, vehicle_key, vin, phone";
private final TdengineHistorySchema schema;
public TdengineHistoryQueries(TdengineHistorySchema schema) {
if (schema == null) {
throw new IllegalArgumentException("schema must not be null");
}
this.schema = schema;
}
public TdengineQueryStatement rawFrames(TdengineRawFrameQuery query) {
String table = schema.rawFrameTable(query.protocol(), query.vehicleKey());
return query(table, RAW_COLUMNS, "frame_id", query.from(), query.to(),
query.order(), query.limit(), query.cursor());
}
public TdengineQueryStatement locations(TdengineLocationQuery query) {
String table = schema.locationTable(query.protocol(), query.vehicleKey());
return query(table, LOCATION_COLUMNS, "fact_id", query.from(), query.to(),
query.order(), query.limit(), query.cursor());
}
public TdengineQueryStatement telemetryFields(TdengineTelemetryFieldQuery query) {
String table = schema.telemetryFieldTable(query.protocol(), query.vehicleKey(), query.fieldKey());
return query(table, TELEMETRY_FIELD_COLUMNS, "fact_id", query.from(), query.to(),
query.order(), query.limit(), query.cursor());
}
private static TdengineQueryStatement query(String table,
String columns,
String tieBreakerColumn,
Object from,
Object to,
TdengineQueryOrder order,
int limit,
TdenginePageCursor cursor) {
List<Object> values = new ArrayList<>();
StringBuilder sql = new StringBuilder(256)
.append("SELECT ").append(columns)
.append(" FROM ").append(table)
.append(" WHERE ts >= ? AND ts < ?");
values.add(from);
values.add(to);
if (cursor != null) {
String op = order == TdengineQueryOrder.ASC ? ">" : "<";
sql.append(" AND (ts ").append(op).append(" ? OR (ts = ? AND ")
.append(tieBreakerColumn).append(' ').append(op).append(" ?))");
values.add(cursor.ts());
values.add(cursor.ts());
values.add(cursor.tieBreaker());
}
sql.append(" ORDER BY ts ").append(order.name())
.append(", ").append(tieBreakerColumn).append(' ').append(order.name())
.append(" LIMIT ?");
values.add(limit);
return new TdengineQueryStatement(sql.toString(), values);
}
}

View File

@@ -0,0 +1,12 @@
package com.lingniu.ingest.tdenginehistory;
import java.io.IOException;
public interface TdengineHistoryReader {
TdenginePage<TdengineRawFrameRow> queryRawFrames(TdengineRawFrameQuery query) throws IOException;
TdenginePage<TdengineLocationRow> queryLocations(TdengineLocationQuery query) throws IOException;
TdenginePage<TdengineTelemetryFieldRow> queryTelemetryFields(TdengineTelemetryFieldQuery query) throws IOException;
}

View File

@@ -0,0 +1,113 @@
package com.lingniu.ingest.tdenginehistory;
import java.util.List;
/**
* SQL model for the TDengine hot history store.
*/
public final class TdengineHistorySchema {
private final String database;
public TdengineHistorySchema(String database) {
this.database = TdengineIdentifier.database(database);
}
public List<String> bootstrapSql() {
return List.of(
"CREATE DATABASE IF NOT EXISTS " + database + " PRECISION 'ms'",
"USE " + database,
rawFramesStableSql(),
vehicleLocationsStableSql(),
telemetryFieldsStableSql()
);
}
public String rawFrameTable(String protocol, String vehicleKey) {
return "raw_" + TdengineIdentifier.fragment(protocol) + "_" + TdengineIdentifier.hash16(vehicleKey);
}
public String locationTable(String protocol, String vehicleKey) {
return "loc_" + TdengineIdentifier.fragment(protocol) + "_" + TdengineIdentifier.hash16(vehicleKey);
}
public String telemetryFieldTable(String protocol, String vehicleKey, String fieldKey) {
return "tf_" + TdengineIdentifier.fragment(protocol)
+ "_" + TdengineIdentifier.hash16(vehicleKey)
+ "_" + TdengineIdentifier.hash16(fieldKey);
}
private static String rawFramesStableSql() {
return """
CREATE STABLE IF NOT EXISTS raw_frames (
ts TIMESTAMP,
frame_id NCHAR(64),
received_at TIMESTAMP,
message_id INT,
sub_type INT,
event_time TIMESTAMP,
raw_uri NCHAR(512),
checksum NCHAR(128),
raw_size_bytes BIGINT,
parse_status NCHAR(16),
parse_error NCHAR(512),
peer NCHAR(128),
metadata_json NCHAR(4096)
) TAGS (
protocol NCHAR(16),
vehicle_key NCHAR(128),
vin NCHAR(64),
phone NCHAR(32)
)""";
}
private static String vehicleLocationsStableSql() {
return """
CREATE STABLE IF NOT EXISTS vehicle_locations (
ts TIMESTAMP,
fact_id NCHAR(64),
frame_id NCHAR(64),
received_at TIMESTAMP,
longitude DOUBLE,
latitude DOUBLE,
altitude_m DOUBLE,
speed_kmh DOUBLE,
direction_deg DOUBLE,
alarm_flag BIGINT,
status_flag BIGINT,
total_mileage_km DOUBLE,
raw_uri NCHAR(512),
metadata_json NCHAR(4096)
) TAGS (
protocol NCHAR(16),
vehicle_key NCHAR(128),
vin NCHAR(64),
phone NCHAR(32)
)""";
}
private static String telemetryFieldsStableSql() {
return """
CREATE STABLE IF NOT EXISTS telemetry_fields (
ts TIMESTAMP,
fact_id NCHAR(96),
frame_id NCHAR(64),
received_at TIMESTAMP,
value_type NCHAR(16),
value_text NCHAR(1024),
value_double DOUBLE,
value_long BIGINT,
unit NCHAR(32),
quality NCHAR(16),
source_path NCHAR(256),
raw_uri NCHAR(512),
metadata_json NCHAR(4096)
) TAGS (
protocol NCHAR(16),
vehicle_key NCHAR(128),
vin NCHAR(64),
phone NCHAR(32),
field_key NCHAR(256)
)""";
}
}

View File

@@ -0,0 +1,87 @@
package com.lingniu.ingest.tdenginehistory;
import java.util.List;
import java.util.Arrays;
public final class TdengineHistoryStatements {
private static final String RAW_FRAME_COLUMNS = "ts, frame_id, received_at, message_id, sub_type, event_time, "
+ "raw_uri, checksum, raw_size_bytes, parse_status, parse_error, peer, metadata_json";
private static final String RAW_FRAME_PLACEHOLDERS = "?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?";
private static final String LOCATION_COLUMNS = "ts, fact_id, frame_id, received_at, longitude, latitude, "
+ "altitude_m, speed_kmh, direction_deg, alarm_flag, status_flag, total_mileage_km, raw_uri, metadata_json";
private static final String LOCATION_PLACEHOLDERS = "?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?";
private static final String TELEMETRY_FIELD_COLUMNS = "ts, fact_id, frame_id, received_at, value_type, "
+ "value_text, value_double, value_long, unit, quality, source_path, raw_uri, metadata_json";
private static final String TELEMETRY_FIELD_PLACEHOLDERS = "?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?";
private final TdengineHistorySchema schema;
public TdengineHistoryStatements(TdengineHistorySchema schema) {
if (schema == null) {
throw new IllegalArgumentException("schema must not be null");
}
this.schema = schema;
}
public TdengineBatchStatement rawFrame(TdengineRawFrameRow row) {
String table = schema.rawFrameTable(row.protocol(), row.vehicleKey());
return new TdengineBatchStatement(
"CREATE TABLE IF NOT EXISTS " + table + " USING raw_frames TAGS ("
+ tags(row.protocol(), row.vehicleKey(), row.vin(), row.phone()) + ")",
"INSERT INTO " + table + " (" + RAW_FRAME_COLUMNS + ") VALUES (" + RAW_FRAME_PLACEHOLDERS + ")",
values(
row.ts(), row.frameId(), row.receivedAt(), row.messageId(), row.subType(),
row.eventTime(), row.rawUri(), row.checksum(), row.rawSizeBytes(), row.parseStatus(),
row.parseError(), row.peer(), row.metadataJson()
)
);
}
public TdengineBatchStatement location(TdengineLocationRow row) {
String table = schema.locationTable(row.protocol(), row.vehicleKey());
return new TdengineBatchStatement(
"CREATE TABLE IF NOT EXISTS " + table + " USING vehicle_locations TAGS ("
+ tags(row.protocol(), row.vehicleKey(), row.vin(), row.phone()) + ")",
"INSERT INTO " + table + " (" + LOCATION_COLUMNS + ") VALUES (" + LOCATION_PLACEHOLDERS + ")",
values(
row.ts(), row.factId(), row.frameId(), row.receivedAt(), row.longitude(), row.latitude(),
row.altitudeM(), row.speedKmh(), row.directionDeg(), row.alarmFlag(), row.statusFlag(),
row.totalMileageKm(), row.rawUri(), row.metadataJson()
)
);
}
public TdengineBatchStatement telemetryField(TdengineTelemetryFieldRow row) {
String table = schema.telemetryFieldTable(row.protocol(), row.vehicleKey(), row.fieldKey());
return new TdengineBatchStatement(
"CREATE TABLE IF NOT EXISTS " + table + " USING telemetry_fields TAGS ("
+ tags(row.protocol(), row.vehicleKey(), row.vin(), row.phone(), row.fieldKey()) + ")",
"INSERT INTO " + table + " (" + TELEMETRY_FIELD_COLUMNS + ") VALUES ("
+ TELEMETRY_FIELD_PLACEHOLDERS + ")",
values(
row.ts(), row.factId(), row.frameId(), row.receivedAt(), row.valueType(), row.valueText(),
row.valueDouble(), row.valueLong(), row.unit(), row.quality(), row.sourcePath(),
row.rawUri(), row.metadataJson()
)
);
}
private static String tags(String protocol, String vehicleKey, String vin, String phone) {
return quote(protocol) + ", " + quote(vehicleKey) + ", " + quote(vin) + ", " + quote(phone);
}
private static String tags(String protocol, String vehicleKey, String vin, String phone, String fieldKey) {
return tags(protocol, vehicleKey, vin, phone) + ", " + quote(fieldKey);
}
private static String quote(String value) {
return "'" + (value == null ? "" : value.replace("'", "''")) + "'";
}
private static List<Object> values(Object... values) {
return Arrays.asList(values);
}
}

View File

@@ -0,0 +1,25 @@
package com.lingniu.ingest.tdenginehistory;
import java.io.IOException;
import java.util.List;
public interface TdengineHistoryWriter {
default void appendRawFrame(TdengineRawFrameRow row) throws IOException {
appendRawFrames(List.of(row));
}
void appendRawFrames(List<TdengineRawFrameRow> rows) throws IOException;
default void appendLocation(TdengineLocationRow row) throws IOException {
appendLocations(List.of(row));
}
void appendLocations(List<TdengineLocationRow> rows) throws IOException;
default void appendTelemetryField(TdengineTelemetryFieldRow row) throws IOException {
appendTelemetryFields(List.of(row));
}
void appendTelemetryFields(List<TdengineTelemetryFieldRow> rows) throws IOException;
}

View File

@@ -0,0 +1,42 @@
package com.lingniu.ingest.tdenginehistory;
import java.nio.charset.StandardCharsets;
import java.security.MessageDigest;
import java.security.NoSuchAlgorithmException;
import java.util.HexFormat;
import java.util.Locale;
final class TdengineIdentifier {
private TdengineIdentifier() {
}
static String database(String value) {
String fragment = fragment(value);
if (fragment.isBlank()) {
throw new IllegalArgumentException("database must contain identifier characters");
}
return fragment;
}
static String fragment(String value) {
if (value == null) {
return "unknown";
}
String normalized = value.trim().toLowerCase(Locale.ROOT)
.replaceAll("[^a-z0-9_]+", "_")
.replaceAll("_+", "_")
.replaceAll("^_|_$", "");
return normalized.isBlank() ? "unknown" : normalized;
}
static String hash16(String value) {
try {
byte[] digest = MessageDigest.getInstance("SHA-256")
.digest(String.valueOf(value).getBytes(StandardCharsets.UTF_8));
return HexFormat.of().formatHex(digest, 0, 8);
} catch (NoSuchAlgorithmException e) {
throw new IllegalStateException("SHA-256 is not available", e);
}
}
}

View File

@@ -0,0 +1,193 @@
package com.lingniu.ingest.tdenginehistory;
import javax.sql.DataSource;
import java.io.IOException;
import java.sql.Connection;
import java.sql.PreparedStatement;
import java.sql.ResultSet;
import java.sql.SQLException;
import java.sql.Timestamp;
import java.time.Instant;
import java.util.ArrayList;
import java.util.List;
import java.util.Optional;
import java.util.function.Function;
public final class TdengineJdbcHistoryReader implements TdengineHistoryReader {
private final DataSource dataSource;
private final TdengineHistoryQueries queries;
public TdengineJdbcHistoryReader(DataSource dataSource, TdengineHistorySchema schema) {
if (dataSource == null) {
throw new IllegalArgumentException("dataSource must not be null");
}
this.dataSource = dataSource;
this.queries = new TdengineHistoryQueries(schema);
}
@Override
public TdenginePage<TdengineRawFrameRow> queryRawFrames(TdengineRawFrameQuery query) throws IOException {
int pageSize = Math.min(query.limit(), 1000);
TdengineQueryStatement statement = queries.rawFrames(query.withLimit(pageSize + 1));
return readPage(statement, pageSize, this::rawFrame, row -> new TdenginePageCursor(row.ts(), row.frameId()));
}
@Override
public TdenginePage<TdengineLocationRow> queryLocations(TdengineLocationQuery query) throws IOException {
int pageSize = Math.min(query.limit(), 1000);
TdengineQueryStatement statement = queries.locations(query.withLimit(pageSize + 1));
return readPage(statement, pageSize, this::location, row -> new TdenginePageCursor(row.ts(), row.factId()));
}
@Override
public TdenginePage<TdengineTelemetryFieldRow> queryTelemetryFields(TdengineTelemetryFieldQuery query)
throws IOException {
int pageSize = Math.min(query.limit(), 1000);
TdengineQueryStatement statement = queries.telemetryFields(query.withLimit(pageSize + 1));
return readPage(statement, pageSize, this::telemetryField,
row -> new TdenginePageCursor(row.ts(), row.factId()));
}
private <T> TdenginePage<T> readPage(TdengineQueryStatement statement,
int pageSize,
SqlRowMapper<T> mapper,
Function<T, TdenginePageCursor> cursor) throws IOException {
List<T> rows = new ArrayList<>();
try (Connection connection = dataSource.getConnection();
PreparedStatement prepared = connection.prepareStatement(statement.sql())) {
bind(prepared, statement.values());
try (ResultSet resultSet = prepared.executeQuery()) {
while (resultSet.next()) {
rows.add(mapper.map(resultSet));
}
}
} catch (SQLException e) {
if (isTableMissing(e)) {
return new TdenginePage<>(List.of(), Optional.empty());
}
throw new IOException("tdengine history query failed", e);
}
boolean hasMore = rows.size() > pageSize;
List<T> items = hasMore ? List.copyOf(rows.subList(0, pageSize)) : List.copyOf(rows);
Optional<TdenginePageCursor> next = hasMore && !items.isEmpty()
? Optional.of(cursor.apply(items.getLast()))
: Optional.empty();
return new TdenginePage<>(items, next);
}
private static boolean isTableMissing(SQLException e) {
for (Throwable current = e; current != null; current = current.getCause()) {
String message = current.getMessage();
if (message != null) {
String normalized = message.toLowerCase();
if (normalized.contains("table does not exist")
|| normalized.contains("0x2603")
|| normalized.contains("table not exist")) {
return true;
}
}
}
return false;
}
private static void bind(PreparedStatement prepared, List<Object> values) throws SQLException {
for (int i = 0; i < values.size(); i++) {
Object value = values.get(i);
int parameterIndex = i + 1;
if (value instanceof Instant instant) {
prepared.setTimestamp(parameterIndex, Timestamp.from(instant));
} else {
prepared.setObject(parameterIndex, value);
}
}
}
private TdengineRawFrameRow rawFrame(ResultSet rs) throws SQLException {
return new TdengineRawFrameRow(
instant(rs, "ts"),
rs.getString("frame_id"),
instant(rs, "received_at"),
rs.getInt("message_id"),
rs.getInt("sub_type"),
instant(rs, "event_time"),
rs.getString("raw_uri"),
rs.getString("checksum"),
rs.getLong("raw_size_bytes"),
rs.getString("parse_status"),
rs.getString("parse_error"),
rs.getString("peer"),
rs.getString("metadata_json"),
rs.getString("protocol"),
rs.getString("vehicle_key"),
rs.getString("vin"),
rs.getString("phone")
);
}
private TdengineLocationRow location(ResultSet rs) throws SQLException {
return new TdengineLocationRow(
instant(rs, "ts"),
rs.getString("fact_id"),
rs.getString("frame_id"),
instant(rs, "received_at"),
rs.getDouble("longitude"),
rs.getDouble("latitude"),
rs.getDouble("altitude_m"),
rs.getDouble("speed_kmh"),
rs.getDouble("direction_deg"),
rs.getLong("alarm_flag"),
rs.getLong("status_flag"),
nullableDouble(rs, "total_mileage_km"),
rs.getString("raw_uri"),
rs.getString("metadata_json"),
rs.getString("protocol"),
rs.getString("vehicle_key"),
rs.getString("vin"),
rs.getString("phone")
);
}
private TdengineTelemetryFieldRow telemetryField(ResultSet rs) throws SQLException {
return new TdengineTelemetryFieldRow(
instant(rs, "ts"),
rs.getString("fact_id"),
rs.getString("frame_id"),
instant(rs, "received_at"),
rs.getString("field_key"),
rs.getString("value_type"),
rs.getString("value_text"),
nullableDouble(rs, "value_double"),
nullableLong(rs, "value_long"),
rs.getString("unit"),
rs.getString("quality"),
rs.getString("source_path"),
rs.getString("raw_uri"),
rs.getString("metadata_json"),
rs.getString("protocol"),
rs.getString("vehicle_key"),
rs.getString("vin"),
rs.getString("phone")
);
}
private static Instant instant(ResultSet rs, String column) throws SQLException {
Timestamp timestamp = rs.getTimestamp(column);
return timestamp == null ? null : timestamp.toInstant();
}
private static Double nullableDouble(ResultSet rs, String column) throws SQLException {
Object value = rs.getObject(column);
return value instanceof Number number ? number.doubleValue() : null;
}
private static Long nullableLong(ResultSet rs, String column) throws SQLException {
Object value = rs.getObject(column);
return value instanceof Number number ? number.longValue() : null;
}
@FunctionalInterface
private interface SqlRowMapper<T> {
T map(ResultSet rs) throws SQLException;
}
}

View File

@@ -0,0 +1,301 @@
package com.lingniu.ingest.tdenginehistory;
import javax.sql.DataSource;
import java.io.IOException;
import java.sql.Connection;
import java.sql.PreparedStatement;
import java.sql.SQLException;
import java.sql.Statement;
import java.sql.Timestamp;
import java.time.Instant;
import java.util.LinkedHashMap;
import java.util.LinkedHashSet;
import java.util.List;
import java.util.Map;
import java.util.Set;
import java.util.function.Function;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
public final class TdengineJdbcHistoryWriter implements TdengineHistoryWriter {
private static final Logger log = LoggerFactory.getLogger(TdengineJdbcHistoryWriter.class);
private static final int MAX_LITERAL_ROWS_PER_STATEMENT = 200;
private final DataSource dataSource;
private final TdengineHistorySchema schema;
private final TdengineHistoryStatements statements;
private final Object schemaInitializationMonitor = new Object();
private volatile boolean schemaInitialized;
private volatile boolean literalFallbackLogged;
public TdengineJdbcHistoryWriter(DataSource dataSource, TdengineHistorySchema schema) {
if (dataSource == null) {
throw new IllegalArgumentException("dataSource must not be null");
}
if (schema == null) {
throw new IllegalArgumentException("schema must not be null");
}
this.dataSource = dataSource;
this.schema = schema;
this.statements = new TdengineHistoryStatements(schema);
}
public void initializeSchema() throws IOException {
inTransaction(connection -> {
try (Statement statement = connection.createStatement()) {
for (String sql : schema.bootstrapSql()) {
statement.execute(sql);
}
}
});
schemaInitialized = true;
}
@Override
public void appendRawFrames(List<TdengineRawFrameRow> rows) throws IOException {
append(rows, statements::rawFrame);
}
@Override
public void appendLocations(List<TdengineLocationRow> rows) throws IOException {
append(rows, statements::location);
}
@Override
public void appendTelemetryFields(List<TdengineTelemetryFieldRow> rows) throws IOException {
append(rows, statements::telemetryField);
}
private <T> void append(List<T> rows, Function<T, TdengineBatchStatement> mapper) throws IOException {
if (rows == null || rows.isEmpty()) {
return;
}
if (!schemaInitialized) {
synchronized (schemaInitializationMonitor) {
if (!schemaInitialized) {
append(rows, mapper, true);
schemaInitialized = true;
return;
}
}
}
append(rows, mapper, false);
}
private <T> void append(List<T> rows,
Function<T, TdengineBatchStatement> mapper,
boolean initializeSchema) throws IOException {
inTransaction(connection -> {
if (initializeSchema) {
try (Statement statement = connection.createStatement()) {
for (String sql : schema.bootstrapSql()) {
statement.execute(sql);
}
}
}
Set<String> createdChildTables = new LinkedHashSet<>();
Map<String, PreparedStatement> preparedStatements = new LinkedHashMap<>();
Map<String, LiteralBatch> literalBatches = new LinkedHashMap<>();
Map<String, LiteralBatch> literalOnlyBatches = new LinkedHashMap<>();
try {
for (T row : rows) {
if (row == null) {
continue;
}
TdengineBatchStatement batch = mapper.apply(row);
if (createdChildTables.add(batch.createChildTableSql())) {
try (Statement statement = connection.createStatement()) {
statement.execute(batch.createChildTableSql());
}
}
if (hasEmptyString(batch.values())) {
literalOnlyBatches.computeIfAbsent(batch.insertSql(), LiteralBatch::new).add(batch.values());
continue;
}
PreparedStatement prepared = preparedStatements.computeIfAbsent(batch.insertSql(), sql -> {
try {
return connection.prepareStatement(sql);
} catch (SQLException e) {
throw new JdbcRuntimeException(e);
}
});
bind(prepared, batch.values());
prepared.addBatch();
literalBatches.computeIfAbsent(batch.insertSql(), LiteralBatch::new).add(batch.values());
}
for (Map.Entry<String, PreparedStatement> entry : preparedStatements.entrySet()) {
try {
entry.getValue().executeBatch();
} catch (SQLException e) {
executeLiteralFallback(connection, literalBatches.get(entry.getKey()), e);
}
}
for (LiteralBatch batch : literalOnlyBatches.values()) {
executeLiteralBatch(connection, batch);
}
} finally {
for (PreparedStatement prepared : preparedStatements.values()) {
prepared.close();
}
}
});
}
private static void bind(PreparedStatement prepared, List<Object> values) throws SQLException {
for (int i = 0; i < values.size(); i++) {
Object value = values.get(i);
int parameterIndex = i + 1;
if (value instanceof Instant instant) {
prepared.setTimestamp(parameterIndex, Timestamp.from(instant));
} else {
prepared.setObject(parameterIndex, value);
}
}
}
private static boolean hasEmptyString(List<Object> values) {
for (Object value : values) {
if (value instanceof String stringValue && stringValue.isEmpty()) {
return true;
}
}
return false;
}
private void executeLiteralFallback(Connection connection,
LiteralBatch batch,
SQLException preparedFailure) throws SQLException {
if (batch == null || batch.rows().isEmpty()) {
throw preparedFailure;
}
if (!literalFallbackLogged) {
literalFallbackLogged = true;
log.warn("TDengine prepared batch failed; falling back to literal inserts: {}",
preparedFailure.getMessage());
log.debug("TDengine prepared batch failure stacktrace", preparedFailure);
}
executeLiteralBatch(connection, batch);
}
private static void executeLiteralBatch(Connection connection, LiteralBatch batch) throws SQLException {
try (Statement statement = connection.createStatement()) {
List<List<Object>> rows = batch.rows();
for (int start = 0; start < rows.size(); start += MAX_LITERAL_ROWS_PER_STATEMENT) {
int end = Math.min(start + MAX_LITERAL_ROWS_PER_STATEMENT, rows.size());
statement.execute(toLiteralInsertSql(batch.insertSql(), rows.subList(start, end)));
}
}
}
private static String toLiteralInsertSql(String insertSql, List<List<Object>> rows) {
int valuesIndex = insertSql.lastIndexOf("VALUES");
if (valuesIndex < 0) {
throw new IllegalArgumentException("insert SQL must contain VALUES: " + insertSql);
}
if (rows == null || rows.isEmpty()) {
throw new IllegalArgumentException("literal rows must not be empty");
}
StringBuilder sql = new StringBuilder(insertSql.substring(0, valuesIndex))
.append("VALUES ");
for (int rowIndex = 0; rowIndex < rows.size(); rowIndex++) {
if (rowIndex > 0) {
sql.append(' ');
}
List<Object> values = rows.get(rowIndex);
sql.append('(');
for (int i = 0; i < values.size(); i++) {
if (i > 0) {
sql.append(", ");
}
sql.append(literal(values.get(i)));
}
sql.append(')');
}
return sql.toString();
}
private static String literal(Object value) {
if (value == null) {
return "NULL";
}
if (value instanceof Instant instant) {
return Long.toString(instant.toEpochMilli());
}
if (value instanceof Timestamp timestamp) {
return Long.toString(timestamp.toInstant().toEpochMilli());
}
if (value instanceof Double doubleValue) {
return Double.isFinite(doubleValue) ? doubleValue.toString() : "NULL";
}
if (value instanceof Float floatValue) {
return Float.isFinite(floatValue) ? floatValue.toString() : "NULL";
}
if (value instanceof Number number) {
return number.toString();
}
return "'" + value.toString().replace("'", "''") + "'";
}
private void inTransaction(SqlWork work) throws IOException {
try (Connection connection = dataSource.getConnection()) {
boolean autoCommit = connection.getAutoCommit();
connection.setAutoCommit(false);
try {
work.execute(connection);
connection.commit();
} catch (JdbcRuntimeException e) {
rollback(connection);
throw e.getCause();
} catch (SQLException e) {
rollback(connection);
throw e;
} finally {
connection.setAutoCommit(autoCommit);
}
} catch (SQLException e) {
throw new IOException("tdengine history write failed", e);
}
}
private static void rollback(Connection connection) throws SQLException {
connection.rollback();
}
@FunctionalInterface
private interface SqlWork {
void execute(Connection connection) throws SQLException;
}
private static final class JdbcRuntimeException extends RuntimeException {
private JdbcRuntimeException(SQLException cause) {
super(cause);
}
@Override
public synchronized SQLException getCause() {
return (SQLException) super.getCause();
}
}
private static final class LiteralBatch {
private final String insertSql;
private final List<List<Object>> rows = new java.util.ArrayList<>();
private LiteralBatch(String insertSql) {
this.insertSql = insertSql;
}
private void add(List<Object> row) {
rows.add(new java.util.ArrayList<>(row));
}
private String insertSql() {
return insertSql;
}
private List<List<Object>> rows() {
return rows;
}
}
}

View File

@@ -0,0 +1,31 @@
package com.lingniu.ingest.tdenginehistory;
import java.time.Instant;
public record TdengineLocationQuery(
String protocol,
String vehicleKey,
Instant from,
Instant to,
TdengineQueryOrder order,
int limit,
TdenginePageCursor cursor
) {
public TdengineLocationQuery {
if (protocol == null || protocol.isBlank()) {
throw new IllegalArgumentException("protocol must not be blank");
}
if (vehicleKey == null || vehicleKey.isBlank()) {
throw new IllegalArgumentException("vehicleKey must not be blank");
}
if (from == null || to == null || !from.isBefore(to)) {
throw new IllegalArgumentException("query time range must be valid");
}
order = order == null ? TdengineQueryOrder.DESC : order;
limit = Math.max(1, Math.min(limit, 1001));
}
TdengineLocationQuery withLimit(int newLimit) {
return new TdengineLocationQuery(protocol, vehicleKey, from, to, order, newLimit, cursor);
}
}

View File

@@ -0,0 +1,25 @@
package com.lingniu.ingest.tdenginehistory;
import java.time.Instant;
public record TdengineLocationRow(
Instant ts,
String factId,
String frameId,
Instant receivedAt,
double longitude,
double latitude,
double altitudeM,
double speedKmh,
double directionDeg,
long alarmFlag,
long statusFlag,
Double totalMileageKm,
String rawUri,
String metadataJson,
String protocol,
String vehicleKey,
String vin,
String phone
) {
}

View File

@@ -0,0 +1,12 @@
package com.lingniu.ingest.tdenginehistory;
import java.util.List;
import java.util.Optional;
public record TdenginePage<T>(List<T> items, Optional<TdenginePageCursor> nextCursor) {
public TdenginePage {
items = items == null ? List.of() : List.copyOf(items);
nextCursor = nextCursor == null ? Optional.empty() : nextCursor;
}
}

View File

@@ -0,0 +1,13 @@
package com.lingniu.ingest.tdenginehistory;
import java.time.Instant;
public record TdenginePageCursor(Instant ts, String tieBreaker) {
public TdenginePageCursor {
if (ts == null) {
throw new IllegalArgumentException("cursor ts must not be null");
}
tieBreaker = tieBreaker == null ? "" : tieBreaker;
}
}

View File

@@ -0,0 +1,6 @@
package com.lingniu.ingest.tdenginehistory;
public enum TdengineQueryOrder {
ASC,
DESC
}

View File

@@ -0,0 +1,15 @@
package com.lingniu.ingest.tdenginehistory;
import java.util.ArrayList;
import java.util.Collections;
import java.util.List;
public record TdengineQueryStatement(String sql, List<Object> values) {
public TdengineQueryStatement {
if (sql == null || sql.isBlank()) {
throw new IllegalArgumentException("sql must not be blank");
}
values = values == null ? List.of() : Collections.unmodifiableList(new ArrayList<>(values));
}
}

View File

@@ -0,0 +1,31 @@
package com.lingniu.ingest.tdenginehistory;
import java.time.Instant;
public record TdengineRawFrameQuery(
String protocol,
String vehicleKey,
Instant from,
Instant to,
TdengineQueryOrder order,
int limit,
TdenginePageCursor cursor
) {
public TdengineRawFrameQuery {
if (protocol == null || protocol.isBlank()) {
throw new IllegalArgumentException("protocol must not be blank");
}
if (vehicleKey == null || vehicleKey.isBlank()) {
throw new IllegalArgumentException("vehicleKey must not be blank");
}
if (from == null || to == null || !from.isBefore(to)) {
throw new IllegalArgumentException("query time range must be valid");
}
order = order == null ? TdengineQueryOrder.DESC : order;
limit = Math.max(1, Math.min(limit, 1001));
}
TdengineRawFrameQuery withLimit(int newLimit) {
return new TdengineRawFrameQuery(protocol, vehicleKey, from, to, order, newLimit, cursor);
}
}

View File

@@ -0,0 +1,24 @@
package com.lingniu.ingest.tdenginehistory;
import java.time.Instant;
public record TdengineRawFrameRow(
Instant ts,
String frameId,
Instant receivedAt,
int messageId,
int subType,
Instant eventTime,
String rawUri,
String checksum,
long rawSizeBytes,
String parseStatus,
String parseError,
String peer,
String metadataJson,
String protocol,
String vehicleKey,
String vin,
String phone
) {
}

View File

@@ -0,0 +1,35 @@
package com.lingniu.ingest.tdenginehistory;
import java.time.Instant;
public record TdengineTelemetryFieldQuery(
String protocol,
String vehicleKey,
String fieldKey,
Instant from,
Instant to,
TdengineQueryOrder order,
int limit,
TdenginePageCursor cursor
) {
public TdengineTelemetryFieldQuery {
if (protocol == null || protocol.isBlank()) {
throw new IllegalArgumentException("protocol must not be blank");
}
if (vehicleKey == null || vehicleKey.isBlank()) {
throw new IllegalArgumentException("vehicleKey must not be blank");
}
if (fieldKey == null || fieldKey.isBlank()) {
throw new IllegalArgumentException("fieldKey must not be blank");
}
if (from == null || to == null || !from.isBefore(to)) {
throw new IllegalArgumentException("query time range must be valid");
}
order = order == null ? TdengineQueryOrder.DESC : order;
limit = Math.max(1, Math.min(limit, 1001));
}
TdengineTelemetryFieldQuery withLimit(int newLimit) {
return new TdengineTelemetryFieldQuery(protocol, vehicleKey, fieldKey, from, to, order, newLimit, cursor);
}
}

View File

@@ -0,0 +1,25 @@
package com.lingniu.ingest.tdenginehistory;
import java.time.Instant;
public record TdengineTelemetryFieldRow(
Instant ts,
String factId,
String frameId,
Instant receivedAt,
String fieldKey,
String valueType,
String valueText,
Double valueDouble,
Long valueLong,
String unit,
String quality,
String sourcePath,
String rawUri,
String metadataJson,
String protocol,
String vehicleKey,
String vin,
String phone
) {
}

View File

@@ -0,0 +1,85 @@
package com.lingniu.ingest.tdenginehistory.config;
import com.lingniu.ingest.tdenginehistory.TdengineHistorySchema;
import com.lingniu.ingest.tdenginehistory.TdengineHistoryStatements;
import com.lingniu.ingest.tdenginehistory.TdengineHistoryReader;
import com.lingniu.ingest.tdenginehistory.TdengineHistoryWriter;
import com.lingniu.ingest.tdenginehistory.TdengineJdbcHistoryReader;
import com.lingniu.ingest.tdenginehistory.TdengineJdbcHistoryWriter;
import com.zaxxer.hikari.HikariConfig;
import com.zaxxer.hikari.HikariDataSource;
import org.springframework.boot.autoconfigure.AutoConfiguration;
import org.springframework.boot.autoconfigure.condition.ConditionalOnBean;
import org.springframework.boot.autoconfigure.condition.ConditionalOnMissingBean;
import org.springframework.boot.autoconfigure.condition.ConditionalOnProperty;
import org.springframework.boot.context.properties.EnableConfigurationProperties;
import org.springframework.context.annotation.Bean;
import javax.sql.DataSource;
@AutoConfiguration
@EnableConfigurationProperties(TdengineHistoryProperties.class)
public class TdengineHistoryAutoConfiguration {
@Bean
@ConditionalOnMissingBean
@ConditionalOnProperty(
prefix = "lingniu.ingest.tdengine-history",
name = "enabled",
havingValue = "true")
public TdengineHistorySchema tdengineHistorySchema(TdengineHistoryProperties properties) {
return new TdengineHistorySchema(properties.getDatabase());
}
@Bean
@ConditionalOnMissingBean
@ConditionalOnProperty(
prefix = "lingniu.ingest.tdengine-history",
name = "enabled",
havingValue = "true")
public TdengineHistoryStatements tdengineHistoryStatements(TdengineHistorySchema schema) {
return new TdengineHistoryStatements(schema);
}
@Bean
@ConditionalOnMissingBean(DataSource.class)
@ConditionalOnProperty(
prefix = "lingniu.ingest.tdengine-history",
name = "enabled",
havingValue = "true")
public DataSource tdengineDataSource(TdengineHistoryProperties properties) {
HikariConfig config = new HikariConfig();
config.setPoolName("tdengine-history");
config.setDriverClassName(properties.getDriverClassName());
config.setJdbcUrl(properties.getJdbcUrl());
config.setUsername(properties.getUsername());
config.setPassword(properties.getPassword());
config.setMaximumPoolSize(Math.max(1, properties.getMaximumPoolSize()));
config.setMinimumIdle(Math.max(0, Math.min(properties.getMinimumIdle(), properties.getMaximumPoolSize())));
config.setConnectionTimeout(Math.max(250, properties.getConnectionTimeoutMillis()));
config.setInitializationFailTimeout(properties.getInitializationFailTimeoutMillis());
return new HikariDataSource(config);
}
@Bean
@ConditionalOnMissingBean
@ConditionalOnBean(DataSource.class)
@ConditionalOnProperty(
prefix = "lingniu.ingest.tdengine-history",
name = "enabled",
havingValue = "true")
public TdengineHistoryWriter tdengineHistoryWriter(DataSource dataSource, TdengineHistorySchema schema) {
return new TdengineJdbcHistoryWriter(dataSource, schema);
}
@Bean
@ConditionalOnMissingBean
@ConditionalOnBean(DataSource.class)
@ConditionalOnProperty(
prefix = "lingniu.ingest.tdengine-history",
name = "enabled",
havingValue = "true")
public TdengineHistoryReader tdengineHistoryReader(DataSource dataSource, TdengineHistorySchema schema) {
return new TdengineJdbcHistoryReader(dataSource, schema);
}
}

View File

@@ -0,0 +1,119 @@
package com.lingniu.ingest.tdenginehistory.config;
import org.springframework.boot.context.properties.ConfigurationProperties;
@ConfigurationProperties(prefix = "lingniu.ingest.tdengine-history")
public class TdengineHistoryProperties {
/**
* 默认关闭,避免未配置 TDengine 连接时影响现有历史查询服务启动。
*/
private boolean enabled = false;
/**
* TDengine 历史库名。
*/
private String database = "vehicle_history";
/**
* TDengine RESTful JDBC 地址。默认走 6041部署时可用 TDENGINE_JDBC_URL 覆盖。
*/
private String jdbcUrl;
private String username = "root";
private String password = "taosdata";
private String driverClassName = "com.taosdata.jdbc.rs.RestfulDriver";
private int maximumPoolSize = 16;
private int minimumIdle = 0;
private long connectionTimeoutMillis = 5000;
private long initializationFailTimeoutMillis = -1;
public boolean isEnabled() {
return enabled;
}
public void setEnabled(boolean enabled) {
this.enabled = enabled;
}
public String getDatabase() {
return database;
}
public void setDatabase(String database) {
this.database = database;
}
public String getJdbcUrl() {
if (jdbcUrl == null || jdbcUrl.isBlank()) {
return "jdbc:TAOS-RS://127.0.0.1:6041/" + database;
}
return jdbcUrl;
}
public void setJdbcUrl(String jdbcUrl) {
this.jdbcUrl = jdbcUrl;
}
public String getUsername() {
return username;
}
public void setUsername(String username) {
this.username = username;
}
public String getPassword() {
return password;
}
public void setPassword(String password) {
this.password = password;
}
public String getDriverClassName() {
return driverClassName;
}
public void setDriverClassName(String driverClassName) {
this.driverClassName = driverClassName;
}
public int getMaximumPoolSize() {
return maximumPoolSize;
}
public void setMaximumPoolSize(int maximumPoolSize) {
this.maximumPoolSize = maximumPoolSize;
}
public int getMinimumIdle() {
return minimumIdle;
}
public void setMinimumIdle(int minimumIdle) {
this.minimumIdle = minimumIdle;
}
public long getConnectionTimeoutMillis() {
return connectionTimeoutMillis;
}
public void setConnectionTimeoutMillis(long connectionTimeoutMillis) {
this.connectionTimeoutMillis = connectionTimeoutMillis;
}
public long getInitializationFailTimeoutMillis() {
return initializationFailTimeoutMillis;
}
public void setInitializationFailTimeoutMillis(long initializationFailTimeoutMillis) {
this.initializationFailTimeoutMillis = initializationFailTimeoutMillis;
}
}

View File

@@ -0,0 +1 @@
com.lingniu.ingest.tdenginehistory.config.TdengineHistoryAutoConfiguration

View File

@@ -0,0 +1,193 @@
package com.lingniu.ingest.tdenginehistory;
import com.lingniu.ingest.sink.mq.proto.LocationPayload;
import com.lingniu.ingest.sink.mq.proto.ParseStatusProto;
import com.lingniu.ingest.sink.mq.proto.RawArchiveRef;
import com.lingniu.ingest.sink.mq.proto.RawFrameFactPayload;
import com.lingniu.ingest.sink.mq.proto.TelemetryField;
import com.lingniu.ingest.sink.mq.proto.TelemetrySnapshot;
import com.lingniu.ingest.sink.mq.proto.VehicleEnvelope;
import org.junit.jupiter.api.Test;
import java.time.Instant;
import static org.assertj.core.api.Assertions.assertThat;
class TdengineEnvelopeRowsTest {
@Test
void mapsRawFrameFactEnvelopeToRawFrameRow() {
VehicleEnvelope envelope = VehicleEnvelope.newBuilder()
.setEventId("evt-raw-1")
.setVin("VIN123")
.setSource("JT808")
.setEventTimeMs(1_772_000_001_234L)
.setIngestTimeMs(1_772_000_001_999L)
.putMetadata("channel", "tcp-808")
.setRawFrameFact(RawFrameFactPayload.newBuilder()
.setFrameId("frame-1")
.setVehicleKey("jt808:g7gps")
.setPhone("013800000000")
.setMessageId(0x0200)
.setSubType(0)
.setRawUri("archive://jt808/2026/06/29/frame-1.bin")
.setChecksum("sha256:abc")
.setRawSizeBytes(67)
.setParseStatus(ParseStatusProto.PARSE_STATUS_SUCCEEDED)
.setPeer("10.0.0.8:30001")
.putMetadata("auth", "passed"))
.build();
TdengineRawFrameRow row = TdengineEnvelopeRows.rawFrame(envelope).orElseThrow();
assertThat(row.ts()).isEqualTo(Instant.ofEpochMilli(1_772_000_001_234L));
assertThat(row.receivedAt()).isEqualTo(Instant.ofEpochMilli(1_772_000_001_999L));
assertThat(row.protocol()).isEqualTo("JT808");
assertThat(row.vehicleKey()).isEqualTo("jt808:g7gps");
assertThat(row.vin()).isEqualTo("VIN123");
assertThat(row.phone()).isEqualTo("013800000000");
assertThat(row.messageId()).isEqualTo(0x0200);
assertThat(row.rawUri()).isEqualTo("archive://jt808/2026/06/29/frame-1.bin");
assertThat(row.metadataJson()).contains("\"channel\":\"tcp-808\"", "\"auth\":\"passed\"");
}
@Test
void mapsLocationEnvelopeToVehicleLocationRow() {
VehicleEnvelope envelope = VehicleEnvelope.newBuilder()
.setEventId("evt-location-1")
.setVin("VIN32960")
.setSource("GB32960")
.setEventTimeMs(1_772_000_002_000L)
.setIngestTimeMs(1_772_000_002_321L)
.putMetadata("vehicle_key", "vin:VIN32960")
.putMetadata("frame_id", "frame-location-1")
.setRawArchive(RawArchiveRef.newBuilder()
.setUri("archive://gb32960/2026/06/29/frame-location-1.bin")
.setChecksum("sha256:def")
.setSizeBytes(98))
.setLocation(LocationPayload.newBuilder()
.setLongitude(113.12)
.setLatitude(23.45)
.setAltitudeM(8.0)
.setSpeedKmh(42.5)
.setDirectionDeg(91.0)
.setAlarmFlag(1)
.setStatusFlag(3))
.build();
TdengineLocationRow row = TdengineEnvelopeRows.location(envelope).orElseThrow();
assertThat(row.ts()).isEqualTo(Instant.ofEpochMilli(1_772_000_002_000L));
assertThat(row.factId()).isEqualTo("evt-location-1");
assertThat(row.frameId()).isEqualTo("frame-location-1");
assertThat(row.protocol()).isEqualTo("GB32960");
assertThat(row.vehicleKey()).isEqualTo("vin:VIN32960");
assertThat(row.longitude()).isEqualTo(113.12);
assertThat(row.latitude()).isEqualTo(23.45);
assertThat(row.speedKmh()).isEqualTo(42.5);
assertThat(row.totalMileageKm()).isNull();
assertThat(row.rawUri()).isEqualTo("archive://gb32960/2026/06/29/frame-location-1.bin");
}
@Test
void mapsLocationRawUriFromMetadataWhenRawArchiveRefIsAbsent() {
VehicleEnvelope envelope = VehicleEnvelope.newBuilder()
.setEventId("evt-location-metadata-raw")
.setVin("unknown")
.setSource("JT808")
.setEventTimeMs(1_772_000_002_000L)
.setIngestTimeMs(1_772_000_002_321L)
.putMetadata("phone", "13079963291")
.putMetadata("rawArchiveUri", "archive://jt808/metadata-only.bin")
.setLocation(LocationPayload.newBuilder()
.setLongitude(119.12)
.setLatitude(29.45))
.build();
TdengineLocationRow row = TdengineEnvelopeRows.location(envelope).orElseThrow();
assertThat(row.rawUri()).isEqualTo("archive://jt808/metadata-only.bin");
}
@Test
void mapsTelemetrySnapshotEnvelopeToFieldRows() {
VehicleEnvelope envelope = VehicleEnvelope.newBuilder()
.setEventId("evt-telemetry-1")
.setVin("VIN32960")
.setSource("GB32960")
.setEventTimeMs(1_772_000_003_000L)
.setIngestTimeMs(1_772_000_003_456L)
.putMetadata("vehicle_key", "VIN32960")
.putMetadata("frame_id", "frame-telemetry-1")
.setTelemetrySnapshot(TelemetrySnapshot.newBuilder()
.setEventType("REALTIME")
.setRawArchiveUri("archive://gb32960/2026/06/29/frame-telemetry-1.bin")
.addFields(TelemetryField.newBuilder()
.setKey("VEHICLE.totalMileageKm")
.setValueType("DOUBLE")
.setValue("123.4")
.setUnit("km")
.setQuality("GOOD")
.setSourcePath("VEHICLE.totalMileageKm"))
.addFields(TelemetryField.newBuilder()
.setKey("VEHICLE.runningMode")
.setValueType("STRING")
.setValue("FUEL_CELL")
.setQuality("GOOD")
.setSourcePath("VEHICLE.runningMode")))
.build();
var rows = TdengineEnvelopeRows.telemetryFields(envelope);
assertThat(rows).hasSize(2);
TdengineTelemetryFieldRow totalMileage = rows.getFirst();
assertThat(totalMileage.ts()).isEqualTo(Instant.ofEpochMilli(1_772_000_003_000L));
assertThat(totalMileage.factId()).isEqualTo("evt-telemetry-1#0");
assertThat(totalMileage.frameId()).isEqualTo("frame-telemetry-1");
assertThat(totalMileage.protocol()).isEqualTo("GB32960");
assertThat(totalMileage.vehicleKey()).isEqualTo("VIN32960");
assertThat(totalMileage.fieldKey()).isEqualTo("VEHICLE.totalMileageKm");
assertThat(totalMileage.valueText()).isEqualTo("123.4");
assertThat(totalMileage.valueDouble()).isEqualTo(123.4);
assertThat(totalMileage.valueLong()).isNull();
assertThat(totalMileage.rawUri()).isEqualTo("archive://gb32960/2026/06/29/frame-telemetry-1.bin");
assertThat(rows.get(1).valueDouble()).isNull();
assertThat(rows.get(1).valueText()).isEqualTo("FUEL_CELL");
}
@Test
void mapsJt808UnknownVinRowsToPhoneVehicleKey() {
VehicleEnvelope envelope = VehicleEnvelope.newBuilder()
.setEventId("evt-jt808-phone-1")
.setVin("unknown")
.setSource("JT808")
.setEventTimeMs(1_772_000_004_000L)
.setIngestTimeMs(1_772_000_004_500L)
.putMetadata("phone", "13079963291")
.putMetadata("frame_id", "frame-jt808-phone-1")
.setRawFrameFact(RawFrameFactPayload.newBuilder()
.setFrameId("frame-jt808-phone-1")
.setPhone("13079963291")
.setMessageId(0x0200)
.setRawUri("archive://jt808/frame-jt808-phone-1.bin")
.setChecksum("sha256:phone")
.setRawSizeBytes(88)
.setParseStatus(ParseStatusProto.PARSE_STATUS_SUCCEEDED))
.setLocation(LocationPayload.newBuilder()
.setLongitude(119.1)
.setLatitude(29.2))
.setTelemetrySnapshot(TelemetrySnapshot.newBuilder()
.addFields(TelemetryField.newBuilder()
.setKey("speed_kmh")
.setValueType("DOUBLE")
.setValue("12.3")))
.build();
assertThat(TdengineEnvelopeRows.rawFrame(envelope).orElseThrow().vehicleKey())
.isEqualTo("jt808:13079963291");
assertThat(TdengineEnvelopeRows.location(envelope).orElseThrow().vehicleKey())
.isEqualTo("jt808:13079963291");
assertThat(TdengineEnvelopeRows.telemetryFields(envelope).getFirst().vehicleKey())
.isEqualTo("jt808:13079963291");
}
}

View File

@@ -0,0 +1,86 @@
package com.lingniu.ingest.tdenginehistory;
import org.junit.jupiter.api.Test;
import java.time.Instant;
import static org.assertj.core.api.Assertions.assertThat;
class TdengineHistoryQueriesTest {
private final TdengineHistorySchema schema = new TdengineHistorySchema("vehicle_history");
private final TdengineHistoryQueries queries = new TdengineHistoryQueries(schema);
@Test
void rawFrameQueryUsesVehicleChildTableAndKeysetPagination() {
TdengineRawFrameQuery query = new TdengineRawFrameQuery(
"JT808",
"jt808:g7gps",
Instant.parse("2026-06-29T00:00:00Z"),
Instant.parse("2026-06-30T00:00:00Z"),
TdengineQueryOrder.ASC,
101,
new TdenginePageCursor(Instant.parse("2026-06-29T01:00:00Z"), "frame-100"));
TdengineQueryStatement statement = queries.rawFrames(query);
assertThat(statement.sql())
.startsWith("SELECT ts, frame_id, received_at, message_id")
.contains(" FROM raw_jt808_")
.contains(" WHERE ts >= ? AND ts < ?")
.contains(" AND (ts > ? OR (ts = ? AND frame_id > ?))")
.contains(" ORDER BY ts ASC, frame_id ASC LIMIT ?");
assertThat(statement.values())
.containsExactly(
query.from(), query.to(),
query.cursor().ts(), query.cursor().ts(), query.cursor().tieBreaker(),
101);
}
@Test
void locationQueryCapsLimitAndUsesDescendingCursor() {
TdengineLocationQuery query = new TdengineLocationQuery(
"GB32960",
"vin:VIN123",
Instant.parse("2026-06-29T00:00:00Z"),
Instant.parse("2026-06-30T00:00:00Z"),
TdengineQueryOrder.DESC,
5000,
new TdenginePageCursor(Instant.parse("2026-06-29T20:00:00Z"), "fact-200"));
TdengineQueryStatement statement = queries.locations(query);
assertThat(statement.sql())
.contains(" FROM loc_gb32960_")
.contains(" AND (ts < ? OR (ts = ? AND fact_id < ?))")
.contains(" ORDER BY ts DESC, fact_id DESC LIMIT ?");
assertThat(statement.values().getLast()).isEqualTo(1001);
}
@Test
void telemetryFieldQueryUsesFieldChildTableAndKeysetPagination() {
TdengineTelemetryFieldQuery query = new TdengineTelemetryFieldQuery(
"GB32960",
"VIN123",
"VEHICLE.totalMileageKm",
Instant.parse("2026-06-29T00:00:00Z"),
Instant.parse("2026-06-30T00:00:00Z"),
TdengineQueryOrder.ASC,
100,
new TdenginePageCursor(Instant.parse("2026-06-29T01:00:00Z"), "evt-1#0"));
TdengineQueryStatement statement = queries.telemetryFields(query);
assertThat(statement.sql())
.startsWith("SELECT ts, fact_id, frame_id, received_at, field_key")
.contains(" FROM tf_gb32960_")
.contains(" WHERE ts >= ? AND ts < ?")
.contains(" AND (ts > ? OR (ts = ? AND fact_id > ?))")
.contains(" ORDER BY ts ASC, fact_id ASC LIMIT ?");
assertThat(statement.values())
.containsExactly(
query.from(), query.to(),
query.cursor().ts(), query.cursor().ts(), query.cursor().tieBreaker(),
100);
}
}

View File

@@ -0,0 +1,50 @@
package com.lingniu.ingest.tdenginehistory;
import org.junit.jupiter.api.Test;
import static org.assertj.core.api.Assertions.assertThat;
class TdengineHistorySchemaTest {
@Test
void bootstrapSqlCreatesRawLocationAndTelemetryStablesFromSpec() {
TdengineHistorySchema schema = new TdengineHistorySchema("vehicle_history");
assertThat(schema.bootstrapSql())
.first()
.isEqualTo("CREATE DATABASE IF NOT EXISTS vehicle_history PRECISION 'ms'");
assertThat(schema.bootstrapSql())
.contains("USE vehicle_history");
assertThat(schema.bootstrapSql().get(2))
.contains("CREATE STABLE IF NOT EXISTS raw_frames")
.contains("frame_id NCHAR(64)")
.contains("raw_uri NCHAR(512)")
.contains("metadata_json NCHAR(4096)")
.contains("TAGS (")
.contains("protocol NCHAR(16)")
.contains("vehicle_key NCHAR(128)")
.contains("phone NCHAR(32)");
assertThat(schema.bootstrapSql().get(3))
.contains("CREATE STABLE IF NOT EXISTS vehicle_locations")
.contains("longitude DOUBLE")
.contains("total_mileage_km DOUBLE")
.contains("raw_uri NCHAR(512)");
assertThat(schema.bootstrapSql().get(4))
.contains("CREATE STABLE IF NOT EXISTS telemetry_fields")
.contains("value_double DOUBLE")
.contains("value_long BIGINT")
.contains("field_key NCHAR(256)");
}
@Test
void childTableNamesAreStableAndSanitized() {
TdengineHistorySchema schema = new TdengineHistorySchema("vehicle_history");
assertThat(schema.rawFrameTable("JT808", "jt808:g7gps/013800000000"))
.matches("raw_jt808_[0-9a-f]{16}");
assertThat(schema.locationTable("GB32960", "VIN WITH SPACE"))
.matches("loc_gb32960_[0-9a-f]{16}");
assertThat(schema.telemetryFieldTable("GB32960", "VIN WITH SPACE", "VEHICLE.totalMileageKm"))
.matches("tf_gb32960_[0-9a-f]{16}_[0-9a-f]{16}");
}
}

View File

@@ -0,0 +1,121 @@
package com.lingniu.ingest.tdenginehistory;
import org.junit.jupiter.api.Test;
import java.time.Instant;
import static org.assertj.core.api.Assertions.assertThat;
class TdengineHistoryStatementsTest {
private final TdengineHistorySchema schema = new TdengineHistorySchema("vehicle_history");
private final TdengineHistoryStatements statements = new TdengineHistoryStatements(schema);
@Test
void rawFrameBatchStatementCreatesChildTableAndParameterizedInsert() {
TdengineRawFrameRow row = new TdengineRawFrameRow(
Instant.parse("2026-06-29T05:00:01Z"),
"frame-1",
Instant.parse("2026-06-29T05:00:02Z"),
0x0200,
0,
Instant.parse("2026-06-29T05:00:01Z"),
"archive://jt808/frame-1.bin",
"sha256:abc",
67,
"SUCCEEDED",
"",
"10.0.0.1:808",
"{\"auth\":\"passed\"}",
"JT808",
"jt808:g7gps",
"VIN123",
"013800000000");
TdengineBatchStatement batch = statements.rawFrame(row);
assertThat(batch.createChildTableSql())
.startsWith("CREATE TABLE IF NOT EXISTS raw_jt808_")
.contains(" USING raw_frames TAGS ('JT808', 'jt808:g7gps', 'VIN123', '013800000000')");
assertThat(batch.insertSql())
.startsWith("INSERT INTO raw_jt808_")
.contains("(ts, frame_id, received_at, message_id, sub_type, event_time, raw_uri, checksum, raw_size_bytes, parse_status, parse_error, peer, metadata_json)")
.endsWith("VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)");
assertThat(batch.values())
.containsExactly(
row.ts(), row.frameId(), row.receivedAt(), row.messageId(), row.subType(),
row.eventTime(), row.rawUri(), row.checksum(), row.rawSizeBytes(), row.parseStatus(),
row.parseError(), row.peer(), row.metadataJson());
}
@Test
void locationBatchStatementEscapesTags() {
TdengineLocationRow row = new TdengineLocationRow(
Instant.parse("2026-06-29T05:00:01Z"),
"fact-1",
"frame-1",
Instant.parse("2026-06-29T05:00:02Z"),
113.1,
23.4,
8.0,
42.0,
90.0,
1,
3,
null,
"archive://gb32960/frame-1.bin",
"{}",
"GB32960",
"vin:O'HARE",
"O'HARE",
"");
TdengineBatchStatement batch = statements.location(row);
assertThat(batch.createChildTableSql())
.startsWith("CREATE TABLE IF NOT EXISTS loc_gb32960_")
.contains(" USING vehicle_locations TAGS ('GB32960', 'vin:O''HARE', 'O''HARE', '')");
assertThat(batch.values())
.containsExactly(
row.ts(), row.factId(), row.frameId(), row.receivedAt(), row.longitude(), row.latitude(),
row.altitudeM(), row.speedKmh(), row.directionDeg(), row.alarmFlag(), row.statusFlag(),
row.totalMileageKm(), row.rawUri(), row.metadataJson());
}
@Test
void telemetryFieldBatchStatementUsesFieldKeyTagAndTypedValues() {
TdengineTelemetryFieldRow row = new TdengineTelemetryFieldRow(
Instant.parse("2026-06-29T05:00:01Z"),
"evt-1#0",
"frame-1",
Instant.parse("2026-06-29T05:00:02Z"),
"VEHICLE.totalMileageKm",
"DOUBLE",
"123.4",
123.4,
null,
"km",
"GOOD",
"VEHICLE.totalMileageKm",
"archive://gb32960/frame-1.bin",
"{}",
"GB32960",
"VIN123",
"VIN123",
"");
TdengineBatchStatement batch = statements.telemetryField(row);
assertThat(batch.createChildTableSql())
.startsWith("CREATE TABLE IF NOT EXISTS tf_gb32960_")
.contains(" USING telemetry_fields TAGS ('GB32960', 'VIN123', 'VIN123', '', 'VEHICLE.totalMileageKm')");
assertThat(batch.insertSql())
.contains("(ts, fact_id, frame_id, received_at, value_type, value_text, value_double, value_long, unit, quality, source_path, raw_uri, metadata_json)")
.endsWith("VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)");
assertThat(batch.values())
.containsExactly(
row.ts(), row.factId(), row.frameId(), row.receivedAt(), row.valueType(), row.valueText(),
row.valueDouble(), row.valueLong(), row.unit(), row.quality(), row.sourcePath(),
row.rawUri(), row.metadataJson());
}
}

View File

@@ -0,0 +1,270 @@
package com.lingniu.ingest.tdenginehistory;
import org.junit.jupiter.api.Test;
import javax.sql.DataSource;
import java.lang.reflect.Proxy;
import java.sql.Connection;
import java.sql.PreparedStatement;
import java.sql.ResultSet;
import java.sql.SQLException;
import java.sql.Timestamp;
import java.time.Instant;
import java.util.ArrayList;
import java.util.HashMap;
import java.util.LinkedHashMap;
import java.util.List;
import java.util.Map;
import static org.assertj.core.api.Assertions.assertThat;
class TdengineJdbcHistoryReaderTest {
@Test
void readsRawFramesAndReturnsNextCursorWhenLimitHasMoreRow() throws Exception {
RecordingQueryJdbc jdbc = new RecordingQueryJdbc();
jdbc.rows.add(rawFrameResult("frame-1", "2026-06-29T05:00:01Z"));
jdbc.rows.add(rawFrameResult("frame-2", "2026-06-29T05:00:02Z"));
TdengineJdbcHistoryReader reader = new TdengineJdbcHistoryReader(
jdbc.dataSource(), new TdengineHistorySchema("vehicle_history"));
TdenginePage<TdengineRawFrameRow> page = reader.queryRawFrames(new TdengineRawFrameQuery(
"JT808",
"jt808:g7gps",
Instant.parse("2026-06-29T00:00:00Z"),
Instant.parse("2026-06-30T00:00:00Z"),
TdengineQueryOrder.ASC,
1,
null));
assertThat(page.items())
.extracting(TdengineRawFrameRow::frameId)
.containsExactly("frame-1");
assertThat(page.nextCursor())
.contains(new TdenginePageCursor(Instant.parse("2026-06-29T05:00:01Z"), "frame-1"));
assertThat(jdbc.preparedSql).contains("LIMIT ?");
assertThat(jdbc.boundValues.get(3)).isEqualTo(2);
}
@Test
void readsLocationRowsWithoutNextCursorWhenPageIsComplete() throws Exception {
RecordingQueryJdbc jdbc = new RecordingQueryJdbc();
jdbc.rows.add(locationResult("fact-1", "2026-06-29T05:00:01Z"));
TdengineJdbcHistoryReader reader = new TdengineJdbcHistoryReader(
jdbc.dataSource(), new TdengineHistorySchema("vehicle_history"));
TdenginePage<TdengineLocationRow> page = reader.queryLocations(new TdengineLocationQuery(
"GB32960",
"vin:VIN123",
Instant.parse("2026-06-29T00:00:00Z"),
Instant.parse("2026-06-30T00:00:00Z"),
TdengineQueryOrder.DESC,
10,
null));
assertThat(page.items())
.extracting(TdengineLocationRow::factId)
.containsExactly("fact-1");
assertThat(page.nextCursor()).isEmpty();
assertThat(page.items().getFirst().longitude()).isEqualTo(113.12);
}
@Test
void readsTelemetryFieldRowsWithNextCursor() throws Exception {
RecordingQueryJdbc jdbc = new RecordingQueryJdbc();
jdbc.rows.add(telemetryFieldResult("evt-1#0", "2026-06-29T05:00:01Z"));
jdbc.rows.add(telemetryFieldResult("evt-2#0", "2026-06-29T05:00:02Z"));
TdengineJdbcHistoryReader reader = new TdengineJdbcHistoryReader(
jdbc.dataSource(), new TdengineHistorySchema("vehicle_history"));
TdenginePage<TdengineTelemetryFieldRow> page = reader.queryTelemetryFields(new TdengineTelemetryFieldQuery(
"GB32960",
"VIN123",
"VEHICLE.totalMileageKm",
Instant.parse("2026-06-29T00:00:00Z"),
Instant.parse("2026-06-30T00:00:00Z"),
TdengineQueryOrder.ASC,
1,
null));
assertThat(page.items())
.extracting(TdengineTelemetryFieldRow::factId)
.containsExactly("evt-1#0");
assertThat(page.items().getFirst().fieldKey()).isEqualTo("VEHICLE.totalMileageKm");
assertThat(page.items().getFirst().valueDouble()).isEqualTo(123.4);
assertThat(page.nextCursor())
.contains(new TdenginePageCursor(Instant.parse("2026-06-29T05:00:01Z"), "evt-1#0"));
}
@Test
void returnsEmptyPageWhenTdengineChildTableDoesNotExist() throws Exception {
RecordingQueryJdbc jdbc = new RecordingQueryJdbc();
jdbc.executeQueryFailure = new SQLException("(0x2603):Fail to get table info, error: Table does not exist");
TdengineJdbcHistoryReader reader = new TdengineJdbcHistoryReader(
jdbc.dataSource(), new TdengineHistorySchema("vehicle_history"));
TdenginePage<TdengineTelemetryFieldRow> page = reader.queryTelemetryFields(new TdengineTelemetryFieldQuery(
"JT808",
"jt808:g7gps",
"location.speed_kmh",
Instant.parse("2026-06-29T00:00:00Z"),
Instant.parse("2026-06-30T00:00:00Z"),
TdengineQueryOrder.ASC,
10,
null));
assertThat(page.items()).isEmpty();
assertThat(page.nextCursor()).isEmpty();
}
@Test
void returnsEmptyPageWhenTdengineChildTableDoesNotExistInNestedCause() throws Exception {
RecordingQueryJdbc jdbc = new RecordingQueryJdbc();
jdbc.executeQueryFailure = new SQLException("wrapper",
new RuntimeException("TDengine error: table not exist"));
TdengineJdbcHistoryReader reader = new TdengineJdbcHistoryReader(
jdbc.dataSource(), new TdengineHistorySchema("vehicle_history"));
TdenginePage<TdengineLocationRow> page = reader.queryLocations(new TdengineLocationQuery(
"JT808",
"jt808:g7gps",
Instant.parse("2026-06-29T00:00:00Z"),
Instant.parse("2026-06-30T00:00:00Z"),
TdengineQueryOrder.ASC,
10,
null));
assertThat(page.items()).isEmpty();
assertThat(page.nextCursor()).isEmpty();
}
private static Map<String, Object> rawFrameResult(String frameId, String ts) {
Map<String, Object> row = new LinkedHashMap<>();
row.put("ts", Timestamp.from(Instant.parse(ts)));
row.put("frame_id", frameId);
row.put("received_at", Timestamp.from(Instant.parse(ts).plusMillis(500)));
row.put("message_id", 0x0200);
row.put("sub_type", 0);
row.put("event_time", Timestamp.from(Instant.parse(ts)));
row.put("raw_uri", "archive://jt808/" + frameId + ".bin");
row.put("checksum", "sha256:" + frameId);
row.put("raw_size_bytes", 67L);
row.put("parse_status", "SUCCEEDED");
row.put("parse_error", "");
row.put("peer", "10.0.0.1:808");
row.put("metadata_json", "{}");
row.put("protocol", "JT808");
row.put("vehicle_key", "jt808:g7gps");
row.put("vin", "VIN123");
row.put("phone", "013800000000");
return row;
}
private static Map<String, Object> locationResult(String factId, String ts) {
Map<String, Object> row = new LinkedHashMap<>();
row.put("ts", Timestamp.from(Instant.parse(ts)));
row.put("fact_id", factId);
row.put("frame_id", "frame-1");
row.put("received_at", Timestamp.from(Instant.parse(ts).plusMillis(500)));
row.put("longitude", 113.12);
row.put("latitude", 23.45);
row.put("altitude_m", 8.0);
row.put("speed_kmh", 42.5);
row.put("direction_deg", 90.0);
row.put("alarm_flag", 1L);
row.put("status_flag", 3L);
row.put("total_mileage_km", null);
row.put("raw_uri", "archive://gb32960/frame-1.bin");
row.put("metadata_json", "{}");
row.put("protocol", "GB32960");
row.put("vehicle_key", "vin:VIN123");
row.put("vin", "VIN123");
row.put("phone", "");
return row;
}
private static Map<String, Object> telemetryFieldResult(String factId, String ts) {
Map<String, Object> row = new LinkedHashMap<>();
row.put("ts", Timestamp.from(Instant.parse(ts)));
row.put("fact_id", factId);
row.put("frame_id", "frame-1");
row.put("received_at", Timestamp.from(Instant.parse(ts).plusMillis(500)));
row.put("field_key", "VEHICLE.totalMileageKm");
row.put("value_type", "DOUBLE");
row.put("value_text", "123.4");
row.put("value_double", 123.4);
row.put("value_long", null);
row.put("unit", "km");
row.put("quality", "GOOD");
row.put("source_path", "VEHICLE.totalMileageKm");
row.put("raw_uri", "archive://gb32960/frame-1.bin");
row.put("metadata_json", "{}");
row.put("protocol", "GB32960");
row.put("vehicle_key", "VIN123");
row.put("vin", "VIN123");
row.put("phone", "");
return row;
}
private static final class RecordingQueryJdbc {
private final List<Map<String, Object>> rows = new ArrayList<>();
private final Map<Integer, Object> boundValues = new HashMap<>();
private SQLException executeQueryFailure;
private String preparedSql;
DataSource dataSource() {
return (DataSource) Proxy.newProxyInstance(
getClass().getClassLoader(),
new Class<?>[]{DataSource.class},
(proxy, method, args) -> switch (method.getName()) {
case "getConnection" -> connection();
default -> null;
});
}
private Connection connection() {
return (Connection) Proxy.newProxyInstance(
getClass().getClassLoader(),
new Class<?>[]{Connection.class},
(proxy, method, args) -> switch (method.getName()) {
case "prepareStatement" -> preparedStatement((String) args[0]);
case "getAutoCommit" -> true;
default -> null;
});
}
private PreparedStatement preparedStatement(String sql) {
preparedSql = sql;
return (PreparedStatement) Proxy.newProxyInstance(
getClass().getClassLoader(),
new Class<?>[]{PreparedStatement.class},
(proxy, method, args) -> switch (method.getName()) {
case "setTimestamp", "setObject" -> {
boundValues.put((Integer) args[0], args[1]);
yield null;
}
case "executeQuery" -> {
if (executeQueryFailure != null) {
throw executeQueryFailure;
}
yield resultSet();
}
default -> null;
});
}
private ResultSet resultSet() {
final int[] index = {-1};
return (ResultSet) Proxy.newProxyInstance(
getClass().getClassLoader(),
new Class<?>[]{ResultSet.class},
(proxy, method, args) -> switch (method.getName()) {
case "next" -> ++index[0] < rows.size();
case "getTimestamp", "getString", "getInt", "getLong", "getDouble", "getObject" ->
rows.get(index[0]).get((String) args[0]);
case "wasNull" -> false;
default -> null;
});
}
}
}

View File

@@ -0,0 +1,331 @@
package com.lingniu.ingest.tdenginehistory;
import org.junit.jupiter.api.Test;
import javax.sql.DataSource;
import java.lang.reflect.Proxy;
import java.sql.Connection;
import java.sql.PreparedStatement;
import java.sql.SQLException;
import java.sql.Statement;
import java.sql.Timestamp;
import java.time.Instant;
import java.util.ArrayList;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
import static org.assertj.core.api.Assertions.assertThat;
class TdengineJdbcHistoryWriterTest {
private final TdengineHistorySchema schema = new TdengineHistorySchema("vehicle_history");
@Test
void initializesSchemaWithBootstrapSql() throws Exception {
RecordingJdbc jdbc = new RecordingJdbc();
TdengineJdbcHistoryWriter writer = new TdengineJdbcHistoryWriter(jdbc.dataSource(), schema);
writer.initializeSchema();
assertThat(jdbc.executedSql)
.containsExactlyElementsOf(schema.bootstrapSql());
assertThat(jdbc.commits).isEqualTo(1);
}
@Test
void writesRawFramesInBatchesPerChildTable() throws Exception {
RecordingJdbc jdbc = new RecordingJdbc();
TdengineJdbcHistoryWriter writer = new TdengineJdbcHistoryWriter(jdbc.dataSource(), schema);
TdengineRawFrameRow first = rawFrame("frame-1", Instant.parse("2026-06-29T05:00:01Z"), "none");
TdengineRawFrameRow second = rawFrame("frame-2", Instant.parse("2026-06-29T05:00:02Z"), "none");
writer.appendRawFrames(List.of(first, second));
assertThat(jdbc.executedSql).startsWith(schema.bootstrapSql().toArray(String[]::new));
assertThat(jdbc.executedSql)
.filteredOn(sql -> sql.contains("USING raw_frames TAGS"))
.hasSize(1);
assertThat(jdbc.preparedBatches).hasSize(1);
RecordingJdbc.PreparedBatch batch = jdbc.preparedBatches.values().iterator().next();
assertThat(batch.sql()).contains("INSERT INTO raw_jt808_");
assertThat(batch.rows()).hasSize(2);
assertThat(batch.rows().getFirst().get(1)).isEqualTo(Timestamp.from(first.ts()));
assertThat(batch.rows().getFirst().get(2)).isEqualTo("frame-1");
assertThat(batch.rows().get(1).get(2)).isEqualTo("frame-2");
assertThat(jdbc.commits).isEqualTo(1);
}
@Test
void writesLocationRowsWithNullableMileage() throws Exception {
RecordingJdbc jdbc = new RecordingJdbc();
TdengineJdbcHistoryWriter writer = new TdengineJdbcHistoryWriter(jdbc.dataSource(), schema);
TdengineLocationRow location = new TdengineLocationRow(
Instant.parse("2026-06-29T05:00:01Z"),
"fact-1",
"frame-1",
Instant.parse("2026-06-29T05:00:02Z"),
113.1,
23.4,
8.0,
42.0,
90.0,
1,
3,
null,
"archive://gb32960/frame-1.bin",
"{}",
"GB32960",
"vin:VIN123",
"VIN123",
"");
writer.appendLocations(List.of(location));
assertThat(jdbc.executedSql).startsWith(schema.bootstrapSql().toArray(String[]::new));
assertThat(jdbc.executedSql)
.anyMatch(sql -> sql.contains("USING vehicle_locations TAGS"));
RecordingJdbc.PreparedBatch batch = jdbc.preparedBatches.values().iterator().next();
assertThat(batch.sql()).contains("INSERT INTO loc_gb32960_");
assertThat(batch.rows().getFirst().get(12)).isNull();
}
@Test
void writesTelemetryFieldsPerVehicleAndFieldChildTable() throws Exception {
RecordingJdbc jdbc = new RecordingJdbc();
TdengineJdbcHistoryWriter writer = new TdengineJdbcHistoryWriter(jdbc.dataSource(), schema);
TdengineTelemetryFieldRow first = telemetryField("evt-1#0", "VEHICLE.totalMileageKm", "123.4", 123.4);
TdengineTelemetryFieldRow second = telemetryField("evt-2#0", "VEHICLE.totalMileageKm", "124.5", 124.5);
writer.appendTelemetryFields(List.of(first, second));
assertThat(jdbc.executedSql).startsWith(schema.bootstrapSql().toArray(String[]::new));
assertThat(jdbc.executedSql)
.filteredOn(sql -> sql.contains("USING telemetry_fields TAGS"))
.hasSize(1);
RecordingJdbc.PreparedBatch batch = jdbc.preparedBatches.values().iterator().next();
assertThat(batch.sql()).contains("INSERT INTO tf_gb32960_");
assertThat(batch.rows()).hasSize(2);
assertThat(batch.rows().getFirst().get(2)).isEqualTo("evt-1#0");
assertThat(batch.rows().getFirst().get(7)).isEqualTo(123.4);
assertThat(batch.rows().get(1).get(6)).isEqualTo("124.5");
}
@Test
void schemaBootstrapRunsOnlyOnceAcrossWrites() throws Exception {
RecordingJdbc jdbc = new RecordingJdbc();
TdengineJdbcHistoryWriter writer = new TdengineJdbcHistoryWriter(jdbc.dataSource(), schema);
writer.appendRawFrames(List.of(rawFrame("frame-1", Instant.parse("2026-06-29T05:00:01Z"))));
writer.appendRawFrames(List.of(rawFrame("frame-2", Instant.parse("2026-06-29T05:00:02Z"))));
assertThat(jdbc.executedSql)
.filteredOn(sql -> sql.startsWith("CREATE DATABASE"))
.hasSize(1);
assertThat(jdbc.commits).isEqualTo(2);
}
@Test
void fallsBackToLiteralInsertWhenPreparedBatchFails() throws Exception {
RecordingJdbc jdbc = new RecordingJdbc();
jdbc.failPreparedBatches = true;
TdengineJdbcHistoryWriter writer = new TdengineJdbcHistoryWriter(jdbc.dataSource(), schema);
writer.appendRawFrames(List.of(rawFrame("frame-1", Instant.parse("2026-06-29T05:00:01Z"), "none")));
assertThat(jdbc.preparedBatches).hasSize(1);
assertThat(jdbc.executedSql)
.anyMatch(sql -> sql.startsWith("INSERT INTO raw_jt808_")
&& sql.contains("parse_error")
&& sql.contains("'SUCCEEDED', 'none', '10.0.0.1:808'"));
assertThat(jdbc.commits).isEqualTo(1);
}
@Test
void usesLiteralInsertForRowsWithEmptyStrings() throws Exception {
RecordingJdbc jdbc = new RecordingJdbc();
TdengineJdbcHistoryWriter writer = new TdengineJdbcHistoryWriter(jdbc.dataSource(), schema);
writer.appendRawFrames(List.of(rawFrame("frame-1", Instant.parse("2026-06-29T05:00:01Z"))));
assertThat(jdbc.preparedBatches).isEmpty();
assertThat(jdbc.executedSql)
.anyMatch(sql -> sql.startsWith("INSERT INTO raw_jt808_")
&& sql.contains("parse_error")
&& sql.contains("'SUCCEEDED', '', '10.0.0.1:808'"));
assertThat(jdbc.commits).isEqualTo(1);
}
@Test
void groupsLiteralRowsForSameChildTableIntoOneInsert() throws Exception {
RecordingJdbc jdbc = new RecordingJdbc();
TdengineJdbcHistoryWriter writer = new TdengineJdbcHistoryWriter(jdbc.dataSource(), schema);
writer.appendRawFrames(List.of(
rawFrame("frame-1", Instant.parse("2026-06-29T05:00:01Z")),
rawFrame("frame-2", Instant.parse("2026-06-29T05:00:02Z"))));
List<String> inserts = jdbc.executedSql.stream()
.filter(sql -> sql.startsWith("INSERT INTO raw_jt808_"))
.toList();
assertThat(inserts).hasSize(1);
assertThat(inserts.getFirst())
.contains("frame-1")
.contains("frame-2")
.contains(") (");
}
private static TdengineRawFrameRow rawFrame(String frameId, Instant ts) {
return rawFrame(frameId, ts, "");
}
private static TdengineRawFrameRow rawFrame(String frameId, Instant ts, String parseError) {
return new TdengineRawFrameRow(
ts,
frameId,
ts.plusSeconds(1),
0x0200,
0,
ts,
"archive://jt808/" + frameId + ".bin",
"sha256:" + frameId,
67,
"SUCCEEDED",
parseError,
"10.0.0.1:808",
"{}",
"JT808",
"jt808:g7gps",
"VIN123",
"013800000000");
}
private static TdengineTelemetryFieldRow telemetryField(String factId,
String fieldKey,
String value,
Double valueDouble) {
return new TdengineTelemetryFieldRow(
Instant.parse("2026-06-29T05:00:01Z"),
factId,
"frame-1",
Instant.parse("2026-06-29T05:00:02Z"),
fieldKey,
"DOUBLE",
value,
valueDouble,
null,
"km",
"GOOD",
fieldKey,
"archive://gb32960/frame-1.bin",
"{}",
"GB32960",
"VIN123",
"VIN123",
"");
}
private static final class RecordingJdbc {
private final List<String> executedSql = new ArrayList<>();
private final Map<String, PreparedBatch> preparedBatches = new HashMap<>();
private boolean failPreparedBatches;
private int commits;
private int rollbacks;
DataSource dataSource() {
return (DataSource) Proxy.newProxyInstance(
getClass().getClassLoader(),
new Class<?>[]{DataSource.class},
(proxy, method, args) -> switch (method.getName()) {
case "getConnection" -> connection();
case "unwrap" -> null;
case "isWrapperFor" -> false;
default -> defaultValue(method.getReturnType());
});
}
private Connection connection() {
return (Connection) Proxy.newProxyInstance(
getClass().getClassLoader(),
new Class<?>[]{Connection.class},
(proxy, method, args) -> switch (method.getName()) {
case "createStatement" -> statement();
case "prepareStatement" -> preparedStatement((String) args[0]);
case "commit" -> {
commits++;
yield null;
}
case "rollback" -> {
rollbacks++;
yield null;
}
case "getAutoCommit" -> true;
case "unwrap" -> null;
case "isWrapperFor" -> false;
default -> defaultValue(method.getReturnType());
});
}
private Statement statement() {
return (Statement) Proxy.newProxyInstance(
getClass().getClassLoader(),
new Class<?>[]{Statement.class},
(proxy, method, args) -> switch (method.getName()) {
case "execute", "executeUpdate" -> {
executedSql.add((String) args[0]);
yield method.getReturnType() == boolean.class ? true : 1;
}
case "unwrap" -> null;
case "isWrapperFor" -> false;
default -> defaultValue(method.getReturnType());
});
}
private PreparedStatement preparedStatement(String sql) {
PreparedBatch batch = preparedBatches.computeIfAbsent(sql, PreparedBatch::new);
Map<Integer, Object> current = new HashMap<>();
return (PreparedStatement) Proxy.newProxyInstance(
getClass().getClassLoader(),
new Class<?>[]{PreparedStatement.class},
(proxy, method, args) -> switch (method.getName()) {
case "setTimestamp", "setObject" -> {
current.put((Integer) args[0], args[1]);
yield null;
}
case "addBatch" -> {
batch.rows().add(new HashMap<>(current));
current.clear();
yield null;
}
case "executeBatch" -> {
if (failPreparedBatches) {
throw new SQLException("simulated prepared batch failure");
}
yield new int[batch.rows().size()];
}
case "unwrap" -> null;
case "isWrapperFor" -> false;
default -> defaultValue(method.getReturnType());
});
}
private static Object defaultValue(Class<?> type) {
if (type == boolean.class) {
return false;
}
if (type == int.class) {
return 0;
}
if (type == long.class) {
return 0L;
}
return null;
}
record PreparedBatch(String sql, List<Map<Integer, Object>> rows) {
PreparedBatch(String sql) {
this(sql, new ArrayList<>());
}
}
}
}

View File

@@ -0,0 +1,81 @@
package com.lingniu.ingest.tdenginehistory.config;
import com.lingniu.ingest.tdenginehistory.TdengineHistorySchema;
import com.lingniu.ingest.tdenginehistory.TdengineHistoryReader;
import com.lingniu.ingest.tdenginehistory.TdengineHistoryWriter;
import com.zaxxer.hikari.HikariDataSource;
import org.junit.jupiter.api.Test;
import org.springframework.boot.autoconfigure.AutoConfigurations;
import org.springframework.boot.test.context.runner.ApplicationContextRunner;
import javax.sql.DataSource;
import static org.assertj.core.api.Assertions.assertThat;
import static org.mockito.Mockito.mock;
class TdengineHistoryAutoConfigurationTest {
private final ApplicationContextRunner contextRunner = new ApplicationContextRunner()
.withConfiguration(AutoConfigurations.of(TdengineHistoryAutoConfiguration.class));
@Test
void staysOffByDefault() {
contextRunner.run(context -> {
assertThat(context).doesNotHaveBean(TdengineHistorySchema.class);
assertThat(context).doesNotHaveBean(DataSource.class);
assertThat(context.getBean(TdengineHistoryProperties.class).isEnabled()).isFalse();
});
}
@Test
void createsSchemaWhenEnabled() {
contextRunner
.withPropertyValues(
"lingniu.ingest.tdengine-history.enabled=true",
"lingniu.ingest.tdengine-history.database=vehicle_history_hot")
.run(context -> {
assertThat(context).hasSingleBean(TdengineHistorySchema.class);
assertThat(context.getBean(TdengineHistorySchema.class).bootstrapSql().getFirst())
.isEqualTo("CREATE DATABASE IF NOT EXISTS vehicle_history_hot PRECISION 'ms'");
});
}
@Test
void createsWriterWhenEnabledAndDataSourceExists() {
contextRunner
.withBean(DataSource.class, () -> mock(DataSource.class))
.withPropertyValues("lingniu.ingest.tdengine-history.enabled=true")
.run(context -> assertThat(context).hasSingleBean(TdengineHistoryWriter.class));
}
@Test
void createsReaderWhenEnabledAndDataSourceExists() {
contextRunner
.withBean(DataSource.class, () -> mock(DataSource.class))
.withPropertyValues("lingniu.ingest.tdengine-history.enabled=true")
.run(context -> assertThat(context).hasSingleBean(TdengineHistoryReader.class));
}
@Test
void createsTdengineDataSourceWhenEnabled() {
contextRunner
.withPropertyValues(
"lingniu.ingest.tdengine-history.enabled=true",
"lingniu.ingest.tdengine-history.jdbc-url=jdbc:TAOS-RS://tdengine:6041/vehicle_history",
"lingniu.ingest.tdengine-history.username=root",
"lingniu.ingest.tdengine-history.password=secret",
"lingniu.ingest.tdengine-history.maximum-pool-size=24",
"lingniu.ingest.tdengine-history.minimum-idle=3",
"lingniu.ingest.tdengine-history.initialization-fail-timeout-millis=-1")
.run(context -> {
assertThat(context).hasSingleBean(DataSource.class);
HikariDataSource dataSource = context.getBean(HikariDataSource.class);
assertThat(dataSource.getJdbcUrl()).isEqualTo("jdbc:TAOS-RS://tdengine:6041/vehicle_history");
assertThat(dataSource.getUsername()).isEqualTo("root");
assertThat(dataSource.getMaximumPoolSize()).isEqualTo(24);
assertThat(dataSource.getMinimumIdle()).isEqualTo(3);
assertThat(context).hasSingleBean(TdengineHistoryWriter.class);
assertThat(context).hasSingleBean(TdengineHistoryReader.class);
});
}
}

Some files were not shown because too many files have changed in this diff Show More