chore: keep go branch go-only
This commit is contained in:
lingniu
@@ -1,600 +0,0 @@
|
||||
# GB32960 Body Parser — Per-Block Exception Isolation Plan
|
||||
|
||||
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
|
||||
|
||||
**Goal:** `Gb32960BodyParser` 主循环对单信息块的解析异常实现**单块隔离**：一块解析失败不再整帧放弃，改为把失败块兜成 `InfoBlock.Raw` 并继续解析后续信息块。
|
||||
|
||||
**Architecture:** 在主循环每次 `parser.parse(body)` 外面套 try/catch（仅捕获 `DecodeException` / `BufferUnderflowException` / `IndexOutOfBoundsException`，放行 RuntimeException 以免掩盖 parser bug）。失败时：
|
||||
- 若 `fixedLen ≥ 0` 且剩余字节 ≥ fixedLen → 回滚 reader → 按 fixedLen 截取 Raw → position 前进 fixedLen → `continue` 循环
|
||||
- 否则（变长 parser 或剩余不足）→ 剩余字节全部兜成 Raw → `break` 循环（变长块无法安全找下一块边界）
|
||||
|
||||
通过 `Gb32960Properties.Parse.lenientBlockFailure`（默认 true）控制开关，可以回退到严格模式。**不改 `InfoBlock.Raw` record 结构**（避免 downstream 兼容风险——`Gb32960EventMapper` 按 `findBlock(Class)` 类型查找，新增 Raw 不影响其行为）。
|
||||
|
||||
**Tech Stack:** Java 25, JUnit 5, AssertJ, Spring Boot ConfigurationProperties, Maven。仅改动 `protocol-gb32960` 模块。
|
||||
|
||||
---
|
||||
|
||||
## File Structure
|
||||
|
||||
- **Modify** `protocol-gb32960/src/main/java/com/lingniu/ingest/protocol/gb32960/config/Gb32960Properties.java` — 新增 `Parse` 内嵌类 + `parse` 字段/getter/setter
|
||||
- **Modify** `protocol-gb32960/src/main/java/com/lingniu/ingest/protocol/gb32960/codec/Gb32960BodyParser.java` — 主循环 try/catch + recovery 逻辑 + `lenientBlockFailure` 字段
|
||||
- **Modify** `protocol-gb32960/src/main/java/com/lingniu/ingest/protocol/gb32960/config/Gb32960AutoConfiguration.java` — 把配置穿给 BodyParser
|
||||
- **Create** `protocol-gb32960/src/test/java/com/lingniu/ingest/protocol/gb32960/codec/Gb32960BodyParserIsolationTest.java` — 3 个隔离场景 + 1 个严格模式回退场景
|
||||
- **Modify** `bootstrap-all/src/main/resources/application.yml` — 添加 `parse:` 注释段示例
|
||||
- **Modify** `CHANGELOG.md` — 追加本次变更条目
|
||||
|
||||
---
|
||||
|
||||
## Task 1: 基线验证 —— 现有测试全绿
|
||||
|
||||
**Files:** 无
|
||||
|
||||
- [ ] **Step 1.1: 跑 protocol-gb32960 模块测试**
|
||||
|
||||
Run: `mvn -pl protocol-gb32960 test -q`
|
||||
Expected: BUILD SUCCESS，所有现有测试通过（`Gb32960DecoderGoldenTest`、`Gb32960FullBlocksTest`、`Gb32960DecoderTest`、`GuangdongFcEndToEndTest`、parser/* 各 Block Parser 单测、profile/* 选择器单测）。
|
||||
|
||||
如果基线红了，**先停下来修复基线**再进入 Task 2。
|
||||
|
||||
---
|
||||
|
||||
## Task 2: 添加 Parse 配置子节点
|
||||
|
||||
**Files:**
|
||||
- Modify: `protocol-gb32960/src/main/java/com/lingniu/ingest/protocol/gb32960/config/Gb32960Properties.java`
|
||||
|
||||
- [ ] **Step 2.1: 在 `Gb32960Properties` 类体中（在 `Auth` 字段后、`Tls` 字段前）新增 `parse` 字段**
|
||||
|
||||
位置参考：现在 L24 `private Auth auth = new Auth();` 下面一行，在 L27 `private Tls tls = new Tls();` 之前插入：
|
||||
|
||||
```java
|
||||
/**
|
||||
* 报文解析行为配置。
|
||||
*
|
||||
* <p>默认 {@code lenientBlockFailure=true}：单个信息块解析异常时兜成
|
||||
* {@link com.lingniu.ingest.protocol.gb32960.model.InfoBlock.Raw} 继续解析，
|
||||
* 不再整帧放弃。关闭后恢复旧行为（任意异常直接抛 {@code DecodeException}
|
||||
* 放弃整帧），仅在灰度回滚时使用。
|
||||
*/
|
||||
private Parse parse = new Parse();
|
||||
```
|
||||
|
||||
- [ ] **Step 2.2: 在类底部（`Tls` 静态类之后、`VendorExtension` 静态类之前）添加 `Parse` 静态内嵌类**
|
||||
|
||||
```java
|
||||
/**
|
||||
* 报文解析容错策略。
|
||||
*/
|
||||
public static class Parse {
|
||||
/**
|
||||
* 单块解析异常时是否兜底为 {@link com.lingniu.ingest.protocol.gb32960.model.InfoBlock.Raw}
|
||||
* 继续解析。默认开启。
|
||||
*
|
||||
* <ul>
|
||||
* <li>{@code true}（默认）：parser 抛 DecodeException / BufferUnderflowException /
|
||||
* IndexOutOfBoundsException 时，固定长度块按 fixedLen 截取 Raw 后 continue；
|
||||
* 变长块或剩余字节不足时，剩余全部兜成 Raw 后 break 循环。
|
||||
* <li>{@code false}：保留旧行为，任意异常直接抛出 DecodeException 放弃整帧。
|
||||
* </ul>
|
||||
*/
|
||||
private boolean lenientBlockFailure = true;
|
||||
|
||||
public boolean isLenientBlockFailure() { return lenientBlockFailure; }
|
||||
public void setLenientBlockFailure(boolean lenientBlockFailure) {
|
||||
this.lenientBlockFailure = lenientBlockFailure;
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
- [ ] **Step 2.3: 在属性 getter/setter 区域（现有 `getTls` 之后）补充 `getParse` / `setParse`**
|
||||
|
||||
```java
|
||||
public Parse getParse() { return parse; }
|
||||
public void setParse(Parse parse) { this.parse = parse; }
|
||||
```
|
||||
|
||||
- [ ] **Step 2.4: 编译验证**
|
||||
|
||||
Run: `mvn -pl protocol-gb32960 compile -q`
|
||||
Expected: BUILD SUCCESS。
|
||||
|
||||
- [ ] **Step 2.5: 提交**
|
||||
|
||||
```bash
|
||||
git add protocol-gb32960/src/main/java/com/lingniu/ingest/protocol/gb32960/config/Gb32960Properties.java
|
||||
git commit -m "$(cat <<'EOF'
|
||||
config(gb32960): add parse.lenientBlockFailure toggle
|
||||
|
||||
新增 lingniu.ingest.gb32960.parse.lenientBlockFailure 配置开关（默认 true），
|
||||
为 Gb32960BodyParser 的单块异常隔离行为做回退开关。实现在后续 commit。
|
||||
|
||||
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||||
EOF
|
||||
)"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Task 3: RED —— 写三个失败测试覆盖隔离场景
|
||||
|
||||
**Files:**
|
||||
- Create: `protocol-gb32960/src/test/java/com/lingniu/ingest/protocol/gb32960/codec/Gb32960BodyParserIsolationTest.java`
|
||||
|
||||
- [ ] **Step 3.1: 创建测试文件**
|
||||
|
||||
完整内容：
|
||||
|
||||
```java
|
||||
package com.lingniu.ingest.protocol.gb32960.codec;
|
||||
|
||||
import com.lingniu.ingest.api.spi.DecodeException;
|
||||
import com.lingniu.ingest.protocol.gb32960.codec.parser.v2016.AlarmV2016BlockParser;
|
||||
import com.lingniu.ingest.protocol.gb32960.codec.parser.v2016.PositionV2016BlockParser;
|
||||
import com.lingniu.ingest.protocol.gb32960.codec.parser.v2016.VehicleV2016BlockParser;
|
||||
import com.lingniu.ingest.protocol.gb32960.model.InfoBlock;
|
||||
import com.lingniu.ingest.protocol.gb32960.model.InfoBlockType;
|
||||
import com.lingniu.ingest.protocol.gb32960.model.ProtocolVersion;
|
||||
import org.junit.jupiter.api.Test;
|
||||
|
||||
import java.io.ByteArrayOutputStream;
|
||||
import java.nio.ByteBuffer;
|
||||
import java.util.List;
|
||||
|
||||
import static org.assertj.core.api.Assertions.assertThat;
|
||||
import static org.assertj.core.api.Assertions.assertThatThrownBy;
|
||||
|
||||
/**
|
||||
* 验证 {@link Gb32960BodyParser} 单块异常隔离行为。
|
||||
*
|
||||
* <p>三个隔离场景：
|
||||
* <ol>
|
||||
* <li>固定长度块 parser 抛异常 → 失败块兜 Raw（按 fixedLen 截取）+ 后续块继续解析
|
||||
* <li>固定长度块剩余字节不足（截断帧） → 兜 Raw(剩余) + break 循环
|
||||
* <li>变长块 parser 抛异常 → 兜 Raw(从失败块起剩余全部) + break 循环
|
||||
* </ol>
|
||||
*
|
||||
* <p>外加一个严格模式回退测试：{@code lenientBlockFailure=false} 时任意异常应抛 DecodeException。
|
||||
*/
|
||||
class Gb32960BodyParserIsolationTest {
|
||||
|
||||
/** 构造一个"看起来合法但中途 parser 失败"用来注入失败的 Position parser。 */
|
||||
private static final InfoBlockParser EXPLODING_FIXED_LEN_POSITION = new InfoBlockParser() {
|
||||
@Override public ProtocolVersion version() { return ProtocolVersion.V2016; }
|
||||
@Override public int typeCode() { return 0x05; }
|
||||
@Override public int fixedLength() { return 9; }
|
||||
@Override public InfoBlock parse(ByteBuffer buffer) {
|
||||
// 模拟 parser 读了几字节后发现不合法就抛
|
||||
buffer.get(); buffer.get();
|
||||
throw new DecodeException("simulated parser failure in Position");
|
||||
}
|
||||
};
|
||||
|
||||
/** 变长块（fixedLength=-1），parse 时消费若干字节后抛。模拟 Alarm/Voltage 类列表读越界。 */
|
||||
private static final InfoBlockParser EXPLODING_VAR_LEN_ALARM = new InfoBlockParser() {
|
||||
@Override public ProtocolVersion version() { return ProtocolVersion.V2016; }
|
||||
@Override public int typeCode() { return 0x07; }
|
||||
@Override public int fixedLength() { return -1; }
|
||||
@Override public InfoBlock parse(ByteBuffer buffer) {
|
||||
// 假装读了个长度字段就爆
|
||||
buffer.get();
|
||||
throw new DecodeException("simulated parser failure in Alarm list-length read");
|
||||
}
|
||||
};
|
||||
|
||||
@Test
|
||||
void fixedLengthBlockFailure_isIsolated_subsequentBlocksStillParsed() {
|
||||
// 帧布局：Vehicle(0x01, 20B) + Position(0x05, 9B 但 parser 爆) + Engine(0x04) 不构造，
|
||||
// 简化为 Vehicle + FailingPosition + Vehicle 再次，验证"Position 之后还能继续"。
|
||||
InfoBlockParserRegistry registry = new InfoBlockParserRegistry(List.of(
|
||||
new VehicleV2016BlockParser(),
|
||||
EXPLODING_FIXED_LEN_POSITION));
|
||||
|
||||
Gb32960BodyParser parser = new Gb32960BodyParser(registry);
|
||||
|
||||
ByteArrayOutputStream os = new ByteArrayOutputStream();
|
||||
writeValidVehicle(os); // 1+20 = 21B
|
||||
writePositionTypeAnd9ByteBody(os); // 1+9 = 10B (parser 会爆)
|
||||
writeValidVehicle(os); // 1+20 = 21B
|
||||
ByteBuffer body = ByteBuffer.wrap(os.toByteArray());
|
||||
|
||||
var result = parser.parse(ProtocolVersion.V2016, body);
|
||||
|
||||
assertThat(result.blocks()).hasSize(3);
|
||||
assertThat(result.blocks().get(0)).isInstanceOf(InfoBlock.Gb32960V2016.Vehicle.class);
|
||||
assertThat(result.blocks().get(1)).isInstanceOfSatisfying(InfoBlock.Raw.class, raw -> {
|
||||
assertThat(raw.typeCode()).isEqualTo(0x05);
|
||||
assertThat(raw.type()).isEqualTo(InfoBlockType.RAW);
|
||||
assertThat(raw.bytes()).hasSize(9); // 按 fixedLen 截取
|
||||
});
|
||||
assertThat(result.blocks().get(2)).isInstanceOf(InfoBlock.Gb32960V2016.Vehicle.class);
|
||||
assertThat(body.hasRemaining()).isFalse();
|
||||
}
|
||||
|
||||
@Test
|
||||
void truncatedFixedLengthBlock_isWrappedAsRaw_loopTerminates() {
|
||||
// Vehicle(21B) + Position(type 0x05)但 body 只给 3B —— 不足 9B
|
||||
InfoBlockParserRegistry registry = new InfoBlockParserRegistry(List.of(
|
||||
new VehicleV2016BlockParser(),
|
||||
new PositionV2016BlockParser()));
|
||||
|
||||
Gb32960BodyParser parser = new Gb32960BodyParser(registry);
|
||||
|
||||
ByteArrayOutputStream os = new ByteArrayOutputStream();
|
||||
writeValidVehicle(os);
|
||||
os.write(0x05); // Position typeCode
|
||||
os.write(0); os.write(0); os.write(0); // 只 3B，远远不够 9B
|
||||
ByteBuffer body = ByteBuffer.wrap(os.toByteArray());
|
||||
|
||||
var result = parser.parse(ProtocolVersion.V2016, body);
|
||||
|
||||
assertThat(result.blocks()).hasSize(2);
|
||||
assertThat(result.blocks().get(0)).isInstanceOf(InfoBlock.Gb32960V2016.Vehicle.class);
|
||||
assertThat(result.blocks().get(1)).isInstanceOfSatisfying(InfoBlock.Raw.class, raw -> {
|
||||
assertThat(raw.typeCode()).isEqualTo(0x05);
|
||||
assertThat(raw.bytes()).hasSize(3); // 剩余字节全兜成 Raw
|
||||
});
|
||||
}
|
||||
|
||||
@Test
|
||||
void variableLengthBlockFailure_swallowsRemainderAsRaw_loopBreaks() {
|
||||
// Vehicle(21B) + FailingAlarm(0x07)后面还有一个 Vehicle，但因 Alarm 变长无法安全跳过，
|
||||
// 整段 Alarm 起的剩余字节都被兜成 Raw 后 break。
|
||||
InfoBlockParserRegistry registry = new InfoBlockParserRegistry(List.of(
|
||||
new VehicleV2016BlockParser(),
|
||||
EXPLODING_VAR_LEN_ALARM));
|
||||
|
||||
Gb32960BodyParser parser = new Gb32960BodyParser(registry);
|
||||
|
||||
ByteArrayOutputStream os = new ByteArrayOutputStream();
|
||||
writeValidVehicle(os); // 21B
|
||||
os.write(0x07); // Alarm typeCode
|
||||
for (int i = 0; i < 10; i++) os.write(0xAA); // 10B 的 alarm body（parser 爆）
|
||||
writeValidVehicle(os); // 21B（应当不再被解析）
|
||||
ByteBuffer body = ByteBuffer.wrap(os.toByteArray());
|
||||
|
||||
var result = parser.parse(ProtocolVersion.V2016, body);
|
||||
|
||||
assertThat(result.blocks()).hasSize(2);
|
||||
assertThat(result.blocks().get(0)).isInstanceOf(InfoBlock.Gb32960V2016.Vehicle.class);
|
||||
assertThat(result.blocks().get(1)).isInstanceOfSatisfying(InfoBlock.Raw.class, raw -> {
|
||||
assertThat(raw.typeCode()).isEqualTo(0x07);
|
||||
// 剩余 10B alarm body + 1+20=21B 后续 Vehicle = 31B
|
||||
assertThat(raw.bytes()).hasSize(31);
|
||||
});
|
||||
}
|
||||
|
||||
@Test
|
||||
void strictMode_throwsOnAnyBlockFailure() {
|
||||
InfoBlockParserRegistry registry = new InfoBlockParserRegistry(List.of(
|
||||
new VehicleV2016BlockParser(),
|
||||
EXPLODING_FIXED_LEN_POSITION));
|
||||
|
||||
Gb32960BodyParser parser = new Gb32960BodyParser(registry);
|
||||
parser.setLenientBlockFailure(false);
|
||||
|
||||
ByteArrayOutputStream os = new ByteArrayOutputStream();
|
||||
writeValidVehicle(os);
|
||||
writePositionTypeAnd9ByteBody(os);
|
||||
ByteBuffer body = ByteBuffer.wrap(os.toByteArray());
|
||||
|
||||
assertThatThrownBy(() -> parser.parse(ProtocolVersion.V2016, body))
|
||||
.isInstanceOf(DecodeException.class);
|
||||
}
|
||||
|
||||
// ------------------------------------------------------------------------
|
||||
// helpers
|
||||
// ------------------------------------------------------------------------
|
||||
|
||||
/** 写一个完整的合法 V2016 Vehicle 块（typeCode + 20B body）。 */
|
||||
private static void writeValidVehicle(ByteArrayOutputStream os) {
|
||||
os.write(0x01); // typeCode Vehicle
|
||||
os.write(0x01); // vehicleState=1
|
||||
os.write(0x01); // chargingState=1
|
||||
os.write(0x01); // runningMode=1
|
||||
os.write(0); os.write(0); // speed=0
|
||||
os.write(0); os.write(0); os.write(0); os.write(0); // mileage=0
|
||||
os.write(0); os.write(0); // totalVoltage=0
|
||||
os.write(0); os.write(0); // totalCurrent=0（offset 1000，原值0）
|
||||
os.write(50); // soc=50%
|
||||
os.write(0x01); // dcdc
|
||||
os.write(0); // gear
|
||||
os.write(0); os.write(0); // insulation
|
||||
os.write(0); // accelerator
|
||||
os.write(0); // brake
|
||||
}
|
||||
|
||||
/** 写 Position typeCode(0x05) + 9 字节 body（内容不重要，parser 替换成 EXPLODING 的）。 */
|
||||
private static void writePositionTypeAnd9ByteBody(ByteArrayOutputStream os) {
|
||||
os.write(0x05);
|
||||
for (int i = 0; i < 9; i++) os.write(0);
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
- [ ] **Step 3.2: 跑测试验证 RED**
|
||||
|
||||
Run: `mvn -pl protocol-gb32960 test -Dtest=Gb32960BodyParserIsolationTest -q`
|
||||
Expected:
|
||||
- `fixedLengthBlockFailure_isIsolated_subsequentBlocksStillParsed` FAIL（当前会抛 DecodeException）
|
||||
- `truncatedFixedLengthBlock_isWrappedAsRaw_loopTerminates` FAIL（当前 L132 会抛 DecodeException）
|
||||
- `variableLengthBlockFailure_swallowsRemainderAsRaw_loopBreaks` FAIL（异常冒出整帧失败）
|
||||
- `strictMode_throwsOnAnyBlockFailure` FAIL（没有 `setLenientBlockFailure` 方法，编译就红）
|
||||
|
||||
**验证点**：编译失败（strictMode test 里 `setLenientBlockFailure` 尚未存在）是**预期** RED 信号——下一步实现。
|
||||
|
||||
---
|
||||
|
||||
## Task 4: GREEN —— 实现单块异常隔离
|
||||
|
||||
**Files:**
|
||||
- Modify: `protocol-gb32960/src/main/java/com/lingniu/ingest/protocol/gb32960/codec/Gb32960BodyParser.java`
|
||||
|
||||
- [ ] **Step 4.1: 在类字段区新增 `lenientBlockFailure` 字段 + setter**
|
||||
|
||||
在 `private final VendorExtensionSelector selector;`（现 L54）下面插入：
|
||||
|
||||
```java
|
||||
/**
|
||||
* 单块解析异常时是否兜底为 Raw 后继续。由
|
||||
* {@link com.lingniu.ingest.protocol.gb32960.config.Gb32960Properties.Parse#isLenientBlockFailure()}
|
||||
* 注入；默认 true。
|
||||
*/
|
||||
private boolean lenientBlockFailure = true;
|
||||
|
||||
public void setLenientBlockFailure(boolean lenientBlockFailure) {
|
||||
this.lenientBlockFailure = lenientBlockFailure;
|
||||
}
|
||||
```
|
||||
|
||||
- [ ] **Step 4.2: 改造主循环：替换 L130~L150 的整个 "parse + 长度校验" 段**
|
||||
|
||||
**完整替换块**：以下代码替换原来从 `int fixedLen = parser.fixedLength();`（L130）开始到 `blocks.add(block);`（L150）结束的整块。
|
||||
|
||||
```java
|
||||
int fixedLen = parser.fixedLength();
|
||||
int posBefore = body.position();
|
||||
|
||||
// 旧行为：fixedLen 预检不足 → 直接抛。新行为（lenient 模式）：走统一恢复路径。
|
||||
if (fixedLen >= 0 && body.remaining() < fixedLen) {
|
||||
if (!lenientBlockFailure) {
|
||||
throw new DecodeException(
|
||||
"info block 0x" + Integer.toHexString(typeCode)
|
||||
+ " needs " + fixedLen + " bytes but got " + body.remaining());
|
||||
}
|
||||
// 剩余字节数不足 fixedLen —— 无法按块截取，整尾兜 Raw + break
|
||||
int remaining = body.remaining();
|
||||
byte[] tail = new byte[remaining];
|
||||
body.get(tail);
|
||||
log.warn("[gb32960] truncated block typeCode=0x{} declaredFixedLen={} remaining={} — wrapping tail as Raw",
|
||||
Integer.toHexString(typeCode), fixedLen, remaining);
|
||||
blocks.add(new InfoBlock.Raw(typeCode, InfoBlockType.RAW, tail));
|
||||
break;
|
||||
}
|
||||
|
||||
InfoBlock block;
|
||||
try {
|
||||
block = parser.parse(body);
|
||||
} catch (DecodeException | java.nio.BufferUnderflowException | IndexOutOfBoundsException e) {
|
||||
if (!lenientBlockFailure) {
|
||||
if (e instanceof DecodeException de) throw de;
|
||||
throw new DecodeException(
|
||||
"parser " + parser.getClass().getSimpleName() + " failed: " + e.getMessage(), e);
|
||||
}
|
||||
// 恢复路径：先回滚 position 到 parse 入口
|
||||
body.position(posBefore);
|
||||
if (fixedLen >= 0) {
|
||||
// 固定长度块：按 fixedLen 截取 → 兜 Raw → continue 循环
|
||||
byte[] corrupt = new byte[fixedLen];
|
||||
body.get(corrupt);
|
||||
log.warn("[gb32960] block parse failed typeCode=0x{} parser={} fixedLen={} — isolated as Raw, continuing",
|
||||
Integer.toHexString(typeCode), parser.getClass().getSimpleName(), fixedLen, e);
|
||||
blocks.add(new InfoBlock.Raw(typeCode, InfoBlockType.RAW, corrupt));
|
||||
continue;
|
||||
} else {
|
||||
// 变长块：无法安全找下一块边界 → 剩余整段兜 Raw + break
|
||||
int remaining = body.remaining();
|
||||
byte[] tail = new byte[remaining];
|
||||
body.get(tail);
|
||||
log.warn("[gb32960] variable-length block parse failed typeCode=0x{} parser={} remaining={} — wrapping remainder as Raw, stopping loop",
|
||||
Integer.toHexString(typeCode), parser.getClass().getSimpleName(), remaining, e);
|
||||
blocks.add(new InfoBlock.Raw(typeCode, InfoBlockType.RAW, tail));
|
||||
break;
|
||||
}
|
||||
}
|
||||
|
||||
int consumed = body.position() - posBefore;
|
||||
if (fixedLen >= 0 && consumed != fixedLen) {
|
||||
// 这是 parser 契约违反（声明 fixedLen=X 但实际读了 Y），保持抛异常以暴露 parser bug。
|
||||
throw new DecodeException(
|
||||
"parser " + parser.getClass().getSimpleName()
|
||||
+ " consumed " + consumed
|
||||
+ " bytes but declared " + fixedLen);
|
||||
}
|
||||
if (log.isDebugEnabled()) {
|
||||
log.debug("[gb32960] block typeCode=0x{} parser={} pos {}→{} consumed={} declaredFixed={}",
|
||||
Integer.toHexString(typeCode), parser.getClass().getSimpleName(),
|
||||
typeStartPos, body.position(), consumed, fixedLen);
|
||||
}
|
||||
blocks.add(block);
|
||||
```
|
||||
|
||||
- [ ] **Step 4.3: 运行隔离测试验证 GREEN**
|
||||
|
||||
Run: `mvn -pl protocol-gb32960 test -Dtest=Gb32960BodyParserIsolationTest -q`
|
||||
Expected: 4 个测试全部 PASS。
|
||||
|
||||
- [ ] **Step 4.4: 运行全模块测试检查回归**
|
||||
|
||||
Run: `mvn -pl protocol-gb32960 test -q`
|
||||
Expected: BUILD SUCCESS，所有原有测试（含 Golden、FullBlocks、GuangdongFcEndToEnd）依旧通过。
|
||||
|
||||
如果有原有测试红了：**停下来诊断**。最可能的原因是某个现有测试之前靠"抛异常"行为验证错误帧的，需要手动判断这是测试问题还是实现问题。
|
||||
|
||||
- [ ] **Step 4.5: 提交**
|
||||
|
||||
```bash
|
||||
git add protocol-gb32960/src/main/java/com/lingniu/ingest/protocol/gb32960/codec/Gb32960BodyParser.java \
|
||||
protocol-gb32960/src/test/java/com/lingniu/ingest/protocol/gb32960/codec/Gb32960BodyParserIsolationTest.java
|
||||
git commit -m "$(cat <<'EOF'
|
||||
feat(gb32960): isolate per-block parse failures in body parser
|
||||
|
||||
主循环改造：单信息块 parser 抛 DecodeException / BufferUnderflowException /
|
||||
IndexOutOfBoundsException 时不再放弃整帧。
|
||||
|
||||
- 固定长度块（fixedLen≥0）：回滚 reader，按 fixedLen 截取字节兜成 InfoBlock.Raw，
|
||||
position 前进 fixedLen，continue 循环继续解析后续块
|
||||
- 变长块（fixedLen=-1）或剩余不足 fixedLen：剩余字节全部兜成 Raw 后 break
|
||||
- parser 契约违反（consumed != declared fixedLen）仍抛异常，以暴露 parser bug
|
||||
- 通过 lingniu.ingest.gb32960.parse.lenientBlockFailure=false 可回退严格模式
|
||||
- 新增 Gb32960BodyParserIsolationTest 覆盖 3 种失败场景 + 1 严格模式
|
||||
|
||||
不改 InfoBlock.Raw record 结构；下游 Gb32960EventMapper 按 findBlock(Class) 类型
|
||||
查找，新增 Raw 不影响其行为。
|
||||
|
||||
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||||
EOF
|
||||
)"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Task 5: 把配置注入 BodyParser（AutoConfiguration 连线）
|
||||
|
||||
**Files:**
|
||||
- Modify: `protocol-gb32960/src/main/java/com/lingniu/ingest/protocol/gb32960/config/Gb32960AutoConfiguration.java`
|
||||
|
||||
- [ ] **Step 5.1: 定位 `Gb32960BodyParser` bean 定义**
|
||||
|
||||
先查看：`grep -n "Gb32960BodyParser" protocol-gb32960/src/main/java/com/lingniu/ingest/protocol/gb32960/config/Gb32960AutoConfiguration.java`
|
||||
|
||||
根据返回的行号，在构造 `Gb32960BodyParser` 的 @Bean 方法里，构造完成后立即调用：
|
||||
|
||||
```java
|
||||
Gb32960BodyParser parser = new Gb32960BodyParser(profileRegistry, selector);
|
||||
parser.setLenientBlockFailure(properties.getParse().isLenientBlockFailure());
|
||||
return parser;
|
||||
```
|
||||
|
||||
（如果目前的构造返回一行表达式，改成先赋给 local 变量再设 flag 再 return。）
|
||||
|
||||
- [ ] **Step 5.2: 编译并运行全模块测试**
|
||||
|
||||
Run: `mvn -pl protocol-gb32960 test -q`
|
||||
Expected: BUILD SUCCESS。
|
||||
|
||||
- [ ] **Step 5.3: 提交**
|
||||
|
||||
```bash
|
||||
git add protocol-gb32960/src/main/java/com/lingniu/ingest/protocol/gb32960/config/Gb32960AutoConfiguration.java
|
||||
git commit -m "$(cat <<'EOF'
|
||||
config(gb32960): wire parse.lenientBlockFailure into BodyParser bean
|
||||
|
||||
Gb32960AutoConfiguration 在创建 Gb32960BodyParser bean 后，将
|
||||
Gb32960Properties.Parse.lenientBlockFailure 通过 setter 注入，让运行期配置生效。
|
||||
|
||||
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||||
EOF
|
||||
)"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Task 6: bootstrap-all 配置文档化
|
||||
|
||||
**Files:**
|
||||
- Modify: `bootstrap-all/src/main/resources/application.yml`
|
||||
|
||||
- [ ] **Step 6.1: 在 `lingniu.ingest.gb32960` 节点的 `vendor-extensions` 段之后、`jt808` 之前，插入 parse 配置示例**
|
||||
|
||||
参考位置：现有 application.yml L57 末尾（`vendor-extensions` list 结束）。在 L58 `jt808:` 之前加入：
|
||||
|
||||
```yaml
|
||||
# 报文解析容错。默认启用单块异常隔离：某个信息块解析失败时兜成 Raw 后继续，
|
||||
# 不再整帧丢弃。仅在需要严格失败语义（灰度回滚或抓虫）时设为 false。
|
||||
parse:
|
||||
lenient-block-failure: true
|
||||
```
|
||||
|
||||
- [ ] **Step 6.2: 启动一次 bootstrap-all 的 dry-run 编译**
|
||||
|
||||
Run: `mvn -pl bootstrap-all compile -q`
|
||||
Expected: BUILD SUCCESS（只是配置段注释变化，不会影响编译）。
|
||||
|
||||
- [ ] **Step 6.3: 提交**
|
||||
|
||||
```bash
|
||||
git add bootstrap-all/src/main/resources/application.yml
|
||||
git commit -m "$(cat <<'EOF'
|
||||
config(gb32960): document parse.lenient-block-failure in application.yml
|
||||
|
||||
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||||
EOF
|
||||
)"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Task 7: 全仓库回归测试
|
||||
|
||||
**Files:** 无（只是验证）
|
||||
|
||||
- [ ] **Step 7.1: 跑完整项目测试**
|
||||
|
||||
Run: `mvn test -q`
|
||||
Expected: BUILD SUCCESS。重点关注：
|
||||
- `protocol-gb32960` 模块全绿（Golden、FullBlocks、Isolation、GuangdongFcEndToEnd、Decoder、Mapper、所有 parser 单测、profile）
|
||||
- 其它模块（`ingest-core`、`sink-kafka` 等）不受影响（无改动应自动通过）
|
||||
|
||||
若有红，停下来诊断。
|
||||
|
||||
---
|
||||
|
||||
## Task 8: 更新 CHANGELOG
|
||||
|
||||
**Files:**
|
||||
- Modify: `CHANGELOG.md`
|
||||
|
||||
- [ ] **Step 8.1: 在 CHANGELOG.md 文件顶部（在 `## [0.1.0] — 2026-04-15` 标题之前）新增一个 `## [Unreleased]` 段落；如已存在则追加**
|
||||
|
||||
新增段落内容（如已存在 Unreleased 段则在其 `### Added` / `### Changed` 内部追加）：
|
||||
|
||||
```markdown
|
||||
## [Unreleased]
|
||||
|
||||
### Changed —— GB/T 32960 Body Parser 单块异常隔离
|
||||
- `Gb32960BodyParser` 主循环对单信息块的 parser 异常不再放弃整帧：
|
||||
- 固定长度块（`fixedLength ≥ 0`）：回滚 reader → 按 fixedLen 截取字节兜成
|
||||
`InfoBlock.Raw` → 继续解析后续块；
|
||||
- 变长块或剩余字节不足：剩余字节全部兜成 `Raw` 后终止循环；
|
||||
- `parser` 契约违反（声明 fixedLen=X 但实际读了 Y）仍抛 `DecodeException`，
|
||||
以暴露 parser bug，不被静默吞掉；
|
||||
- 只捕获 `DecodeException` / `BufferUnderflowException` / `IndexOutOfBoundsException`，
|
||||
`RuntimeException` 继续向上抛，保留 bug 可见性。
|
||||
- 新增配置 `lingniu.ingest.gb32960.parse.lenient-block-failure`（默认 `true`）；
|
||||
回退严格模式用 `false`。
|
||||
- `InfoBlock.Raw` record 结构**未变**，下游 `Gb32960EventMapper` 按 `findBlock(Class)`
|
||||
类型匹配，新增 Raw 不影响业务事件映射。
|
||||
- 覆盖测试：`Gb32960BodyParserIsolationTest` —— 3 种隔离场景 + 严格模式回退。
|
||||
```
|
||||
|
||||
- [ ] **Step 8.2: 提交**
|
||||
|
||||
```bash
|
||||
git add CHANGELOG.md
|
||||
git commit -m "$(cat <<'EOF'
|
||||
docs(changelog): record gb32960 body parser block isolation
|
||||
|
||||
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||||
EOF
|
||||
)"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Self-Review Checklist
|
||||
|
||||
- [x] **Spec coverage**: 原讨论的 A1~A7 全部覆盖——A1 状态机（Task 4 Step 4.2）、A2 三类错误策略（同）、A3 收窄异常种类（同，只 catch 三种）、A4 日志分类（warn 带 parser 名/typeCode/原因）、A5 配置开关（Task 2）、A6 合成帧测试（Task 3 三场景 + 严格回退）、A7 下游兼容（不改 Raw record，`findBlock` 类型匹配不受影响，plan 中已说明）。
|
||||
- [x] **Placeholder scan**: 无 TBD / TODO / "实现上面的" 等占位；每个 Step 含实际代码 or 命令。
|
||||
- [x] **Type consistency**: `lenientBlockFailure` 在 Properties / BodyParser 字段 / setter / 测试中拼写一致。`InfoBlockType.RAW` 枚举值依赖当前存在（已核对 InfoBlock.Raw record）。
|
||||
- [x] **测试命名一致**：`Gb32960BodyParserIsolationTest` 在 Task 3 创建、Task 4/7 引用一致。
|
||||
- [x] **未引入新枚举/proto 变更**：InfoBlock.Raw 沿用现有 `(int typeCode, InfoBlockType type, byte[] bytes)` 构造。
|
||||
@@ -1,817 +0,0 @@
|
||||
> **Superseded:** This 2026-06-23 DuckDB hot-store plan is historical context.
|
||||
> Use `docs/target-architecture.md` and
|
||||
> `docs/superpowers/specs/2026-06-29-vehicle-ingest-redesign.md` for the
|
||||
> current production architecture: TDengine stores hot history, `sink-archive`
|
||||
> owns raw bytes, and `event-file-store` / DuckDB are no longer part of the
|
||||
> current build surface.
|
||||
|
||||
# GB32960 History DuckDB Hot Store Implementation Plan
|
||||
|
||||
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
|
||||
|
||||
**Goal:** Replace the current high-write-risk Parquet rewrite history path with a DuckDB hot history store that supports fast `vin + time` queries and full RAW frame replay.
|
||||
|
||||
**Architecture:** Add a new `DuckDbHotEventFileStore` implementation behind the existing `EventFileStore` interface, backed by append/upsert DuckDB tables instead of rewriting per-vehicle Parquet files. Keep the current HTTP history API and RAW archive reader intact, then switch `vehicle-history-app` to the hot DuckDB store by configuration.
|
||||
|
||||
**Tech Stack:** Java 26, Spring Boot, DuckDB JDBC, JUnit 5, AssertJ, existing Kafka `VehicleEnvelope`, existing `ArchiveStore`.
|
||||
|
||||
---
|
||||
|
||||
## Scope
|
||||
|
||||
This plan implements Phase 1 from `docs/superpowers/specs/2026-06-23-gb32960-production-readiness-design.md`.
|
||||
|
||||
Included:
|
||||
|
||||
- DuckDB hot event table for `EventFileRecord`.
|
||||
- Idempotent writes by `event_id`.
|
||||
- Query by protocol/date/VIN/eventType/eventTime/order/limit.
|
||||
- Lookup by `rawArchiveUri`.
|
||||
- Spring configuration to select the hot store.
|
||||
- Tests that prove history endpoints still replay RAW frames.
|
||||
- Local verification using the existing `vehicle-history-app`.
|
||||
|
||||
Not included in this plan:
|
||||
|
||||
- MySQL daily statistics.
|
||||
- VIN-to-platform local mapping.
|
||||
- Full telemetry point columnar table.
|
||||
- Alarm timeline MySQL tables.
|
||||
- Cold Parquet export.
|
||||
|
||||
Those will be separate plans after this foundation lands.
|
||||
|
||||
## Files
|
||||
|
||||
- Create: `modules/sinks/event-file-store/src/main/java/com/lingniu/ingest/eventfilestore/DuckDbHotEventFileStore.java`
|
||||
- Create: `modules/sinks/event-file-store/src/test/java/com/lingniu/ingest/eventfilestore/DuckDbHotEventFileStoreTest.java`
|
||||
- Modify: `modules/sinks/event-file-store/src/main/java/com/lingniu/ingest/eventfilestore/config/EventFileStoreProperties.java`
|
||||
- Modify: `modules/sinks/event-file-store/src/main/java/com/lingniu/ingest/eventfilestore/config/EventFileStoreAutoConfiguration.java`
|
||||
- Modify: `modules/sinks/event-file-store/src/test/java/com/lingniu/ingest/eventfilestore/config/EventFileStoreAutoConfigurationTest.java`
|
||||
- Modify: `modules/apps/vehicle-history-app/src/main/resources/application.yml`
|
||||
- Modify: `modules/apps/vehicle-history-app/src/test/java/com/lingniu/ingest/historyapp/VehicleHistoryAppDefaultsTest.java`
|
||||
- Modify: `modules/services/event-history-service/src/test/java/com/lingniu/ingest/eventhistory/Gb32960DecodedFrameServiceTest.java`
|
||||
|
||||
## Task 1: Add Hot Store Configuration
|
||||
|
||||
**Files:**
|
||||
- Modify: `modules/sinks/event-file-store/src/main/java/com/lingniu/ingest/eventfilestore/config/EventFileStoreProperties.java`
|
||||
- Modify: `modules/sinks/event-file-store/src/main/java/com/lingniu/ingest/eventfilestore/config/EventFileStoreAutoConfiguration.java`
|
||||
- Test: `modules/sinks/event-file-store/src/test/java/com/lingniu/ingest/eventfilestore/config/EventFileStoreAutoConfigurationTest.java`
|
||||
|
||||
- [ ] **Step 1: Write the failing auto-configuration test**
|
||||
|
||||
Add a test that sets `lingniu.ingest.event-file-store.storage=duckdb-hot` and expects the bean class to be `DuckDbHotEventFileStore`.
|
||||
|
||||
```java
|
||||
@Test
|
||||
void createsDuckDbHotStoreWhenStorageIsDuckDbHot() {
|
||||
contextRunner
|
||||
.withPropertyValues(
|
||||
"lingniu.ingest.event-file-store.enabled=true",
|
||||
"lingniu.ingest.event-file-store.storage=duckdb-hot",
|
||||
"lingniu.ingest.event-file-store.path=" + tempDir.resolve("history"))
|
||||
.run(context -> assertThat(context)
|
||||
.hasSingleBean(EventFileStore.class)
|
||||
.getBean(EventFileStore.class)
|
||||
.isInstanceOf(DuckDbHotEventFileStore.class));
|
||||
}
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Run the new test and verify it fails**
|
||||
|
||||
Run:
|
||||
|
||||
```bash
|
||||
mvn -pl :event-file-store -Dtest=EventFileStoreAutoConfigurationTest#createsDuckDbHotStoreWhenStorageIsDuckDbHot test
|
||||
```
|
||||
|
||||
Expected: compilation failure because `DuckDbHotEventFileStore` and `storage` property do not exist.
|
||||
|
||||
- [ ] **Step 3: Add the `storage` property**
|
||||
|
||||
Add to `EventFileStoreProperties`:
|
||||
|
||||
```java
|
||||
/**
|
||||
* Storage backend. `duckdb-hot` is the production backend; `parquet-sidecar`
|
||||
* keeps the previous implementation available for compatibility tests.
|
||||
*/
|
||||
private String storage = "duckdb-hot";
|
||||
|
||||
public String getStorage() {
|
||||
return storage;
|
||||
}
|
||||
|
||||
public void setStorage(String storage) {
|
||||
this.storage = storage;
|
||||
}
|
||||
```
|
||||
|
||||
- [ ] **Step 4: Switch auto-configuration by storage mode**
|
||||
|
||||
Change `eventFileStore(...)` to:
|
||||
|
||||
```java
|
||||
@Bean
|
||||
@ConditionalOnMissingBean
|
||||
public EventFileStore eventFileStore(EventFileStoreProperties properties,
|
||||
ObjectProvider<ObjectMapper> objectMapper) {
|
||||
ObjectMapper mapper = mapper(objectMapper);
|
||||
Path root = Path.of(properties.getPath());
|
||||
ZoneId zoneId = ZoneId.of(properties.getZoneId());
|
||||
String storage = properties.getStorage() == null ? "" : properties.getStorage().trim();
|
||||
return switch (storage) {
|
||||
case "", "duckdb-hot" -> new DuckDbHotEventFileStore(root, zoneId, mapper);
|
||||
case "parquet-sidecar" -> new DuckDbParquetEventFileStore(root, zoneId, mapper);
|
||||
default -> throw new IllegalStateException(
|
||||
"unsupported event-file-store storage: " + properties.getStorage());
|
||||
};
|
||||
}
|
||||
```
|
||||
|
||||
Import `DuckDbHotEventFileStore`.
|
||||
|
||||
- [ ] **Step 5: Run configuration tests**
|
||||
|
||||
Run:
|
||||
|
||||
```bash
|
||||
mvn -pl :event-file-store -Dtest=EventFileStoreAutoConfigurationTest test
|
||||
```
|
||||
|
||||
Expected: tests pass after the store class exists in Task 2.
|
||||
|
||||
## Task 2: Implement DuckDB Hot Store Schema and Writes
|
||||
|
||||
**Files:**
|
||||
- Create: `modules/sinks/event-file-store/src/main/java/com/lingniu/ingest/eventfilestore/DuckDbHotEventFileStore.java`
|
||||
- Create: `modules/sinks/event-file-store/src/test/java/com/lingniu/ingest/eventfilestore/DuckDbHotEventFileStoreTest.java`
|
||||
|
||||
- [ ] **Step 1: Write failing append/query/idempotency tests**
|
||||
|
||||
Create `DuckDbHotEventFileStoreTest`:
|
||||
|
||||
```java
|
||||
class DuckDbHotEventFileStoreTest {
|
||||
|
||||
@TempDir
|
||||
Path tempDir;
|
||||
|
||||
@Test
|
||||
void appendsRecordsAndQueriesByVinTypeAndTime() throws Exception {
|
||||
EventFileStore store = store();
|
||||
store.append(rawRecord("raw-1", "VIN001", "2026-06-23T01:00:00Z"));
|
||||
store.append(rawRecord("raw-2", "VIN002", "2026-06-23T01:00:01Z"));
|
||||
store.append(rawRecord("raw-3", "VIN001", "2026-06-23T01:00:02Z"));
|
||||
|
||||
EventFileQuery query = new EventFileQuery(
|
||||
ProtocolId.GB32960,
|
||||
LocalDate.parse("2026-06-23"),
|
||||
LocalDate.parse("2026-06-23"),
|
||||
Instant.parse("2026-06-23T01:00:00Z"),
|
||||
Instant.parse("2026-06-23T01:00:03Z"),
|
||||
EventFileQuery.Order.DESC,
|
||||
2,
|
||||
"VIN001",
|
||||
"RAW_ARCHIVE");
|
||||
|
||||
assertThat(store.query(query))
|
||||
.extracting(EventFileRecord::eventId)
|
||||
.containsExactly("raw-3", "raw-1");
|
||||
}
|
||||
|
||||
@Test
|
||||
void appendAllIsIdempotentByEventId() throws Exception {
|
||||
EventFileStore store = store();
|
||||
EventFileRecord original = rawRecord("same-id", "VIN001", "2026-06-23T01:00:00Z");
|
||||
EventFileRecord replacement = new EventFileRecord(
|
||||
"same-id",
|
||||
ProtocolId.GB32960,
|
||||
"RAW_ARCHIVE",
|
||||
"VIN001",
|
||||
Instant.parse("2026-06-23T01:00:05Z"),
|
||||
Instant.parse("2026-06-23T01:00:06Z"),
|
||||
"archive://replacement.bin",
|
||||
Map.of("source", "replacement"),
|
||||
"{\"replacement\":true}");
|
||||
|
||||
store.appendAll(List.of(original, replacement));
|
||||
|
||||
assertThat(store.query(new EventFileQuery(
|
||||
ProtocolId.GB32960,
|
||||
LocalDate.parse("2026-06-23"),
|
||||
LocalDate.parse("2026-06-23"),
|
||||
EventFileQuery.Order.ASC,
|
||||
10,
|
||||
"VIN001",
|
||||
"RAW_ARCHIVE")))
|
||||
.singleElement()
|
||||
.satisfies(record -> {
|
||||
assertThat(record.eventId()).isEqualTo("same-id");
|
||||
assertThat(record.rawArchiveUri()).isEqualTo("archive://replacement.bin");
|
||||
});
|
||||
}
|
||||
|
||||
@Test
|
||||
void findsRecordByRawArchiveUri() throws Exception {
|
||||
EventFileStore store = store();
|
||||
EventFileRecord record = rawRecord("raw-uri", "VIN001", "2026-06-23T01:00:00Z");
|
||||
store.append(record);
|
||||
|
||||
EventFileRecord found = store.findByRawArchiveUri(record.rawArchiveUri());
|
||||
|
||||
assertThat(found).isNotNull();
|
||||
assertThat(found.eventId()).isEqualTo("raw-uri");
|
||||
}
|
||||
|
||||
private EventFileStore store() {
|
||||
return new DuckDbHotEventFileStore(tempDir, ZoneId.of("Asia/Shanghai"), new ObjectMapper());
|
||||
}
|
||||
|
||||
private static EventFileRecord rawRecord(String id, String vin, String eventTime) {
|
||||
return new EventFileRecord(
|
||||
id,
|
||||
ProtocolId.GB32960,
|
||||
"RAW_ARCHIVE",
|
||||
vin,
|
||||
Instant.parse(eventTime),
|
||||
Instant.parse(eventTime).plusMillis(100),
|
||||
"archive://" + id + ".bin",
|
||||
Map.of("platformAccount", "Hyundai", "command", "REALTIME_REPORT"),
|
||||
"{\"eventId\":\"" + id + "\"}");
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Run the test and verify it fails**
|
||||
|
||||
Run:
|
||||
|
||||
```bash
|
||||
mvn -pl :event-file-store -Dtest=DuckDbHotEventFileStoreTest test
|
||||
```
|
||||
|
||||
Expected: compilation failure because `DuckDbHotEventFileStore` does not exist.
|
||||
|
||||
- [ ] **Step 3: Create the hot store class**
|
||||
|
||||
Create `DuckDbHotEventFileStore` with this structure:
|
||||
|
||||
```java
|
||||
public final class DuckDbHotEventFileStore implements EventFileStore {
|
||||
|
||||
private static final TypeReference<Map<String, String>> STRING_MAP = new TypeReference<>() {};
|
||||
|
||||
private final Path root;
|
||||
private final Path dbPath;
|
||||
private final ZoneId partitionZone;
|
||||
private final ObjectMapper objectMapper;
|
||||
private volatile boolean initialized;
|
||||
|
||||
public DuckDbHotEventFileStore(Path root, ZoneId partitionZone) {
|
||||
this(root, partitionZone, new ObjectMapper());
|
||||
}
|
||||
|
||||
public DuckDbHotEventFileStore(Path root, ZoneId partitionZone, ObjectMapper objectMapper) {
|
||||
if (root == null) {
|
||||
throw new IllegalArgumentException("root must not be null");
|
||||
}
|
||||
this.root = root.toAbsolutePath();
|
||||
this.dbPath = this.root.resolve("events.duckdb");
|
||||
this.partitionZone = partitionZone == null ? ZoneId.of("Asia/Shanghai") : partitionZone;
|
||||
this.objectMapper = objectMapper == null ? new ObjectMapper() : objectMapper;
|
||||
}
|
||||
|
||||
@Override
|
||||
public synchronized void appendAll(List<EventFileRecord> records) throws IOException {
|
||||
if (records == null || records.isEmpty()) {
|
||||
return;
|
||||
}
|
||||
ensureInitialized();
|
||||
try (Connection connection = DriverManager.getConnection(jdbcUrl())) {
|
||||
connection.setAutoCommit(false);
|
||||
try {
|
||||
upsertRecords(connection, records);
|
||||
connection.commit();
|
||||
} catch (SQLException | IOException e) {
|
||||
connection.rollback();
|
||||
throw e;
|
||||
} finally {
|
||||
connection.setAutoCommit(true);
|
||||
}
|
||||
} catch (SQLException e) {
|
||||
throw new IOException("write duckdb hot event store failed", e);
|
||||
}
|
||||
}
|
||||
|
||||
@Override
|
||||
public List<EventFileRecord> query(EventFileQuery query) throws IOException {
|
||||
ensureInitialized();
|
||||
// Implement in Task 3.
|
||||
return List.of();
|
||||
}
|
||||
|
||||
@Override
|
||||
public EventFileRecord findByRawArchiveUri(String rawArchiveUri) throws IOException {
|
||||
ensureInitialized();
|
||||
// Implement in Task 3.
|
||||
return null;
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
- [ ] **Step 4: Add schema initialization**
|
||||
|
||||
Add:
|
||||
|
||||
```java
|
||||
private void ensureInitialized() throws IOException {
|
||||
if (initialized) {
|
||||
return;
|
||||
}
|
||||
synchronized (this) {
|
||||
if (initialized) {
|
||||
return;
|
||||
}
|
||||
Files.createDirectories(root);
|
||||
try (Connection connection = DriverManager.getConnection(jdbcUrl());
|
||||
Statement statement = connection.createStatement()) {
|
||||
statement.execute("""
|
||||
CREATE TABLE IF NOT EXISTS event_records (
|
||||
event_id VARCHAR PRIMARY KEY,
|
||||
protocol VARCHAR NOT NULL,
|
||||
event_type VARCHAR NOT NULL,
|
||||
vin VARCHAR NOT NULL,
|
||||
event_time_ms BIGINT NOT NULL,
|
||||
ingest_time_ms BIGINT NOT NULL,
|
||||
partition_date DATE NOT NULL,
|
||||
raw_archive_uri VARCHAR NOT NULL,
|
||||
metadata_json VARCHAR NOT NULL,
|
||||
payload_json VARCHAR NOT NULL
|
||||
)
|
||||
""");
|
||||
statement.execute("""
|
||||
CREATE INDEX IF NOT EXISTS event_records_protocol_date_time_idx
|
||||
ON event_records(protocol, partition_date, event_time_ms)
|
||||
""");
|
||||
statement.execute("""
|
||||
CREATE INDEX IF NOT EXISTS event_records_vin_date_time_idx
|
||||
ON event_records(protocol, vin, partition_date, event_time_ms)
|
||||
""");
|
||||
statement.execute("""
|
||||
CREATE INDEX IF NOT EXISTS event_records_vin_type_date_time_idx
|
||||
ON event_records(protocol, vin, event_type, partition_date, event_time_ms)
|
||||
""");
|
||||
statement.execute("""
|
||||
CREATE INDEX IF NOT EXISTS event_records_raw_archive_uri_idx
|
||||
ON event_records(raw_archive_uri)
|
||||
""");
|
||||
initialized = true;
|
||||
} catch (SQLException e) {
|
||||
throw new IOException("initialize duckdb hot event store failed", e);
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
- [ ] **Step 5: Add idempotent batch upsert**
|
||||
|
||||
Use DuckDB `INSERT OR REPLACE` inside one transaction:
|
||||
|
||||
```java
|
||||
private void upsertRecords(Connection connection, List<EventFileRecord> records)
|
||||
throws SQLException, IOException {
|
||||
try (PreparedStatement ps = connection.prepareStatement("""
|
||||
INSERT OR REPLACE INTO event_records VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
|
||||
""")) {
|
||||
for (EventFileRecord record : records) {
|
||||
ps.setString(1, record.eventId());
|
||||
ps.setString(2, record.protocol().name());
|
||||
ps.setString(3, record.eventType());
|
||||
ps.setString(4, record.vin());
|
||||
ps.setLong(5, record.eventTime().toEpochMilli());
|
||||
ps.setLong(6, record.ingestTime().toEpochMilli());
|
||||
ps.setString(7, LocalDate.ofInstant(record.eventTime(), partitionZone).toString());
|
||||
ps.setString(8, record.rawArchiveUri());
|
||||
ps.setString(9, objectMapper.writeValueAsString(record.metadata()));
|
||||
ps.setString(10, record.payloadJson());
|
||||
ps.addBatch();
|
||||
}
|
||||
ps.executeBatch();
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
- [ ] **Step 6: Run the hot store tests**
|
||||
|
||||
Run:
|
||||
|
||||
```bash
|
||||
mvn -pl :event-file-store -Dtest=DuckDbHotEventFileStoreTest test
|
||||
```
|
||||
|
||||
Expected: query tests still fail until Task 3 implements reads; append initialization should compile.
|
||||
|
||||
## Task 3: Implement Hot Store Queries and Raw URI Lookup
|
||||
|
||||
**Files:**
|
||||
- Modify: `modules/sinks/event-file-store/src/main/java/com/lingniu/ingest/eventfilestore/DuckDbHotEventFileStore.java`
|
||||
- Test: `modules/sinks/event-file-store/src/test/java/com/lingniu/ingest/eventfilestore/DuckDbHotEventFileStoreTest.java`
|
||||
|
||||
- [ ] **Step 1: Implement `query(EventFileQuery)`**
|
||||
|
||||
Use prepared statements for every external value:
|
||||
|
||||
```java
|
||||
@Override
|
||||
public List<EventFileRecord> query(EventFileQuery query) throws IOException {
|
||||
ensureInitialized();
|
||||
String order = query.order() == EventFileQuery.Order.DESC ? "DESC" : "ASC";
|
||||
StringBuilder where = new StringBuilder("""
|
||||
WHERE protocol = ?
|
||||
AND partition_date BETWEEN CAST(? AS DATE) AND CAST(? AS DATE)
|
||||
""");
|
||||
if (query.vin() != null) {
|
||||
where.append(" AND vin = ?\n");
|
||||
}
|
||||
if (query.eventType() != null) {
|
||||
where.append(" AND event_type = ?\n");
|
||||
}
|
||||
if (query.eventTimeFrom() != null) {
|
||||
where.append(" AND event_time_ms >= ?\n");
|
||||
}
|
||||
if (query.eventTimeTo() != null) {
|
||||
where.append(" AND event_time_ms <= ?\n");
|
||||
}
|
||||
String sql = """
|
||||
SELECT event_id, protocol, event_type, vin, event_time_ms, ingest_time_ms,
|
||||
raw_archive_uri, metadata_json, payload_json
|
||||
FROM event_records
|
||||
%s
|
||||
ORDER BY event_time_ms %s, ingest_time_ms %s, event_id %s
|
||||
LIMIT ?
|
||||
""".formatted(where, order, order, order);
|
||||
try (Connection connection = DriverManager.getConnection(jdbcUrl());
|
||||
PreparedStatement ps = connection.prepareStatement(sql)) {
|
||||
bindQuery(ps, query);
|
||||
try (ResultSet rs = ps.executeQuery()) {
|
||||
List<EventFileRecord> out = new ArrayList<>();
|
||||
while (rs.next()) {
|
||||
out.add(record(rs));
|
||||
}
|
||||
return out;
|
||||
}
|
||||
} catch (SQLException e) {
|
||||
throw new IOException("query duckdb hot event store failed", e);
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Add query binding helper**
|
||||
|
||||
```java
|
||||
private static void bindQuery(PreparedStatement ps, EventFileQuery query) throws SQLException {
|
||||
int index = 1;
|
||||
ps.setString(index++, query.protocol().name());
|
||||
ps.setString(index++, query.dateFrom().toString());
|
||||
ps.setString(index++, query.dateTo().toString());
|
||||
if (query.vin() != null) {
|
||||
ps.setString(index++, query.vin());
|
||||
}
|
||||
if (query.eventType() != null) {
|
||||
ps.setString(index++, query.eventType());
|
||||
}
|
||||
if (query.eventTimeFrom() != null) {
|
||||
ps.setLong(index++, query.eventTimeFrom().toEpochMilli());
|
||||
}
|
||||
if (query.eventTimeTo() != null) {
|
||||
ps.setLong(index++, query.eventTimeTo().toEpochMilli());
|
||||
}
|
||||
ps.setInt(index, query.limit());
|
||||
}
|
||||
```
|
||||
|
||||
- [ ] **Step 3: Implement `findByRawArchiveUri`**
|
||||
|
||||
```java
|
||||
@Override
|
||||
public EventFileRecord findByRawArchiveUri(String rawArchiveUri) throws IOException {
|
||||
if (rawArchiveUri == null || rawArchiveUri.isBlank()) {
|
||||
return null;
|
||||
}
|
||||
ensureInitialized();
|
||||
try (Connection connection = DriverManager.getConnection(jdbcUrl());
|
||||
PreparedStatement ps = connection.prepareStatement("""
|
||||
SELECT event_id, protocol, event_type, vin, event_time_ms, ingest_time_ms,
|
||||
raw_archive_uri, metadata_json, payload_json
|
||||
FROM event_records
|
||||
WHERE raw_archive_uri = ?
|
||||
ORDER BY event_type = 'RAW_ARCHIVE' DESC, ingest_time_ms DESC, event_id DESC
|
||||
LIMIT 1
|
||||
""")) {
|
||||
ps.setString(1, rawArchiveUri);
|
||||
try (ResultSet rs = ps.executeQuery()) {
|
||||
return rs.next() ? record(rs) : null;
|
||||
}
|
||||
} catch (SQLException e) {
|
||||
throw new IOException("query duckdb hot store by raw archive uri failed", e);
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
- [ ] **Step 4: Add record mapper and helpers**
|
||||
|
||||
```java
|
||||
private EventFileRecord record(ResultSet rs) throws SQLException, IOException {
|
||||
return new EventFileRecord(
|
||||
rs.getString("event_id"),
|
||||
protocol(rs.getString("protocol")),
|
||||
rs.getString("event_type"),
|
||||
rs.getString("vin"),
|
||||
Instant.ofEpochMilli(rs.getLong("event_time_ms")),
|
||||
Instant.ofEpochMilli(rs.getLong("ingest_time_ms")),
|
||||
rs.getString("raw_archive_uri"),
|
||||
readMetadata(rs.getString("metadata_json")),
|
||||
rs.getString("payload_json"));
|
||||
}
|
||||
|
||||
private Map<String, String> readMetadata(String json) throws IOException {
|
||||
if (json == null || json.isBlank()) {
|
||||
return Map.of();
|
||||
}
|
||||
return objectMapper.readValue(json, STRING_MAP);
|
||||
}
|
||||
|
||||
private static ProtocolId protocol(String value) {
|
||||
if (value == null || value.isBlank()) {
|
||||
return ProtocolId.UNKNOWN;
|
||||
}
|
||||
try {
|
||||
return ProtocolId.valueOf(value);
|
||||
} catch (IllegalArgumentException ex) {
|
||||
return ProtocolId.UNKNOWN;
|
||||
}
|
||||
}
|
||||
|
||||
private String jdbcUrl() {
|
||||
return "jdbc:duckdb:" + dbPath;
|
||||
}
|
||||
```
|
||||
|
||||
- [ ] **Step 5: Run hot store tests**
|
||||
|
||||
Run:
|
||||
|
||||
```bash
|
||||
mvn -pl :event-file-store -Dtest=DuckDbHotEventFileStoreTest test
|
||||
```
|
||||
|
||||
Expected: all `DuckDbHotEventFileStoreTest` tests pass.
|
||||
|
||||
## Task 4: Preserve Legacy Store Tests and Update Defaults
|
||||
|
||||
**Files:**
|
||||
- Modify: `modules/sinks/event-file-store/src/test/java/com/lingniu/ingest/eventfilestore/DuckDbParquetEventFileStoreTest.java`
|
||||
- Modify: `modules/apps/vehicle-history-app/src/main/resources/application.yml`
|
||||
- Modify: `modules/apps/vehicle-history-app/src/test/java/com/lingniu/ingest/historyapp/VehicleHistoryAppDefaultsTest.java`
|
||||
|
||||
- [ ] **Step 1: Keep Parquet tests explicitly legacy**
|
||||
|
||||
No behavior change is needed in `DuckDbParquetEventFileStoreTest`; leave it instantiating `DuckDbParquetEventFileStore` directly. Add a class comment:
|
||||
|
||||
```java
|
||||
/**
|
||||
* Compatibility coverage for the legacy Parquet sidecar backend.
|
||||
* Production history uses DuckDbHotEventFileStore through auto-configuration.
|
||||
*/
|
||||
class DuckDbParquetEventFileStoreTest {
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Set vehicle-history-app storage default**
|
||||
|
||||
Add to `modules/apps/vehicle-history-app/src/main/resources/application.yml`:
|
||||
|
||||
```yaml
|
||||
event-file-store:
|
||||
enabled: ${EVENT_FILE_STORE_ENABLED:true}
|
||||
storage: ${EVENT_FILE_STORE_STORAGE:duckdb-hot}
|
||||
path: ${EVENT_FILE_STORE_PATH:./target/event-store/}
|
||||
zone-id: ${EVENT_FILE_STORE_ZONE_ID:Asia/Shanghai}
|
||||
batch-size: ${EVENT_FILE_STORE_BATCH_SIZE:1000}
|
||||
flush-interval-millis: ${EVENT_FILE_STORE_FLUSH_INTERVAL_MILLIS:1000}
|
||||
```
|
||||
|
||||
Keep existing indentation and only add `storage`; if `batch-size` already exists, update it to `1000`.
|
||||
|
||||
- [ ] **Step 3: Update app default test**
|
||||
|
||||
In `VehicleHistoryAppDefaultsTest`, assert:
|
||||
|
||||
```java
|
||||
assertThat(context.getEnvironment()
|
||||
.getProperty("lingniu.ingest.event-file-store.storage"))
|
||||
.isEqualTo("duckdb-hot");
|
||||
```
|
||||
|
||||
- [ ] **Step 4: Run app default tests**
|
||||
|
||||
Run:
|
||||
|
||||
```bash
|
||||
mvn -pl :vehicle-history-app -Dtest=VehicleHistoryAppDefaultsTest test
|
||||
```
|
||||
|
||||
Expected: default configuration test passes.
|
||||
|
||||
## Task 5: Verify History Ingest Still Archives RAW Bytes
|
||||
|
||||
**Files:**
|
||||
- Modify: `modules/services/event-history-service/src/test/java/com/lingniu/ingest/eventhistory/EventHistoryEnvelopeIngestorTest.java`
|
||||
- Test: `modules/services/event-history-service/src/test/java/com/lingniu/ingest/eventhistory/Gb32960DecodedFrameServiceTest.java`
|
||||
|
||||
- [ ] **Step 1: Add an integration-style test using the hot store**
|
||||
|
||||
Add a test that writes a RAW envelope through `EventHistoryEnvelopeIngestor` into `DuckDbHotEventFileStore`, then finds it by URI.
|
||||
|
||||
```java
|
||||
@Test
|
||||
void rawArchiveEnvelopeCanBeFoundFromDuckDbHotStoreByUri(@TempDir Path tempDir) throws Exception {
|
||||
EventFileStore store = new DuckDbHotEventFileStore(tempDir, ZoneId.of("Asia/Shanghai"), OBJECT_MAPPER);
|
||||
CapturingArchiveStore archive = new CapturingArchiveStore();
|
||||
EventHistoryEnvelopeIngestor ingestor =
|
||||
new EventHistoryEnvelopeIngestor(store, new TelemetryEnvelopeRecordMapper(), archive);
|
||||
String rawArchiveKey = "2026/06/23/GB32960/VINRAW001/raw-event-hot.bin";
|
||||
String rawArchiveUri = "archive://" + rawArchiveKey;
|
||||
byte[] rawBytes = new byte[]{0x23, 0x23, 0x02, 0x01};
|
||||
|
||||
VehicleEnvelope envelope = VehicleEnvelope.newBuilder()
|
||||
.setSchemaVersion("1.0")
|
||||
.setEventId("raw-event-hot")
|
||||
.setVin("VINRAW001")
|
||||
.setSource("GB32960")
|
||||
.setProtocolVersion("V2016")
|
||||
.setEventTimeMs(1_782_112_400_000L)
|
||||
.setIngestTimeMs(1_782_112_401_000L)
|
||||
.putMetadata(RawArchiveKeys.META_KEY, rawArchiveKey)
|
||||
.putMetadata(RawArchiveKeys.META_URI, rawArchiveUri)
|
||||
.setRawArchive(RawArchiveRef.newBuilder()
|
||||
.setUri(rawArchiveUri)
|
||||
.setSizeBytes(rawBytes.length)
|
||||
.setData(ByteString.copyFrom(rawBytes))
|
||||
.build())
|
||||
.build();
|
||||
|
||||
EnvelopeIngestResult result = ingestor.tryIngest(envelope.toByteArray());
|
||||
|
||||
assertThat(result.status()).isEqualTo(EnvelopeIngestResult.Status.STORED);
|
||||
assertThat(store.findByRawArchiveUri(rawArchiveUri))
|
||||
.isNotNull()
|
||||
.extracting(EventFileRecord::eventId)
|
||||
.isEqualTo("raw-event-hot");
|
||||
assertThat(archive.bytesByKey).containsEntry(rawArchiveKey, rawBytes);
|
||||
}
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Run the event history ingestor test**
|
||||
|
||||
Run:
|
||||
|
||||
```bash
|
||||
mvn -pl :event-history-service -Dtest=EventHistoryEnvelopeIngestorTest test
|
||||
```
|
||||
|
||||
Expected: test passes.
|
||||
|
||||
- [ ] **Step 3: Run decoded frame service tests**
|
||||
|
||||
Run:
|
||||
|
||||
```bash
|
||||
mvn -pl :event-history-service -Dtest=Gb32960DecodedFrameServiceTest test
|
||||
```
|
||||
|
||||
Expected: existing replay/snapshot tests pass. If they use a fake store, no change is needed.
|
||||
|
||||
## Task 6: Full Module Verification
|
||||
|
||||
**Files:**
|
||||
- No source changes unless failures expose missing imports or config assertions.
|
||||
|
||||
- [ ] **Step 1: Run sink module tests**
|
||||
|
||||
Run:
|
||||
|
||||
```bash
|
||||
mvn -pl :event-file-store test
|
||||
```
|
||||
|
||||
Expected: all event-file-store tests pass, including both hot and legacy stores.
|
||||
|
||||
- [ ] **Step 2: Run event-history-service tests**
|
||||
|
||||
Run:
|
||||
|
||||
```bash
|
||||
mvn -pl :event-history-service test
|
||||
```
|
||||
|
||||
Expected: all event-history-service tests pass.
|
||||
|
||||
- [ ] **Step 3: Package vehicle-history-app**
|
||||
|
||||
Run:
|
||||
|
||||
```bash
|
||||
mvn -pl :vehicle-history-app -am package -DskipTests
|
||||
```
|
||||
|
||||
Expected: package succeeds.
|
||||
|
||||
## Task 7: Local Runtime Verification
|
||||
|
||||
**Files:**
|
||||
- No committed source changes.
|
||||
|
||||
- [ ] **Step 1: Stop any old history service on port 20200**
|
||||
|
||||
Run:
|
||||
|
||||
```bash
|
||||
lsof -tiTCP:20200 -sTCP:LISTEN | xargs -r kill
|
||||
```
|
||||
|
||||
Expected: no command output, or the old process exits.
|
||||
|
||||
- [ ] **Step 2: Start vehicle-history-app with hot DuckDB store**
|
||||
|
||||
Run:
|
||||
|
||||
```bash
|
||||
EVENT_FILE_STORE_STORAGE=duckdb-hot \
|
||||
EVENT_FILE_STORE_PATH=./target/live-history-event-store \
|
||||
SINK_ARCHIVE_PATH=./target/live-history-archive \
|
||||
KAFKA_BROKERS=114.55.58.251:9092 \
|
||||
KAFKA_CONSUMER_ENABLED=true \
|
||||
KAFKA_GROUP_HISTORY=vehicle-history-hot-$(date +%Y%m%d%H%M%S) \
|
||||
java --sun-misc-unsafe-memory-access=allow \
|
||||
-jar modules/apps/vehicle-history-app/target/vehicle-history-app.jar
|
||||
```
|
||||
|
||||
Expected:
|
||||
|
||||
- app starts on `http://127.0.0.1:20200`;
|
||||
- logs show Kafka consumer subscribed;
|
||||
- `target/live-history-event-store/events.duckdb` is created.
|
||||
|
||||
- [ ] **Step 3: Verify health**
|
||||
|
||||
Run:
|
||||
|
||||
```bash
|
||||
curl -sS http://127.0.0.1:20200/actuator/health
|
||||
```
|
||||
|
||||
Expected:
|
||||
|
||||
```json
|
||||
{"status":"UP"}
|
||||
```
|
||||
|
||||
- [ ] **Step 4: Verify a recent VIN query**
|
||||
|
||||
Run with a VIN observed in live archive:
|
||||
|
||||
```bash
|
||||
curl -sS 'http://127.0.0.1:20200/api/event-history/gb32960/telemetry-snapshots?vin=LNXNEGRR9SR318194&platformAccount=Hyundai&dateFrom=2026-06-23&dateTo=2026-06-24&order=DESC&limit=3'
|
||||
```
|
||||
|
||||
Expected:
|
||||
|
||||
- response is a JSON array;
|
||||
- if live traffic for that VIN exists after service start, at least one snapshot appears;
|
||||
- `rawArchiveUris` point to existing files under `target/live-history-archive`.
|
||||
|
||||
- [ ] **Step 5: Verify raw frame replay**
|
||||
|
||||
Use one `rawArchiveUri` from Step 4:
|
||||
|
||||
```bash
|
||||
curl -sS 'http://127.0.0.1:20200/api/event-history/gb32960/frame?rawArchiveUri=archive://REPLACE_ME&platformAccount=Hyundai'
|
||||
```
|
||||
|
||||
Expected:
|
||||
|
||||
- response contains `vin`, `command`, `eventTime`, and parsed `blocks`;
|
||||
- no `raw archive is missing` warning for the selected URI.
|
||||
|
||||
## Self-Review Checklist
|
||||
|
||||
- [ ] The new hot store does not rewrite Parquet files on every append.
|
||||
- [ ] `appendAll` writes one batch in one transaction.
|
||||
- [ ] Duplicate `event_id` replay is idempotent.
|
||||
- [ ] Existing history HTTP APIs still use `EventFileStore`, so controller contracts are unchanged.
|
||||
- [ ] RAW bytes still land in `ArchiveStore`.
|
||||
- [ ] `findByRawArchiveUri` is backed by DuckDB index.
|
||||
- [ ] No MySQL credential, RDS host, username, or password appears in the plan or source files.
|
||||
- [ ] This plan does not claim the full production goal is complete; it only lands the history foundation.
|
||||
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
@@ -1,46 +0,0 @@
|
||||
# JT808 Kafka Streaming Mileage Plan
|
||||
|
||||
## Current Goal
|
||||
|
||||
Consume JT808 Kafka location events and write the derived daily mileage into the common vehicle-stat metric repository.
|
||||
|
||||
## Current Design
|
||||
|
||||
- Only supported message backbone: Kafka.
|
||||
- Source topic: `vehicle.event.jt808.v1`
|
||||
- Runtime app: `vehicle-analytics-app`
|
||||
- Runtime state: none outside `vehicle_stat_metric`
|
||||
- Metric output: `VehicleStatRepository.recordDailyMileageSample(...)`
|
||||
- Production metric storage: JDBC/MySQL `vehicle_stat_metric`
|
||||
- Date boundary: `Asia/Shanghai`
|
||||
- Calculation method: `JT808_TOTAL_MILEAGE_DIFF`
|
||||
|
||||
JT808 daily mileage is calculated only from the GPS total mileage reported by JT808 location additional information:
|
||||
|
||||
```text
|
||||
daily_mileage_km = max_total_mileage_km - min_total_mileage_km
|
||||
```
|
||||
|
||||
The first valid local-day sample writes `daily_mileage_km=0.0`. Later ordered or replayed samples update the same metric row with:
|
||||
|
||||
```text
|
||||
metric_value = max_total_mileage_km - min_total_mileage_km
|
||||
metric_key = daily_mileage_km
|
||||
```
|
||||
|
||||
The local-day minimum and maximum GPS total mileage values stay on that same metric row as calculation source columns so restarts recover from MySQL without Redis or another state table.
|
||||
|
||||
## Runtime Properties
|
||||
|
||||
```text
|
||||
KAFKA_TOPIC_JT808_EVENT=vehicle.event.jt808.v1
|
||||
VEHICLE_STAT_ENABLED=true
|
||||
VEHICLE_STAT_JT808_MILEAGE_ENABLED=true
|
||||
MYSQL_JDBC_URL=<jdbc-url>
|
||||
MYSQL_USERNAME=<user>
|
||||
MYSQL_PASSWORD=<password>
|
||||
```
|
||||
|
||||
## Notes
|
||||
|
||||
Do not create or write a protocol-specific JT808 daily-mileage table. Do not add another message backbone, distance accumulation, integral calculation, Redis state, or memory state back into this path unless the mileage definition changes again.
|
||||
@@ -1,720 +0,0 @@
|
||||
# Go Vehicle Ingest Redesign Phase 1 Implementation Plan
|
||||
|
||||
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
|
||||
|
||||
**Goal:** Build the first production-capable Go runtime for GB32960, JT808, and Yutong MQTT ingestion with unified Kafka, TDengine, MySQL statistics, and Redis realtime state boundaries.
|
||||
|
||||
**Architecture:** Create a new single Go module at `go/vehicle-gateway` and migrate only the useful pieces from the existing `go/ingest-edge` and `go/vehicle-state` prototypes. The gateway produces unified envelopes to Kafka; independent consumers write TDengine history, MySQL daily metrics, and Redis realtime state.
|
||||
|
||||
**Tech Stack:** Go 1.26+, Kafka (`segmentio/kafka-go`), Redis (`redis/go-redis`), MySQL (`go-sql-driver/mysql`), TDengine official Go connector, MQTT (`eclipse/paho.mqtt.golang`), standard library TCP.
|
||||
|
||||
---
|
||||
|
||||
## File Structure
|
||||
|
||||
- Create: `go/vehicle-gateway/go.mod`
|
||||
- Create: `go/vehicle-gateway/cmd/gateway/main.go`
|
||||
- Create: `go/vehicle-gateway/cmd/history-writer/main.go`
|
||||
- Create: `go/vehicle-gateway/cmd/stat-writer/main.go`
|
||||
- Create: `go/vehicle-gateway/cmd/realtime-api/main.go`
|
||||
- Create: `go/vehicle-gateway/internal/envelope/envelope.go`
|
||||
- Create: `go/vehicle-gateway/internal/envelope/envelope_test.go`
|
||||
- Create: `go/vehicle-gateway/internal/protocol/jt808/*`
|
||||
- Create: `go/vehicle-gateway/internal/protocol/gb32960/*`
|
||||
- Create: `go/vehicle-gateway/internal/protocol/yutongmqtt/*`
|
||||
- Create: `go/vehicle-gateway/internal/gateway/*`
|
||||
- Create: `go/vehicle-gateway/internal/identity/*`
|
||||
- Create: `go/vehicle-gateway/internal/eventbus/*`
|
||||
- Create: `go/vehicle-gateway/internal/history/*`
|
||||
- Create: `go/vehicle-gateway/internal/stats/*`
|
||||
- Create: `go/vehicle-gateway/internal/realtime/*`
|
||||
- Create: `go/vehicle-gateway/internal/observability/*`
|
||||
- Modify: `README.md`
|
||||
- Modify: `docs/target-architecture.md`
|
||||
- Create: `deploy/portainer/docker-compose-go.yml`
|
||||
|
||||
The existing `go/ingest-edge` and `go/vehicle-state` directories are migration sources only. After phase 1 is verified, remove or mark them superseded.
|
||||
|
||||
---
|
||||
|
||||
### Task 1: Create Single Go Module
|
||||
|
||||
**Files:**
|
||||
- Create: `go/vehicle-gateway/go.mod`
|
||||
- Create: `go/vehicle-gateway/internal/observability/logger.go`
|
||||
- Create: `go/vehicle-gateway/cmd/gateway/main.go`
|
||||
|
||||
- [ ] **Step 1: Create module manifest**
|
||||
|
||||
Add `go/vehicle-gateway/go.mod`:
|
||||
|
||||
```go
|
||||
module lingniu-vehicle-ingest/go/vehicle-gateway
|
||||
|
||||
go 1.26
|
||||
|
||||
require (
|
||||
github.com/eclipse/paho.mqtt.golang v1.5.1
|
||||
github.com/go-sql-driver/mysql v1.9.3
|
||||
github.com/redis/go-redis/v9 v9.17.2
|
||||
github.com/segmentio/kafka-go v0.4.49
|
||||
github.com/taosdata/driver-go/v3 v3.8.1
|
||||
)
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Add logger helper**
|
||||
|
||||
Add `go/vehicle-gateway/internal/observability/logger.go`:
|
||||
|
||||
```go
|
||||
package observability
|
||||
|
||||
import (
|
||||
"log/slog"
|
||||
"os"
|
||||
)
|
||||
|
||||
func NewLogger(service string) *slog.Logger {
|
||||
handler := slog.NewJSONHandler(os.Stdout, &slog.HandlerOptions{AddSource: true})
|
||||
return slog.New(handler).With("service", service)
|
||||
}
|
||||
```
|
||||
|
||||
- [ ] **Step 3: Add temporary gateway entrypoint**
|
||||
|
||||
Add `go/vehicle-gateway/cmd/gateway/main.go`:
|
||||
|
||||
```go
|
||||
package main
|
||||
|
||||
import "lingniu-vehicle-ingest/go/vehicle-gateway/internal/observability"
|
||||
|
||||
func main() {
|
||||
logger := observability.NewLogger("vehicle-gateway")
|
||||
logger.Info("vehicle gateway scaffold started")
|
||||
}
|
||||
```
|
||||
|
||||
- [ ] **Step 4: Verify scaffold builds**
|
||||
|
||||
Run:
|
||||
|
||||
```bash
|
||||
cd go/vehicle-gateway
|
||||
go mod tidy
|
||||
go test ./...
|
||||
go build ./cmd/gateway
|
||||
```
|
||||
|
||||
Expected: all commands exit `0`.
|
||||
|
||||
---
|
||||
|
||||
### Task 2: Define Unified Envelope
|
||||
|
||||
**Files:**
|
||||
- Create: `go/vehicle-gateway/internal/envelope/envelope.go`
|
||||
- Create: `go/vehicle-gateway/internal/envelope/envelope_test.go`
|
||||
|
||||
- [ ] **Step 1: Write envelope tests**
|
||||
|
||||
Add `go/vehicle-gateway/internal/envelope/envelope_test.go`:
|
||||
|
||||
```go
|
||||
package envelope
|
||||
|
||||
import "testing"
|
||||
|
||||
func TestFrameEnvelopeVehicleKeyPrefersVIN(t *testing.T) {
|
||||
e := FrameEnvelope{Protocol: ProtocolJT808, VIN: "LNBVIN00000000001", Phone: "013307795425"}
|
||||
if got := e.VehicleKey(); got != "LNBVIN00000000001" {
|
||||
t.Fatalf("VehicleKey() = %q", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestFrameEnvelopeVehicleKeyFallsBackToPhone(t *testing.T) {
|
||||
e := FrameEnvelope{Protocol: ProtocolJT808, Phone: "013307795425"}
|
||||
if got := e.VehicleKey(); got != "JT808:013307795425" {
|
||||
t.Fatalf("VehicleKey() = %q", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestFrameEnvelopeEventIDStable(t *testing.T) {
|
||||
e := FrameEnvelope{
|
||||
Protocol: ProtocolJT808,
|
||||
MessageID: "0x0200",
|
||||
Phone: "013307795425",
|
||||
Sequence: 1,
|
||||
EventTimeMS: 1782745114000,
|
||||
ReceivedAtMS: 1782745114999,
|
||||
RawHex: "7e02000000ff7e",
|
||||
}
|
||||
a := e.StableEventID()
|
||||
b := e.StableEventID()
|
||||
if a == "" || a != b {
|
||||
t.Fatalf("event id must be non-empty and stable: %q %q", a, b)
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Run tests and confirm failure**
|
||||
|
||||
Run:
|
||||
|
||||
```bash
|
||||
cd go/vehicle-gateway
|
||||
go test ./internal/envelope
|
||||
```
|
||||
|
||||
Expected: fail because `FrameEnvelope` is not defined.
|
||||
|
||||
- [ ] **Step 3: Implement envelope**
|
||||
|
||||
Add `go/vehicle-gateway/internal/envelope/envelope.go`:
|
||||
|
||||
```go
|
||||
package envelope
|
||||
|
||||
import (
|
||||
"crypto/sha256"
|
||||
"encoding/hex"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"strings"
|
||||
)
|
||||
|
||||
type Protocol string
|
||||
|
||||
const (
|
||||
ProtocolGB32960 Protocol = "GB32960"
|
||||
ProtocolJT808 Protocol = "JT808"
|
||||
ProtocolYutongMQTT Protocol = "YUTONG_MQTT"
|
||||
)
|
||||
|
||||
type ParseStatus string
|
||||
|
||||
const (
|
||||
ParseOK ParseStatus = "OK"
|
||||
ParsePartial ParseStatus = "PARTIAL"
|
||||
ParseBadFrame ParseStatus = "BAD_FRAME"
|
||||
)
|
||||
|
||||
type FrameEnvelope struct {
|
||||
EventID string `json:"event_id"`
|
||||
TraceID string `json:"trace_id"`
|
||||
Protocol Protocol `json:"protocol"`
|
||||
MessageID string `json:"message_id"`
|
||||
Sequence uint16 `json:"sequence"`
|
||||
VIN string `json:"vin,omitempty"`
|
||||
VehicleKeyHint string `json:"vehicle_key,omitempty"`
|
||||
Phone string `json:"phone,omitempty"`
|
||||
DeviceID string `json:"device_id,omitempty"`
|
||||
Plate string `json:"plate,omitempty"`
|
||||
SourceEndpoint string `json:"source_endpoint,omitempty"`
|
||||
EventTimeMS int64 `json:"event_time_ms"`
|
||||
ReceivedAtMS int64 `json:"received_at_ms"`
|
||||
RawHex string `json:"raw_hex,omitempty"`
|
||||
RawText string `json:"raw_text,omitempty"`
|
||||
Parsed map[string]any `json:"parsed,omitempty"`
|
||||
Fields map[string]any `json:"fields,omitempty"`
|
||||
ParseStatus ParseStatus `json:"parse_status"`
|
||||
ParseError string `json:"parse_error,omitempty"`
|
||||
}
|
||||
|
||||
func (e FrameEnvelope) VehicleKey() string {
|
||||
if key := strings.TrimSpace(e.VIN); key != "" {
|
||||
return key
|
||||
}
|
||||
if key := strings.TrimSpace(e.VehicleKeyHint); key != "" {
|
||||
return key
|
||||
}
|
||||
if key := strings.TrimSpace(e.Phone); key != "" {
|
||||
return string(e.Protocol) + ":" + key
|
||||
}
|
||||
if key := strings.TrimSpace(e.DeviceID); key != "" {
|
||||
return string(e.Protocol) + ":" + key
|
||||
}
|
||||
return string(e.Protocol) + ":unknown"
|
||||
}
|
||||
|
||||
func (e FrameEnvelope) StableEventID() string {
|
||||
if strings.TrimSpace(e.EventID) != "" {
|
||||
return e.EventID
|
||||
}
|
||||
input := fmt.Sprintf("%s|%s|%s|%d|%d|%s",
|
||||
e.Protocol, e.MessageID, e.VehicleKey(), e.Sequence, e.EventTimeMS, e.RawHex)
|
||||
sum := sha256.Sum256([]byte(input))
|
||||
return hex.EncodeToString(sum[:16])
|
||||
}
|
||||
|
||||
func (e FrameEnvelope) MarshalJSONBytes() ([]byte, error) {
|
||||
if e.EventID == "" {
|
||||
e.EventID = e.StableEventID()
|
||||
}
|
||||
if e.ParseStatus == "" {
|
||||
e.ParseStatus = ParseOK
|
||||
}
|
||||
return json.Marshal(e)
|
||||
}
|
||||
```
|
||||
|
||||
- [ ] **Step 4: Verify tests pass**
|
||||
|
||||
Run:
|
||||
|
||||
```bash
|
||||
cd go/vehicle-gateway
|
||||
go test ./internal/envelope
|
||||
```
|
||||
|
||||
Expected: pass.
|
||||
|
||||
---
|
||||
|
||||
### Task 3: Implement JT808 Frame and 0200 Parser
|
||||
|
||||
**Files:**
|
||||
- Create: `go/vehicle-gateway/internal/protocol/jt808/parser.go`
|
||||
- Create: `go/vehicle-gateway/internal/protocol/jt808/parser_test.go`
|
||||
|
||||
- [ ] **Step 1: Add sample frame tests**
|
||||
|
||||
Use the production sample provided in the thread:
|
||||
|
||||
```text
|
||||
7E020000320133077954250001000000000048000301D2C4C707376139000A00E6004F26063016235701040001900C2504000000000202000030011F31010F867E
|
||||
```
|
||||
|
||||
Expected assertions:
|
||||
|
||||
- message id is `0x0200`
|
||||
- phone keeps normalized BCD string `013307795425`
|
||||
- sequence is `1`
|
||||
- `fields.total_mileage_km` is parsed from additional item `0x01`
|
||||
- latitude, longitude, speed, direction, alarm, status, and device time are present
|
||||
|
||||
- [ ] **Step 2: Implement parser**
|
||||
|
||||
Implementation rules:
|
||||
|
||||
- strip `0x7e` start/end delimiters
|
||||
- unescape `0x7d 0x02 -> 0x7e`
|
||||
- unescape `0x7d 0x01 -> 0x7d`
|
||||
- verify XOR checksum
|
||||
- parse 2011/2013 common header
|
||||
- parse BCD phone from 6 bytes and keep both raw and normalized values in `parsed.header`
|
||||
- parse 0200 fixed body
|
||||
- parse additional item list as `id,length,value_hex`
|
||||
- parse additional `0x01` as `total_mileage_km = uint32 / 10`
|
||||
|
||||
- [ ] **Step 3: Verify**
|
||||
|
||||
Run:
|
||||
|
||||
```bash
|
||||
cd go/vehicle-gateway
|
||||
go test ./internal/protocol/jt808
|
||||
```
|
||||
|
||||
Expected: pass.
|
||||
|
||||
---
|
||||
|
||||
### Task 4: Implement GB32960 Frame and Data Unit Parser
|
||||
|
||||
**Files:**
|
||||
- Create: `go/vehicle-gateway/internal/protocol/gb32960/parser.go`
|
||||
- Create: `go/vehicle-gateway/internal/protocol/gb32960/parser_test.go`
|
||||
|
||||
- [ ] **Step 1: Add parser tests**
|
||||
|
||||
Tests must cover:
|
||||
|
||||
- `##` frame boundary
|
||||
- command id
|
||||
- response flag
|
||||
- 17-byte VIN
|
||||
- encryption flag
|
||||
- payload length
|
||||
- BCC verification
|
||||
- realtime data command `0x02`
|
||||
- reissue data command `0x03`
|
||||
- data unit `0x01` vehicle status fields
|
||||
- data unit `0x05` position fields
|
||||
|
||||
- [ ] **Step 2: Implement parser**
|
||||
|
||||
Implementation rules:
|
||||
|
||||
- parse header without allocating large temporary buffers
|
||||
- keep RAW as hex in envelope
|
||||
- write all recognized data units to `Parsed`
|
||||
- write core fields to `Fields`
|
||||
- unsupported data unit stays in `Parsed["unknown_units"]`
|
||||
|
||||
- [ ] **Step 3: Verify**
|
||||
|
||||
Run:
|
||||
|
||||
```bash
|
||||
cd go/vehicle-gateway
|
||||
go test ./internal/protocol/gb32960
|
||||
```
|
||||
|
||||
Expected: pass.
|
||||
|
||||
---
|
||||
|
||||
### Task 5: Implement Kafka Sink
|
||||
|
||||
**Files:**
|
||||
- Create: `go/vehicle-gateway/internal/eventbus/kafka_sink.go`
|
||||
- Create: `go/vehicle-gateway/internal/eventbus/kafka_sink_test.go`
|
||||
|
||||
- [ ] **Step 1: Add sink interface**
|
||||
|
||||
Create a sink interface:
|
||||
|
||||
```go
|
||||
package eventbus
|
||||
|
||||
import (
|
||||
"context"
|
||||
"lingniu-vehicle-ingest/go/vehicle-gateway/internal/envelope"
|
||||
)
|
||||
|
||||
type Sink interface {
|
||||
PublishRaw(context.Context, envelope.FrameEnvelope) error
|
||||
PublishUnified(context.Context, envelope.FrameEnvelope) error
|
||||
Close() error
|
||||
}
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Implement Kafka topic routing**
|
||||
|
||||
Rules:
|
||||
|
||||
- `GB32960 -> vehicle.raw.gb32960.v1`
|
||||
- `JT808 -> vehicle.raw.jt808.v1`
|
||||
- `YUTONG_MQTT -> vehicle.raw.yutong-mqtt.v1`
|
||||
- unified topic is `vehicle.event.unified.v1`
|
||||
- message key is `env.VehicleKey()`
|
||||
|
||||
- [ ] **Step 3: Verify routing with unit tests**
|
||||
|
||||
Run:
|
||||
|
||||
```bash
|
||||
cd go/vehicle-gateway
|
||||
go test ./internal/eventbus
|
||||
```
|
||||
|
||||
Expected: pass.
|
||||
|
||||
---
|
||||
|
||||
### Task 6: Implement TDengine History Writer
|
||||
|
||||
**Files:**
|
||||
- Create: `go/vehicle-gateway/internal/history/schema.go`
|
||||
- Create: `go/vehicle-gateway/internal/history/writer.go`
|
||||
- Create: `go/vehicle-gateway/internal/history/writer_test.go`
|
||||
- Create: `go/vehicle-gateway/cmd/history-writer/main.go`
|
||||
|
||||
- [ ] **Step 1: Add schema bootstrap SQL**
|
||||
|
||||
Implement schema strings for:
|
||||
|
||||
- database `lingniu_vehicle_ts`
|
||||
- stable `raw_frames`
|
||||
- stable `vehicle_locations`
|
||||
- stable `vehicle_mileage_points`
|
||||
|
||||
- [ ] **Step 2: Implement writer**
|
||||
|
||||
Rules:
|
||||
|
||||
- `AppendRawFrame` always writes one raw row.
|
||||
- `AppendLocation` writes only when longitude and latitude exist.
|
||||
- `AppendMileagePoint` writes only when `total_mileage_km` exists.
|
||||
- child table name is deterministic hash of `protocol + vehicle_key`.
|
||||
- escape tag values.
|
||||
|
||||
- [ ] **Step 3: Use TDengine official driver**
|
||||
|
||||
Import WebSocket driver in command:
|
||||
|
||||
```go
|
||||
import _ "github.com/taosdata/driver-go/v3/taosWS"
|
||||
```
|
||||
|
||||
Default driver name:
|
||||
|
||||
```text
|
||||
TDENGINE_DRIVER=taosWS
|
||||
```
|
||||
|
||||
Short-term compatibility:
|
||||
|
||||
```text
|
||||
TDENGINE_DRIVER=taosSql
|
||||
```
|
||||
|
||||
Only use `taosSql` when ECS TDengine WebSocket is not available.
|
||||
|
||||
- [ ] **Step 4: Verify**
|
||||
|
||||
Run:
|
||||
|
||||
```bash
|
||||
cd go/vehicle-gateway
|
||||
go test ./internal/history
|
||||
go build ./cmd/history-writer
|
||||
```
|
||||
|
||||
Expected: pass.
|
||||
|
||||
---
|
||||
|
||||
### Task 7: Implement MySQL Daily Metric Writer
|
||||
|
||||
**Files:**
|
||||
- Create: `go/vehicle-gateway/internal/stats/schema.go`
|
||||
- Create: `go/vehicle-gateway/internal/stats/daily_metric.go`
|
||||
- Create: `go/vehicle-gateway/internal/stats/daily_metric_test.go`
|
||||
- Create: `go/vehicle-gateway/cmd/stat-writer/main.go`
|
||||
|
||||
- [ ] **Step 1: Add schema bootstrap**
|
||||
|
||||
Implement `vehicle_daily_metric` schema from the design spec.
|
||||
|
||||
- [ ] **Step 2: Add metric derivation tests**
|
||||
|
||||
Test cases:
|
||||
|
||||
- no `total_mileage_km` produces no metric
|
||||
- one sample produces:
|
||||
- `daily_mileage_km = 0`
|
||||
- `daily_total_mileage_km = sample`
|
||||
- later larger sample updates:
|
||||
- `latest_total_mileage_km`
|
||||
- `daily_mileage_km`
|
||||
- `daily_total_mileage_km`
|
||||
- out-of-order smaller sample updates:
|
||||
- `first_total_mileage_km`
|
||||
- `daily_mileage_km`
|
||||
|
||||
- [ ] **Step 3: Implement MySQL upsert**
|
||||
|
||||
Use one table and one idempotent upsert:
|
||||
|
||||
```sql
|
||||
INSERT INTO vehicle_daily_metric
|
||||
(vin, stat_date, protocol, metric_key, metric_value, metric_unit,
|
||||
first_total_mileage_km, latest_total_mileage_km, sample_count, calculation_method)
|
||||
VALUES (?, ?, ?, ?, ?, 'km', ?, ?, 1, 'TOTAL_MILEAGE_DIFF')
|
||||
ON DUPLICATE KEY UPDATE
|
||||
first_total_mileage_km = LEAST(first_total_mileage_km, VALUES(first_total_mileage_km)),
|
||||
latest_total_mileage_km = GREATEST(latest_total_mileage_km, VALUES(latest_total_mileage_km)),
|
||||
metric_value = CASE
|
||||
WHEN metric_key = 'daily_mileage_km'
|
||||
THEN GREATEST(latest_total_mileage_km, VALUES(latest_total_mileage_km))
|
||||
- LEAST(first_total_mileage_km, VALUES(first_total_mileage_km))
|
||||
WHEN metric_key = 'daily_total_mileage_km'
|
||||
THEN GREATEST(latest_total_mileage_km, VALUES(latest_total_mileage_km))
|
||||
ELSE VALUES(metric_value)
|
||||
END,
|
||||
sample_count = sample_count + 1,
|
||||
updated_at = CURRENT_TIMESTAMP
|
||||
```
|
||||
|
||||
- [ ] **Step 4: Verify**
|
||||
|
||||
Run:
|
||||
|
||||
```bash
|
||||
cd go/vehicle-gateway
|
||||
go test ./internal/stats
|
||||
go build ./cmd/stat-writer
|
||||
```
|
||||
|
||||
Expected: pass.
|
||||
|
||||
---
|
||||
|
||||
### Task 8: Implement Redis Realtime API
|
||||
|
||||
**Files:**
|
||||
- Create: `go/vehicle-gateway/internal/realtime/repository.go`
|
||||
- Create: `go/vehicle-gateway/internal/realtime/repository_test.go`
|
||||
- Create: `go/vehicle-gateway/internal/realtime/http.go`
|
||||
- Create: `go/vehicle-gateway/cmd/realtime-api/main.go`
|
||||
|
||||
- [ ] **Step 1: Add repository contract**
|
||||
|
||||
Methods:
|
||||
|
||||
- `Update(ctx, envelope.FrameEnvelope) error`
|
||||
- `GetMerged(ctx, vin string) (Snapshot, error)`
|
||||
- `GetProtocol(ctx, vin string, protocol envelope.Protocol) (Snapshot, error)`
|
||||
- `IsOnline(ctx, vin string) (OnlineStatus, error)`
|
||||
|
||||
- [ ] **Step 2: Implement merge logic**
|
||||
|
||||
Rules:
|
||||
|
||||
- update `vehicle:latest:{vin}:{protocol}`
|
||||
- update `vehicle:latest:{vin}` with newest fields
|
||||
- update `vehicle:online:{vin}`
|
||||
- update sorted set `vehicle:last_seen`
|
||||
|
||||
- [ ] **Step 3: Implement HTTP API**
|
||||
|
||||
Routes:
|
||||
|
||||
- `GET /api/realtime/vehicles/{vin}`
|
||||
- `GET /api/realtime/vehicles/{vin}/online`
|
||||
- `GET /api/realtime/vehicles/{vin}/protocols/{protocol}`
|
||||
|
||||
- [ ] **Step 4: Verify**
|
||||
|
||||
Run:
|
||||
|
||||
```bash
|
||||
cd go/vehicle-gateway
|
||||
go test ./internal/realtime
|
||||
go build ./cmd/realtime-api
|
||||
```
|
||||
|
||||
Expected: pass.
|
||||
|
||||
---
|
||||
|
||||
### Task 9: Wire Gateway Runtime
|
||||
|
||||
**Files:**
|
||||
- Create: `go/vehicle-gateway/internal/gateway/tcp_server.go`
|
||||
- Create: `go/vehicle-gateway/internal/gateway/mqtt_client.go`
|
||||
- Modify: `go/vehicle-gateway/cmd/gateway/main.go`
|
||||
|
||||
- [ ] **Step 1: Implement TCP server**
|
||||
|
||||
Rules:
|
||||
|
||||
- one goroutine per accepted connection
|
||||
- bounded max connections
|
||||
- read timeout and idle timeout
|
||||
- protocol-specific frame extractor
|
||||
- structured peer endpoint
|
||||
- graceful shutdown on SIGTERM
|
||||
|
||||
- [ ] **Step 2: Implement MQTT client**
|
||||
|
||||
Rules:
|
||||
|
||||
- connect with official production config from environment
|
||||
- subscribe configured topic list
|
||||
- convert each message into envelope
|
||||
- publish raw and unified events
|
||||
- reconnect with backoff
|
||||
|
||||
- [ ] **Step 3: Verify local JSON mode**
|
||||
|
||||
Run without Kafka:
|
||||
|
||||
```bash
|
||||
cd go/vehicle-gateway
|
||||
GB32960_TCP_ADDR=:132960 JT808_TCP_ADDR=:18080 go run ./cmd/gateway
|
||||
```
|
||||
|
||||
Expected: service starts and logs configured listeners.
|
||||
|
||||
---
|
||||
|
||||
### Task 10: Docker and ECS Deployment
|
||||
|
||||
**Files:**
|
||||
- Create: `go/vehicle-gateway/Dockerfile`
|
||||
- Create: `deploy/portainer/docker-compose-go.yml`
|
||||
- Modify: `docs/operations/current-ecs-deployment.md`
|
||||
|
||||
- [ ] **Step 1: Add multi-stage Dockerfile**
|
||||
|
||||
Build all commands:
|
||||
|
||||
- `gateway`
|
||||
- `history-writer`
|
||||
- `stat-writer`
|
||||
- `realtime-api`
|
||||
|
||||
- [ ] **Step 2: Add Portainer compose**
|
||||
|
||||
Services:
|
||||
|
||||
- `go-vehicle-gateway`
|
||||
- `go-history-writer`
|
||||
- `go-stat-writer`
|
||||
- `go-realtime-api`
|
||||
|
||||
Each service must include:
|
||||
|
||||
- restart policy
|
||||
- memory limit
|
||||
- Kafka env
|
||||
- MySQL/TDengine/Redis env as needed
|
||||
- logging options
|
||||
|
||||
- [ ] **Step 3: Verify Linux build**
|
||||
|
||||
Run:
|
||||
|
||||
```bash
|
||||
cd go/vehicle-gateway
|
||||
GOOS=linux GOARCH=amd64 go build ./cmd/gateway
|
||||
GOOS=linux GOARCH=amd64 go build ./cmd/history-writer
|
||||
GOOS=linux GOARCH=amd64 go build ./cmd/stat-writer
|
||||
GOOS=linux GOARCH=amd64 go build ./cmd/realtime-api
|
||||
```
|
||||
|
||||
Expected: all commands exit `0`.
|
||||
|
||||
---
|
||||
|
||||
### Task 11: Production Verification
|
||||
|
||||
**Files:**
|
||||
- Create: `docs/operations/go-vehicle-gateway-verification.md`
|
||||
|
||||
- [ ] **Step 1: Record test commands**
|
||||
|
||||
Document commands to verify:
|
||||
|
||||
- gateway process health
|
||||
- Kafka topic consumption
|
||||
- TDengine row counts
|
||||
- MySQL daily metric rows
|
||||
- Redis realtime lookup
|
||||
|
||||
- [ ] **Step 2: Validate real traffic**
|
||||
|
||||
Evidence required:
|
||||
|
||||
- one real 32960 VIN with RAW, location, mileage point, daily metric, Redis snapshot
|
||||
- one real JT808 phone/VIN with RAW, location, mileage point, daily metric, Redis snapshot
|
||||
- one real Yutong MQTT VIN with RAW and Redis snapshot
|
||||
|
||||
- [ ] **Step 3: Keep old Java services until evidence is captured**
|
||||
|
||||
Only disable Java equivalents after the evidence file contains successful command outputs and timestamps.
|
||||
|
||||
---
|
||||
|
||||
## Self-Review Checklist
|
||||
|
||||
- The plan creates one new Go module instead of extending scattered prototypes.
|
||||
- The plan covers GB32960, JT808, and Yutong MQTT ingress.
|
||||
- The plan covers Kafka, TDengine, MySQL, and Redis.
|
||||
- The plan includes 32960 and 808 daily mileage and daily total mileage.
|
||||
- The plan includes local and ECS verification.
|
||||
- The plan does not restore Xinda Push.
|
||||
- The plan keeps Java services untouched until Go evidence exists.
|
||||
@@ -1,481 +0,0 @@
|
||||
> **Superseded:** This 2026-06-23 DuckDB-based design is historical context.
|
||||
> Use `docs/target-architecture.md` and
|
||||
> `docs/superpowers/specs/2026-06-29-vehicle-ingest-redesign.md` for the
|
||||
> current production architecture: TDengine is the default hot history store,
|
||||
> `event-file-store` and Xinda Push have been removed from the production codebase.
|
||||
|
||||
# GB32960 Production Readiness Design
|
||||
|
||||
## Goal
|
||||
|
||||
Make the GB32960 ingestion, history, and analytics services production-ready:
|
||||
|
||||
- receive and parse GB32960 traffic quickly and accurately;
|
||||
- maximize VIN-to-platform/vendor-profile association through platform login and local mappings;
|
||||
- store complete RAW frames and queryable telemetry history with DuckDB;
|
||||
- support full frame replay from persisted RAW bytes;
|
||||
- compute real-time daily vehicle metrics and alarm timelines into MySQL;
|
||||
- reduce service coupling and improve cohesion inside each module.
|
||||
|
||||
## Non-Goals
|
||||
|
||||
- Do not put database credentials in code, YAML files, tests, docs, or commits.
|
||||
- Do not make MySQL the source for full telemetry history. MySQL is for derived daily metrics and alarm timelines.
|
||||
- Do not require every private GB32960 extension to be perfect before storing data. RAW bytes must always be kept when a valid frame is received.
|
||||
|
||||
## Production Architecture
|
||||
|
||||
```mermaid
|
||||
flowchart LR
|
||||
tcp["GB32960 TCP:32960"] --> ingest["gb32960-ingest-app"]
|
||||
mapping["local vin-platform-profile.jsonl"] --> ingest
|
||||
ingest --> kafkaEvent["Kafka vehicle.event.gb32960.v1"]
|
||||
ingest --> kafkaRaw["Kafka vehicle.raw.gb32960.v1"]
|
||||
kafkaEvent --> history["vehicle-history-app"]
|
||||
kafkaRaw --> history
|
||||
history --> duckdb["DuckDB hot history"]
|
||||
history --> raw["RAW frame archive"]
|
||||
kafkaEvent --> analytics["vehicle-analytics-app"]
|
||||
analytics --> mysql["MySQL daily stats + alarms"]
|
||||
```
|
||||
|
||||
## Service Boundaries
|
||||
|
||||
### 1. `gb32960-ingest-app`
|
||||
|
||||
Responsibilities:
|
||||
|
||||
- Netty TCP accept, frame splitting, GB32960 decode, auth, ACK, and Kafka production.
|
||||
- Platform login handling and connection-local `platformAccount`.
|
||||
- VIN-to-platform/vendor-profile resolution.
|
||||
- Produce structured telemetry, session, alarm, diagnostics, and RAW envelope messages.
|
||||
|
||||
It must not:
|
||||
|
||||
- write DuckDB, MySQL, or local archive files;
|
||||
- run historical queries;
|
||||
- run daily statistics.
|
||||
|
||||
ACK policy:
|
||||
|
||||
- login/report-like GB32960 commands use durable ACK: ACK only after Kafka dispatch succeeds;
|
||||
- malformed bytes that cannot form a valid frame are dropped with diagnostics;
|
||||
- valid frames with partial private-block parse failures still go to Kafka with `parseStatus=PARTIAL`.
|
||||
|
||||
VIN/profile resolution priority:
|
||||
|
||||
1. active platform login on the current TCP channel;
|
||||
2. exact VIN entry from local mapping file;
|
||||
3. VIN prefix entry from local mapping file;
|
||||
4. configured platform extension rule;
|
||||
5. standard GB32960 profile.
|
||||
|
||||
Local mapping file format:
|
||||
|
||||
```jsonl
|
||||
{"vin":"LNXNEGRR2SR321390","platformAccount":"Hyundai","vendorProfile":"guangdong-fc","enabled":true}
|
||||
{"vinPrefix":"LNXNEGRR","platformAccount":"Hyundai","vendorProfile":"guangdong-fc","enabled":true}
|
||||
```
|
||||
|
||||
The mapping file is loaded at startup and can be reloaded by an actuator endpoint or file watcher in a later phase.
|
||||
|
||||
### 2. `vehicle-history-app`
|
||||
|
||||
Responsibilities:
|
||||
|
||||
- Consume Kafka event and RAW topics.
|
||||
- Persist RAW bytes to an append-only archive.
|
||||
- Persist query indexes and compact telemetry points to DuckDB.
|
||||
- Serve high-performance query APIs:
|
||||
- full frame list by `vin + eventTime`;
|
||||
- telemetry fields by `vin + eventTime`;
|
||||
- one-frame replay by `rawArchiveUri`;
|
||||
- parse diagnostics and missing-profile troubleshooting.
|
||||
|
||||
It must not:
|
||||
|
||||
- listen on GB32960 TCP;
|
||||
- run MySQL statistics;
|
||||
- mutate daily business metrics.
|
||||
|
||||
### 3. `vehicle-analytics-app`
|
||||
|
||||
Responsibilities:
|
||||
|
||||
- Consume Kafka event topic.
|
||||
- Maintain real-time daily stats in MySQL.
|
||||
- Maintain full alarm timelines in MySQL.
|
||||
- Expose metrics query APIs and operational health.
|
||||
|
||||
It must not:
|
||||
|
||||
- parse RAW frames for normal operation;
|
||||
- write DuckDB history;
|
||||
- own GB32960 TCP connectivity.
|
||||
|
||||
## Kafka Contract
|
||||
|
||||
Keep Protobuf `VehicleEnvelope`, but make the producer contract stricter:
|
||||
|
||||
- `eventId`: stable idempotency key, deterministic from protocol, VIN, command, event time, raw checksum, and sequence when available.
|
||||
- `vin`: required for vehicle frames; platform-only commands use `_platform` or empty plus `platformAccount`.
|
||||
- `eventTimeMs`: GB32960 data collection time when available.
|
||||
- `ingestTimeMs`: service receive time.
|
||||
- `metadata.platformAccount`: resolved platform account.
|
||||
- `metadata.vendorProfile`: selected parser profile.
|
||||
- `metadata.parseStatus`: `OK`, `PARTIAL`, or `FAILED`.
|
||||
- `metadata.parseErrorCode`: stable code for failures.
|
||||
- `raw_archive.data`: present on RAW topic only; event topic keeps `rawArchiveUri` pointer.
|
||||
- `telemetry_snapshot`: standard field projection for downstream services.
|
||||
|
||||
Consumer rule:
|
||||
|
||||
- History consumes both event and RAW topics.
|
||||
- Analytics consumes only event topic.
|
||||
- Consumer offset is committed only after the target store commit succeeds.
|
||||
- Reprocessing must be safe through deterministic ids and database upsert/ignore semantics.
|
||||
|
||||
## DuckDB History Design
|
||||
|
||||
Use DuckDB as the hot historical database, not a Parquet-rewrite sidecar.
|
||||
|
||||
DuckDB official guidance says row-by-row insert loops are inefficient for bulk loading, and Appender/batched writes are the intended high-throughput path. Therefore history writes use a single writer worker with bounded batches and one DuckDB connection.
|
||||
|
||||
### Tables
|
||||
|
||||
`gb32960_frame_index`
|
||||
|
||||
```sql
|
||||
CREATE TABLE IF NOT EXISTS gb32960_frame_index (
|
||||
event_id VARCHAR PRIMARY KEY,
|
||||
vin VARCHAR NOT NULL,
|
||||
platform_account VARCHAR,
|
||||
vendor_profile VARCHAR,
|
||||
command VARCHAR NOT NULL,
|
||||
command_code INTEGER NOT NULL,
|
||||
event_time TIMESTAMP NOT NULL,
|
||||
event_time_ms BIGINT NOT NULL,
|
||||
ingest_time TIMESTAMP NOT NULL,
|
||||
ingest_time_ms BIGINT NOT NULL,
|
||||
partition_date DATE NOT NULL,
|
||||
raw_archive_uri VARCHAR NOT NULL,
|
||||
raw_checksum VARCHAR,
|
||||
raw_size_bytes BIGINT NOT NULL,
|
||||
parse_status VARCHAR NOT NULL,
|
||||
parse_error_code VARCHAR,
|
||||
block_count INTEGER NOT NULL,
|
||||
metadata_json VARCHAR NOT NULL
|
||||
);
|
||||
|
||||
CREATE INDEX IF NOT EXISTS gb32960_frame_vin_time_idx
|
||||
ON gb32960_frame_index(vin, event_time_ms);
|
||||
|
||||
CREATE INDEX IF NOT EXISTS gb32960_frame_raw_uri_idx
|
||||
ON gb32960_frame_index(raw_archive_uri);
|
||||
|
||||
CREATE INDEX IF NOT EXISTS gb32960_frame_partition_time_idx
|
||||
ON gb32960_frame_index(partition_date, event_time_ms);
|
||||
```
|
||||
|
||||
`gb32960_telemetry_point`
|
||||
|
||||
```sql
|
||||
CREATE TABLE IF NOT EXISTS gb32960_telemetry_point (
|
||||
event_id VARCHAR NOT NULL,
|
||||
vin VARCHAR NOT NULL,
|
||||
event_time TIMESTAMP NOT NULL,
|
||||
event_time_ms BIGINT NOT NULL,
|
||||
ingest_time TIMESTAMP NOT NULL,
|
||||
platform_account VARCHAR,
|
||||
vendor_profile VARCHAR,
|
||||
field_key VARCHAR NOT NULL,
|
||||
value_type VARCHAR NOT NULL,
|
||||
value_text VARCHAR NOT NULL,
|
||||
value_num DOUBLE,
|
||||
unit VARCHAR,
|
||||
quality VARCHAR NOT NULL,
|
||||
raw_archive_uri VARCHAR NOT NULL,
|
||||
PRIMARY KEY(event_id, field_key)
|
||||
);
|
||||
|
||||
CREATE INDEX IF NOT EXISTS gb32960_point_vin_field_time_idx
|
||||
ON gb32960_telemetry_point(vin, field_key, event_time_ms);
|
||||
```
|
||||
|
||||
`gb32960_alarm_frame`
|
||||
|
||||
```sql
|
||||
CREATE TABLE IF NOT EXISTS gb32960_alarm_frame (
|
||||
event_id VARCHAR PRIMARY KEY,
|
||||
vin VARCHAR NOT NULL,
|
||||
event_time TIMESTAMP NOT NULL,
|
||||
event_time_ms BIGINT NOT NULL,
|
||||
level VARCHAR NOT NULL,
|
||||
active_bits_json VARCHAR NOT NULL,
|
||||
fault_codes_json VARCHAR NOT NULL,
|
||||
raw_archive_uri VARCHAR NOT NULL
|
||||
);
|
||||
|
||||
CREATE INDEX IF NOT EXISTS gb32960_alarm_vin_time_idx
|
||||
ON gb32960_alarm_frame(vin, event_time_ms);
|
||||
```
|
||||
|
||||
### RAW Archive
|
||||
|
||||
RAW archive remains file/object based:
|
||||
|
||||
```text
|
||||
archive://yyyy/MM/dd/GB32960/{vin}/{event_id}.bin
|
||||
```
|
||||
|
||||
The archive writer verifies:
|
||||
|
||||
- stored byte length equals envelope `raw_archive.size_bytes`;
|
||||
- checksum matches;
|
||||
- URI is unique for the deterministic event id.
|
||||
|
||||
Frame replay path:
|
||||
|
||||
1. query `gb32960_frame_index` by `rawArchiveUri`;
|
||||
2. read RAW bytes from archive;
|
||||
3. resolve `platformAccount/vendorProfile` from index metadata;
|
||||
4. decode using current parser;
|
||||
5. return decoded frame plus original indexed parse diagnostics.
|
||||
|
||||
## Query APIs
|
||||
|
||||
Keep existing endpoints but back them by the new DuckDB tables:
|
||||
|
||||
- `GET /api/event-history/gb32960/frames`
|
||||
- full frame query by VIN/time.
|
||||
- `GET /api/event-history/gb32960/frame`
|
||||
- one RAW frame replay by `rawArchiveUri`.
|
||||
- `GET /api/event-history/gb32960/telemetry-snapshots`
|
||||
- compact telemetry snapshot by VIN/time.
|
||||
- `GET /api/event-history/gb32960/telemetry-fields`
|
||||
- field projection optimized by `gb32960_telemetry_point`.
|
||||
- `GET /api/event-history/gb32960/diagnostics`
|
||||
- parse status and missing-profile investigation by VIN/time.
|
||||
|
||||
Time semantics:
|
||||
|
||||
- `eventTime` is business query time.
|
||||
- `ingestTime` is kept for operational replay, late-arrival detection, and Kafka lag investigation.
|
||||
- Dates without timezone are interpreted in `Asia/Shanghai`.
|
||||
|
||||
## MySQL Analytics Design
|
||||
|
||||
Credentials are supplied only by environment variables:
|
||||
|
||||
```text
|
||||
MYSQL_HOST
|
||||
MYSQL_PORT
|
||||
MYSQL_DATABASE
|
||||
MYSQL_USERNAME
|
||||
MYSQL_PASSWORD
|
||||
```
|
||||
|
||||
Daily derived metrics are stored in the common metric table, not in a
|
||||
protocol-specific daily-mileage table.
|
||||
|
||||
```sql
|
||||
CREATE TABLE IF NOT EXISTS vehicle_stat_metric (
|
||||
vin VARCHAR(64) NOT NULL,
|
||||
stat_date DATE NOT NULL,
|
||||
metric_key VARCHAR(64) NOT NULL,
|
||||
metric_value DECIMAL(18,6) NULL,
|
||||
metric_unit VARCHAR(16) NOT NULL,
|
||||
calculation_method VARCHAR(64) NOT NULL,
|
||||
created_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
|
||||
updated_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
|
||||
PRIMARY KEY (vin, stat_date, metric_key)
|
||||
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4;
|
||||
```
|
||||
|
||||
`vehicle_alarm_timeline`
|
||||
|
||||
```sql
|
||||
CREATE TABLE IF NOT EXISTS vehicle_alarm_timeline (
|
||||
alarm_id VARCHAR(128) NOT NULL,
|
||||
vin VARCHAR(32) NOT NULL,
|
||||
alarm_key VARCHAR(128) NOT NULL,
|
||||
level VARCHAR(32) NOT NULL,
|
||||
start_time DATETIME(3) NOT NULL,
|
||||
end_time DATETIME(3),
|
||||
last_seen_time DATETIME(3) NOT NULL,
|
||||
start_raw_archive_uri VARCHAR(512),
|
||||
last_raw_archive_uri VARCHAR(512),
|
||||
status VARCHAR(16) NOT NULL,
|
||||
details_json JSON,
|
||||
updated_at DATETIME(3) NOT NULL,
|
||||
PRIMARY KEY (alarm_id),
|
||||
KEY vehicle_alarm_vin_time_idx (vin, start_time),
|
||||
KEY vehicle_alarm_open_idx (vin, status, alarm_key)
|
||||
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4;
|
||||
```
|
||||
|
||||
`vehicle_stat_event_dedup`
|
||||
|
||||
```sql
|
||||
CREATE TABLE IF NOT EXISTS vehicle_stat_event_dedup (
|
||||
event_id VARCHAR(128) NOT NULL,
|
||||
consumer_group VARCHAR(128) NOT NULL,
|
||||
processed_at DATETIME(3) NOT NULL,
|
||||
PRIMARY KEY (event_id, consumer_group)
|
||||
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4;
|
||||
```
|
||||
|
||||
Writes use `INSERT ... ON DUPLICATE KEY UPDATE` so replayed Kafka messages update the same daily row instead of creating duplicates.
|
||||
|
||||
## Daily Metric Algorithms
|
||||
|
||||
All daily statistics are keyed by `eventTime` in `Asia/Shanghai`.
|
||||
|
||||
### Total Mileage
|
||||
|
||||
Use the latest valid `totalMileageKm` seen for the day.
|
||||
|
||||
Validation:
|
||||
|
||||
- reject negative values;
|
||||
- reject impossible jumps based on configurable max speed and elapsed time;
|
||||
- keep the sample as diagnostic if rejected.
|
||||
|
||||
### Daily Mileage
|
||||
|
||||
JT808 daily mileage uses the GPS total mileage reported in location additional
|
||||
information. The local-day minimum and maximum GPS total mileage values rewrite
|
||||
the same metric row, so ordered streaming and historical replay use one
|
||||
calculation path.
|
||||
|
||||
```text
|
||||
dailyMileage = today.maxTotalMileage - today.minTotalMileage
|
||||
calculationMethod = JT808_TOTAL_MILEAGE_DIFF
|
||||
```
|
||||
|
||||
The result is stored in `vehicle_stat_metric` with
|
||||
`metric_key=daily_mileage_km`. There is no JT808-specific daily mileage table,
|
||||
no Redis mileage state, and no previous-day baseline calculation.
|
||||
|
||||
### Daily Power Consumption
|
||||
|
||||
Preferred:
|
||||
|
||||
- vendor cumulative power-consumption field if available.
|
||||
|
||||
Fallback:
|
||||
|
||||
```text
|
||||
deltaKwh = sum(max(0, totalVoltageV * totalCurrentA) * deltaSeconds / 3600000)
|
||||
```
|
||||
|
||||
The sign convention is configurable per vendor profile because GB32960 vehicle current may be positive or negative depending on charging/discharging representation.
|
||||
|
||||
### Daily Hydrogen Consumption
|
||||
|
||||
Use a tiered market-practical algorithm instead of a single fixed formula:
|
||||
|
||||
1. `DIRECT_COUNTER`: vendor cumulative hydrogen consumption or remaining hydrogen mass delta.
|
||||
2. `PVT_MASS_DELTA`: pressure/temperature/tank-volume mass estimate when storage data is available.
|
||||
3. `FUEL_CELL_ENERGY_MODEL`: integrate fuel-cell electric output and divide by calibrated efficiency.
|
||||
4. `UNKNOWN`: keep null when required signals are missing.
|
||||
|
||||
The PVT approach follows the industry pattern behind compressed hydrogen consumption tests, where pressure, temperature, and tank volume can be used to estimate hydrogen mass. The implementation must expose calibration constants per vehicle model/profile:
|
||||
|
||||
```json
|
||||
{
|
||||
"vinPrefix": "LNXNEGRR",
|
||||
"tankVolumeLiter": 140,
|
||||
"hydrogenModel": "PVT_MASS_DELTA",
|
||||
"fuelCellEfficiency": 0.52,
|
||||
"currentDischargeSign": "POSITIVE"
|
||||
}
|
||||
```
|
||||
|
||||
### Daily Hydrogen Consumption Rate
|
||||
|
||||
```text
|
||||
dailyHydrogenKgPer100km = dailyHydrogenKg / dailyMileageKm * 100
|
||||
```
|
||||
|
||||
If mileage is missing or less than the configured minimum distance, keep the rate null and set quality to `INSUFFICIENT_DISTANCE`.
|
||||
|
||||
### Alarm Timeline
|
||||
|
||||
For each telemetry frame:
|
||||
|
||||
- derive active alarm keys from alarm bits and fault code arrays;
|
||||
- open a timeline row when a key first appears;
|
||||
- update `last_seen_time` while it remains active;
|
||||
- close open alarms when the key disappears after a configurable grace window;
|
||||
- keep start/end raw archive URIs for replay.
|
||||
|
||||
## Reliability and Observability
|
||||
|
||||
Required metrics:
|
||||
|
||||
- TCP active connections;
|
||||
- frames received per command;
|
||||
- decode failures by code;
|
||||
- missing platform login rejections;
|
||||
- vendor profile resolution source;
|
||||
- Kafka send latency and failures;
|
||||
- history write batch size, latency, failure count;
|
||||
- DuckDB query latency by endpoint;
|
||||
- MySQL upsert latency and failure count;
|
||||
- consumer lag by group/topic/partition;
|
||||
- late sample count by VIN/date.
|
||||
|
||||
Required logs:
|
||||
|
||||
- one structured line for every dropped or rejected valid frame;
|
||||
- no per-frame INFO spam for successful reports after first frame per VIN/profile unless debug is enabled;
|
||||
- include `vin`, `platformAccount`, `command`, `eventId`, `reasonCode`, and `peer`.
|
||||
|
||||
## Migration Plan
|
||||
|
||||
Phase 1: History foundation
|
||||
|
||||
- Replace Parquet rewrite store with DuckDB hot tables and batch writer.
|
||||
- Keep RAW bytes archive.
|
||||
- Move existing history endpoints onto the new store.
|
||||
- Add diagnostic endpoint for missing VIN/profile cases.
|
||||
|
||||
Phase 2: Ingest association and diagnostics
|
||||
|
||||
- Add local `vin-platform-profile.jsonl` loader.
|
||||
- Use resolved profile before private block parsing.
|
||||
- Add structured parse/reject diagnostics to Kafka metadata and logs.
|
||||
- Add tests for platform login, mapping fallback, and partial private parsing.
|
||||
|
||||
Phase 3: MySQL analytics
|
||||
|
||||
- Add MySQL repository and migrations.
|
||||
- Implement daily stat accumulator with idempotent upsert.
|
||||
- Implement alarm timeline open/update/close logic.
|
||||
- Add integration tests using Testcontainers or a local MySQL profile.
|
||||
|
||||
Phase 4: Production hardening
|
||||
|
||||
- Add backfill/replay commands from DuckDB RAW archive to analytics.
|
||||
- Add retention/export from DuckDB to Parquet for cold storage.
|
||||
- Add dashboards and operational runbook.
|
||||
|
||||
## Acceptance Criteria
|
||||
|
||||
- A valid GB32960 frame received on port 32960 appears in Kafka with RAW bytes or RAW URI.
|
||||
- A valid frame is queryable by `vin + eventTime` from history.
|
||||
- A returned frame can be replayed from `rawArchiveUri` and decoded with the stored profile.
|
||||
- A VIN with only local mapping, and no current platform login, can still use the correct private parser profile.
|
||||
- A private block parse failure does not drop the frame.
|
||||
- Daily MySQL rows update in real time for mileage, total mileage, power consumption, hydrogen consumption, hydrogen rate, and sample metadata.
|
||||
- Alarm timeline rows preserve full alarm start, update, and close history.
|
||||
- Replaying the same Kafka event does not duplicate daily stats or alarm rows.
|
||||
- No secret appears in committed files.
|
||||
|
||||
## References
|
||||
|
||||
- DuckDB Appender: https://duckdb.org/docs/current/data/appender
|
||||
- DuckDB INSERT performance guidance: https://duckdb.org/docs/current/data/insert
|
||||
- MySQL `INSERT ... ON DUPLICATE KEY UPDATE`: https://dev.mysql.com/doc/refman/9.7/en/insert-on-duplicate.html
|
||||
- SAE J2572 overview: https://h2tools.org/fuel-cell-codes-and-standards/sae-j2572-measuring-exhaust-emissions-energy-consumption-and-range
|
||||
@@ -1,411 +0,0 @@
|
||||
> **Superseded:** This 2026-06-23 split design is historical context.
|
||||
> Use `docs/target-architecture.md` and
|
||||
> `docs/superpowers/specs/2026-06-29-vehicle-ingest-redesign.md` for the
|
||||
> current production architecture: TDengine is the default hot history store,
|
||||
> `sink-archive` owns raw bytes, and `event-file-store` is optional
|
||||
> compatibility only.
|
||||
|
||||
# GB32960 Service Split Design
|
||||
|
||||
Date: 2026-06-23
|
||||
|
||||
## Goal
|
||||
|
||||
Split the current all-in-one vehicle ingest runtime into lower-coupled, higher-cohesion services. The first production slice focuses on GB/T 32960 because it is the active high-pressure path: persistent TCP connections, protocol ACKs, raw frame volume, history queries, and downstream analytics are currently assembled in one `bootstrap-all` process.
|
||||
|
||||
The target outcome is:
|
||||
|
||||
- Keep GB32960 ingest reliable and fast.
|
||||
- Move disk writes and history queries out of the protocol process.
|
||||
- Move statistics and analysis out of the protocol process.
|
||||
- Use Kafka as the durable boundary between ingest, history, and analytics.
|
||||
- Preserve `bootstrap-all` during migration as a local development and rollback entry point.
|
||||
|
||||
## Decision
|
||||
|
||||
Use ACK semantics A:
|
||||
|
||||
> A GB32960 frame is ACKed successfully after the protocol service parses/accepts it and writes the required Kafka message(s) successfully.
|
||||
|
||||
The protocol service must not wait for raw archive, DuckDB/Parquet writes, snapshot generation, or statistics computation before sending the GB32960 response frame.
|
||||
|
||||
## Service Boundaries
|
||||
|
||||
### `gb32960-ingest-app`
|
||||
|
||||
Purpose: own GB32960 connection handling and Kafka production.
|
||||
|
||||
Responsibilities:
|
||||
|
||||
- Listen on the GB32960 TCP port.
|
||||
- Handle platform login, VIN authorization, TLS if enabled, and session state required for protocol responses.
|
||||
- Decode GB32960 frames and apply vendor extension parsing.
|
||||
- Produce raw frame records and normalized event records to Kafka.
|
||||
- ACK successful reports only after Kafka production succeeds.
|
||||
- Publish malformed or rejected frames to a DLQ when possible.
|
||||
- Expose operational health and metrics for connections, decode rate, Kafka latency, ACK latency, and DLQ count.
|
||||
|
||||
Non-responsibilities:
|
||||
|
||||
- No local raw archive writes.
|
||||
- No DuckDB or Parquet history writes.
|
||||
- No snapshot query API.
|
||||
- No vehicle statistics or analysis.
|
||||
- No long-running analytical calculations.
|
||||
|
||||
Initial module dependencies:
|
||||
|
||||
- `ingest-api`
|
||||
- `ingest-core`
|
||||
- `ingest-codec-common`
|
||||
- `session-core`
|
||||
- `protocol-gb32960`
|
||||
- `sink-kafka`
|
||||
- `observability`
|
||||
|
||||
### `vehicle-history-app`
|
||||
|
||||
Purpose: own raw storage, historical index storage, and history query APIs.
|
||||
|
||||
Responsibilities:
|
||||
|
||||
- Consume GB32960 raw records and normalized event records from Kafka.
|
||||
- Write raw `.bin` payloads through `sink-archive`.
|
||||
- Write queryable event indexes through `event-file-store`.
|
||||
- Provide APIs for raw frame lookup, decoded frame lookup, history query, and snapshot/source-frame lookup.
|
||||
- Rebuild or backfill the DuckDB sidecar index from stored Parquet files when needed.
|
||||
- Track consumer lag, write latency, archive failures, index failures, and query latency.
|
||||
|
||||
Non-responsibilities:
|
||||
|
||||
- No TCP protocol listener.
|
||||
- No GB32960 ACK decisions.
|
||||
- No online statistics computation.
|
||||
- No direct coupling to `gb32960-ingest-app` internals.
|
||||
|
||||
Initial module dependencies:
|
||||
|
||||
- `ingest-api`
|
||||
- `sink-kafka`
|
||||
- `sink-archive`
|
||||
- `event-file-store`
|
||||
- `event-history-service`
|
||||
- `protocol-gb32960` only for raw-frame decode/query support
|
||||
- `observability`
|
||||
|
||||
### `vehicle-analytics-app`
|
||||
|
||||
Purpose: own vehicle state, daily statistics, alarms, and later analytics.
|
||||
|
||||
Responsibilities:
|
||||
|
||||
- Consume normalized vehicle event records from Kafka.
|
||||
- Maintain latest vehicle state if enabled.
|
||||
- Compute daily vehicle statistics and alarm-derived metrics.
|
||||
- Persist analytical outputs to the selected backend.
|
||||
- Support reprocessing from Kafka offsets when analytical logic changes.
|
||||
- Track consumer lag, per-VIN processing latency, aggregation failures, and output write latency.
|
||||
|
||||
Non-responsibilities:
|
||||
|
||||
- No protocol listener.
|
||||
- No GB32960 ACK decisions.
|
||||
- No raw archive writes.
|
||||
- No normal-path dependency on raw archive or history APIs.
|
||||
|
||||
Initial module dependencies:
|
||||
|
||||
- `ingest-api`
|
||||
- `sink-kafka`
|
||||
- `vehicle-state-service`
|
||||
- `vehicle-stat-service`
|
||||
- `observability`
|
||||
|
||||
## Kafka Contract
|
||||
|
||||
Kafka is the service boundary. Messages must be stable enough that `vehicle-history-app` and `vehicle-analytics-app` do not depend on protocol service internals.
|
||||
|
||||
### Topics
|
||||
|
||||
Use versioned topics:
|
||||
|
||||
- `vehicle.raw.gb32960.v1`
|
||||
- `vehicle.event.gb32960.v1`
|
||||
- `vehicle.dlq.gb32960.v1`
|
||||
|
||||
The current topic names under `lingniu.ingest.sink.kafka.topics` can remain temporarily for compatibility, but the split apps should converge on versioned topic names.
|
||||
|
||||
### Keys
|
||||
|
||||
Use VIN as the Kafka key whenever a VIN is available.
|
||||
|
||||
Benefits:
|
||||
|
||||
- Preserves per-vehicle ordering within a partition.
|
||||
- Lets history and analytics scale by partition.
|
||||
- Keeps stateful analytics simpler.
|
||||
|
||||
When VIN is unavailable, use a stable fallback key such as platform account plus peer address plus receive time bucket.
|
||||
|
||||
### Raw Record
|
||||
|
||||
Topic: `vehicle.raw.gb32960.v1`
|
||||
|
||||
Minimum fields:
|
||||
|
||||
- `schemaVersion`
|
||||
- `protocol = GB32960`
|
||||
- `eventId`
|
||||
- `receiveTime`
|
||||
- `eventTime` if decoded
|
||||
- `peer`
|
||||
- `platformAccount`
|
||||
- `vin` if decoded
|
||||
- `command`
|
||||
- `protocolVersion`
|
||||
- `rawBytes`
|
||||
- `checksumStatus`
|
||||
- `decodeStatus`
|
||||
- `metadata`
|
||||
|
||||
Consumers:
|
||||
|
||||
- `vehicle-history-app` writes archive `.bin` files and raw-frame query indexes.
|
||||
- Future replay tools can regenerate normalized events from raw records.
|
||||
|
||||
### Normalized Event Record
|
||||
|
||||
Topic: `vehicle.event.gb32960.v1`
|
||||
|
||||
Minimum fields:
|
||||
|
||||
- `schemaVersion`
|
||||
- `protocol = GB32960`
|
||||
- `eventId`
|
||||
- `sourceRawEventId`
|
||||
- `receiveTime`
|
||||
- `eventTime`
|
||||
- `platformAccount`
|
||||
- `vin`
|
||||
- `command`
|
||||
- `eventType`
|
||||
- `normalizedFields`
|
||||
- `vendorFields`
|
||||
- `alarmFields`
|
||||
- `metadata`
|
||||
|
||||
Consumers:
|
||||
|
||||
- `vehicle-history-app` writes queryable history records.
|
||||
- `vehicle-analytics-app` updates state and statistics.
|
||||
|
||||
### DLQ Record
|
||||
|
||||
Topic: `vehicle.dlq.gb32960.v1`
|
||||
|
||||
Minimum fields:
|
||||
|
||||
- `schemaVersion`
|
||||
- `stage`
|
||||
- `errorCode`
|
||||
- `errorMessage`
|
||||
- `receiveTime`
|
||||
- `peer`
|
||||
- `platformAccount` if known
|
||||
- `vin` if known
|
||||
- `rawBytes` when available
|
||||
- `metadata`
|
||||
|
||||
Typical stages:
|
||||
|
||||
- `FRAME_DECODE`
|
||||
- `BODY_PARSE`
|
||||
- `AUTH`
|
||||
- `KAFKA_PRODUCE`
|
||||
- `ACK_WRITE`
|
||||
|
||||
## ACK and Failure Semantics
|
||||
|
||||
### Success Path
|
||||
|
||||
1. Receive TCP bytes.
|
||||
2. Decode a complete GB32960 frame.
|
||||
3. Validate checksum/auth/session rules.
|
||||
4. Build raw and normalized Kafka records.
|
||||
5. Produce required Kafka records successfully.
|
||||
6. Send GB32960 success ACK.
|
||||
|
||||
### Kafka Produce Failure
|
||||
|
||||
If required Kafka production fails, the service must not send a success ACK.
|
||||
|
||||
Allowed behavior:
|
||||
|
||||
- Retry within a bounded timeout.
|
||||
- Return a GB32960 failure response when the protocol allows it.
|
||||
- Close the channel after repeated produce failures.
|
||||
- Emit local error logs and metrics.
|
||||
|
||||
Do not silently ACK frames that have not crossed the Kafka durability boundary.
|
||||
|
||||
### History or Analytics Failure
|
||||
|
||||
Failures in `vehicle-history-app` or `vehicle-analytics-app` must not affect GB32960 ACKs. They are handled by Kafka offset retry, DLQ, operational alerting, and backfill/replay.
|
||||
|
||||
## Current Coupling to Remove
|
||||
|
||||
`bootstrap-all` currently assembles protocol modules, Kafka producer/consumer, archive sink, event file store, history API, vehicle state, vehicle statistics, command gateway, and multiple inbound protocols in one application.
|
||||
|
||||
The split should remove these production couplings:
|
||||
|
||||
- Protocol runtime directly containing archive and event-file-store writers.
|
||||
- Protocol runtime directly containing statistics processors.
|
||||
- History query API sharing the same process as TCP ingest.
|
||||
- Kafka consumer workers living in the same all-in-one app as protocol listeners.
|
||||
- Operational scaling tied to one JVM for ingest, storage, queries, and analytics.
|
||||
|
||||
## Migration Plan
|
||||
|
||||
### Phase 1: Add Split App Entrypoints
|
||||
|
||||
Add new app modules:
|
||||
|
||||
- `modules/apps/gb32960-ingest-app`
|
||||
- `modules/apps/vehicle-history-app`
|
||||
- `modules/apps/vehicle-analytics-app`
|
||||
|
||||
Keep `modules/apps/bootstrap-all` unchanged as the fallback runtime.
|
||||
|
||||
### Phase 2: GB32960 Ingest App
|
||||
|
||||
Create a production profile for `gb32960-ingest-app`:
|
||||
|
||||
- Enable `protocol-gb32960`.
|
||||
- Enable Kafka producer.
|
||||
- Disable local archive.
|
||||
- Disable event-file-store.
|
||||
- Disable history API.
|
||||
- Disable vehicle state/stat services.
|
||||
- Disable unrelated protocols.
|
||||
|
||||
Acceptance:
|
||||
|
||||
- GB32960 TCP port starts.
|
||||
- Platform login works.
|
||||
- Realtime report frames are parsed.
|
||||
- Kafka records are produced with VIN keys.
|
||||
- ACK is sent only after Kafka success.
|
||||
|
||||
### Phase 3: History App
|
||||
|
||||
Create `vehicle-history-app`:
|
||||
|
||||
- Enable Kafka consumer.
|
||||
- Enable archive sink.
|
||||
- Enable event-file-store.
|
||||
- Enable event-history HTTP API.
|
||||
- Disable protocol listeners.
|
||||
- Disable analytics processors.
|
||||
|
||||
Acceptance:
|
||||
|
||||
- Consumes `vehicle.raw.gb32960.v1`.
|
||||
- Writes raw `.bin` files.
|
||||
- Consumes `vehicle.event.gb32960.v1`.
|
||||
- Writes Parquet/DuckDB indexes.
|
||||
- Queries can locate raw frames and decoded snapshots.
|
||||
|
||||
### Phase 4: Analytics App
|
||||
|
||||
Create `vehicle-analytics-app`:
|
||||
|
||||
- Enable Kafka consumer.
|
||||
- Enable vehicle state if a state backend is configured.
|
||||
- Enable vehicle statistics.
|
||||
- Disable protocol listeners.
|
||||
- Disable archive and event-file-store.
|
||||
|
||||
Acceptance:
|
||||
|
||||
- Consumes `vehicle.event.gb32960.v1`.
|
||||
- Maintains expected per-VIN state/stat outputs.
|
||||
- Can be restarted and resume from Kafka offsets.
|
||||
- Does not affect ingest ACK latency.
|
||||
|
||||
### Phase 5: Parallel Run and Cutover
|
||||
|
||||
Run old and new paths in parallel for a bounded validation window.
|
||||
|
||||
Compare:
|
||||
|
||||
- Received frame count.
|
||||
- Kafka produced record count.
|
||||
- Raw archive file count.
|
||||
- Unique VIN count.
|
||||
- History query count and sample query correctness.
|
||||
- Analytics output count and sample values.
|
||||
- GB32960 ACK latency before and after split.
|
||||
|
||||
Cut over only after counts and sample queries match within the agreed tolerance.
|
||||
|
||||
## Operational Guidance
|
||||
|
||||
Scale services independently:
|
||||
|
||||
- Scale `gb32960-ingest-app` by connection count and Kafka produce latency.
|
||||
- Scale `vehicle-history-app` by disk throughput, consumer lag, and query latency.
|
||||
- Scale `vehicle-analytics-app` by consumer lag and computation latency.
|
||||
|
||||
Monitor:
|
||||
|
||||
- Kafka producer error rate and p99 produce latency.
|
||||
- ACK p99 latency.
|
||||
- TCP active connections.
|
||||
- Decode failures and DLQ counts.
|
||||
- Consumer lag per group.
|
||||
- Archive write latency and failure rate.
|
||||
- Event-file-store flush latency and failure rate.
|
||||
- Analytics processing latency.
|
||||
|
||||
## Risks and Mitigations
|
||||
|
||||
Risk: Kafka outage blocks GB32960 ACKs.
|
||||
|
||||
Mitigation: bounded producer retry, clear failure ACK/close behavior, Kafka cluster monitoring, and optional local emergency spool only as a later explicit design.
|
||||
|
||||
Risk: Two Kafka topics for raw and normalized events can diverge.
|
||||
|
||||
Mitigation: include `eventId` and `sourceRawEventId`, produce records transactionally if required, or publish a single envelope containing both raw and normalized sections in the first implementation if transaction support is not ready.
|
||||
|
||||
Risk: History app reprocessing duplicates archive/index records.
|
||||
|
||||
Mitigation: deterministic archive keys and idempotent index writes keyed by `eventId` or `rawArchiveUri`.
|
||||
|
||||
Risk: Analytics logic changes require backfill.
|
||||
|
||||
Mitigation: keep raw and normalized topics with sufficient retention; allow analytics consumers to reset offsets or run a backfill group.
|
||||
|
||||
Risk: Query APIs accidentally depend on ingest internals.
|
||||
|
||||
Mitigation: history APIs should depend on `event-file-store`, `sink-archive`, and decode libraries only, not `gb32960-ingest-app`.
|
||||
|
||||
## Open Decisions
|
||||
|
||||
These are intentionally left for implementation planning:
|
||||
|
||||
- Whether Kafka production uses two independent sends or a transaction for raw + normalized records.
|
||||
- Exact protobuf/JSON schema shape for `vehicle.raw.gb32960.v1` and `vehicle.event.gb32960.v1`.
|
||||
- Whether latest snapshot state belongs only to `vehicle-analytics-app` or is also materialized by `vehicle-history-app` for query convenience.
|
||||
- Initial Kafka partition count and retention duration.
|
||||
- Whether command downlink stays with `command-gateway` or gets a separate command service later.
|
||||
|
||||
## Completion Criteria
|
||||
|
||||
The split is complete when:
|
||||
|
||||
- `gb32960-ingest-app`, `vehicle-history-app`, and `vehicle-analytics-app` are separate runnable app modules.
|
||||
- `gb32960-ingest-app` can receive live GB32960 data and ACK after Kafka success.
|
||||
- `vehicle-history-app` can consume Kafka and provide raw/history/snapshot query APIs.
|
||||
- `vehicle-analytics-app` can consume Kafka and compute state/stat outputs.
|
||||
- `bootstrap-all` remains available for development and rollback.
|
||||
- Verification covers build, app startup, Kafka production/consumption, raw archive writes, history queries, analytics outputs, and ACK latency.
|
||||
@@ -1,621 +0,0 @@
|
||||
# Vehicle Ingest Redesign Design
|
||||
|
||||
## Background
|
||||
|
||||
Current GB32960 and JT808 ingestion can receive, decode, publish, and query data, but the responsibilities are split across several partially overlapping paths:
|
||||
|
||||
- Netty handlers produce `RawFrame` and the dispatcher emits both raw archive events and normalized vehicle events.
|
||||
- Kafka envelopes carry raw archive references, while raw bytes are stored separately by the archive sink.
|
||||
- GB32960 historical queries use raw archive records plus file replay decoding.
|
||||
- JT808 location history is materialized separately into TDengine.
|
||||
- DuckDB/Parquet, local raw archive, Kafka, and TDengine each own part of the story, so query and durability semantics are hard to reason about.
|
||||
|
||||
The new design intentionally keeps useful protocol parsers and app deployment knowledge, but replaces the ingestion, durability, storage, and query boundaries with a first-principles model.
|
||||
|
||||
## Goals
|
||||
|
||||
1. Store every original inbound protocol frame, including malformed frames when bytes are available.
|
||||
2. Query historical uploaded data for GB32960 and JT808 by vehicle, protocol, message type, and time range.
|
||||
3. Extend to new fields, protocol messages, and vendor profiles without rewriting the core pipeline.
|
||||
4. Sustain production load for 10,000 vehicles and 1,000 concurrent historical query requests per second.
|
||||
5. Make ACK and durability rules explicit enough that an acknowledged frame is not silently lost.
|
||||
|
||||
## Non-Goals
|
||||
|
||||
- Do not make Kafka the long-term historical query database.
|
||||
- Do not store large raw byte arrays directly in Kafka topics.
|
||||
- Do not keep DuckDB/Parquet as the production hot query store for GB32960 and JT808.
|
||||
- Do not build one generic API that hides protocol-specific semantics. Shared internals are preferred; external APIs may remain protocol-aware.
|
||||
|
||||
## Recommended Architecture
|
||||
|
||||
Use `Raw Archive + Kafka + TDengine` as the core architecture.
|
||||
|
||||
```mermaid
|
||||
flowchart LR
|
||||
GB["GB32960 TCP"] --> Edge["Protocol Edge"]
|
||||
JT["JT808 TCP"] --> Edge
|
||||
Edge --> RawStore["Raw Archive"]
|
||||
RawStore --> RawTopic["Kafka vehicle.raw-frame.v1"]
|
||||
Edge --> Decode["Protocol Decoder"]
|
||||
Decode --> FactTopic["Kafka vehicle.decoded-fact.v1"]
|
||||
RawTopic --> Writer["History Writer"]
|
||||
FactTopic --> Writer
|
||||
Writer --> TD["TDengine"]
|
||||
Writer --> Redis["Realtime State"]
|
||||
API["History API"] --> TD
|
||||
API --> RawStore
|
||||
RealtimeAPI["Realtime API"] --> Redis
|
||||
```
|
||||
|
||||
The architecture has three facts:
|
||||
|
||||
- `RawFrameFact`: one row per inbound frame, whether parsing succeeds or fails.
|
||||
- `DecodedFact`: zero or more normalized facts derived from a raw frame.
|
||||
- `RawArchiveObject`: immutable raw bytes addressed by `raw_uri`.
|
||||
|
||||
The raw archive is the forensic source of truth. TDengine is the query source of truth. Kafka is the decoupling and replay pipe.
|
||||
|
||||
## Module Boundaries
|
||||
|
||||
### Protocol Edge Apps
|
||||
|
||||
Apps:
|
||||
|
||||
- `gb32960-ingest-app`
|
||||
- `jt808-ingest-app`
|
||||
|
||||
Responsibilities:
|
||||
|
||||
- Accept TCP connections.
|
||||
- Split frames.
|
||||
- Perform protocol-level validation, authentication, session binding, and ACK.
|
||||
- Create `RawFrameFact`.
|
||||
- Persist raw bytes through `RawArchiveWriter`.
|
||||
- Publish raw frame fact to Kafka.
|
||||
- Decode frames and publish decoded facts.
|
||||
|
||||
They do not query history, export data, or write business tables directly.
|
||||
|
||||
### Ingest Core
|
||||
|
||||
New module:
|
||||
|
||||
- `modules/core/ingest-facts`
|
||||
|
||||
Core types:
|
||||
|
||||
```java
|
||||
public record RawFrameFact(
|
||||
String frameId,
|
||||
ProtocolId protocol,
|
||||
String vehicleKey,
|
||||
String vin,
|
||||
String phone,
|
||||
int messageId,
|
||||
int subType,
|
||||
Instant eventTime,
|
||||
Instant receivedAt,
|
||||
String peer,
|
||||
String rawUri,
|
||||
String checksum,
|
||||
long rawSizeBytes,
|
||||
ParseStatus parseStatus,
|
||||
String parseError,
|
||||
Map<String, String> metadata
|
||||
) {}
|
||||
```
|
||||
|
||||
```java
|
||||
public sealed interface DecodedFact permits
|
||||
DecodedFact.Location,
|
||||
DecodedFact.Realtime,
|
||||
DecodedFact.Alarm,
|
||||
DecodedFact.Session,
|
||||
DecodedFact.Register,
|
||||
DecodedFact.Extension {
|
||||
|
||||
String factId();
|
||||
String frameId();
|
||||
ProtocolId protocol();
|
||||
String vehicleKey();
|
||||
String vin();
|
||||
Instant eventTime();
|
||||
Instant receivedAt();
|
||||
String rawUri();
|
||||
Map<String, String> metadata();
|
||||
}
|
||||
```
|
||||
|
||||
`vehicleKey` is the stable partition key:
|
||||
|
||||
- GB32960: VIN.
|
||||
- JT808: resolved VIN when known, otherwise `jt808:<phone>`.
|
||||
- Unknown/malformed: `unknown:<protocol>:<hash>`.
|
||||
|
||||
### Raw Archive
|
||||
|
||||
New or refactored module:
|
||||
|
||||
- `modules/sinks/raw-archive-store`
|
||||
|
||||
Interface:
|
||||
|
||||
```java
|
||||
public interface RawArchiveWriter {
|
||||
RawArchiveReceipt write(RawArchiveWriteRequest request) throws IOException;
|
||||
}
|
||||
```
|
||||
|
||||
Receipt:
|
||||
|
||||
```java
|
||||
public record RawArchiveReceipt(
|
||||
String rawUri,
|
||||
String checksum,
|
||||
long sizeBytes
|
||||
) {}
|
||||
```
|
||||
|
||||
Rules:
|
||||
|
||||
- Raw bytes are immutable once written.
|
||||
- Write path is content-addressed or deterministic by `protocol/date/vehicleKey/frameId.bin`.
|
||||
- Write must be fsync-capable for local storage and pluggable for object storage.
|
||||
- `rawUri` must be resolvable by history API for single-frame replay.
|
||||
- Malformed frames are written with `parseStatus=FAILED` when bytes are available.
|
||||
|
||||
Deployment constraints:
|
||||
|
||||
- Raw archive, TDengine, and the history service may run on different ECS instances, but should use private-network access in the same VPC and availability zone.
|
||||
- High-frequency historical queries only read TDengine and must not read raw archive.
|
||||
- Raw archive is only used by single-frame detail, troubleshooting, replay, and asynchronous export paths.
|
||||
- `rawUri` must be globally resolvable by services and must not be a private local absolute path on one ECS instance.
|
||||
- The first local raw archive implementation must stay behind a replaceable interface so it can later move to OSS, MinIO, or another object store.
|
||||
|
||||
### Kafka Topics
|
||||
|
||||
Use stable versioned topics:
|
||||
|
||||
- `vehicle.raw-frame.v1`
|
||||
- `vehicle.decoded-fact.v1`
|
||||
- `vehicle.dlq.v1`
|
||||
|
||||
Kafka key:
|
||||
|
||||
```text
|
||||
<protocol>:<vehicleKey>
|
||||
```
|
||||
|
||||
`vehicle.raw-frame.v1` payload contains `RawFrameFact` without raw bytes. It carries `rawUri`, checksum, byte size, message id, parse status, and metadata.
|
||||
|
||||
`vehicle.decoded-fact.v1` payload contains one `DecodedFact`. It always references `frameId` and `rawUri`.
|
||||
|
||||
`vehicle.dlq.v1` carries failed store, publish, decode, and write attempts with enough metadata to replay from raw archive when possible.
|
||||
|
||||
### History Writer
|
||||
|
||||
New app or refactored app:
|
||||
|
||||
- `vehicle-history-app`
|
||||
|
||||
Responsibilities:
|
||||
|
||||
- Consume `vehicle.raw-frame.v1`.
|
||||
- Consume `vehicle.decoded-fact.v1`.
|
||||
- Batch write TDengine.
|
||||
- Update realtime state store for latest vehicle state.
|
||||
- Write DLQ on invalid payloads or storage failures.
|
||||
|
||||
The writer is idempotent by `(frame_id)` for raw facts and `(fact_id)` for decoded facts.
|
||||
|
||||
## TDengine Model
|
||||
|
||||
TDengine is the production historical query store.
|
||||
|
||||
### Raw Frames Super Table
|
||||
|
||||
```sql
|
||||
CREATE STABLE IF NOT EXISTS raw_frames (
|
||||
ts TIMESTAMP,
|
||||
frame_id NCHAR(64),
|
||||
received_at TIMESTAMP,
|
||||
message_id INT,
|
||||
sub_type INT,
|
||||
event_time TIMESTAMP,
|
||||
raw_uri NCHAR(512),
|
||||
checksum NCHAR(128),
|
||||
raw_size_bytes BIGINT,
|
||||
parse_status NCHAR(16),
|
||||
parse_error NCHAR(512),
|
||||
peer NCHAR(128),
|
||||
metadata_json NCHAR(4096)
|
||||
) TAGS (
|
||||
protocol NCHAR(16),
|
||||
vehicle_key NCHAR(128),
|
||||
vin NCHAR(64),
|
||||
phone NCHAR(32)
|
||||
);
|
||||
```
|
||||
|
||||
Child table:
|
||||
|
||||
```text
|
||||
raw_<protocol>_<hash(vehicleKey)>
|
||||
```
|
||||
|
||||
Primary query patterns:
|
||||
|
||||
- Raw frames by vehicle and time.
|
||||
- Raw frames by protocol, message id, parse status, and time.
|
||||
- Single raw frame by `frame_id` or `raw_uri`.
|
||||
|
||||
### Location Super Table
|
||||
|
||||
```sql
|
||||
CREATE STABLE IF NOT EXISTS vehicle_locations (
|
||||
ts TIMESTAMP,
|
||||
fact_id NCHAR(64),
|
||||
frame_id NCHAR(64),
|
||||
received_at TIMESTAMP,
|
||||
longitude DOUBLE,
|
||||
latitude DOUBLE,
|
||||
altitude_m DOUBLE,
|
||||
speed_kmh DOUBLE,
|
||||
direction_deg DOUBLE,
|
||||
alarm_flag BIGINT,
|
||||
status_flag BIGINT,
|
||||
total_mileage_km DOUBLE,
|
||||
raw_uri NCHAR(512),
|
||||
metadata_json NCHAR(4096)
|
||||
) TAGS (
|
||||
protocol NCHAR(16),
|
||||
vehicle_key NCHAR(128),
|
||||
vin NCHAR(64),
|
||||
phone NCHAR(32)
|
||||
);
|
||||
```
|
||||
|
||||
### Realtime Super Table
|
||||
|
||||
```sql
|
||||
CREATE STABLE IF NOT EXISTS vehicle_realtime (
|
||||
ts TIMESTAMP,
|
||||
fact_id NCHAR(64),
|
||||
frame_id NCHAR(64),
|
||||
received_at TIMESTAMP,
|
||||
speed_kmh DOUBLE,
|
||||
total_mileage_km DOUBLE,
|
||||
battery_soc DOUBLE,
|
||||
total_voltage_v DOUBLE,
|
||||
total_current_a DOUBLE,
|
||||
running_mode NCHAR(64),
|
||||
vehicle_state NCHAR(64),
|
||||
charging_state NCHAR(64),
|
||||
raw_uri NCHAR(512),
|
||||
fields_json NCHAR(8192),
|
||||
metadata_json NCHAR(4096)
|
||||
) TAGS (
|
||||
protocol NCHAR(16),
|
||||
vehicle_key NCHAR(128),
|
||||
vin NCHAR(64),
|
||||
phone NCHAR(32)
|
||||
);
|
||||
```
|
||||
|
||||
`fields_json` stores sparse full-field values for low-frequency inspection. High-frequency queried fields should be promoted to explicit columns or a dedicated extension table.
|
||||
|
||||
### Alarm Super Table
|
||||
|
||||
```sql
|
||||
CREATE STABLE IF NOT EXISTS vehicle_alarms (
|
||||
ts TIMESTAMP,
|
||||
fact_id NCHAR(64),
|
||||
frame_id NCHAR(64),
|
||||
received_at TIMESTAMP,
|
||||
alarm_level NCHAR(32),
|
||||
alarm_code BIGINT,
|
||||
alarm_name NCHAR(128),
|
||||
longitude DOUBLE,
|
||||
latitude DOUBLE,
|
||||
raw_uri NCHAR(512),
|
||||
fields_json NCHAR(8192),
|
||||
metadata_json NCHAR(4096)
|
||||
) TAGS (
|
||||
protocol NCHAR(16),
|
||||
vehicle_key NCHAR(128),
|
||||
vin NCHAR(64),
|
||||
phone NCHAR(32)
|
||||
);
|
||||
```
|
||||
|
||||
### Session and Registration Tables
|
||||
|
||||
```sql
|
||||
CREATE STABLE IF NOT EXISTS vehicle_sessions (
|
||||
ts TIMESTAMP,
|
||||
fact_id NCHAR(64),
|
||||
frame_id NCHAR(64),
|
||||
received_at TIMESTAMP,
|
||||
session_type NCHAR(32),
|
||||
result_code INT,
|
||||
raw_uri NCHAR(512),
|
||||
fields_json NCHAR(8192),
|
||||
metadata_json NCHAR(4096)
|
||||
) TAGS (
|
||||
protocol NCHAR(16),
|
||||
vehicle_key NCHAR(128),
|
||||
vin NCHAR(64),
|
||||
phone NCHAR(32)
|
||||
);
|
||||
```
|
||||
|
||||
JT808 registration details go into `vehicle_sessions` first and may later be projected into MySQL identity bindings for durable identity lookup.
|
||||
|
||||
## ACK and Durability Rules
|
||||
|
||||
For frames that require a protocol ACK:
|
||||
|
||||
1. Frame bytes are accepted from Netty.
|
||||
2. Raw bytes are written to raw archive.
|
||||
3. `RawFrameFact` is published to Kafka.
|
||||
4. Protocol-specific ACK is sent.
|
||||
5. Decoding and `DecodedFact` publication may run before or after ACK, but failures must emit a DLQ event and update raw frame parse status.
|
||||
|
||||
For GB32960 realtime/report frames, this prevents acknowledging a frame whose original bytes cannot be recovered.
|
||||
|
||||
For JT808 register/auth/heartbeat, ACK follows the same raw archive and raw fact boundary. Register ACK can still include token generation before raw fact publish, but must not be flushed to the channel until the durability boundary succeeds.
|
||||
|
||||
If raw archive write fails, close or keep the channel according to protocol tolerance, but do not ACK success.
|
||||
|
||||
If Kafka raw fact publish fails, do not ACK success for durable-ACK frames. Non-durable frames go to local DLQ when configured.
|
||||
|
||||
## Protocol Decoding and Extension
|
||||
|
||||
Use a plugin contract:
|
||||
|
||||
```java
|
||||
public interface ProtocolFrameDecoder {
|
||||
ProtocolId protocol();
|
||||
DecodeResult decode(RawFrameFact rawFact, byte[] rawBytes);
|
||||
}
|
||||
```
|
||||
|
||||
```java
|
||||
public record DecodeResult(
|
||||
ParseStatus status,
|
||||
List<DecodedFact> facts,
|
||||
String errorMessage,
|
||||
Map<String, String> metadata
|
||||
) {}
|
||||
```
|
||||
|
||||
GB32960:
|
||||
|
||||
- Frame command maps to session, realtime, location, alarm, and extension facts.
|
||||
- Vendor extensions are selected by platform account, VIN, or configured profile.
|
||||
- Full raw replay remains possible by reading `rawUri` and running the current decoder.
|
||||
|
||||
JT808:
|
||||
|
||||
- Message id maps to register, auth, heartbeat, location, batch location, media metadata, and extension facts.
|
||||
- Identity resolution updates `vehicleKey`.
|
||||
- Registration facts carry province, city, maker, device id, plate, plate color, and auth token metadata.
|
||||
|
||||
New protocol messages do not require TDengine schema changes unless they become high-frequency query dimensions. Unknown fields are stored as extension facts with `fields_json`.
|
||||
|
||||
## Query APIs
|
||||
|
||||
Keep API surfaces explicit.
|
||||
|
||||
Raw frame APIs:
|
||||
|
||||
- `GET /api/history/raw-frames`
|
||||
- Filters: `protocol`, `vin`, `phone`, `vehicleKey`, `messageId`, `parseStatus`, `dateFrom`, `dateTo`, `pageSize`, `cursor`.
|
||||
- Returns frame metadata, raw URI, checksum, parse status, and compact decoded summary when available.
|
||||
|
||||
Single raw frame API:
|
||||
|
||||
- `GET /api/history/raw-frames/{frameId}`
|
||||
- Returns raw metadata and decoded detail.
|
||||
- Supports `includeRawHex=true` with a strict size limit.
|
||||
|
||||
Location APIs:
|
||||
|
||||
- `GET /api/history/locations`
|
||||
- Filters: `protocol`, `vin`, `phone`, `vehicleKey`, time range, cursor.
|
||||
|
||||
Realtime APIs:
|
||||
|
||||
- `GET /api/history/realtime`
|
||||
- Historical time-series query.
|
||||
|
||||
Realtime state APIs:
|
||||
|
||||
- `GET /api/realtime/vehicles/{vehicleKey}`
|
||||
- Reads Redis latest state, not TDengine.
|
||||
|
||||
Export:
|
||||
|
||||
- Export is asynchronous.
|
||||
- Request creates an export job.
|
||||
- Worker streams TDengine query results to object storage or local export files.
|
||||
- API returns job status and download URI.
|
||||
|
||||
## Pagination
|
||||
|
||||
Use cursor pagination, not deep offset, for high-QPS history queries.
|
||||
|
||||
Cursor fields:
|
||||
|
||||
```text
|
||||
ts, received_at, fact_id
|
||||
```
|
||||
|
||||
Descending query predicate:
|
||||
|
||||
```sql
|
||||
WHERE (
|
||||
ts < :cursorTs
|
||||
OR (ts = :cursorTs AND received_at < :cursorReceivedAt)
|
||||
OR (ts = :cursorTs AND received_at = :cursorReceivedAt AND fact_id < :cursorFactId)
|
||||
)
|
||||
```
|
||||
|
||||
Page size defaults to 100 and is capped at 1000.
|
||||
|
||||
## Capacity Design
|
||||
|
||||
Assumptions:
|
||||
|
||||
- 10,000 vehicles.
|
||||
- Normal report interval: 5 to 10 seconds.
|
||||
- Expected writes: 1,000 to 2,000 frames per second.
|
||||
- Peak burst factor: 3x.
|
||||
- Historical query target: 1,000 requests per second.
|
||||
|
||||
Write path:
|
||||
|
||||
- Netty worker threads stay CPU-light.
|
||||
- Raw archive writes use bounded async workers.
|
||||
- Kafka producer uses compression, batching, idempotence, and vehicle-key partitioning.
|
||||
- History writer batches TDengine inserts by super table and child table.
|
||||
- TDengine child tables are partitioned by vehicle key to make single-vehicle queries cheap.
|
||||
|
||||
Read path:
|
||||
|
||||
- Most high-QPS queries require vehicle key and time range.
|
||||
- Unbounded cross-vehicle queries are admin-only and capped.
|
||||
- Realtime latest state is served from Redis latest-state cache.
|
||||
- Raw detail replay reads one raw object and decodes on demand.
|
||||
- Common dictionary and metadata responses are cached in memory.
|
||||
|
||||
Backpressure:
|
||||
|
||||
- Raw archive worker queue has a fixed bound.
|
||||
- Kafka publish futures are tracked.
|
||||
- If durability boundary cannot complete within timeout, durable ACK fails and the channel is closed or throttled.
|
||||
- History writer lag is observable by Kafka consumer lag and TDengine batch latency.
|
||||
|
||||
## Migration Plan
|
||||
|
||||
Phase 1: Build new fact model beside existing code.
|
||||
|
||||
- Add `ingest-facts`.
|
||||
- Add raw archive receipt contract.
|
||||
- Add serializers for raw frame and decoded fact Kafka messages.
|
||||
- Add tests for stable ids, vehicle key, and raw URI generation.
|
||||
|
||||
Phase 2: Harden raw durability boundary.
|
||||
|
||||
- Refactor GB32960 and JT808 handlers so ACK waits for raw archive and raw fact publish.
|
||||
- Keep existing parsers.
|
||||
- Emit decoded facts to the new Kafka topic.
|
||||
|
||||
Phase 3: Build TDengine writer.
|
||||
|
||||
- Create TDengine schema manager.
|
||||
- Write raw frame facts.
|
||||
- Write location, realtime, alarm, session facts.
|
||||
- Add idempotent insert behavior by stable ids.
|
||||
|
||||
Phase 4: Build query APIs.
|
||||
|
||||
- Raw frame query.
|
||||
- Single frame replay.
|
||||
- Location history query.
|
||||
- Realtime history query.
|
||||
- Latest realtime state query.
|
||||
|
||||
Phase 5: Cut over and retire old hot paths.
|
||||
|
||||
- Stop using DuckDB/Parquet as production hot history for GB32960/JT808.
|
||||
- Keep raw replay compatibility while historical data migrates.
|
||||
- Remove protocol-specific ad hoc history storage once TDengine coverage is verified.
|
||||
|
||||
## Testing Strategy
|
||||
|
||||
Unit tests:
|
||||
|
||||
- `RawFrameFact` validation and vehicle key derivation.
|
||||
- Raw archive path generation and checksum.
|
||||
- GB32960 decoder facts from golden frames.
|
||||
- JT808 decoder facts from sample frames.
|
||||
- ACK boundary success and failure cases.
|
||||
- TDengine SQL generation.
|
||||
|
||||
Integration tests:
|
||||
|
||||
- Start app with embedded or mocked Kafka producer and fake raw archive.
|
||||
- Send GB32960 sample frame and verify raw fact, decoded fact, raw archive receipt.
|
||||
- Send JT808 register/auth/location frames and verify registration/session/location facts.
|
||||
- Simulate raw archive failure and verify no success ACK.
|
||||
- Simulate Kafka publish failure and verify durable ACK failure.
|
||||
|
||||
Storage tests:
|
||||
|
||||
- TDengine schema initialization.
|
||||
- Batch insert raw frames and decoded facts.
|
||||
- Query by vehicle key and time range.
|
||||
- Cursor pagination correctness.
|
||||
- Single raw frame replay by `rawUri`.
|
||||
|
||||
Load tests:
|
||||
|
||||
- 2,000 frames/s sustained write for at least 30 minutes.
|
||||
- 6,000 frames/s burst for 5 minutes.
|
||||
- 1,000 QPS location/realtime history queries with bounded p95 latency.
|
||||
- Verify no raw frame count gap between raw archive receipts, Kafka raw facts, and TDengine raw frame rows.
|
||||
|
||||
## Operational Requirements
|
||||
|
||||
Metrics:
|
||||
|
||||
- TCP connections.
|
||||
- Frame receive rate.
|
||||
- Raw archive write latency and failure count.
|
||||
- Kafka publish latency and failure count.
|
||||
- Decoder success/failure count by protocol and message id.
|
||||
- TDengine batch size, latency, and failure count.
|
||||
- Query QPS and latency by endpoint.
|
||||
- Kafka consumer lag.
|
||||
|
||||
Logs:
|
||||
|
||||
- One structured log per failed frame with `frameId`, protocol, vehicle key, message id, raw URI, and error.
|
||||
- No full raw bytes in normal logs.
|
||||
- Raw hex only in explicit debug tools with size limits.
|
||||
|
||||
Alerts:
|
||||
|
||||
- Raw archive write failure.
|
||||
- Durable ACK timeout.
|
||||
- Kafka producer failure.
|
||||
- History writer lag.
|
||||
- TDengine write failure.
|
||||
- Raw archive count and TDengine raw frame row count divergence.
|
||||
|
||||
## Acceptance Criteria
|
||||
|
||||
The redesign is complete only when all of the following are true:
|
||||
|
||||
1. Every accepted GB32960 and JT808 frame creates a raw archive object and a TDengine `raw_frames` row.
|
||||
2. Malformed frames with bytes are queryable from raw history with parse status and error reason.
|
||||
3. GB32960 and JT808 location history can be queried by vehicle and time range with cursor pagination.
|
||||
4. GB32960 realtime history can be queried by vehicle and time range.
|
||||
5. JT808 registration/session data is queryable and can update identity binding.
|
||||
6. Single-frame detail can reload raw bytes by `rawUri` and decode with the current decoder.
|
||||
7. ACK-required frames do not receive success ACK before raw archive and raw fact publication succeed.
|
||||
8. New protocol fields can be added by implementing a decoder/projector and, only when needed, a TDengine table migration.
|
||||
9. Load tests demonstrate sustained 2,000 frames/s writes and 1,000 QPS bounded historical queries.
|
||||
10. Old DuckDB/Parquet hot-history paths are no longer required for GB32960/JT808 production queries.
|
||||
|
||||
## Open Decisions
|
||||
|
||||
The following choices are intentionally fixed for the first implementation:
|
||||
|
||||
- Use local raw archive first, with the interface shaped for object storage later.
|
||||
- Use TDengine as the only production hot historical query store for GB32960/JT808.
|
||||
- Use explicit protocol-aware query APIs instead of a single generic query API.
|
||||
- Use cursor pagination for all high-QPS historical queries.
|
||||
- Keep Kafka payloads byte-light by sending raw references instead of raw bytes.
|
||||
@@ -1,624 +0,0 @@
|
||||
# 车辆接入重设计方案
|
||||
|
||||
## 背景
|
||||
|
||||
当前 GB32960 和 JT808 链路已经能够完成接收、解析、发布和查询，但职责边界被拆散在多条路径里：
|
||||
|
||||
- Netty Handler 生成 `RawFrame`，Dispatcher 同时发出原始归档事件和标准化车辆事件。
|
||||
- Kafka Envelope 只携带 raw archive 引用，原始 bytes 由 archive sink 另行存储。
|
||||
- GB32960 历史查询依赖 raw archive 记录，再回读文件重新解析。
|
||||
- JT808 位置历史又单独物化到 TDengine。
|
||||
- DuckDB/Parquet、本地 raw archive、Kafka、TDengine 分别承担一部分职责，导致查询语义和持久化边界不好推理。
|
||||
|
||||
新设计保留已有协议解析器和部署经验，但重新定义接入、持久化、存储和查询边界。
|
||||
|
||||
## 目标
|
||||
|
||||
1. 保存每一条入站协议原始帧，包括可拿到 bytes 的异常帧。
|
||||
2. 支持按车辆、协议、消息类型、时间范围查询 GB32960 和 JT808 历史上报数据。
|
||||
3. 新增字段、协议消息、厂商扩展时，不需要重写核心管线。
|
||||
4. 支撑 10,000 辆车生产写入，以及 1,000 QPS 历史查询。
|
||||
5. 明确 ACK 和持久化规则，避免“已经 ACK 但数据悄悄丢失”。
|
||||
|
||||
## 非目标
|
||||
|
||||
- 不把 Kafka 作为长期历史查询数据库。
|
||||
- 不把大块原始 bytes 直接塞进 Kafka topic。
|
||||
- 不继续把 DuckDB/Parquet 作为 GB32960/JT808 的生产热查询库。
|
||||
- 不做一个掩盖协议差异的万能接口。内部可以统一，外部 API 保持协议语义清晰。
|
||||
|
||||
## 推荐架构
|
||||
|
||||
采用 `Raw Archive + Kafka + TDengine` 作为核心架构。
|
||||
|
||||
```mermaid
|
||||
flowchart LR
|
||||
GB["GB32960 TCP"] --> Edge["协议接入边界"]
|
||||
JT["JT808 TCP"] --> Edge
|
||||
Edge --> RawStore["Raw Archive"]
|
||||
RawStore --> RawTopic["Kafka vehicle.raw-frame.v1"]
|
||||
Edge --> Decode["协议解析器"]
|
||||
Decode --> FactTopic["Kafka vehicle.decoded-fact.v1"]
|
||||
RawTopic --> Writer["History Writer"]
|
||||
FactTopic --> Writer
|
||||
Writer --> TD["TDengine"]
|
||||
Writer --> Redis["实时状态"]
|
||||
API["历史 API"] --> TD
|
||||
API --> RawStore
|
||||
RealtimeAPI["实时 API"] --> Redis
|
||||
```
|
||||
|
||||
架构里只有三类事实：
|
||||
|
||||
- `RawFrameFact`：每一条入站帧一条记录，不管解析成功还是失败。
|
||||
- `DecodedFact`：从 raw frame 派生出来的零条或多条标准事实。
|
||||
- `RawArchiveObject`：不可变原始 bytes，通过 `raw_uri` 寻址。
|
||||
|
||||
raw archive 是排障和回放的事实源。TDengine 是查询事实源。Kafka 是解耦和重放管道。
|
||||
|
||||
## 模块边界
|
||||
|
||||
### 协议接入 APP
|
||||
|
||||
APP：
|
||||
|
||||
- `gb32960-ingest-app`
|
||||
- `jt808-ingest-app`
|
||||
|
||||
职责：
|
||||
|
||||
- 接收 TCP 连接。
|
||||
- 完成帧切分。
|
||||
- 执行协议层校验、鉴权、会话绑定和 ACK。
|
||||
- 创建 `RawFrameFact`。
|
||||
- 通过 `RawArchiveWriter` 持久化原始 bytes。
|
||||
- 发布 raw frame fact 到 Kafka。
|
||||
- 解析协议帧并发布 decoded facts。
|
||||
|
||||
接入 APP 不负责历史查询、导出，也不直接写业务查询表。
|
||||
|
||||
### Ingest Core
|
||||
|
||||
新增模块：
|
||||
|
||||
- `modules/core/ingest-facts`
|
||||
|
||||
核心类型：
|
||||
|
||||
```java
|
||||
public record RawFrameFact(
|
||||
String frameId,
|
||||
ProtocolId protocol,
|
||||
String vehicleKey,
|
||||
String vin,
|
||||
String phone,
|
||||
int messageId,
|
||||
int subType,
|
||||
Instant eventTime,
|
||||
Instant receivedAt,
|
||||
String peer,
|
||||
String rawUri,
|
||||
String checksum,
|
||||
long rawSizeBytes,
|
||||
ParseStatus parseStatus,
|
||||
String parseError,
|
||||
Map<String, String> metadata
|
||||
) {}
|
||||
```
|
||||
|
||||
```java
|
||||
public sealed interface DecodedFact permits
|
||||
DecodedFact.Location,
|
||||
DecodedFact.Realtime,
|
||||
DecodedFact.Alarm,
|
||||
DecodedFact.Session,
|
||||
DecodedFact.Register,
|
||||
DecodedFact.Extension {
|
||||
|
||||
String factId();
|
||||
String frameId();
|
||||
ProtocolId protocol();
|
||||
String vehicleKey();
|
||||
String vin();
|
||||
Instant eventTime();
|
||||
Instant receivedAt();
|
||||
String rawUri();
|
||||
Map<String, String> metadata();
|
||||
}
|
||||
```
|
||||
|
||||
`vehicleKey` 是稳定分区键：
|
||||
|
||||
- GB32960：VIN。
|
||||
- JT808：已解析到 VIN 时使用 VIN，否则使用 `jt808:<phone>`。
|
||||
- 未知或异常帧：使用 `unknown:<protocol>:<hash>`。
|
||||
|
||||
### Raw Archive
|
||||
|
||||
新增或重构模块：
|
||||
|
||||
- `modules/sinks/raw-archive-store`
|
||||
|
||||
接口：
|
||||
|
||||
```java
|
||||
public interface RawArchiveWriter {
|
||||
RawArchiveReceipt write(RawArchiveWriteRequest request) throws IOException;
|
||||
}
|
||||
```
|
||||
|
||||
写入回执：
|
||||
|
||||
```java
|
||||
public record RawArchiveReceipt(
|
||||
String rawUri,
|
||||
String checksum,
|
||||
long sizeBytes
|
||||
) {}
|
||||
```
|
||||
|
||||
规则：
|
||||
|
||||
- 原始 bytes 一旦写入，不允许原地修改。
|
||||
- 写入路径可以按内容寻址，也可以按 `protocol/date/vehicleKey/frameId.bin` 生成。
|
||||
- 本地存储需要支持 fsync 能力，接口要预留对象存储实现。
|
||||
- `rawUri` 必须能被历史 API 解析，用于单帧回放。
|
||||
- 异常帧只要有 bytes，也必须写入，并记录 `parseStatus=FAILED`。
|
||||
|
||||
部署约束：
|
||||
|
||||
- Raw archive、TDengine、history 服务可以不在同一台 ECS，但应在同一 VPC 和同一可用区内通过内网访问。
|
||||
- 高频历史查询只访问 TDengine，不访问 raw archive。
|
||||
- Raw archive 只进入单帧详情、问题排查、回放和异步导出路径。
|
||||
- `rawUri` 必须是全局可解析地址，不能是某台 ECS 私有的本地绝对路径。
|
||||
- 第一版本地 raw archive 需要通过可替换接口封装，后续可以切换到 OSS、MinIO 或其它对象存储。
|
||||
|
||||
### Kafka Topic
|
||||
|
||||
使用稳定的版本化 topic：
|
||||
|
||||
- `vehicle.raw-frame.v1`
|
||||
- `vehicle.decoded-fact.v1`
|
||||
- `vehicle.dlq.v1`
|
||||
|
||||
Kafka key：
|
||||
|
||||
```text
|
||||
<protocol>:<vehicleKey>
|
||||
```
|
||||
|
||||
`vehicle.raw-frame.v1` 的 payload 是不含 raw bytes 的 `RawFrameFact`，携带 `rawUri`、checksum、byte size、message id、parse status 和 metadata。
|
||||
|
||||
`vehicle.decoded-fact.v1` 的 payload 是单条 `DecodedFact`，必须引用 `frameId` 和 `rawUri`。
|
||||
|
||||
`vehicle.dlq.v1` 保存存储失败、发布失败、解析失败和写入失败事件，并携带足够元数据，尽量支持从 raw archive 重放。
|
||||
|
||||
### History Writer
|
||||
|
||||
新增或重构 APP：
|
||||
|
||||
- `vehicle-history-app`
|
||||
|
||||
职责：
|
||||
|
||||
- 消费 `vehicle.raw-frame.v1`。
|
||||
- 消费 `vehicle.decoded-fact.v1`。
|
||||
- 批量写 TDengine。
|
||||
- 更新实时状态存储。
|
||||
- 对非法 payload 或存储失败写 DLQ。
|
||||
|
||||
Writer 写入需要幂等：
|
||||
|
||||
- raw fact 以 `(frame_id)` 幂等。
|
||||
- decoded fact 以 `(fact_id)` 幂等。
|
||||
|
||||
## TDengine 模型
|
||||
|
||||
TDengine 是生产历史查询主库。
|
||||
|
||||
### 原始帧超级表
|
||||
|
||||
```sql
|
||||
CREATE STABLE IF NOT EXISTS raw_frames (
|
||||
ts TIMESTAMP,
|
||||
frame_id NCHAR(64),
|
||||
received_at TIMESTAMP,
|
||||
message_id INT,
|
||||
sub_type INT,
|
||||
event_time TIMESTAMP,
|
||||
raw_uri NCHAR(512),
|
||||
checksum NCHAR(128),
|
||||
raw_size_bytes BIGINT,
|
||||
parse_status NCHAR(16),
|
||||
parse_error NCHAR(512),
|
||||
peer NCHAR(128),
|
||||
metadata_json NCHAR(4096)
|
||||
) TAGS (
|
||||
protocol NCHAR(16),
|
||||
vehicle_key NCHAR(128),
|
||||
vin NCHAR(64),
|
||||
phone NCHAR(32)
|
||||
);
|
||||
```
|
||||
|
||||
子表命名：
|
||||
|
||||
```text
|
||||
raw_<protocol>_<hash(vehicleKey)>
|
||||
```
|
||||
|
||||
主要查询模式：
|
||||
|
||||
- 按车辆和时间查询原始帧。
|
||||
- 按协议、消息 ID、解析状态和时间查询原始帧。
|
||||
- 按 `frame_id` 或 `raw_uri` 查询单帧。
|
||||
|
||||
### 位置超级表
|
||||
|
||||
```sql
|
||||
CREATE STABLE IF NOT EXISTS vehicle_locations (
|
||||
ts TIMESTAMP,
|
||||
fact_id NCHAR(64),
|
||||
frame_id NCHAR(64),
|
||||
received_at TIMESTAMP,
|
||||
longitude DOUBLE,
|
||||
latitude DOUBLE,
|
||||
altitude_m DOUBLE,
|
||||
speed_kmh DOUBLE,
|
||||
direction_deg DOUBLE,
|
||||
alarm_flag BIGINT,
|
||||
status_flag BIGINT,
|
||||
total_mileage_km DOUBLE,
|
||||
raw_uri NCHAR(512),
|
||||
metadata_json NCHAR(4096)
|
||||
) TAGS (
|
||||
protocol NCHAR(16),
|
||||
vehicle_key NCHAR(128),
|
||||
vin NCHAR(64),
|
||||
phone NCHAR(32)
|
||||
);
|
||||
```
|
||||
|
||||
### 实时数据超级表
|
||||
|
||||
```sql
|
||||
CREATE STABLE IF NOT EXISTS vehicle_realtime (
|
||||
ts TIMESTAMP,
|
||||
fact_id NCHAR(64),
|
||||
frame_id NCHAR(64),
|
||||
received_at TIMESTAMP,
|
||||
speed_kmh DOUBLE,
|
||||
total_mileage_km DOUBLE,
|
||||
battery_soc DOUBLE,
|
||||
total_voltage_v DOUBLE,
|
||||
total_current_a DOUBLE,
|
||||
running_mode NCHAR(64),
|
||||
vehicle_state NCHAR(64),
|
||||
charging_state NCHAR(64),
|
||||
raw_uri NCHAR(512),
|
||||
fields_json NCHAR(8192),
|
||||
metadata_json NCHAR(4096)
|
||||
) TAGS (
|
||||
protocol NCHAR(16),
|
||||
vehicle_key NCHAR(128),
|
||||
vin NCHAR(64),
|
||||
phone NCHAR(32)
|
||||
);
|
||||
```
|
||||
|
||||
`fields_json` 存放稀疏全字段值，用于低频排查。高频查询字段应提升为明确列，或放入专用扩展表。
|
||||
|
||||
### 报警超级表
|
||||
|
||||
```sql
|
||||
CREATE STABLE IF NOT EXISTS vehicle_alarms (
|
||||
ts TIMESTAMP,
|
||||
fact_id NCHAR(64),
|
||||
frame_id NCHAR(64),
|
||||
received_at TIMESTAMP,
|
||||
alarm_level NCHAR(32),
|
||||
alarm_code BIGINT,
|
||||
alarm_name NCHAR(128),
|
||||
longitude DOUBLE,
|
||||
latitude DOUBLE,
|
||||
raw_uri NCHAR(512),
|
||||
fields_json NCHAR(8192),
|
||||
metadata_json NCHAR(4096)
|
||||
) TAGS (
|
||||
protocol NCHAR(16),
|
||||
vehicle_key NCHAR(128),
|
||||
vin NCHAR(64),
|
||||
phone NCHAR(32)
|
||||
);
|
||||
```
|
||||
|
||||
### 会话和注册表
|
||||
|
||||
```sql
|
||||
CREATE STABLE IF NOT EXISTS vehicle_sessions (
|
||||
ts TIMESTAMP,
|
||||
fact_id NCHAR(64),
|
||||
frame_id NCHAR(64),
|
||||
received_at TIMESTAMP,
|
||||
session_type NCHAR(32),
|
||||
result_code INT,
|
||||
raw_uri NCHAR(512),
|
||||
fields_json NCHAR(8192),
|
||||
metadata_json NCHAR(4096)
|
||||
) TAGS (
|
||||
protocol NCHAR(16),
|
||||
vehicle_key NCHAR(128),
|
||||
vin NCHAR(64),
|
||||
phone NCHAR(32)
|
||||
);
|
||||
```
|
||||
|
||||
JT808 注册详情先进入 `vehicle_sessions`，后续可以投影到 MySQL 车辆身份绑定表，用于持久身份解析。
|
||||
|
||||
## ACK 和持久化规则
|
||||
|
||||
对需要协议 ACK 的帧：
|
||||
|
||||
1. Netty 接收到 frame bytes。
|
||||
2. 原始 bytes 写入 raw archive。
|
||||
3. `RawFrameFact` 发布到 Kafka。
|
||||
4. 发送协议 ACK。
|
||||
5. 解码和 `DecodedFact` 发布可以在 ACK 前或 ACK 后执行，但失败必须发 DLQ，并更新 raw frame 解析状态。
|
||||
|
||||
对 GB32960 实时上报和补发帧，这条规则保证不会 ACK 一条无法恢复原始 bytes 的帧。
|
||||
|
||||
对 JT808 注册、鉴权、心跳，也遵循相同 raw archive 和 raw fact 边界。注册 ACK 可以先生成 token，但在 durability boundary 成功前不能 flush 到 channel。
|
||||
|
||||
如果 raw archive 写入失败，不返回成功 ACK。具体是关闭连接还是保留连接，由协议容忍度决定。
|
||||
|
||||
如果 Kafka raw fact 发布失败，durable ACK 帧不返回成功 ACK。非 durable 帧可以写本地 DLQ。
|
||||
|
||||
## 协议解析和扩展
|
||||
|
||||
使用插件契约：
|
||||
|
||||
```java
|
||||
public interface ProtocolFrameDecoder {
|
||||
ProtocolId protocol();
|
||||
DecodeResult decode(RawFrameFact rawFact, byte[] rawBytes);
|
||||
}
|
||||
```
|
||||
|
||||
```java
|
||||
public record DecodeResult(
|
||||
ParseStatus status,
|
||||
List<DecodedFact> facts,
|
||||
String errorMessage,
|
||||
Map<String, String> metadata
|
||||
) {}
|
||||
```
|
||||
|
||||
GB32960：
|
||||
|
||||
- 命令帧映射为会话、实时、位置、报警和扩展事实。
|
||||
- 厂商扩展按平台账号、VIN 或配置 profile 选择。
|
||||
- 全量 raw replay 仍通过读取 `rawUri` 并运行当前解析器完成。
|
||||
|
||||
JT808：
|
||||
|
||||
- 消息 ID 映射为注册、鉴权、心跳、位置、批量位置、多媒体元数据和扩展事实。
|
||||
- 身份解析会更新 `vehicleKey`。
|
||||
- 注册事实携带省、市、厂商、设备 ID、车牌、车牌颜色和 auth token metadata。
|
||||
|
||||
新增协议消息通常不需要改 TDengine schema，除非它成为高频查询维度。未知字段先作为 extension fact 存入 `fields_json`。
|
||||
|
||||
## 查询 API
|
||||
|
||||
API 保持明确，不做一个含糊的统一接口。
|
||||
|
||||
原始帧查询：
|
||||
|
||||
- `GET /api/history/raw-frames`
|
||||
- 过滤条件：`protocol`、`vin`、`phone`、`vehicleKey`、`messageId`、`parseStatus`、`dateFrom`、`dateTo`、`pageSize`、`cursor`。
|
||||
- 返回 frame metadata、raw URI、checksum、parse status，以及可选的紧凑解析摘要。
|
||||
|
||||
单帧详情：
|
||||
|
||||
- `GET /api/history/raw-frames/{frameId}`
|
||||
- 返回 raw metadata 和解码详情。
|
||||
- 支持 `includeRawHex=true`，但必须有严格大小限制。
|
||||
|
||||
位置历史：
|
||||
|
||||
- `GET /api/history/locations`
|
||||
- 过滤条件：`protocol`、`vin`、`phone`、`vehicleKey`、时间范围、cursor。
|
||||
|
||||
实时历史：
|
||||
|
||||
- `GET /api/history/realtime`
|
||||
- 查询历史时序数据。
|
||||
|
||||
实时状态：
|
||||
|
||||
- `GET /api/realtime/vehicles/{vehicleKey}`
|
||||
- 从 Redis 或内存 latest state 读取，不扫 TDengine。
|
||||
|
||||
导出：
|
||||
|
||||
- 导出必须异步。
|
||||
- 请求创建 export job。
|
||||
- Worker 将 TDengine 查询结果流式写入对象存储或本地导出文件。
|
||||
- API 返回 job 状态和下载 URI。
|
||||
|
||||
## 分页
|
||||
|
||||
高 QPS 历史查询使用 cursor pagination，不使用深 offset。
|
||||
|
||||
Cursor 字段：
|
||||
|
||||
```text
|
||||
ts, received_at, fact_id
|
||||
```
|
||||
|
||||
倒序查询谓词：
|
||||
|
||||
```sql
|
||||
WHERE (
|
||||
ts < :cursorTs
|
||||
OR (ts = :cursorTs AND received_at < :cursorReceivedAt)
|
||||
OR (ts = :cursorTs AND received_at = :cursorReceivedAt AND fact_id < :cursorFactId)
|
||||
)
|
||||
```
|
||||
|
||||
默认 page size 为 100，上限为 1000。
|
||||
|
||||
## 容量设计
|
||||
|
||||
假设：
|
||||
|
||||
- 10,000 辆车。
|
||||
- 常规上报间隔 5 到 10 秒。
|
||||
- 预计写入 1,000 到 2,000 frames/s。
|
||||
- 峰值突发系数 3 倍。
|
||||
- 历史查询目标 1,000 requests/s。
|
||||
|
||||
写路径：
|
||||
|
||||
- Netty worker 保持 CPU-light。
|
||||
- Raw archive 写入使用有界异步 worker。
|
||||
- Kafka producer 开启压缩、批量、幂等，并按 vehicle key 分区。
|
||||
- History writer 按超级表和子表批量写 TDengine。
|
||||
- TDengine 子表按 vehicle key 分区，使单车查询足够便宜。
|
||||
|
||||
读路径：
|
||||
|
||||
- 高频查询必须带 vehicle key 和时间范围。
|
||||
- 无边界跨车查询仅限管理接口，并强制限流和限制返回量。
|
||||
- 实时 latest state 从 Redis 或内存状态读取。
|
||||
- Raw 详情只读取单个 raw object 并按需解析。
|
||||
- 字典和元数据接口使用内存缓存。
|
||||
|
||||
背压：
|
||||
|
||||
- Raw archive worker queue 固定上限。
|
||||
- Kafka publish future 必须被跟踪。
|
||||
- Durability boundary 超时时，durable ACK 失败，channel 关闭或被限流。
|
||||
- History writer lag 通过 Kafka consumer lag 和 TDengine batch latency 观察。
|
||||
|
||||
## 迁移计划
|
||||
|
||||
阶段 1：在现有代码旁边建立新的 fact model。
|
||||
|
||||
- 新增 `ingest-facts`。
|
||||
- 新增 raw archive receipt contract。
|
||||
- 新增 raw frame 和 decoded fact 的 Kafka 序列化。
|
||||
- 为 stable id、vehicle key、raw URI 生成规则加测试。
|
||||
|
||||
阶段 2：强化 raw durability boundary。
|
||||
|
||||
- 重构 GB32960 和 JT808 handler，让 ACK 等待 raw archive 和 raw fact publish。
|
||||
- 保留现有解析器。
|
||||
- 向新 Kafka topic 发 decoded facts。
|
||||
|
||||
阶段 3：构建 TDengine writer。
|
||||
|
||||
- 创建 TDengine schema manager。
|
||||
- 写入 raw frame facts。
|
||||
- 写入 location、realtime、alarm、session facts。
|
||||
- 按 stable id 实现幂等写入。
|
||||
|
||||
阶段 4：构建查询 API。
|
||||
|
||||
- Raw frame 查询。
|
||||
- 单帧回放。
|
||||
- 位置历史查询。
|
||||
- 实时历史查询。
|
||||
- 最新实时状态查询。
|
||||
|
||||
阶段 5：切换并下线旧热路径。
|
||||
|
||||
- GB32960/JT808 生产热历史不再依赖 DuckDB/Parquet。
|
||||
- 历史迁移期间保留 raw replay 兼容。
|
||||
- TDengine 覆盖验证完成后，移除协议特定的临时历史存储。
|
||||
|
||||
## 测试策略
|
||||
|
||||
单元测试：
|
||||
|
||||
- `RawFrameFact` 校验和 vehicle key 派生。
|
||||
- Raw archive 路径生成和 checksum。
|
||||
- GB32960 golden frame 到 decoded facts。
|
||||
- JT808 sample frame 到 decoded facts。
|
||||
- ACK boundary 成功和失败场景。
|
||||
- TDengine SQL 生成。
|
||||
|
||||
集成测试：
|
||||
|
||||
- 使用 mocked Kafka producer 和 fake raw archive 启动 APP。
|
||||
- 发送 GB32960 sample frame，验证 raw fact、decoded fact 和 raw archive receipt。
|
||||
- 发送 JT808 注册、鉴权、位置帧，验证 registration、session、location facts。
|
||||
- 模拟 raw archive 失败，验证不发送成功 ACK。
|
||||
- 模拟 Kafka publish 失败，验证 durable ACK 失败。
|
||||
|
||||
存储测试：
|
||||
|
||||
- TDengine schema 初始化。
|
||||
- 批量写入 raw frames 和 decoded facts。
|
||||
- 按 vehicle key 和时间范围查询。
|
||||
- Cursor 分页正确性。
|
||||
- 按 `rawUri` 单帧回放。
|
||||
|
||||
压测：
|
||||
|
||||
- 持续 30 分钟 2,000 frames/s 写入。
|
||||
- 持续 5 分钟 6,000 frames/s 突发写入。
|
||||
- 1,000 QPS location/realtime 历史查询，p95 延迟受控。
|
||||
- 校验 raw archive receipt、Kafka raw fact、TDengine raw frame row 三者数量无缺口。
|
||||
|
||||
## 运维要求
|
||||
|
||||
指标：
|
||||
|
||||
- TCP 连接数。
|
||||
- Frame 接收速率。
|
||||
- Raw archive 写入延迟和失败数。
|
||||
- Kafka 发布延迟和失败数。
|
||||
- 按协议和 message id 统计 decoder 成功/失败数。
|
||||
- TDengine batch size、延迟和失败数。
|
||||
- 各查询 endpoint 的 QPS 和延迟。
|
||||
- Kafka consumer lag。
|
||||
|
||||
日志：
|
||||
|
||||
- 每个失败帧输出一条结构化日志，包含 `frameId`、protocol、vehicle key、message id、raw URI 和 error。
|
||||
- 正常日志不输出完整 raw bytes。
|
||||
- Raw hex 只允许在显式 debug 工具里输出，并限制大小。
|
||||
|
||||
告警：
|
||||
|
||||
- Raw archive 写入失败。
|
||||
- Durable ACK 超时。
|
||||
- Kafka producer 失败。
|
||||
- History writer lag。
|
||||
- TDengine 写入失败。
|
||||
- Raw archive 数量和 TDengine raw frame row 数量不一致。
|
||||
|
||||
## 验收标准
|
||||
|
||||
重设计只有满足以下条件才算完成：
|
||||
|
||||
1. 每一条被接收的 GB32960 和 JT808 帧都会创建 raw archive object 和 TDengine `raw_frames` 行。
|
||||
2. 有 bytes 的异常帧可以从 raw history 查询到，包含 parse status 和错误原因。
|
||||
3. GB32960 和 JT808 位置历史可以按车辆和时间范围 cursor 分页查询。
|
||||
4. GB32960 实时历史可以按车辆和时间范围查询。
|
||||
5. JT808 注册和会话数据可查询，并且可以更新 identity binding。
|
||||
6. 单帧详情可以通过 `rawUri` 重新读取 raw bytes，并使用当前解析器解码。
|
||||
7. 需要 ACK 的帧，在 raw archive 和 raw fact publish 成功前，不会发送成功 ACK。
|
||||
8. 新协议字段可以通过新增 decoder/projector 扩展；只有成为高频查询字段时才需要 TDengine 表迁移。
|
||||
9. 压测证明可以持续 2,000 frames/s 写入，并支持 1,000 QPS 有边界历史查询。
|
||||
10. GB32960/JT808 生产查询不再依赖旧 DuckDB/Parquet 热历史路径。
|
||||
|
||||
## 已固定决策
|
||||
|
||||
第一版实现固定以下选择：
|
||||
|
||||
- 先使用本地 raw archive，实现接口时预留对象存储。
|
||||
- TDengine 是 GB32960/JT808 唯一生产热历史查询库。
|
||||
- 使用协议明确的查询 API，不做一个泛化万能查询 API。
|
||||
- 高频历史查询统一使用 cursor pagination。
|
||||
- Kafka payload 保持轻量，只传 raw 引用，不传 raw bytes。
|
||||
@@ -1,498 +0,0 @@
|
||||
# Go 车辆接入重构设计规格
|
||||
|
||||
## 目标
|
||||
|
||||
使用 Go 重新设计车辆数据接入运行面，让系统回到第一性原则：
|
||||
|
||||
- 接入层只负责可靠收包、协议解析、应答、身份解析和投递。
|
||||
- Kafka 是唯一在线消息通道。
|
||||
- TDengine 保存高吞吐时序明细和完整 RAW 解析 JSON。
|
||||
- MySQL 保存低频配置、身份映射和每日统计窄表。
|
||||
- Redis 保存可重建的准实时状态。
|
||||
- 32960、JT808、宇通 MQTT 使用同一套数据层接口，避免每个协议各自落库。
|
||||
|
||||
第一阶段只做生产链路的骨架和核心数据闭环：接收、解析、Kafka、TDengine、MySQL 统计、Redis 实时查询、ECS 验证。历史查询 API 和业务侧复杂分析放在后续阶段。
|
||||
|
||||
## 参考原则
|
||||
|
||||
本设计只吸收开源项目的边界思想，不复制外部实现代码。
|
||||
|
||||
- GB32960 参考 `DarkInno/gb32960-go-sdk`：连接管理、Handler、Forwarder、鉴权接口分离。
|
||||
- JT808 参考 `cuteLittleDevil/go-jt808`：高并发 Go 原生 TCP、协议核心小而可扩展、事件/适配器机制。
|
||||
- JT808 参考 `fakeyanss/jt808-server-go`：`FramePayload -> PacketData -> JT808Msg -> Reply` 的分层处理、Gateway 模式、保活和版本兼容。
|
||||
- TDengine 参考官方数据模型：按数据采集点建立 supertable，静态维度做 tags，明细数据按子表写入；Go 连接优先使用 WebSocket 驱动。
|
||||
|
||||
## 非目标
|
||||
|
||||
- 第一阶段不继续扩展 Java 服务。
|
||||
- 第一阶段不做统一大查询 API。
|
||||
- 第一阶段不恢复信达 Push。
|
||||
- 第一阶段不把所有协议字段展开成一张超宽 TDengine 表。
|
||||
- Redis 不作为历史事实源。
|
||||
- MySQL 不保存高频原始明细。
|
||||
|
||||
## 目标目录
|
||||
|
||||
新的 Go 运行面放在 `go/vehicle-gateway`，不继续沿用实验性的多 module 分散结构。现有 `go/ingest-edge` 和 `go/vehicle-state` 可以作为迁移素材，最终归并或删除。
|
||||
|
||||
```text
|
||||
go/vehicle-gateway/
|
||||
cmd/
|
||||
gateway/ # 协议接入进程：32960 TCP、JT808 TCP、宇通 MQTT
|
||||
history-writer/ # Kafka -> TDengine RAW/location/mileage points
|
||||
stat-writer/ # Kafka -> MySQL 每日统计窄表
|
||||
realtime-api/ # Redis 准实时查询 API
|
||||
internal/
|
||||
config/ # 环境变量、配置文件、校验
|
||||
gateway/ # TCP/MQTT 生命周期、连接限制、优雅停机
|
||||
protocol/
|
||||
gb32960/ # 32960 编解码
|
||||
jt808/ # 808 编解码
|
||||
yutongmqtt/ # 宇通 MQTT payload 解析
|
||||
identity/ # VIN、phone、device_id、plate 绑定
|
||||
envelope/ # 统一事件模型
|
||||
eventbus/ # Kafka producer/consumer
|
||||
history/ # TDengine adapter
|
||||
stats/ # 每日里程聚合
|
||||
realtime/ # Redis adapter 与快照合并
|
||||
observability/ # 日志、metrics、健康检查
|
||||
```
|
||||
|
||||
## 协议接入设计
|
||||
|
||||
### GB32960
|
||||
|
||||
职责：
|
||||
|
||||
- 监听 TCP `32960`。
|
||||
- 支持车辆登入、实时信息上报、补发信息上报、车辆登出、平台登入/登出、心跳、校时。
|
||||
- 正确返回协议应答。
|
||||
- 完整解析 2016 数据单元：
|
||||
- 整车数据
|
||||
- 驱动电机数据
|
||||
- 燃料电池数据
|
||||
- 发动机数据
|
||||
- 车辆位置数据
|
||||
- 极值数据
|
||||
- 报警数据
|
||||
- 可充电储能装置电压数据
|
||||
- 可充电储能装置温度数据
|
||||
- RAW 中保存完整 parsed JSON。
|
||||
- 关键字段标准化为统一 field：
|
||||
- `speed_kmh`
|
||||
- `total_mileage_km`
|
||||
- `soc_percent`
|
||||
- `longitude`
|
||||
- `latitude`
|
||||
- `vehicle_status`
|
||||
- `charge_status`
|
||||
|
||||
### JT808
|
||||
|
||||
职责：
|
||||
|
||||
- 监听 TCP `808`。
|
||||
- 支持 2011/2013 基础兼容，预留 2019 版本标记。
|
||||
- 分层处理：
|
||||
- Frame：`0x7e` 包边界、转义、校验。
|
||||
- Packet：消息头、手机号 BCD、流水号、分包。
|
||||
- Message：注册、鉴权、心跳、注销、位置、批量位置、未知上行。
|
||||
- Reply：注册应答、平台通用应答。
|
||||
- 终端手机号按协议头 BCD 解析，内部同时保留原始 BCD、规范化 phone、去前导零 phone。
|
||||
- 注册帧和鉴权帧写入 808 registration 表。
|
||||
- 如果设备直接上报位置帧，也要走 identity 解析，尝试用 phone、device_id、plate 找到 VIN。
|
||||
- 0200 附加信息必须解析表 27：
|
||||
- `0x01` GPS 总里程，DWORD，单位 0.1 km。
|
||||
- 其他已知附加项结构化放入 `parsed_json.additional`。
|
||||
- 标准化字段：
|
||||
- `speed_kmh`
|
||||
- `total_mileage_km`
|
||||
- `longitude`
|
||||
- `latitude`
|
||||
- `altitude_m`
|
||||
- `direction_deg`
|
||||
- `alarm_flag`
|
||||
- `status_flag`
|
||||
|
||||
### 宇通 MQTT
|
||||
|
||||
职责：
|
||||
|
||||
- 使用正式 MQTT 配置订阅生产 topic。
|
||||
- 接收 payload 后解析为统一 envelope。
|
||||
- 按 payload 中的车辆标识解析 VIN、车牌、设备号。
|
||||
- 保存完整 payload 和 parsed JSON。
|
||||
- 标准化字段向 32960/808 对齐：
|
||||
- `speed_kmh`
|
||||
- `total_mileage_km`
|
||||
- `longitude`
|
||||
- `latitude`
|
||||
- `soc_percent`
|
||||
- `vehicle_status`
|
||||
|
||||
## 统一 Envelope
|
||||
|
||||
所有协议接入后输出同一结构。
|
||||
|
||||
```json
|
||||
{
|
||||
"event_id": "string",
|
||||
"trace_id": "string",
|
||||
"protocol": "GB32960|JT808|YUTONG_MQTT",
|
||||
"message_id": "string",
|
||||
"vin": "string",
|
||||
"vehicle_key": "string",
|
||||
"phone": "string",
|
||||
"device_id": "string",
|
||||
"plate": "string",
|
||||
"source_endpoint": "ip:port or mqtt://broker/topic",
|
||||
"event_time_ms": 0,
|
||||
"received_at_ms": 0,
|
||||
"raw_hex": "string",
|
||||
"raw_text": "string",
|
||||
"parsed": {},
|
||||
"fields": {
|
||||
"speed_kmh": 0,
|
||||
"total_mileage_km": 0,
|
||||
"longitude": 0,
|
||||
"latitude": 0
|
||||
},
|
||||
"parse_status": "OK|PARTIAL|BAD_FRAME",
|
||||
"parse_error": ""
|
||||
}
|
||||
```
|
||||
|
||||
规则：
|
||||
|
||||
- `event_id` 使用协议、设备标识、消息流水、事件时间、RAW hash 生成，保证幂等。
|
||||
- Kafka partition key 优先使用 VIN；没有 VIN 时使用 `protocol:phone/device_id`。
|
||||
- `parsed` 保存协议完整结构化字段。
|
||||
- `fields` 只保存跨协议核心字段，用于统计、位置和实时快照。
|
||||
|
||||
## Kafka Topic
|
||||
|
||||
生产只支持 Kafka。
|
||||
|
||||
| Topic | 内容 | Key |
|
||||
|---|---|---|
|
||||
| `vehicle.raw.gb32960.v1` | 32960 完整 RAW envelope | VIN 或 vehicle_key |
|
||||
| `vehicle.raw.jt808.v1` | 808 完整 RAW envelope | VIN 或 phone |
|
||||
| `vehicle.raw.yutong-mqtt.v1` | 宇通 MQTT 完整 RAW envelope | VIN 或 vehicle_key |
|
||||
| `vehicle.event.unified.v1` | 统一轻量事件，用于业务消费 | VIN 或 vehicle_key |
|
||||
|
||||
接入层必须先写 RAW topic，再写 unified event。后续消费者只依赖 Kafka，不直接依赖接入进程内存。
|
||||
|
||||
发布可靠性：
|
||||
|
||||
- Kafka 生产端必须开启同步写入，RAW 写成功后才允许写 unified event。
|
||||
- Kafka 写入必须支持可配置重试、单次写超时和短退避，默认值为 `KAFKA_PUBLISH_ATTEMPTS=3`、`KAFKA_PUBLISH_TIMEOUT_MS=3000`、`KAFKA_PUBLISH_BACKOFF_MS=100`。
|
||||
- gateway 支持本地磁盘 spool/WAL，配置 `KAFKA_SPOOL_DIR` 后启用；Kafka 长时间不可用时，发布失败的 envelope 先原子写入本地 JSON 文件。
|
||||
- spool 补发按文件名顺序执行，补发成功后删除文件；默认补发间隔由 `KAFKA_SPOOL_REPLAY_INTERVAL_MS=1000` 控制。
|
||||
- 如果 RAW 写 Kafka 失败并已落盘，同一个 event 的 unified event 不允许抢先写 Kafka，也必须进入 spool，确保恢复后按 RAW -> unified 顺序补发。
|
||||
- Kafka topic 不允许自动创建，topic 和分区数由部署脚本或运维初始化，避免生产拼写错误造成隐性分流。
|
||||
|
||||
## TDengine 数据库设计
|
||||
|
||||
如果现有 TDengine 库不符合目标，可以新建库并迁移。建议库名：
|
||||
|
||||
```sql
|
||||
CREATE DATABASE IF NOT EXISTS lingniu_vehicle_ts KEEP 7300 DURATION 10 BUFFER 256;
|
||||
```
|
||||
|
||||
### raw_frames
|
||||
|
||||
保存完整 RAW 和完整 parsed JSON。
|
||||
|
||||
```sql
|
||||
CREATE STABLE IF NOT EXISTS raw_frames (
|
||||
ts TIMESTAMP,
|
||||
frame_id NCHAR(64),
|
||||
event_id NCHAR(64),
|
||||
message_id INT,
|
||||
event_time TIMESTAMP,
|
||||
received_at TIMESTAMP,
|
||||
raw_size_bytes INT,
|
||||
raw_hex BINARY(16374),
|
||||
raw_text BINARY(16374),
|
||||
parsed_json BINARY(16374),
|
||||
fields_json BINARY(4096),
|
||||
parse_status NCHAR(16),
|
||||
parse_error BINARY(1024),
|
||||
source_endpoint NCHAR(128)
|
||||
) TAGS (
|
||||
protocol NCHAR(32),
|
||||
vehicle_key NCHAR(64),
|
||||
vin NCHAR(32),
|
||||
phone NCHAR(32),
|
||||
device_id NCHAR(64)
|
||||
);
|
||||
```
|
||||
|
||||
说明：
|
||||
|
||||
- 子表按 `protocol + vehicle_key hash` 创建。
|
||||
- RAW 查询优先按 tags 和时间范围过滤。
|
||||
- `parsed_json` 是完整协议字段；查询 API 可选择是否返回，避免默认大字段拖慢。
|
||||
|
||||
### vehicle_locations
|
||||
|
||||
只保存位置查询核心字段。
|
||||
|
||||
```sql
|
||||
CREATE STABLE IF NOT EXISTS vehicle_locations (
|
||||
ts TIMESTAMP,
|
||||
event_id NCHAR(64),
|
||||
frame_id NCHAR(64),
|
||||
received_at TIMESTAMP,
|
||||
longitude DOUBLE,
|
||||
latitude DOUBLE,
|
||||
altitude_m DOUBLE,
|
||||
speed_kmh DOUBLE,
|
||||
direction_deg INT,
|
||||
alarm_flag BIGINT,
|
||||
status_flag BIGINT,
|
||||
total_mileage_km DOUBLE
|
||||
) TAGS (
|
||||
protocol NCHAR(32),
|
||||
vehicle_key NCHAR(64),
|
||||
vin NCHAR(32),
|
||||
phone NCHAR(32),
|
||||
device_id NCHAR(64)
|
||||
);
|
||||
```
|
||||
|
||||
### vehicle_mileage_points
|
||||
|
||||
保存总里程采样点，用于回放重算和查询。
|
||||
|
||||
```sql
|
||||
CREATE STABLE IF NOT EXISTS vehicle_mileage_points (
|
||||
ts TIMESTAMP,
|
||||
event_id NCHAR(64),
|
||||
frame_id NCHAR(64),
|
||||
received_at TIMESTAMP,
|
||||
total_mileage_km DOUBLE,
|
||||
speed_kmh DOUBLE,
|
||||
longitude DOUBLE,
|
||||
latitude DOUBLE
|
||||
) TAGS (
|
||||
protocol NCHAR(32),
|
||||
vehicle_key NCHAR(64),
|
||||
vin NCHAR(32),
|
||||
phone NCHAR(32),
|
||||
device_id NCHAR(64)
|
||||
);
|
||||
```
|
||||
|
||||
## MySQL 数据库设计
|
||||
|
||||
MySQL 只保存配置、身份映射、808 注册鉴权和每日统计窄表。
|
||||
|
||||
### vehicle_identity_binding
|
||||
|
||||
用于把外部标识定位到 VIN。
|
||||
|
||||
```sql
|
||||
CREATE TABLE vehicle_identity_binding (
|
||||
id BIGINT PRIMARY KEY AUTO_INCREMENT,
|
||||
vin VARCHAR(32) NOT NULL,
|
||||
plate VARCHAR(32) NULL,
|
||||
phone VARCHAR(32) NULL,
|
||||
device_id VARCHAR(64) NULL,
|
||||
remark VARCHAR(255) NULL,
|
||||
created_at DATETIME NOT NULL DEFAULT CURRENT_TIMESTAMP,
|
||||
updated_at DATETIME NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
|
||||
UNIQUE KEY uk_vin (vin),
|
||||
KEY idx_plate (plate),
|
||||
KEY idx_phone (phone),
|
||||
KEY idx_device_id (device_id)
|
||||
);
|
||||
```
|
||||
|
||||
### jt808_registration
|
||||
|
||||
只记录 808 独有的注册、鉴权和在线身份信息。
|
||||
|
||||
```sql
|
||||
CREATE TABLE jt808_registration (
|
||||
phone VARCHAR(32) PRIMARY KEY,
|
||||
vin VARCHAR(32) NULL,
|
||||
plate VARCHAR(32) NULL,
|
||||
device_id VARCHAR(64) NULL,
|
||||
manufacturer_id VARCHAR(32) NULL,
|
||||
terminal_model VARCHAR(64) NULL,
|
||||
terminal_id VARCHAR(64) NULL,
|
||||
auth_code VARCHAR(128) NULL,
|
||||
source_endpoint VARCHAR(128) NULL,
|
||||
first_registered_at DATETIME NULL,
|
||||
latest_registered_at DATETIME NULL,
|
||||
latest_authed_at DATETIME NULL,
|
||||
latest_seen_at DATETIME NULL,
|
||||
created_at DATETIME NOT NULL DEFAULT CURRENT_TIMESTAMP,
|
||||
updated_at DATETIME NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
|
||||
KEY idx_vin (vin),
|
||||
KEY idx_plate (plate),
|
||||
KEY idx_device_id (device_id)
|
||||
);
|
||||
```
|
||||
|
||||
### vehicle_daily_metric
|
||||
|
||||
32960 和 808 的每日里程、每日总里程都进入同一张窄表。
|
||||
|
||||
```sql
|
||||
CREATE TABLE vehicle_daily_metric (
|
||||
id BIGINT PRIMARY KEY AUTO_INCREMENT,
|
||||
vin VARCHAR(32) NOT NULL,
|
||||
stat_date DATE NOT NULL,
|
||||
protocol VARCHAR(32) NOT NULL,
|
||||
metric_key VARCHAR(64) NOT NULL,
|
||||
metric_value DECIMAL(18,3) NOT NULL,
|
||||
metric_unit VARCHAR(16) NOT NULL,
|
||||
first_total_mileage_km DECIMAL(18,3) NULL,
|
||||
latest_total_mileage_km DECIMAL(18,3) NULL,
|
||||
sample_count BIGINT NOT NULL DEFAULT 0,
|
||||
calculation_method VARCHAR(64) NOT NULL,
|
||||
created_at DATETIME NOT NULL DEFAULT CURRENT_TIMESTAMP,
|
||||
updated_at DATETIME NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
|
||||
UNIQUE KEY uk_daily_metric (vin, stat_date, protocol, metric_key),
|
||||
KEY idx_stat_date (stat_date),
|
||||
KEY idx_protocol_metric (protocol, metric_key)
|
||||
);
|
||||
```
|
||||
|
||||
指标：
|
||||
|
||||
- `daily_mileage_km = max(total_mileage_km) - min(total_mileage_km)`
|
||||
- `daily_total_mileage_km = max(total_mileage_km)`
|
||||
|
||||
统计消费者可幂等重放。每个样本 upsert 时只维护当天 `min/max total_mileage_km`，不依赖进程内状态。
|
||||
|
||||
## Redis 实时数据设计
|
||||
|
||||
Redis 保存可从 Kafka 重建的数据。
|
||||
|
||||
| Key | 内容 |
|
||||
|---|---|
|
||||
| `vehicle:online:{vin}` | 在线状态、最后活跃时间、协议列表 |
|
||||
| `vehicle:latest:{vin}` | 合并后的 VIN 最新快照 |
|
||||
| `vehicle:latest:{vin}:{protocol}` | 单协议最新快照 |
|
||||
| `vehicle:last_seen` | VIN 到最后接收时间的 sorted set |
|
||||
|
||||
合并规则：
|
||||
|
||||
- 位置以最新 event_time 为准。
|
||||
- 总里程取最新上报值，不做递增假设。
|
||||
- 32960 多帧字段按 field timestamp 合并。
|
||||
- 808 和 MQTT 不覆盖其他协议独有字段。
|
||||
- Redis TTL 用于在线判断，不作为数据删除依据。
|
||||
|
||||
## 数据流
|
||||
|
||||
```mermaid
|
||||
flowchart LR
|
||||
source["车辆 / 平台"] --> gateway["Go gateway"]
|
||||
gateway --> parser["protocol parser"]
|
||||
parser --> identity["identity resolver"]
|
||||
identity --> kafkaRaw["Kafka RAW topics"]
|
||||
identity --> kafkaEvent["Kafka unified event"]
|
||||
kafkaRaw --> history["history-writer"]
|
||||
kafkaRaw --> stats["stat-writer"]
|
||||
kafkaEvent --> realtime["realtime consumer"]
|
||||
history --> td["TDengine raw_frames / locations / mileage_points"]
|
||||
stats --> mysql["MySQL vehicle_daily_metric"]
|
||||
realtime --> redis["Redis latest state"]
|
||||
```
|
||||
|
||||
## 错误处理
|
||||
|
||||
- 协议坏帧仍写 RAW topic，`parse_status=BAD_FRAME`。
|
||||
- 可部分解析的帧写 `parse_status=PARTIAL`，保留 parse error。
|
||||
- Kafka 写失败时接入层先按配置重试；重试耗尽后不写 unified event，必须打错误日志和 metrics。
|
||||
- 启用 `KAFKA_SPOOL_DIR` 后，Kafka 重试耗尽会落本地 spool；如果 spool 写失败，才视为接入层最终失败。
|
||||
- TDengine 写失败不提交 Kafka offset。
|
||||
- MySQL 统计写失败不提交 Kafka offset。
|
||||
- Redis 写失败不影响历史和统计，但要通过 metrics 暴露。
|
||||
- 808 相同 phone 新连接时，旧连接应被替换并记录 latest_seen。
|
||||
|
||||
## 部署设计
|
||||
|
||||
生产 ECS 运行 Go 二进制或容器：
|
||||
|
||||
- `vehicle-gateway`
|
||||
- `vehicle-history-writer`
|
||||
- `vehicle-stat-writer`
|
||||
- `vehicle-realtime-api`
|
||||
|
||||
配置来源：
|
||||
|
||||
- 环境变量优先。
|
||||
- Nacos 可作为后续配置中心，但 Go 第一阶段必须能只靠环境变量启动，便于 Portainer 部署和故障恢复。
|
||||
|
||||
TDengine 连接：
|
||||
|
||||
- 优先 WebSocket `taosWS`。
|
||||
- 若当前 ECS 只开放原生端口，可短期兼容 `taosSql`，但代码配置必须显式标记为兼容模式。
|
||||
|
||||
## 验证计划
|
||||
|
||||
### 本地验证
|
||||
|
||||
- 用样例 808 报文验证：
|
||||
- phone BCD 解析。
|
||||
- 0200 经纬度、速度、方向、时间。
|
||||
- 表 27 `0x01` GPS 总里程。
|
||||
- 用真实 32960 报文验证：
|
||||
- 登录应答正确。
|
||||
- 实时帧和补发帧都能写 RAW。
|
||||
- 位置和总里程字段进入统一 fields。
|
||||
- 用宇通 MQTT payload 验证：
|
||||
- payload 完整保存。
|
||||
- 核心字段归一化。
|
||||
- 不连 Kafka 时可输出 JSON log。
|
||||
- 连接 Kafka 时 RAW topic 可消费到 envelope。
|
||||
|
||||
### ECS 验证
|
||||
|
||||
- 部署前先旁路接收或短窗口切流。
|
||||
- 验证端口：
|
||||
- `32960`
|
||||
- `808`
|
||||
- MQTT 订阅连接。
|
||||
- 验证 Kafka topic 有持续写入。
|
||||
- 验证 TDengine：
|
||||
- `raw_frames` 有三种协议数据。
|
||||
- `vehicle_locations` 能按 VIN、协议、时间分页查询。
|
||||
- `vehicle_mileage_points` 有 32960 和 808 总里程采样。
|
||||
- 验证 MySQL：
|
||||
- `vehicle_daily_metric` 有 32960 和 808 的 `daily_mileage_km`、`daily_total_mileage_km`。
|
||||
- 验证 Redis：
|
||||
- 查询 VIN 在线。
|
||||
- 查询 VIN 合并实时快照。
|
||||
- 查询 VIN 分协议实时快照。
|
||||
|
||||
## 验收标准
|
||||
|
||||
- Go gateway 能在 ECS 上接收真实 32960、808、宇通 MQTT 数据。
|
||||
- 三种协议 RAW 都进入 Kafka。
|
||||
- Kafka 短暂写失败时，gateway 按配置重试；单元测试覆盖首次失败后成功和重试耗尽返回错误。
|
||||
- Kafka 长时间不可用时，gateway 可把 RAW/unified 写入本地 spool；单元测试覆盖 RAW 已落盘时 unified 也落盘、恢复后按顺序补发并删除文件。
|
||||
- 三种协议 RAW 和 parsed JSON 都进入 TDengine。
|
||||
- 32960 和 808 的位置进入 `vehicle_locations`。
|
||||
- 32960 和 808 的总里程采样进入 `vehicle_mileage_points`。
|
||||
- 32960 和 808 的每日里程、每日总里程进入 MySQL 窄表。
|
||||
- Redis 可以查询 VIN 在线和实时数据。
|
||||
- 任一消费者宕机后可通过 Kafka offset 继续恢复。
|
||||
- 查询和统计验证使用 ECS 上的真实数据完成。
|
||||
|
||||
## 实施顺序
|
||||
|
||||
1. 创建 `go/vehicle-gateway` 单 module，迁移现有 Go 原型中可用的解析、Kafka、TDengine、MySQL、Redis 代码。
|
||||
2. 先写协议解析单元测试，覆盖 808 样例报文和 32960 核心字段。
|
||||
3. 完成统一 envelope 和 Kafka sink。
|
||||
4. 完成 TDengine schema bootstrap 与 history writer。
|
||||
5. 完成 MySQL schema bootstrap 与 stat writer。
|
||||
6. 完成 Redis realtime writer/API。
|
||||
7. 本地启动 gateway，用 ECS 转发流量验证。
|
||||
8. 构建 Linux 镜像，部署 ECS。
|
||||
9. 按验收标准逐项验证并记录证据。
|
||||
Reference in New Issue
Block a user