shishengliang/lingniu-vehicle-ingest
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
62 Commits 2 Branches 0 Tags
8d69296b093e5040bc3a9c1fba155ac74081c3a4
Commit Graph

62 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
lingniu
8d69296b09 docs: add chinese vehicle ingest redesign spec 2026-06-29 12:35:31 +08:00
lingniu
cd953e7cd7 docs: add vehicle ingest redesign spec 2026-06-29 12:32:05 +08:00
lingniu
ec555cd67a fix: avoid jt808 fallback ids as vin
All checks were successful
ci/woodpecker/push/woodpecker Pipeline was successful
Details
2026-06-26 11:37:37 +08:00
lingniu
5e619c3634 feat: add jt808 ingest app
All checks were successful
ci/woodpecker/push/woodpecker Pipeline was successful
Details
2026-06-26 11:17:27 +08:00
lingniu
db871b7531 chore: use public aliyun acr in woodpecker
All checks were successful
ci/woodpecker/push/woodpecker Pipeline was successful
Details
2026-06-24 17:00:30 +08:00
lingniu
e5628bd2f3 chore: use aliyun acr vpc registry
Some checks failed
ci/woodpecker/manual/woodpecker Pipeline failed
Details
2026-06-24 16:52:33 +08:00
lingniu
76325af982 chore: push images to aliyun acr 2026-06-24 16:52:05 +08:00
lingniu
310cf9526c chore: skip tests in woodpecker build
All checks were successful
ci/woodpecker/push/woodpecker Pipeline was successful
Details
2026-06-24 16:41:27 +08:00
lingniu
a628728296 chore: add portainer compose for split apps
All checks were successful
ci/woodpecker/push/woodpecker Pipeline was successful
Details
2026-06-24 14:47:02 +08:00
lingniu
22468e78b4 chore: wire split apps deployment config
Some checks failed
ci/woodpecker/manual/woodpecker Pipeline was canceled
Details
2026-06-24 14:38:59 +08:00
lingniu
a70f8f39ad Merge branch 'codex/gb32960-service-split' 2026-06-24 14:32:59 +08:00
lingniu
b983f77c23 chore: make gb32960 diagnostics frame key limit configurable 2026-06-24 11:56:04 +08:00
lingniu
620821b624 docs: plan gb32960 duckdb hot history 2026-06-23 19:39:06 +08:00
lingniu
9d7af780cd docs: design gb32960 production readiness 2026-06-23 19:35:35 +08:00
lingniu
29be62656f docs: record remote kafka live verification 2026-06-23 18:27:52 +08:00
lingniu
ca10ea0d4d docs: record gb32960 split verification 2026-06-23 18:15:59 +08:00
lingniu
8c6168e592 docs: clarify gb32960 split verification limits 2026-06-23 18:13:42 +08:00
lingniu
b17889c2e2 docs: add gb32960 split service runbook 2026-06-23 18:07:21 +08:00
lingniu
c46b46beb8 test: cover split consumer default toggles 2026-06-23 18:00:34 +08:00
lingniu
2431c14934 test: assert split app defaults 2026-06-23 17:57:52 +08:00
lingniu
6a8b53bab6 feat: split history and analytics consumers 2026-06-23 17:43:28 +08:00
lingniu
f9aa214dc2 fix: preflight required kafka sink 2026-06-23 17:36:24 +08:00
lingniu
a80b38bd04 fix: enforce gb32960 kafka ack boundary 2026-06-23 17:27:12 +08:00
lingniu
633b3ea9c9 feat: ack gb32960 after kafka boundary 2026-06-23 17:10:46 +08:00
lingniu
6b82144f3c feat: ingest raw archive envelopes in history 2026-06-23 16:58:45 +08:00
lingniu
66807fdf3d feat: route gb32960 kafka topics 2026-06-23 16:50:07 +08:00
lingniu
8e9afd79ac test: make split composition tests hermetic 2026-06-23 16:45:32 +08:00
lingniu
6423987ddf test: cover split service composition 2026-06-23 16:37:05 +08:00
lingniu
06cb6aa4b9 fix: align split app runtime dependencies 2026-06-23 16:24:32 +08:00
lingniu
f79d995b8c feat: add split service entrypoints 2026-06-23 16:14:32 +08:00
lingniu
4ff481c324 build: defer split app repackaging 2026-06-23 16:04:07 +08:00
lingniu
27b421d49b build: add split service app modules 2026-06-23 15:58:51 +08:00
lingniu
d7afb8e599 fix: harden local archive sink 2026-06-23 15:54:41 +08:00
lingniu
019512bac2 fix: restore sink archive implementation 2026-06-23 15:42:35 +08:00
lingniu
58d46bc372 chore: ignore local worktrees 2026-06-23 15:39:21 +08:00
lingniu
662d04714d docs: plan gb32960 service split 2026-06-23 15:35:51 +08:00
lingniu
33c9931057 docs: design gb32960 service split 2026-06-23 15:28:18 +08:00
kkfluous
a096e4ce0e docs: add detailed 32960 pipeline comments 2026-06-23 13:17:37 +08:00
kkfluous
ba68ffe061 feat: make gb32960 archive history query production ready 2026-06-23 11:55:44 +08:00
kkfluous
b14871ff1c test(bootstrap): e2e verify gb32960 frame lands in archive store 
Spring Boot Test 启用 archive sink 到 JUnit @TempDir，发一条 GB32960 实时上报帧：
  TCP → Gb32960 Netty → Dispatcher emitRawArchive → Disruptor 虚拟线程
     → ArchiveEventSink.publish → LocalArchiveStore.put → 文件落盘

断言：
- 文件出现在 <tempDir>/yyyy/MM/dd/GB32960/<vin>/*.bin 路径下
- 文件内容字节级等于原始发送的 62 字节帧

原始 IngestEndToEndTest 禁用 archive 只验业务事件路径；两个 E2E 互补，保证后续谁
动 Dispatcher / AutoConfig / RawFrame 破坏 archive 扇出都会被 CI 立刻红线。

@DynamicPropertySource 把 @TempDir 的路径注入 lingniu.ingest.sink.archive.path，
避免硬编码路径导致并行测试相互污染。

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-04-20 16:26:04 +08:00
kkfluous
1c1d400784 docs(changelog): record raw archive event sink integration 
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-04-20 15:55:14 +08:00
kkfluous
812b8877d4 feat(ingest-core): Dispatcher emits RawArchive event per inbound frame 
Dispatcher.dispatch 顶部（interceptor 之前）在 RawFrame.rawBytes() 非空时通过
EventBus 发一条 VehicleEvent.RawArchive，由 ArchiveEventSink 消费落盘。

放在 interceptor 之前的理由：原始可回放的目标是"每一条成功解码的帧都要留痕"，
dedup / rate-limit 不应过滤 archive。archive sink 的写失败只影响冷存，不影响
业务事件流。

RawArchive 的 VIN 从 RawFrame.sourceMeta["vin"] 取；缺失时留空字符串，ArchiveSink
落盘时会用 "unknown-vin" 占位（见 ArchiveEventSink.buildKey）。

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-04-20 15:54:59 +08:00
kkfluous
dd838b4f73 feat(sink-archive): ArchiveEventSink consumes RawArchive events to cold storage 
新 EventSink 实现：
- accepts 仅 VehicleEvent.RawArchive
- publish 写 ArchiveStore，key = yyyy/MM/dd/<source>/<vin>/<eventId>.bin
  （基于 ingestTime 的 UTC 日期分片，便于按车按天回放）
- 相同 eventId 重复写按 put 语义覆盖
- 写失败在返回的 CompletableFuture 上 completeExceptionally，EventBus 打 WARN

AutoConfig 在 ArchiveStore bean 存在时装配 ArchiveEventSink，Disruptor EventBus 启动时
自动把它纳入扇出 handler 列表。Disruptor 使用虚拟线程消费，本地/S3/OSS 的 blocking
IO 不会 pin 平台线程。

单测覆盖 accepts 过滤、稳定 key 写入、幂等覆盖三种情形。

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-04-20 15:54:54 +08:00
kkfluous
bfc3c29c40 feat(api): add VehicleEvent.RawArchive sealed variant for raw cold-storage 
新增 VehicleEvent.RawArchive 事件类型，承载一帧入站的原始字节，供 ArchiveEventSink
写冷存、后续迭代可回放。携带 command / infoType / rawBytes 三个字段，VIN 与
timestamps 沿用基接口约定。

同步更新两处穷尽 switch：
- TopicRouter.route：RawArchive → topics.getRawArchive()（未来 Kafka 需要时会用到）
- EnvelopeMapper.toEnvelope：只填 RawArchiveRef.size_bytes，payload oneof 留空

本期 Kafka sink 不处理 raw archive——KafkaEventSink.accepts(RawArchive) 返回 false，
原始字节只落 ArchiveStore，避免 Kafka 消息体被原始报文撑大。未来若接 URI 回填
（archive 写成功后把 URI 放进 Envelope.raw_archive 送 vehicle.raw.archive topic），
再放开过滤。

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-04-20 15:54:46 +08:00
kkfluous
4d8dd54e59 refactor(gb32960): extract channelRead0 command branches into private methods 
把 Gb32960ChannelHandler.channelRead0 的 140 行 if/else-if 级联拆成按 CommandType
的 switch 分派到 7 个独立私有方法：

- handleVehicleLogin       （0x01）
- handleVehicleLogout      （0x04）
- handlePlatformLogin      （0x05，返回 boolean 表示是否继续 dispatch）
- handlePlatformLogout     （0x06）
- handleReportOrHeartbeat  （0x02/0x03/0x07）
- handleTimeCalibration    （0x08）
- logOtherFrame            （其它命令的调试日志）

同时抽出 eventOrNow(msg) 静态 helper，消除 6 处
"msg.header().eventTime() != null ? msg.header().eventTime() : Instant.now()"
的重复。

纯重构，语义零变化：ACK 写入顺序、PlatformLogin 鉴权失败短路、日志 tag、VIN 白名单
预检都保持原行为。全模块 41 tests + 全仓 E2E 全绿，无回归。

channelRead0 从 140 行收敛到约 45 行，每个分支处理器 20~40 行并带 §6.3.2 规范注释，
未来新增命令或调整 ack 时序只动对应私有方法。

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-04-20 15:41:59 +08:00
kkfluous
da28a0c8d5 fix(gb32960): emit platform login/logout events instead of swallowing them 
Gb32960RealtimeHandler.onPlatform 之前返回 List.of() 吞掉所有 0x05/0x06 事件，
而 Gb32960EventMapper.toEvents 里 PLATFORM_LOGIN / PLATFORM_LOGOUT 分支实际构造了
VehicleEvent.Login/Logout（vin="platform:<username>"，metadata.kind="platform"）。
两边语义冲突导致平台登入事件永远不会上送下游——mapper 里那段是死代码。

修正：
- onPlatform 改为委托 mapper.toEvents(msg)，让平台会话事件真正发布到
  vehicle.session topic，下游按 metadata.kind 区分车辆/平台会话
- 加 @EventEmit 声明 Login/Logout 事件类，保持元数据一致
- 去掉 @ProtocolHandler(version = "2017")：2017 不是真实的 GB/T 32960 版本（标准
  仅 2016 / 2025），且 version 属性在 HandlerRegistry 路由里未使用，纯噪音
- JavaDoc 从"示例 Handler：演示..."改为生产路径描述，说明它是 GB32960 在
  Dispatcher 管线里的唯一入口

新增 Gb32960RealtimeHandlerTest 2 case 覆盖平台登入/登出事件产出。
全模块 41 tests 全绿（原 39 + 新 2），无回归。

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-04-20 15:27:52 +08:00
kkfluous
ce87c8ee26 docs(changelog): record gb32960 body parser block isolation 
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-04-20 15:18:26 +08:00
kkfluous
f9b9c2a6a7 config(gb32960): document parse.lenient-block-failure in application.yml 
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-04-20 15:17:07 +08:00
kkfluous
310d1d46f1 config(gb32960): wire parse.lenientBlockFailure into BodyParser bean 
Gb32960AutoConfiguration.gb32960BodyParser 现在注入 Gb32960Properties，
构造完 BodyParser 后把 props.getParse().isLenientBlockFailure() 通过 setter 注入，
让运行期配置 lingniu.ingest.gb32960.parse.lenient-block-failure 生效。

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-04-20 15:16:31 +08:00
kkfluous
89a7c07106 feat(gb32960): isolate per-block parse failures in body parser 
主循环改造：单信息块 parser 抛 DecodeException / BufferUnderflowException /
IndexOutOfBoundsException 时不再放弃整帧。

- 固定长度块（fixedLen≥0）：回滚 reader，按 fixedLen 截取字节兜成 InfoBlock.Raw，
  position 前进 fixedLen，continue 循环继续解析后续块
- 变长块（fixedLen=-1）或剩余不足 fixedLen：剩余字节全部兜成 Raw 后 break
- parser 契约违反（consumed != declared fixedLen）仍抛异常，以暴露 parser bug，
  不走 lenient 兜底，避免静默误解析长期误判
- 仅捕获 DecodeException / BufferUnderflowException / IndexOutOfBoundsException，
  RuntimeException 继续向上抛，保留程序 bug 可见性
- 通过 lingniu.ingest.gb32960.parse.lenientBlockFailure=false 可回退严格模式

不改 InfoBlock.Raw record 结构；下游 Gb32960EventMapper 按 findBlock(Class) 类型
查找，新增 Raw 不影响业务事件映射。

Gb32960BodyParserIsolationTest 4/4 绿；全模块 39 tests 全绿，无回归。

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-04-20 15:15:41 +08:00