shishengliang/lingniu-vehicle-ingest
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
109 Commits 2 Branches 0 Tags
80e96d80755d1f2b7a4a4f74f0ff86c5b50246fd
Commit Graph

109 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
lingniu
5e619c3634 feat: add jt808 ingest app
All checks were successful
ci/woodpecker/push/woodpecker Pipeline was successful
Details
2026-06-26 11:17:27 +08:00
lingniu
db871b7531 chore: use public aliyun acr in woodpecker
All checks were successful
ci/woodpecker/push/woodpecker Pipeline was successful
Details
2026-06-24 17:00:30 +08:00
lingniu
e5628bd2f3 chore: use aliyun acr vpc registry
Some checks failed
ci/woodpecker/manual/woodpecker Pipeline failed
Details
2026-06-24 16:52:33 +08:00
lingniu
76325af982 chore: push images to aliyun acr 2026-06-24 16:52:05 +08:00
lingniu
310cf9526c chore: skip tests in woodpecker build
All checks were successful
ci/woodpecker/push/woodpecker Pipeline was successful
Details
2026-06-24 16:41:27 +08:00
lingniu
a628728296 chore: add portainer compose for split apps
All checks were successful
ci/woodpecker/push/woodpecker Pipeline was successful
Details
2026-06-24 14:47:02 +08:00
lingniu
22468e78b4 chore: wire split apps deployment config
Some checks failed
ci/woodpecker/manual/woodpecker Pipeline was canceled
Details
2026-06-24 14:38:59 +08:00
lingniu
a70f8f39ad Merge branch 'codex/gb32960-service-split' 2026-06-24 14:32:59 +08:00
lingniu
b983f77c23 chore: make gb32960 diagnostics frame key limit configurable 2026-06-24 11:56:04 +08:00
lingniu
620821b624 docs: plan gb32960 duckdb hot history 2026-06-23 19:39:06 +08:00
lingniu
9d7af780cd docs: design gb32960 production readiness 2026-06-23 19:35:35 +08:00
lingniu
29be62656f docs: record remote kafka live verification 2026-06-23 18:27:52 +08:00
lingniu
ca10ea0d4d docs: record gb32960 split verification 2026-06-23 18:15:59 +08:00
lingniu
8c6168e592 docs: clarify gb32960 split verification limits 2026-06-23 18:13:42 +08:00
lingniu
b17889c2e2 docs: add gb32960 split service runbook 2026-06-23 18:07:21 +08:00
lingniu
c46b46beb8 test: cover split consumer default toggles 2026-06-23 18:00:34 +08:00
lingniu
2431c14934 test: assert split app defaults 2026-06-23 17:57:52 +08:00
lingniu
6a8b53bab6 feat: split history and analytics consumers 2026-06-23 17:43:28 +08:00
lingniu
f9aa214dc2 fix: preflight required kafka sink 2026-06-23 17:36:24 +08:00
lingniu
a80b38bd04 fix: enforce gb32960 kafka ack boundary 2026-06-23 17:27:12 +08:00
lingniu
633b3ea9c9 feat: ack gb32960 after kafka boundary 2026-06-23 17:10:46 +08:00
lingniu
6b82144f3c feat: ingest raw archive envelopes in history 2026-06-23 16:58:45 +08:00
lingniu
66807fdf3d feat: route gb32960 kafka topics 2026-06-23 16:50:07 +08:00
lingniu
8e9afd79ac test: make split composition tests hermetic 2026-06-23 16:45:32 +08:00
lingniu
6423987ddf test: cover split service composition 2026-06-23 16:37:05 +08:00
lingniu
06cb6aa4b9 fix: align split app runtime dependencies 2026-06-23 16:24:32 +08:00
lingniu
f79d995b8c feat: add split service entrypoints 2026-06-23 16:14:32 +08:00
lingniu
4ff481c324 build: defer split app repackaging 2026-06-23 16:04:07 +08:00
lingniu
27b421d49b build: add split service app modules 2026-06-23 15:58:51 +08:00
lingniu
d7afb8e599 fix: harden local archive sink 2026-06-23 15:54:41 +08:00
lingniu
019512bac2 fix: restore sink archive implementation 2026-06-23 15:42:35 +08:00
lingniu
58d46bc372 chore: ignore local worktrees 2026-06-23 15:39:21 +08:00
lingniu
662d04714d docs: plan gb32960 service split 2026-06-23 15:35:51 +08:00
lingniu
33c9931057 docs: design gb32960 service split 2026-06-23 15:28:18 +08:00
kkfluous
a096e4ce0e docs: add detailed 32960 pipeline comments 2026-06-23 13:17:37 +08:00
kkfluous
ba68ffe061 feat: make gb32960 archive history query production ready 2026-06-23 11:55:44 +08:00
kkfluous
b14871ff1c test(bootstrap): e2e verify gb32960 frame lands in archive store 
Spring Boot Test 启用 archive sink 到 JUnit @TempDir，发一条 GB32960 实时上报帧：
  TCP → Gb32960 Netty → Dispatcher emitRawArchive → Disruptor 虚拟线程
     → ArchiveEventSink.publish → LocalArchiveStore.put → 文件落盘

断言：
- 文件出现在 <tempDir>/yyyy/MM/dd/GB32960/<vin>/*.bin 路径下
- 文件内容字节级等于原始发送的 62 字节帧

原始 IngestEndToEndTest 禁用 archive 只验业务事件路径；两个 E2E 互补，保证后续谁
动 Dispatcher / AutoConfig / RawFrame 破坏 archive 扇出都会被 CI 立刻红线。

@DynamicPropertySource 把 @TempDir 的路径注入 lingniu.ingest.sink.archive.path，
避免硬编码路径导致并行测试相互污染。

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-04-20 16:26:04 +08:00
kkfluous
1c1d400784 docs(changelog): record raw archive event sink integration 
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-04-20 15:55:14 +08:00
kkfluous
812b8877d4 feat(ingest-core): Dispatcher emits RawArchive event per inbound frame 
Dispatcher.dispatch 顶部（interceptor 之前）在 RawFrame.rawBytes() 非空时通过
EventBus 发一条 VehicleEvent.RawArchive，由 ArchiveEventSink 消费落盘。

放在 interceptor 之前的理由：原始可回放的目标是"每一条成功解码的帧都要留痕"，
dedup / rate-limit 不应过滤 archive。archive sink 的写失败只影响冷存，不影响
业务事件流。

RawArchive 的 VIN 从 RawFrame.sourceMeta["vin"] 取；缺失时留空字符串，ArchiveSink
落盘时会用 "unknown-vin" 占位（见 ArchiveEventSink.buildKey）。

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-04-20 15:54:59 +08:00
kkfluous
dd838b4f73 feat(sink-archive): ArchiveEventSink consumes RawArchive events to cold storage 
新 EventSink 实现：
- accepts 仅 VehicleEvent.RawArchive
- publish 写 ArchiveStore，key = yyyy/MM/dd/<source>/<vin>/<eventId>.bin
  （基于 ingestTime 的 UTC 日期分片，便于按车按天回放）
- 相同 eventId 重复写按 put 语义覆盖
- 写失败在返回的 CompletableFuture 上 completeExceptionally，EventBus 打 WARN

AutoConfig 在 ArchiveStore bean 存在时装配 ArchiveEventSink，Disruptor EventBus 启动时
自动把它纳入扇出 handler 列表。Disruptor 使用虚拟线程消费，本地/S3/OSS 的 blocking
IO 不会 pin 平台线程。

单测覆盖 accepts 过滤、稳定 key 写入、幂等覆盖三种情形。

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-04-20 15:54:54 +08:00
kkfluous
bfc3c29c40 feat(api): add VehicleEvent.RawArchive sealed variant for raw cold-storage 
新增 VehicleEvent.RawArchive 事件类型，承载一帧入站的原始字节，供 ArchiveEventSink
写冷存、后续迭代可回放。携带 command / infoType / rawBytes 三个字段，VIN 与
timestamps 沿用基接口约定。

同步更新两处穷尽 switch：
- TopicRouter.route：RawArchive → topics.getRawArchive()（未来 Kafka 需要时会用到）
- EnvelopeMapper.toEnvelope：只填 RawArchiveRef.size_bytes，payload oneof 留空

本期 Kafka sink 不处理 raw archive——KafkaEventSink.accepts(RawArchive) 返回 false，
原始字节只落 ArchiveStore，避免 Kafka 消息体被原始报文撑大。未来若接 URI 回填
（archive 写成功后把 URI 放进 Envelope.raw_archive 送 vehicle.raw.archive topic），
再放开过滤。

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-04-20 15:54:46 +08:00
kkfluous
4d8dd54e59 refactor(gb32960): extract channelRead0 command branches into private methods 
把 Gb32960ChannelHandler.channelRead0 的 140 行 if/else-if 级联拆成按 CommandType
的 switch 分派到 7 个独立私有方法：

- handleVehicleLogin       （0x01）
- handleVehicleLogout      （0x04）
- handlePlatformLogin      （0x05，返回 boolean 表示是否继续 dispatch）
- handlePlatformLogout     （0x06）
- handleReportOrHeartbeat  （0x02/0x03/0x07）
- handleTimeCalibration    （0x08）
- logOtherFrame            （其它命令的调试日志）

同时抽出 eventOrNow(msg) 静态 helper，消除 6 处
"msg.header().eventTime() != null ? msg.header().eventTime() : Instant.now()"
的重复。

纯重构，语义零变化：ACK 写入顺序、PlatformLogin 鉴权失败短路、日志 tag、VIN 白名单
预检都保持原行为。全模块 41 tests + 全仓 E2E 全绿，无回归。

channelRead0 从 140 行收敛到约 45 行，每个分支处理器 20~40 行并带 §6.3.2 规范注释，
未来新增命令或调整 ack 时序只动对应私有方法。

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-04-20 15:41:59 +08:00
kkfluous
da28a0c8d5 fix(gb32960): emit platform login/logout events instead of swallowing them 
Gb32960RealtimeHandler.onPlatform 之前返回 List.of() 吞掉所有 0x05/0x06 事件，
而 Gb32960EventMapper.toEvents 里 PLATFORM_LOGIN / PLATFORM_LOGOUT 分支实际构造了
VehicleEvent.Login/Logout（vin="platform:<username>"，metadata.kind="platform"）。
两边语义冲突导致平台登入事件永远不会上送下游——mapper 里那段是死代码。

修正：
- onPlatform 改为委托 mapper.toEvents(msg)，让平台会话事件真正发布到
  vehicle.session topic，下游按 metadata.kind 区分车辆/平台会话
- 加 @EventEmit 声明 Login/Logout 事件类，保持元数据一致
- 去掉 @ProtocolHandler(version = "2017")：2017 不是真实的 GB/T 32960 版本（标准
  仅 2016 / 2025），且 version 属性在 HandlerRegistry 路由里未使用，纯噪音
- JavaDoc 从"示例 Handler：演示..."改为生产路径描述，说明它是 GB32960 在
  Dispatcher 管线里的唯一入口

新增 Gb32960RealtimeHandlerTest 2 case 覆盖平台登入/登出事件产出。
全模块 41 tests 全绿（原 39 + 新 2），无回归。

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-04-20 15:27:52 +08:00
kkfluous
ce87c8ee26 docs(changelog): record gb32960 body parser block isolation 
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-04-20 15:18:26 +08:00
kkfluous
f9b9c2a6a7 config(gb32960): document parse.lenient-block-failure in application.yml 
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-04-20 15:17:07 +08:00
kkfluous
310d1d46f1 config(gb32960): wire parse.lenientBlockFailure into BodyParser bean 
Gb32960AutoConfiguration.gb32960BodyParser 现在注入 Gb32960Properties，
构造完 BodyParser 后把 props.getParse().isLenientBlockFailure() 通过 setter 注入，
让运行期配置 lingniu.ingest.gb32960.parse.lenient-block-failure 生效。

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-04-20 15:16:31 +08:00
kkfluous
89a7c07106 feat(gb32960): isolate per-block parse failures in body parser 
主循环改造：单信息块 parser 抛 DecodeException / BufferUnderflowException /
IndexOutOfBoundsException 时不再放弃整帧。

- 固定长度块（fixedLen≥0）：回滚 reader，按 fixedLen 截取字节兜成 InfoBlock.Raw，
  position 前进 fixedLen，continue 循环继续解析后续块
- 变长块（fixedLen=-1）或剩余不足 fixedLen：剩余字节全部兜成 Raw 后 break
- parser 契约违反（consumed != declared fixedLen）仍抛异常，以暴露 parser bug，
  不走 lenient 兜底，避免静默误解析长期误判
- 仅捕获 DecodeException / BufferUnderflowException / IndexOutOfBoundsException，
  RuntimeException 继续向上抛，保留程序 bug 可见性
- 通过 lingniu.ingest.gb32960.parse.lenientBlockFailure=false 可回退严格模式

不改 InfoBlock.Raw record 结构；下游 Gb32960EventMapper 按 findBlock(Class) 类型
查找，新增 Raw 不影响业务事件映射。

Gb32960BodyParserIsolationTest 4/4 绿；全模块 39 tests 全绿，无回归。

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-04-20 15:15:41 +08:00
kkfluous
9cbc19578b test(gb32960): red — isolation test suite for body parser block failures 
4 个测试覆盖：
- fixedLengthBlockFailure_isIsolated_subsequentBlocksStillParsed
- truncatedFixedLengthBlock_isWrappedAsRaw_loopTerminates
- variableLengthBlockFailure_swallowsRemainderAsRaw_loopBreaks
- strictMode_throwsOnAnyBlockFailure（当前行为已经抛异常，即通过）

外加在 Gb32960BodyParser 上加了 lenientBlockFailure 字段 + setter 的 stub（默认 true，
尚未接入主循环逻辑；行为变化由下一个 commit 实现），确保测试文件能编译。

当前运行：Tests run: 4, Errors: 3（3 个隔离场景未实现），strictMode PASS。

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-04-20 15:13:53 +08:00
kkfluous
fa552f4b20 config(gb32960): add parse.lenientBlockFailure toggle 
新增 lingniu.ingest.gb32960.parse.lenientBlockFailure 配置开关（默认 true），
为 Gb32960BodyParser 的单块异常隔离行为做回退开关。实现在后续 commit。

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-04-20 15:12:16 +08:00
lingniu-dev
d8279ac3b7 chore(gb32960): demote high-frequency report/heartbeat ack logs to DEBUG 
Every 0x02/0x03/0x07 ack produced an INFO line that, at one report per
vehicle every few seconds across many vehicles, drowns out meaningful
events in the log stream.

writeAck now checks the tag string: tags starting with "report-ack-" or
"heartbeat-ack" log their success at DEBUG (guarded by isDebugEnabled to
avoid building the hex dump string when DEBUG is off). Sparse ack tags
-- vehicle-login-ack / vehicle-logout-ack / platform-login-ack /
platform-logout-ack / time-calibration-ack -- keep INFO for audit.
Flush *failure* is still WARN for every tag so problems are never
silenced.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
 2026-04-15 17:43:52 +08:00