shishengliang/lingniu-vehicle-ingest
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
69 Commits 2 Branches 0 Tags
9c2779f82bf04791027b81cd139d7d50a087ffe8
Commit Graph

69 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
lingniu
9c2779f82b feat: add fact payloads to kafka envelope 2026-06-29 12:58:27 +08:00
lingniu
0ec29a732c feat: add raw archive store contract 2026-06-29 12:56:50 +08:00
lingniu
969a245454 feat: add decoded fact model 2026-06-29 12:55:20 +08:00
lingniu
b5453ff888 feat: add raw frame fact model 2026-06-29 12:54:16 +08:00
lingniu
aed023043c feat: add vehicle key derivation 2026-06-29 12:53:08 +08:00
lingniu
4feb1ec829 build: add ingest facts module 2026-06-29 12:52:17 +08:00
lingniu
310ef94a6e docs: plan vehicle ingest redesign phase 1 2026-06-29 12:50:41 +08:00
lingniu
8d69296b09 docs: add chinese vehicle ingest redesign spec 2026-06-29 12:35:31 +08:00
lingniu
cd953e7cd7 docs: add vehicle ingest redesign spec 2026-06-29 12:32:05 +08:00
lingniu
ec555cd67a fix: avoid jt808 fallback ids as vin
All checks were successful
ci/woodpecker/push/woodpecker Pipeline was successful
Details
2026-06-26 11:37:37 +08:00
lingniu
5e619c3634 feat: add jt808 ingest app
All checks were successful
ci/woodpecker/push/woodpecker Pipeline was successful
Details
2026-06-26 11:17:27 +08:00
lingniu
db871b7531 chore: use public aliyun acr in woodpecker
All checks were successful
ci/woodpecker/push/woodpecker Pipeline was successful
Details
2026-06-24 17:00:30 +08:00
lingniu
e5628bd2f3 chore: use aliyun acr vpc registry
Some checks failed
ci/woodpecker/manual/woodpecker Pipeline failed
Details
2026-06-24 16:52:33 +08:00
lingniu
76325af982 chore: push images to aliyun acr 2026-06-24 16:52:05 +08:00
lingniu
310cf9526c chore: skip tests in woodpecker build
All checks were successful
ci/woodpecker/push/woodpecker Pipeline was successful
Details
2026-06-24 16:41:27 +08:00
lingniu
a628728296 chore: add portainer compose for split apps
All checks were successful
ci/woodpecker/push/woodpecker Pipeline was successful
Details
2026-06-24 14:47:02 +08:00
lingniu
22468e78b4 chore: wire split apps deployment config
Some checks failed
ci/woodpecker/manual/woodpecker Pipeline was canceled
Details
2026-06-24 14:38:59 +08:00
lingniu
a70f8f39ad Merge branch 'codex/gb32960-service-split' 2026-06-24 14:32:59 +08:00
lingniu
b983f77c23 chore: make gb32960 diagnostics frame key limit configurable 2026-06-24 11:56:04 +08:00
lingniu
620821b624 docs: plan gb32960 duckdb hot history 2026-06-23 19:39:06 +08:00
lingniu
9d7af780cd docs: design gb32960 production readiness 2026-06-23 19:35:35 +08:00
lingniu
29be62656f docs: record remote kafka live verification 2026-06-23 18:27:52 +08:00
lingniu
ca10ea0d4d docs: record gb32960 split verification 2026-06-23 18:15:59 +08:00
lingniu
8c6168e592 docs: clarify gb32960 split verification limits 2026-06-23 18:13:42 +08:00
lingniu
b17889c2e2 docs: add gb32960 split service runbook 2026-06-23 18:07:21 +08:00
lingniu
c46b46beb8 test: cover split consumer default toggles 2026-06-23 18:00:34 +08:00
lingniu
2431c14934 test: assert split app defaults 2026-06-23 17:57:52 +08:00
lingniu
6a8b53bab6 feat: split history and analytics consumers 2026-06-23 17:43:28 +08:00
lingniu
f9aa214dc2 fix: preflight required kafka sink 2026-06-23 17:36:24 +08:00
lingniu
a80b38bd04 fix: enforce gb32960 kafka ack boundary 2026-06-23 17:27:12 +08:00
lingniu
633b3ea9c9 feat: ack gb32960 after kafka boundary 2026-06-23 17:10:46 +08:00
lingniu
6b82144f3c feat: ingest raw archive envelopes in history 2026-06-23 16:58:45 +08:00
lingniu
66807fdf3d feat: route gb32960 kafka topics 2026-06-23 16:50:07 +08:00
lingniu
8e9afd79ac test: make split composition tests hermetic 2026-06-23 16:45:32 +08:00
lingniu
6423987ddf test: cover split service composition 2026-06-23 16:37:05 +08:00
lingniu
06cb6aa4b9 fix: align split app runtime dependencies 2026-06-23 16:24:32 +08:00
lingniu
f79d995b8c feat: add split service entrypoints 2026-06-23 16:14:32 +08:00
lingniu
4ff481c324 build: defer split app repackaging 2026-06-23 16:04:07 +08:00
lingniu
27b421d49b build: add split service app modules 2026-06-23 15:58:51 +08:00
lingniu
d7afb8e599 fix: harden local archive sink 2026-06-23 15:54:41 +08:00
lingniu
019512bac2 fix: restore sink archive implementation 2026-06-23 15:42:35 +08:00
lingniu
58d46bc372 chore: ignore local worktrees 2026-06-23 15:39:21 +08:00
lingniu
662d04714d docs: plan gb32960 service split 2026-06-23 15:35:51 +08:00
lingniu
33c9931057 docs: design gb32960 service split 2026-06-23 15:28:18 +08:00
kkfluous
a096e4ce0e docs: add detailed 32960 pipeline comments 2026-06-23 13:17:37 +08:00
kkfluous
ba68ffe061 feat: make gb32960 archive history query production ready 2026-06-23 11:55:44 +08:00
kkfluous
b14871ff1c test(bootstrap): e2e verify gb32960 frame lands in archive store 
Spring Boot Test 启用 archive sink 到 JUnit @TempDir，发一条 GB32960 实时上报帧：
  TCP → Gb32960 Netty → Dispatcher emitRawArchive → Disruptor 虚拟线程
     → ArchiveEventSink.publish → LocalArchiveStore.put → 文件落盘

断言：
- 文件出现在 <tempDir>/yyyy/MM/dd/GB32960/<vin>/*.bin 路径下
- 文件内容字节级等于原始发送的 62 字节帧

原始 IngestEndToEndTest 禁用 archive 只验业务事件路径；两个 E2E 互补，保证后续谁
动 Dispatcher / AutoConfig / RawFrame 破坏 archive 扇出都会被 CI 立刻红线。

@DynamicPropertySource 把 @TempDir 的路径注入 lingniu.ingest.sink.archive.path，
避免硬编码路径导致并行测试相互污染。

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-04-20 16:26:04 +08:00
kkfluous
1c1d400784 docs(changelog): record raw archive event sink integration 
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-04-20 15:55:14 +08:00
kkfluous
812b8877d4 feat(ingest-core): Dispatcher emits RawArchive event per inbound frame 
Dispatcher.dispatch 顶部（interceptor 之前）在 RawFrame.rawBytes() 非空时通过
EventBus 发一条 VehicleEvent.RawArchive，由 ArchiveEventSink 消费落盘。

放在 interceptor 之前的理由：原始可回放的目标是"每一条成功解码的帧都要留痕"，
dedup / rate-limit 不应过滤 archive。archive sink 的写失败只影响冷存，不影响
业务事件流。

RawArchive 的 VIN 从 RawFrame.sourceMeta["vin"] 取；缺失时留空字符串，ArchiveSink
落盘时会用 "unknown-vin" 占位（见 ArchiveEventSink.buildKey）。

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-04-20 15:54:59 +08:00
kkfluous
dd838b4f73 feat(sink-archive): ArchiveEventSink consumes RawArchive events to cold storage 
新 EventSink 实现：
- accepts 仅 VehicleEvent.RawArchive
- publish 写 ArchiveStore，key = yyyy/MM/dd/<source>/<vin>/<eventId>.bin
  （基于 ingestTime 的 UTC 日期分片，便于按车按天回放）
- 相同 eventId 重复写按 put 语义覆盖
- 写失败在返回的 CompletableFuture 上 completeExceptionally，EventBus 打 WARN

AutoConfig 在 ArchiveStore bean 存在时装配 ArchiveEventSink，Disruptor EventBus 启动时
自动把它纳入扇出 handler 列表。Disruptor 使用虚拟线程消费，本地/S3/OSS 的 blocking
IO 不会 pin 平台线程。

单测覆盖 accepts 过滤、稳定 key 写入、幂等覆盖三种情形。

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-04-20 15:54:54 +08:00