shishengliang/lingniu-vehicle-ingest
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

refactor: rename kafka sink module

Browse Source
This commit is contained in:
lingniu
2026-07-01 10:56:33 +08:00
parent 849e8f8a8b
commit e80fa6f38c
43 changed files with 87 additions and 66 deletions
Download Patch File Download Diff File Expand all files Collapse all files

6
docs/module-data-flow.html
View File

@@ -317,7 +317,7 @@ flowchart TB
end
subgraph sink["输出与明细存储 modules/sinks"]
mq["sink-mq<br/>VehicleEvent 到 Protobuf Envelope 到 Kafka"]
mq["sink-kafka<br/>VehicleEvent 到 Protobuf Envelope 到 Kafka"]
archive["sink-archive<br/>RawArchive 到本地/S3/OSS 冷存"]
tdengineStore["tdengine-history-store<br/>TDengine raw_frames + locations"]
fileStore["event-file-store<br/>Parquet/DuckDB 兼容可选"]
@@ -416,7 +416,7 @@ sequenceDiagram
participant Dispatcher as Dispatcher
participant Handler as Protocol Handler<br/>Mapper
participant EventBus as DisruptorEventBus
participant MQ as sink-mq<br/>Kafka Envelope
participant MQ as sink-kafka<br/>Kafka Envelope
participant Archive as sink-archive<br/>ArchiveStore
participant History as vehicle-history-app
participant TDengine as tdengine-history-store<br/>raw_frames + locations
@@ -578,7 +578,7 @@ flowchart LR
<td><span class="tag">车企平台接入</span><span class="tag green">profile 扩展</span><span class="tag green">身份映射</span><span class="tag green">双向 TLS</span><span class="tag green">错误隔离</span></td>
</tr>
<tr>
<td>modules/sinks/sink-mq</td>
<td>modules/sinks/sink-kafka</td>
<td>Kafka Sink。把 VehicleEvent 转成 Protobuf Envelope，按 VIN 分区投递；业务事件携带全字段 TelemetrySnapshot 和 rawArchiveUri，显式 RawArchive Envelope 会填充 archive:// 逻辑 URI 与 size，默认 Kafka Sink 仍不发送原始 bytes 本体；同时提供 KafkaEnvelopeConsumerFactory、KafkaEnvelopeConsumerRunner、KafkaEnvelopeConsumerWorker 和 KafkaEnvelopeDeadLetterSink，统一服务侧 Kafka 消费启动、独立 group 绑定、死信发布和 topic 到 EnvelopeConsumerProcessor 的分发。</td>
<td>VehicleEvent。</td>
<td>Kafka Protobuf 消息、消费侧 DLQ 记录。</td>

2
docs/operations/gb32960-service-split-runbook.md
View File

@@ -251,7 +251,7 @@ Observed on 2026-06-23 in worktree `.worktrees/gb32960-service-split`:
Observed on 2026-06-23 in worktree `.worktrees/gb32960-service-split`:
- Targeted module tests: `mvn -pl :sink-mq,:ingest-core,:protocol-gb32960,:event-history-service,:vehicle-state-service,:vehicle-stat-service test` ended with `BUILD SUCCESS`; Maven reported 55 protocol/sink/core tests, 30 event-history tests, 14 vehicle-state tests, and 19 vehicle-stat tests with 0 failures/errors/skips.
- Targeted module tests: `mvn -pl :sink-kafka,:ingest-core,:protocol-gb32960,:event-history-service,:vehicle-state-service,:vehicle-stat-service test` ended with `BUILD SUCCESS`; Maven reported 55 protocol/sink/core tests, 30 event-history tests, 14 vehicle-state tests, and 19 vehicle-stat tests with 0 failures/errors/skips.
- Split app tests: `mvn -pl :gb32960-ingest-app,:vehicle-history-app,:vehicle-analytics-app test` ended with `BUILD SUCCESS`; Maven reported 6 app composition/default tests with 0 failures/errors/skips.
- Package verification: `mvn -pl :gb32960-ingest-app,:vehicle-history-app,:vehicle-analytics-app -am package -Dmaven.test.skip=true` ended with `BUILD SUCCESS` and repackaged all three app jars.
- Split app startup: not run in this verification pass because local Kafka was still absent.

2
docs/superpowers/plans/2026-04-20-gb32960-body-parser-block-isolation.md
View File

@@ -543,7 +543,7 @@ EOF
Run: `mvn test -q`
Expected: BUILD SUCCESS。重点关注：
- `protocol-gb32960` 模块全绿（Golden、FullBlocks、Isolation、GuangdongFcEndToEnd、Decoder、Mapper、所有 parser 单测、profile）
- 其它模块（`ingest-core`、`sink-mq` 等）不受影响（无改动应自动通过）
- 其它模块（`ingest-core`、`sink-kafka` 等）不受影响（无改动应自动通过）
若有红，停下来诊断。

46
docs/superpowers/plans/2026-06-23-gb32960-service-split.md
View File

@@ -48,11 +48,11 @@ Create app modules:
Modify shared build and Kafka contract files:
- `pom.xml`
- `modules/sinks/sink-mq/src/main/java/com/lingniu/ingest/sink/mq/KafkaSinkProperties.java`
- `modules/sinks/sink-mq/src/main/java/com/lingniu/ingest/sink/mq/TopicRouter.java`
- `modules/sinks/sink-mq/src/main/java/com/lingniu/ingest/sink/mq/KafkaEventSink.java`
- `modules/sinks/sink-mq/src/test/java/com/lingniu/ingest/sink/mq/TopicRouterTest.java`
- `modules/sinks/sink-mq/src/test/java/com/lingniu/ingest/sink/mq/KafkaEventSinkTest.java`
- `modules/sinks/sink-kafka/src/main/java/com/lingniu/ingest/sink/kafka/KafkaSinkProperties.java`
- `modules/sinks/sink-kafka/src/main/java/com/lingniu/ingest/sink/kafka/TopicRouter.java`
- `modules/sinks/sink-kafka/src/main/java/com/lingniu/ingest/sink/kafka/KafkaEventSink.java`
- `modules/sinks/sink-kafka/src/test/java/com/lingniu/ingest/sink/kafka/TopicRouterTest.java`
- `modules/sinks/sink-kafka/src/test/java/com/lingniu/ingest/sink/kafka/KafkaEventSinkTest.java`
Modify ACK/Kafka durability boundary files:
@@ -157,7 +157,7 @@ Create `modules/apps/gb32960-ingest-app/pom.xml`:
</dependency>
<dependency>
<groupId>com.lingniu.ingest</groupId>
<artifactId>sink-mq</artifactId>
<artifactId>sink-kafka</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
@@ -222,7 +222,7 @@ Create `modules/apps/vehicle-history-app/pom.xml`:
</dependency>
<dependency>
<groupId>com.lingniu.ingest</groupId>
<artifactId>sink-mq</artifactId>
<artifactId>sink-kafka</artifactId>
</dependency>
<dependency>
<groupId>com.lingniu.ingest</groupId>
@@ -303,7 +303,7 @@ Create `modules/apps/vehicle-analytics-app/pom.xml`:
</dependency>
<dependency>
<groupId>com.lingniu.ingest</groupId>
<artifactId>sink-mq</artifactId>
<artifactId>sink-kafka</artifactId>
</dependency>
<dependency>
<groupId>com.lingniu.ingest</groupId>
@@ -933,15 +933,15 @@ git commit -m "test: cover split service composition"
### Task 4: Version Kafka Topics for GB32960 Raw/Event/DLQ
**Files:**
- Modify: `modules/sinks/sink-mq/src/main/java/com/lingniu/ingest/sink/mq/KafkaSinkProperties.java`
- Modify: `modules/sinks/sink-mq/src/main/java/com/lingniu/ingest/sink/mq/TopicRouter.java`
- Modify: `modules/sinks/sink-mq/src/main/java/com/lingniu/ingest/sink/mq/KafkaEventSink.java`
- Create or modify: `modules/sinks/sink-mq/src/test/java/com/lingniu/ingest/sink/mq/TopicRouterTest.java`
- Create or modify: `modules/sinks/sink-mq/src/test/java/com/lingniu/ingest/sink/mq/KafkaEventSinkTest.java`
- Modify: `modules/sinks/sink-kafka/src/main/java/com/lingniu/ingest/sink/kafka/KafkaSinkProperties.java`
- Modify: `modules/sinks/sink-kafka/src/main/java/com/lingniu/ingest/sink/kafka/TopicRouter.java`
- Modify: `modules/sinks/sink-kafka/src/main/java/com/lingniu/ingest/sink/kafka/KafkaEventSink.java`
- Create or modify: `modules/sinks/sink-kafka/src/test/java/com/lingniu/ingest/sink/kafka/TopicRouterTest.java`
- Create or modify: `modules/sinks/sink-kafka/src/test/java/com/lingniu/ingest/sink/kafka/KafkaEventSinkTest.java`
- [ ] **Step 1: Write topic routing tests**
Create `modules/sinks/sink-mq/src/test/java/com/lingniu/ingest/sink/mq/TopicRouterTest.java` if it does not exist. Add tests proving GB32960 raw and normalized events can route to versioned topics. Use the existing `VehicleEvent` constructors from `EventFileStoreSinkTest` and `EnvelopeMapperTelemetrySnapshotTest` as examples.
Create `modules/sinks/sink-kafka/src/test/java/com/lingniu/ingest/sink/kafka/TopicRouterTest.java` if it does not exist. Add tests proving GB32960 raw and normalized events can route to versioned topics. Use the existing `VehicleEvent` constructors from `EventFileStoreSinkTest` and `EnvelopeMapperTelemetrySnapshotTest` as examples.
Expected test names:
@@ -996,7 +996,7 @@ Adjust constructor arguments to match the current sealed `VehicleEvent` definiti
Run:
```bash
mvn -pl :sink-mq -Dtest=TopicRouterTest test
mvn -pl :sink-kafka -Dtest=TopicRouterTest test
```
Expected before implementation: compile failure if constructors are wrong or assertion failure if defaults are old. Fix constructors first; keep routing assertions.
@@ -1028,12 +1028,12 @@ public boolean accepts(VehicleEvent event) {
Update the JavaDoc above it to say Kafka carries both normalized events and raw archive records in split-service mode.
- [ ] **Step 5: Run sink-mq tests**
- [ ] **Step 5: Run sink-kafka tests**
Run:
```bash
mvn -pl :sink-mq test
mvn -pl :sink-kafka test
```
Expected: PASS.
@@ -1041,11 +1041,11 @@ Expected: PASS.
- [ ] **Step 6: Commit**
```bash
git add modules/sinks/sink-mq/src/main/java/com/lingniu/ingest/sink/mq/KafkaSinkProperties.java \
modules/sinks/sink-mq/src/main/java/com/lingniu/ingest/sink/mq/TopicRouter.java \
modules/sinks/sink-mq/src/main/java/com/lingniu/ingest/sink/mq/KafkaEventSink.java \
modules/sinks/sink-mq/src/test/java/com/lingniu/ingest/sink/mq/TopicRouterTest.java \
modules/sinks/sink-mq/src/test/java/com/lingniu/ingest/sink/mq/KafkaEventSinkTest.java
git add modules/sinks/sink-kafka/src/main/java/com/lingniu/ingest/sink/kafka/KafkaSinkProperties.java \
modules/sinks/sink-kafka/src/main/java/com/lingniu/ingest/sink/kafka/TopicRouter.java \
modules/sinks/sink-kafka/src/main/java/com/lingniu/ingest/sink/kafka/KafkaEventSink.java \
modules/sinks/sink-kafka/src/test/java/com/lingniu/ingest/sink/kafka/TopicRouterTest.java \
modules/sinks/sink-kafka/src/test/java/com/lingniu/ingest/sink/kafka/KafkaEventSinkTest.java
git commit -m "feat: route gb32960 kafka topics"
```
@@ -1567,7 +1567,7 @@ git commit -m "docs: add gb32960 split service runbook"
Run:
```bash
mvn -pl :sink-mq,:ingest-core,:protocol-gb32960,:event-history-service,:vehicle-state-service,:vehicle-stat-service test
mvn -pl :sink-kafka,:ingest-core,:protocol-gb32960,:event-history-service,:vehicle-state-service,:vehicle-stat-service test
```
Expected: BUILD SUCCESS.

38
docs/superpowers/plans/2026-06-29-vehicle-ingest-redesign-phase1.md
View File

@@ -16,7 +16,7 @@ This plan implements only the first slice of the redesign:
- `modules/core/ingest-facts`: protocol-neutral fact records and helpers.
- `modules/sinks/raw-archive-store`: raw archive write/read contract and local URI-safe implementation skeleton.
- `modules/sinks/sink-mq`: protobuf messages for raw frame facts and decoded facts.
- `modules/sinks/sink-kafka`: protobuf messages for raw frame facts and decoded facts.
- Parent Maven wiring and focused tests.
It intentionally does not yet modify:
@@ -53,9 +53,9 @@ Create:
Modify:
- `pom.xml`: add modules and dependency management entries.
- `modules/sinks/sink-mq/pom.xml`: add dependency on `ingest-facts`.
- `modules/sinks/sink-mq/src/main/proto/vehicle_envelope.proto`: add `RawFrameFactPayload`, `DecodedFactPayload`, and related enums/messages.
- `modules/sinks/sink-mq/src/test/java/com/lingniu/ingest/sink/mq/VehicleEnvelopeProtoCompatibilityTest.java`: verify new proto messages can round-trip.
- `modules/sinks/sink-kafka/pom.xml`: add dependency on `ingest-facts`.
- `modules/sinks/sink-kafka/src/main/proto/vehicle_envelope.proto`: add `RawFrameFactPayload`, `DecodedFactPayload`, and related enums/messages.
- `modules/sinks/sink-kafka/src/test/java/com/lingniu/ingest/sink/kafka/VehicleEnvelopeProtoCompatibilityTest.java`: verify new proto messages can round-trip.
---
@@ -1260,9 +1260,9 @@ git commit -m "feat: add raw archive store contract"
**Files:**
- Modify: `modules/sinks/sink-mq/pom.xml`
- Modify: `modules/sinks/sink-mq/src/main/proto/vehicle_envelope.proto`
- Create: `modules/sinks/sink-mq/src/test/java/com/lingniu/ingest/sink/mq/VehicleEnvelopeProtoCompatibilityTest.java`
- Modify: `modules/sinks/sink-kafka/pom.xml`
- Modify: `modules/sinks/sink-kafka/src/main/proto/vehicle_envelope.proto`
- Create: `modules/sinks/sink-kafka/src/test/java/com/lingniu/ingest/sink/kafka/VehicleEnvelopeProtoCompatibilityTest.java`
- [ ] **Step 1: Write failing proto compatibility test**
@@ -1341,14 +1341,14 @@ class VehicleEnvelopeProtoCompatibilityTest {
Run:
```bash
mvn -pl modules/sinks/sink-mq -Dtest=VehicleEnvelopeProtoCompatibilityTest test
mvn -pl modules/sinks/sink-kafka -Dtest=VehicleEnvelopeProtoCompatibilityTest test
```
Expected: FAIL because proto generated classes do not exist.
- [ ] **Step 3: Add `ingest-facts` dependency to sink-mq**
- [ ] **Step 3: Add `ingest-facts` dependency to sink-kafka**
Add this dependency to `modules/sinks/sink-mq/pom.xml` after `ingest-api`:
Add this dependency to `modules/sinks/sink-kafka/pom.xml` after `ingest-api`:
```xml
<dependency>
@@ -1412,17 +1412,17 @@ message DecodedFactPayload {
Run:
```bash
mvn -pl modules/sinks/sink-mq -Dtest=VehicleEnvelopeProtoCompatibilityTest test
mvn -pl modules/sinks/sink-kafka -Dtest=VehicleEnvelopeProtoCompatibilityTest test
```
Expected: PASS.
- [ ] **Step 6: Run existing sink-mq tests**
- [ ] **Step 6: Run existing sink-kafka tests**
Run:
```bash
mvn -pl modules/sinks/sink-mq test
mvn -pl modules/sinks/sink-kafka test
```
Expected: PASS. Existing envelope tests must continue to pass because new proto fields are additive.
@@ -1430,9 +1430,9 @@ Expected: PASS. Existing envelope tests must continue to pass because new proto
- [ ] **Step 7: Commit**
```bash
git add modules/sinks/sink-mq/pom.xml \
modules/sinks/sink-mq/src/main/proto/vehicle_envelope.proto \
modules/sinks/sink-mq/src/test/java/com/lingniu/ingest/sink/mq/VehicleEnvelopeProtoCompatibilityTest.java
git add modules/sinks/sink-kafka/pom.xml \
modules/sinks/sink-kafka/src/main/proto/vehicle_envelope.proto \
modules/sinks/sink-kafka/src/test/java/com/lingniu/ingest/sink/kafka/VehicleEnvelopeProtoCompatibilityTest.java
git commit -m "feat: add fact payloads to kafka envelope"
```
@@ -1449,7 +1449,7 @@ git commit -m "feat: add fact payloads to kafka envelope"
Run:
```bash
mvn -pl modules/core/ingest-facts,modules/sinks/raw-archive-store,modules/sinks/sink-mq -am test
mvn -pl modules/core/ingest-facts,modules/sinks/raw-archive-store,modules/sinks/sink-kafka -am test
```
Expected: PASS.
@@ -1459,7 +1459,7 @@ Expected: PASS.
Run:
```bash
mvn -pl modules/sinks/sink-mq -am -DskipTests package
mvn -pl modules/sinks/sink-kafka -am -DskipTests package
```
Expected: PASS. Generated protobuf Java sources include `RawFrameFactPayload`, `DecodedFactPayload`, and `ParseStatusProto`.
@@ -1473,7 +1473,7 @@ rg -n "RawFrameFact|DecodedFact|RawArchiveWriter|vehicle.raw-frame.v1|vehicle.de
modules/apps modules/protocols modules/services modules/core modules/sinks
```
Expected: Matches only in the new modules, sink-mq proto/test, and planned references. No existing handler behavior should be changed in Phase 1.
Expected: Matches only in the new modules, sink-kafka proto/test, and planned references. No existing handler behavior should be changed in Phase 1.
- [ ] **Step 4: Commit final verification note if any docs changed**

6
docs/superpowers/specs/2026-06-23-gb32960-service-split-design.md
View File

@@ -60,7 +60,7 @@ Initial module dependencies:
- `ingest-codec-common`
- `session-core`
- `protocol-gb32960`
- `sink-mq`
- `sink-kafka`
- `observability`
### `vehicle-history-app`
@@ -86,7 +86,7 @@ Non-responsibilities:
Initial module dependencies:
- `ingest-api`
- `sink-mq`
- `sink-kafka`
- `sink-archive`
- `event-file-store`
- `event-history-service`
@@ -116,7 +116,7 @@ Non-responsibilities:
Initial module dependencies:
- `ingest-api`
- `sink-mq`
- `sink-kafka`
- `vehicle-state-service`
- `vehicle-stat-service`
- `observability`