shishengliang/lingniu-vehicle-ingest
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

refactor: rename kafka sink module

Browse Source
This commit is contained in:
lingniu
2026-07-01 10:56:33 +08:00
parent 849e8f8a8b
commit e80fa6f38c
43 changed files with 87 additions and 66 deletions
Download Patch File Download Diff File Expand all files Collapse all files

2
docs/superpowers/plans/2026-04-20-gb32960-body-parser-block-isolation.md
View File

@@ -543,7 +543,7 @@ EOF
Run: `mvn test -q`
Expected: BUILD SUCCESS。重点关注：
- `protocol-gb32960` 模块全绿（Golden、FullBlocks、Isolation、GuangdongFcEndToEnd、Decoder、Mapper、所有 parser 单测、profile）
- 其它模块（`ingest-core`、`sink-mq` 等）不受影响（无改动应自动通过）
- 其它模块（`ingest-core`、`sink-kafka` 等）不受影响（无改动应自动通过）
若有红，停下来诊断。

46
docs/superpowers/plans/2026-06-23-gb32960-service-split.md
View File

@@ -48,11 +48,11 @@ Create app modules:
Modify shared build and Kafka contract files:
- `pom.xml`
- `modules/sinks/sink-mq/src/main/java/com/lingniu/ingest/sink/mq/KafkaSinkProperties.java`
- `modules/sinks/sink-mq/src/main/java/com/lingniu/ingest/sink/mq/TopicRouter.java`
- `modules/sinks/sink-mq/src/main/java/com/lingniu/ingest/sink/mq/KafkaEventSink.java`
- `modules/sinks/sink-mq/src/test/java/com/lingniu/ingest/sink/mq/TopicRouterTest.java`
- `modules/sinks/sink-mq/src/test/java/com/lingniu/ingest/sink/mq/KafkaEventSinkTest.java`
- `modules/sinks/sink-kafka/src/main/java/com/lingniu/ingest/sink/kafka/KafkaSinkProperties.java`
- `modules/sinks/sink-kafka/src/main/java/com/lingniu/ingest/sink/kafka/TopicRouter.java`
- `modules/sinks/sink-kafka/src/main/java/com/lingniu/ingest/sink/kafka/KafkaEventSink.java`
- `modules/sinks/sink-kafka/src/test/java/com/lingniu/ingest/sink/kafka/TopicRouterTest.java`
- `modules/sinks/sink-kafka/src/test/java/com/lingniu/ingest/sink/kafka/KafkaEventSinkTest.java`
Modify ACK/Kafka durability boundary files:
@@ -157,7 +157,7 @@ Create `modules/apps/gb32960-ingest-app/pom.xml`:
</dependency>
<dependency>
<groupId>com.lingniu.ingest</groupId>
<artifactId>sink-mq</artifactId>
<artifactId>sink-kafka</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
@@ -222,7 +222,7 @@ Create `modules/apps/vehicle-history-app/pom.xml`:
</dependency>
<dependency>
<groupId>com.lingniu.ingest</groupId>
<artifactId>sink-mq</artifactId>
<artifactId>sink-kafka</artifactId>
</dependency>
<dependency>
<groupId>com.lingniu.ingest</groupId>
@@ -303,7 +303,7 @@ Create `modules/apps/vehicle-analytics-app/pom.xml`:
</dependency>
<dependency>
<groupId>com.lingniu.ingest</groupId>
<artifactId>sink-mq</artifactId>
<artifactId>sink-kafka</artifactId>
</dependency>
<dependency>
<groupId>com.lingniu.ingest</groupId>
@@ -933,15 +933,15 @@ git commit -m "test: cover split service composition"
### Task 4: Version Kafka Topics for GB32960 Raw/Event/DLQ
**Files:**
- Modify: `modules/sinks/sink-mq/src/main/java/com/lingniu/ingest/sink/mq/KafkaSinkProperties.java`
- Modify: `modules/sinks/sink-mq/src/main/java/com/lingniu/ingest/sink/mq/TopicRouter.java`
- Modify: `modules/sinks/sink-mq/src/main/java/com/lingniu/ingest/sink/mq/KafkaEventSink.java`
- Create or modify: `modules/sinks/sink-mq/src/test/java/com/lingniu/ingest/sink/mq/TopicRouterTest.java`
- Create or modify: `modules/sinks/sink-mq/src/test/java/com/lingniu/ingest/sink/mq/KafkaEventSinkTest.java`
- Modify: `modules/sinks/sink-kafka/src/main/java/com/lingniu/ingest/sink/kafka/KafkaSinkProperties.java`
- Modify: `modules/sinks/sink-kafka/src/main/java/com/lingniu/ingest/sink/kafka/TopicRouter.java`
- Modify: `modules/sinks/sink-kafka/src/main/java/com/lingniu/ingest/sink/kafka/KafkaEventSink.java`
- Create or modify: `modules/sinks/sink-kafka/src/test/java/com/lingniu/ingest/sink/kafka/TopicRouterTest.java`
- Create or modify: `modules/sinks/sink-kafka/src/test/java/com/lingniu/ingest/sink/kafka/KafkaEventSinkTest.java`
- [ ] **Step 1: Write topic routing tests**
Create `modules/sinks/sink-mq/src/test/java/com/lingniu/ingest/sink/mq/TopicRouterTest.java` if it does not exist. Add tests proving GB32960 raw and normalized events can route to versioned topics. Use the existing `VehicleEvent` constructors from `EventFileStoreSinkTest` and `EnvelopeMapperTelemetrySnapshotTest` as examples.
Create `modules/sinks/sink-kafka/src/test/java/com/lingniu/ingest/sink/kafka/TopicRouterTest.java` if it does not exist. Add tests proving GB32960 raw and normalized events can route to versioned topics. Use the existing `VehicleEvent` constructors from `EventFileStoreSinkTest` and `EnvelopeMapperTelemetrySnapshotTest` as examples.
Expected test names:
@@ -996,7 +996,7 @@ Adjust constructor arguments to match the current sealed `VehicleEvent` definiti
Run:
```bash
mvn -pl :sink-mq -Dtest=TopicRouterTest test
mvn -pl :sink-kafka -Dtest=TopicRouterTest test
```
Expected before implementation: compile failure if constructors are wrong or assertion failure if defaults are old. Fix constructors first; keep routing assertions.
@@ -1028,12 +1028,12 @@ public boolean accepts(VehicleEvent event) {
Update the JavaDoc above it to say Kafka carries both normalized events and raw archive records in split-service mode.
- [ ] **Step 5: Run sink-mq tests**
- [ ] **Step 5: Run sink-kafka tests**
Run:
```bash
mvn -pl :sink-mq test
mvn -pl :sink-kafka test
```
Expected: PASS.
@@ -1041,11 +1041,11 @@ Expected: PASS.
- [ ] **Step 6: Commit**
```bash
git add modules/sinks/sink-mq/src/main/java/com/lingniu/ingest/sink/mq/KafkaSinkProperties.java \
modules/sinks/sink-mq/src/main/java/com/lingniu/ingest/sink/mq/TopicRouter.java \
modules/sinks/sink-mq/src/main/java/com/lingniu/ingest/sink/mq/KafkaEventSink.java \
modules/sinks/sink-mq/src/test/java/com/lingniu/ingest/sink/mq/TopicRouterTest.java \
modules/sinks/sink-mq/src/test/java/com/lingniu/ingest/sink/mq/KafkaEventSinkTest.java
git add modules/sinks/sink-kafka/src/main/java/com/lingniu/ingest/sink/kafka/KafkaSinkProperties.java \
modules/sinks/sink-kafka/src/main/java/com/lingniu/ingest/sink/kafka/TopicRouter.java \
modules/sinks/sink-kafka/src/main/java/com/lingniu/ingest/sink/kafka/KafkaEventSink.java \
modules/sinks/sink-kafka/src/test/java/com/lingniu/ingest/sink/kafka/TopicRouterTest.java \
modules/sinks/sink-kafka/src/test/java/com/lingniu/ingest/sink/kafka/KafkaEventSinkTest.java
git commit -m "feat: route gb32960 kafka topics"
```
@@ -1567,7 +1567,7 @@ git commit -m "docs: add gb32960 split service runbook"
Run:
```bash
mvn -pl :sink-mq,:ingest-core,:protocol-gb32960,:event-history-service,:vehicle-state-service,:vehicle-stat-service test
mvn -pl :sink-kafka,:ingest-core,:protocol-gb32960,:event-history-service,:vehicle-state-service,:vehicle-stat-service test
```
Expected: BUILD SUCCESS.

38
docs/superpowers/plans/2026-06-29-vehicle-ingest-redesign-phase1.md
View File

@@ -16,7 +16,7 @@ This plan implements only the first slice of the redesign:
- `modules/core/ingest-facts`: protocol-neutral fact records and helpers.
- `modules/sinks/raw-archive-store`: raw archive write/read contract and local URI-safe implementation skeleton.
- `modules/sinks/sink-mq`: protobuf messages for raw frame facts and decoded facts.
- `modules/sinks/sink-kafka`: protobuf messages for raw frame facts and decoded facts.
- Parent Maven wiring and focused tests.
It intentionally does not yet modify:
@@ -53,9 +53,9 @@ Create:
Modify:
- `pom.xml`: add modules and dependency management entries.
- `modules/sinks/sink-mq/pom.xml`: add dependency on `ingest-facts`.
- `modules/sinks/sink-mq/src/main/proto/vehicle_envelope.proto`: add `RawFrameFactPayload`, `DecodedFactPayload`, and related enums/messages.
- `modules/sinks/sink-mq/src/test/java/com/lingniu/ingest/sink/mq/VehicleEnvelopeProtoCompatibilityTest.java`: verify new proto messages can round-trip.
- `modules/sinks/sink-kafka/pom.xml`: add dependency on `ingest-facts`.
- `modules/sinks/sink-kafka/src/main/proto/vehicle_envelope.proto`: add `RawFrameFactPayload`, `DecodedFactPayload`, and related enums/messages.
- `modules/sinks/sink-kafka/src/test/java/com/lingniu/ingest/sink/kafka/VehicleEnvelopeProtoCompatibilityTest.java`: verify new proto messages can round-trip.
---
@@ -1260,9 +1260,9 @@ git commit -m "feat: add raw archive store contract"
**Files:**
- Modify: `modules/sinks/sink-mq/pom.xml`
- Modify: `modules/sinks/sink-mq/src/main/proto/vehicle_envelope.proto`
- Create: `modules/sinks/sink-mq/src/test/java/com/lingniu/ingest/sink/mq/VehicleEnvelopeProtoCompatibilityTest.java`
- Modify: `modules/sinks/sink-kafka/pom.xml`
- Modify: `modules/sinks/sink-kafka/src/main/proto/vehicle_envelope.proto`
- Create: `modules/sinks/sink-kafka/src/test/java/com/lingniu/ingest/sink/kafka/VehicleEnvelopeProtoCompatibilityTest.java`
- [ ] **Step 1: Write failing proto compatibility test**
@@ -1341,14 +1341,14 @@ class VehicleEnvelopeProtoCompatibilityTest {
Run:
```bash
mvn -pl modules/sinks/sink-mq -Dtest=VehicleEnvelopeProtoCompatibilityTest test
mvn -pl modules/sinks/sink-kafka -Dtest=VehicleEnvelopeProtoCompatibilityTest test
```
Expected: FAIL because proto generated classes do not exist.
- [ ] **Step 3: Add `ingest-facts` dependency to sink-mq**
- [ ] **Step 3: Add `ingest-facts` dependency to sink-kafka**
Add this dependency to `modules/sinks/sink-mq/pom.xml` after `ingest-api`:
Add this dependency to `modules/sinks/sink-kafka/pom.xml` after `ingest-api`:
```xml
<dependency>
@@ -1412,17 +1412,17 @@ message DecodedFactPayload {
Run:
```bash
mvn -pl modules/sinks/sink-mq -Dtest=VehicleEnvelopeProtoCompatibilityTest test
mvn -pl modules/sinks/sink-kafka -Dtest=VehicleEnvelopeProtoCompatibilityTest test
```
Expected: PASS.
- [ ] **Step 6: Run existing sink-mq tests**
- [ ] **Step 6: Run existing sink-kafka tests**
Run:
```bash
mvn -pl modules/sinks/sink-mq test
mvn -pl modules/sinks/sink-kafka test
```
Expected: PASS. Existing envelope tests must continue to pass because new proto fields are additive.
@@ -1430,9 +1430,9 @@ Expected: PASS. Existing envelope tests must continue to pass because new proto
- [ ] **Step 7: Commit**
```bash
git add modules/sinks/sink-mq/pom.xml \
modules/sinks/sink-mq/src/main/proto/vehicle_envelope.proto \
modules/sinks/sink-mq/src/test/java/com/lingniu/ingest/sink/mq/VehicleEnvelopeProtoCompatibilityTest.java
git add modules/sinks/sink-kafka/pom.xml \
modules/sinks/sink-kafka/src/main/proto/vehicle_envelope.proto \
modules/sinks/sink-kafka/src/test/java/com/lingniu/ingest/sink/kafka/VehicleEnvelopeProtoCompatibilityTest.java
git commit -m "feat: add fact payloads to kafka envelope"
```
@@ -1449,7 +1449,7 @@ git commit -m "feat: add fact payloads to kafka envelope"
Run:
```bash
mvn -pl modules/core/ingest-facts,modules/sinks/raw-archive-store,modules/sinks/sink-mq -am test
mvn -pl modules/core/ingest-facts,modules/sinks/raw-archive-store,modules/sinks/sink-kafka -am test
```
Expected: PASS.
@@ -1459,7 +1459,7 @@ Expected: PASS.
Run:
```bash
mvn -pl modules/sinks/sink-mq -am -DskipTests package
mvn -pl modules/sinks/sink-kafka -am -DskipTests package
```
Expected: PASS. Generated protobuf Java sources include `RawFrameFactPayload`, `DecodedFactPayload`, and `ParseStatusProto`.
@@ -1473,7 +1473,7 @@ rg -n "RawFrameFact|DecodedFact|RawArchiveWriter|vehicle.raw-frame.v1|vehicle.de
modules/apps modules/protocols modules/services modules/core modules/sinks
```
Expected: Matches only in the new modules, sink-mq proto/test, and planned references. No existing handler behavior should be changed in Phase 1.
Expected: Matches only in the new modules, sink-kafka proto/test, and planned references. No existing handler behavior should be changed in Phase 1.
- [ ] **Step 4: Commit final verification note if any docs changed**