Merge branch 'codex/gb32960-service-split'
This commit is contained in:
lingniu
224
docs/operations/gb32960-service-split-runbook.md
Normal file
224
docs/operations/gb32960-service-split-runbook.md
Normal file
@@ -0,0 +1,224 @@
|
||||
# GB32960 Split Service Runbook
|
||||
|
||||
This runbook covers the split GB32960 ingest, history, and analytics runtimes. Kafka is the durability boundary between the protocol service and downstream business services.
|
||||
|
||||
## Service Roles
|
||||
|
||||
| Service | Module | Role | Local ports |
|
||||
| --- | --- | --- | --- |
|
||||
| GB32960 ingest | `:gb32960-ingest-app` | Accept GB32960 TCP connections, decode/authenticate frames, produce raw and normalized records to Kafka, and send protocol ACKs after required Kafka production succeeds. | TCP `32960`, HTTP `20100` |
|
||||
| Vehicle history | `:vehicle-history-app` | Consume GB32960 raw/event Kafka records, store event-history records and raw archive references, and expose event/history/GB32960 query APIs. Kafka raw envelopes carry archive URI/size metadata, not raw frame bytes. | HTTP `20200` |
|
||||
| Vehicle analytics | `:vehicle-analytics-app` | Consume GB32960 event Kafka records and update analytics outputs such as daily vehicle statistics. Vehicle state can also be enabled from this runtime. | HTTP `20300` |
|
||||
|
||||
## Kafka Contract
|
||||
|
||||
| Topic | Producer | Consumers | Purpose |
|
||||
| --- | --- | --- | --- |
|
||||
| `vehicle.raw.gb32960.v1` | `gb32960-ingest-app` | `vehicle-history-app` | Raw archive reference records. The envelope contains URI/size metadata, not the raw `.bin` payload. |
|
||||
| `vehicle.event.gb32960.v1` | `gb32960-ingest-app` | `vehicle-history-app`, `vehicle-analytics-app` | Normalized vehicle events keyed by VIN when available. |
|
||||
| `vehicle.dlq.gb32960.v1` | Kafka producer/consumer error paths | Operators/replay tooling | Dead-letter records for failed production or consumer processing. |
|
||||
|
||||
Default local consumer groups:
|
||||
|
||||
- History: `vehicle-history`
|
||||
- Analytics state: `vehicle-state` when `VEHICLE_STATE_ENABLED=true`
|
||||
- Analytics statistics: `vehicle-stat`
|
||||
|
||||
## ACK Semantics
|
||||
|
||||
GB32960 success ACKs are sent only after the ingest service completes the required Kafka production boundary for the accepted frame. If required Kafka production fails, the ingest service must not claim a successful ACK for that frame.
|
||||
|
||||
History and analytics failures do not block GB32960 ACKs. They are downstream Kafka consumer failures and should be handled through retry, DLQ inspection, offset replay, and service-specific recovery.
|
||||
|
||||
Authorization failures are rejected before the Kafka durability boundary and can be ACKed/rejected immediately according to protocol handling.
|
||||
|
||||
## Local Prerequisites
|
||||
|
||||
- Java 25 and Maven available on PATH.
|
||||
- Local Kafka reachable at `127.0.0.1:9092`, plus Kafka CLI tools such as `kafka-topics` and optionally `kafka-console-consumer`.
|
||||
- No repository-local Kafka bootstrap script or compose file was found during Task 8 verification on 2026-06-23.
|
||||
|
||||
If your shell needs an explicit JDK:
|
||||
|
||||
```bash
|
||||
export JAVA_HOME=/opt/homebrew/opt/openjdk@25/libexec/openjdk.jdk/Contents/Home
|
||||
export PATH="$JAVA_HOME/bin:/opt/homebrew/bin:$PATH"
|
||||
```
|
||||
|
||||
## Build
|
||||
|
||||
From the repository root:
|
||||
|
||||
```bash
|
||||
mvn -pl :gb32960-ingest-app,:vehicle-history-app,:vehicle-analytics-app -am package -Dmaven.test.skip=true
|
||||
```
|
||||
|
||||
Expected result: `BUILD SUCCESS` and these runnable jars:
|
||||
|
||||
- `modules/apps/gb32960-ingest-app/target/gb32960-ingest-app.jar`
|
||||
- `modules/apps/vehicle-history-app/target/vehicle-history-app.jar`
|
||||
- `modules/apps/vehicle-analytics-app/target/vehicle-analytics-app.jar`
|
||||
|
||||
## Create Topics
|
||||
|
||||
Run these only after Kafka is reachable at `127.0.0.1:9092`:
|
||||
|
||||
```bash
|
||||
kafka-topics --bootstrap-server 127.0.0.1:9092 --create --if-not-exists --topic vehicle.raw.gb32960.v1 --partitions 12 --replication-factor 1
|
||||
kafka-topics --bootstrap-server 127.0.0.1:9092 --create --if-not-exists --topic vehicle.event.gb32960.v1 --partitions 12 --replication-factor 1
|
||||
kafka-topics --bootstrap-server 127.0.0.1:9092 --create --if-not-exists --topic vehicle.dlq.gb32960.v1 --partitions 3 --replication-factor 1
|
||||
```
|
||||
|
||||
Optional sanity check:
|
||||
|
||||
```bash
|
||||
kafka-topics --bootstrap-server 127.0.0.1:9092 --list | grep -E 'vehicle\.(raw|event|dlq)\.gb32960\.v1'
|
||||
```
|
||||
|
||||
## Start Split Services Locally
|
||||
|
||||
Run each command in a separate terminal from the repository root.
|
||||
|
||||
GB32960 ingest:
|
||||
|
||||
```bash
|
||||
KAFKA_BROKERS=127.0.0.1:9092 \
|
||||
GB32960_PORT=32960 \
|
||||
HTTP_PORT=20100 \
|
||||
java --sun-misc-unsafe-memory-access=allow \
|
||||
-jar modules/apps/gb32960-ingest-app/target/gb32960-ingest-app.jar
|
||||
```
|
||||
|
||||
Vehicle history:
|
||||
|
||||
```bash
|
||||
KAFKA_BROKERS=127.0.0.1:9092 \
|
||||
HTTP_PORT=20200 \
|
||||
EVENT_FILE_STORE_PATH=./target/split-event-store \
|
||||
java --sun-misc-unsafe-memory-access=allow \
|
||||
-jar modules/apps/vehicle-history-app/target/vehicle-history-app.jar
|
||||
```
|
||||
|
||||
`SINK_ARCHIVE_PATH` is intentionally omitted in the split-service Kafka verification path. Current Kafka raw envelopes contain `RawArchiveRef` URI/size metadata only; they do not contain raw bytes that `RawArchiveEventSink` could materialize into `.bin` files in the history JVM.
|
||||
|
||||
Vehicle analytics:
|
||||
|
||||
```bash
|
||||
KAFKA_BROKERS=127.0.0.1:9092 \
|
||||
HTTP_PORT=20300 \
|
||||
VEHICLE_STAT_FILE_PATH=./target/split-vehicle-stat \
|
||||
java --sun-misc-unsafe-memory-access=allow \
|
||||
-jar modules/apps/vehicle-analytics-app/target/vehicle-analytics-app.jar
|
||||
```
|
||||
|
||||
To enable the latest-state API in analytics, also set `VEHICLE_STATE_ENABLED=true` and configure the state repository requirements used by `vehicle-state-service`.
|
||||
|
||||
## Health Verification
|
||||
|
||||
```bash
|
||||
curl -sS http://127.0.0.1:20100/actuator/health
|
||||
curl -sS http://127.0.0.1:20200/actuator/health
|
||||
curl -sS http://127.0.0.1:20300/actuator/health
|
||||
```
|
||||
|
||||
Expected: each endpoint returns `{"status":"UP"}` or an equivalent Spring Boot health JSON with top-level status `UP`.
|
||||
|
||||
## End-to-End Verification
|
||||
|
||||
Use a known-valid GB32960 fixture, for example one of the hex samples under `modules/protocols/protocol-gb32960/src/test/resources/samples/gb32960/`.
|
||||
|
||||
Example send command:
|
||||
|
||||
```bash
|
||||
xxd -r -p modules/protocols/protocol-gb32960/src/test/resources/samples/gb32960/realtime_001.hex | nc 127.0.0.1 32960
|
||||
```
|
||||
|
||||
To capture and inspect the binary ACK, write the response to a file:
|
||||
|
||||
```bash
|
||||
rm -f /tmp/gb32960-ack.bin
|
||||
xxd -r -p modules/protocols/protocol-gb32960/src/test/resources/samples/gb32960/realtime_001.hex \
|
||||
| nc -w 3 127.0.0.1 32960 > /tmp/gb32960-ack.bin
|
||||
xxd -p /tmp/gb32960-ack.bin
|
||||
```
|
||||
|
||||
Expected: the ACK file is non-empty and the decoded bytes begin with the GB32960 frame marker `2323`. Treat the ACK as verified only after checking the response bytes from your run.
|
||||
|
||||
Verify only what was actually run in your environment:
|
||||
|
||||
```bash
|
||||
kafka-console-consumer --bootstrap-server 127.0.0.1:9092 --topic vehicle.raw.gb32960.v1 --from-beginning --max-messages 1 --timeout-ms 10000
|
||||
kafka-console-consumer --bootstrap-server 127.0.0.1:9092 --topic vehicle.event.gb32960.v1 --from-beginning --max-messages 1 --timeout-ms 10000
|
||||
find target/split-event-store -type f | head
|
||||
find target/split-vehicle-stat -type f | head
|
||||
```
|
||||
|
||||
Useful HTTP query checks after events are consumed. For `realtime_001.hex`, the fixture VIN is `LTEST000000000001`; replace `<date-from>`, `<date-to>`, and `<stat-date>` with values that cover the consumed record's event time:
|
||||
|
||||
```bash
|
||||
curl -sS 'http://127.0.0.1:20200/api/event-history/records?protocol=GB32960&dateFrom=<date-from>&dateTo=<date-to>&vin=LTEST000000000001&limit=10'
|
||||
curl -sS 'http://127.0.0.1:20200/api/event-history/gb32960/dictionary'
|
||||
curl -sS 'http://127.0.0.1:20300/api/vehicle-stat/LTEST000000000001/daily?date=<stat-date>'
|
||||
```
|
||||
|
||||
Expected E2E result when Kafka and all services are running:
|
||||
|
||||
- A GB32960 client receives a binary success ACK only after required Kafka production succeeds, and the captured ACK bytes are inspected.
|
||||
- `vehicle.raw.gb32960.v1` receives a raw archive reference envelope.
|
||||
- `vehicle.event.gb32960.v1` receives one or more normalized records.
|
||||
- `target/split-event-store` receives event-store files after the history service flushes consumed Kafka records, including raw archive reference records.
|
||||
- `target/split-vehicle-stat` receives stat output after analytics consumes applicable event records.
|
||||
|
||||
Do not expect `vehicle-history-app` to create raw `.bin` archive files from Kafka raw records in the current implementation. `RawArchiveEventSink` can write archive files only when it receives `VehicleEvent.RawArchive.rawBytes()` inside the same JVM; the Kafka envelope carries only `RawArchiveRef` metadata.
|
||||
|
||||
Do not mark any of these as verified unless the matching command was run and the output was inspected.
|
||||
|
||||
## Rollback Guidance
|
||||
|
||||
If the split deployment is unhealthy:
|
||||
|
||||
1. Stop `gb32960-ingest-app` first so new GB32960 ACKs are not issued against an unhealthy Kafka boundary.
|
||||
2. Keep Kafka topics intact for replay unless storage or privacy policy requires deletion.
|
||||
3. Restart or roll back `vehicle-history-app` and `vehicle-analytics-app` independently; their failures do not require protocol ACK rollback because they consume from Kafka offsets.
|
||||
4. If Kafka itself is unavailable, route GB32960 traffic back to the previous all-in-one runtime only if that runtime is configured with its known-good durability behavior.
|
||||
5. After rollback, compare Kafka consumer group lag and DLQ contents before resuming the split services.
|
||||
|
||||
Rollback commands depend on the deployment environment. For local testing, stop the three Java processes and restart the previous `bootstrap-all` workflow if needed.
|
||||
|
||||
## Task 8 Verification Notes
|
||||
|
||||
Observed on 2026-06-23 in worktree `.worktrees/gb32960-service-split`:
|
||||
|
||||
- Package build ran with `mvn -pl :gb32960-ingest-app,:vehicle-history-app,:vehicle-analytics-app -am package -Dmaven.test.skip=true` and ended with `BUILD SUCCESS`.
|
||||
- Repository-local Kafka setup inspection found no Kafka script, no Docker Compose file, and no compose YAML within the searched repository paths.
|
||||
- `nc -z -w 2 127.0.0.1 9092` exited `1`, so no local Kafka broker was reachable at `127.0.0.1:9092`.
|
||||
- `kafka-topics`, `kafka-topics.sh`, `docker`, and `docker-compose` were not found on PATH.
|
||||
- Kafka topic creation, service startup, health checks, Kafka record checks, archive checks, event-store checks, stat output checks, and ACK observation were not run because the local Kafka prerequisite was absent.
|
||||
|
||||
## Latest Local Verification
|
||||
|
||||
Observed on 2026-06-23 in worktree `.worktrees/gb32960-service-split`:
|
||||
|
||||
- Targeted module tests: `mvn -pl :sink-mq,:ingest-core,:protocol-gb32960,:event-history-service,:vehicle-state-service,:vehicle-stat-service test` ended with `BUILD SUCCESS`; Maven reported 55 protocol/sink/core tests, 30 event-history tests, 14 vehicle-state tests, and 19 vehicle-stat tests with 0 failures/errors/skips.
|
||||
- Split app tests: `mvn -pl :gb32960-ingest-app,:vehicle-history-app,:vehicle-analytics-app test` ended with `BUILD SUCCESS`; Maven reported 6 app composition/default tests with 0 failures/errors/skips.
|
||||
- Package verification: `mvn -pl :gb32960-ingest-app,:vehicle-history-app,:vehicle-analytics-app -am package -Dmaven.test.skip=true` ended with `BUILD SUCCESS` and repackaged all three app jars.
|
||||
- Split app startup: not run in this verification pass because local Kafka was still absent.
|
||||
- Kafka raw records: not verified; no local Kafka broker or Kafka CLI tools were available.
|
||||
- Kafka event records: not verified; no local Kafka broker or Kafka CLI tools were available.
|
||||
- Event-store files: not verified; downstream services were not started without Kafka.
|
||||
- Analytics output: not verified; downstream services were not started without Kafka.
|
||||
- ACK behavior: not verified against a live broker in this pass. Unit tests cover the GB32960 ACK boundary and ordering, but an operator should still run the ACK capture command above in a Kafka-backed local or staging environment.
|
||||
|
||||
## Remote Kafka Live Verification
|
||||
|
||||
Observed on 2026-06-23 with Kafka `114.55.58.251:9092`:
|
||||
|
||||
- Replaced the local `bootstrap-all` listener on TCP `32960` with `gb32960-ingest-app` on TCP `32960` and HTTP `20100`.
|
||||
- Ingest app health check `curl -sS http://127.0.0.1:20100/actuator/health` returned `{"status":"UP"}`.
|
||||
- Kafka AdminClient confirmed topics `vehicle.raw.gb32960.v1` and `vehicle.event.gb32960.v1` existed, and created `vehicle.dlq.gb32960.v1`.
|
||||
- The app accepted the live Hyundai platform connection from `8.134.95.166`, authenticated platform login, and received real GB32960 realtime/report/login/logout frames.
|
||||
- Probe consumer with a temporary group read records from `vehicle.event.gb32960.v1`, including VIN keys such as `LB9A32A2XR0LS1575` and `LNXNEGRR3SR319485`.
|
||||
- Probe consumer with a temporary group read records from `vehicle.raw.gb32960.v1`, including VIN keys such as `LNXNEGRR0SR319444`, `LNXNEGRR2SR318201`, and `LB9A32A23R0LS1532`.
|
||||
- Runtime logs showed successful GB32960 ACK flushes for platform login, vehicle login, and vehicle logout after the Kafka-backed split ingest service was running.
|
||||
|
||||
This verifies the live path `GB32960 TCP 32960 -> gb32960-ingest-app -> Kafka raw/event topics` against the remote Kafka broker. Downstream `vehicle-history-app` and `vehicle-analytics-app` were not started in this pass.
|
||||
@@ -0,0 +1,810 @@
|
||||
# GB32960 History DuckDB Hot Store Implementation Plan
|
||||
|
||||
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
|
||||
|
||||
**Goal:** Replace the current high-write-risk Parquet rewrite history path with a DuckDB hot history store that supports fast `vin + time` queries and full RAW frame replay.
|
||||
|
||||
**Architecture:** Add a new `DuckDbHotEventFileStore` implementation behind the existing `EventFileStore` interface, backed by append/upsert DuckDB tables instead of rewriting per-vehicle Parquet files. Keep the current HTTP history API and RAW archive reader intact, then switch `vehicle-history-app` to the hot DuckDB store by configuration.
|
||||
|
||||
**Tech Stack:** Java 26, Spring Boot, DuckDB JDBC, JUnit 5, AssertJ, existing Kafka `VehicleEnvelope`, existing `ArchiveStore`.
|
||||
|
||||
---
|
||||
|
||||
## Scope
|
||||
|
||||
This plan implements Phase 1 from `docs/superpowers/specs/2026-06-23-gb32960-production-readiness-design.md`.
|
||||
|
||||
Included:
|
||||
|
||||
- DuckDB hot event table for `EventFileRecord`.
|
||||
- Idempotent writes by `event_id`.
|
||||
- Query by protocol/date/VIN/eventType/eventTime/order/limit.
|
||||
- Lookup by `rawArchiveUri`.
|
||||
- Spring configuration to select the hot store.
|
||||
- Tests that prove history endpoints still replay RAW frames.
|
||||
- Local verification using the existing `vehicle-history-app`.
|
||||
|
||||
Not included in this plan:
|
||||
|
||||
- MySQL daily statistics.
|
||||
- VIN-to-platform local mapping.
|
||||
- Full telemetry point columnar table.
|
||||
- Alarm timeline MySQL tables.
|
||||
- Cold Parquet export.
|
||||
|
||||
Those will be separate plans after this foundation lands.
|
||||
|
||||
## Files
|
||||
|
||||
- Create: `modules/sinks/event-file-store/src/main/java/com/lingniu/ingest/eventfilestore/DuckDbHotEventFileStore.java`
|
||||
- Create: `modules/sinks/event-file-store/src/test/java/com/lingniu/ingest/eventfilestore/DuckDbHotEventFileStoreTest.java`
|
||||
- Modify: `modules/sinks/event-file-store/src/main/java/com/lingniu/ingest/eventfilestore/config/EventFileStoreProperties.java`
|
||||
- Modify: `modules/sinks/event-file-store/src/main/java/com/lingniu/ingest/eventfilestore/config/EventFileStoreAutoConfiguration.java`
|
||||
- Modify: `modules/sinks/event-file-store/src/test/java/com/lingniu/ingest/eventfilestore/config/EventFileStoreAutoConfigurationTest.java`
|
||||
- Modify: `modules/apps/vehicle-history-app/src/main/resources/application.yml`
|
||||
- Modify: `modules/apps/vehicle-history-app/src/test/java/com/lingniu/ingest/historyapp/VehicleHistoryAppDefaultsTest.java`
|
||||
- Modify: `modules/services/event-history-service/src/test/java/com/lingniu/ingest/eventhistory/Gb32960DecodedFrameServiceTest.java`
|
||||
|
||||
## Task 1: Add Hot Store Configuration
|
||||
|
||||
**Files:**
|
||||
- Modify: `modules/sinks/event-file-store/src/main/java/com/lingniu/ingest/eventfilestore/config/EventFileStoreProperties.java`
|
||||
- Modify: `modules/sinks/event-file-store/src/main/java/com/lingniu/ingest/eventfilestore/config/EventFileStoreAutoConfiguration.java`
|
||||
- Test: `modules/sinks/event-file-store/src/test/java/com/lingniu/ingest/eventfilestore/config/EventFileStoreAutoConfigurationTest.java`
|
||||
|
||||
- [ ] **Step 1: Write the failing auto-configuration test**
|
||||
|
||||
Add a test that sets `lingniu.ingest.event-file-store.storage=duckdb-hot` and expects the bean class to be `DuckDbHotEventFileStore`.
|
||||
|
||||
```java
|
||||
@Test
|
||||
void createsDuckDbHotStoreWhenStorageIsDuckDbHot() {
|
||||
contextRunner
|
||||
.withPropertyValues(
|
||||
"lingniu.ingest.event-file-store.enabled=true",
|
||||
"lingniu.ingest.event-file-store.storage=duckdb-hot",
|
||||
"lingniu.ingest.event-file-store.path=" + tempDir.resolve("history"))
|
||||
.run(context -> assertThat(context)
|
||||
.hasSingleBean(EventFileStore.class)
|
||||
.getBean(EventFileStore.class)
|
||||
.isInstanceOf(DuckDbHotEventFileStore.class));
|
||||
}
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Run the new test and verify it fails**
|
||||
|
||||
Run:
|
||||
|
||||
```bash
|
||||
mvn -pl :event-file-store -Dtest=EventFileStoreAutoConfigurationTest#createsDuckDbHotStoreWhenStorageIsDuckDbHot test
|
||||
```
|
||||
|
||||
Expected: compilation failure because `DuckDbHotEventFileStore` and `storage` property do not exist.
|
||||
|
||||
- [ ] **Step 3: Add the `storage` property**
|
||||
|
||||
Add to `EventFileStoreProperties`:
|
||||
|
||||
```java
|
||||
/**
|
||||
* Storage backend. `duckdb-hot` is the production backend; `parquet-sidecar`
|
||||
* keeps the previous implementation available for compatibility tests.
|
||||
*/
|
||||
private String storage = "duckdb-hot";
|
||||
|
||||
public String getStorage() {
|
||||
return storage;
|
||||
}
|
||||
|
||||
public void setStorage(String storage) {
|
||||
this.storage = storage;
|
||||
}
|
||||
```
|
||||
|
||||
- [ ] **Step 4: Switch auto-configuration by storage mode**
|
||||
|
||||
Change `eventFileStore(...)` to:
|
||||
|
||||
```java
|
||||
@Bean
|
||||
@ConditionalOnMissingBean
|
||||
public EventFileStore eventFileStore(EventFileStoreProperties properties,
|
||||
ObjectProvider<ObjectMapper> objectMapper) {
|
||||
ObjectMapper mapper = mapper(objectMapper);
|
||||
Path root = Path.of(properties.getPath());
|
||||
ZoneId zoneId = ZoneId.of(properties.getZoneId());
|
||||
String storage = properties.getStorage() == null ? "" : properties.getStorage().trim();
|
||||
return switch (storage) {
|
||||
case "", "duckdb-hot" -> new DuckDbHotEventFileStore(root, zoneId, mapper);
|
||||
case "parquet-sidecar" -> new DuckDbParquetEventFileStore(root, zoneId, mapper);
|
||||
default -> throw new IllegalStateException(
|
||||
"unsupported event-file-store storage: " + properties.getStorage());
|
||||
};
|
||||
}
|
||||
```
|
||||
|
||||
Import `DuckDbHotEventFileStore`.
|
||||
|
||||
- [ ] **Step 5: Run configuration tests**
|
||||
|
||||
Run:
|
||||
|
||||
```bash
|
||||
mvn -pl :event-file-store -Dtest=EventFileStoreAutoConfigurationTest test
|
||||
```
|
||||
|
||||
Expected: tests pass after the store class exists in Task 2.
|
||||
|
||||
## Task 2: Implement DuckDB Hot Store Schema and Writes
|
||||
|
||||
**Files:**
|
||||
- Create: `modules/sinks/event-file-store/src/main/java/com/lingniu/ingest/eventfilestore/DuckDbHotEventFileStore.java`
|
||||
- Create: `modules/sinks/event-file-store/src/test/java/com/lingniu/ingest/eventfilestore/DuckDbHotEventFileStoreTest.java`
|
||||
|
||||
- [ ] **Step 1: Write failing append/query/idempotency tests**
|
||||
|
||||
Create `DuckDbHotEventFileStoreTest`:
|
||||
|
||||
```java
|
||||
class DuckDbHotEventFileStoreTest {
|
||||
|
||||
@TempDir
|
||||
Path tempDir;
|
||||
|
||||
@Test
|
||||
void appendsRecordsAndQueriesByVinTypeAndTime() throws Exception {
|
||||
EventFileStore store = store();
|
||||
store.append(rawRecord("raw-1", "VIN001", "2026-06-23T01:00:00Z"));
|
||||
store.append(rawRecord("raw-2", "VIN002", "2026-06-23T01:00:01Z"));
|
||||
store.append(rawRecord("raw-3", "VIN001", "2026-06-23T01:00:02Z"));
|
||||
|
||||
EventFileQuery query = new EventFileQuery(
|
||||
ProtocolId.GB32960,
|
||||
LocalDate.parse("2026-06-23"),
|
||||
LocalDate.parse("2026-06-23"),
|
||||
Instant.parse("2026-06-23T01:00:00Z"),
|
||||
Instant.parse("2026-06-23T01:00:03Z"),
|
||||
EventFileQuery.Order.DESC,
|
||||
2,
|
||||
"VIN001",
|
||||
"RAW_ARCHIVE");
|
||||
|
||||
assertThat(store.query(query))
|
||||
.extracting(EventFileRecord::eventId)
|
||||
.containsExactly("raw-3", "raw-1");
|
||||
}
|
||||
|
||||
@Test
|
||||
void appendAllIsIdempotentByEventId() throws Exception {
|
||||
EventFileStore store = store();
|
||||
EventFileRecord original = rawRecord("same-id", "VIN001", "2026-06-23T01:00:00Z");
|
||||
EventFileRecord replacement = new EventFileRecord(
|
||||
"same-id",
|
||||
ProtocolId.GB32960,
|
||||
"RAW_ARCHIVE",
|
||||
"VIN001",
|
||||
Instant.parse("2026-06-23T01:00:05Z"),
|
||||
Instant.parse("2026-06-23T01:00:06Z"),
|
||||
"archive://replacement.bin",
|
||||
Map.of("source", "replacement"),
|
||||
"{\"replacement\":true}");
|
||||
|
||||
store.appendAll(List.of(original, replacement));
|
||||
|
||||
assertThat(store.query(new EventFileQuery(
|
||||
ProtocolId.GB32960,
|
||||
LocalDate.parse("2026-06-23"),
|
||||
LocalDate.parse("2026-06-23"),
|
||||
EventFileQuery.Order.ASC,
|
||||
10,
|
||||
"VIN001",
|
||||
"RAW_ARCHIVE")))
|
||||
.singleElement()
|
||||
.satisfies(record -> {
|
||||
assertThat(record.eventId()).isEqualTo("same-id");
|
||||
assertThat(record.rawArchiveUri()).isEqualTo("archive://replacement.bin");
|
||||
});
|
||||
}
|
||||
|
||||
@Test
|
||||
void findsRecordByRawArchiveUri() throws Exception {
|
||||
EventFileStore store = store();
|
||||
EventFileRecord record = rawRecord("raw-uri", "VIN001", "2026-06-23T01:00:00Z");
|
||||
store.append(record);
|
||||
|
||||
EventFileRecord found = store.findByRawArchiveUri(record.rawArchiveUri());
|
||||
|
||||
assertThat(found).isNotNull();
|
||||
assertThat(found.eventId()).isEqualTo("raw-uri");
|
||||
}
|
||||
|
||||
private EventFileStore store() {
|
||||
return new DuckDbHotEventFileStore(tempDir, ZoneId.of("Asia/Shanghai"), new ObjectMapper());
|
||||
}
|
||||
|
||||
private static EventFileRecord rawRecord(String id, String vin, String eventTime) {
|
||||
return new EventFileRecord(
|
||||
id,
|
||||
ProtocolId.GB32960,
|
||||
"RAW_ARCHIVE",
|
||||
vin,
|
||||
Instant.parse(eventTime),
|
||||
Instant.parse(eventTime).plusMillis(100),
|
||||
"archive://" + id + ".bin",
|
||||
Map.of("platformAccount", "Hyundai", "command", "REALTIME_REPORT"),
|
||||
"{\"eventId\":\"" + id + "\"}");
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Run the test and verify it fails**
|
||||
|
||||
Run:
|
||||
|
||||
```bash
|
||||
mvn -pl :event-file-store -Dtest=DuckDbHotEventFileStoreTest test
|
||||
```
|
||||
|
||||
Expected: compilation failure because `DuckDbHotEventFileStore` does not exist.
|
||||
|
||||
- [ ] **Step 3: Create the hot store class**
|
||||
|
||||
Create `DuckDbHotEventFileStore` with this structure:
|
||||
|
||||
```java
|
||||
public final class DuckDbHotEventFileStore implements EventFileStore {
|
||||
|
||||
private static final TypeReference<Map<String, String>> STRING_MAP = new TypeReference<>() {};
|
||||
|
||||
private final Path root;
|
||||
private final Path dbPath;
|
||||
private final ZoneId partitionZone;
|
||||
private final ObjectMapper objectMapper;
|
||||
private volatile boolean initialized;
|
||||
|
||||
public DuckDbHotEventFileStore(Path root, ZoneId partitionZone) {
|
||||
this(root, partitionZone, new ObjectMapper());
|
||||
}
|
||||
|
||||
public DuckDbHotEventFileStore(Path root, ZoneId partitionZone, ObjectMapper objectMapper) {
|
||||
if (root == null) {
|
||||
throw new IllegalArgumentException("root must not be null");
|
||||
}
|
||||
this.root = root.toAbsolutePath();
|
||||
this.dbPath = this.root.resolve("events.duckdb");
|
||||
this.partitionZone = partitionZone == null ? ZoneId.of("Asia/Shanghai") : partitionZone;
|
||||
this.objectMapper = objectMapper == null ? new ObjectMapper() : objectMapper;
|
||||
}
|
||||
|
||||
@Override
|
||||
public synchronized void appendAll(List<EventFileRecord> records) throws IOException {
|
||||
if (records == null || records.isEmpty()) {
|
||||
return;
|
||||
}
|
||||
ensureInitialized();
|
||||
try (Connection connection = DriverManager.getConnection(jdbcUrl())) {
|
||||
connection.setAutoCommit(false);
|
||||
try {
|
||||
upsertRecords(connection, records);
|
||||
connection.commit();
|
||||
} catch (SQLException | IOException e) {
|
||||
connection.rollback();
|
||||
throw e;
|
||||
} finally {
|
||||
connection.setAutoCommit(true);
|
||||
}
|
||||
} catch (SQLException e) {
|
||||
throw new IOException("write duckdb hot event store failed", e);
|
||||
}
|
||||
}
|
||||
|
||||
@Override
|
||||
public List<EventFileRecord> query(EventFileQuery query) throws IOException {
|
||||
ensureInitialized();
|
||||
// Implement in Task 3.
|
||||
return List.of();
|
||||
}
|
||||
|
||||
@Override
|
||||
public EventFileRecord findByRawArchiveUri(String rawArchiveUri) throws IOException {
|
||||
ensureInitialized();
|
||||
// Implement in Task 3.
|
||||
return null;
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
- [ ] **Step 4: Add schema initialization**
|
||||
|
||||
Add:
|
||||
|
||||
```java
|
||||
private void ensureInitialized() throws IOException {
|
||||
if (initialized) {
|
||||
return;
|
||||
}
|
||||
synchronized (this) {
|
||||
if (initialized) {
|
||||
return;
|
||||
}
|
||||
Files.createDirectories(root);
|
||||
try (Connection connection = DriverManager.getConnection(jdbcUrl());
|
||||
Statement statement = connection.createStatement()) {
|
||||
statement.execute("""
|
||||
CREATE TABLE IF NOT EXISTS event_records (
|
||||
event_id VARCHAR PRIMARY KEY,
|
||||
protocol VARCHAR NOT NULL,
|
||||
event_type VARCHAR NOT NULL,
|
||||
vin VARCHAR NOT NULL,
|
||||
event_time_ms BIGINT NOT NULL,
|
||||
ingest_time_ms BIGINT NOT NULL,
|
||||
partition_date DATE NOT NULL,
|
||||
raw_archive_uri VARCHAR NOT NULL,
|
||||
metadata_json VARCHAR NOT NULL,
|
||||
payload_json VARCHAR NOT NULL
|
||||
)
|
||||
""");
|
||||
statement.execute("""
|
||||
CREATE INDEX IF NOT EXISTS event_records_protocol_date_time_idx
|
||||
ON event_records(protocol, partition_date, event_time_ms)
|
||||
""");
|
||||
statement.execute("""
|
||||
CREATE INDEX IF NOT EXISTS event_records_vin_date_time_idx
|
||||
ON event_records(protocol, vin, partition_date, event_time_ms)
|
||||
""");
|
||||
statement.execute("""
|
||||
CREATE INDEX IF NOT EXISTS event_records_vin_type_date_time_idx
|
||||
ON event_records(protocol, vin, event_type, partition_date, event_time_ms)
|
||||
""");
|
||||
statement.execute("""
|
||||
CREATE INDEX IF NOT EXISTS event_records_raw_archive_uri_idx
|
||||
ON event_records(raw_archive_uri)
|
||||
""");
|
||||
initialized = true;
|
||||
} catch (SQLException e) {
|
||||
throw new IOException("initialize duckdb hot event store failed", e);
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
- [ ] **Step 5: Add idempotent batch upsert**
|
||||
|
||||
Use DuckDB `INSERT OR REPLACE` inside one transaction:
|
||||
|
||||
```java
|
||||
private void upsertRecords(Connection connection, List<EventFileRecord> records)
|
||||
throws SQLException, IOException {
|
||||
try (PreparedStatement ps = connection.prepareStatement("""
|
||||
INSERT OR REPLACE INTO event_records VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
|
||||
""")) {
|
||||
for (EventFileRecord record : records) {
|
||||
ps.setString(1, record.eventId());
|
||||
ps.setString(2, record.protocol().name());
|
||||
ps.setString(3, record.eventType());
|
||||
ps.setString(4, record.vin());
|
||||
ps.setLong(5, record.eventTime().toEpochMilli());
|
||||
ps.setLong(6, record.ingestTime().toEpochMilli());
|
||||
ps.setString(7, LocalDate.ofInstant(record.eventTime(), partitionZone).toString());
|
||||
ps.setString(8, record.rawArchiveUri());
|
||||
ps.setString(9, objectMapper.writeValueAsString(record.metadata()));
|
||||
ps.setString(10, record.payloadJson());
|
||||
ps.addBatch();
|
||||
}
|
||||
ps.executeBatch();
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
- [ ] **Step 6: Run the hot store tests**
|
||||
|
||||
Run:
|
||||
|
||||
```bash
|
||||
mvn -pl :event-file-store -Dtest=DuckDbHotEventFileStoreTest test
|
||||
```
|
||||
|
||||
Expected: query tests still fail until Task 3 implements reads; append initialization should compile.
|
||||
|
||||
## Task 3: Implement Hot Store Queries and Raw URI Lookup
|
||||
|
||||
**Files:**
|
||||
- Modify: `modules/sinks/event-file-store/src/main/java/com/lingniu/ingest/eventfilestore/DuckDbHotEventFileStore.java`
|
||||
- Test: `modules/sinks/event-file-store/src/test/java/com/lingniu/ingest/eventfilestore/DuckDbHotEventFileStoreTest.java`
|
||||
|
||||
- [ ] **Step 1: Implement `query(EventFileQuery)`**
|
||||
|
||||
Use prepared statements for every external value:
|
||||
|
||||
```java
|
||||
@Override
|
||||
public List<EventFileRecord> query(EventFileQuery query) throws IOException {
|
||||
ensureInitialized();
|
||||
String order = query.order() == EventFileQuery.Order.DESC ? "DESC" : "ASC";
|
||||
StringBuilder where = new StringBuilder("""
|
||||
WHERE protocol = ?
|
||||
AND partition_date BETWEEN CAST(? AS DATE) AND CAST(? AS DATE)
|
||||
""");
|
||||
if (query.vin() != null) {
|
||||
where.append(" AND vin = ?\n");
|
||||
}
|
||||
if (query.eventType() != null) {
|
||||
where.append(" AND event_type = ?\n");
|
||||
}
|
||||
if (query.eventTimeFrom() != null) {
|
||||
where.append(" AND event_time_ms >= ?\n");
|
||||
}
|
||||
if (query.eventTimeTo() != null) {
|
||||
where.append(" AND event_time_ms <= ?\n");
|
||||
}
|
||||
String sql = """
|
||||
SELECT event_id, protocol, event_type, vin, event_time_ms, ingest_time_ms,
|
||||
raw_archive_uri, metadata_json, payload_json
|
||||
FROM event_records
|
||||
%s
|
||||
ORDER BY event_time_ms %s, ingest_time_ms %s, event_id %s
|
||||
LIMIT ?
|
||||
""".formatted(where, order, order, order);
|
||||
try (Connection connection = DriverManager.getConnection(jdbcUrl());
|
||||
PreparedStatement ps = connection.prepareStatement(sql)) {
|
||||
bindQuery(ps, query);
|
||||
try (ResultSet rs = ps.executeQuery()) {
|
||||
List<EventFileRecord> out = new ArrayList<>();
|
||||
while (rs.next()) {
|
||||
out.add(record(rs));
|
||||
}
|
||||
return out;
|
||||
}
|
||||
} catch (SQLException e) {
|
||||
throw new IOException("query duckdb hot event store failed", e);
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Add query binding helper**
|
||||
|
||||
```java
|
||||
private static void bindQuery(PreparedStatement ps, EventFileQuery query) throws SQLException {
|
||||
int index = 1;
|
||||
ps.setString(index++, query.protocol().name());
|
||||
ps.setString(index++, query.dateFrom().toString());
|
||||
ps.setString(index++, query.dateTo().toString());
|
||||
if (query.vin() != null) {
|
||||
ps.setString(index++, query.vin());
|
||||
}
|
||||
if (query.eventType() != null) {
|
||||
ps.setString(index++, query.eventType());
|
||||
}
|
||||
if (query.eventTimeFrom() != null) {
|
||||
ps.setLong(index++, query.eventTimeFrom().toEpochMilli());
|
||||
}
|
||||
if (query.eventTimeTo() != null) {
|
||||
ps.setLong(index++, query.eventTimeTo().toEpochMilli());
|
||||
}
|
||||
ps.setInt(index, query.limit());
|
||||
}
|
||||
```
|
||||
|
||||
- [ ] **Step 3: Implement `findByRawArchiveUri`**
|
||||
|
||||
```java
|
||||
@Override
|
||||
public EventFileRecord findByRawArchiveUri(String rawArchiveUri) throws IOException {
|
||||
if (rawArchiveUri == null || rawArchiveUri.isBlank()) {
|
||||
return null;
|
||||
}
|
||||
ensureInitialized();
|
||||
try (Connection connection = DriverManager.getConnection(jdbcUrl());
|
||||
PreparedStatement ps = connection.prepareStatement("""
|
||||
SELECT event_id, protocol, event_type, vin, event_time_ms, ingest_time_ms,
|
||||
raw_archive_uri, metadata_json, payload_json
|
||||
FROM event_records
|
||||
WHERE raw_archive_uri = ?
|
||||
ORDER BY event_type = 'RAW_ARCHIVE' DESC, ingest_time_ms DESC, event_id DESC
|
||||
LIMIT 1
|
||||
""")) {
|
||||
ps.setString(1, rawArchiveUri);
|
||||
try (ResultSet rs = ps.executeQuery()) {
|
||||
return rs.next() ? record(rs) : null;
|
||||
}
|
||||
} catch (SQLException e) {
|
||||
throw new IOException("query duckdb hot store by raw archive uri failed", e);
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
- [ ] **Step 4: Add record mapper and helpers**
|
||||
|
||||
```java
|
||||
private EventFileRecord record(ResultSet rs) throws SQLException, IOException {
|
||||
return new EventFileRecord(
|
||||
rs.getString("event_id"),
|
||||
protocol(rs.getString("protocol")),
|
||||
rs.getString("event_type"),
|
||||
rs.getString("vin"),
|
||||
Instant.ofEpochMilli(rs.getLong("event_time_ms")),
|
||||
Instant.ofEpochMilli(rs.getLong("ingest_time_ms")),
|
||||
rs.getString("raw_archive_uri"),
|
||||
readMetadata(rs.getString("metadata_json")),
|
||||
rs.getString("payload_json"));
|
||||
}
|
||||
|
||||
private Map<String, String> readMetadata(String json) throws IOException {
|
||||
if (json == null || json.isBlank()) {
|
||||
return Map.of();
|
||||
}
|
||||
return objectMapper.readValue(json, STRING_MAP);
|
||||
}
|
||||
|
||||
private static ProtocolId protocol(String value) {
|
||||
if (value == null || value.isBlank()) {
|
||||
return ProtocolId.UNKNOWN;
|
||||
}
|
||||
try {
|
||||
return ProtocolId.valueOf(value);
|
||||
} catch (IllegalArgumentException ex) {
|
||||
return ProtocolId.UNKNOWN;
|
||||
}
|
||||
}
|
||||
|
||||
private String jdbcUrl() {
|
||||
return "jdbc:duckdb:" + dbPath;
|
||||
}
|
||||
```
|
||||
|
||||
- [ ] **Step 5: Run hot store tests**
|
||||
|
||||
Run:
|
||||
|
||||
```bash
|
||||
mvn -pl :event-file-store -Dtest=DuckDbHotEventFileStoreTest test
|
||||
```
|
||||
|
||||
Expected: all `DuckDbHotEventFileStoreTest` tests pass.
|
||||
|
||||
## Task 4: Preserve Legacy Store Tests and Update Defaults
|
||||
|
||||
**Files:**
|
||||
- Modify: `modules/sinks/event-file-store/src/test/java/com/lingniu/ingest/eventfilestore/DuckDbParquetEventFileStoreTest.java`
|
||||
- Modify: `modules/apps/vehicle-history-app/src/main/resources/application.yml`
|
||||
- Modify: `modules/apps/vehicle-history-app/src/test/java/com/lingniu/ingest/historyapp/VehicleHistoryAppDefaultsTest.java`
|
||||
|
||||
- [ ] **Step 1: Keep Parquet tests explicitly legacy**
|
||||
|
||||
No behavior change is needed in `DuckDbParquetEventFileStoreTest`; leave it instantiating `DuckDbParquetEventFileStore` directly. Add a class comment:
|
||||
|
||||
```java
|
||||
/**
|
||||
* Compatibility coverage for the legacy Parquet sidecar backend.
|
||||
* Production history uses DuckDbHotEventFileStore through auto-configuration.
|
||||
*/
|
||||
class DuckDbParquetEventFileStoreTest {
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Set vehicle-history-app storage default**
|
||||
|
||||
Add to `modules/apps/vehicle-history-app/src/main/resources/application.yml`:
|
||||
|
||||
```yaml
|
||||
event-file-store:
|
||||
enabled: ${EVENT_FILE_STORE_ENABLED:true}
|
||||
storage: ${EVENT_FILE_STORE_STORAGE:duckdb-hot}
|
||||
path: ${EVENT_FILE_STORE_PATH:./target/event-store/}
|
||||
zone-id: ${EVENT_FILE_STORE_ZONE_ID:Asia/Shanghai}
|
||||
batch-size: ${EVENT_FILE_STORE_BATCH_SIZE:1000}
|
||||
flush-interval-millis: ${EVENT_FILE_STORE_FLUSH_INTERVAL_MILLIS:1000}
|
||||
```
|
||||
|
||||
Keep existing indentation and only add `storage`; if `batch-size` already exists, update it to `1000`.
|
||||
|
||||
- [ ] **Step 3: Update app default test**
|
||||
|
||||
In `VehicleHistoryAppDefaultsTest`, assert:
|
||||
|
||||
```java
|
||||
assertThat(context.getEnvironment()
|
||||
.getProperty("lingniu.ingest.event-file-store.storage"))
|
||||
.isEqualTo("duckdb-hot");
|
||||
```
|
||||
|
||||
- [ ] **Step 4: Run app default tests**
|
||||
|
||||
Run:
|
||||
|
||||
```bash
|
||||
mvn -pl :vehicle-history-app -Dtest=VehicleHistoryAppDefaultsTest test
|
||||
```
|
||||
|
||||
Expected: default configuration test passes.
|
||||
|
||||
## Task 5: Verify History Ingest Still Archives RAW Bytes
|
||||
|
||||
**Files:**
|
||||
- Modify: `modules/services/event-history-service/src/test/java/com/lingniu/ingest/eventhistory/EventHistoryEnvelopeIngestorTest.java`
|
||||
- Test: `modules/services/event-history-service/src/test/java/com/lingniu/ingest/eventhistory/Gb32960DecodedFrameServiceTest.java`
|
||||
|
||||
- [ ] **Step 1: Add an integration-style test using the hot store**
|
||||
|
||||
Add a test that writes a RAW envelope through `EventHistoryEnvelopeIngestor` into `DuckDbHotEventFileStore`, then finds it by URI.
|
||||
|
||||
```java
|
||||
@Test
|
||||
void rawArchiveEnvelopeCanBeFoundFromDuckDbHotStoreByUri(@TempDir Path tempDir) throws Exception {
|
||||
EventFileStore store = new DuckDbHotEventFileStore(tempDir, ZoneId.of("Asia/Shanghai"), OBJECT_MAPPER);
|
||||
CapturingArchiveStore archive = new CapturingArchiveStore();
|
||||
EventHistoryEnvelopeIngestor ingestor =
|
||||
new EventHistoryEnvelopeIngestor(store, new TelemetryEnvelopeRecordMapper(), archive);
|
||||
String rawArchiveKey = "2026/06/23/GB32960/VINRAW001/raw-event-hot.bin";
|
||||
String rawArchiveUri = "archive://" + rawArchiveKey;
|
||||
byte[] rawBytes = new byte[]{0x23, 0x23, 0x02, 0x01};
|
||||
|
||||
VehicleEnvelope envelope = VehicleEnvelope.newBuilder()
|
||||
.setSchemaVersion("1.0")
|
||||
.setEventId("raw-event-hot")
|
||||
.setVin("VINRAW001")
|
||||
.setSource("GB32960")
|
||||
.setProtocolVersion("V2016")
|
||||
.setEventTimeMs(1_782_112_400_000L)
|
||||
.setIngestTimeMs(1_782_112_401_000L)
|
||||
.putMetadata(RawArchiveKeys.META_KEY, rawArchiveKey)
|
||||
.putMetadata(RawArchiveKeys.META_URI, rawArchiveUri)
|
||||
.setRawArchive(RawArchiveRef.newBuilder()
|
||||
.setUri(rawArchiveUri)
|
||||
.setSizeBytes(rawBytes.length)
|
||||
.setData(ByteString.copyFrom(rawBytes))
|
||||
.build())
|
||||
.build();
|
||||
|
||||
EnvelopeIngestResult result = ingestor.tryIngest(envelope.toByteArray());
|
||||
|
||||
assertThat(result.status()).isEqualTo(EnvelopeIngestResult.Status.STORED);
|
||||
assertThat(store.findByRawArchiveUri(rawArchiveUri))
|
||||
.isNotNull()
|
||||
.extracting(EventFileRecord::eventId)
|
||||
.isEqualTo("raw-event-hot");
|
||||
assertThat(archive.bytesByKey).containsEntry(rawArchiveKey, rawBytes);
|
||||
}
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Run the event history ingestor test**
|
||||
|
||||
Run:
|
||||
|
||||
```bash
|
||||
mvn -pl :event-history-service -Dtest=EventHistoryEnvelopeIngestorTest test
|
||||
```
|
||||
|
||||
Expected: test passes.
|
||||
|
||||
- [ ] **Step 3: Run decoded frame service tests**
|
||||
|
||||
Run:
|
||||
|
||||
```bash
|
||||
mvn -pl :event-history-service -Dtest=Gb32960DecodedFrameServiceTest test
|
||||
```
|
||||
|
||||
Expected: existing replay/snapshot tests pass. If they use a fake store, no change is needed.
|
||||
|
||||
## Task 6: Full Module Verification
|
||||
|
||||
**Files:**
|
||||
- No source changes unless failures expose missing imports or config assertions.
|
||||
|
||||
- [ ] **Step 1: Run sink module tests**
|
||||
|
||||
Run:
|
||||
|
||||
```bash
|
||||
mvn -pl :event-file-store test
|
||||
```
|
||||
|
||||
Expected: all event-file-store tests pass, including both hot and legacy stores.
|
||||
|
||||
- [ ] **Step 2: Run event-history-service tests**
|
||||
|
||||
Run:
|
||||
|
||||
```bash
|
||||
mvn -pl :event-history-service test
|
||||
```
|
||||
|
||||
Expected: all event-history-service tests pass.
|
||||
|
||||
- [ ] **Step 3: Package vehicle-history-app**
|
||||
|
||||
Run:
|
||||
|
||||
```bash
|
||||
mvn -pl :vehicle-history-app -am package -DskipTests
|
||||
```
|
||||
|
||||
Expected: package succeeds.
|
||||
|
||||
## Task 7: Local Runtime Verification
|
||||
|
||||
**Files:**
|
||||
- No committed source changes.
|
||||
|
||||
- [ ] **Step 1: Stop any old history service on port 20200**
|
||||
|
||||
Run:
|
||||
|
||||
```bash
|
||||
lsof -tiTCP:20200 -sTCP:LISTEN | xargs -r kill
|
||||
```
|
||||
|
||||
Expected: no command output, or the old process exits.
|
||||
|
||||
- [ ] **Step 2: Start vehicle-history-app with hot DuckDB store**
|
||||
|
||||
Run:
|
||||
|
||||
```bash
|
||||
EVENT_FILE_STORE_STORAGE=duckdb-hot \
|
||||
EVENT_FILE_STORE_PATH=./target/live-history-event-store \
|
||||
SINK_ARCHIVE_PATH=./target/live-history-archive \
|
||||
KAFKA_BROKERS=114.55.58.251:9092 \
|
||||
KAFKA_CONSUMER_ENABLED=true \
|
||||
KAFKA_GROUP_HISTORY=vehicle-history-hot-$(date +%Y%m%d%H%M%S) \
|
||||
java --sun-misc-unsafe-memory-access=allow \
|
||||
-jar modules/apps/vehicle-history-app/target/vehicle-history-app.jar
|
||||
```
|
||||
|
||||
Expected:
|
||||
|
||||
- app starts on `http://127.0.0.1:20200`;
|
||||
- logs show Kafka consumer subscribed;
|
||||
- `target/live-history-event-store/events.duckdb` is created.
|
||||
|
||||
- [ ] **Step 3: Verify health**
|
||||
|
||||
Run:
|
||||
|
||||
```bash
|
||||
curl -sS http://127.0.0.1:20200/actuator/health
|
||||
```
|
||||
|
||||
Expected:
|
||||
|
||||
```json
|
||||
{"status":"UP"}
|
||||
```
|
||||
|
||||
- [ ] **Step 4: Verify a recent VIN query**
|
||||
|
||||
Run with a VIN observed in live archive:
|
||||
|
||||
```bash
|
||||
curl -sS 'http://127.0.0.1:20200/api/event-history/gb32960/telemetry-snapshots?vin=LNXNEGRR9SR318194&platformAccount=Hyundai&dateFrom=2026-06-23&dateTo=2026-06-24&order=DESC&limit=3'
|
||||
```
|
||||
|
||||
Expected:
|
||||
|
||||
- response is a JSON array;
|
||||
- if live traffic for that VIN exists after service start, at least one snapshot appears;
|
||||
- `rawArchiveUris` point to existing files under `target/live-history-archive`.
|
||||
|
||||
- [ ] **Step 5: Verify raw frame replay**
|
||||
|
||||
Use one `rawArchiveUri` from Step 4:
|
||||
|
||||
```bash
|
||||
curl -sS 'http://127.0.0.1:20200/api/event-history/gb32960/frame?rawArchiveUri=archive://REPLACE_ME&platformAccount=Hyundai'
|
||||
```
|
||||
|
||||
Expected:
|
||||
|
||||
- response contains `vin`, `command`, `eventTime`, and parsed `blocks`;
|
||||
- no `raw archive is missing` warning for the selected URI.
|
||||
|
||||
## Self-Review Checklist
|
||||
|
||||
- [ ] The new hot store does not rewrite Parquet files on every append.
|
||||
- [ ] `appendAll` writes one batch in one transaction.
|
||||
- [ ] Duplicate `event_id` replay is idempotent.
|
||||
- [ ] Existing history HTTP APIs still use `EventFileStore`, so controller contracts are unchanged.
|
||||
- [ ] RAW bytes still land in `ArchiveStore`.
|
||||
- [ ] `findByRawArchiveUri` is backed by DuckDB index.
|
||||
- [ ] No MySQL credential, RDS host, username, or password appears in the plan or source files.
|
||||
- [ ] This plan does not claim the full production goal is complete; it only lands the history foundation.
|
||||
@@ -0,0 +1,487 @@
|
||||
# GB32960 Production Readiness Design
|
||||
|
||||
## Goal
|
||||
|
||||
Make the GB32960 ingestion, history, and analytics services production-ready:
|
||||
|
||||
- receive and parse GB32960 traffic quickly and accurately;
|
||||
- maximize VIN-to-platform/vendor-profile association through platform login and local mappings;
|
||||
- store complete RAW frames and queryable telemetry history with DuckDB;
|
||||
- support full frame replay from persisted RAW bytes;
|
||||
- compute real-time daily vehicle metrics and alarm timelines into MySQL;
|
||||
- reduce service coupling and improve cohesion inside each module.
|
||||
|
||||
## Non-Goals
|
||||
|
||||
- Do not put database credentials in code, YAML files, tests, docs, or commits.
|
||||
- Do not make MySQL the source for full telemetry history. MySQL is for derived daily metrics and alarm timelines.
|
||||
- Do not require every private GB32960 extension to be perfect before storing data. RAW bytes must always be kept when a valid frame is received.
|
||||
|
||||
## Production Architecture
|
||||
|
||||
```mermaid
|
||||
flowchart LR
|
||||
tcp["GB32960 TCP:32960"] --> ingest["gb32960-ingest-app"]
|
||||
mapping["local vin-platform-profile.jsonl"] --> ingest
|
||||
ingest --> kafkaEvent["Kafka vehicle.event.gb32960.v1"]
|
||||
ingest --> kafkaRaw["Kafka vehicle.raw.gb32960.v1"]
|
||||
kafkaEvent --> history["vehicle-history-app"]
|
||||
kafkaRaw --> history
|
||||
history --> duckdb["DuckDB hot history"]
|
||||
history --> raw["RAW frame archive"]
|
||||
kafkaEvent --> analytics["vehicle-analytics-app"]
|
||||
analytics --> mysql["MySQL daily stats + alarms"]
|
||||
```
|
||||
|
||||
## Service Boundaries
|
||||
|
||||
### 1. `gb32960-ingest-app`
|
||||
|
||||
Responsibilities:
|
||||
|
||||
- Netty TCP accept, frame splitting, GB32960 decode, auth, ACK, and Kafka production.
|
||||
- Platform login handling and connection-local `platformAccount`.
|
||||
- VIN-to-platform/vendor-profile resolution.
|
||||
- Produce structured telemetry, session, alarm, diagnostics, and RAW envelope messages.
|
||||
|
||||
It must not:
|
||||
|
||||
- write DuckDB, MySQL, or local archive files;
|
||||
- run historical queries;
|
||||
- run daily statistics.
|
||||
|
||||
ACK policy:
|
||||
|
||||
- login/report-like GB32960 commands use durable ACK: ACK only after Kafka dispatch succeeds;
|
||||
- malformed bytes that cannot form a valid frame are dropped with diagnostics;
|
||||
- valid frames with partial private-block parse failures still go to Kafka with `parseStatus=PARTIAL`.
|
||||
|
||||
VIN/profile resolution priority:
|
||||
|
||||
1. active platform login on the current TCP channel;
|
||||
2. exact VIN entry from local mapping file;
|
||||
3. VIN prefix entry from local mapping file;
|
||||
4. configured platform extension rule;
|
||||
5. standard GB32960 profile.
|
||||
|
||||
Local mapping file format:
|
||||
|
||||
```jsonl
|
||||
{"vin":"LNXNEGRR2SR321390","platformAccount":"Hyundai","vendorProfile":"guangdong-fc","enabled":true}
|
||||
{"vinPrefix":"LNXNEGRR","platformAccount":"Hyundai","vendorProfile":"guangdong-fc","enabled":true}
|
||||
```
|
||||
|
||||
The mapping file is loaded at startup and can be reloaded by an actuator endpoint or file watcher in a later phase.
|
||||
|
||||
### 2. `vehicle-history-app`
|
||||
|
||||
Responsibilities:
|
||||
|
||||
- Consume Kafka event and RAW topics.
|
||||
- Persist RAW bytes to an append-only archive.
|
||||
- Persist query indexes and compact telemetry points to DuckDB.
|
||||
- Serve high-performance query APIs:
|
||||
- full frame list by `vin + eventTime`;
|
||||
- telemetry fields by `vin + eventTime`;
|
||||
- one-frame replay by `rawArchiveUri`;
|
||||
- parse diagnostics and missing-profile troubleshooting.
|
||||
|
||||
It must not:
|
||||
|
||||
- listen on GB32960 TCP;
|
||||
- run MySQL statistics;
|
||||
- mutate daily business metrics.
|
||||
|
||||
### 3. `vehicle-analytics-app`
|
||||
|
||||
Responsibilities:
|
||||
|
||||
- Consume Kafka event topic.
|
||||
- Maintain real-time daily stats in MySQL.
|
||||
- Maintain full alarm timelines in MySQL.
|
||||
- Expose metrics query APIs and operational health.
|
||||
|
||||
It must not:
|
||||
|
||||
- parse RAW frames for normal operation;
|
||||
- write DuckDB history;
|
||||
- own GB32960 TCP connectivity.
|
||||
|
||||
## Kafka Contract
|
||||
|
||||
Keep Protobuf `VehicleEnvelope`, but make the producer contract stricter:
|
||||
|
||||
- `eventId`: stable idempotency key, deterministic from protocol, VIN, command, event time, raw checksum, and sequence when available.
|
||||
- `vin`: required for vehicle frames; platform-only commands use `_platform` or empty plus `platformAccount`.
|
||||
- `eventTimeMs`: GB32960 data collection time when available.
|
||||
- `ingestTimeMs`: service receive time.
|
||||
- `metadata.platformAccount`: resolved platform account.
|
||||
- `metadata.vendorProfile`: selected parser profile.
|
||||
- `metadata.parseStatus`: `OK`, `PARTIAL`, or `FAILED`.
|
||||
- `metadata.parseErrorCode`: stable code for failures.
|
||||
- `raw_archive.data`: present on RAW topic only; event topic keeps `rawArchiveUri` pointer.
|
||||
- `telemetry_snapshot`: standard field projection for downstream services.
|
||||
|
||||
Consumer rule:
|
||||
|
||||
- History consumes both event and RAW topics.
|
||||
- Analytics consumes only event topic.
|
||||
- Consumer offset is committed only after the target store commit succeeds.
|
||||
- Reprocessing must be safe through deterministic ids and database upsert/ignore semantics.
|
||||
|
||||
## DuckDB History Design
|
||||
|
||||
Use DuckDB as the hot historical database, not a Parquet-rewrite sidecar.
|
||||
|
||||
DuckDB official guidance says row-by-row insert loops are inefficient for bulk loading, and Appender/batched writes are the intended high-throughput path. Therefore history writes use a single writer worker with bounded batches and one DuckDB connection.
|
||||
|
||||
### Tables
|
||||
|
||||
`gb32960_frame_index`
|
||||
|
||||
```sql
|
||||
CREATE TABLE IF NOT EXISTS gb32960_frame_index (
|
||||
event_id VARCHAR PRIMARY KEY,
|
||||
vin VARCHAR NOT NULL,
|
||||
platform_account VARCHAR,
|
||||
vendor_profile VARCHAR,
|
||||
command VARCHAR NOT NULL,
|
||||
command_code INTEGER NOT NULL,
|
||||
event_time TIMESTAMP NOT NULL,
|
||||
event_time_ms BIGINT NOT NULL,
|
||||
ingest_time TIMESTAMP NOT NULL,
|
||||
ingest_time_ms BIGINT NOT NULL,
|
||||
partition_date DATE NOT NULL,
|
||||
raw_archive_uri VARCHAR NOT NULL,
|
||||
raw_checksum VARCHAR,
|
||||
raw_size_bytes BIGINT NOT NULL,
|
||||
parse_status VARCHAR NOT NULL,
|
||||
parse_error_code VARCHAR,
|
||||
block_count INTEGER NOT NULL,
|
||||
metadata_json VARCHAR NOT NULL
|
||||
);
|
||||
|
||||
CREATE INDEX IF NOT EXISTS gb32960_frame_vin_time_idx
|
||||
ON gb32960_frame_index(vin, event_time_ms);
|
||||
|
||||
CREATE INDEX IF NOT EXISTS gb32960_frame_raw_uri_idx
|
||||
ON gb32960_frame_index(raw_archive_uri);
|
||||
|
||||
CREATE INDEX IF NOT EXISTS gb32960_frame_partition_time_idx
|
||||
ON gb32960_frame_index(partition_date, event_time_ms);
|
||||
```
|
||||
|
||||
`gb32960_telemetry_point`
|
||||
|
||||
```sql
|
||||
CREATE TABLE IF NOT EXISTS gb32960_telemetry_point (
|
||||
event_id VARCHAR NOT NULL,
|
||||
vin VARCHAR NOT NULL,
|
||||
event_time TIMESTAMP NOT NULL,
|
||||
event_time_ms BIGINT NOT NULL,
|
||||
ingest_time TIMESTAMP NOT NULL,
|
||||
platform_account VARCHAR,
|
||||
vendor_profile VARCHAR,
|
||||
field_key VARCHAR NOT NULL,
|
||||
value_type VARCHAR NOT NULL,
|
||||
value_text VARCHAR NOT NULL,
|
||||
value_num DOUBLE,
|
||||
unit VARCHAR,
|
||||
quality VARCHAR NOT NULL,
|
||||
raw_archive_uri VARCHAR NOT NULL,
|
||||
PRIMARY KEY(event_id, field_key)
|
||||
);
|
||||
|
||||
CREATE INDEX IF NOT EXISTS gb32960_point_vin_field_time_idx
|
||||
ON gb32960_telemetry_point(vin, field_key, event_time_ms);
|
||||
```
|
||||
|
||||
`gb32960_alarm_frame`
|
||||
|
||||
```sql
|
||||
CREATE TABLE IF NOT EXISTS gb32960_alarm_frame (
|
||||
event_id VARCHAR PRIMARY KEY,
|
||||
vin VARCHAR NOT NULL,
|
||||
event_time TIMESTAMP NOT NULL,
|
||||
event_time_ms BIGINT NOT NULL,
|
||||
level VARCHAR NOT NULL,
|
||||
active_bits_json VARCHAR NOT NULL,
|
||||
fault_codes_json VARCHAR NOT NULL,
|
||||
raw_archive_uri VARCHAR NOT NULL
|
||||
);
|
||||
|
||||
CREATE INDEX IF NOT EXISTS gb32960_alarm_vin_time_idx
|
||||
ON gb32960_alarm_frame(vin, event_time_ms);
|
||||
```
|
||||
|
||||
### RAW Archive
|
||||
|
||||
RAW archive remains file/object based:
|
||||
|
||||
```text
|
||||
archive://yyyy/MM/dd/GB32960/{vin}/{event_id}.bin
|
||||
```
|
||||
|
||||
The archive writer verifies:
|
||||
|
||||
- stored byte length equals envelope `raw_archive.size_bytes`;
|
||||
- checksum matches;
|
||||
- URI is unique for the deterministic event id.
|
||||
|
||||
Frame replay path:
|
||||
|
||||
1. query `gb32960_frame_index` by `rawArchiveUri`;
|
||||
2. read RAW bytes from archive;
|
||||
3. resolve `platformAccount/vendorProfile` from index metadata;
|
||||
4. decode using current parser;
|
||||
5. return decoded frame plus original indexed parse diagnostics.
|
||||
|
||||
## Query APIs
|
||||
|
||||
Keep existing endpoints but back them by the new DuckDB tables:
|
||||
|
||||
- `GET /api/event-history/gb32960/frames`
|
||||
- full frame query by VIN/time.
|
||||
- `GET /api/event-history/gb32960/frame`
|
||||
- one RAW frame replay by `rawArchiveUri`.
|
||||
- `GET /api/event-history/gb32960/telemetry-snapshots`
|
||||
- compact telemetry snapshot by VIN/time.
|
||||
- `GET /api/event-history/gb32960/telemetry-fields`
|
||||
- field projection optimized by `gb32960_telemetry_point`.
|
||||
- `GET /api/event-history/gb32960/diagnostics`
|
||||
- parse status and missing-profile investigation by VIN/time.
|
||||
|
||||
Time semantics:
|
||||
|
||||
- `eventTime` is business query time.
|
||||
- `ingestTime` is kept for operational replay, late-arrival detection, and Kafka lag investigation.
|
||||
- Dates without timezone are interpreted in `Asia/Shanghai`.
|
||||
|
||||
## MySQL Analytics Design
|
||||
|
||||
Credentials are supplied only by environment variables:
|
||||
|
||||
```text
|
||||
MYSQL_HOST
|
||||
MYSQL_PORT
|
||||
MYSQL_DATABASE
|
||||
MYSQL_USERNAME
|
||||
MYSQL_PASSWORD
|
||||
```
|
||||
|
||||
`vehicle_daily_stat`
|
||||
|
||||
```sql
|
||||
CREATE TABLE IF NOT EXISTS vehicle_daily_stat (
|
||||
vin VARCHAR(32) NOT NULL,
|
||||
stat_date DATE NOT NULL,
|
||||
platform_account VARCHAR(64),
|
||||
total_mileage_km DECIMAL(14,3),
|
||||
daily_mileage_km DECIMAL(14,3),
|
||||
daily_power_kwh DECIMAL(14,3),
|
||||
daily_hydrogen_kg DECIMAL(14,4),
|
||||
daily_hydrogen_kg_per_100km DECIMAL(14,4),
|
||||
first_event_time DATETIME(3),
|
||||
last_event_time DATETIME(3),
|
||||
first_total_mileage_km DECIMAL(14,3),
|
||||
last_total_mileage_km DECIMAL(14,3),
|
||||
last_soc_percent DECIMAL(8,3),
|
||||
sample_count BIGINT NOT NULL DEFAULT 0,
|
||||
late_sample_count BIGINT NOT NULL DEFAULT 0,
|
||||
calculation_quality VARCHAR(32) NOT NULL,
|
||||
updated_at DATETIME(3) NOT NULL,
|
||||
PRIMARY KEY (vin, stat_date)
|
||||
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4;
|
||||
```
|
||||
|
||||
`vehicle_alarm_timeline`
|
||||
|
||||
```sql
|
||||
CREATE TABLE IF NOT EXISTS vehicle_alarm_timeline (
|
||||
alarm_id VARCHAR(128) NOT NULL,
|
||||
vin VARCHAR(32) NOT NULL,
|
||||
alarm_key VARCHAR(128) NOT NULL,
|
||||
level VARCHAR(32) NOT NULL,
|
||||
start_time DATETIME(3) NOT NULL,
|
||||
end_time DATETIME(3),
|
||||
last_seen_time DATETIME(3) NOT NULL,
|
||||
start_raw_archive_uri VARCHAR(512),
|
||||
last_raw_archive_uri VARCHAR(512),
|
||||
status VARCHAR(16) NOT NULL,
|
||||
details_json JSON,
|
||||
updated_at DATETIME(3) NOT NULL,
|
||||
PRIMARY KEY (alarm_id),
|
||||
KEY vehicle_alarm_vin_time_idx (vin, start_time),
|
||||
KEY vehicle_alarm_open_idx (vin, status, alarm_key)
|
||||
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4;
|
||||
```
|
||||
|
||||
`vehicle_stat_event_dedup`
|
||||
|
||||
```sql
|
||||
CREATE TABLE IF NOT EXISTS vehicle_stat_event_dedup (
|
||||
event_id VARCHAR(128) NOT NULL,
|
||||
consumer_group VARCHAR(128) NOT NULL,
|
||||
processed_at DATETIME(3) NOT NULL,
|
||||
PRIMARY KEY (event_id, consumer_group)
|
||||
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4;
|
||||
```
|
||||
|
||||
Writes use `INSERT ... ON DUPLICATE KEY UPDATE` so replayed Kafka messages update the same daily row instead of creating duplicates.
|
||||
|
||||
## Daily Metric Algorithms
|
||||
|
||||
All daily statistics are keyed by `eventTime` in `Asia/Shanghai`.
|
||||
|
||||
### Total Mileage
|
||||
|
||||
Use the latest valid `totalMileageKm` seen for the day.
|
||||
|
||||
Validation:
|
||||
|
||||
- reject negative values;
|
||||
- reject impossible jumps based on configurable max speed and elapsed time;
|
||||
- keep the sample as diagnostic if rejected.
|
||||
|
||||
### Daily Mileage
|
||||
|
||||
Preferred:
|
||||
|
||||
```text
|
||||
dailyMileage = today.lastTotalMileage - previousDay.lastTotalMileage
|
||||
```
|
||||
|
||||
Fallback:
|
||||
|
||||
```text
|
||||
dailyMileage = today.lastTotalMileage - today.firstTotalMileage
|
||||
```
|
||||
|
||||
Quality:
|
||||
|
||||
- `PREVIOUS_DAY_BASELINE` when previous day baseline exists;
|
||||
- `INTRADAY_DELTA` when only same-day baseline exists;
|
||||
- `INSUFFICIENT_DATA` when neither is reliable.
|
||||
|
||||
### Daily Power Consumption
|
||||
|
||||
Preferred:
|
||||
|
||||
- vendor cumulative power-consumption field if available.
|
||||
|
||||
Fallback:
|
||||
|
||||
```text
|
||||
deltaKwh = sum(max(0, totalVoltageV * totalCurrentA) * deltaSeconds / 3600000)
|
||||
```
|
||||
|
||||
The sign convention is configurable per vendor profile because GB32960 vehicle current may be positive or negative depending on charging/discharging representation.
|
||||
|
||||
### Daily Hydrogen Consumption
|
||||
|
||||
Use a tiered market-practical algorithm instead of a single fixed formula:
|
||||
|
||||
1. `DIRECT_COUNTER`: vendor cumulative hydrogen consumption or remaining hydrogen mass delta.
|
||||
2. `PVT_MASS_DELTA`: pressure/temperature/tank-volume mass estimate when storage data is available.
|
||||
3. `FUEL_CELL_ENERGY_MODEL`: integrate fuel-cell electric output and divide by calibrated efficiency.
|
||||
4. `UNKNOWN`: keep null when required signals are missing.
|
||||
|
||||
The PVT approach follows the industry pattern behind compressed hydrogen consumption tests, where pressure, temperature, and tank volume can be used to estimate hydrogen mass. The implementation must expose calibration constants per vehicle model/profile:
|
||||
|
||||
```json
|
||||
{
|
||||
"vinPrefix": "LNXNEGRR",
|
||||
"tankVolumeLiter": 140,
|
||||
"hydrogenModel": "PVT_MASS_DELTA",
|
||||
"fuelCellEfficiency": 0.52,
|
||||
"currentDischargeSign": "POSITIVE"
|
||||
}
|
||||
```
|
||||
|
||||
### Daily Hydrogen Consumption Rate
|
||||
|
||||
```text
|
||||
dailyHydrogenKgPer100km = dailyHydrogenKg / dailyMileageKm * 100
|
||||
```
|
||||
|
||||
If mileage is missing or less than the configured minimum distance, keep the rate null and set quality to `INSUFFICIENT_DISTANCE`.
|
||||
|
||||
### Alarm Timeline
|
||||
|
||||
For each telemetry frame:
|
||||
|
||||
- derive active alarm keys from alarm bits and fault code arrays;
|
||||
- open a timeline row when a key first appears;
|
||||
- update `last_seen_time` while it remains active;
|
||||
- close open alarms when the key disappears after a configurable grace window;
|
||||
- keep start/end raw archive URIs for replay.
|
||||
|
||||
## Reliability and Observability
|
||||
|
||||
Required metrics:
|
||||
|
||||
- TCP active connections;
|
||||
- frames received per command;
|
||||
- decode failures by code;
|
||||
- missing platform login rejections;
|
||||
- vendor profile resolution source;
|
||||
- Kafka send latency and failures;
|
||||
- history write batch size, latency, failure count;
|
||||
- DuckDB query latency by endpoint;
|
||||
- MySQL upsert latency and failure count;
|
||||
- consumer lag by group/topic/partition;
|
||||
- late sample count by VIN/date.
|
||||
|
||||
Required logs:
|
||||
|
||||
- one structured line for every dropped or rejected valid frame;
|
||||
- no per-frame INFO spam for successful reports after first frame per VIN/profile unless debug is enabled;
|
||||
- include `vin`, `platformAccount`, `command`, `eventId`, `reasonCode`, and `peer`.
|
||||
|
||||
## Migration Plan
|
||||
|
||||
Phase 1: History foundation
|
||||
|
||||
- Replace Parquet rewrite store with DuckDB hot tables and batch writer.
|
||||
- Keep RAW bytes archive.
|
||||
- Move existing history endpoints onto the new store.
|
||||
- Add diagnostic endpoint for missing VIN/profile cases.
|
||||
|
||||
Phase 2: Ingest association and diagnostics
|
||||
|
||||
- Add local `vin-platform-profile.jsonl` loader.
|
||||
- Use resolved profile before private block parsing.
|
||||
- Add structured parse/reject diagnostics to Kafka metadata and logs.
|
||||
- Add tests for platform login, mapping fallback, and partial private parsing.
|
||||
|
||||
Phase 3: MySQL analytics
|
||||
|
||||
- Add MySQL repository and migrations.
|
||||
- Implement daily stat accumulator with idempotent upsert.
|
||||
- Implement alarm timeline open/update/close logic.
|
||||
- Add integration tests using Testcontainers or a local MySQL profile.
|
||||
|
||||
Phase 4: Production hardening
|
||||
|
||||
- Add backfill/replay commands from DuckDB RAW archive to analytics.
|
||||
- Add retention/export from DuckDB to Parquet for cold storage.
|
||||
- Add dashboards and operational runbook.
|
||||
|
||||
## Acceptance Criteria
|
||||
|
||||
- A valid GB32960 frame received on port 32960 appears in Kafka with RAW bytes or RAW URI.
|
||||
- A valid frame is queryable by `vin + eventTime` from history.
|
||||
- A returned frame can be replayed from `rawArchiveUri` and decoded with the stored profile.
|
||||
- A VIN with only local mapping, and no current platform login, can still use the correct private parser profile.
|
||||
- A private block parse failure does not drop the frame.
|
||||
- Daily MySQL rows update in real time for mileage, total mileage, power consumption, hydrogen consumption, hydrogen rate, and sample metadata.
|
||||
- Alarm timeline rows preserve full alarm start, update, and close history.
|
||||
- Replaying the same Kafka event does not duplicate daily stats or alarm rows.
|
||||
- No secret appears in committed files.
|
||||
|
||||
## References
|
||||
|
||||
- DuckDB Appender: https://duckdb.org/docs/current/data/appender
|
||||
- DuckDB INSERT performance guidance: https://duckdb.org/docs/current/data/insert
|
||||
- MySQL `INSERT ... ON DUPLICATE KEY UPDATE`: https://dev.mysql.com/doc/refman/9.7/en/insert-on-duplicate.html
|
||||
- SAE J2572 overview: https://h2tools.org/fuel-cell-codes-and-standards/sae-j2572-measuring-exhaust-emissions-energy-consumption-and-range
|
||||
66
modules/apps/gb32960-ingest-app/pom.xml
Normal file
66
modules/apps/gb32960-ingest-app/pom.xml
Normal file
@@ -0,0 +1,66 @@
|
||||
<?xml version="1.0" encoding="UTF-8"?>
|
||||
<project xmlns="http://maven.apache.org/POM/4.0.0">
|
||||
<modelVersion>4.0.0</modelVersion>
|
||||
<parent>
|
||||
<groupId>com.lingniu.ingest</groupId>
|
||||
<artifactId>lingniu-vehicle-ingest</artifactId>
|
||||
<version>0.1.0-SNAPSHOT</version>
|
||||
<relativePath>../../../pom.xml</relativePath>
|
||||
</parent>
|
||||
|
||||
<artifactId>gb32960-ingest-app</artifactId>
|
||||
<name>gb32960-ingest-app</name>
|
||||
<description>GB32960 protocol ingress runtime: TCP decode, auth, Kafka production, and ACK.</description>
|
||||
|
||||
<dependencies>
|
||||
<dependency>
|
||||
<groupId>com.lingniu.ingest</groupId>
|
||||
<artifactId>ingest-core</artifactId>
|
||||
</dependency>
|
||||
<dependency>
|
||||
<groupId>com.lingniu.ingest</groupId>
|
||||
<artifactId>session-core</artifactId>
|
||||
</dependency>
|
||||
<dependency>
|
||||
<groupId>com.lingniu.ingest</groupId>
|
||||
<artifactId>observability</artifactId>
|
||||
</dependency>
|
||||
<dependency>
|
||||
<groupId>com.lingniu.ingest</groupId>
|
||||
<artifactId>protocol-gb32960</artifactId>
|
||||
</dependency>
|
||||
<dependency>
|
||||
<groupId>com.lingniu.ingest</groupId>
|
||||
<artifactId>sink-mq</artifactId>
|
||||
</dependency>
|
||||
<dependency>
|
||||
<groupId>org.springframework.boot</groupId>
|
||||
<artifactId>spring-boot-starter-web</artifactId>
|
||||
</dependency>
|
||||
<dependency>
|
||||
<groupId>org.springframework.boot</groupId>
|
||||
<artifactId>spring-boot-starter-test</artifactId>
|
||||
<scope>test</scope>
|
||||
</dependency>
|
||||
</dependencies>
|
||||
|
||||
<build>
|
||||
<finalName>gb32960-ingest-app</finalName>
|
||||
<plugins>
|
||||
<plugin>
|
||||
<groupId>org.springframework.boot</groupId>
|
||||
<artifactId>spring-boot-maven-plugin</artifactId>
|
||||
<configuration>
|
||||
<jvmArguments>--sun-misc-unsafe-memory-access=allow</jvmArguments>
|
||||
</configuration>
|
||||
<executions>
|
||||
<execution>
|
||||
<goals>
|
||||
<goal>repackage</goal>
|
||||
</goals>
|
||||
</execution>
|
||||
</executions>
|
||||
</plugin>
|
||||
</plugins>
|
||||
</build>
|
||||
</project>
|
||||
@@ -0,0 +1,21 @@
|
||||
package com.lingniu.ingest.gb32960app;
|
||||
|
||||
import org.slf4j.Logger;
|
||||
import org.slf4j.LoggerFactory;
|
||||
import org.springframework.boot.SpringApplication;
|
||||
import org.springframework.boot.autoconfigure.SpringBootApplication;
|
||||
|
||||
@SpringBootApplication(scanBasePackages = "com.lingniu.ingest")
|
||||
public class Gb32960IngestApplication {
|
||||
|
||||
private static final Logger log = LoggerFactory.getLogger(Gb32960IngestApplication.class);
|
||||
|
||||
public static void main(String[] args) {
|
||||
try {
|
||||
SpringApplication.run(Gb32960IngestApplication.class, args);
|
||||
} catch (Throwable t) {
|
||||
log.error("GB32960 ingest application failed to start, forcing JVM exit", t);
|
||||
System.exit(1);
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,110 @@
|
||||
spring:
|
||||
application:
|
||||
name: gb32960-ingest-app
|
||||
threads:
|
||||
virtual:
|
||||
enabled: true
|
||||
|
||||
server:
|
||||
port: ${HTTP_PORT:20100}
|
||||
|
||||
lingniu:
|
||||
ingest:
|
||||
gb32960:
|
||||
enabled: true
|
||||
server:
|
||||
enabled: true
|
||||
port: ${GB32960_PORT:32960}
|
||||
boss-threads: ${GB32960_BOSS_THREADS:1}
|
||||
worker-threads: ${GB32960_WORKER_THREADS:0}
|
||||
auth:
|
||||
enabled: ${GB32960_AUTH_ENABLED:false}
|
||||
case-sensitive: false
|
||||
whitelist: []
|
||||
platforms:
|
||||
- username: ${GB32960_PLATFORM_USER_HYUNDAI:Hyundai}
|
||||
password: ${GB32960_PLATFORM_PWD_HYUNDAI:f2e3445d7cda409fb4f278f6fb890734}
|
||||
allowed-ips:
|
||||
- ${GB32960_PLATFORM_IP_HYUNDAI:8.134.95.166}
|
||||
description: 外部下级平台 - Hyundai
|
||||
tls:
|
||||
enabled: ${GB32960_TLS_ENABLED:false}
|
||||
cert-path: ${GB32960_TLS_CERT:}
|
||||
key-path: ${GB32960_TLS_KEY:}
|
||||
trust-cert-path: ${GB32960_TLS_CA:}
|
||||
require-client-auth: true
|
||||
vendor-extensions:
|
||||
- name: guangdong-fc
|
||||
match:
|
||||
platform-accounts:
|
||||
- Hyundai
|
||||
vin-prefixes: []
|
||||
vins: []
|
||||
parse:
|
||||
lenient-block-failure: true
|
||||
diagnostics:
|
||||
max-logged-frame-keys-per-channel: ${GB32960_DIAGNOSTICS_MAX_LOGGED_FRAME_KEYS_PER_CHANNEL:4096}
|
||||
pipeline:
|
||||
disruptor:
|
||||
ring-buffer-size: ${PIPELINE_RING_BUFFER_SIZE:131072}
|
||||
wait-strategy: ${PIPELINE_WAIT_STRATEGY:yielding}
|
||||
producer-type: multi
|
||||
dedup:
|
||||
enabled: true
|
||||
cache-size: 200000
|
||||
ttl-seconds: 600
|
||||
rate-limit:
|
||||
per-vin-qps: 50
|
||||
session:
|
||||
store: ${SESSION_STORE:memory}
|
||||
ttl: ${SESSION_TTL:30m}
|
||||
identity:
|
||||
store: ${VEHICLE_IDENTITY_STORE:file}
|
||||
file:
|
||||
path: ${VEHICLE_IDENTITY_FILE:./data/vehicle-identity.jsonl}
|
||||
sink:
|
||||
mq:
|
||||
enabled: ${KAFKA_ENABLED:true}
|
||||
type: kafka
|
||||
bootstrap-servers: ${KAFKA_BROKERS:114.55.58.251:9092}
|
||||
compression-type: zstd
|
||||
linger-ms: 20
|
||||
batch-size: 65536
|
||||
acks: all
|
||||
enable-idempotence: true
|
||||
node-id: ${KAFKA_NODE_ID:gb32960-ingest-local}
|
||||
topics:
|
||||
realtime: ${KAFKA_TOPIC_GB32960_EVENT:vehicle.event.gb32960.v1}
|
||||
location: ${KAFKA_TOPIC_GB32960_EVENT:vehicle.event.gb32960.v1}
|
||||
alarm: ${KAFKA_TOPIC_GB32960_EVENT:vehicle.event.gb32960.v1}
|
||||
session: ${KAFKA_TOPIC_GB32960_EVENT:vehicle.event.gb32960.v1}
|
||||
media-meta: ${KAFKA_TOPIC_MEDIA_META:vehicle.media.meta.v1}
|
||||
raw-archive: ${KAFKA_TOPIC_GB32960_RAW:vehicle.raw.gb32960.v1}
|
||||
dlq: ${KAFKA_TOPIC_GB32960_DLQ:vehicle.dlq.gb32960.v1}
|
||||
consumer:
|
||||
enabled: false
|
||||
event-file-store:
|
||||
enabled: false
|
||||
event-history:
|
||||
enabled: false
|
||||
vehicle-state:
|
||||
enabled: false
|
||||
vehicle-stat:
|
||||
enabled: false
|
||||
|
||||
management:
|
||||
endpoints:
|
||||
web:
|
||||
exposure:
|
||||
include: health,info,metrics,prometheus,env
|
||||
health:
|
||||
redis:
|
||||
enabled: ${MANAGEMENT_HEALTH_REDIS_ENABLED:false}
|
||||
metrics:
|
||||
tags:
|
||||
application: gb32960-ingest-app
|
||||
|
||||
logging:
|
||||
level:
|
||||
root: INFO
|
||||
com.lingniu.ingest: INFO
|
||||
@@ -0,0 +1,68 @@
|
||||
package com.lingniu.ingest.gb32960app;
|
||||
|
||||
import com.lingniu.ingest.core.config.IngestCoreAutoConfiguration;
|
||||
import com.lingniu.ingest.protocol.gb32960.codec.Gb32960MessageDecoder;
|
||||
import com.lingniu.ingest.protocol.gb32960.config.Gb32960AutoConfiguration;
|
||||
import com.lingniu.ingest.protocol.gb32960.inbound.Gb32960NettyServer;
|
||||
import com.lingniu.ingest.session.config.SessionCoreAutoConfiguration;
|
||||
import com.lingniu.ingest.sink.mq.KafkaEventSink;
|
||||
import com.lingniu.ingest.sink.mq.KafkaEnvelopeDeadLetterSink;
|
||||
import com.lingniu.ingest.sink.mq.SinkMqAutoConfiguration;
|
||||
import org.apache.kafka.clients.producer.KafkaProducer;
|
||||
import org.junit.jupiter.api.Test;
|
||||
import org.springframework.boot.autoconfigure.AutoConfigurations;
|
||||
import org.springframework.boot.test.context.runner.ApplicationContextRunner;
|
||||
import org.springframework.context.ApplicationContext;
|
||||
import org.springframework.util.ClassUtils;
|
||||
|
||||
import static org.assertj.core.api.Assertions.assertThat;
|
||||
import static org.mockito.Mockito.mock;
|
||||
|
||||
class Gb32960IngestAppCompositionTest {
|
||||
|
||||
private final ApplicationContextRunner contextRunner = new ApplicationContextRunner()
|
||||
.withConfiguration(AutoConfigurations.of(
|
||||
IngestCoreAutoConfiguration.class,
|
||||
SessionCoreAutoConfiguration.class,
|
||||
SinkMqAutoConfiguration.class,
|
||||
Gb32960AutoConfiguration.class))
|
||||
.withAllowBeanDefinitionOverriding(true)
|
||||
.withBean("kafkaProducer", KafkaProducer.class, Gb32960IngestAppCompositionTest::kafkaProducer)
|
||||
.withPropertyValues(
|
||||
"lingniu.ingest.gb32960.enabled=true",
|
||||
"lingniu.ingest.gb32960.server.enabled=true",
|
||||
"lingniu.ingest.gb32960.port=0",
|
||||
"lingniu.ingest.session.store=memory",
|
||||
"lingniu.ingest.sink.mq.enabled=true",
|
||||
"lingniu.ingest.sink.mq.type=kafka",
|
||||
"lingniu.ingest.sink.mq.bootstrap-servers=localhost:9092",
|
||||
"lingniu.ingest.sink.mq.consumer.enabled=false",
|
||||
"lingniu.ingest.event-file-store.enabled=false",
|
||||
"lingniu.ingest.event-history.enabled=false",
|
||||
"lingniu.ingest.vehicle-state.enabled=false",
|
||||
"lingniu.ingest.vehicle-stat.enabled=false");
|
||||
|
||||
@Test
|
||||
void createsGb32960ListenerWithoutHistoryStorageBoundaries() {
|
||||
contextRunner.run(context -> {
|
||||
assertThat(context).hasSingleBean(Gb32960MessageDecoder.class);
|
||||
assertThat(context).hasSingleBean(Gb32960NettyServer.class);
|
||||
assertThat(context).hasSingleBean(KafkaEventSink.class);
|
||||
assertThat(context).hasSingleBean(KafkaEnvelopeDeadLetterSink.class);
|
||||
|
||||
assertTypeNotPresent(context, "com.lingniu.ingest.sink.archive.ArchiveStore");
|
||||
assertTypeNotPresent(context, "com.lingniu.ingest.eventfilestore.EventFileStore");
|
||||
});
|
||||
}
|
||||
|
||||
@SuppressWarnings("unchecked")
|
||||
private static KafkaProducer<String, byte[]> kafkaProducer() {
|
||||
return mock(KafkaProducer.class);
|
||||
}
|
||||
|
||||
private static void assertTypeNotPresent(ApplicationContext context, String className) {
|
||||
assertThat(ClassUtils.isPresent(className, context.getClassLoader()))
|
||||
.as("%s must stay outside the GB32960 ingest runtime", className)
|
||||
.isFalse();
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,39 @@
|
||||
package com.lingniu.ingest.gb32960app;
|
||||
|
||||
import org.junit.jupiter.api.Test;
|
||||
import org.springframework.beans.factory.config.YamlPropertiesFactoryBean;
|
||||
import org.springframework.core.io.ClassPathResource;
|
||||
|
||||
import java.util.Properties;
|
||||
|
||||
import static org.assertj.core.api.Assertions.assertThat;
|
||||
|
||||
class Gb32960IngestAppDefaultsTest {
|
||||
|
||||
@Test
|
||||
void applicationDefaultsKeepGb32960IngestAsTcpKafkaProducerOnly() {
|
||||
Properties properties = applicationProperties();
|
||||
|
||||
assertThat(properties)
|
||||
.containsEntry("spring.application.name", "gb32960-ingest-app")
|
||||
.containsEntry("lingniu.ingest.gb32960.enabled", true)
|
||||
.containsEntry("lingniu.ingest.gb32960.server.enabled", true)
|
||||
.containsEntry("lingniu.ingest.gb32960.port", "${GB32960_PORT:32960}")
|
||||
.containsEntry("lingniu.ingest.sink.mq.enabled", "${KAFKA_ENABLED:true}")
|
||||
.containsEntry("lingniu.ingest.sink.mq.consumer.enabled", false)
|
||||
.containsEntry("lingniu.ingest.event-file-store.enabled", false)
|
||||
.containsEntry("lingniu.ingest.event-history.enabled", false)
|
||||
.containsEntry("lingniu.ingest.vehicle-state.enabled", false)
|
||||
.containsEntry("lingniu.ingest.vehicle-stat.enabled", false);
|
||||
}
|
||||
|
||||
private static Properties applicationProperties() {
|
||||
YamlPropertiesFactoryBean factory = new YamlPropertiesFactoryBean();
|
||||
factory.setResources(new ClassPathResource("application.yml"));
|
||||
|
||||
Properties properties = factory.getObject();
|
||||
|
||||
assertThat(properties).isNotNull();
|
||||
return properties;
|
||||
}
|
||||
}
|
||||
66
modules/apps/vehicle-analytics-app/pom.xml
Normal file
66
modules/apps/vehicle-analytics-app/pom.xml
Normal file
@@ -0,0 +1,66 @@
|
||||
<?xml version="1.0" encoding="UTF-8"?>
|
||||
<project xmlns="http://maven.apache.org/POM/4.0.0">
|
||||
<modelVersion>4.0.0</modelVersion>
|
||||
<parent>
|
||||
<groupId>com.lingniu.ingest</groupId>
|
||||
<artifactId>lingniu-vehicle-ingest</artifactId>
|
||||
<version>0.1.0-SNAPSHOT</version>
|
||||
<relativePath>../../../pom.xml</relativePath>
|
||||
</parent>
|
||||
|
||||
<artifactId>vehicle-analytics-app</artifactId>
|
||||
<name>vehicle-analytics-app</name>
|
||||
<description>Vehicle state, daily statistics, alarms, and analytics runtime.</description>
|
||||
|
||||
<dependencies>
|
||||
<dependency>
|
||||
<groupId>com.lingniu.ingest</groupId>
|
||||
<artifactId>ingest-api</artifactId>
|
||||
</dependency>
|
||||
<dependency>
|
||||
<groupId>com.lingniu.ingest</groupId>
|
||||
<artifactId>observability</artifactId>
|
||||
</dependency>
|
||||
<dependency>
|
||||
<groupId>com.lingniu.ingest</groupId>
|
||||
<artifactId>sink-mq</artifactId>
|
||||
</dependency>
|
||||
<dependency>
|
||||
<groupId>com.lingniu.ingest</groupId>
|
||||
<artifactId>vehicle-state-service</artifactId>
|
||||
</dependency>
|
||||
<dependency>
|
||||
<groupId>com.lingniu.ingest</groupId>
|
||||
<artifactId>vehicle-stat-service</artifactId>
|
||||
</dependency>
|
||||
<dependency>
|
||||
<groupId>org.springframework.boot</groupId>
|
||||
<artifactId>spring-boot-starter-web</artifactId>
|
||||
</dependency>
|
||||
<dependency>
|
||||
<groupId>org.springframework.boot</groupId>
|
||||
<artifactId>spring-boot-starter-test</artifactId>
|
||||
<scope>test</scope>
|
||||
</dependency>
|
||||
</dependencies>
|
||||
|
||||
<build>
|
||||
<finalName>vehicle-analytics-app</finalName>
|
||||
<plugins>
|
||||
<plugin>
|
||||
<groupId>org.springframework.boot</groupId>
|
||||
<artifactId>spring-boot-maven-plugin</artifactId>
|
||||
<configuration>
|
||||
<jvmArguments>--sun-misc-unsafe-memory-access=allow</jvmArguments>
|
||||
</configuration>
|
||||
<executions>
|
||||
<execution>
|
||||
<goals>
|
||||
<goal>repackage</goal>
|
||||
</goals>
|
||||
</execution>
|
||||
</executions>
|
||||
</plugin>
|
||||
</plugins>
|
||||
</build>
|
||||
</project>
|
||||
@@ -0,0 +1,21 @@
|
||||
package com.lingniu.ingest.analyticsapp;
|
||||
|
||||
import org.slf4j.Logger;
|
||||
import org.slf4j.LoggerFactory;
|
||||
import org.springframework.boot.SpringApplication;
|
||||
import org.springframework.boot.autoconfigure.SpringBootApplication;
|
||||
|
||||
@SpringBootApplication(scanBasePackages = "com.lingniu.ingest")
|
||||
public class VehicleAnalyticsApplication {
|
||||
|
||||
private static final Logger log = LoggerFactory.getLogger(VehicleAnalyticsApplication.class);
|
||||
|
||||
public static void main(String[] args) {
|
||||
try {
|
||||
SpringApplication.run(VehicleAnalyticsApplication.class, args);
|
||||
} catch (Throwable t) {
|
||||
log.error("Vehicle analytics application failed to start, forcing JVM exit", t);
|
||||
System.exit(1);
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,62 @@
|
||||
spring:
|
||||
application:
|
||||
name: vehicle-analytics-app
|
||||
threads:
|
||||
virtual:
|
||||
enabled: true
|
||||
|
||||
server:
|
||||
port: ${HTTP_PORT:20300}
|
||||
|
||||
lingniu:
|
||||
ingest:
|
||||
gb32960:
|
||||
enabled: false
|
||||
sink:
|
||||
mq:
|
||||
enabled: ${KAFKA_ENABLED:true}
|
||||
type: kafka
|
||||
bootstrap-servers: ${KAFKA_BROKERS:114.55.58.251:9092}
|
||||
topics:
|
||||
realtime: ${KAFKA_TOPIC_GB32960_EVENT:vehicle.event.gb32960.v1}
|
||||
dlq: ${KAFKA_TOPIC_GB32960_DLQ:vehicle.dlq.gb32960.v1}
|
||||
consumer:
|
||||
enabled: ${KAFKA_CONSUMER_ENABLED:true}
|
||||
client-id-prefix: ${KAFKA_CONSUMER_CLIENT_ID_PREFIX:vehicle-analytics}
|
||||
auto-offset-reset: ${KAFKA_CONSUMER_AUTO_OFFSET_RESET:earliest}
|
||||
max-poll-records: ${KAFKA_CONSUMER_MAX_POLL_RECORDS:500}
|
||||
bindings:
|
||||
vehicleStateEnvelopeConsumerProcessor:
|
||||
enabled: ${VEHICLE_STATE_ENABLED:false}
|
||||
group-id: ${KAFKA_GROUP_STATE:vehicle-state}
|
||||
topics:
|
||||
- ${KAFKA_TOPIC_GB32960_EVENT:vehicle.event.gb32960.v1}
|
||||
vehicleStatEnvelopeConsumerProcessor:
|
||||
enabled: ${VEHICLE_STAT_ENABLED:true}
|
||||
group-id: ${KAFKA_GROUP_STAT:vehicle-stat}
|
||||
topics:
|
||||
- ${KAFKA_TOPIC_GB32960_EVENT:vehicle.event.gb32960.v1}
|
||||
archive:
|
||||
enabled: false
|
||||
event-file-store:
|
||||
enabled: false
|
||||
event-history:
|
||||
enabled: false
|
||||
vehicle-state:
|
||||
enabled: ${VEHICLE_STATE_ENABLED:false}
|
||||
vehicle-stat:
|
||||
enabled: ${VEHICLE_STAT_ENABLED:true}
|
||||
file-path: ${VEHICLE_STAT_FILE_PATH:./target/vehicle-stat/}
|
||||
zone-id: ${VEHICLE_STAT_ZONE_ID:Asia/Shanghai}
|
||||
|
||||
management:
|
||||
endpoints:
|
||||
web:
|
||||
exposure:
|
||||
include: health,info,metrics,prometheus,env
|
||||
health:
|
||||
redis:
|
||||
enabled: ${MANAGEMENT_HEALTH_REDIS_ENABLED:false}
|
||||
metrics:
|
||||
tags:
|
||||
application: vehicle-analytics-app
|
||||
@@ -0,0 +1,85 @@
|
||||
package com.lingniu.ingest.analyticsapp;
|
||||
|
||||
import com.lingniu.ingest.api.consumer.EnvelopeConsumerProcessor;
|
||||
import com.lingniu.ingest.sink.mq.KafkaEnvelopeDeadLetterSink;
|
||||
import com.lingniu.ingest.sink.mq.KafkaEventSink;
|
||||
import com.lingniu.ingest.sink.mq.SinkMqAutoConfiguration;
|
||||
import com.lingniu.ingest.vehiclestate.VehicleStateEnvelopeIngestor;
|
||||
import com.lingniu.ingest.vehiclestate.config.VehicleStateAutoConfiguration;
|
||||
import com.lingniu.ingest.vehiclestat.DailyVehicleStatService;
|
||||
import com.lingniu.ingest.vehiclestat.FileVehicleStatRepository;
|
||||
import com.lingniu.ingest.vehiclestat.VehicleStatController;
|
||||
import com.lingniu.ingest.vehiclestat.VehicleStatEnvelopeIngestor;
|
||||
import com.lingniu.ingest.vehiclestat.VehicleStatEventProcessor;
|
||||
import com.lingniu.ingest.vehiclestat.VehicleStatEventSink;
|
||||
import com.lingniu.ingest.vehiclestat.VehicleStatRepository;
|
||||
import com.lingniu.ingest.vehiclestat.config.VehicleStatAutoConfiguration;
|
||||
import org.apache.kafka.clients.producer.KafkaProducer;
|
||||
import org.junit.jupiter.api.Test;
|
||||
import org.junit.jupiter.api.io.TempDir;
|
||||
import org.springframework.boot.autoconfigure.AutoConfigurations;
|
||||
import org.springframework.boot.test.context.runner.ApplicationContextRunner;
|
||||
import org.springframework.context.ApplicationContext;
|
||||
import org.springframework.util.ClassUtils;
|
||||
|
||||
import java.nio.file.Path;
|
||||
|
||||
import static org.assertj.core.api.Assertions.assertThat;
|
||||
import static org.mockito.Mockito.mock;
|
||||
|
||||
class VehicleAnalyticsAppCompositionTest {
|
||||
|
||||
@TempDir
|
||||
Path tempDir;
|
||||
|
||||
@Test
|
||||
void createsStatsBeansWithoutProtocolListenerOrEventFileStore() {
|
||||
new ApplicationContextRunner()
|
||||
.withConfiguration(AutoConfigurations.of(
|
||||
SinkMqAutoConfiguration.class,
|
||||
VehicleStateAutoConfiguration.class,
|
||||
VehicleStatAutoConfiguration.class))
|
||||
.withAllowBeanDefinitionOverriding(true)
|
||||
.withBean("kafkaProducer", KafkaProducer.class, VehicleAnalyticsAppCompositionTest::kafkaProducer)
|
||||
.withPropertyValues(
|
||||
"lingniu.ingest.sink.mq.enabled=true",
|
||||
"lingniu.ingest.sink.mq.type=kafka",
|
||||
"lingniu.ingest.sink.mq.bootstrap-servers=localhost:9092",
|
||||
"lingniu.ingest.sink.mq.consumer.enabled=false",
|
||||
"lingniu.ingest.vehicle-state.enabled=false",
|
||||
"lingniu.ingest.vehicle-stat.enabled=true",
|
||||
"lingniu.ingest.vehicle-stat.file-path=" + tempDir.resolve("vehicle-stat"),
|
||||
"lingniu.ingest.event-file-store.enabled=false",
|
||||
"lingniu.ingest.event-history.enabled=false",
|
||||
"lingniu.ingest.gb32960.enabled=false")
|
||||
.run(context -> {
|
||||
assertThat(context).hasSingleBean(VehicleStatRepository.class);
|
||||
assertThat(context).hasSingleBean(FileVehicleStatRepository.class);
|
||||
assertThat(context).hasSingleBean(DailyVehicleStatService.class);
|
||||
assertThat(context).hasSingleBean(VehicleStatEventProcessor.class);
|
||||
assertThat(context).hasSingleBean(VehicleStatEnvelopeIngestor.class);
|
||||
assertThat(context).hasSingleBean(VehicleStatEventSink.class);
|
||||
assertThat(context).hasSingleBean(VehicleStatController.class);
|
||||
assertThat(context).hasSingleBean(KafkaEventSink.class);
|
||||
assertThat(context).hasSingleBean(KafkaEnvelopeDeadLetterSink.class);
|
||||
assertThat(context).hasBean("vehicleStatEnvelopeConsumerProcessor");
|
||||
assertThat(context.getBean("vehicleStatEnvelopeConsumerProcessor"))
|
||||
.isInstanceOf(EnvelopeConsumerProcessor.class);
|
||||
|
||||
assertThat(context).doesNotHaveBean(VehicleStateEnvelopeIngestor.class);
|
||||
assertTypeNotPresent(context, "com.lingniu.ingest.protocol.gb32960.inbound.Gb32960NettyServer");
|
||||
assertTypeNotPresent(context, "com.lingniu.ingest.eventfilestore.EventFileStore");
|
||||
});
|
||||
}
|
||||
|
||||
@SuppressWarnings("unchecked")
|
||||
private static KafkaProducer<String, byte[]> kafkaProducer() {
|
||||
return mock(KafkaProducer.class);
|
||||
}
|
||||
|
||||
private static void assertTypeNotPresent(ApplicationContext context, String className) {
|
||||
assertThat(ClassUtils.isPresent(className, context.getClassLoader()))
|
||||
.as("%s must stay outside the vehicle analytics runtime", className)
|
||||
.isFalse();
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,49 @@
|
||||
package com.lingniu.ingest.analyticsapp;
|
||||
|
||||
import org.junit.jupiter.api.Test;
|
||||
import org.springframework.beans.factory.config.YamlPropertiesFactoryBean;
|
||||
import org.springframework.core.io.ClassPathResource;
|
||||
|
||||
import java.util.Properties;
|
||||
|
||||
import static org.assertj.core.api.Assertions.assertThat;
|
||||
|
||||
class VehicleAnalyticsAppDefaultsTest {
|
||||
|
||||
@Test
|
||||
void applicationDefaultsKeepAnalyticsAsStatConsumerRuntime() {
|
||||
Properties properties = applicationProperties();
|
||||
|
||||
assertThat(properties)
|
||||
.containsEntry("spring.application.name", "vehicle-analytics-app")
|
||||
.containsEntry("lingniu.ingest.gb32960.enabled", false)
|
||||
.containsEntry("lingniu.ingest.sink.archive.enabled", false)
|
||||
.containsEntry("lingniu.ingest.event-file-store.enabled", false)
|
||||
.containsEntry("lingniu.ingest.event-history.enabled", false)
|
||||
.containsEntry("lingniu.ingest.vehicle-stat.enabled", "${VEHICLE_STAT_ENABLED:true}")
|
||||
.containsEntry("lingniu.ingest.vehicle-state.enabled", "${VEHICLE_STATE_ENABLED:false}")
|
||||
.containsEntry("lingniu.ingest.sink.mq.consumer.enabled", "${KAFKA_CONSUMER_ENABLED:true}")
|
||||
.containsEntry(
|
||||
"lingniu.ingest.sink.mq.consumer.bindings.vehicleStatEnvelopeConsumerProcessor.enabled",
|
||||
"${VEHICLE_STAT_ENABLED:true}")
|
||||
.containsEntry(
|
||||
"lingniu.ingest.sink.mq.consumer.bindings.vehicleStateEnvelopeConsumerProcessor.enabled",
|
||||
"${VEHICLE_STATE_ENABLED:false}")
|
||||
.containsEntry(
|
||||
"lingniu.ingest.sink.mq.consumer.bindings.vehicleStatEnvelopeConsumerProcessor.group-id",
|
||||
"${KAFKA_GROUP_STAT:vehicle-stat}")
|
||||
.containsEntry(
|
||||
"lingniu.ingest.sink.mq.consumer.bindings.vehicleStateEnvelopeConsumerProcessor.group-id",
|
||||
"${KAFKA_GROUP_STATE:vehicle-state}");
|
||||
}
|
||||
|
||||
private static Properties applicationProperties() {
|
||||
YamlPropertiesFactoryBean factory = new YamlPropertiesFactoryBean();
|
||||
factory.setResources(new ClassPathResource("application.yml"));
|
||||
|
||||
Properties properties = factory.getObject();
|
||||
|
||||
assertThat(properties).isNotNull();
|
||||
return properties;
|
||||
}
|
||||
}
|
||||
74
modules/apps/vehicle-history-app/pom.xml
Normal file
74
modules/apps/vehicle-history-app/pom.xml
Normal file
@@ -0,0 +1,74 @@
|
||||
<?xml version="1.0" encoding="UTF-8"?>
|
||||
<project xmlns="http://maven.apache.org/POM/4.0.0">
|
||||
<modelVersion>4.0.0</modelVersion>
|
||||
<parent>
|
||||
<groupId>com.lingniu.ingest</groupId>
|
||||
<artifactId>lingniu-vehicle-ingest</artifactId>
|
||||
<version>0.1.0-SNAPSHOT</version>
|
||||
<relativePath>../../../pom.xml</relativePath>
|
||||
</parent>
|
||||
|
||||
<artifactId>vehicle-history-app</artifactId>
|
||||
<name>vehicle-history-app</name>
|
||||
<description>Vehicle raw archive, event history, and GB32960 frame query runtime.</description>
|
||||
|
||||
<dependencies>
|
||||
<dependency>
|
||||
<groupId>com.lingniu.ingest</groupId>
|
||||
<artifactId>ingest-api</artifactId>
|
||||
</dependency>
|
||||
<dependency>
|
||||
<groupId>com.lingniu.ingest</groupId>
|
||||
<artifactId>observability</artifactId>
|
||||
</dependency>
|
||||
<dependency>
|
||||
<groupId>com.lingniu.ingest</groupId>
|
||||
<artifactId>sink-mq</artifactId>
|
||||
</dependency>
|
||||
<dependency>
|
||||
<groupId>com.lingniu.ingest</groupId>
|
||||
<artifactId>sink-archive</artifactId>
|
||||
</dependency>
|
||||
<dependency>
|
||||
<groupId>com.lingniu.ingest</groupId>
|
||||
<artifactId>event-file-store</artifactId>
|
||||
</dependency>
|
||||
<dependency>
|
||||
<groupId>com.lingniu.ingest</groupId>
|
||||
<artifactId>event-history-service</artifactId>
|
||||
</dependency>
|
||||
<dependency>
|
||||
<groupId>com.lingniu.ingest</groupId>
|
||||
<artifactId>protocol-gb32960</artifactId>
|
||||
</dependency>
|
||||
<dependency>
|
||||
<groupId>org.springframework.boot</groupId>
|
||||
<artifactId>spring-boot-starter-web</artifactId>
|
||||
</dependency>
|
||||
<dependency>
|
||||
<groupId>org.springframework.boot</groupId>
|
||||
<artifactId>spring-boot-starter-test</artifactId>
|
||||
<scope>test</scope>
|
||||
</dependency>
|
||||
</dependencies>
|
||||
|
||||
<build>
|
||||
<finalName>vehicle-history-app</finalName>
|
||||
<plugins>
|
||||
<plugin>
|
||||
<groupId>org.springframework.boot</groupId>
|
||||
<artifactId>spring-boot-maven-plugin</artifactId>
|
||||
<configuration>
|
||||
<jvmArguments>--sun-misc-unsafe-memory-access=allow</jvmArguments>
|
||||
</configuration>
|
||||
<executions>
|
||||
<execution>
|
||||
<goals>
|
||||
<goal>repackage</goal>
|
||||
</goals>
|
||||
</execution>
|
||||
</executions>
|
||||
</plugin>
|
||||
</plugins>
|
||||
</build>
|
||||
</project>
|
||||
@@ -0,0 +1,21 @@
|
||||
package com.lingniu.ingest.historyapp;
|
||||
|
||||
import org.slf4j.Logger;
|
||||
import org.slf4j.LoggerFactory;
|
||||
import org.springframework.boot.SpringApplication;
|
||||
import org.springframework.boot.autoconfigure.SpringBootApplication;
|
||||
|
||||
@SpringBootApplication(scanBasePackages = "com.lingniu.ingest")
|
||||
public class VehicleHistoryApplication {
|
||||
|
||||
private static final Logger log = LoggerFactory.getLogger(VehicleHistoryApplication.class);
|
||||
|
||||
public static void main(String[] args) {
|
||||
try {
|
||||
SpringApplication.run(VehicleHistoryApplication.class, args);
|
||||
} catch (Throwable t) {
|
||||
log.error("Vehicle history application failed to start, forcing JVM exit", t);
|
||||
System.exit(1);
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,65 @@
|
||||
spring:
|
||||
application:
|
||||
name: vehicle-history-app
|
||||
threads:
|
||||
virtual:
|
||||
enabled: true
|
||||
|
||||
server:
|
||||
port: ${HTTP_PORT:20200}
|
||||
|
||||
lingniu:
|
||||
ingest:
|
||||
gb32960:
|
||||
enabled: true
|
||||
server:
|
||||
enabled: false
|
||||
sink:
|
||||
mq:
|
||||
enabled: ${KAFKA_ENABLED:true}
|
||||
type: kafka
|
||||
bootstrap-servers: ${KAFKA_BROKERS:114.55.58.251:9092}
|
||||
topics:
|
||||
realtime: ${KAFKA_TOPIC_GB32960_EVENT:vehicle.event.gb32960.v1}
|
||||
raw-archive: ${KAFKA_TOPIC_GB32960_RAW:vehicle.raw.gb32960.v1}
|
||||
dlq: ${KAFKA_TOPIC_GB32960_DLQ:vehicle.dlq.gb32960.v1}
|
||||
consumer:
|
||||
enabled: ${KAFKA_CONSUMER_ENABLED:true}
|
||||
client-id-prefix: ${KAFKA_CONSUMER_CLIENT_ID_PREFIX:vehicle-history}
|
||||
auto-offset-reset: ${KAFKA_CONSUMER_AUTO_OFFSET_RESET:earliest}
|
||||
max-poll-records: ${KAFKA_CONSUMER_MAX_POLL_RECORDS:500}
|
||||
bindings:
|
||||
eventHistoryEnvelopeConsumerProcessor:
|
||||
enabled: true
|
||||
group-id: ${KAFKA_GROUP_HISTORY:vehicle-history}
|
||||
topics:
|
||||
- ${KAFKA_TOPIC_GB32960_EVENT:vehicle.event.gb32960.v1}
|
||||
- ${KAFKA_TOPIC_GB32960_RAW:vehicle.raw.gb32960.v1}
|
||||
archive:
|
||||
enabled: ${SINK_ARCHIVE_ENABLED:true}
|
||||
type: local
|
||||
path: ${SINK_ARCHIVE_PATH:./archive/}
|
||||
event-file-store:
|
||||
enabled: ${EVENT_FILE_STORE_ENABLED:true}
|
||||
path: ${EVENT_FILE_STORE_PATH:./target/event-store/}
|
||||
zone-id: ${EVENT_FILE_STORE_ZONE_ID:Asia/Shanghai}
|
||||
batch-size: ${EVENT_FILE_STORE_BATCH_SIZE:500}
|
||||
flush-interval-millis: ${EVENT_FILE_STORE_FLUSH_INTERVAL_MILLIS:1000}
|
||||
event-history:
|
||||
enabled: true
|
||||
vehicle-state:
|
||||
enabled: false
|
||||
vehicle-stat:
|
||||
enabled: false
|
||||
|
||||
management:
|
||||
endpoints:
|
||||
web:
|
||||
exposure:
|
||||
include: health,info,metrics,prometheus,env
|
||||
health:
|
||||
redis:
|
||||
enabled: ${MANAGEMENT_HEALTH_REDIS_ENABLED:false}
|
||||
metrics:
|
||||
tags:
|
||||
application: vehicle-history-app
|
||||
@@ -0,0 +1,83 @@
|
||||
package com.lingniu.ingest.historyapp;
|
||||
|
||||
import com.lingniu.ingest.eventfilestore.EventFileStore;
|
||||
import com.lingniu.ingest.eventfilestore.EventFileStoreSink;
|
||||
import com.lingniu.ingest.eventfilestore.config.EventFileStoreAutoConfiguration;
|
||||
import com.lingniu.ingest.eventhistory.EventHistoryController;
|
||||
import com.lingniu.ingest.eventhistory.EventHistoryEnvelopeIngestor;
|
||||
import com.lingniu.ingest.eventhistory.Gb32960DecodedFrameService;
|
||||
import com.lingniu.ingest.eventhistory.Gb32960FrameController;
|
||||
import com.lingniu.ingest.eventhistory.config.EventHistoryAutoConfiguration;
|
||||
import com.lingniu.ingest.protocol.gb32960.codec.Gb32960MessageDecoder;
|
||||
import com.lingniu.ingest.protocol.gb32960.config.Gb32960AutoConfiguration;
|
||||
import com.lingniu.ingest.protocol.gb32960.inbound.Gb32960NettyServer;
|
||||
import com.lingniu.ingest.sink.archive.ArchiveStore;
|
||||
import com.lingniu.ingest.sink.archive.RawArchiveEventSink;
|
||||
import com.lingniu.ingest.sink.archive.config.SinkArchiveAutoConfiguration;
|
||||
import com.lingniu.ingest.sink.mq.KafkaEnvelopeDeadLetterSink;
|
||||
import com.lingniu.ingest.sink.mq.KafkaEventSink;
|
||||
import com.lingniu.ingest.sink.mq.SinkMqAutoConfiguration;
|
||||
import org.apache.kafka.clients.producer.KafkaProducer;
|
||||
import org.junit.jupiter.api.Test;
|
||||
import org.junit.jupiter.api.io.TempDir;
|
||||
import org.springframework.boot.autoconfigure.AutoConfigurations;
|
||||
import org.springframework.boot.test.context.runner.ApplicationContextRunner;
|
||||
|
||||
import java.nio.file.Path;
|
||||
|
||||
import static org.assertj.core.api.Assertions.assertThat;
|
||||
import static org.mockito.Mockito.mock;
|
||||
|
||||
class VehicleHistoryAppCompositionTest {
|
||||
|
||||
@TempDir
|
||||
Path tempDir;
|
||||
|
||||
@Test
|
||||
void createsHistoryStorageAndQueryBeansWithoutGb32960TcpServer() {
|
||||
new ApplicationContextRunner()
|
||||
.withConfiguration(AutoConfigurations.of(
|
||||
SinkArchiveAutoConfiguration.class,
|
||||
EventFileStoreAutoConfiguration.class,
|
||||
SinkMqAutoConfiguration.class,
|
||||
Gb32960AutoConfiguration.class,
|
||||
EventHistoryAutoConfiguration.class))
|
||||
.withAllowBeanDefinitionOverriding(true)
|
||||
.withBean("kafkaProducer", KafkaProducer.class, VehicleHistoryAppCompositionTest::kafkaProducer)
|
||||
.withPropertyValues(
|
||||
"lingniu.ingest.sink.archive.enabled=true",
|
||||
"lingniu.ingest.sink.archive.type=local",
|
||||
"lingniu.ingest.sink.archive.path=" + tempDir.resolve("archive"),
|
||||
"lingniu.ingest.event-file-store.enabled=true",
|
||||
"lingniu.ingest.event-file-store.path=" + tempDir.resolve("event-store"),
|
||||
"lingniu.ingest.event-history.enabled=true",
|
||||
"lingniu.ingest.gb32960.enabled=true",
|
||||
"lingniu.ingest.gb32960.server.enabled=false",
|
||||
"lingniu.ingest.sink.mq.enabled=true",
|
||||
"lingniu.ingest.sink.mq.type=kafka",
|
||||
"lingniu.ingest.sink.mq.bootstrap-servers=localhost:9092",
|
||||
"lingniu.ingest.sink.mq.consumer.enabled=false",
|
||||
"lingniu.ingest.vehicle-state.enabled=false",
|
||||
"lingniu.ingest.vehicle-stat.enabled=false")
|
||||
.run(context -> {
|
||||
assertThat(context).hasSingleBean(ArchiveStore.class);
|
||||
assertThat(context).hasSingleBean(RawArchiveEventSink.class);
|
||||
assertThat(context).hasSingleBean(EventFileStore.class);
|
||||
assertThat(context).hasSingleBean(EventFileStoreSink.class);
|
||||
assertThat(context).hasSingleBean(KafkaEventSink.class);
|
||||
assertThat(context).hasSingleBean(KafkaEnvelopeDeadLetterSink.class);
|
||||
|
||||
assertThat(context).hasSingleBean(EventHistoryEnvelopeIngestor.class);
|
||||
assertThat(context).hasSingleBean(EventHistoryController.class);
|
||||
assertThat(context).hasSingleBean(Gb32960MessageDecoder.class);
|
||||
assertThat(context).hasSingleBean(Gb32960DecodedFrameService.class);
|
||||
assertThat(context).hasSingleBean(Gb32960FrameController.class);
|
||||
assertThat(context).doesNotHaveBean(Gb32960NettyServer.class);
|
||||
});
|
||||
}
|
||||
|
||||
@SuppressWarnings("unchecked")
|
||||
private static KafkaProducer<String, byte[]> kafkaProducer() {
|
||||
return mock(KafkaProducer.class);
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,50 @@
|
||||
package com.lingniu.ingest.historyapp;
|
||||
|
||||
import org.junit.jupiter.api.Test;
|
||||
import org.springframework.beans.factory.config.YamlPropertiesFactoryBean;
|
||||
import org.springframework.core.io.ClassPathResource;
|
||||
|
||||
import java.util.Properties;
|
||||
|
||||
import static org.assertj.core.api.Assertions.assertThat;
|
||||
|
||||
class VehicleHistoryAppDefaultsTest {
|
||||
|
||||
@Test
|
||||
void applicationDefaultsKeepHistoryAsDecoderConsumerAndStorageRuntime() {
|
||||
Properties properties = applicationProperties();
|
||||
|
||||
assertThat(properties)
|
||||
.containsEntry("spring.application.name", "vehicle-history-app")
|
||||
.containsEntry("lingniu.ingest.gb32960.enabled", true)
|
||||
.containsEntry("lingniu.ingest.gb32960.server.enabled", false)
|
||||
.containsEntry("lingniu.ingest.sink.archive.enabled", "${SINK_ARCHIVE_ENABLED:true}")
|
||||
.containsEntry("lingniu.ingest.event-file-store.enabled", "${EVENT_FILE_STORE_ENABLED:true}")
|
||||
.containsEntry("lingniu.ingest.event-history.enabled", true)
|
||||
.containsEntry("lingniu.ingest.vehicle-state.enabled", false)
|
||||
.containsEntry("lingniu.ingest.vehicle-stat.enabled", false)
|
||||
.containsEntry("lingniu.ingest.sink.mq.consumer.enabled", "${KAFKA_CONSUMER_ENABLED:true}")
|
||||
.containsEntry(
|
||||
"lingniu.ingest.sink.mq.consumer.bindings.eventHistoryEnvelopeConsumerProcessor.enabled",
|
||||
true)
|
||||
.containsEntry(
|
||||
"lingniu.ingest.sink.mq.consumer.bindings.eventHistoryEnvelopeConsumerProcessor.group-id",
|
||||
"${KAFKA_GROUP_HISTORY:vehicle-history}")
|
||||
.containsEntry(
|
||||
"lingniu.ingest.sink.mq.consumer.bindings.eventHistoryEnvelopeConsumerProcessor.topics[0]",
|
||||
"${KAFKA_TOPIC_GB32960_EVENT:vehicle.event.gb32960.v1}")
|
||||
.containsEntry(
|
||||
"lingniu.ingest.sink.mq.consumer.bindings.eventHistoryEnvelopeConsumerProcessor.topics[1]",
|
||||
"${KAFKA_TOPIC_GB32960_RAW:vehicle.raw.gb32960.v1}");
|
||||
}
|
||||
|
||||
private static Properties applicationProperties() {
|
||||
YamlPropertiesFactoryBean factory = new YamlPropertiesFactoryBean();
|
||||
factory.setResources(new ClassPathResource("application.yml"));
|
||||
|
||||
Properties properties = factory.getObject();
|
||||
|
||||
assertThat(properties).isNotNull();
|
||||
return properties;
|
||||
}
|
||||
}
|
||||
@@ -13,7 +13,12 @@ import com.lmax.disruptor.dsl.ProducerType;
|
||||
import org.slf4j.Logger;
|
||||
import org.slf4j.LoggerFactory;
|
||||
|
||||
import java.util.ArrayList;
|
||||
import java.util.List;
|
||||
import java.util.Objects;
|
||||
import java.util.concurrent.CompletableFuture;
|
||||
import java.util.concurrent.ExecutorService;
|
||||
import java.util.concurrent.Executors;
|
||||
import java.util.concurrent.ThreadFactory;
|
||||
import java.util.concurrent.atomic.AtomicLong;
|
||||
|
||||
@@ -28,10 +33,14 @@ public final class DisruptorEventBus implements AutoCloseable {
|
||||
private static final Logger log = LoggerFactory.getLogger(DisruptorEventBus.class);
|
||||
|
||||
private final Disruptor<VehicleEventSlot> disruptor;
|
||||
private final List<EventSink> sinks;
|
||||
private final ExecutorService awaitExecutor;
|
||||
private final AtomicLong published = new AtomicLong();
|
||||
private final AtomicLong failed = new AtomicLong();
|
||||
|
||||
public DisruptorEventBus(int ringBufferSize, String waitStrategyName, List<EventSink> sinks) {
|
||||
this.sinks = List.copyOf(sinks);
|
||||
this.awaitExecutor = Executors.newVirtualThreadPerTaskExecutor();
|
||||
ThreadFactory tf = Thread.ofVirtual().name("ingest-bus-", 0).factory();
|
||||
this.disruptor = new Disruptor<>(
|
||||
VehicleEventSlot::new,
|
||||
@@ -40,14 +49,14 @@ public final class DisruptorEventBus implements AutoCloseable {
|
||||
ProducerType.MULTI,
|
||||
waitStrategy(waitStrategyName));
|
||||
|
||||
EventHandler<VehicleEventSlot>[] handlers = sinks.stream()
|
||||
EventHandler<VehicleEventSlot>[] handlers = this.sinks.stream()
|
||||
.map(this::toHandler)
|
||||
.toArray(EventHandler[]::new);
|
||||
// 每个 sink 一个独立 handler，事件按扇出模式同时写 Kafka、event-file-store 等目标。
|
||||
disruptor.handleEventsWith(handlers);
|
||||
disruptor.start();
|
||||
log.info("DisruptorEventBus started ringBuffer={} wait={} sinks={}",
|
||||
ringBufferSize, waitStrategyName, sinks.size());
|
||||
ringBufferSize, waitStrategyName, this.sinks.size());
|
||||
}
|
||||
|
||||
public void publish(VehicleEvent event) {
|
||||
@@ -55,9 +64,36 @@ public final class DisruptorEventBus implements AutoCloseable {
|
||||
published.incrementAndGet();
|
||||
}
|
||||
|
||||
public CompletableFuture<Void> publishAndAwait(VehicleEvent event, String requiredSinkName) {
|
||||
String requiredSink = Objects.requireNonNull(requiredSinkName, "requiredSinkName");
|
||||
published.incrementAndGet();
|
||||
List<EventSink> acceptingSinks = sinks.stream()
|
||||
.filter(sink -> sink.accepts(event))
|
||||
.toList();
|
||||
List<EventSink> requiredSinks = acceptingSinks.stream()
|
||||
.filter(sink -> requiredSink.equals(sink.name()))
|
||||
.toList();
|
||||
if (requiredSinks.isEmpty()) {
|
||||
return CompletableFuture.failedFuture(new IllegalStateException(
|
||||
"required sink '" + requiredSink + "' did not accept event " + event.eventId()));
|
||||
}
|
||||
|
||||
List<CompletableFuture<Void>> requiredFutures = new ArrayList<>();
|
||||
for (EventSink sink : acceptingSinks) {
|
||||
CompletableFuture<Void> future = CompletableFuture
|
||||
.supplyAsync(() -> publishToSinkAndTrack(sink, event), awaitExecutor)
|
||||
.thenCompose(f -> f);
|
||||
if (requiredSink.equals(sink.name())) {
|
||||
requiredFutures.add(future);
|
||||
}
|
||||
}
|
||||
return CompletableFuture.allOf(requiredFutures.toArray(CompletableFuture[]::new));
|
||||
}
|
||||
|
||||
@Override
|
||||
public void close() {
|
||||
disruptor.shutdown();
|
||||
awaitExecutor.shutdown();
|
||||
log.info("DisruptorEventBus stopped published={} failed={}", published.get(), failed.get());
|
||||
}
|
||||
|
||||
@@ -66,18 +102,28 @@ public final class DisruptorEventBus implements AutoCloseable {
|
||||
VehicleEvent e = slot.event;
|
||||
if (e == null || !sink.accepts(e)) return;
|
||||
try {
|
||||
sink.publish(e).exceptionally(ex -> {
|
||||
// sink 异步失败只计数和打日志，不反向阻塞入站连接；生产侧靠 sink 自身重试/DLQ 保证可观测。
|
||||
failed.incrementAndGet();
|
||||
log.warn("sink {} publish failed", sink.name(), ex);
|
||||
return null;
|
||||
});
|
||||
publishToSinkAndTrack(sink, e);
|
||||
} finally {
|
||||
if (endOfBatch) slot.clear();
|
||||
}
|
||||
};
|
||||
}
|
||||
|
||||
private CompletableFuture<Void> publishToSinkAndTrack(EventSink sink, VehicleEvent event) {
|
||||
try {
|
||||
return sink.publish(event).whenComplete((ignored, ex) -> {
|
||||
if (ex == null) return;
|
||||
// sink 异步失败只计数和打日志；await 调用方仍会收到异常并决定是否 ACK。
|
||||
failed.incrementAndGet();
|
||||
log.warn("sink {} publish failed", sink.name(), ex);
|
||||
});
|
||||
} catch (Throwable t) {
|
||||
failed.incrementAndGet();
|
||||
log.warn("sink {} publish failed", sink.name(), t);
|
||||
return CompletableFuture.failedFuture(t);
|
||||
}
|
||||
}
|
||||
|
||||
private static WaitStrategy waitStrategy(String name) {
|
||||
return switch (name == null ? "yielding" : name.toLowerCase()) {
|
||||
case "blocking" -> new BlockingWaitStrategy();
|
||||
|
||||
@@ -10,11 +10,16 @@ import com.lingniu.ingest.core.pipeline.InterceptorChain;
|
||||
import org.slf4j.Logger;
|
||||
import org.slf4j.LoggerFactory;
|
||||
|
||||
import java.time.Duration;
|
||||
import java.time.Instant;
|
||||
import java.util.ArrayList;
|
||||
import java.util.LinkedHashMap;
|
||||
import java.util.List;
|
||||
import java.util.Map;
|
||||
import java.util.UUID;
|
||||
import java.util.concurrent.CompletableFuture;
|
||||
import java.util.concurrent.CompletionException;
|
||||
import java.util.concurrent.TimeUnit;
|
||||
import java.util.concurrent.atomic.AtomicLong;
|
||||
|
||||
/**
|
||||
@@ -30,6 +35,7 @@ public final class Dispatcher {
|
||||
|
||||
private static final Logger log = LoggerFactory.getLogger(Dispatcher.class);
|
||||
private static final AtomicLong RAW_ARCHIVE_SEQUENCE = new AtomicLong();
|
||||
private static final String REQUIRED_DURABLE_SINK = "kafka";
|
||||
|
||||
private final HandlerRegistry registry;
|
||||
private final InterceptorChain interceptors;
|
||||
@@ -52,13 +58,67 @@ public final class Dispatcher {
|
||||
public void dispatch(RawFrame frame) {
|
||||
IngestContext ctx = new IngestContext(UUID.randomUUID().toString());
|
||||
try {
|
||||
// 在 interceptor 之前发 RawArchive，保证原始字节被无条件落盘（dedup/rate-limit
|
||||
// 不会过滤它），满足"原始可回放"目标。archive 的写盘靠下游 ArchiveEventSink 消费。
|
||||
RawArchiveLookup rawArchive = emitRawArchive(frame, ctx);
|
||||
DispatchPlan plan = mapFrameToEvents(frame, ctx, false);
|
||||
for (VehicleEvent e : plan.events()) {
|
||||
eventBus.publish(e);
|
||||
}
|
||||
if (plan.failure() != null) {
|
||||
throw plan.failure();
|
||||
}
|
||||
} catch (Throwable t) {
|
||||
log.error("dispatch failure traceId={}", ctx.traceId(), t);
|
||||
interceptors.onError(t, ctx);
|
||||
}
|
||||
}
|
||||
|
||||
public CompletableFuture<Void> dispatchAndAwait(RawFrame frame) {
|
||||
IngestContext ctx = new IngestContext(UUID.randomUUID().toString());
|
||||
try {
|
||||
DispatchPlan plan = mapFrameToEvents(frame, ctx, true);
|
||||
if (plan.events().isEmpty() && plan.failure() == null) {
|
||||
return CompletableFuture.failedFuture(new IllegalStateException(
|
||||
"awaited dispatch produced no events; required sink '" + REQUIRED_DURABLE_SINK + "' was not reached"));
|
||||
}
|
||||
CompletableFuture<?>[] futures = plan.events().stream()
|
||||
.map(event -> eventBus.publishAndAwait(event, REQUIRED_DURABLE_SINK))
|
||||
.toArray(CompletableFuture[]::new);
|
||||
CompletableFuture<Void> boundary = CompletableFuture.allOf(futures);
|
||||
if (plan.failure() == null) {
|
||||
return boundary;
|
||||
}
|
||||
return boundary.handle((ignored, publishFailure) -> {
|
||||
log.error("dispatch failure traceId={}", ctx.traceId(), plan.failure());
|
||||
interceptors.onError(plan.failure(), ctx);
|
||||
if (publishFailure != null) {
|
||||
throw new CompletionException(publishFailure);
|
||||
}
|
||||
throw new CompletionException(plan.failure());
|
||||
});
|
||||
} catch (Throwable t) {
|
||||
log.error("dispatch failure traceId={}", ctx.traceId(), t);
|
||||
interceptors.onError(t, ctx);
|
||||
return CompletableFuture.failedFuture(t);
|
||||
}
|
||||
}
|
||||
|
||||
public CompletableFuture<Void> dispatchAndAwait(RawFrame frame, Duration timeout) {
|
||||
CompletableFuture<Void> future = dispatchAndAwait(frame);
|
||||
if (timeout == null || timeout.isZero() || timeout.isNegative()) {
|
||||
return future;
|
||||
}
|
||||
return future.orTimeout(timeout.toMillis(), TimeUnit.MILLISECONDS);
|
||||
}
|
||||
|
||||
private DispatchPlan mapFrameToEvents(RawFrame frame, IngestContext ctx, boolean awaitDurability) {
|
||||
List<VehicleEvent> out = new ArrayList<>();
|
||||
// 在 interceptor 之前发 RawArchive，保证原始字节被无条件落盘（dedup/rate-limit
|
||||
// 不会过滤它），满足"原始可回放"目标。archive 的写盘靠下游 Sink 消费。
|
||||
RawArchiveLookup rawArchive = appendRawArchive(frame, ctx, out);
|
||||
|
||||
try {
|
||||
if (!interceptors.before(frame, ctx)) {
|
||||
log.debug("frame aborted: {}", ctx.abortReason());
|
||||
return;
|
||||
return new DispatchPlan(out, null);
|
||||
}
|
||||
|
||||
List<HandlerDefinition> handlers = registry.resolve(
|
||||
@@ -67,11 +127,16 @@ public final class Dispatcher {
|
||||
log.debug("no handler for {} cmd=0x{} info=0x{}",
|
||||
frame.protocolId(), Integer.toHexString(frame.command()),
|
||||
Integer.toHexString(frame.infoType()));
|
||||
return;
|
||||
return new DispatchPlan(out, null);
|
||||
}
|
||||
|
||||
for (HandlerDefinition def : handlers) {
|
||||
if (def.asyncBatch() != null) {
|
||||
if (awaitDurability) {
|
||||
throw new AwaitedAsyncBatchUnsupportedException(
|
||||
"@AsyncBatch handler cannot be used with dispatchAndAwait until batch completion is awaited: "
|
||||
+ def.method());
|
||||
}
|
||||
batchExecutor.submit(def, frame.payload(), event -> enrichWithRawArchive(event, rawArchive));
|
||||
continue;
|
||||
}
|
||||
@@ -79,12 +144,15 @@ public final class Dispatcher {
|
||||
for (VehicleEvent e : events) {
|
||||
e = enrichWithRawArchive(e, rawArchive);
|
||||
interceptors.after(e, ctx);
|
||||
eventBus.publish(e);
|
||||
out.add(e);
|
||||
}
|
||||
}
|
||||
return new DispatchPlan(out, null);
|
||||
} catch (Throwable t) {
|
||||
log.error("dispatch failure traceId={}", ctx.traceId(), t);
|
||||
interceptors.onError(t, ctx);
|
||||
if (t instanceof AwaitedAsyncBatchUnsupportedException) {
|
||||
return new DispatchPlan(List.of(), t);
|
||||
}
|
||||
return new DispatchPlan(out, t);
|
||||
}
|
||||
}
|
||||
|
||||
@@ -95,7 +163,7 @@ public final class Dispatcher {
|
||||
* <p>VIN 取自 sourceMeta 里的 {@code vin} key（由各入站适配器负责填充）；缺失时
|
||||
* 留空字符串，archive sink 会用 "unknown-vin" 占位保证 key 可解析。
|
||||
*/
|
||||
private RawArchiveLookup emitRawArchive(RawFrame frame, IngestContext ctx) {
|
||||
private RawArchiveLookup appendRawArchive(RawFrame frame, IngestContext ctx, List<VehicleEvent> out) {
|
||||
byte[] bytes = frame.rawBytes();
|
||||
if (bytes == null || bytes.length == 0) return RawArchiveLookup.empty();
|
||||
|
||||
@@ -117,7 +185,7 @@ public final class Dispatcher {
|
||||
frame.command(),
|
||||
frame.infoType(),
|
||||
bytes);
|
||||
eventBus.publish(raw);
|
||||
out.add(raw);
|
||||
return new RawArchiveLookup(eventId, key, RawArchiveKeys.logicalUri(key));
|
||||
}
|
||||
|
||||
@@ -182,4 +250,13 @@ public final class Dispatcher {
|
||||
return key == null || key.isBlank();
|
||||
}
|
||||
}
|
||||
|
||||
private record DispatchPlan(List<VehicleEvent> events, Throwable failure) {
|
||||
}
|
||||
|
||||
private static final class AwaitedAsyncBatchUnsupportedException extends IllegalStateException {
|
||||
private AwaitedAsyncBatchUnsupportedException(String message) {
|
||||
super(message);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@@ -0,0 +1,81 @@
|
||||
package com.lingniu.ingest.core.concurrency;
|
||||
|
||||
import com.lingniu.ingest.api.ProtocolId;
|
||||
import com.lingniu.ingest.api.event.VehicleEvent;
|
||||
import com.lingniu.ingest.api.sink.EventSink;
|
||||
import org.junit.jupiter.api.Test;
|
||||
|
||||
import java.time.Instant;
|
||||
import java.util.List;
|
||||
import java.util.Map;
|
||||
import java.util.concurrent.CompletableFuture;
|
||||
import java.util.concurrent.CompletionException;
|
||||
import java.util.concurrent.CountDownLatch;
|
||||
import java.util.concurrent.TimeUnit;
|
||||
import java.util.concurrent.atomic.AtomicLong;
|
||||
|
||||
import static org.assertj.core.api.Assertions.assertThat;
|
||||
import static org.assertj.core.api.Assertions.assertThatThrownBy;
|
||||
|
||||
class DisruptorEventBusAwaitTest {
|
||||
|
||||
@Test
|
||||
void publishAndAwaitDoesNotInvokeSinkPublishOnCallerThread() throws Exception {
|
||||
long callerThreadId = Thread.currentThread().threadId();
|
||||
CapturingSink sink = new CapturingSink("kafka");
|
||||
|
||||
try (DisruptorEventBus bus = new DisruptorEventBus(1024, "blocking", java.util.List.of(sink))) {
|
||||
bus.publishAndAwait(event(), "kafka").get(3, TimeUnit.SECONDS);
|
||||
}
|
||||
|
||||
assertThat(sink.publishThreadId.get()).isNotEqualTo(callerThreadId);
|
||||
}
|
||||
|
||||
@Test
|
||||
void publishAndAwaitDoesNotInvokeOptionalSinksWhenRequiredSinkIsAbsent() throws Exception {
|
||||
CapturingSink optionalSink = new CapturingSink("event-file-store");
|
||||
|
||||
try (DisruptorEventBus bus = new DisruptorEventBus(1024, "blocking", List.of(optionalSink))) {
|
||||
assertThatThrownBy(() -> bus.publishAndAwait(event(), "kafka").join())
|
||||
.isInstanceOf(CompletionException.class)
|
||||
.hasRootCauseInstanceOf(IllegalStateException.class)
|
||||
.hasMessageContaining("kafka");
|
||||
|
||||
assertThat(optionalSink.published.await(200, TimeUnit.MILLISECONDS)).isFalse();
|
||||
}
|
||||
}
|
||||
|
||||
private static VehicleEvent.Heartbeat event() {
|
||||
Instant now = Instant.parse("2026-06-23T10:00:00Z");
|
||||
return new VehicleEvent.Heartbeat(
|
||||
"heartbeat-1",
|
||||
"VIN001",
|
||||
ProtocolId.GB32960,
|
||||
now,
|
||||
now,
|
||||
"trace-1",
|
||||
Map.of());
|
||||
}
|
||||
|
||||
private static final class CapturingSink implements EventSink {
|
||||
private final String name;
|
||||
private final AtomicLong publishThreadId = new AtomicLong(-1);
|
||||
private final CountDownLatch published = new CountDownLatch(1);
|
||||
|
||||
private CapturingSink(String name) {
|
||||
this.name = name;
|
||||
}
|
||||
|
||||
@Override
|
||||
public String name() {
|
||||
return name;
|
||||
}
|
||||
|
||||
@Override
|
||||
public CompletableFuture<Void> publish(VehicleEvent event) {
|
||||
publishThreadId.set(Thread.currentThread().threadId());
|
||||
published.countDown();
|
||||
return CompletableFuture.completedFuture(null);
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,217 @@
|
||||
package com.lingniu.ingest.core.dispatcher;
|
||||
|
||||
import com.lingniu.ingest.api.ProtocolId;
|
||||
import com.lingniu.ingest.api.annotation.AsyncBatch;
|
||||
import com.lingniu.ingest.api.annotation.MessageMapping;
|
||||
import com.lingniu.ingest.api.annotation.ProtocolHandler;
|
||||
import com.lingniu.ingest.api.event.VehicleEvent;
|
||||
import com.lingniu.ingest.api.pipeline.IngestContext;
|
||||
import com.lingniu.ingest.api.pipeline.IngestInterceptor;
|
||||
import com.lingniu.ingest.api.pipeline.RawFrame;
|
||||
import com.lingniu.ingest.api.sink.EventSink;
|
||||
import com.lingniu.ingest.core.concurrency.AsyncBatchExecutor;
|
||||
import com.lingniu.ingest.core.concurrency.DisruptorEventBus;
|
||||
import com.lingniu.ingest.core.pipeline.InterceptorChain;
|
||||
import org.junit.jupiter.api.Test;
|
||||
|
||||
import java.lang.reflect.Method;
|
||||
import java.time.Instant;
|
||||
import java.util.List;
|
||||
import java.util.Map;
|
||||
import java.util.concurrent.CompletableFuture;
|
||||
import java.util.concurrent.CompletionException;
|
||||
import java.util.concurrent.CompletionStage;
|
||||
import java.util.concurrent.atomic.AtomicInteger;
|
||||
import java.util.concurrent.atomic.AtomicReference;
|
||||
|
||||
import static org.assertj.core.api.Assertions.assertThat;
|
||||
import static org.assertj.core.api.Assertions.assertThatThrownBy;
|
||||
|
||||
class DispatcherDurableAckBoundaryTest {
|
||||
|
||||
@Test
|
||||
void dispatchAndAwaitFailsWhenNoKafkaSinkAcceptsEvent() throws Exception {
|
||||
EventSink fileSink = new NamedSink("event-file-store", true);
|
||||
|
||||
try (Harness harness = new Harness(registryWithHeartbeatHandler(), List.of(fileSink), List.of())) {
|
||||
CompletionStage<Void> future = harness.dispatcher.dispatchAndAwait(frame(0x07, "payload"));
|
||||
|
||||
assertThatThrownBy(() -> future.toCompletableFuture().join())
|
||||
.isInstanceOf(CompletionException.class)
|
||||
.hasRootCauseInstanceOf(IllegalStateException.class)
|
||||
.hasMessageContaining("kafka");
|
||||
}
|
||||
}
|
||||
|
||||
@Test
|
||||
void dispatchAndAwaitNotifiesInterceptorWhenPlanCapturesFailure() throws Exception {
|
||||
CapturingInterceptor interceptor = new CapturingInterceptor();
|
||||
|
||||
try (Harness harness = new Harness(registryWithThrowingHandler(), List.of(new NamedSink("kafka", true)),
|
||||
List.of(interceptor))) {
|
||||
CompletionStage<Void> future = harness.dispatcher.dispatchAndAwait(frame(0x08, "payload"));
|
||||
|
||||
assertThatThrownBy(() -> future.toCompletableFuture().join())
|
||||
.isInstanceOf(CompletionException.class)
|
||||
.hasRootCauseInstanceOf(IllegalStateException.class)
|
||||
.rootCause()
|
||||
.hasMessageContaining("handler failed");
|
||||
assertThat(interceptor.error.get())
|
||||
.isInstanceOf(RuntimeException.class)
|
||||
.hasMessageContaining("Handler threw exception");
|
||||
}
|
||||
}
|
||||
|
||||
@Test
|
||||
void dispatchAndAwaitFailsFastForAsyncBatchHandler() throws Exception {
|
||||
NamedSink kafka = new NamedSink("kafka", true);
|
||||
try (Harness harness = new Harness(registryWithAsyncBatchHandler(), List.of(kafka), List.of())) {
|
||||
CompletionStage<Void> future = harness.dispatcher.dispatchAndAwait(
|
||||
frame(0x09, "payload", new byte[]{0x23, 0x23, 0x09}));
|
||||
|
||||
assertThatThrownBy(() -> future.toCompletableFuture().join())
|
||||
.isInstanceOf(CompletionException.class)
|
||||
.hasRootCauseInstanceOf(IllegalStateException.class)
|
||||
.hasMessageContaining("@AsyncBatch");
|
||||
assertThat(kafka.published.get()).isZero();
|
||||
}
|
||||
}
|
||||
|
||||
private static HandlerRegistry registryWithHeartbeatHandler() throws NoSuchMethodException {
|
||||
HeartbeatHandler bean = new HeartbeatHandler();
|
||||
Method method = HeartbeatHandler.class.getDeclaredMethod("onHeartbeat", Object.class);
|
||||
return registry(bean, method, 0x07, method.getAnnotation(AsyncBatch.class));
|
||||
}
|
||||
|
||||
private static HandlerRegistry registryWithThrowingHandler() throws NoSuchMethodException {
|
||||
ThrowingHandler bean = new ThrowingHandler();
|
||||
Method method = ThrowingHandler.class.getDeclaredMethod("onFrame", Object.class);
|
||||
return registry(bean, method, 0x08, method.getAnnotation(AsyncBatch.class));
|
||||
}
|
||||
|
||||
private static HandlerRegistry registryWithAsyncBatchHandler() throws NoSuchMethodException {
|
||||
AsyncHandler bean = new AsyncHandler();
|
||||
Method method = AsyncHandler.class.getDeclaredMethod("onBatch", List.class);
|
||||
return registry(bean, method, 0x09, method.getAnnotation(AsyncBatch.class));
|
||||
}
|
||||
|
||||
private static HandlerRegistry registry(Object bean, Method method, int command, AsyncBatch asyncBatch) {
|
||||
HandlerRegistry registry = new HandlerRegistry();
|
||||
registry.register(new HandlerDefinition(
|
||||
ProtocolId.GB32960,
|
||||
command,
|
||||
0,
|
||||
method.getName(),
|
||||
bean,
|
||||
method,
|
||||
Object.class,
|
||||
null,
|
||||
null,
|
||||
asyncBatch));
|
||||
return registry;
|
||||
}
|
||||
|
||||
private static RawFrame frame(int command, Object payload) {
|
||||
return frame(command, payload, null);
|
||||
}
|
||||
|
||||
private static RawFrame frame(int command, Object payload, byte[] rawBytes) {
|
||||
return new RawFrame(
|
||||
ProtocolId.GB32960,
|
||||
command,
|
||||
0,
|
||||
payload,
|
||||
rawBytes,
|
||||
Map.of("vin", "VIN001"),
|
||||
Instant.parse("2026-06-23T10:00:00Z"));
|
||||
}
|
||||
|
||||
private static VehicleEvent.Heartbeat event(String id) {
|
||||
Instant now = Instant.parse("2026-06-23T10:00:00Z");
|
||||
return new VehicleEvent.Heartbeat(id, "VIN001", ProtocolId.GB32960, now, now, "trace-" + id, Map.of());
|
||||
}
|
||||
|
||||
private static final class Harness implements AutoCloseable {
|
||||
private final DisruptorEventBus eventBus;
|
||||
private final AsyncBatchExecutor batchExecutor;
|
||||
private final Dispatcher dispatcher;
|
||||
|
||||
private Harness(HandlerRegistry registry, List<EventSink> sinks, List<IngestInterceptor> interceptors) {
|
||||
this.eventBus = new DisruptorEventBus(1024, "blocking", sinks);
|
||||
this.batchExecutor = new AsyncBatchExecutor(eventBus::publish);
|
||||
this.dispatcher = new Dispatcher(
|
||||
registry,
|
||||
new InterceptorChain(interceptors),
|
||||
new HandlerInvoker(),
|
||||
eventBus,
|
||||
batchExecutor);
|
||||
}
|
||||
|
||||
@Override
|
||||
public void close() {
|
||||
batchExecutor.close();
|
||||
eventBus.close();
|
||||
}
|
||||
}
|
||||
|
||||
private static final class NamedSink implements EventSink {
|
||||
private final String name;
|
||||
private final boolean accepts;
|
||||
private final AtomicInteger published = new AtomicInteger();
|
||||
|
||||
private NamedSink(String name, boolean accepts) {
|
||||
this.name = name;
|
||||
this.accepts = accepts;
|
||||
}
|
||||
|
||||
@Override
|
||||
public String name() {
|
||||
return name;
|
||||
}
|
||||
|
||||
@Override
|
||||
public CompletableFuture<Void> publish(VehicleEvent event) {
|
||||
published.incrementAndGet();
|
||||
return CompletableFuture.completedFuture(null);
|
||||
}
|
||||
|
||||
@Override
|
||||
public boolean accepts(VehicleEvent event) {
|
||||
return accepts;
|
||||
}
|
||||
}
|
||||
|
||||
private static final class CapturingInterceptor implements IngestInterceptor {
|
||||
private final AtomicReference<Throwable> error = new AtomicReference<>();
|
||||
|
||||
@Override
|
||||
public void onError(Throwable error, IngestContext ctx) {
|
||||
this.error.set(error);
|
||||
}
|
||||
}
|
||||
|
||||
@ProtocolHandler(protocol = ProtocolId.GB32960)
|
||||
static final class HeartbeatHandler {
|
||||
@MessageMapping(command = 0x07)
|
||||
VehicleEvent.Heartbeat onHeartbeat(Object ignored) {
|
||||
return event("heartbeat-1");
|
||||
}
|
||||
}
|
||||
|
||||
@ProtocolHandler(protocol = ProtocolId.GB32960)
|
||||
static final class ThrowingHandler {
|
||||
@MessageMapping(command = 0x08)
|
||||
VehicleEvent.Heartbeat onFrame(Object ignored) {
|
||||
throw new IllegalStateException("handler failed");
|
||||
}
|
||||
}
|
||||
|
||||
@ProtocolHandler(protocol = ProtocolId.GB32960)
|
||||
public static final class AsyncHandler {
|
||||
@MessageMapping(command = 0x09)
|
||||
@AsyncBatch(size = 2, waitMs = 10, poolSize = 1)
|
||||
public List<VehicleEvent> onBatch(List<Object> ignored) {
|
||||
return List.of(event("async-1"));
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -55,5 +55,10 @@
|
||||
<artifactId>assertj-core</artifactId>
|
||||
<scope>test</scope>
|
||||
</dependency>
|
||||
<dependency>
|
||||
<groupId>org.springframework.boot</groupId>
|
||||
<artifactId>spring-boot-test</artifactId>
|
||||
<scope>test</scope>
|
||||
</dependency>
|
||||
</dependencies>
|
||||
</project>
|
||||
|
||||
@@ -59,8 +59,8 @@ import java.util.List;
|
||||
* <p>每个 Parser Bean 都会被 {@link InfoBlockParserRegistry} 按
|
||||
* {@code (ProtocolVersion, typeCode)} 注册。2016/2025 可同时加载。
|
||||
*
|
||||
* <p>只有 {@code lingniu.ingest.gb32960.enabled=true} 时才会启动 32960 TCP 服务。
|
||||
* 解析、鉴权、ACK、RAW 入库都从 {@link Gb32960NettyServer} 建立的 Netty pipeline 进入。
|
||||
* <p>只有 {@code lingniu.ingest.gb32960.enabled=true} 时才会装配 32960 协议解析组件。
|
||||
* TCP 监听由 {@code lingniu.ingest.gb32960.server.enabled} 单独控制。
|
||||
*/
|
||||
@AutoConfiguration
|
||||
@EnableConfigurationProperties(Gb32960Properties.class)
|
||||
@@ -222,6 +222,8 @@ public class Gb32960AutoConfiguration {
|
||||
|
||||
@Bean
|
||||
@ConditionalOnMissingBean
|
||||
@ConditionalOnProperty(prefix = "lingniu.ingest.gb32960.server", name = "enabled",
|
||||
havingValue = "true", matchIfMissing = true)
|
||||
public Gb32960NettyServer gb32960NettyServer(Gb32960Properties props,
|
||||
Gb32960MessageDecoder decoder,
|
||||
Dispatcher dispatcher,
|
||||
|
||||
@@ -10,8 +10,10 @@ import java.util.Set;
|
||||
@ConfigurationProperties(prefix = "lingniu.ingest.gb32960")
|
||||
public class Gb32960Properties {
|
||||
|
||||
/** 是否启动 32960 TCP 监听。生产 32960 线路打开后，此开关决定端口是否真正 bind。 */
|
||||
/** 是否装配 32960 协议解析组件。TCP 监听由 {@link #server} 单独控制。 */
|
||||
private boolean enabled = false;
|
||||
/** 32960 TCP 监听开关。默认跟随协议启用后启动，查询类应用可单独关闭。 */
|
||||
private Server server = new Server();
|
||||
/** 32960 TCP 监听端口；当前生产验证线使用 32960。 */
|
||||
private int port = 9000;
|
||||
/** Netty worker 线程数；0 表示按 Netty/CPU 默认策略分配。 */
|
||||
@@ -63,6 +65,8 @@ public class Gb32960Properties {
|
||||
|
||||
public boolean isEnabled() { return enabled; }
|
||||
public void setEnabled(boolean enabled) { this.enabled = enabled; }
|
||||
public Server getServer() { return server; }
|
||||
public void setServer(Server server) { this.server = server; }
|
||||
public int getPort() { return port; }
|
||||
public void setPort(int port) { this.port = port; }
|
||||
public int getWorkerThreads() { return workerThreads; }
|
||||
@@ -82,6 +86,13 @@ public class Gb32960Properties {
|
||||
this.vendorExtensions = vendorExtensions;
|
||||
}
|
||||
|
||||
public static class Server {
|
||||
private boolean enabled = true;
|
||||
|
||||
public boolean isEnabled() { return enabled; }
|
||||
public void setEnabled(boolean enabled) { this.enabled = enabled; }
|
||||
}
|
||||
|
||||
/**
|
||||
* VIN 白名单认证。
|
||||
*
|
||||
|
||||
@@ -20,6 +20,7 @@ import org.slf4j.LoggerFactory;
|
||||
import java.net.InetSocketAddress;
|
||||
import java.nio.ByteBuffer;
|
||||
import java.time.Instant;
|
||||
import java.util.concurrent.CompletableFuture;
|
||||
import java.util.HashMap;
|
||||
import java.util.List;
|
||||
import java.util.Map;
|
||||
@@ -55,6 +56,9 @@ public class Gb32960ChannelHandler extends SimpleChannelInboundHandler<byte[]> {
|
||||
public static final AttributeKey<String> PLATFORM_ACCOUNT_ATTR =
|
||||
Gb32960AccessService.PLATFORM_ACCOUNT_ATTR;
|
||||
|
||||
private static final AttributeKey<CompletableFuture<Void>> DURABLE_ACK_CHAIN_ATTR =
|
||||
AttributeKey.valueOf("gb32960.durableAckChain");
|
||||
|
||||
private final Gb32960MessageDecoder decoder;
|
||||
private final Dispatcher dispatcher;
|
||||
private final Gb32960AccessService accessService;
|
||||
@@ -122,41 +126,137 @@ public class Gb32960ChannelHandler extends SimpleChannelInboundHandler<byte[]> {
|
||||
return;
|
||||
}
|
||||
|
||||
// 鉴权通过：按命令类型回 ACK，写成功后统一 dispatch 到通用管线。
|
||||
// 鉴权通过：先完成命令侧状态变更/日志，再统一 dispatch 到通用管线。
|
||||
// 需要成功 ACK 的已接收帧，等待配置的 Kafka dispatch 边界完成后再回 ACK。
|
||||
// 平台登入鉴权失败会短路 return，不进 dispatch。
|
||||
switch (cmd) {
|
||||
case VEHICLE_LOGIN -> handleVehicleLogin(ctx, msg, rawVin);
|
||||
case VEHICLE_LOGOUT -> handleVehicleLogout(ctx, msg, rawVin);
|
||||
case VEHICLE_LOGIN -> logVehicleLogin(ctx, msg);
|
||||
case VEHICLE_LOGOUT -> logVehicleLogout(ctx, msg);
|
||||
case PLATFORM_LOGIN -> {
|
||||
if (!handlePlatformLogin(ctx, msg, rawVin)) return;
|
||||
}
|
||||
case PLATFORM_LOGOUT -> handlePlatformLogout(ctx, msg, rawVin);
|
||||
case REALTIME_REPORT, RESEND_REPORT, HEARTBEAT -> handleReportOrHeartbeat(ctx, msg, rawVin);
|
||||
case TIME_CALIBRATION -> handleTimeCalibration(ctx, msg, rawVin);
|
||||
case PLATFORM_LOGOUT -> logPlatformLogout(ctx);
|
||||
case REALTIME_REPORT, RESEND_REPORT, HEARTBEAT -> logReportOrHeartbeat(ctx, msg);
|
||||
case TIME_CALIBRATION -> { }
|
||||
default -> logOtherFrame(ctx, msg);
|
||||
}
|
||||
|
||||
RawFrame rf = rawFrame(ctx, msg, frame);
|
||||
if (!requiresDurableAck(cmd)) {
|
||||
dispatcher.dispatch(rf);
|
||||
return;
|
||||
}
|
||||
|
||||
enqueueDurableAck(ctx, msg, rawVin, vin, cmd, dispatcher.dispatchAndAwait(rf));
|
||||
}
|
||||
|
||||
private void enqueueDurableAck(ChannelHandlerContext ctx,
|
||||
Gb32960Message msg,
|
||||
byte[] rawVin,
|
||||
String vin,
|
||||
CommandType cmd,
|
||||
CompletableFuture<Void> dispatchFuture) {
|
||||
CompletableFuture<Void> previous = ctx.channel().attr(DURABLE_ACK_CHAIN_ATTR).get();
|
||||
if (previous == null) {
|
||||
previous = CompletableFuture.completedFuture(null);
|
||||
}
|
||||
|
||||
CompletableFuture<Void> next = previous
|
||||
.handle((ignored, previousFailure) -> null)
|
||||
.thenCompose(ignored -> dispatchFuture.handle((ok, failure) -> failure))
|
||||
.thenCompose(failure -> runOnEventLoop(ctx,
|
||||
() -> handleDurableAckResult(ctx, msg, rawVin, vin, cmd, failure)));
|
||||
ctx.channel().attr(DURABLE_ACK_CHAIN_ATTR).set(next);
|
||||
}
|
||||
|
||||
private CompletableFuture<Void> runOnEventLoop(ChannelHandlerContext ctx, Runnable task) {
|
||||
CompletableFuture<Void> done = new CompletableFuture<>();
|
||||
Runnable wrapped = () -> {
|
||||
try {
|
||||
task.run();
|
||||
done.complete(null);
|
||||
} catch (Throwable t) {
|
||||
done.completeExceptionally(t);
|
||||
}
|
||||
};
|
||||
if (ctx.executor().inEventLoop()) {
|
||||
wrapped.run();
|
||||
} else {
|
||||
ctx.executor().execute(wrapped);
|
||||
}
|
||||
return done;
|
||||
}
|
||||
|
||||
private void handleDurableAckResult(ChannelHandlerContext ctx,
|
||||
Gb32960Message msg,
|
||||
byte[] rawVin,
|
||||
String vin,
|
||||
CommandType cmd,
|
||||
Throwable failure) {
|
||||
if (!ctx.channel().isActive()) {
|
||||
return;
|
||||
}
|
||||
if (failure != null) {
|
||||
log.warn("[gb32960] dispatch failed before ack peer={} vin={} cmd=0x{}",
|
||||
addr(ctx), vin, Integer.toHexString(cmd.code()), failure);
|
||||
ctx.close();
|
||||
return;
|
||||
}
|
||||
writeAckForCommand(ctx, msg, rawVin);
|
||||
}
|
||||
|
||||
private RawFrame rawFrame(ChannelHandlerContext ctx, Gb32960Message msg, byte[] frame) {
|
||||
Map<String, String> sourceMeta = new HashMap<>(4);
|
||||
sourceMeta.put("vin", vin);
|
||||
sourceMeta.put("vin", msg.header().vin());
|
||||
sourceMeta.put("peer", addr(ctx));
|
||||
String platformAccount = accessService.platformAccount(ctx.channel());
|
||||
if (platformAccount != null && !platformAccount.isBlank()) {
|
||||
sourceMeta.put("platformAccount", platformAccount);
|
||||
}
|
||||
// RawFrame.rawBytes 是后续 raw archive 和按 rawArchiveUri 回查的源头；不要在这里裁剪或重编码。
|
||||
RawFrame rf = new RawFrame(
|
||||
return new RawFrame(
|
||||
ProtocolId.GB32960,
|
||||
cmd.code(),
|
||||
msg.header().command().code(),
|
||||
0,
|
||||
msg,
|
||||
frame,
|
||||
sourceMeta,
|
||||
Instant.now());
|
||||
dispatcher.dispatch(rf);
|
||||
}
|
||||
|
||||
private static boolean requiresDurableAck(CommandType cmd) {
|
||||
return switch (cmd) {
|
||||
case VEHICLE_LOGIN,
|
||||
VEHICLE_LOGOUT,
|
||||
PLATFORM_LOGIN,
|
||||
PLATFORM_LOGOUT,
|
||||
REALTIME_REPORT,
|
||||
RESEND_REPORT,
|
||||
HEARTBEAT,
|
||||
TIME_CALIBRATION -> true;
|
||||
default -> false;
|
||||
};
|
||||
}
|
||||
|
||||
private void writeAckForCommand(ChannelHandlerContext ctx, Gb32960Message msg, byte[] rawVin) {
|
||||
switch (msg.header().command()) {
|
||||
case VEHICLE_LOGIN -> writeVehicleLoginAck(ctx, msg, rawVin);
|
||||
case VEHICLE_LOGOUT -> writeVehicleLogoutAck(ctx, msg, rawVin);
|
||||
case PLATFORM_LOGIN -> writePlatformLoginAck(ctx, msg, rawVin);
|
||||
case PLATFORM_LOGOUT -> writePlatformLogoutAck(ctx, msg, rawVin);
|
||||
case REALTIME_REPORT, RESEND_REPORT, HEARTBEAT -> writeReportOrHeartbeatAck(ctx, msg, rawVin);
|
||||
case TIME_CALIBRATION -> writeTimeCalibrationAck(ctx, msg, rawVin);
|
||||
default -> { }
|
||||
}
|
||||
}
|
||||
|
||||
/** 0x01 车辆登入：按 §6.3.2 带原采集时间回成功 ACK + INFO 日志。 */
|
||||
private void handleVehicleLogin(ChannelHandlerContext ctx, Gb32960Message msg, byte[] rawVin) {
|
||||
private void logVehicleLogin(ChannelHandlerContext ctx, Gb32960Message msg) {
|
||||
log.info("[gb32960] vehicle login peer={} vin={} protocolVersion={}",
|
||||
addr(ctx), msg.header().vin(), msg.header().protocolVersion());
|
||||
}
|
||||
|
||||
private void writeVehicleLoginAck(ChannelHandlerContext ctx, Gb32960Message msg, byte[] rawVin) {
|
||||
ackService.writeResponse(
|
||||
ctx,
|
||||
msg.header().protocolVersion(),
|
||||
@@ -166,13 +266,14 @@ public class Gb32960ChannelHandler extends SimpleChannelInboundHandler<byte[]> {
|
||||
eventOrNow(msg),
|
||||
null,
|
||||
"vehicle-login-ack");
|
||||
log.info("[gb32960] vehicle login peer={} vin={} protocolVersion={}",
|
||||
addr(ctx), msg.header().vin(), msg.header().protocolVersion());
|
||||
}
|
||||
|
||||
/** 0x04 车辆登出：INFO 日志 + 带原采集时间回成功 ACK。 */
|
||||
private void handleVehicleLogout(ChannelHandlerContext ctx, Gb32960Message msg, byte[] rawVin) {
|
||||
private void logVehicleLogout(ChannelHandlerContext ctx, Gb32960Message msg) {
|
||||
log.info("[gb32960] vehicle logout peer={} vin={}", addr(ctx), msg.header().vin());
|
||||
}
|
||||
|
||||
private void writeVehicleLogoutAck(ChannelHandlerContext ctx, Gb32960Message msg, byte[] rawVin) {
|
||||
ackService.writeResponse(ctx,
|
||||
msg.header().protocolVersion(),
|
||||
CommandType.VEHICLE_LOGOUT,
|
||||
@@ -189,8 +290,8 @@ public class Gb32960ChannelHandler extends SimpleChannelInboundHandler<byte[]> {
|
||||
* <ul>
|
||||
* <li>消息体解析缺失：关闭连接，返回 false 短路
|
||||
* <li>鉴权失败：写 {@code 0x02 OTHER_ERROR} NACK + 关闭连接，返回 false 短路
|
||||
* <li>鉴权通过：把 username 钉到 channel attribute（供后续 vendor profile 路由），
|
||||
* 写成功 ACK + INFO 日志，返回 true 继续 dispatch
|
||||
* <li>鉴权通过：把 username 钉到 channel attribute（供后续 vendor profile 路由），INFO 日志，
|
||||
* 返回 true 继续 dispatch；成功 ACK 等 dispatch 持久化边界完成后再写
|
||||
* </ul>
|
||||
*
|
||||
* @return true 表示鉴权通过应继续 dispatch；false 表示调用方应 short-circuit return
|
||||
@@ -221,6 +322,10 @@ public class Gb32960ChannelHandler extends SimpleChannelInboundHandler<byte[]> {
|
||||
addr(ctx), pl.username(), pl.encryptRule(), pl.serialNo(), pl.loginTime(), result);
|
||||
// 把 username 钉到 channel attribute，供后续 0x02/0x03 帧的 vendor profile 路由
|
||||
accessService.bindPlatformAccount(ctx.channel(), pl.username());
|
||||
return true;
|
||||
}
|
||||
|
||||
private void writePlatformLoginAck(ChannelHandlerContext ctx, Gb32960Message msg, byte[] rawVin) {
|
||||
ackService.writeResponse(
|
||||
ctx,
|
||||
msg.header().protocolVersion(),
|
||||
@@ -230,12 +335,14 @@ public class Gb32960ChannelHandler extends SimpleChannelInboundHandler<byte[]> {
|
||||
eventOrNow(msg),
|
||||
null,
|
||||
"platform-login-ack");
|
||||
return true;
|
||||
}
|
||||
|
||||
/** 0x06 平台登出：INFO 日志 + 带原采集时间回成功 ACK。 */
|
||||
private void handlePlatformLogout(ChannelHandlerContext ctx, Gb32960Message msg, byte[] rawVin) {
|
||||
private void logPlatformLogout(ChannelHandlerContext ctx) {
|
||||
log.info("[gb32960] platform logout peer={}", addr(ctx));
|
||||
}
|
||||
|
||||
private void writePlatformLogoutAck(ChannelHandlerContext ctx, Gb32960Message msg, byte[] rawVin) {
|
||||
ackService.writeResponse(ctx,
|
||||
msg.header().protocolVersion(),
|
||||
CommandType.PLATFORM_LOGOUT,
|
||||
@@ -247,11 +354,11 @@ public class Gb32960ChannelHandler extends SimpleChannelInboundHandler<byte[]> {
|
||||
}
|
||||
|
||||
/**
|
||||
* 0x02/0x03/0x07 实时 / 补发 / 心跳：按 §6.3.2 回应答帧，保留原帧采集时间。
|
||||
* 0x02/0x03/0x07 实时 / 补发 / 心跳：记录帧诊断；成功 ACK 在配置的 Kafka dispatch 边界后写回。
|
||||
* 部分车端实现严格按规范，没收到应答会停止后续上报或重连。ack-tag 按 cmd.code 区分便于日志筛选。
|
||||
*
|
||||
* <p>注意：应答成功不代表下游已完成持久化，只表示平台已接收并通过 Dispatcher 投递。
|
||||
* 生产排查存储问题时，应继续检查 raw archive sink、EventFileStoreSink 和 Kafka sink 的日志。
|
||||
* <p>注意：应答成功表示当前配置的 Kafka sink 已完成该帧映射事件的 dispatch 边界。
|
||||
* 生产排查其他存储问题时，应继续检查 raw archive sink、EventFileStoreSink 等可选 sink 的日志。
|
||||
*
|
||||
* <p>日志策略（按 (channel, vin, rawTypeSignature) 去重，避免高频帧刷屏）：
|
||||
* <ul>
|
||||
@@ -264,17 +371,9 @@ public class Gb32960ChannelHandler extends SimpleChannelInboundHandler<byte[]> {
|
||||
* </ul>
|
||||
* 帧内字段全量 JSON 仍保留 DEBUG。
|
||||
*/
|
||||
private void handleReportOrHeartbeat(ChannelHandlerContext ctx, Gb32960Message msg, byte[] rawVin) {
|
||||
private void logReportOrHeartbeat(ChannelHandlerContext ctx, Gb32960Message msg) {
|
||||
CommandType cmd = msg.header().command();
|
||||
String platformAccount = accessService.platformAccount(ctx.channel());
|
||||
ackService.writeResponse(ctx,
|
||||
msg.header().protocolVersion(),
|
||||
cmd,
|
||||
ResponseFlag.SUCCESS,
|
||||
rawVin,
|
||||
eventOrNow(msg),
|
||||
null,
|
||||
"report-ack-0x" + Integer.toHexString(cmd.code()));
|
||||
if (cmd == CommandType.HEARTBEAT) {
|
||||
if (log.isDebugEnabled()) {
|
||||
log.debug("[gb32960] HEARTBEAT peer={} vin={} platformAccount={}",
|
||||
@@ -304,8 +403,20 @@ public class Gb32960ChannelHandler extends SimpleChannelInboundHandler<byte[]> {
|
||||
}
|
||||
}
|
||||
|
||||
private void writeReportOrHeartbeatAck(ChannelHandlerContext ctx, Gb32960Message msg, byte[] rawVin) {
|
||||
CommandType cmd = msg.header().command();
|
||||
ackService.writeResponse(ctx,
|
||||
msg.header().protocolVersion(),
|
||||
cmd,
|
||||
ResponseFlag.SUCCESS,
|
||||
rawVin,
|
||||
eventOrNow(msg),
|
||||
null,
|
||||
"report-ack-0x" + Integer.toHexString(cmd.code()));
|
||||
}
|
||||
|
||||
/** 0x08 终端校时：平台应答 data 段为平台当前时间 6B（用 Instant.now()）。 */
|
||||
private void handleTimeCalibration(ChannelHandlerContext ctx, Gb32960Message msg, byte[] rawVin) {
|
||||
private void writeTimeCalibrationAck(ChannelHandlerContext ctx, Gb32960Message msg, byte[] rawVin) {
|
||||
ackService.writeResponse(ctx,
|
||||
msg.header().protocolVersion(),
|
||||
CommandType.TIME_CALIBRATION,
|
||||
|
||||
@@ -0,0 +1,37 @@
|
||||
package com.lingniu.ingest.protocol.gb32960.config;
|
||||
|
||||
import com.lingniu.ingest.core.dispatcher.Dispatcher;
|
||||
import com.lingniu.ingest.protocol.gb32960.codec.Gb32960MessageDecoder;
|
||||
import com.lingniu.ingest.protocol.gb32960.inbound.Gb32960NettyServer;
|
||||
import org.junit.jupiter.api.Test;
|
||||
import org.springframework.boot.autoconfigure.AutoConfigurations;
|
||||
import org.springframework.boot.test.context.runner.ApplicationContextRunner;
|
||||
|
||||
import static org.assertj.core.api.Assertions.assertThat;
|
||||
|
||||
class Gb32960AutoConfigurationTest {
|
||||
|
||||
private final ApplicationContextRunner runner = new ApplicationContextRunner()
|
||||
.withConfiguration(AutoConfigurations.of(Gb32960AutoConfiguration.class))
|
||||
.withBean(Dispatcher.class, () -> new Dispatcher(null, null, null, null, null))
|
||||
.withPropertyValues(
|
||||
"lingniu.ingest.gb32960.enabled=true",
|
||||
"lingniu.ingest.gb32960.port=0");
|
||||
|
||||
@Test
|
||||
void createsDecoderWithoutTcpServerWhenServerDisabled() {
|
||||
runner.withPropertyValues("lingniu.ingest.gb32960.server.enabled=false")
|
||||
.run(context -> {
|
||||
assertThat(context).hasSingleBean(Gb32960MessageDecoder.class);
|
||||
assertThat(context).doesNotHaveBean(Gb32960NettyServer.class);
|
||||
});
|
||||
}
|
||||
|
||||
@Test
|
||||
void createsTcpServerByDefaultWhenProtocolEnabled() {
|
||||
runner.run(context -> {
|
||||
assertThat(context).hasSingleBean(Gb32960MessageDecoder.class);
|
||||
assertThat(context).hasSingleBean(Gb32960NettyServer.class);
|
||||
});
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,212 @@
|
||||
package com.lingniu.ingest.protocol.gb32960.inbound;
|
||||
|
||||
import com.lingniu.ingest.api.event.VehicleEvent;
|
||||
import com.lingniu.ingest.api.sink.EventSink;
|
||||
import com.lingniu.ingest.codec.BccChecksum;
|
||||
import com.lingniu.ingest.core.concurrency.AsyncBatchExecutor;
|
||||
import com.lingniu.ingest.core.concurrency.DisruptorEventBus;
|
||||
import com.lingniu.ingest.core.dispatcher.Dispatcher;
|
||||
import com.lingniu.ingest.core.dispatcher.HandlerInvoker;
|
||||
import com.lingniu.ingest.core.dispatcher.HandlerRegistry;
|
||||
import com.lingniu.ingest.core.pipeline.InterceptorChain;
|
||||
import com.lingniu.ingest.protocol.gb32960.auth.Gb32960PlatformAuthorizer;
|
||||
import com.lingniu.ingest.protocol.gb32960.auth.Gb32960VinAuthorizer;
|
||||
import com.lingniu.ingest.protocol.gb32960.codec.Gb32960BodyParser;
|
||||
import com.lingniu.ingest.protocol.gb32960.codec.Gb32960DecoderTest;
|
||||
import com.lingniu.ingest.protocol.gb32960.codec.Gb32960MessageDecoder;
|
||||
import com.lingniu.ingest.protocol.gb32960.codec.InfoBlockParserRegistry;
|
||||
import com.lingniu.ingest.protocol.gb32960.codec.parser.v2016.PositionV2016BlockParser;
|
||||
import com.lingniu.ingest.protocol.gb32960.codec.parser.v2016.VehicleV2016BlockParser;
|
||||
import com.lingniu.ingest.protocol.gb32960.config.Gb32960Properties;
|
||||
import io.netty.buffer.ByteBuf;
|
||||
import io.netty.channel.embedded.EmbeddedChannel;
|
||||
import org.junit.jupiter.api.Test;
|
||||
|
||||
import java.io.ByteArrayOutputStream;
|
||||
import java.nio.charset.StandardCharsets;
|
||||
import java.util.ArrayList;
|
||||
import java.util.List;
|
||||
import java.util.concurrent.CompletableFuture;
|
||||
|
||||
import static org.assertj.core.api.Assertions.assertThat;
|
||||
|
||||
class Gb32960ChannelHandlerAckBoundaryTest {
|
||||
|
||||
@Test
|
||||
void realtimeReportAckWaitsForDispatchFutureCompletion() {
|
||||
try (DispatchHarness dispatch = new DispatchHarness()) {
|
||||
EmbeddedChannel channel = new EmbeddedChannel(handlerWith(dispatch.dispatcher()));
|
||||
|
||||
assertThat(channel.writeInbound(Gb32960DecoderTest.buildRealtimeFrame("LTEST000000000001"))).isFalse();
|
||||
assertThat((Object) channel.readOutbound()).isNull();
|
||||
|
||||
dispatch.awaitPublishCount(1);
|
||||
dispatch.completeDurability();
|
||||
channel.runPendingTasks();
|
||||
|
||||
ByteBuf ack = channel.readOutbound();
|
||||
assertThat(ack).isNotNull();
|
||||
assertThat(ack.getUnsignedByte(2)).isEqualTo((short) 0x02);
|
||||
assertThat(ack.getUnsignedByte(3)).isEqualTo((short) 0x01);
|
||||
ack.release();
|
||||
channel.finishAndReleaseAll();
|
||||
}
|
||||
}
|
||||
|
||||
@Test
|
||||
void realtimeReportAckIsNotWrittenWhenDispatchFutureFails() {
|
||||
try (DispatchHarness dispatch = new DispatchHarness()) {
|
||||
EmbeddedChannel channel = new EmbeddedChannel(handlerWith(dispatch.dispatcher()));
|
||||
|
||||
assertThat(channel.writeInbound(Gb32960DecoderTest.buildRealtimeFrame("LTEST000000000001"))).isFalse();
|
||||
|
||||
dispatch.awaitPublishCount(1);
|
||||
dispatch.failDurability();
|
||||
channel.runPendingTasks();
|
||||
|
||||
assertThat((Object) channel.readOutbound()).isNull();
|
||||
assertThat(channel.isActive()).isFalse();
|
||||
channel.finishAndReleaseAll();
|
||||
}
|
||||
}
|
||||
|
||||
@Test
|
||||
void durableAcksAreWrittenInInboundOrderWhenLaterDispatchCompletesFirst() {
|
||||
try (DispatchHarness dispatch = new DispatchHarness()) {
|
||||
EmbeddedChannel channel = new EmbeddedChannel(handlerWith(dispatch.dispatcher()));
|
||||
|
||||
assertThat(channel.writeInbound(Gb32960DecoderTest.buildRealtimeFrame("LTEST000000000001"))).isFalse();
|
||||
assertThat(channel.writeInbound(buildHeartbeatFrame("LTEST000000000001"))).isFalse();
|
||||
assertThat((Object) channel.readOutbound()).isNull();
|
||||
|
||||
dispatch.awaitPublishCount(2);
|
||||
dispatch.completeDurability(1);
|
||||
channel.runPendingTasks();
|
||||
assertThat((Object) channel.readOutbound()).isNull();
|
||||
|
||||
dispatch.completeDurability(0);
|
||||
channel.runPendingTasks();
|
||||
|
||||
ByteBuf firstAck = channel.readOutbound();
|
||||
ByteBuf secondAck = channel.readOutbound();
|
||||
assertThat(firstAck).isNotNull();
|
||||
assertThat(secondAck).isNotNull();
|
||||
assertThat(firstAck.getUnsignedByte(2)).isEqualTo((short) 0x02);
|
||||
assertThat(secondAck.getUnsignedByte(2)).isEqualTo((short) 0x07);
|
||||
firstAck.release();
|
||||
secondAck.release();
|
||||
channel.finishAndReleaseAll();
|
||||
}
|
||||
}
|
||||
|
||||
private static Gb32960ChannelHandler handlerWith(Dispatcher dispatcher) {
|
||||
return new Gb32960ChannelHandler(
|
||||
decoder(),
|
||||
dispatcher,
|
||||
accessService(),
|
||||
new Gb32960AckService(),
|
||||
new Gb32960FrameDiagnostics(16));
|
||||
}
|
||||
|
||||
private static Gb32960MessageDecoder decoder() {
|
||||
return new Gb32960MessageDecoder(new Gb32960BodyParser(new InfoBlockParserRegistry(List.of(
|
||||
new VehicleV2016BlockParser(),
|
||||
new PositionV2016BlockParser()))));
|
||||
}
|
||||
|
||||
private static Gb32960AccessService accessService() {
|
||||
Gb32960Properties.Auth auth = new Gb32960Properties.Auth();
|
||||
auth.setEnabled(false);
|
||||
return new Gb32960AccessService(
|
||||
new Gb32960VinAuthorizer(auth),
|
||||
new Gb32960PlatformAuthorizer(auth.getPlatforms()));
|
||||
}
|
||||
|
||||
private static byte[] buildHeartbeatFrame(String vin) {
|
||||
ByteArrayOutputStream out = new ByteArrayOutputStream();
|
||||
out.write(0x23);
|
||||
out.write(0x23);
|
||||
out.write(0x07);
|
||||
out.write(0xFE);
|
||||
byte[] vinBytes = vin.getBytes(StandardCharsets.US_ASCII);
|
||||
out.write(vinBytes, 0, 17);
|
||||
out.write(0x01);
|
||||
out.write(0x00);
|
||||
out.write(0x00);
|
||||
|
||||
byte[] almost = out.toByteArray();
|
||||
out.write(BccChecksum.compute(almost, 2, almost.length - 2) & 0xFF);
|
||||
return out.toByteArray();
|
||||
}
|
||||
|
||||
private static final class DispatchHarness implements AutoCloseable {
|
||||
private final ControlledSink sink = new ControlledSink();
|
||||
private final DisruptorEventBus eventBus = new DisruptorEventBus(1024, "blocking", List.of(sink));
|
||||
private final AsyncBatchExecutor batchExecutor = new AsyncBatchExecutor(eventBus::publish);
|
||||
private final Dispatcher dispatcher = new Dispatcher(
|
||||
new HandlerRegistry(),
|
||||
new InterceptorChain(List.of()),
|
||||
new HandlerInvoker(),
|
||||
eventBus,
|
||||
batchExecutor);
|
||||
|
||||
Dispatcher dispatcher() {
|
||||
return dispatcher;
|
||||
}
|
||||
|
||||
void completeDurability() {
|
||||
completeDurability(0);
|
||||
}
|
||||
|
||||
void completeDurability(int index) {
|
||||
sink.future(index).complete(null);
|
||||
}
|
||||
|
||||
void failDurability() {
|
||||
sink.future(0).completeExceptionally(new IllegalStateException("durability failed"));
|
||||
}
|
||||
|
||||
void awaitPublishCount(int count) {
|
||||
long deadline = System.currentTimeMillis() + 3000;
|
||||
while (System.currentTimeMillis() < deadline) {
|
||||
synchronized (sink.futures) {
|
||||
if (sink.futures.size() >= count) {
|
||||
return;
|
||||
}
|
||||
}
|
||||
Thread.onSpinWait();
|
||||
}
|
||||
throw new AssertionError("expected " + count + " sink publishes, actual=" + sink.futures.size());
|
||||
}
|
||||
|
||||
@Override
|
||||
public void close() {
|
||||
batchExecutor.close();
|
||||
eventBus.close();
|
||||
}
|
||||
}
|
||||
|
||||
private static final class ControlledSink implements EventSink {
|
||||
private final List<CompletableFuture<Void>> futures = new ArrayList<>();
|
||||
|
||||
@Override
|
||||
public String name() {
|
||||
return "kafka";
|
||||
}
|
||||
|
||||
@Override
|
||||
public CompletableFuture<Void> publish(VehicleEvent event) {
|
||||
CompletableFuture<Void> future = new CompletableFuture<>();
|
||||
synchronized (futures) {
|
||||
futures.add(future);
|
||||
}
|
||||
return future;
|
||||
}
|
||||
|
||||
private CompletableFuture<Void> future(int index) {
|
||||
synchronized (futures) {
|
||||
return futures.get(index);
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -1,7 +1,11 @@
|
||||
package com.lingniu.ingest.eventhistory;
|
||||
|
||||
import com.fasterxml.jackson.core.JsonProcessingException;
|
||||
import com.fasterxml.jackson.databind.ObjectMapper;
|
||||
import com.lingniu.ingest.api.ProtocolId;
|
||||
import com.lingniu.ingest.api.event.RawArchiveKeys;
|
||||
import com.lingniu.ingest.eventfilestore.EventFileRecord;
|
||||
import com.lingniu.ingest.sink.mq.proto.RawArchiveRef;
|
||||
import com.lingniu.ingest.sink.mq.proto.VehicleEnvelope;
|
||||
import com.google.protobuf.InvalidProtocolBufferException;
|
||||
import com.google.protobuf.util.JsonFormat;
|
||||
@@ -21,20 +25,24 @@ import java.util.Map;
|
||||
public final class TelemetryEnvelopeRecordMapper {
|
||||
|
||||
private static final JsonFormat.Printer JSON_PRINTER = JsonFormat.printer();
|
||||
private static final ObjectMapper OBJECT_MAPPER = new ObjectMapper();
|
||||
|
||||
public EventFileRecord toRecord(VehicleEnvelope envelope) {
|
||||
if (envelope == null) {
|
||||
throw new IllegalArgumentException("envelope must not be null");
|
||||
}
|
||||
if (!envelope.hasTelemetrySnapshot()) {
|
||||
throw new IllegalArgumentException("envelope telemetry_snapshot is required");
|
||||
if (envelope.hasTelemetrySnapshot()) {
|
||||
return telemetryRecord(envelope);
|
||||
}
|
||||
if (envelope.hasRawArchive()) {
|
||||
return rawArchiveRecord(envelope);
|
||||
}
|
||||
throw new IllegalArgumentException("envelope telemetry_snapshot or raw_archive is required");
|
||||
}
|
||||
|
||||
private static EventFileRecord telemetryRecord(VehicleEnvelope envelope) {
|
||||
ProtocolId protocol = protocol(envelope.getSource());
|
||||
Map<String, String> metadata = new LinkedHashMap<>(envelope.getMetadataMap());
|
||||
if (protocol == ProtocolId.UNKNOWN && !envelope.getSource().isBlank()) {
|
||||
// 保留未知 source 原文，避免历史库标准化成 UNKNOWN 后丢失排障线索。
|
||||
metadata.putIfAbsent("originalSource", envelope.getSource());
|
||||
}
|
||||
Map<String, String> metadata = metadata(envelope, protocol);
|
||||
return new EventFileRecord(
|
||||
envelope.getEventId(),
|
||||
protocol,
|
||||
@@ -48,6 +56,57 @@ public final class TelemetryEnvelopeRecordMapper {
|
||||
);
|
||||
}
|
||||
|
||||
private static EventFileRecord rawArchiveRecord(VehicleEnvelope envelope) {
|
||||
ProtocolId protocol = protocol(envelope.getSource());
|
||||
RawArchiveRef rawArchive = envelope.getRawArchive();
|
||||
Map<String, String> metadata = metadata(envelope, protocol);
|
||||
String rawArchiveKey = rawArchiveKey(rawArchive.getUri(), metadata);
|
||||
String rawArchiveUri = rawArchive.getUri().isBlank() && !rawArchiveKey.isBlank()
|
||||
? RawArchiveKeys.logicalUri(rawArchiveKey)
|
||||
: rawArchive.getUri();
|
||||
metadata.putIfAbsent(RawArchiveKeys.META_EVENT_ID, envelope.getEventId());
|
||||
if (!rawArchiveKey.isBlank()) {
|
||||
metadata.putIfAbsent(RawArchiveKeys.META_KEY, rawArchiveKey);
|
||||
}
|
||||
if (!rawArchiveUri.isBlank()) {
|
||||
metadata.putIfAbsent(RawArchiveKeys.META_URI, rawArchiveUri);
|
||||
}
|
||||
return new EventFileRecord(
|
||||
envelope.getEventId(),
|
||||
protocol,
|
||||
"RAW_ARCHIVE",
|
||||
envelope.getVin(),
|
||||
Instant.ofEpochMilli(envelope.getEventTimeMs()),
|
||||
Instant.ofEpochMilli(envelope.getIngestTimeMs()),
|
||||
rawArchiveUri,
|
||||
metadata,
|
||||
rawArchiveJson(envelope, metadata, rawArchiveKey, rawArchiveUri)
|
||||
);
|
||||
}
|
||||
|
||||
private static String rawArchiveKey(String rawArchiveUri, Map<String, String> metadata) {
|
||||
String key = metadata.getOrDefault(RawArchiveKeys.META_KEY, "");
|
||||
if (!key.isBlank()) {
|
||||
return key;
|
||||
}
|
||||
if (rawArchiveUri != null && rawArchiveUri.startsWith("archive://")) {
|
||||
return rawArchiveUri.substring("archive://".length());
|
||||
}
|
||||
return "";
|
||||
}
|
||||
|
||||
private static Map<String, String> metadata(VehicleEnvelope envelope, ProtocolId protocol) {
|
||||
Map<String, String> metadata = new LinkedHashMap<>(envelope.getMetadataMap());
|
||||
if (!envelope.getProtocolVersion().isBlank()) {
|
||||
metadata.putIfAbsent("protocolVersion", envelope.getProtocolVersion());
|
||||
}
|
||||
if (protocol == ProtocolId.UNKNOWN && !envelope.getSource().isBlank()) {
|
||||
// 保留未知 source 原文，避免历史库标准化成 UNKNOWN 后丢失排障线索。
|
||||
metadata.putIfAbsent("originalSource", envelope.getSource());
|
||||
}
|
||||
return metadata;
|
||||
}
|
||||
|
||||
private static ProtocolId protocol(String source) {
|
||||
if (source == null || source.isBlank()) {
|
||||
return ProtocolId.UNKNOWN;
|
||||
@@ -66,4 +125,36 @@ public final class TelemetryEnvelopeRecordMapper {
|
||||
throw new IllegalArgumentException("failed to serialize telemetry_snapshot", e);
|
||||
}
|
||||
}
|
||||
|
||||
private static String rawArchiveJson(VehicleEnvelope envelope,
|
||||
Map<String, String> metadata,
|
||||
String rawArchiveKey,
|
||||
String rawArchiveUri) {
|
||||
Map<String, Object> payload = new LinkedHashMap<>();
|
||||
RawArchiveRef rawArchive = envelope.getRawArchive();
|
||||
payload.put("eventId", envelope.getEventId());
|
||||
payload.put("vin", envelope.getVin());
|
||||
payload.put("protocol", protocol(envelope.getSource()).name());
|
||||
if (!envelope.getSource().isBlank()) {
|
||||
payload.put("source", envelope.getSource());
|
||||
}
|
||||
payload.put("eventType", "RAW_ARCHIVE");
|
||||
payload.put("eventTime", Instant.ofEpochMilli(envelope.getEventTimeMs()).toString());
|
||||
payload.put("ingestTime", Instant.ofEpochMilli(envelope.getIngestTimeMs()).toString());
|
||||
payload.put("rawArchiveUri", rawArchiveUri);
|
||||
payload.put("rawArchiveKey", rawArchiveKey);
|
||||
if (metadata.containsKey("command")) {
|
||||
payload.put("command", metadata.get("command"));
|
||||
}
|
||||
if (metadata.containsKey("infoType")) {
|
||||
payload.put("infoType", metadata.get("infoType"));
|
||||
}
|
||||
payload.put("rawSizeBytes", rawArchive.getSizeBytes());
|
||||
payload.put("metadata", metadata);
|
||||
try {
|
||||
return OBJECT_MAPPER.writeValueAsString(payload);
|
||||
} catch (JsonProcessingException e) {
|
||||
throw new IllegalArgumentException("failed to serialize raw_archive", e);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@@ -1,16 +1,22 @@
|
||||
package com.lingniu.ingest.eventhistory;
|
||||
|
||||
import com.fasterxml.jackson.databind.JsonNode;
|
||||
import com.fasterxml.jackson.databind.ObjectMapper;
|
||||
import com.lingniu.ingest.api.ProtocolId;
|
||||
import com.lingniu.ingest.api.consumer.EnvelopeIngestResult;
|
||||
import com.lingniu.ingest.api.consumer.EnvelopeIngestor;
|
||||
import com.lingniu.ingest.api.event.RawArchiveKeys;
|
||||
import com.lingniu.ingest.eventfilestore.EventFileQuery;
|
||||
import com.lingniu.ingest.eventfilestore.EventFileRecord;
|
||||
import com.lingniu.ingest.eventfilestore.EventFileStore;
|
||||
import com.lingniu.ingest.sink.mq.proto.RawArchiveRef;
|
||||
import com.lingniu.ingest.sink.mq.proto.TelemetryField;
|
||||
import com.lingniu.ingest.sink.mq.proto.TelemetrySnapshot;
|
||||
import com.lingniu.ingest.sink.mq.proto.VehicleEnvelope;
|
||||
import org.junit.jupiter.api.Test;
|
||||
|
||||
import java.io.IOException;
|
||||
import java.time.Instant;
|
||||
import java.util.ArrayList;
|
||||
import java.util.List;
|
||||
|
||||
@@ -19,6 +25,8 @@ import static org.assertj.core.api.Assertions.assertThatThrownBy;
|
||||
|
||||
class EventHistoryEnvelopeIngestorTest {
|
||||
|
||||
private static final ObjectMapper OBJECT_MAPPER = new ObjectMapper();
|
||||
|
||||
@Test
|
||||
void parsesKafkaEnvelopeBytesAndAppendsRecord() throws Exception {
|
||||
CapturingStore store = new CapturingStore();
|
||||
@@ -55,6 +63,61 @@ class EventHistoryEnvelopeIngestorTest {
|
||||
assertThat(store.records).isEmpty();
|
||||
}
|
||||
|
||||
@Test
|
||||
void tryIngestRawArchiveEnvelopeAppendsRawArchiveRecord() throws Exception {
|
||||
CapturingStore store = new CapturingStore();
|
||||
EventHistoryEnvelopeIngestor ingestor =
|
||||
new EventHistoryEnvelopeIngestor(store, new TelemetryEnvelopeRecordMapper());
|
||||
String rawArchiveKey = "2026/06/23/GB32960/VINRAW001/raw-event-1.bin";
|
||||
String rawArchiveUri = "archive://" + rawArchiveKey;
|
||||
|
||||
VehicleEnvelope envelope = VehicleEnvelope.newBuilder()
|
||||
.setSchemaVersion("1.0")
|
||||
.setEventId("raw-event-1")
|
||||
.setVin("VINRAW001")
|
||||
.setSource("GB32960")
|
||||
.setProtocolVersion("V2016")
|
||||
.setEventTimeMs(1_782_112_400_000L)
|
||||
.setIngestTimeMs(1_782_112_401_000L)
|
||||
.putMetadata(RawArchiveKeys.META_KEY, rawArchiveKey)
|
||||
.putMetadata(RawArchiveKeys.META_URI, rawArchiveUri)
|
||||
.putMetadata("command", "0x0002")
|
||||
.setRawArchive(RawArchiveRef.newBuilder()
|
||||
.setUri(rawArchiveUri)
|
||||
.setSizeBytes(128)
|
||||
.build())
|
||||
.build();
|
||||
|
||||
EnvelopeIngestResult result = ingestor.tryIngest(envelope.toByteArray());
|
||||
|
||||
assertThat(result.status()).isEqualTo(EnvelopeIngestResult.Status.STORED);
|
||||
assertThat(store.records).hasSize(1);
|
||||
EventFileRecord record = store.records.getFirst();
|
||||
assertThat(record.eventId()).isEqualTo("raw-event-1");
|
||||
assertThat(record.protocol()).isEqualTo(ProtocolId.GB32960);
|
||||
assertThat(record.eventType()).isEqualTo("RAW_ARCHIVE");
|
||||
assertThat(record.vin()).isEqualTo("VINRAW001");
|
||||
assertThat(record.eventTime()).isEqualTo(Instant.ofEpochMilli(1_782_112_400_000L));
|
||||
assertThat(record.rawArchiveUri()).isEqualTo(rawArchiveUri);
|
||||
assertThat(record.metadata())
|
||||
.containsEntry("protocolVersion", "V2016")
|
||||
.containsEntry(RawArchiveKeys.META_EVENT_ID, "raw-event-1")
|
||||
.containsEntry(RawArchiveKeys.META_KEY, rawArchiveKey)
|
||||
.containsEntry(RawArchiveKeys.META_URI, rawArchiveUri)
|
||||
.containsEntry("command", "0x0002");
|
||||
JsonNode payload = OBJECT_MAPPER.readTree(record.payloadJson());
|
||||
assertThat(payload.get("eventId").asText()).isEqualTo("raw-event-1");
|
||||
assertThat(payload.get("protocol").asText()).isEqualTo("GB32960");
|
||||
assertThat(payload.get("eventType").asText()).isEqualTo("RAW_ARCHIVE");
|
||||
assertThat(payload.get("vin").asText()).isEqualTo("VINRAW001");
|
||||
assertThat(payload.get("eventTime").asText()).isEqualTo("2026-06-22T07:13:20Z");
|
||||
assertThat(payload.get("rawArchiveUri").asText()).isEqualTo(rawArchiveUri);
|
||||
assertThat(payload.get("rawArchiveKey").asText()).isEqualTo(rawArchiveKey);
|
||||
assertThat(payload.get("rawSizeBytes").asLong()).isEqualTo(128);
|
||||
assertThat(payload.get("metadata").get("command").asText()).isEqualTo("0x0002");
|
||||
assertThat(payload.toString()).doesNotContain("rawBytes");
|
||||
}
|
||||
|
||||
private static VehicleEnvelope envelope(String eventId) {
|
||||
return VehicleEnvelope.newBuilder()
|
||||
.setEventId(eventId)
|
||||
|
||||
@@ -17,7 +17,11 @@ public final class VehicleStatEnvelopeIngestor implements EnvelopeIngestor {
|
||||
}
|
||||
|
||||
public void ingest(byte[] kafkaValue) {
|
||||
processor.process(parse(kafkaValue));
|
||||
VehicleEnvelope envelope = parse(kafkaValue);
|
||||
if (!envelope.hasTelemetrySnapshot()) {
|
||||
return;
|
||||
}
|
||||
processor.process(envelope);
|
||||
}
|
||||
|
||||
@Override
|
||||
|
||||
@@ -2,6 +2,7 @@ package com.lingniu.ingest.vehiclestat;
|
||||
|
||||
import com.lingniu.ingest.api.consumer.EnvelopeIngestResult;
|
||||
import com.lingniu.ingest.api.consumer.EnvelopeIngestor;
|
||||
import com.lingniu.ingest.sink.mq.proto.RawArchiveRef;
|
||||
import com.lingniu.ingest.sink.mq.proto.TelemetryField;
|
||||
import com.lingniu.ingest.sink.mq.proto.TelemetrySnapshot;
|
||||
import com.lingniu.ingest.sink.mq.proto.VehicleEnvelope;
|
||||
@@ -74,6 +75,23 @@ class VehicleStatEnvelopeIngestorTest {
|
||||
assertThat(repository.mileagePoints("VIN001", LocalDate.of(2026, 6, 22))).isEmpty();
|
||||
}
|
||||
|
||||
@Test
|
||||
void rawArchiveEnvelopeIsIgnoredWithoutAppendingMileagePoint() {
|
||||
InMemoryVehicleStatRepository repository = new InMemoryVehicleStatRepository();
|
||||
VehicleStatEnvelopeIngestor ingestor =
|
||||
new VehicleStatEnvelopeIngestor(new VehicleStatEventProcessor(repository));
|
||||
VehicleEnvelope envelope = rawArchiveEnvelope();
|
||||
|
||||
ingestor.ingest(envelope.toByteArray());
|
||||
var result = ingestor.tryIngest(envelope.toByteArray());
|
||||
|
||||
assertThat(result.status()).isEqualTo(EnvelopeIngestResult.Status.SKIPPED);
|
||||
assertThat(result.eventId()).isEqualTo("raw-event-1");
|
||||
assertThat(result.vin()).isEqualTo("VIN001");
|
||||
assertThat(result.message()).contains("telemetry_snapshot");
|
||||
assertThat(repository.mileagePoints("VIN001", LocalDate.of(2026, 6, 22))).isEmpty();
|
||||
}
|
||||
|
||||
private static VehicleEnvelope envelope() {
|
||||
return VehicleEnvelope.newBuilder()
|
||||
.setEventId("event-1")
|
||||
@@ -91,6 +109,20 @@ class VehicleStatEnvelopeIngestorTest {
|
||||
.build();
|
||||
}
|
||||
|
||||
private static VehicleEnvelope rawArchiveEnvelope() {
|
||||
return VehicleEnvelope.newBuilder()
|
||||
.setEventId("raw-event-1")
|
||||
.setVin("VIN001")
|
||||
.setSource("GB32960")
|
||||
.setEventTimeMs(1_782_112_400_000L)
|
||||
.setIngestTimeMs(1_782_112_401_000L)
|
||||
.setRawArchive(RawArchiveRef.newBuilder()
|
||||
.setUri("archive://2026/06/23/GB32960/VIN001/raw-event-1.bin")
|
||||
.setSizeBytes(128)
|
||||
.build())
|
||||
.build();
|
||||
}
|
||||
|
||||
private static final class InMemoryVehicleStatRepository implements VehicleStatRepository {
|
||||
private final java.util.Map<String, java.util.List<MileagePoint>> points = new java.util.HashMap<>();
|
||||
|
||||
|
||||
@@ -17,7 +17,11 @@ public final class VehicleStateEnvelopeIngestor implements EnvelopeIngestor {
|
||||
}
|
||||
|
||||
public void ingest(byte[] kafkaValue) {
|
||||
updater.update(parse(kafkaValue));
|
||||
VehicleEnvelope envelope = parse(kafkaValue);
|
||||
if (!envelope.hasTelemetrySnapshot()) {
|
||||
return;
|
||||
}
|
||||
updater.update(envelope);
|
||||
}
|
||||
|
||||
@Override
|
||||
@@ -25,6 +29,10 @@ public final class VehicleStateEnvelopeIngestor implements EnvelopeIngestor {
|
||||
VehicleEnvelope envelope = null;
|
||||
try {
|
||||
envelope = parse(kafkaValue);
|
||||
if (!envelope.hasTelemetrySnapshot()) {
|
||||
return EnvelopeIngestResult.skipped(
|
||||
envelope.getEventId(), envelope.getVin(), "envelope telemetry_snapshot is required");
|
||||
}
|
||||
// Kafka 消费路径只接受标准 VehicleEnvelope；坏消息返回明确结果给处理器写 DLQ。
|
||||
updater.update(envelope);
|
||||
return EnvelopeIngestResult.processed(envelope.getEventId(), envelope.getVin());
|
||||
|
||||
@@ -2,6 +2,7 @@ package com.lingniu.ingest.vehiclestate;
|
||||
|
||||
import com.lingniu.ingest.api.consumer.EnvelopeIngestResult;
|
||||
import com.lingniu.ingest.api.consumer.EnvelopeIngestor;
|
||||
import com.lingniu.ingest.sink.mq.proto.RawArchiveRef;
|
||||
import com.lingniu.ingest.sink.mq.proto.TelemetryField;
|
||||
import com.lingniu.ingest.sink.mq.proto.TelemetrySnapshot;
|
||||
import com.lingniu.ingest.sink.mq.proto.VehicleEnvelope;
|
||||
@@ -47,6 +48,25 @@ class VehicleStateEnvelopeIngestorTest {
|
||||
assertThat(repository.getState("VIN001")).isEmpty();
|
||||
}
|
||||
|
||||
@Test
|
||||
void rawArchiveEnvelopeIsIgnoredWithoutUpdatingRepository() {
|
||||
InMemoryVehicleStateRepository repository = new InMemoryVehicleStateRepository();
|
||||
VehicleStateEnvelopeIngestor ingestor = new VehicleStateEnvelopeIngestor(new VehicleStateUpdater(repository));
|
||||
VehicleEnvelope envelope = rawArchiveEnvelope();
|
||||
|
||||
ingestor.ingest(envelope.toByteArray());
|
||||
var result = ingestor.tryIngest(envelope.toByteArray());
|
||||
|
||||
assertThat(result.status()).isEqualTo(EnvelopeIngestResult.Status.SKIPPED);
|
||||
assertThat(result.eventId()).isEqualTo("raw-event-1");
|
||||
assertThat(result.vin()).isEqualTo("VIN001");
|
||||
assertThat(result.message()).contains("telemetry_snapshot");
|
||||
assertThat(repository.getState("VIN001")).isEmpty();
|
||||
assertThat(repository.getLocation("VIN001")).isEmpty();
|
||||
assertThat(repository.getSafety("VIN001")).isEmpty();
|
||||
assertThat(repository.getLastEvent("VIN001")).isEmpty();
|
||||
}
|
||||
|
||||
private static VehicleEnvelope envelope() {
|
||||
return VehicleEnvelope.newBuilder()
|
||||
.setEventId("event-1")
|
||||
@@ -64,6 +84,20 @@ class VehicleStateEnvelopeIngestorTest {
|
||||
.build();
|
||||
}
|
||||
|
||||
private static VehicleEnvelope rawArchiveEnvelope() {
|
||||
return VehicleEnvelope.newBuilder()
|
||||
.setEventId("raw-event-1")
|
||||
.setVin("VIN001")
|
||||
.setSource("GB32960")
|
||||
.setEventTimeMs(1_782_112_400_000L)
|
||||
.setIngestTimeMs(1_782_112_401_000L)
|
||||
.setRawArchive(RawArchiveRef.newBuilder()
|
||||
.setUri("archive://2026/06/23/GB32960/VIN001/raw-event-1.bin")
|
||||
.setSizeBytes(128)
|
||||
.build())
|
||||
.build();
|
||||
}
|
||||
|
||||
private static final class EmptyRepository implements VehicleStateRepository {
|
||||
@Override public void putState(String vin, String json) {}
|
||||
@Override public void putLocation(String vin, String json) {}
|
||||
|
||||
@@ -0,0 +1,17 @@
|
||||
package com.lingniu.ingest.sink.archive;
|
||||
|
||||
import java.io.IOException;
|
||||
import java.io.InputStream;
|
||||
|
||||
public interface ArchiveStore {
|
||||
|
||||
String put(String key, InputStream data, long length) throws IOException;
|
||||
|
||||
String append(String key, byte[] chunk) throws IOException;
|
||||
|
||||
InputStream get(String key) throws IOException;
|
||||
|
||||
boolean exists(String key);
|
||||
|
||||
long size(String key) throws IOException;
|
||||
}
|
||||
@@ -0,0 +1,150 @@
|
||||
package com.lingniu.ingest.sink.archive;
|
||||
|
||||
import java.io.IOException;
|
||||
import java.io.InputStream;
|
||||
import java.io.OutputStream;
|
||||
import java.net.URI;
|
||||
import java.nio.file.Files;
|
||||
import java.nio.file.LinkOption;
|
||||
import java.nio.file.Path;
|
||||
import java.nio.file.StandardOpenOption;
|
||||
import java.util.concurrent.ConcurrentHashMap;
|
||||
import java.util.concurrent.ConcurrentMap;
|
||||
|
||||
public final class LocalArchiveStore implements ArchiveStore {
|
||||
|
||||
private final Path root;
|
||||
private final ConcurrentMap<Path, Object> appendLocks = new ConcurrentHashMap<>();
|
||||
|
||||
public LocalArchiveStore(String root) {
|
||||
this(rootPath(root));
|
||||
}
|
||||
|
||||
public LocalArchiveStore(Path root) {
|
||||
this.root = root.toAbsolutePath().normalize();
|
||||
}
|
||||
|
||||
@Override
|
||||
public String put(String key, InputStream data, long length) throws IOException {
|
||||
if (data == null) {
|
||||
throw new IllegalArgumentException("data must not be null");
|
||||
}
|
||||
Path target = resolve(key);
|
||||
createDirectoriesInsideRoot(target.getParent());
|
||||
rejectSymlinkPath(target);
|
||||
try (OutputStream out = Files.newOutputStream(
|
||||
target,
|
||||
StandardOpenOption.CREATE,
|
||||
StandardOpenOption.TRUNCATE_EXISTING,
|
||||
StandardOpenOption.WRITE,
|
||||
LinkOption.NOFOLLOW_LINKS)) {
|
||||
data.transferTo(out);
|
||||
}
|
||||
return target.toUri().toString();
|
||||
}
|
||||
|
||||
@Override
|
||||
public String append(String key, byte[] chunk) throws IOException {
|
||||
Path target = resolve(key);
|
||||
synchronized (appendLocks.computeIfAbsent(target, ignored -> new Object())) {
|
||||
createDirectoriesInsideRoot(target.getParent());
|
||||
rejectSymlinkPath(target);
|
||||
Files.write(
|
||||
target,
|
||||
chunk == null ? new byte[0] : chunk,
|
||||
StandardOpenOption.CREATE,
|
||||
StandardOpenOption.APPEND,
|
||||
StandardOpenOption.WRITE,
|
||||
LinkOption.NOFOLLOW_LINKS);
|
||||
}
|
||||
return target.toUri().toString();
|
||||
}
|
||||
|
||||
@Override
|
||||
public InputStream get(String key) throws IOException {
|
||||
Path target = resolveExisting(key);
|
||||
return Files.newInputStream(target, StandardOpenOption.READ, LinkOption.NOFOLLOW_LINKS);
|
||||
}
|
||||
|
||||
@Override
|
||||
public boolean exists(String key) {
|
||||
try {
|
||||
return Files.exists(resolveExisting(key), LinkOption.NOFOLLOW_LINKS);
|
||||
} catch (IOException | IllegalArgumentException e) {
|
||||
return false;
|
||||
}
|
||||
}
|
||||
|
||||
@Override
|
||||
public long size(String key) throws IOException {
|
||||
return Files.size(resolveExisting(key));
|
||||
}
|
||||
|
||||
private Path resolve(String key) {
|
||||
if (key == null || key.isBlank()) {
|
||||
throw new IllegalArgumentException("archive key must not be blank");
|
||||
}
|
||||
Path target = root.resolve(key).normalize();
|
||||
if (!target.startsWith(root)) {
|
||||
throw new IllegalArgumentException("archive key escapes root: " + key);
|
||||
}
|
||||
return target;
|
||||
}
|
||||
|
||||
private Path resolveExisting(String key) throws IOException {
|
||||
Path target = resolve(key);
|
||||
rejectSymlinkPath(target);
|
||||
return target;
|
||||
}
|
||||
|
||||
private void createDirectoriesInsideRoot(Path directory) throws IOException {
|
||||
rejectEscapedPath(directory);
|
||||
if (Files.notExists(root, LinkOption.NOFOLLOW_LINKS)) {
|
||||
Files.createDirectories(root);
|
||||
}
|
||||
Path current = root;
|
||||
Path relative = root.relativize(directory);
|
||||
for (Path segment : relative) {
|
||||
current = current.resolve(segment);
|
||||
if (Files.isSymbolicLink(current)) {
|
||||
throw new IOException("archive path traverses symlink: " + current);
|
||||
}
|
||||
if (Files.notExists(current, LinkOption.NOFOLLOW_LINKS)) {
|
||||
Files.createDirectory(current);
|
||||
} else if (!Files.isDirectory(current, LinkOption.NOFOLLOW_LINKS)) {
|
||||
throw new IOException("archive path segment is not a directory: " + current);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
private void rejectSymlinkPath(Path target) throws IOException {
|
||||
rejectEscapedPath(target);
|
||||
Path current = root;
|
||||
Path relative = root.relativize(target);
|
||||
for (Path segment : relative) {
|
||||
current = current.resolve(segment);
|
||||
if (Files.isSymbolicLink(current)) {
|
||||
throw new IOException("archive path traverses symlink: " + current);
|
||||
}
|
||||
if (Files.notExists(current, LinkOption.NOFOLLOW_LINKS)) {
|
||||
return;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
private void rejectEscapedPath(Path path) {
|
||||
if (!path.normalize().startsWith(root)) {
|
||||
throw new IllegalArgumentException("archive path escapes root: " + path);
|
||||
}
|
||||
}
|
||||
|
||||
private static Path rootPath(String value) {
|
||||
if (value == null || value.isBlank()) {
|
||||
return Path.of(System.getProperty("java.io.tmpdir"), "lingniu-archive");
|
||||
}
|
||||
if (value.startsWith("file://")) {
|
||||
return Path.of(URI.create(value));
|
||||
}
|
||||
return Path.of(value);
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,44 @@
|
||||
package com.lingniu.ingest.sink.archive;
|
||||
|
||||
import com.lingniu.ingest.api.event.RawArchiveKeys;
|
||||
import com.lingniu.ingest.api.event.VehicleEvent;
|
||||
import com.lingniu.ingest.api.sink.EventSink;
|
||||
|
||||
import java.io.ByteArrayInputStream;
|
||||
import java.util.concurrent.CompletableFuture;
|
||||
|
||||
public final class RawArchiveEventSink implements EventSink {
|
||||
|
||||
private final ArchiveStore store;
|
||||
|
||||
public RawArchiveEventSink(ArchiveStore store) {
|
||||
this.store = store;
|
||||
}
|
||||
|
||||
@Override
|
||||
public String name() {
|
||||
return "raw-archive";
|
||||
}
|
||||
|
||||
@Override
|
||||
public CompletableFuture<Void> publish(VehicleEvent event) {
|
||||
if (!(event instanceof VehicleEvent.RawArchive raw)) {
|
||||
return CompletableFuture.completedFuture(null);
|
||||
}
|
||||
try {
|
||||
byte[] bytes = raw.rawBytes();
|
||||
if (bytes == null) {
|
||||
throw new IllegalArgumentException("raw archive bytes must not be null");
|
||||
}
|
||||
store.put(RawArchiveKeys.key(raw), new ByteArrayInputStream(bytes), bytes.length);
|
||||
return CompletableFuture.completedFuture(null);
|
||||
} catch (Exception e) {
|
||||
return CompletableFuture.failedFuture(e);
|
||||
}
|
||||
}
|
||||
|
||||
@Override
|
||||
public boolean accepts(VehicleEvent event) {
|
||||
return event instanceof VehicleEvent.RawArchive;
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,35 @@
|
||||
package com.lingniu.ingest.sink.archive.config;
|
||||
|
||||
import com.lingniu.ingest.sink.archive.ArchiveStore;
|
||||
import com.lingniu.ingest.sink.archive.LocalArchiveStore;
|
||||
import com.lingniu.ingest.sink.archive.RawArchiveEventSink;
|
||||
import org.springframework.boot.autoconfigure.AutoConfiguration;
|
||||
import org.springframework.boot.autoconfigure.condition.ConditionalOnMissingBean;
|
||||
import org.springframework.boot.autoconfigure.condition.ConditionalOnBean;
|
||||
import org.springframework.boot.autoconfigure.condition.ConditionalOnProperty;
|
||||
import org.springframework.boot.context.properties.EnableConfigurationProperties;
|
||||
import org.springframework.context.annotation.Bean;
|
||||
|
||||
@AutoConfiguration
|
||||
@EnableConfigurationProperties(SinkArchiveProperties.class)
|
||||
@ConditionalOnProperty(
|
||||
prefix = "lingniu.ingest.sink.archive",
|
||||
name = "enabled",
|
||||
havingValue = "true",
|
||||
matchIfMissing = true)
|
||||
public class SinkArchiveAutoConfiguration {
|
||||
|
||||
@Bean
|
||||
@ConditionalOnMissingBean
|
||||
@ConditionalOnProperty(prefix = "lingniu.ingest.sink.archive", name = "type", havingValue = "local", matchIfMissing = true)
|
||||
public ArchiveStore archiveStore(SinkArchiveProperties properties) {
|
||||
return new LocalArchiveStore(properties.getPath());
|
||||
}
|
||||
|
||||
@Bean
|
||||
@ConditionalOnMissingBean
|
||||
@ConditionalOnBean(ArchiveStore.class)
|
||||
public RawArchiveEventSink rawArchiveEventSink(ArchiveStore store) {
|
||||
return new RawArchiveEventSink(store);
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,35 @@
|
||||
package com.lingniu.ingest.sink.archive.config;
|
||||
|
||||
import org.springframework.boot.context.properties.ConfigurationProperties;
|
||||
|
||||
@ConfigurationProperties(prefix = "lingniu.ingest.sink.archive")
|
||||
public class SinkArchiveProperties {
|
||||
|
||||
private boolean enabled = true;
|
||||
private String type = "local";
|
||||
private String path = System.getProperty("java.io.tmpdir") + "/lingniu-archive";
|
||||
|
||||
public boolean isEnabled() {
|
||||
return enabled;
|
||||
}
|
||||
|
||||
public void setEnabled(boolean enabled) {
|
||||
this.enabled = enabled;
|
||||
}
|
||||
|
||||
public String getType() {
|
||||
return type;
|
||||
}
|
||||
|
||||
public void setType(String type) {
|
||||
this.type = type;
|
||||
}
|
||||
|
||||
public String getPath() {
|
||||
return path;
|
||||
}
|
||||
|
||||
public void setPath(String path) {
|
||||
this.path = path;
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,74 @@
|
||||
package com.lingniu.ingest.sink.archive;
|
||||
|
||||
import org.junit.jupiter.api.Test;
|
||||
import org.junit.jupiter.api.io.TempDir;
|
||||
|
||||
import java.io.ByteArrayInputStream;
|
||||
import java.nio.charset.StandardCharsets;
|
||||
import java.nio.file.Files;
|
||||
import java.nio.file.Path;
|
||||
import java.util.ArrayList;
|
||||
import java.util.List;
|
||||
import java.util.concurrent.CompletableFuture;
|
||||
import java.util.concurrent.Executors;
|
||||
|
||||
import static org.assertj.core.api.Assertions.assertThat;
|
||||
import static org.assertj.core.api.Assertions.assertThatThrownBy;
|
||||
|
||||
class LocalArchiveStoreTest {
|
||||
|
||||
@TempDir
|
||||
Path tempDir;
|
||||
|
||||
@Test
|
||||
void rejectsSymlinkTraversalOutsideRootForWriteReadAndSize() throws Exception {
|
||||
Path root = tempDir.resolve("archive");
|
||||
Path outside = tempDir.resolve("outside");
|
||||
Files.createDirectories(root);
|
||||
Files.createDirectories(outside);
|
||||
Files.writeString(outside.resolve("existing.bin"), "secret", StandardCharsets.UTF_8);
|
||||
Files.createSymbolicLink(root.resolve("link"), outside);
|
||||
|
||||
LocalArchiveStore store = new LocalArchiveStore(root);
|
||||
|
||||
assertThatThrownBy(() -> store.put(
|
||||
"link/new.bin",
|
||||
new ByteArrayInputStream("payload".getBytes(StandardCharsets.UTF_8)),
|
||||
7))
|
||||
.isInstanceOfAny(IllegalArgumentException.class, java.io.IOException.class);
|
||||
assertThat(outside.resolve("new.bin")).doesNotExist();
|
||||
|
||||
assertThatThrownBy(() -> store.get("link/existing.bin"))
|
||||
.isInstanceOfAny(IllegalArgumentException.class, java.io.IOException.class);
|
||||
assertThatThrownBy(() -> store.size("link/existing.bin"))
|
||||
.isInstanceOfAny(IllegalArgumentException.class, java.io.IOException.class);
|
||||
}
|
||||
|
||||
@Test
|
||||
void appendsConcurrentChunksWithoutLosingData() throws Exception {
|
||||
LocalArchiveStore store = new LocalArchiveStore(tempDir.resolve("archive"));
|
||||
int chunks = 96;
|
||||
|
||||
try (var executor = Executors.newFixedThreadPool(12)) {
|
||||
List<CompletableFuture<Void>> futures = new ArrayList<>();
|
||||
for (int i = 0; i < chunks; i++) {
|
||||
String chunk = "%03d\n".formatted(i);
|
||||
futures.add(CompletableFuture.runAsync(() -> {
|
||||
try {
|
||||
store.append("same/key.bin", chunk.getBytes(StandardCharsets.UTF_8));
|
||||
} catch (Exception e) {
|
||||
throw new RuntimeException(e);
|
||||
}
|
||||
}, executor));
|
||||
}
|
||||
CompletableFuture.allOf(futures.toArray(CompletableFuture[]::new)).join();
|
||||
}
|
||||
|
||||
String content = Files.readString(tempDir.resolve("archive/same/key.bin"), StandardCharsets.UTF_8);
|
||||
assertThat(content).hasSize(chunks * 4);
|
||||
assertThat(content.lines()).containsExactlyInAnyOrderElementsOf(
|
||||
java.util.stream.IntStream.range(0, chunks)
|
||||
.mapToObj("%03d"::formatted)
|
||||
.toList());
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,63 @@
|
||||
package com.lingniu.ingest.sink.archive;
|
||||
|
||||
import com.lingniu.ingest.api.ProtocolId;
|
||||
import com.lingniu.ingest.api.event.VehicleEvent;
|
||||
import org.junit.jupiter.api.Test;
|
||||
|
||||
import java.io.InputStream;
|
||||
import java.time.Instant;
|
||||
import java.util.Map;
|
||||
import java.util.concurrent.CompletionException;
|
||||
|
||||
import static org.assertj.core.api.Assertions.assertThatThrownBy;
|
||||
|
||||
class RawArchiveEventSinkTest {
|
||||
|
||||
@Test
|
||||
void failsFastWhenRawBytesAreNull() {
|
||||
RawArchiveEventSink sink = new RawArchiveEventSink(new CapturingArchiveStore());
|
||||
VehicleEvent.RawArchive raw = new VehicleEvent.RawArchive(
|
||||
"event-1",
|
||||
"VIN123",
|
||||
ProtocolId.GB32960,
|
||||
Instant.parse("2026-06-23T00:00:00Z"),
|
||||
Instant.parse("2026-06-23T00:00:01Z"),
|
||||
"trace-1",
|
||||
Map.of(),
|
||||
2,
|
||||
1,
|
||||
null);
|
||||
|
||||
assertThatThrownBy(() -> sink.publish(raw).join())
|
||||
.isInstanceOf(CompletionException.class)
|
||||
.hasCauseInstanceOf(IllegalArgumentException.class);
|
||||
}
|
||||
|
||||
private static final class CapturingArchiveStore implements ArchiveStore {
|
||||
|
||||
@Override
|
||||
public String put(String key, InputStream data, long length) {
|
||||
return "archive://" + key;
|
||||
}
|
||||
|
||||
@Override
|
||||
public String append(String key, byte[] chunk) {
|
||||
return "archive://" + key;
|
||||
}
|
||||
|
||||
@Override
|
||||
public InputStream get(String key) {
|
||||
throw new UnsupportedOperationException();
|
||||
}
|
||||
|
||||
@Override
|
||||
public boolean exists(String key) {
|
||||
return false;
|
||||
}
|
||||
|
||||
@Override
|
||||
public long size(String key) {
|
||||
return 0;
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,40 @@
|
||||
package com.lingniu.ingest.sink.archive.config;
|
||||
|
||||
import com.lingniu.ingest.sink.archive.ArchiveStore;
|
||||
import com.lingniu.ingest.sink.archive.RawArchiveEventSink;
|
||||
import org.junit.jupiter.api.Test;
|
||||
import org.springframework.context.annotation.AnnotationConfigApplicationContext;
|
||||
import org.springframework.core.env.MapPropertySource;
|
||||
|
||||
import static org.assertj.core.api.Assertions.assertThat;
|
||||
|
||||
class SinkArchiveAutoConfigurationTest {
|
||||
|
||||
@Test
|
||||
void localArchiveTypeCreatesStoreAndSink() {
|
||||
try (AnnotationConfigApplicationContext context = new AnnotationConfigApplicationContext()) {
|
||||
context.getEnvironment().getPropertySources().addFirst(new MapPropertySource(
|
||||
"test",
|
||||
java.util.Map.of("lingniu.ingest.sink.archive.path", "target/test-archive")));
|
||||
context.register(SinkArchiveAutoConfiguration.class);
|
||||
context.refresh();
|
||||
|
||||
assertThat(context.getBeansOfType(ArchiveStore.class)).hasSize(1);
|
||||
assertThat(context.getBeansOfType(RawArchiveEventSink.class)).hasSize(1);
|
||||
}
|
||||
}
|
||||
|
||||
@Test
|
||||
void nonLocalArchiveTypeDoesNotCreateSinkWithoutStore() {
|
||||
try (AnnotationConfigApplicationContext context = new AnnotationConfigApplicationContext()) {
|
||||
context.getEnvironment().getPropertySources().addFirst(new MapPropertySource(
|
||||
"test",
|
||||
java.util.Map.of("lingniu.ingest.sink.archive.type", "oss")));
|
||||
context.register(SinkArchiveAutoConfiguration.class);
|
||||
context.refresh();
|
||||
|
||||
assertThat(context.getBeansOfType(ArchiveStore.class)).isEmpty();
|
||||
assertThat(context.getBeansOfType(RawArchiveEventSink.class)).isEmpty();
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -71,8 +71,7 @@ public final class EnvelopeMapper {
|
||||
int size = ra.rawBytes() == null ? 0 : ra.rawBytes().length;
|
||||
String key = rawArchiveKey(ra);
|
||||
String uri = rawArchiveUri(ra, key);
|
||||
// RAW envelope 只携带引用信息，不把完整原始字节塞进 Kafka，避免 topic 膨胀。
|
||||
// 注意 KafkaEventSink 当前仍过滤 RawArchive；这段用于未来放开 raw 引用转发或消费者测试。
|
||||
// RAW envelope 只携带 URI/size 引用信息，不把完整原始字节塞进 Kafka，避免 topic 膨胀。
|
||||
b.putMetadata(RawArchiveKeys.META_KEY, key);
|
||||
b.putMetadata(RawArchiveKeys.META_URI, uri);
|
||||
b.putMetadata(RawArchiveKeys.META_EVENT_ID, ra.eventId());
|
||||
|
||||
@@ -43,13 +43,11 @@ public final class KafkaEventSink implements EventSink, AutoCloseable {
|
||||
}
|
||||
|
||||
/**
|
||||
* 拒绝 {@link VehicleEvent.RawArchive} —— 当前生产 Kafka sink 只投递实时、会话、告警等轻量事件。
|
||||
* {@link EnvelopeMapper} 和 {@link TopicRouter} 已具备 RAW envelope/topic 的映射能力，但这里仍
|
||||
* 显式过滤；如需把 32960 raw 引用转发到 {@code vehicle.raw.archive}，应先移除此过滤并补充测试。
|
||||
* Split-service mode uses Kafka for both normalized events and raw archive records.
|
||||
*/
|
||||
@Override
|
||||
public boolean accepts(VehicleEvent event) {
|
||||
return !(event instanceof VehicleEvent.RawArchive);
|
||||
return true;
|
||||
}
|
||||
|
||||
@Override
|
||||
|
||||
@@ -57,16 +57,16 @@ public class SinkMqProperties {
|
||||
|
||||
public static class Topics {
|
||||
/** 实时遥测事件 topic；GB32960 RAW-only 架构下可逐步弱化该 topic。 */
|
||||
private String realtime = "vehicle.realtime";
|
||||
private String realtime = "vehicle.event.gb32960.v1";
|
||||
/** 位置事件 topic；通常可由 realtime/RAW 派生。 */
|
||||
private String location = "vehicle.location";
|
||||
private String alarm = "vehicle.alarm";
|
||||
private String session = "vehicle.session";
|
||||
private String mediaMeta = "vehicle.media.meta";
|
||||
private String location = "vehicle.event.gb32960.v1";
|
||||
private String alarm = "vehicle.event.gb32960.v1";
|
||||
private String session = "vehicle.event.gb32960.v1";
|
||||
private String mediaMeta = "vehicle.media.meta.v1";
|
||||
/** RAW 归档引用 topic，payload 应携带 archive URI/size，不建议携带完整 raw bytes。 */
|
||||
private String rawArchive = "vehicle.raw.archive";
|
||||
private String rawArchive = "vehicle.raw.gb32960.v1";
|
||||
/** producer 熔断或 consumer 处理失败时的死信 topic。 */
|
||||
private String dlq = "vehicle.dlq";
|
||||
private String dlq = "vehicle.dlq.gb32960.v1";
|
||||
|
||||
public String getRealtime() { return realtime; }
|
||||
public void setRealtime(String realtime) { this.realtime = realtime; }
|
||||
|
||||
@@ -0,0 +1,40 @@
|
||||
package com.lingniu.ingest.sink.mq;
|
||||
|
||||
import com.lingniu.ingest.api.ProtocolId;
|
||||
import com.lingniu.ingest.api.event.VehicleEvent;
|
||||
import io.github.resilience4j.circuitbreaker.CircuitBreaker;
|
||||
import org.junit.jupiter.api.Test;
|
||||
|
||||
import java.time.Instant;
|
||||
import java.util.Map;
|
||||
|
||||
import static org.assertj.core.api.Assertions.assertThat;
|
||||
|
||||
class KafkaEventSinkTest {
|
||||
|
||||
@Test
|
||||
void acceptsRawArchiveRecordsForSplitServiceKafkaBoundary() {
|
||||
KafkaEventSink sink = new KafkaEventSink(
|
||||
null,
|
||||
new EnvelopeMapper("node-1"),
|
||||
new TopicRouter(new SinkMqProperties.Topics()),
|
||||
"vehicle.dlq.gb32960.v1",
|
||||
CircuitBreaker.ofDefaults("kafka-test"));
|
||||
|
||||
assertThat(sink.accepts(rawArchive())).isTrue();
|
||||
}
|
||||
|
||||
private static VehicleEvent.RawArchive rawArchive() {
|
||||
return new VehicleEvent.RawArchive(
|
||||
"raw-1",
|
||||
"VIN001",
|
||||
ProtocolId.GB32960,
|
||||
Instant.parse("2026-06-23T07:00:00Z"),
|
||||
Instant.parse("2026-06-23T07:00:01Z"),
|
||||
"trace-raw-1",
|
||||
Map.of("platformAccount", "Hyundai"),
|
||||
0x02,
|
||||
0,
|
||||
new byte[] {0x23, 0x23});
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,147 @@
|
||||
package com.lingniu.ingest.sink.mq;
|
||||
|
||||
import com.lingniu.ingest.api.ProtocolId;
|
||||
import com.lingniu.ingest.api.event.AlarmPayload;
|
||||
import com.lingniu.ingest.api.event.LocationPayload;
|
||||
import com.lingniu.ingest.api.event.RealtimePayload;
|
||||
import com.lingniu.ingest.api.event.VehicleEvent;
|
||||
import org.junit.jupiter.api.Test;
|
||||
|
||||
import java.time.Instant;
|
||||
import java.util.List;
|
||||
import java.util.Map;
|
||||
import java.util.Set;
|
||||
|
||||
import static org.assertj.core.api.Assertions.assertThat;
|
||||
|
||||
class TopicRouterTest {
|
||||
|
||||
@Test
|
||||
void rawArchiveRoutesToVersionedGb32960RawTopicByDefault() {
|
||||
TopicRouter router = new TopicRouter(new SinkMqProperties.Topics());
|
||||
|
||||
assertThat(router.route(rawArchive())).isEqualTo("vehicle.raw.gb32960.v1");
|
||||
}
|
||||
|
||||
@Test
|
||||
void normalizedGb32960EventsRouteToVersionedEventTopicByDefault() {
|
||||
TopicRouter router = new TopicRouter(new SinkMqProperties.Topics());
|
||||
|
||||
assertThat(router.route(realtime())).isEqualTo("vehicle.event.gb32960.v1");
|
||||
assertThat(router.route(location())).isEqualTo("vehicle.event.gb32960.v1");
|
||||
assertThat(router.route(alarm())).isEqualTo("vehicle.event.gb32960.v1");
|
||||
assertThat(router.route(login())).isEqualTo("vehicle.event.gb32960.v1");
|
||||
assertThat(router.route(logout())).isEqualTo("vehicle.event.gb32960.v1");
|
||||
assertThat(router.route(heartbeat())).isEqualTo("vehicle.event.gb32960.v1");
|
||||
}
|
||||
|
||||
@Test
|
||||
void dlqDefaultsToVersionedGb32960DlqTopic() {
|
||||
SinkMqProperties.Topics topics = new SinkMqProperties.Topics();
|
||||
|
||||
assertThat(topics.getDlq()).isEqualTo("vehicle.dlq.gb32960.v1");
|
||||
}
|
||||
|
||||
private static VehicleEvent.RawArchive rawArchive() {
|
||||
return new VehicleEvent.RawArchive(
|
||||
"raw-1",
|
||||
"VIN001",
|
||||
ProtocolId.GB32960,
|
||||
Instant.parse("2026-06-23T07:00:00Z"),
|
||||
Instant.parse("2026-06-23T07:00:01Z"),
|
||||
"trace-raw-1",
|
||||
Map.of("platformAccount", "Hyundai"),
|
||||
0x02,
|
||||
0,
|
||||
new byte[] {0x23, 0x23});
|
||||
}
|
||||
|
||||
private static VehicleEvent.Realtime realtime() {
|
||||
return new VehicleEvent.Realtime(
|
||||
"event-1",
|
||||
"VIN001",
|
||||
ProtocolId.GB32960,
|
||||
Instant.parse("2026-06-23T07:00:00Z"),
|
||||
Instant.parse("2026-06-23T07:00:01Z"),
|
||||
"trace-1",
|
||||
Map.of("platformAccount", "Hyundai"),
|
||||
new RealtimePayload(
|
||||
42.1, 1234.5, 80.0, null, null,
|
||||
null, null, null, null, null, null,
|
||||
RealtimePayload.VehicleState.STARTED,
|
||||
RealtimePayload.ChargingState.UNCHARGED,
|
||||
RealtimePayload.RunningMode.ELECTRIC,
|
||||
null, null, null,
|
||||
113.12, 23.45, null, null, null, null));
|
||||
}
|
||||
|
||||
private static VehicleEvent.Location location() {
|
||||
return new VehicleEvent.Location(
|
||||
"event-2",
|
||||
"VIN001",
|
||||
ProtocolId.GB32960,
|
||||
Instant.parse("2026-06-23T07:00:00Z"),
|
||||
Instant.parse("2026-06-23T07:00:01Z"),
|
||||
"trace-2",
|
||||
Map.of("platformAccount", "Hyundai"),
|
||||
new LocationPayload(113.12, 23.45, 0, 42.1, 90, 0, 0));
|
||||
}
|
||||
|
||||
private static VehicleEvent.Alarm alarm() {
|
||||
return new VehicleEvent.Alarm(
|
||||
"event-3",
|
||||
"VIN001",
|
||||
ProtocolId.GB32960,
|
||||
Instant.parse("2026-06-23T07:00:00Z"),
|
||||
Instant.parse("2026-06-23T07:00:01Z"),
|
||||
"trace-3",
|
||||
Map.of("platformAccount", "Hyundai"),
|
||||
new AlarmPayload(
|
||||
AlarmPayload.AlarmLevel.MINOR,
|
||||
1,
|
||||
"GB32960_ALARM",
|
||||
List.of("1"),
|
||||
Set.of("bit0"),
|
||||
113.12,
|
||||
23.45,
|
||||
AlarmPayload.SafetyCategory.GENERAL,
|
||||
false,
|
||||
AlarmPayload.HydrogenLeakLevel.NONE,
|
||||
false));
|
||||
}
|
||||
|
||||
private static VehicleEvent.Login login() {
|
||||
return new VehicleEvent.Login(
|
||||
"event-4",
|
||||
"VIN001",
|
||||
ProtocolId.GB32960,
|
||||
Instant.parse("2026-06-23T07:00:00Z"),
|
||||
Instant.parse("2026-06-23T07:00:01Z"),
|
||||
"trace-4",
|
||||
Map.of("platformAccount", "Hyundai"),
|
||||
"iccid-1",
|
||||
"V2016");
|
||||
}
|
||||
|
||||
private static VehicleEvent.Logout logout() {
|
||||
return new VehicleEvent.Logout(
|
||||
"event-5",
|
||||
"VIN001",
|
||||
ProtocolId.GB32960,
|
||||
Instant.parse("2026-06-23T07:00:00Z"),
|
||||
Instant.parse("2026-06-23T07:00:01Z"),
|
||||
"trace-5",
|
||||
Map.of("platformAccount", "Hyundai"));
|
||||
}
|
||||
|
||||
private static VehicleEvent.Heartbeat heartbeat() {
|
||||
return new VehicleEvent.Heartbeat(
|
||||
"event-6",
|
||||
"VIN001",
|
||||
ProtocolId.GB32960,
|
||||
Instant.parse("2026-06-23T07:00:00Z"),
|
||||
Instant.parse("2026-06-23T07:00:01Z"),
|
||||
"trace-6",
|
||||
Map.of("platformAccount", "Hyundai"));
|
||||
}
|
||||
}
|
||||
18
pom.xml
18
pom.xml
@@ -36,6 +36,9 @@
|
||||
<module>modules/inbound/inbound-mqtt</module>
|
||||
<module>modules/inbound/inbound-xinda-push</module>
|
||||
<module>modules/apps/command-gateway</module>
|
||||
<module>modules/apps/gb32960-ingest-app</module>
|
||||
<module>modules/apps/vehicle-history-app</module>
|
||||
<module>modules/apps/vehicle-analytics-app</module>
|
||||
<module>modules/apps/bootstrap-all</module>
|
||||
</modules>
|
||||
|
||||
@@ -222,6 +225,21 @@
|
||||
<artifactId>command-gateway</artifactId>
|
||||
<version>${project.version}</version>
|
||||
</dependency>
|
||||
<dependency>
|
||||
<groupId>com.lingniu.ingest</groupId>
|
||||
<artifactId>gb32960-ingest-app</artifactId>
|
||||
<version>${project.version}</version>
|
||||
</dependency>
|
||||
<dependency>
|
||||
<groupId>com.lingniu.ingest</groupId>
|
||||
<artifactId>vehicle-history-app</artifactId>
|
||||
<version>${project.version}</version>
|
||||
</dependency>
|
||||
<dependency>
|
||||
<groupId>com.lingniu.ingest</groupId>
|
||||
<artifactId>vehicle-analytics-app</artifactId>
|
||||
<version>${project.version}</version>
|
||||
</dependency>
|
||||
|
||||
<!-- 第三方（非 BOM 管理） -->
|
||||
<dependency>
|
||||
|
||||
Reference in New Issue
Block a user