Merge branch 'codex/gb32960-service-split'

This commit is contained in:
lingniu
2026-06-24 14:32:59 +08:00
48 changed files with 4166 additions and 74 deletions

View File

@@ -0,0 +1,224 @@
# GB32960 Split Service Runbook
This runbook covers the split GB32960 ingest, history, and analytics runtimes. Kafka is the durability boundary between the protocol service and downstream business services.
## Service Roles
| Service | Module | Role | Local ports |
| --- | --- | --- | --- |
| GB32960 ingest | `:gb32960-ingest-app` | Accept GB32960 TCP connections, decode/authenticate frames, produce raw and normalized records to Kafka, and send protocol ACKs after required Kafka production succeeds. | TCP `32960`, HTTP `20100` |
| Vehicle history | `:vehicle-history-app` | Consume GB32960 raw/event Kafka records, store event-history records and raw archive references, and expose event/history/GB32960 query APIs. Kafka raw envelopes carry archive URI/size metadata, not raw frame bytes. | HTTP `20200` |
| Vehicle analytics | `:vehicle-analytics-app` | Consume GB32960 event Kafka records and update analytics outputs such as daily vehicle statistics. Vehicle state can also be enabled from this runtime. | HTTP `20300` |
## Kafka Contract
| Topic | Producer | Consumers | Purpose |
| --- | --- | --- | --- |
| `vehicle.raw.gb32960.v1` | `gb32960-ingest-app` | `vehicle-history-app` | Raw archive reference records. The envelope contains URI/size metadata, not the raw `.bin` payload. |
| `vehicle.event.gb32960.v1` | `gb32960-ingest-app` | `vehicle-history-app`, `vehicle-analytics-app` | Normalized vehicle events keyed by VIN when available. |
| `vehicle.dlq.gb32960.v1` | Kafka producer/consumer error paths | Operators/replay tooling | Dead-letter records for failed production or consumer processing. |
Default local consumer groups:
- History: `vehicle-history`
- Analytics state: `vehicle-state` when `VEHICLE_STATE_ENABLED=true`
- Analytics statistics: `vehicle-stat`
## ACK Semantics
GB32960 success ACKs are sent only after the ingest service completes the required Kafka production boundary for the accepted frame. If required Kafka production fails, the ingest service must not claim a successful ACK for that frame.
History and analytics failures do not block GB32960 ACKs. They are downstream Kafka consumer failures and should be handled through retry, DLQ inspection, offset replay, and service-specific recovery.
Authorization failures are rejected before the Kafka durability boundary and can be ACKed/rejected immediately according to protocol handling.
## Local Prerequisites
- Java 25 and Maven available on PATH.
- Local Kafka reachable at `127.0.0.1:9092`, plus Kafka CLI tools such as `kafka-topics` and optionally `kafka-console-consumer`.
- No repository-local Kafka bootstrap script or compose file was found during Task 8 verification on 2026-06-23.
If your shell needs an explicit JDK:
```bash
export JAVA_HOME=/opt/homebrew/opt/openjdk@25/libexec/openjdk.jdk/Contents/Home
export PATH="$JAVA_HOME/bin:/opt/homebrew/bin:$PATH"
```
## Build
From the repository root:
```bash
mvn -pl :gb32960-ingest-app,:vehicle-history-app,:vehicle-analytics-app -am package -Dmaven.test.skip=true
```
Expected result: `BUILD SUCCESS` and these runnable jars:
- `modules/apps/gb32960-ingest-app/target/gb32960-ingest-app.jar`
- `modules/apps/vehicle-history-app/target/vehicle-history-app.jar`
- `modules/apps/vehicle-analytics-app/target/vehicle-analytics-app.jar`
## Create Topics
Run these only after Kafka is reachable at `127.0.0.1:9092`:
```bash
kafka-topics --bootstrap-server 127.0.0.1:9092 --create --if-not-exists --topic vehicle.raw.gb32960.v1 --partitions 12 --replication-factor 1
kafka-topics --bootstrap-server 127.0.0.1:9092 --create --if-not-exists --topic vehicle.event.gb32960.v1 --partitions 12 --replication-factor 1
kafka-topics --bootstrap-server 127.0.0.1:9092 --create --if-not-exists --topic vehicle.dlq.gb32960.v1 --partitions 3 --replication-factor 1
```
Optional sanity check:
```bash
kafka-topics --bootstrap-server 127.0.0.1:9092 --list | grep -E 'vehicle\.(raw|event|dlq)\.gb32960\.v1'
```
## Start Split Services Locally
Run each command in a separate terminal from the repository root.
GB32960 ingest:
```bash
KAFKA_BROKERS=127.0.0.1:9092 \
GB32960_PORT=32960 \
HTTP_PORT=20100 \
java --sun-misc-unsafe-memory-access=allow \
-jar modules/apps/gb32960-ingest-app/target/gb32960-ingest-app.jar
```
Vehicle history:
```bash
KAFKA_BROKERS=127.0.0.1:9092 \
HTTP_PORT=20200 \
EVENT_FILE_STORE_PATH=./target/split-event-store \
java --sun-misc-unsafe-memory-access=allow \
-jar modules/apps/vehicle-history-app/target/vehicle-history-app.jar
```
`SINK_ARCHIVE_PATH` is intentionally omitted in the split-service Kafka verification path. Current Kafka raw envelopes contain `RawArchiveRef` URI/size metadata only; they do not contain raw bytes that `RawArchiveEventSink` could materialize into `.bin` files in the history JVM.
Vehicle analytics:
```bash
KAFKA_BROKERS=127.0.0.1:9092 \
HTTP_PORT=20300 \
VEHICLE_STAT_FILE_PATH=./target/split-vehicle-stat \
java --sun-misc-unsafe-memory-access=allow \
-jar modules/apps/vehicle-analytics-app/target/vehicle-analytics-app.jar
```
To enable the latest-state API in analytics, also set `VEHICLE_STATE_ENABLED=true` and configure the state repository requirements used by `vehicle-state-service`.
## Health Verification
```bash
curl -sS http://127.0.0.1:20100/actuator/health
curl -sS http://127.0.0.1:20200/actuator/health
curl -sS http://127.0.0.1:20300/actuator/health
```
Expected: each endpoint returns `{"status":"UP"}` or an equivalent Spring Boot health JSON with top-level status `UP`.
## End-to-End Verification
Use a known-valid GB32960 fixture, for example one of the hex samples under `modules/protocols/protocol-gb32960/src/test/resources/samples/gb32960/`.
Example send command:
```bash
xxd -r -p modules/protocols/protocol-gb32960/src/test/resources/samples/gb32960/realtime_001.hex | nc 127.0.0.1 32960
```
To capture and inspect the binary ACK, write the response to a file:
```bash
rm -f /tmp/gb32960-ack.bin
xxd -r -p modules/protocols/protocol-gb32960/src/test/resources/samples/gb32960/realtime_001.hex \
| nc -w 3 127.0.0.1 32960 > /tmp/gb32960-ack.bin
xxd -p /tmp/gb32960-ack.bin
```
Expected: the ACK file is non-empty and the decoded bytes begin with the GB32960 frame marker `2323`. Treat the ACK as verified only after checking the response bytes from your run.
Verify only what was actually run in your environment:
```bash
kafka-console-consumer --bootstrap-server 127.0.0.1:9092 --topic vehicle.raw.gb32960.v1 --from-beginning --max-messages 1 --timeout-ms 10000
kafka-console-consumer --bootstrap-server 127.0.0.1:9092 --topic vehicle.event.gb32960.v1 --from-beginning --max-messages 1 --timeout-ms 10000
find target/split-event-store -type f | head
find target/split-vehicle-stat -type f | head
```
Useful HTTP query checks after events are consumed. For `realtime_001.hex`, the fixture VIN is `LTEST000000000001`; replace `<date-from>`, `<date-to>`, and `<stat-date>` with values that cover the consumed record's event time:
```bash
curl -sS 'http://127.0.0.1:20200/api/event-history/records?protocol=GB32960&dateFrom=<date-from>&dateTo=<date-to>&vin=LTEST000000000001&limit=10'
curl -sS 'http://127.0.0.1:20200/api/event-history/gb32960/dictionary'
curl -sS 'http://127.0.0.1:20300/api/vehicle-stat/LTEST000000000001/daily?date=<stat-date>'
```
Expected E2E result when Kafka and all services are running:
- A GB32960 client receives a binary success ACK only after required Kafka production succeeds, and the captured ACK bytes are inspected.
- `vehicle.raw.gb32960.v1` receives a raw archive reference envelope.
- `vehicle.event.gb32960.v1` receives one or more normalized records.
- `target/split-event-store` receives event-store files after the history service flushes consumed Kafka records, including raw archive reference records.
- `target/split-vehicle-stat` receives stat output after analytics consumes applicable event records.
Do not expect `vehicle-history-app` to create raw `.bin` archive files from Kafka raw records in the current implementation. `RawArchiveEventSink` can write archive files only when it receives `VehicleEvent.RawArchive.rawBytes()` inside the same JVM; the Kafka envelope carries only `RawArchiveRef` metadata.
Do not mark any of these as verified unless the matching command was run and the output was inspected.
## Rollback Guidance
If the split deployment is unhealthy:
1. Stop `gb32960-ingest-app` first so new GB32960 ACKs are not issued against an unhealthy Kafka boundary.
2. Keep Kafka topics intact for replay unless storage or privacy policy requires deletion.
3. Restart or roll back `vehicle-history-app` and `vehicle-analytics-app` independently; their failures do not require protocol ACK rollback because they consume from Kafka offsets.
4. If Kafka itself is unavailable, route GB32960 traffic back to the previous all-in-one runtime only if that runtime is configured with its known-good durability behavior.
5. After rollback, compare Kafka consumer group lag and DLQ contents before resuming the split services.
Rollback commands depend on the deployment environment. For local testing, stop the three Java processes and restart the previous `bootstrap-all` workflow if needed.
## Task 8 Verification Notes
Observed on 2026-06-23 in worktree `.worktrees/gb32960-service-split`:
- Package build ran with `mvn -pl :gb32960-ingest-app,:vehicle-history-app,:vehicle-analytics-app -am package -Dmaven.test.skip=true` and ended with `BUILD SUCCESS`.
- Repository-local Kafka setup inspection found no Kafka script, no Docker Compose file, and no compose YAML within the searched repository paths.
- `nc -z -w 2 127.0.0.1 9092` exited `1`, so no local Kafka broker was reachable at `127.0.0.1:9092`.
- `kafka-topics`, `kafka-topics.sh`, `docker`, and `docker-compose` were not found on PATH.
- Kafka topic creation, service startup, health checks, Kafka record checks, archive checks, event-store checks, stat output checks, and ACK observation were not run because the local Kafka prerequisite was absent.
## Latest Local Verification
Observed on 2026-06-23 in worktree `.worktrees/gb32960-service-split`:
- Targeted module tests: `mvn -pl :sink-mq,:ingest-core,:protocol-gb32960,:event-history-service,:vehicle-state-service,:vehicle-stat-service test` ended with `BUILD SUCCESS`; Maven reported 55 protocol/sink/core tests, 30 event-history tests, 14 vehicle-state tests, and 19 vehicle-stat tests with 0 failures/errors/skips.
- Split app tests: `mvn -pl :gb32960-ingest-app,:vehicle-history-app,:vehicle-analytics-app test` ended with `BUILD SUCCESS`; Maven reported 6 app composition/default tests with 0 failures/errors/skips.
- Package verification: `mvn -pl :gb32960-ingest-app,:vehicle-history-app,:vehicle-analytics-app -am package -Dmaven.test.skip=true` ended with `BUILD SUCCESS` and repackaged all three app jars.
- Split app startup: not run in this verification pass because local Kafka was still absent.
- Kafka raw records: not verified; no local Kafka broker or Kafka CLI tools were available.
- Kafka event records: not verified; no local Kafka broker or Kafka CLI tools were available.
- Event-store files: not verified; downstream services were not started without Kafka.
- Analytics output: not verified; downstream services were not started without Kafka.
- ACK behavior: not verified against a live broker in this pass. Unit tests cover the GB32960 ACK boundary and ordering, but an operator should still run the ACK capture command above in a Kafka-backed local or staging environment.
## Remote Kafka Live Verification
Observed on 2026-06-23 with Kafka `114.55.58.251:9092`:
- Replaced the local `bootstrap-all` listener on TCP `32960` with `gb32960-ingest-app` on TCP `32960` and HTTP `20100`.
- Ingest app health check `curl -sS http://127.0.0.1:20100/actuator/health` returned `{"status":"UP"}`.
- Kafka AdminClient confirmed topics `vehicle.raw.gb32960.v1` and `vehicle.event.gb32960.v1` existed, and created `vehicle.dlq.gb32960.v1`.
- The app accepted the live Hyundai platform connection from `8.134.95.166`, authenticated platform login, and received real GB32960 realtime/report/login/logout frames.
- Probe consumer with a temporary group read records from `vehicle.event.gb32960.v1`, including VIN keys such as `LB9A32A2XR0LS1575` and `LNXNEGRR3SR319485`.
- Probe consumer with a temporary group read records from `vehicle.raw.gb32960.v1`, including VIN keys such as `LNXNEGRR0SR319444`, `LNXNEGRR2SR318201`, and `LB9A32A23R0LS1532`.
- Runtime logs showed successful GB32960 ACK flushes for platform login, vehicle login, and vehicle logout after the Kafka-backed split ingest service was running.
This verifies the live path `GB32960 TCP 32960 -> gb32960-ingest-app -> Kafka raw/event topics` against the remote Kafka broker. Downstream `vehicle-history-app` and `vehicle-analytics-app` were not started in this pass.

View File

@@ -0,0 +1,810 @@
# GB32960 History DuckDB Hot Store Implementation Plan
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
**Goal:** Replace the current high-write-risk Parquet rewrite history path with a DuckDB hot history store that supports fast `vin + time` queries and full RAW frame replay.
**Architecture:** Add a new `DuckDbHotEventFileStore` implementation behind the existing `EventFileStore` interface, backed by append/upsert DuckDB tables instead of rewriting per-vehicle Parquet files. Keep the current HTTP history API and RAW archive reader intact, then switch `vehicle-history-app` to the hot DuckDB store by configuration.
**Tech Stack:** Java 26, Spring Boot, DuckDB JDBC, JUnit 5, AssertJ, existing Kafka `VehicleEnvelope`, existing `ArchiveStore`.
---
## Scope
This plan implements Phase 1 from `docs/superpowers/specs/2026-06-23-gb32960-production-readiness-design.md`.
Included:
- DuckDB hot event table for `EventFileRecord`.
- Idempotent writes by `event_id`.
- Query by protocol/date/VIN/eventType/eventTime/order/limit.
- Lookup by `rawArchiveUri`.
- Spring configuration to select the hot store.
- Tests that prove history endpoints still replay RAW frames.
- Local verification using the existing `vehicle-history-app`.
Not included in this plan:
- MySQL daily statistics.
- VIN-to-platform local mapping.
- Full telemetry point columnar table.
- Alarm timeline MySQL tables.
- Cold Parquet export.
Those will be separate plans after this foundation lands.
## Files
- Create: `modules/sinks/event-file-store/src/main/java/com/lingniu/ingest/eventfilestore/DuckDbHotEventFileStore.java`
- Create: `modules/sinks/event-file-store/src/test/java/com/lingniu/ingest/eventfilestore/DuckDbHotEventFileStoreTest.java`
- Modify: `modules/sinks/event-file-store/src/main/java/com/lingniu/ingest/eventfilestore/config/EventFileStoreProperties.java`
- Modify: `modules/sinks/event-file-store/src/main/java/com/lingniu/ingest/eventfilestore/config/EventFileStoreAutoConfiguration.java`
- Modify: `modules/sinks/event-file-store/src/test/java/com/lingniu/ingest/eventfilestore/config/EventFileStoreAutoConfigurationTest.java`
- Modify: `modules/apps/vehicle-history-app/src/main/resources/application.yml`
- Modify: `modules/apps/vehicle-history-app/src/test/java/com/lingniu/ingest/historyapp/VehicleHistoryAppDefaultsTest.java`
- Modify: `modules/services/event-history-service/src/test/java/com/lingniu/ingest/eventhistory/Gb32960DecodedFrameServiceTest.java`
## Task 1: Add Hot Store Configuration
**Files:**
- Modify: `modules/sinks/event-file-store/src/main/java/com/lingniu/ingest/eventfilestore/config/EventFileStoreProperties.java`
- Modify: `modules/sinks/event-file-store/src/main/java/com/lingniu/ingest/eventfilestore/config/EventFileStoreAutoConfiguration.java`
- Test: `modules/sinks/event-file-store/src/test/java/com/lingniu/ingest/eventfilestore/config/EventFileStoreAutoConfigurationTest.java`
- [ ] **Step 1: Write the failing auto-configuration test**
Add a test that sets `lingniu.ingest.event-file-store.storage=duckdb-hot` and expects the bean class to be `DuckDbHotEventFileStore`.
```java
@Test
void createsDuckDbHotStoreWhenStorageIsDuckDbHot() {
contextRunner
.withPropertyValues(
"lingniu.ingest.event-file-store.enabled=true",
"lingniu.ingest.event-file-store.storage=duckdb-hot",
"lingniu.ingest.event-file-store.path=" + tempDir.resolve("history"))
.run(context -> assertThat(context)
.hasSingleBean(EventFileStore.class)
.getBean(EventFileStore.class)
.isInstanceOf(DuckDbHotEventFileStore.class));
}
```
- [ ] **Step 2: Run the new test and verify it fails**
Run:
```bash
mvn -pl :event-file-store -Dtest=EventFileStoreAutoConfigurationTest#createsDuckDbHotStoreWhenStorageIsDuckDbHot test
```
Expected: compilation failure because `DuckDbHotEventFileStore` and `storage` property do not exist.
- [ ] **Step 3: Add the `storage` property**
Add to `EventFileStoreProperties`:
```java
/**
* Storage backend. `duckdb-hot` is the production backend; `parquet-sidecar`
* keeps the previous implementation available for compatibility tests.
*/
private String storage = "duckdb-hot";
public String getStorage() {
return storage;
}
public void setStorage(String storage) {
this.storage = storage;
}
```
- [ ] **Step 4: Switch auto-configuration by storage mode**
Change `eventFileStore(...)` to:
```java
@Bean
@ConditionalOnMissingBean
public EventFileStore eventFileStore(EventFileStoreProperties properties,
ObjectProvider<ObjectMapper> objectMapper) {
ObjectMapper mapper = mapper(objectMapper);
Path root = Path.of(properties.getPath());
ZoneId zoneId = ZoneId.of(properties.getZoneId());
String storage = properties.getStorage() == null ? "" : properties.getStorage().trim();
return switch (storage) {
case "", "duckdb-hot" -> new DuckDbHotEventFileStore(root, zoneId, mapper);
case "parquet-sidecar" -> new DuckDbParquetEventFileStore(root, zoneId, mapper);
default -> throw new IllegalStateException(
"unsupported event-file-store storage: " + properties.getStorage());
};
}
```
Import `DuckDbHotEventFileStore`.
- [ ] **Step 5: Run configuration tests**
Run:
```bash
mvn -pl :event-file-store -Dtest=EventFileStoreAutoConfigurationTest test
```
Expected: tests pass after the store class exists in Task 2.
## Task 2: Implement DuckDB Hot Store Schema and Writes
**Files:**
- Create: `modules/sinks/event-file-store/src/main/java/com/lingniu/ingest/eventfilestore/DuckDbHotEventFileStore.java`
- Create: `modules/sinks/event-file-store/src/test/java/com/lingniu/ingest/eventfilestore/DuckDbHotEventFileStoreTest.java`
- [ ] **Step 1: Write failing append/query/idempotency tests**
Create `DuckDbHotEventFileStoreTest`:
```java
class DuckDbHotEventFileStoreTest {
@TempDir
Path tempDir;
@Test
void appendsRecordsAndQueriesByVinTypeAndTime() throws Exception {
EventFileStore store = store();
store.append(rawRecord("raw-1", "VIN001", "2026-06-23T01:00:00Z"));
store.append(rawRecord("raw-2", "VIN002", "2026-06-23T01:00:01Z"));
store.append(rawRecord("raw-3", "VIN001", "2026-06-23T01:00:02Z"));
EventFileQuery query = new EventFileQuery(
ProtocolId.GB32960,
LocalDate.parse("2026-06-23"),
LocalDate.parse("2026-06-23"),
Instant.parse("2026-06-23T01:00:00Z"),
Instant.parse("2026-06-23T01:00:03Z"),
EventFileQuery.Order.DESC,
2,
"VIN001",
"RAW_ARCHIVE");
assertThat(store.query(query))
.extracting(EventFileRecord::eventId)
.containsExactly("raw-3", "raw-1");
}
@Test
void appendAllIsIdempotentByEventId() throws Exception {
EventFileStore store = store();
EventFileRecord original = rawRecord("same-id", "VIN001", "2026-06-23T01:00:00Z");
EventFileRecord replacement = new EventFileRecord(
"same-id",
ProtocolId.GB32960,
"RAW_ARCHIVE",
"VIN001",
Instant.parse("2026-06-23T01:00:05Z"),
Instant.parse("2026-06-23T01:00:06Z"),
"archive://replacement.bin",
Map.of("source", "replacement"),
"{\"replacement\":true}");
store.appendAll(List.of(original, replacement));
assertThat(store.query(new EventFileQuery(
ProtocolId.GB32960,
LocalDate.parse("2026-06-23"),
LocalDate.parse("2026-06-23"),
EventFileQuery.Order.ASC,
10,
"VIN001",
"RAW_ARCHIVE")))
.singleElement()
.satisfies(record -> {
assertThat(record.eventId()).isEqualTo("same-id");
assertThat(record.rawArchiveUri()).isEqualTo("archive://replacement.bin");
});
}
@Test
void findsRecordByRawArchiveUri() throws Exception {
EventFileStore store = store();
EventFileRecord record = rawRecord("raw-uri", "VIN001", "2026-06-23T01:00:00Z");
store.append(record);
EventFileRecord found = store.findByRawArchiveUri(record.rawArchiveUri());
assertThat(found).isNotNull();
assertThat(found.eventId()).isEqualTo("raw-uri");
}
private EventFileStore store() {
return new DuckDbHotEventFileStore(tempDir, ZoneId.of("Asia/Shanghai"), new ObjectMapper());
}
private static EventFileRecord rawRecord(String id, String vin, String eventTime) {
return new EventFileRecord(
id,
ProtocolId.GB32960,
"RAW_ARCHIVE",
vin,
Instant.parse(eventTime),
Instant.parse(eventTime).plusMillis(100),
"archive://" + id + ".bin",
Map.of("platformAccount", "Hyundai", "command", "REALTIME_REPORT"),
"{\"eventId\":\"" + id + "\"}");
}
}
```
- [ ] **Step 2: Run the test and verify it fails**
Run:
```bash
mvn -pl :event-file-store -Dtest=DuckDbHotEventFileStoreTest test
```
Expected: compilation failure because `DuckDbHotEventFileStore` does not exist.
- [ ] **Step 3: Create the hot store class**
Create `DuckDbHotEventFileStore` with this structure:
```java
public final class DuckDbHotEventFileStore implements EventFileStore {
private static final TypeReference<Map<String, String>> STRING_MAP = new TypeReference<>() {};
private final Path root;
private final Path dbPath;
private final ZoneId partitionZone;
private final ObjectMapper objectMapper;
private volatile boolean initialized;
public DuckDbHotEventFileStore(Path root, ZoneId partitionZone) {
this(root, partitionZone, new ObjectMapper());
}
public DuckDbHotEventFileStore(Path root, ZoneId partitionZone, ObjectMapper objectMapper) {
if (root == null) {
throw new IllegalArgumentException("root must not be null");
}
this.root = root.toAbsolutePath();
this.dbPath = this.root.resolve("events.duckdb");
this.partitionZone = partitionZone == null ? ZoneId.of("Asia/Shanghai") : partitionZone;
this.objectMapper = objectMapper == null ? new ObjectMapper() : objectMapper;
}
@Override
public synchronized void appendAll(List<EventFileRecord> records) throws IOException {
if (records == null || records.isEmpty()) {
return;
}
ensureInitialized();
try (Connection connection = DriverManager.getConnection(jdbcUrl())) {
connection.setAutoCommit(false);
try {
upsertRecords(connection, records);
connection.commit();
} catch (SQLException | IOException e) {
connection.rollback();
throw e;
} finally {
connection.setAutoCommit(true);
}
} catch (SQLException e) {
throw new IOException("write duckdb hot event store failed", e);
}
}
@Override
public List<EventFileRecord> query(EventFileQuery query) throws IOException {
ensureInitialized();
// Implement in Task 3.
return List.of();
}
@Override
public EventFileRecord findByRawArchiveUri(String rawArchiveUri) throws IOException {
ensureInitialized();
// Implement in Task 3.
return null;
}
}
```
- [ ] **Step 4: Add schema initialization**
Add:
```java
private void ensureInitialized() throws IOException {
if (initialized) {
return;
}
synchronized (this) {
if (initialized) {
return;
}
Files.createDirectories(root);
try (Connection connection = DriverManager.getConnection(jdbcUrl());
Statement statement = connection.createStatement()) {
statement.execute("""
CREATE TABLE IF NOT EXISTS event_records (
event_id VARCHAR PRIMARY KEY,
protocol VARCHAR NOT NULL,
event_type VARCHAR NOT NULL,
vin VARCHAR NOT NULL,
event_time_ms BIGINT NOT NULL,
ingest_time_ms BIGINT NOT NULL,
partition_date DATE NOT NULL,
raw_archive_uri VARCHAR NOT NULL,
metadata_json VARCHAR NOT NULL,
payload_json VARCHAR NOT NULL
)
""");
statement.execute("""
CREATE INDEX IF NOT EXISTS event_records_protocol_date_time_idx
ON event_records(protocol, partition_date, event_time_ms)
""");
statement.execute("""
CREATE INDEX IF NOT EXISTS event_records_vin_date_time_idx
ON event_records(protocol, vin, partition_date, event_time_ms)
""");
statement.execute("""
CREATE INDEX IF NOT EXISTS event_records_vin_type_date_time_idx
ON event_records(protocol, vin, event_type, partition_date, event_time_ms)
""");
statement.execute("""
CREATE INDEX IF NOT EXISTS event_records_raw_archive_uri_idx
ON event_records(raw_archive_uri)
""");
initialized = true;
} catch (SQLException e) {
throw new IOException("initialize duckdb hot event store failed", e);
}
}
}
```
- [ ] **Step 5: Add idempotent batch upsert**
Use DuckDB `INSERT OR REPLACE` inside one transaction:
```java
private void upsertRecords(Connection connection, List<EventFileRecord> records)
throws SQLException, IOException {
try (PreparedStatement ps = connection.prepareStatement("""
INSERT OR REPLACE INTO event_records VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
""")) {
for (EventFileRecord record : records) {
ps.setString(1, record.eventId());
ps.setString(2, record.protocol().name());
ps.setString(3, record.eventType());
ps.setString(4, record.vin());
ps.setLong(5, record.eventTime().toEpochMilli());
ps.setLong(6, record.ingestTime().toEpochMilli());
ps.setString(7, LocalDate.ofInstant(record.eventTime(), partitionZone).toString());
ps.setString(8, record.rawArchiveUri());
ps.setString(9, objectMapper.writeValueAsString(record.metadata()));
ps.setString(10, record.payloadJson());
ps.addBatch();
}
ps.executeBatch();
}
}
```
- [ ] **Step 6: Run the hot store tests**
Run:
```bash
mvn -pl :event-file-store -Dtest=DuckDbHotEventFileStoreTest test
```
Expected: query tests still fail until Task 3 implements reads; append initialization should compile.
## Task 3: Implement Hot Store Queries and Raw URI Lookup
**Files:**
- Modify: `modules/sinks/event-file-store/src/main/java/com/lingniu/ingest/eventfilestore/DuckDbHotEventFileStore.java`
- Test: `modules/sinks/event-file-store/src/test/java/com/lingniu/ingest/eventfilestore/DuckDbHotEventFileStoreTest.java`
- [ ] **Step 1: Implement `query(EventFileQuery)`**
Use prepared statements for every external value:
```java
@Override
public List<EventFileRecord> query(EventFileQuery query) throws IOException {
ensureInitialized();
String order = query.order() == EventFileQuery.Order.DESC ? "DESC" : "ASC";
StringBuilder where = new StringBuilder("""
WHERE protocol = ?
AND partition_date BETWEEN CAST(? AS DATE) AND CAST(? AS DATE)
""");
if (query.vin() != null) {
where.append(" AND vin = ?\n");
}
if (query.eventType() != null) {
where.append(" AND event_type = ?\n");
}
if (query.eventTimeFrom() != null) {
where.append(" AND event_time_ms >= ?\n");
}
if (query.eventTimeTo() != null) {
where.append(" AND event_time_ms <= ?\n");
}
String sql = """
SELECT event_id, protocol, event_type, vin, event_time_ms, ingest_time_ms,
raw_archive_uri, metadata_json, payload_json
FROM event_records
%s
ORDER BY event_time_ms %s, ingest_time_ms %s, event_id %s
LIMIT ?
""".formatted(where, order, order, order);
try (Connection connection = DriverManager.getConnection(jdbcUrl());
PreparedStatement ps = connection.prepareStatement(sql)) {
bindQuery(ps, query);
try (ResultSet rs = ps.executeQuery()) {
List<EventFileRecord> out = new ArrayList<>();
while (rs.next()) {
out.add(record(rs));
}
return out;
}
} catch (SQLException e) {
throw new IOException("query duckdb hot event store failed", e);
}
}
```
- [ ] **Step 2: Add query binding helper**
```java
private static void bindQuery(PreparedStatement ps, EventFileQuery query) throws SQLException {
int index = 1;
ps.setString(index++, query.protocol().name());
ps.setString(index++, query.dateFrom().toString());
ps.setString(index++, query.dateTo().toString());
if (query.vin() != null) {
ps.setString(index++, query.vin());
}
if (query.eventType() != null) {
ps.setString(index++, query.eventType());
}
if (query.eventTimeFrom() != null) {
ps.setLong(index++, query.eventTimeFrom().toEpochMilli());
}
if (query.eventTimeTo() != null) {
ps.setLong(index++, query.eventTimeTo().toEpochMilli());
}
ps.setInt(index, query.limit());
}
```
- [ ] **Step 3: Implement `findByRawArchiveUri`**
```java
@Override
public EventFileRecord findByRawArchiveUri(String rawArchiveUri) throws IOException {
if (rawArchiveUri == null || rawArchiveUri.isBlank()) {
return null;
}
ensureInitialized();
try (Connection connection = DriverManager.getConnection(jdbcUrl());
PreparedStatement ps = connection.prepareStatement("""
SELECT event_id, protocol, event_type, vin, event_time_ms, ingest_time_ms,
raw_archive_uri, metadata_json, payload_json
FROM event_records
WHERE raw_archive_uri = ?
ORDER BY event_type = 'RAW_ARCHIVE' DESC, ingest_time_ms DESC, event_id DESC
LIMIT 1
""")) {
ps.setString(1, rawArchiveUri);
try (ResultSet rs = ps.executeQuery()) {
return rs.next() ? record(rs) : null;
}
} catch (SQLException e) {
throw new IOException("query duckdb hot store by raw archive uri failed", e);
}
}
```
- [ ] **Step 4: Add record mapper and helpers**
```java
private EventFileRecord record(ResultSet rs) throws SQLException, IOException {
return new EventFileRecord(
rs.getString("event_id"),
protocol(rs.getString("protocol")),
rs.getString("event_type"),
rs.getString("vin"),
Instant.ofEpochMilli(rs.getLong("event_time_ms")),
Instant.ofEpochMilli(rs.getLong("ingest_time_ms")),
rs.getString("raw_archive_uri"),
readMetadata(rs.getString("metadata_json")),
rs.getString("payload_json"));
}
private Map<String, String> readMetadata(String json) throws IOException {
if (json == null || json.isBlank()) {
return Map.of();
}
return objectMapper.readValue(json, STRING_MAP);
}
private static ProtocolId protocol(String value) {
if (value == null || value.isBlank()) {
return ProtocolId.UNKNOWN;
}
try {
return ProtocolId.valueOf(value);
} catch (IllegalArgumentException ex) {
return ProtocolId.UNKNOWN;
}
}
private String jdbcUrl() {
return "jdbc:duckdb:" + dbPath;
}
```
- [ ] **Step 5: Run hot store tests**
Run:
```bash
mvn -pl :event-file-store -Dtest=DuckDbHotEventFileStoreTest test
```
Expected: all `DuckDbHotEventFileStoreTest` tests pass.
## Task 4: Preserve Legacy Store Tests and Update Defaults
**Files:**
- Modify: `modules/sinks/event-file-store/src/test/java/com/lingniu/ingest/eventfilestore/DuckDbParquetEventFileStoreTest.java`
- Modify: `modules/apps/vehicle-history-app/src/main/resources/application.yml`
- Modify: `modules/apps/vehicle-history-app/src/test/java/com/lingniu/ingest/historyapp/VehicleHistoryAppDefaultsTest.java`
- [ ] **Step 1: Keep Parquet tests explicitly legacy**
No behavior change is needed in `DuckDbParquetEventFileStoreTest`; leave it instantiating `DuckDbParquetEventFileStore` directly. Add a class comment:
```java
/**
* Compatibility coverage for the legacy Parquet sidecar backend.
* Production history uses DuckDbHotEventFileStore through auto-configuration.
*/
class DuckDbParquetEventFileStoreTest {
```
- [ ] **Step 2: Set vehicle-history-app storage default**
Add to `modules/apps/vehicle-history-app/src/main/resources/application.yml`:
```yaml
event-file-store:
enabled: ${EVENT_FILE_STORE_ENABLED:true}
storage: ${EVENT_FILE_STORE_STORAGE:duckdb-hot}
path: ${EVENT_FILE_STORE_PATH:./target/event-store/}
zone-id: ${EVENT_FILE_STORE_ZONE_ID:Asia/Shanghai}
batch-size: ${EVENT_FILE_STORE_BATCH_SIZE:1000}
flush-interval-millis: ${EVENT_FILE_STORE_FLUSH_INTERVAL_MILLIS:1000}
```
Keep existing indentation and only add `storage`; if `batch-size` already exists, update it to `1000`.
- [ ] **Step 3: Update app default test**
In `VehicleHistoryAppDefaultsTest`, assert:
```java
assertThat(context.getEnvironment()
.getProperty("lingniu.ingest.event-file-store.storage"))
.isEqualTo("duckdb-hot");
```
- [ ] **Step 4: Run app default tests**
Run:
```bash
mvn -pl :vehicle-history-app -Dtest=VehicleHistoryAppDefaultsTest test
```
Expected: default configuration test passes.
## Task 5: Verify History Ingest Still Archives RAW Bytes
**Files:**
- Modify: `modules/services/event-history-service/src/test/java/com/lingniu/ingest/eventhistory/EventHistoryEnvelopeIngestorTest.java`
- Test: `modules/services/event-history-service/src/test/java/com/lingniu/ingest/eventhistory/Gb32960DecodedFrameServiceTest.java`
- [ ] **Step 1: Add an integration-style test using the hot store**
Add a test that writes a RAW envelope through `EventHistoryEnvelopeIngestor` into `DuckDbHotEventFileStore`, then finds it by URI.
```java
@Test
void rawArchiveEnvelopeCanBeFoundFromDuckDbHotStoreByUri(@TempDir Path tempDir) throws Exception {
EventFileStore store = new DuckDbHotEventFileStore(tempDir, ZoneId.of("Asia/Shanghai"), OBJECT_MAPPER);
CapturingArchiveStore archive = new CapturingArchiveStore();
EventHistoryEnvelopeIngestor ingestor =
new EventHistoryEnvelopeIngestor(store, new TelemetryEnvelopeRecordMapper(), archive);
String rawArchiveKey = "2026/06/23/GB32960/VINRAW001/raw-event-hot.bin";
String rawArchiveUri = "archive://" + rawArchiveKey;
byte[] rawBytes = new byte[]{0x23, 0x23, 0x02, 0x01};
VehicleEnvelope envelope = VehicleEnvelope.newBuilder()
.setSchemaVersion("1.0")
.setEventId("raw-event-hot")
.setVin("VINRAW001")
.setSource("GB32960")
.setProtocolVersion("V2016")
.setEventTimeMs(1_782_112_400_000L)
.setIngestTimeMs(1_782_112_401_000L)
.putMetadata(RawArchiveKeys.META_KEY, rawArchiveKey)
.putMetadata(RawArchiveKeys.META_URI, rawArchiveUri)
.setRawArchive(RawArchiveRef.newBuilder()
.setUri(rawArchiveUri)
.setSizeBytes(rawBytes.length)
.setData(ByteString.copyFrom(rawBytes))
.build())
.build();
EnvelopeIngestResult result = ingestor.tryIngest(envelope.toByteArray());
assertThat(result.status()).isEqualTo(EnvelopeIngestResult.Status.STORED);
assertThat(store.findByRawArchiveUri(rawArchiveUri))
.isNotNull()
.extracting(EventFileRecord::eventId)
.isEqualTo("raw-event-hot");
assertThat(archive.bytesByKey).containsEntry(rawArchiveKey, rawBytes);
}
```
- [ ] **Step 2: Run the event history ingestor test**
Run:
```bash
mvn -pl :event-history-service -Dtest=EventHistoryEnvelopeIngestorTest test
```
Expected: test passes.
- [ ] **Step 3: Run decoded frame service tests**
Run:
```bash
mvn -pl :event-history-service -Dtest=Gb32960DecodedFrameServiceTest test
```
Expected: existing replay/snapshot tests pass. If they use a fake store, no change is needed.
## Task 6: Full Module Verification
**Files:**
- No source changes unless failures expose missing imports or config assertions.
- [ ] **Step 1: Run sink module tests**
Run:
```bash
mvn -pl :event-file-store test
```
Expected: all event-file-store tests pass, including both hot and legacy stores.
- [ ] **Step 2: Run event-history-service tests**
Run:
```bash
mvn -pl :event-history-service test
```
Expected: all event-history-service tests pass.
- [ ] **Step 3: Package vehicle-history-app**
Run:
```bash
mvn -pl :vehicle-history-app -am package -DskipTests
```
Expected: package succeeds.
## Task 7: Local Runtime Verification
**Files:**
- No committed source changes.
- [ ] **Step 1: Stop any old history service on port 20200**
Run:
```bash
lsof -tiTCP:20200 -sTCP:LISTEN | xargs -r kill
```
Expected: no command output, or the old process exits.
- [ ] **Step 2: Start vehicle-history-app with hot DuckDB store**
Run:
```bash
EVENT_FILE_STORE_STORAGE=duckdb-hot \
EVENT_FILE_STORE_PATH=./target/live-history-event-store \
SINK_ARCHIVE_PATH=./target/live-history-archive \
KAFKA_BROKERS=114.55.58.251:9092 \
KAFKA_CONSUMER_ENABLED=true \
KAFKA_GROUP_HISTORY=vehicle-history-hot-$(date +%Y%m%d%H%M%S) \
java --sun-misc-unsafe-memory-access=allow \
-jar modules/apps/vehicle-history-app/target/vehicle-history-app.jar
```
Expected:
- app starts on `http://127.0.0.1:20200`;
- logs show Kafka consumer subscribed;
- `target/live-history-event-store/events.duckdb` is created.
- [ ] **Step 3: Verify health**
Run:
```bash
curl -sS http://127.0.0.1:20200/actuator/health
```
Expected:
```json
{"status":"UP"}
```
- [ ] **Step 4: Verify a recent VIN query**
Run with a VIN observed in live archive:
```bash
curl -sS 'http://127.0.0.1:20200/api/event-history/gb32960/telemetry-snapshots?vin=LNXNEGRR9SR318194&platformAccount=Hyundai&dateFrom=2026-06-23&dateTo=2026-06-24&order=DESC&limit=3'
```
Expected:
- response is a JSON array;
- if live traffic for that VIN exists after service start, at least one snapshot appears;
- `rawArchiveUris` point to existing files under `target/live-history-archive`.
- [ ] **Step 5: Verify raw frame replay**
Use one `rawArchiveUri` from Step 4:
```bash
curl -sS 'http://127.0.0.1:20200/api/event-history/gb32960/frame?rawArchiveUri=archive://REPLACE_ME&platformAccount=Hyundai'
```
Expected:
- response contains `vin`, `command`, `eventTime`, and parsed `blocks`;
- no `raw archive is missing` warning for the selected URI.
## Self-Review Checklist
- [ ] The new hot store does not rewrite Parquet files on every append.
- [ ] `appendAll` writes one batch in one transaction.
- [ ] Duplicate `event_id` replay is idempotent.
- [ ] Existing history HTTP APIs still use `EventFileStore`, so controller contracts are unchanged.
- [ ] RAW bytes still land in `ArchiveStore`.
- [ ] `findByRawArchiveUri` is backed by DuckDB index.
- [ ] No MySQL credential, RDS host, username, or password appears in the plan or source files.
- [ ] This plan does not claim the full production goal is complete; it only lands the history foundation.

View File

@@ -0,0 +1,487 @@
# GB32960 Production Readiness Design
## Goal
Make the GB32960 ingestion, history, and analytics services production-ready:
- receive and parse GB32960 traffic quickly and accurately;
- maximize VIN-to-platform/vendor-profile association through platform login and local mappings;
- store complete RAW frames and queryable telemetry history with DuckDB;
- support full frame replay from persisted RAW bytes;
- compute real-time daily vehicle metrics and alarm timelines into MySQL;
- reduce service coupling and improve cohesion inside each module.
## Non-Goals
- Do not put database credentials in code, YAML files, tests, docs, or commits.
- Do not make MySQL the source for full telemetry history. MySQL is for derived daily metrics and alarm timelines.
- Do not require every private GB32960 extension to be perfect before storing data. RAW bytes must always be kept when a valid frame is received.
## Production Architecture
```mermaid
flowchart LR
tcp["GB32960 TCP:32960"] --> ingest["gb32960-ingest-app"]
mapping["local vin-platform-profile.jsonl"] --> ingest
ingest --> kafkaEvent["Kafka vehicle.event.gb32960.v1"]
ingest --> kafkaRaw["Kafka vehicle.raw.gb32960.v1"]
kafkaEvent --> history["vehicle-history-app"]
kafkaRaw --> history
history --> duckdb["DuckDB hot history"]
history --> raw["RAW frame archive"]
kafkaEvent --> analytics["vehicle-analytics-app"]
analytics --> mysql["MySQL daily stats + alarms"]
```
## Service Boundaries
### 1. `gb32960-ingest-app`
Responsibilities:
- Netty TCP accept, frame splitting, GB32960 decode, auth, ACK, and Kafka production.
- Platform login handling and connection-local `platformAccount`.
- VIN-to-platform/vendor-profile resolution.
- Produce structured telemetry, session, alarm, diagnostics, and RAW envelope messages.
It must not:
- write DuckDB, MySQL, or local archive files;
- run historical queries;
- run daily statistics.
ACK policy:
- login/report-like GB32960 commands use durable ACK: ACK only after Kafka dispatch succeeds;
- malformed bytes that cannot form a valid frame are dropped with diagnostics;
- valid frames with partial private-block parse failures still go to Kafka with `parseStatus=PARTIAL`.
VIN/profile resolution priority:
1. active platform login on the current TCP channel;
2. exact VIN entry from local mapping file;
3. VIN prefix entry from local mapping file;
4. configured platform extension rule;
5. standard GB32960 profile.
Local mapping file format:
```jsonl
{"vin":"LNXNEGRR2SR321390","platformAccount":"Hyundai","vendorProfile":"guangdong-fc","enabled":true}
{"vinPrefix":"LNXNEGRR","platformAccount":"Hyundai","vendorProfile":"guangdong-fc","enabled":true}
```
The mapping file is loaded at startup and can be reloaded by an actuator endpoint or file watcher in a later phase.
### 2. `vehicle-history-app`
Responsibilities:
- Consume Kafka event and RAW topics.
- Persist RAW bytes to an append-only archive.
- Persist query indexes and compact telemetry points to DuckDB.
- Serve high-performance query APIs:
- full frame list by `vin + eventTime`;
- telemetry fields by `vin + eventTime`;
- one-frame replay by `rawArchiveUri`;
- parse diagnostics and missing-profile troubleshooting.
It must not:
- listen on GB32960 TCP;
- run MySQL statistics;
- mutate daily business metrics.
### 3. `vehicle-analytics-app`
Responsibilities:
- Consume Kafka event topic.
- Maintain real-time daily stats in MySQL.
- Maintain full alarm timelines in MySQL.
- Expose metrics query APIs and operational health.
It must not:
- parse RAW frames for normal operation;
- write DuckDB history;
- own GB32960 TCP connectivity.
## Kafka Contract
Keep Protobuf `VehicleEnvelope`, but make the producer contract stricter:
- `eventId`: stable idempotency key, deterministic from protocol, VIN, command, event time, raw checksum, and sequence when available.
- `vin`: required for vehicle frames; platform-only commands use `_platform` or empty plus `platformAccount`.
- `eventTimeMs`: GB32960 data collection time when available.
- `ingestTimeMs`: service receive time.
- `metadata.platformAccount`: resolved platform account.
- `metadata.vendorProfile`: selected parser profile.
- `metadata.parseStatus`: `OK`, `PARTIAL`, or `FAILED`.
- `metadata.parseErrorCode`: stable code for failures.
- `raw_archive.data`: present on RAW topic only; event topic keeps `rawArchiveUri` pointer.
- `telemetry_snapshot`: standard field projection for downstream services.
Consumer rule:
- History consumes both event and RAW topics.
- Analytics consumes only event topic.
- Consumer offset is committed only after the target store commit succeeds.
- Reprocessing must be safe through deterministic ids and database upsert/ignore semantics.
## DuckDB History Design
Use DuckDB as the hot historical database, not a Parquet-rewrite sidecar.
DuckDB official guidance says row-by-row insert loops are inefficient for bulk loading, and Appender/batched writes are the intended high-throughput path. Therefore history writes use a single writer worker with bounded batches and one DuckDB connection.
### Tables
`gb32960_frame_index`
```sql
CREATE TABLE IF NOT EXISTS gb32960_frame_index (
event_id VARCHAR PRIMARY KEY,
vin VARCHAR NOT NULL,
platform_account VARCHAR,
vendor_profile VARCHAR,
command VARCHAR NOT NULL,
command_code INTEGER NOT NULL,
event_time TIMESTAMP NOT NULL,
event_time_ms BIGINT NOT NULL,
ingest_time TIMESTAMP NOT NULL,
ingest_time_ms BIGINT NOT NULL,
partition_date DATE NOT NULL,
raw_archive_uri VARCHAR NOT NULL,
raw_checksum VARCHAR,
raw_size_bytes BIGINT NOT NULL,
parse_status VARCHAR NOT NULL,
parse_error_code VARCHAR,
block_count INTEGER NOT NULL,
metadata_json VARCHAR NOT NULL
);
CREATE INDEX IF NOT EXISTS gb32960_frame_vin_time_idx
ON gb32960_frame_index(vin, event_time_ms);
CREATE INDEX IF NOT EXISTS gb32960_frame_raw_uri_idx
ON gb32960_frame_index(raw_archive_uri);
CREATE INDEX IF NOT EXISTS gb32960_frame_partition_time_idx
ON gb32960_frame_index(partition_date, event_time_ms);
```
`gb32960_telemetry_point`
```sql
CREATE TABLE IF NOT EXISTS gb32960_telemetry_point (
event_id VARCHAR NOT NULL,
vin VARCHAR NOT NULL,
event_time TIMESTAMP NOT NULL,
event_time_ms BIGINT NOT NULL,
ingest_time TIMESTAMP NOT NULL,
platform_account VARCHAR,
vendor_profile VARCHAR,
field_key VARCHAR NOT NULL,
value_type VARCHAR NOT NULL,
value_text VARCHAR NOT NULL,
value_num DOUBLE,
unit VARCHAR,
quality VARCHAR NOT NULL,
raw_archive_uri VARCHAR NOT NULL,
PRIMARY KEY(event_id, field_key)
);
CREATE INDEX IF NOT EXISTS gb32960_point_vin_field_time_idx
ON gb32960_telemetry_point(vin, field_key, event_time_ms);
```
`gb32960_alarm_frame`
```sql
CREATE TABLE IF NOT EXISTS gb32960_alarm_frame (
event_id VARCHAR PRIMARY KEY,
vin VARCHAR NOT NULL,
event_time TIMESTAMP NOT NULL,
event_time_ms BIGINT NOT NULL,
level VARCHAR NOT NULL,
active_bits_json VARCHAR NOT NULL,
fault_codes_json VARCHAR NOT NULL,
raw_archive_uri VARCHAR NOT NULL
);
CREATE INDEX IF NOT EXISTS gb32960_alarm_vin_time_idx
ON gb32960_alarm_frame(vin, event_time_ms);
```
### RAW Archive
RAW archive remains file/object based:
```text
archive://yyyy/MM/dd/GB32960/{vin}/{event_id}.bin
```
The archive writer verifies:
- stored byte length equals envelope `raw_archive.size_bytes`;
- checksum matches;
- URI is unique for the deterministic event id.
Frame replay path:
1. query `gb32960_frame_index` by `rawArchiveUri`;
2. read RAW bytes from archive;
3. resolve `platformAccount/vendorProfile` from index metadata;
4. decode using current parser;
5. return decoded frame plus original indexed parse diagnostics.
## Query APIs
Keep existing endpoints but back them by the new DuckDB tables:
- `GET /api/event-history/gb32960/frames`
- full frame query by VIN/time.
- `GET /api/event-history/gb32960/frame`
- one RAW frame replay by `rawArchiveUri`.
- `GET /api/event-history/gb32960/telemetry-snapshots`
- compact telemetry snapshot by VIN/time.
- `GET /api/event-history/gb32960/telemetry-fields`
- field projection optimized by `gb32960_telemetry_point`.
- `GET /api/event-history/gb32960/diagnostics`
- parse status and missing-profile investigation by VIN/time.
Time semantics:
- `eventTime` is business query time.
- `ingestTime` is kept for operational replay, late-arrival detection, and Kafka lag investigation.
- Dates without timezone are interpreted in `Asia/Shanghai`.
## MySQL Analytics Design
Credentials are supplied only by environment variables:
```text
MYSQL_HOST
MYSQL_PORT
MYSQL_DATABASE
MYSQL_USERNAME
MYSQL_PASSWORD
```
`vehicle_daily_stat`
```sql
CREATE TABLE IF NOT EXISTS vehicle_daily_stat (
vin VARCHAR(32) NOT NULL,
stat_date DATE NOT NULL,
platform_account VARCHAR(64),
total_mileage_km DECIMAL(14,3),
daily_mileage_km DECIMAL(14,3),
daily_power_kwh DECIMAL(14,3),
daily_hydrogen_kg DECIMAL(14,4),
daily_hydrogen_kg_per_100km DECIMAL(14,4),
first_event_time DATETIME(3),
last_event_time DATETIME(3),
first_total_mileage_km DECIMAL(14,3),
last_total_mileage_km DECIMAL(14,3),
last_soc_percent DECIMAL(8,3),
sample_count BIGINT NOT NULL DEFAULT 0,
late_sample_count BIGINT NOT NULL DEFAULT 0,
calculation_quality VARCHAR(32) NOT NULL,
updated_at DATETIME(3) NOT NULL,
PRIMARY KEY (vin, stat_date)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4;
```
`vehicle_alarm_timeline`
```sql
CREATE TABLE IF NOT EXISTS vehicle_alarm_timeline (
alarm_id VARCHAR(128) NOT NULL,
vin VARCHAR(32) NOT NULL,
alarm_key VARCHAR(128) NOT NULL,
level VARCHAR(32) NOT NULL,
start_time DATETIME(3) NOT NULL,
end_time DATETIME(3),
last_seen_time DATETIME(3) NOT NULL,
start_raw_archive_uri VARCHAR(512),
last_raw_archive_uri VARCHAR(512),
status VARCHAR(16) NOT NULL,
details_json JSON,
updated_at DATETIME(3) NOT NULL,
PRIMARY KEY (alarm_id),
KEY vehicle_alarm_vin_time_idx (vin, start_time),
KEY vehicle_alarm_open_idx (vin, status, alarm_key)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4;
```
`vehicle_stat_event_dedup`
```sql
CREATE TABLE IF NOT EXISTS vehicle_stat_event_dedup (
event_id VARCHAR(128) NOT NULL,
consumer_group VARCHAR(128) NOT NULL,
processed_at DATETIME(3) NOT NULL,
PRIMARY KEY (event_id, consumer_group)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4;
```
Writes use `INSERT ... ON DUPLICATE KEY UPDATE` so replayed Kafka messages update the same daily row instead of creating duplicates.
## Daily Metric Algorithms
All daily statistics are keyed by `eventTime` in `Asia/Shanghai`.
### Total Mileage
Use the latest valid `totalMileageKm` seen for the day.
Validation:
- reject negative values;
- reject impossible jumps based on configurable max speed and elapsed time;
- keep the sample as diagnostic if rejected.
### Daily Mileage
Preferred:
```text
dailyMileage = today.lastTotalMileage - previousDay.lastTotalMileage
```
Fallback:
```text
dailyMileage = today.lastTotalMileage - today.firstTotalMileage
```
Quality:
- `PREVIOUS_DAY_BASELINE` when previous day baseline exists;
- `INTRADAY_DELTA` when only same-day baseline exists;
- `INSUFFICIENT_DATA` when neither is reliable.
### Daily Power Consumption
Preferred:
- vendor cumulative power-consumption field if available.
Fallback:
```text
deltaKwh = sum(max(0, totalVoltageV * totalCurrentA) * deltaSeconds / 3600000)
```
The sign convention is configurable per vendor profile because GB32960 vehicle current may be positive or negative depending on charging/discharging representation.
### Daily Hydrogen Consumption
Use a tiered market-practical algorithm instead of a single fixed formula:
1. `DIRECT_COUNTER`: vendor cumulative hydrogen consumption or remaining hydrogen mass delta.
2. `PVT_MASS_DELTA`: pressure/temperature/tank-volume mass estimate when storage data is available.
3. `FUEL_CELL_ENERGY_MODEL`: integrate fuel-cell electric output and divide by calibrated efficiency.
4. `UNKNOWN`: keep null when required signals are missing.
The PVT approach follows the industry pattern behind compressed hydrogen consumption tests, where pressure, temperature, and tank volume can be used to estimate hydrogen mass. The implementation must expose calibration constants per vehicle model/profile:
```json
{
"vinPrefix": "LNXNEGRR",
"tankVolumeLiter": 140,
"hydrogenModel": "PVT_MASS_DELTA",
"fuelCellEfficiency": 0.52,
"currentDischargeSign": "POSITIVE"
}
```
### Daily Hydrogen Consumption Rate
```text
dailyHydrogenKgPer100km = dailyHydrogenKg / dailyMileageKm * 100
```
If mileage is missing or less than the configured minimum distance, keep the rate null and set quality to `INSUFFICIENT_DISTANCE`.
### Alarm Timeline
For each telemetry frame:
- derive active alarm keys from alarm bits and fault code arrays;
- open a timeline row when a key first appears;
- update `last_seen_time` while it remains active;
- close open alarms when the key disappears after a configurable grace window;
- keep start/end raw archive URIs for replay.
## Reliability and Observability
Required metrics:
- TCP active connections;
- frames received per command;
- decode failures by code;
- missing platform login rejections;
- vendor profile resolution source;
- Kafka send latency and failures;
- history write batch size, latency, failure count;
- DuckDB query latency by endpoint;
- MySQL upsert latency and failure count;
- consumer lag by group/topic/partition;
- late sample count by VIN/date.
Required logs:
- one structured line for every dropped or rejected valid frame;
- no per-frame INFO spam for successful reports after first frame per VIN/profile unless debug is enabled;
- include `vin`, `platformAccount`, `command`, `eventId`, `reasonCode`, and `peer`.
## Migration Plan
Phase 1: History foundation
- Replace Parquet rewrite store with DuckDB hot tables and batch writer.
- Keep RAW bytes archive.
- Move existing history endpoints onto the new store.
- Add diagnostic endpoint for missing VIN/profile cases.
Phase 2: Ingest association and diagnostics
- Add local `vin-platform-profile.jsonl` loader.
- Use resolved profile before private block parsing.
- Add structured parse/reject diagnostics to Kafka metadata and logs.
- Add tests for platform login, mapping fallback, and partial private parsing.
Phase 3: MySQL analytics
- Add MySQL repository and migrations.
- Implement daily stat accumulator with idempotent upsert.
- Implement alarm timeline open/update/close logic.
- Add integration tests using Testcontainers or a local MySQL profile.
Phase 4: Production hardening
- Add backfill/replay commands from DuckDB RAW archive to analytics.
- Add retention/export from DuckDB to Parquet for cold storage.
- Add dashboards and operational runbook.
## Acceptance Criteria
- A valid GB32960 frame received on port 32960 appears in Kafka with RAW bytes or RAW URI.
- A valid frame is queryable by `vin + eventTime` from history.
- A returned frame can be replayed from `rawArchiveUri` and decoded with the stored profile.
- A VIN with only local mapping, and no current platform login, can still use the correct private parser profile.
- A private block parse failure does not drop the frame.
- Daily MySQL rows update in real time for mileage, total mileage, power consumption, hydrogen consumption, hydrogen rate, and sample metadata.
- Alarm timeline rows preserve full alarm start, update, and close history.
- Replaying the same Kafka event does not duplicate daily stats or alarm rows.
- No secret appears in committed files.
## References
- DuckDB Appender: https://duckdb.org/docs/current/data/appender
- DuckDB INSERT performance guidance: https://duckdb.org/docs/current/data/insert
- MySQL `INSERT ... ON DUPLICATE KEY UPDATE`: https://dev.mysql.com/doc/refman/9.7/en/insert-on-duplicate.html
- SAE J2572 overview: https://h2tools.org/fuel-cell-codes-and-standards/sae-j2572-measuring-exhaust-emissions-energy-consumption-and-range

View File

@@ -0,0 +1,66 @@
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0">
<modelVersion>4.0.0</modelVersion>
<parent>
<groupId>com.lingniu.ingest</groupId>
<artifactId>lingniu-vehicle-ingest</artifactId>
<version>0.1.0-SNAPSHOT</version>
<relativePath>../../../pom.xml</relativePath>
</parent>
<artifactId>gb32960-ingest-app</artifactId>
<name>gb32960-ingest-app</name>
<description>GB32960 protocol ingress runtime: TCP decode, auth, Kafka production, and ACK.</description>
<dependencies>
<dependency>
<groupId>com.lingniu.ingest</groupId>
<artifactId>ingest-core</artifactId>
</dependency>
<dependency>
<groupId>com.lingniu.ingest</groupId>
<artifactId>session-core</artifactId>
</dependency>
<dependency>
<groupId>com.lingniu.ingest</groupId>
<artifactId>observability</artifactId>
</dependency>
<dependency>
<groupId>com.lingniu.ingest</groupId>
<artifactId>protocol-gb32960</artifactId>
</dependency>
<dependency>
<groupId>com.lingniu.ingest</groupId>
<artifactId>sink-mq</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-web</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-test</artifactId>
<scope>test</scope>
</dependency>
</dependencies>
<build>
<finalName>gb32960-ingest-app</finalName>
<plugins>
<plugin>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-maven-plugin</artifactId>
<configuration>
<jvmArguments>--sun-misc-unsafe-memory-access=allow</jvmArguments>
</configuration>
<executions>
<execution>
<goals>
<goal>repackage</goal>
</goals>
</execution>
</executions>
</plugin>
</plugins>
</build>
</project>

View File

@@ -0,0 +1,21 @@
package com.lingniu.ingest.gb32960app;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;
@SpringBootApplication(scanBasePackages = "com.lingniu.ingest")
public class Gb32960IngestApplication {
private static final Logger log = LoggerFactory.getLogger(Gb32960IngestApplication.class);
public static void main(String[] args) {
try {
SpringApplication.run(Gb32960IngestApplication.class, args);
} catch (Throwable t) {
log.error("GB32960 ingest application failed to start, forcing JVM exit", t);
System.exit(1);
}
}
}

View File

@@ -0,0 +1,110 @@
spring:
application:
name: gb32960-ingest-app
threads:
virtual:
enabled: true
server:
port: ${HTTP_PORT:20100}
lingniu:
ingest:
gb32960:
enabled: true
server:
enabled: true
port: ${GB32960_PORT:32960}
boss-threads: ${GB32960_BOSS_THREADS:1}
worker-threads: ${GB32960_WORKER_THREADS:0}
auth:
enabled: ${GB32960_AUTH_ENABLED:false}
case-sensitive: false
whitelist: []
platforms:
- username: ${GB32960_PLATFORM_USER_HYUNDAI:Hyundai}
password: ${GB32960_PLATFORM_PWD_HYUNDAI:f2e3445d7cda409fb4f278f6fb890734}
allowed-ips:
- ${GB32960_PLATFORM_IP_HYUNDAI:8.134.95.166}
description: 外部下级平台 - Hyundai
tls:
enabled: ${GB32960_TLS_ENABLED:false}
cert-path: ${GB32960_TLS_CERT:}
key-path: ${GB32960_TLS_KEY:}
trust-cert-path: ${GB32960_TLS_CA:}
require-client-auth: true
vendor-extensions:
- name: guangdong-fc
match:
platform-accounts:
- Hyundai
vin-prefixes: []
vins: []
parse:
lenient-block-failure: true
diagnostics:
max-logged-frame-keys-per-channel: ${GB32960_DIAGNOSTICS_MAX_LOGGED_FRAME_KEYS_PER_CHANNEL:4096}
pipeline:
disruptor:
ring-buffer-size: ${PIPELINE_RING_BUFFER_SIZE:131072}
wait-strategy: ${PIPELINE_WAIT_STRATEGY:yielding}
producer-type: multi
dedup:
enabled: true
cache-size: 200000
ttl-seconds: 600
rate-limit:
per-vin-qps: 50
session:
store: ${SESSION_STORE:memory}
ttl: ${SESSION_TTL:30m}
identity:
store: ${VEHICLE_IDENTITY_STORE:file}
file:
path: ${VEHICLE_IDENTITY_FILE:./data/vehicle-identity.jsonl}
sink:
mq:
enabled: ${KAFKA_ENABLED:true}
type: kafka
bootstrap-servers: ${KAFKA_BROKERS:114.55.58.251:9092}
compression-type: zstd
linger-ms: 20
batch-size: 65536
acks: all
enable-idempotence: true
node-id: ${KAFKA_NODE_ID:gb32960-ingest-local}
topics:
realtime: ${KAFKA_TOPIC_GB32960_EVENT:vehicle.event.gb32960.v1}
location: ${KAFKA_TOPIC_GB32960_EVENT:vehicle.event.gb32960.v1}
alarm: ${KAFKA_TOPIC_GB32960_EVENT:vehicle.event.gb32960.v1}
session: ${KAFKA_TOPIC_GB32960_EVENT:vehicle.event.gb32960.v1}
media-meta: ${KAFKA_TOPIC_MEDIA_META:vehicle.media.meta.v1}
raw-archive: ${KAFKA_TOPIC_GB32960_RAW:vehicle.raw.gb32960.v1}
dlq: ${KAFKA_TOPIC_GB32960_DLQ:vehicle.dlq.gb32960.v1}
consumer:
enabled: false
event-file-store:
enabled: false
event-history:
enabled: false
vehicle-state:
enabled: false
vehicle-stat:
enabled: false
management:
endpoints:
web:
exposure:
include: health,info,metrics,prometheus,env
health:
redis:
enabled: ${MANAGEMENT_HEALTH_REDIS_ENABLED:false}
metrics:
tags:
application: gb32960-ingest-app
logging:
level:
root: INFO
com.lingniu.ingest: INFO

View File

@@ -0,0 +1,68 @@
package com.lingniu.ingest.gb32960app;
import com.lingniu.ingest.core.config.IngestCoreAutoConfiguration;
import com.lingniu.ingest.protocol.gb32960.codec.Gb32960MessageDecoder;
import com.lingniu.ingest.protocol.gb32960.config.Gb32960AutoConfiguration;
import com.lingniu.ingest.protocol.gb32960.inbound.Gb32960NettyServer;
import com.lingniu.ingest.session.config.SessionCoreAutoConfiguration;
import com.lingniu.ingest.sink.mq.KafkaEventSink;
import com.lingniu.ingest.sink.mq.KafkaEnvelopeDeadLetterSink;
import com.lingniu.ingest.sink.mq.SinkMqAutoConfiguration;
import org.apache.kafka.clients.producer.KafkaProducer;
import org.junit.jupiter.api.Test;
import org.springframework.boot.autoconfigure.AutoConfigurations;
import org.springframework.boot.test.context.runner.ApplicationContextRunner;
import org.springframework.context.ApplicationContext;
import org.springframework.util.ClassUtils;
import static org.assertj.core.api.Assertions.assertThat;
import static org.mockito.Mockito.mock;
class Gb32960IngestAppCompositionTest {
private final ApplicationContextRunner contextRunner = new ApplicationContextRunner()
.withConfiguration(AutoConfigurations.of(
IngestCoreAutoConfiguration.class,
SessionCoreAutoConfiguration.class,
SinkMqAutoConfiguration.class,
Gb32960AutoConfiguration.class))
.withAllowBeanDefinitionOverriding(true)
.withBean("kafkaProducer", KafkaProducer.class, Gb32960IngestAppCompositionTest::kafkaProducer)
.withPropertyValues(
"lingniu.ingest.gb32960.enabled=true",
"lingniu.ingest.gb32960.server.enabled=true",
"lingniu.ingest.gb32960.port=0",
"lingniu.ingest.session.store=memory",
"lingniu.ingest.sink.mq.enabled=true",
"lingniu.ingest.sink.mq.type=kafka",
"lingniu.ingest.sink.mq.bootstrap-servers=localhost:9092",
"lingniu.ingest.sink.mq.consumer.enabled=false",
"lingniu.ingest.event-file-store.enabled=false",
"lingniu.ingest.event-history.enabled=false",
"lingniu.ingest.vehicle-state.enabled=false",
"lingniu.ingest.vehicle-stat.enabled=false");
@Test
void createsGb32960ListenerWithoutHistoryStorageBoundaries() {
contextRunner.run(context -> {
assertThat(context).hasSingleBean(Gb32960MessageDecoder.class);
assertThat(context).hasSingleBean(Gb32960NettyServer.class);
assertThat(context).hasSingleBean(KafkaEventSink.class);
assertThat(context).hasSingleBean(KafkaEnvelopeDeadLetterSink.class);
assertTypeNotPresent(context, "com.lingniu.ingest.sink.archive.ArchiveStore");
assertTypeNotPresent(context, "com.lingniu.ingest.eventfilestore.EventFileStore");
});
}
@SuppressWarnings("unchecked")
private static KafkaProducer<String, byte[]> kafkaProducer() {
return mock(KafkaProducer.class);
}
private static void assertTypeNotPresent(ApplicationContext context, String className) {
assertThat(ClassUtils.isPresent(className, context.getClassLoader()))
.as("%s must stay outside the GB32960 ingest runtime", className)
.isFalse();
}
}

View File

@@ -0,0 +1,39 @@
package com.lingniu.ingest.gb32960app;
import org.junit.jupiter.api.Test;
import org.springframework.beans.factory.config.YamlPropertiesFactoryBean;
import org.springframework.core.io.ClassPathResource;
import java.util.Properties;
import static org.assertj.core.api.Assertions.assertThat;
class Gb32960IngestAppDefaultsTest {
@Test
void applicationDefaultsKeepGb32960IngestAsTcpKafkaProducerOnly() {
Properties properties = applicationProperties();
assertThat(properties)
.containsEntry("spring.application.name", "gb32960-ingest-app")
.containsEntry("lingniu.ingest.gb32960.enabled", true)
.containsEntry("lingniu.ingest.gb32960.server.enabled", true)
.containsEntry("lingniu.ingest.gb32960.port", "${GB32960_PORT:32960}")
.containsEntry("lingniu.ingest.sink.mq.enabled", "${KAFKA_ENABLED:true}")
.containsEntry("lingniu.ingest.sink.mq.consumer.enabled", false)
.containsEntry("lingniu.ingest.event-file-store.enabled", false)
.containsEntry("lingniu.ingest.event-history.enabled", false)
.containsEntry("lingniu.ingest.vehicle-state.enabled", false)
.containsEntry("lingniu.ingest.vehicle-stat.enabled", false);
}
private static Properties applicationProperties() {
YamlPropertiesFactoryBean factory = new YamlPropertiesFactoryBean();
factory.setResources(new ClassPathResource("application.yml"));
Properties properties = factory.getObject();
assertThat(properties).isNotNull();
return properties;
}
}

View File

@@ -0,0 +1,66 @@
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0">
<modelVersion>4.0.0</modelVersion>
<parent>
<groupId>com.lingniu.ingest</groupId>
<artifactId>lingniu-vehicle-ingest</artifactId>
<version>0.1.0-SNAPSHOT</version>
<relativePath>../../../pom.xml</relativePath>
</parent>
<artifactId>vehicle-analytics-app</artifactId>
<name>vehicle-analytics-app</name>
<description>Vehicle state, daily statistics, alarms, and analytics runtime.</description>
<dependencies>
<dependency>
<groupId>com.lingniu.ingest</groupId>
<artifactId>ingest-api</artifactId>
</dependency>
<dependency>
<groupId>com.lingniu.ingest</groupId>
<artifactId>observability</artifactId>
</dependency>
<dependency>
<groupId>com.lingniu.ingest</groupId>
<artifactId>sink-mq</artifactId>
</dependency>
<dependency>
<groupId>com.lingniu.ingest</groupId>
<artifactId>vehicle-state-service</artifactId>
</dependency>
<dependency>
<groupId>com.lingniu.ingest</groupId>
<artifactId>vehicle-stat-service</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-web</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-test</artifactId>
<scope>test</scope>
</dependency>
</dependencies>
<build>
<finalName>vehicle-analytics-app</finalName>
<plugins>
<plugin>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-maven-plugin</artifactId>
<configuration>
<jvmArguments>--sun-misc-unsafe-memory-access=allow</jvmArguments>
</configuration>
<executions>
<execution>
<goals>
<goal>repackage</goal>
</goals>
</execution>
</executions>
</plugin>
</plugins>
</build>
</project>

View File

@@ -0,0 +1,21 @@
package com.lingniu.ingest.analyticsapp;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;
@SpringBootApplication(scanBasePackages = "com.lingniu.ingest")
public class VehicleAnalyticsApplication {
private static final Logger log = LoggerFactory.getLogger(VehicleAnalyticsApplication.class);
public static void main(String[] args) {
try {
SpringApplication.run(VehicleAnalyticsApplication.class, args);
} catch (Throwable t) {
log.error("Vehicle analytics application failed to start, forcing JVM exit", t);
System.exit(1);
}
}
}

View File

@@ -0,0 +1,62 @@
spring:
application:
name: vehicle-analytics-app
threads:
virtual:
enabled: true
server:
port: ${HTTP_PORT:20300}
lingniu:
ingest:
gb32960:
enabled: false
sink:
mq:
enabled: ${KAFKA_ENABLED:true}
type: kafka
bootstrap-servers: ${KAFKA_BROKERS:114.55.58.251:9092}
topics:
realtime: ${KAFKA_TOPIC_GB32960_EVENT:vehicle.event.gb32960.v1}
dlq: ${KAFKA_TOPIC_GB32960_DLQ:vehicle.dlq.gb32960.v1}
consumer:
enabled: ${KAFKA_CONSUMER_ENABLED:true}
client-id-prefix: ${KAFKA_CONSUMER_CLIENT_ID_PREFIX:vehicle-analytics}
auto-offset-reset: ${KAFKA_CONSUMER_AUTO_OFFSET_RESET:earliest}
max-poll-records: ${KAFKA_CONSUMER_MAX_POLL_RECORDS:500}
bindings:
vehicleStateEnvelopeConsumerProcessor:
enabled: ${VEHICLE_STATE_ENABLED:false}
group-id: ${KAFKA_GROUP_STATE:vehicle-state}
topics:
- ${KAFKA_TOPIC_GB32960_EVENT:vehicle.event.gb32960.v1}
vehicleStatEnvelopeConsumerProcessor:
enabled: ${VEHICLE_STAT_ENABLED:true}
group-id: ${KAFKA_GROUP_STAT:vehicle-stat}
topics:
- ${KAFKA_TOPIC_GB32960_EVENT:vehicle.event.gb32960.v1}
archive:
enabled: false
event-file-store:
enabled: false
event-history:
enabled: false
vehicle-state:
enabled: ${VEHICLE_STATE_ENABLED:false}
vehicle-stat:
enabled: ${VEHICLE_STAT_ENABLED:true}
file-path: ${VEHICLE_STAT_FILE_PATH:./target/vehicle-stat/}
zone-id: ${VEHICLE_STAT_ZONE_ID:Asia/Shanghai}
management:
endpoints:
web:
exposure:
include: health,info,metrics,prometheus,env
health:
redis:
enabled: ${MANAGEMENT_HEALTH_REDIS_ENABLED:false}
metrics:
tags:
application: vehicle-analytics-app

View File

@@ -0,0 +1,85 @@
package com.lingniu.ingest.analyticsapp;
import com.lingniu.ingest.api.consumer.EnvelopeConsumerProcessor;
import com.lingniu.ingest.sink.mq.KafkaEnvelopeDeadLetterSink;
import com.lingniu.ingest.sink.mq.KafkaEventSink;
import com.lingniu.ingest.sink.mq.SinkMqAutoConfiguration;
import com.lingniu.ingest.vehiclestate.VehicleStateEnvelopeIngestor;
import com.lingniu.ingest.vehiclestate.config.VehicleStateAutoConfiguration;
import com.lingniu.ingest.vehiclestat.DailyVehicleStatService;
import com.lingniu.ingest.vehiclestat.FileVehicleStatRepository;
import com.lingniu.ingest.vehiclestat.VehicleStatController;
import com.lingniu.ingest.vehiclestat.VehicleStatEnvelopeIngestor;
import com.lingniu.ingest.vehiclestat.VehicleStatEventProcessor;
import com.lingniu.ingest.vehiclestat.VehicleStatEventSink;
import com.lingniu.ingest.vehiclestat.VehicleStatRepository;
import com.lingniu.ingest.vehiclestat.config.VehicleStatAutoConfiguration;
import org.apache.kafka.clients.producer.KafkaProducer;
import org.junit.jupiter.api.Test;
import org.junit.jupiter.api.io.TempDir;
import org.springframework.boot.autoconfigure.AutoConfigurations;
import org.springframework.boot.test.context.runner.ApplicationContextRunner;
import org.springframework.context.ApplicationContext;
import org.springframework.util.ClassUtils;
import java.nio.file.Path;
import static org.assertj.core.api.Assertions.assertThat;
import static org.mockito.Mockito.mock;
class VehicleAnalyticsAppCompositionTest {
@TempDir
Path tempDir;
@Test
void createsStatsBeansWithoutProtocolListenerOrEventFileStore() {
new ApplicationContextRunner()
.withConfiguration(AutoConfigurations.of(
SinkMqAutoConfiguration.class,
VehicleStateAutoConfiguration.class,
VehicleStatAutoConfiguration.class))
.withAllowBeanDefinitionOverriding(true)
.withBean("kafkaProducer", KafkaProducer.class, VehicleAnalyticsAppCompositionTest::kafkaProducer)
.withPropertyValues(
"lingniu.ingest.sink.mq.enabled=true",
"lingniu.ingest.sink.mq.type=kafka",
"lingniu.ingest.sink.mq.bootstrap-servers=localhost:9092",
"lingniu.ingest.sink.mq.consumer.enabled=false",
"lingniu.ingest.vehicle-state.enabled=false",
"lingniu.ingest.vehicle-stat.enabled=true",
"lingniu.ingest.vehicle-stat.file-path=" + tempDir.resolve("vehicle-stat"),
"lingniu.ingest.event-file-store.enabled=false",
"lingniu.ingest.event-history.enabled=false",
"lingniu.ingest.gb32960.enabled=false")
.run(context -> {
assertThat(context).hasSingleBean(VehicleStatRepository.class);
assertThat(context).hasSingleBean(FileVehicleStatRepository.class);
assertThat(context).hasSingleBean(DailyVehicleStatService.class);
assertThat(context).hasSingleBean(VehicleStatEventProcessor.class);
assertThat(context).hasSingleBean(VehicleStatEnvelopeIngestor.class);
assertThat(context).hasSingleBean(VehicleStatEventSink.class);
assertThat(context).hasSingleBean(VehicleStatController.class);
assertThat(context).hasSingleBean(KafkaEventSink.class);
assertThat(context).hasSingleBean(KafkaEnvelopeDeadLetterSink.class);
assertThat(context).hasBean("vehicleStatEnvelopeConsumerProcessor");
assertThat(context.getBean("vehicleStatEnvelopeConsumerProcessor"))
.isInstanceOf(EnvelopeConsumerProcessor.class);
assertThat(context).doesNotHaveBean(VehicleStateEnvelopeIngestor.class);
assertTypeNotPresent(context, "com.lingniu.ingest.protocol.gb32960.inbound.Gb32960NettyServer");
assertTypeNotPresent(context, "com.lingniu.ingest.eventfilestore.EventFileStore");
});
}
@SuppressWarnings("unchecked")
private static KafkaProducer<String, byte[]> kafkaProducer() {
return mock(KafkaProducer.class);
}
private static void assertTypeNotPresent(ApplicationContext context, String className) {
assertThat(ClassUtils.isPresent(className, context.getClassLoader()))
.as("%s must stay outside the vehicle analytics runtime", className)
.isFalse();
}
}

View File

@@ -0,0 +1,49 @@
package com.lingniu.ingest.analyticsapp;
import org.junit.jupiter.api.Test;
import org.springframework.beans.factory.config.YamlPropertiesFactoryBean;
import org.springframework.core.io.ClassPathResource;
import java.util.Properties;
import static org.assertj.core.api.Assertions.assertThat;
class VehicleAnalyticsAppDefaultsTest {
@Test
void applicationDefaultsKeepAnalyticsAsStatConsumerRuntime() {
Properties properties = applicationProperties();
assertThat(properties)
.containsEntry("spring.application.name", "vehicle-analytics-app")
.containsEntry("lingniu.ingest.gb32960.enabled", false)
.containsEntry("lingniu.ingest.sink.archive.enabled", false)
.containsEntry("lingniu.ingest.event-file-store.enabled", false)
.containsEntry("lingniu.ingest.event-history.enabled", false)
.containsEntry("lingniu.ingest.vehicle-stat.enabled", "${VEHICLE_STAT_ENABLED:true}")
.containsEntry("lingniu.ingest.vehicle-state.enabled", "${VEHICLE_STATE_ENABLED:false}")
.containsEntry("lingniu.ingest.sink.mq.consumer.enabled", "${KAFKA_CONSUMER_ENABLED:true}")
.containsEntry(
"lingniu.ingest.sink.mq.consumer.bindings.vehicleStatEnvelopeConsumerProcessor.enabled",
"${VEHICLE_STAT_ENABLED:true}")
.containsEntry(
"lingniu.ingest.sink.mq.consumer.bindings.vehicleStateEnvelopeConsumerProcessor.enabled",
"${VEHICLE_STATE_ENABLED:false}")
.containsEntry(
"lingniu.ingest.sink.mq.consumer.bindings.vehicleStatEnvelopeConsumerProcessor.group-id",
"${KAFKA_GROUP_STAT:vehicle-stat}")
.containsEntry(
"lingniu.ingest.sink.mq.consumer.bindings.vehicleStateEnvelopeConsumerProcessor.group-id",
"${KAFKA_GROUP_STATE:vehicle-state}");
}
private static Properties applicationProperties() {
YamlPropertiesFactoryBean factory = new YamlPropertiesFactoryBean();
factory.setResources(new ClassPathResource("application.yml"));
Properties properties = factory.getObject();
assertThat(properties).isNotNull();
return properties;
}
}

View File

@@ -0,0 +1,74 @@
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0">
<modelVersion>4.0.0</modelVersion>
<parent>
<groupId>com.lingniu.ingest</groupId>
<artifactId>lingniu-vehicle-ingest</artifactId>
<version>0.1.0-SNAPSHOT</version>
<relativePath>../../../pom.xml</relativePath>
</parent>
<artifactId>vehicle-history-app</artifactId>
<name>vehicle-history-app</name>
<description>Vehicle raw archive, event history, and GB32960 frame query runtime.</description>
<dependencies>
<dependency>
<groupId>com.lingniu.ingest</groupId>
<artifactId>ingest-api</artifactId>
</dependency>
<dependency>
<groupId>com.lingniu.ingest</groupId>
<artifactId>observability</artifactId>
</dependency>
<dependency>
<groupId>com.lingniu.ingest</groupId>
<artifactId>sink-mq</artifactId>
</dependency>
<dependency>
<groupId>com.lingniu.ingest</groupId>
<artifactId>sink-archive</artifactId>
</dependency>
<dependency>
<groupId>com.lingniu.ingest</groupId>
<artifactId>event-file-store</artifactId>
</dependency>
<dependency>
<groupId>com.lingniu.ingest</groupId>
<artifactId>event-history-service</artifactId>
</dependency>
<dependency>
<groupId>com.lingniu.ingest</groupId>
<artifactId>protocol-gb32960</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-web</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-test</artifactId>
<scope>test</scope>
</dependency>
</dependencies>
<build>
<finalName>vehicle-history-app</finalName>
<plugins>
<plugin>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-maven-plugin</artifactId>
<configuration>
<jvmArguments>--sun-misc-unsafe-memory-access=allow</jvmArguments>
</configuration>
<executions>
<execution>
<goals>
<goal>repackage</goal>
</goals>
</execution>
</executions>
</plugin>
</plugins>
</build>
</project>

View File

@@ -0,0 +1,21 @@
package com.lingniu.ingest.historyapp;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;
@SpringBootApplication(scanBasePackages = "com.lingniu.ingest")
public class VehicleHistoryApplication {
private static final Logger log = LoggerFactory.getLogger(VehicleHistoryApplication.class);
public static void main(String[] args) {
try {
SpringApplication.run(VehicleHistoryApplication.class, args);
} catch (Throwable t) {
log.error("Vehicle history application failed to start, forcing JVM exit", t);
System.exit(1);
}
}
}

View File

@@ -0,0 +1,65 @@
spring:
application:
name: vehicle-history-app
threads:
virtual:
enabled: true
server:
port: ${HTTP_PORT:20200}
lingniu:
ingest:
gb32960:
enabled: true
server:
enabled: false
sink:
mq:
enabled: ${KAFKA_ENABLED:true}
type: kafka
bootstrap-servers: ${KAFKA_BROKERS:114.55.58.251:9092}
topics:
realtime: ${KAFKA_TOPIC_GB32960_EVENT:vehicle.event.gb32960.v1}
raw-archive: ${KAFKA_TOPIC_GB32960_RAW:vehicle.raw.gb32960.v1}
dlq: ${KAFKA_TOPIC_GB32960_DLQ:vehicle.dlq.gb32960.v1}
consumer:
enabled: ${KAFKA_CONSUMER_ENABLED:true}
client-id-prefix: ${KAFKA_CONSUMER_CLIENT_ID_PREFIX:vehicle-history}
auto-offset-reset: ${KAFKA_CONSUMER_AUTO_OFFSET_RESET:earliest}
max-poll-records: ${KAFKA_CONSUMER_MAX_POLL_RECORDS:500}
bindings:
eventHistoryEnvelopeConsumerProcessor:
enabled: true
group-id: ${KAFKA_GROUP_HISTORY:vehicle-history}
topics:
- ${KAFKA_TOPIC_GB32960_EVENT:vehicle.event.gb32960.v1}
- ${KAFKA_TOPIC_GB32960_RAW:vehicle.raw.gb32960.v1}
archive:
enabled: ${SINK_ARCHIVE_ENABLED:true}
type: local
path: ${SINK_ARCHIVE_PATH:./archive/}
event-file-store:
enabled: ${EVENT_FILE_STORE_ENABLED:true}
path: ${EVENT_FILE_STORE_PATH:./target/event-store/}
zone-id: ${EVENT_FILE_STORE_ZONE_ID:Asia/Shanghai}
batch-size: ${EVENT_FILE_STORE_BATCH_SIZE:500}
flush-interval-millis: ${EVENT_FILE_STORE_FLUSH_INTERVAL_MILLIS:1000}
event-history:
enabled: true
vehicle-state:
enabled: false
vehicle-stat:
enabled: false
management:
endpoints:
web:
exposure:
include: health,info,metrics,prometheus,env
health:
redis:
enabled: ${MANAGEMENT_HEALTH_REDIS_ENABLED:false}
metrics:
tags:
application: vehicle-history-app

View File

@@ -0,0 +1,83 @@
package com.lingniu.ingest.historyapp;
import com.lingniu.ingest.eventfilestore.EventFileStore;
import com.lingniu.ingest.eventfilestore.EventFileStoreSink;
import com.lingniu.ingest.eventfilestore.config.EventFileStoreAutoConfiguration;
import com.lingniu.ingest.eventhistory.EventHistoryController;
import com.lingniu.ingest.eventhistory.EventHistoryEnvelopeIngestor;
import com.lingniu.ingest.eventhistory.Gb32960DecodedFrameService;
import com.lingniu.ingest.eventhistory.Gb32960FrameController;
import com.lingniu.ingest.eventhistory.config.EventHistoryAutoConfiguration;
import com.lingniu.ingest.protocol.gb32960.codec.Gb32960MessageDecoder;
import com.lingniu.ingest.protocol.gb32960.config.Gb32960AutoConfiguration;
import com.lingniu.ingest.protocol.gb32960.inbound.Gb32960NettyServer;
import com.lingniu.ingest.sink.archive.ArchiveStore;
import com.lingniu.ingest.sink.archive.RawArchiveEventSink;
import com.lingniu.ingest.sink.archive.config.SinkArchiveAutoConfiguration;
import com.lingniu.ingest.sink.mq.KafkaEnvelopeDeadLetterSink;
import com.lingniu.ingest.sink.mq.KafkaEventSink;
import com.lingniu.ingest.sink.mq.SinkMqAutoConfiguration;
import org.apache.kafka.clients.producer.KafkaProducer;
import org.junit.jupiter.api.Test;
import org.junit.jupiter.api.io.TempDir;
import org.springframework.boot.autoconfigure.AutoConfigurations;
import org.springframework.boot.test.context.runner.ApplicationContextRunner;
import java.nio.file.Path;
import static org.assertj.core.api.Assertions.assertThat;
import static org.mockito.Mockito.mock;
class VehicleHistoryAppCompositionTest {
@TempDir
Path tempDir;
@Test
void createsHistoryStorageAndQueryBeansWithoutGb32960TcpServer() {
new ApplicationContextRunner()
.withConfiguration(AutoConfigurations.of(
SinkArchiveAutoConfiguration.class,
EventFileStoreAutoConfiguration.class,
SinkMqAutoConfiguration.class,
Gb32960AutoConfiguration.class,
EventHistoryAutoConfiguration.class))
.withAllowBeanDefinitionOverriding(true)
.withBean("kafkaProducer", KafkaProducer.class, VehicleHistoryAppCompositionTest::kafkaProducer)
.withPropertyValues(
"lingniu.ingest.sink.archive.enabled=true",
"lingniu.ingest.sink.archive.type=local",
"lingniu.ingest.sink.archive.path=" + tempDir.resolve("archive"),
"lingniu.ingest.event-file-store.enabled=true",
"lingniu.ingest.event-file-store.path=" + tempDir.resolve("event-store"),
"lingniu.ingest.event-history.enabled=true",
"lingniu.ingest.gb32960.enabled=true",
"lingniu.ingest.gb32960.server.enabled=false",
"lingniu.ingest.sink.mq.enabled=true",
"lingniu.ingest.sink.mq.type=kafka",
"lingniu.ingest.sink.mq.bootstrap-servers=localhost:9092",
"lingniu.ingest.sink.mq.consumer.enabled=false",
"lingniu.ingest.vehicle-state.enabled=false",
"lingniu.ingest.vehicle-stat.enabled=false")
.run(context -> {
assertThat(context).hasSingleBean(ArchiveStore.class);
assertThat(context).hasSingleBean(RawArchiveEventSink.class);
assertThat(context).hasSingleBean(EventFileStore.class);
assertThat(context).hasSingleBean(EventFileStoreSink.class);
assertThat(context).hasSingleBean(KafkaEventSink.class);
assertThat(context).hasSingleBean(KafkaEnvelopeDeadLetterSink.class);
assertThat(context).hasSingleBean(EventHistoryEnvelopeIngestor.class);
assertThat(context).hasSingleBean(EventHistoryController.class);
assertThat(context).hasSingleBean(Gb32960MessageDecoder.class);
assertThat(context).hasSingleBean(Gb32960DecodedFrameService.class);
assertThat(context).hasSingleBean(Gb32960FrameController.class);
assertThat(context).doesNotHaveBean(Gb32960NettyServer.class);
});
}
@SuppressWarnings("unchecked")
private static KafkaProducer<String, byte[]> kafkaProducer() {
return mock(KafkaProducer.class);
}
}

View File

@@ -0,0 +1,50 @@
package com.lingniu.ingest.historyapp;
import org.junit.jupiter.api.Test;
import org.springframework.beans.factory.config.YamlPropertiesFactoryBean;
import org.springframework.core.io.ClassPathResource;
import java.util.Properties;
import static org.assertj.core.api.Assertions.assertThat;
class VehicleHistoryAppDefaultsTest {
@Test
void applicationDefaultsKeepHistoryAsDecoderConsumerAndStorageRuntime() {
Properties properties = applicationProperties();
assertThat(properties)
.containsEntry("spring.application.name", "vehicle-history-app")
.containsEntry("lingniu.ingest.gb32960.enabled", true)
.containsEntry("lingniu.ingest.gb32960.server.enabled", false)
.containsEntry("lingniu.ingest.sink.archive.enabled", "${SINK_ARCHIVE_ENABLED:true}")
.containsEntry("lingniu.ingest.event-file-store.enabled", "${EVENT_FILE_STORE_ENABLED:true}")
.containsEntry("lingniu.ingest.event-history.enabled", true)
.containsEntry("lingniu.ingest.vehicle-state.enabled", false)
.containsEntry("lingniu.ingest.vehicle-stat.enabled", false)
.containsEntry("lingniu.ingest.sink.mq.consumer.enabled", "${KAFKA_CONSUMER_ENABLED:true}")
.containsEntry(
"lingniu.ingest.sink.mq.consumer.bindings.eventHistoryEnvelopeConsumerProcessor.enabled",
true)
.containsEntry(
"lingniu.ingest.sink.mq.consumer.bindings.eventHistoryEnvelopeConsumerProcessor.group-id",
"${KAFKA_GROUP_HISTORY:vehicle-history}")
.containsEntry(
"lingniu.ingest.sink.mq.consumer.bindings.eventHistoryEnvelopeConsumerProcessor.topics[0]",
"${KAFKA_TOPIC_GB32960_EVENT:vehicle.event.gb32960.v1}")
.containsEntry(
"lingniu.ingest.sink.mq.consumer.bindings.eventHistoryEnvelopeConsumerProcessor.topics[1]",
"${KAFKA_TOPIC_GB32960_RAW:vehicle.raw.gb32960.v1}");
}
private static Properties applicationProperties() {
YamlPropertiesFactoryBean factory = new YamlPropertiesFactoryBean();
factory.setResources(new ClassPathResource("application.yml"));
Properties properties = factory.getObject();
assertThat(properties).isNotNull();
return properties;
}
}

View File

@@ -13,7 +13,12 @@ import com.lmax.disruptor.dsl.ProducerType;
import org.slf4j.Logger; import org.slf4j.Logger;
import org.slf4j.LoggerFactory; import org.slf4j.LoggerFactory;
import java.util.ArrayList;
import java.util.List; import java.util.List;
import java.util.Objects;
import java.util.concurrent.CompletableFuture;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;
import java.util.concurrent.ThreadFactory; import java.util.concurrent.ThreadFactory;
import java.util.concurrent.atomic.AtomicLong; import java.util.concurrent.atomic.AtomicLong;
@@ -28,10 +33,14 @@ public final class DisruptorEventBus implements AutoCloseable {
private static final Logger log = LoggerFactory.getLogger(DisruptorEventBus.class); private static final Logger log = LoggerFactory.getLogger(DisruptorEventBus.class);
private final Disruptor<VehicleEventSlot> disruptor; private final Disruptor<VehicleEventSlot> disruptor;
private final List<EventSink> sinks;
private final ExecutorService awaitExecutor;
private final AtomicLong published = new AtomicLong(); private final AtomicLong published = new AtomicLong();
private final AtomicLong failed = new AtomicLong(); private final AtomicLong failed = new AtomicLong();
public DisruptorEventBus(int ringBufferSize, String waitStrategyName, List<EventSink> sinks) { public DisruptorEventBus(int ringBufferSize, String waitStrategyName, List<EventSink> sinks) {
this.sinks = List.copyOf(sinks);
this.awaitExecutor = Executors.newVirtualThreadPerTaskExecutor();
ThreadFactory tf = Thread.ofVirtual().name("ingest-bus-", 0).factory(); ThreadFactory tf = Thread.ofVirtual().name("ingest-bus-", 0).factory();
this.disruptor = new Disruptor<>( this.disruptor = new Disruptor<>(
VehicleEventSlot::new, VehicleEventSlot::new,
@@ -40,14 +49,14 @@ public final class DisruptorEventBus implements AutoCloseable {
ProducerType.MULTI, ProducerType.MULTI,
waitStrategy(waitStrategyName)); waitStrategy(waitStrategyName));
EventHandler<VehicleEventSlot>[] handlers = sinks.stream() EventHandler<VehicleEventSlot>[] handlers = this.sinks.stream()
.map(this::toHandler) .map(this::toHandler)
.toArray(EventHandler[]::new); .toArray(EventHandler[]::new);
// 每个 sink 一个独立 handler事件按扇出模式同时写 Kafka、event-file-store 等目标。 // 每个 sink 一个独立 handler事件按扇出模式同时写 Kafka、event-file-store 等目标。
disruptor.handleEventsWith(handlers); disruptor.handleEventsWith(handlers);
disruptor.start(); disruptor.start();
log.info("DisruptorEventBus started ringBuffer={} wait={} sinks={}", log.info("DisruptorEventBus started ringBuffer={} wait={} sinks={}",
ringBufferSize, waitStrategyName, sinks.size()); ringBufferSize, waitStrategyName, this.sinks.size());
} }
public void publish(VehicleEvent event) { public void publish(VehicleEvent event) {
@@ -55,9 +64,36 @@ public final class DisruptorEventBus implements AutoCloseable {
published.incrementAndGet(); published.incrementAndGet();
} }
public CompletableFuture<Void> publishAndAwait(VehicleEvent event, String requiredSinkName) {
String requiredSink = Objects.requireNonNull(requiredSinkName, "requiredSinkName");
published.incrementAndGet();
List<EventSink> acceptingSinks = sinks.stream()
.filter(sink -> sink.accepts(event))
.toList();
List<EventSink> requiredSinks = acceptingSinks.stream()
.filter(sink -> requiredSink.equals(sink.name()))
.toList();
if (requiredSinks.isEmpty()) {
return CompletableFuture.failedFuture(new IllegalStateException(
"required sink '" + requiredSink + "' did not accept event " + event.eventId()));
}
List<CompletableFuture<Void>> requiredFutures = new ArrayList<>();
for (EventSink sink : acceptingSinks) {
CompletableFuture<Void> future = CompletableFuture
.supplyAsync(() -> publishToSinkAndTrack(sink, event), awaitExecutor)
.thenCompose(f -> f);
if (requiredSink.equals(sink.name())) {
requiredFutures.add(future);
}
}
return CompletableFuture.allOf(requiredFutures.toArray(CompletableFuture[]::new));
}
@Override @Override
public void close() { public void close() {
disruptor.shutdown(); disruptor.shutdown();
awaitExecutor.shutdown();
log.info("DisruptorEventBus stopped published={} failed={}", published.get(), failed.get()); log.info("DisruptorEventBus stopped published={} failed={}", published.get(), failed.get());
} }
@@ -66,18 +102,28 @@ public final class DisruptorEventBus implements AutoCloseable {
VehicleEvent e = slot.event; VehicleEvent e = slot.event;
if (e == null || !sink.accepts(e)) return; if (e == null || !sink.accepts(e)) return;
try { try {
sink.publish(e).exceptionally(ex -> { publishToSinkAndTrack(sink, e);
// sink 异步失败只计数和打日志,不反向阻塞入站连接;生产侧靠 sink 自身重试/DLQ 保证可观测。
failed.incrementAndGet();
log.warn("sink {} publish failed", sink.name(), ex);
return null;
});
} finally { } finally {
if (endOfBatch) slot.clear(); if (endOfBatch) slot.clear();
} }
}; };
} }
private CompletableFuture<Void> publishToSinkAndTrack(EventSink sink, VehicleEvent event) {
try {
return sink.publish(event).whenComplete((ignored, ex) -> {
if (ex == null) return;
// sink 异步失败只计数和打日志await 调用方仍会收到异常并决定是否 ACK。
failed.incrementAndGet();
log.warn("sink {} publish failed", sink.name(), ex);
});
} catch (Throwable t) {
failed.incrementAndGet();
log.warn("sink {} publish failed", sink.name(), t);
return CompletableFuture.failedFuture(t);
}
}
private static WaitStrategy waitStrategy(String name) { private static WaitStrategy waitStrategy(String name) {
return switch (name == null ? "yielding" : name.toLowerCase()) { return switch (name == null ? "yielding" : name.toLowerCase()) {
case "blocking" -> new BlockingWaitStrategy(); case "blocking" -> new BlockingWaitStrategy();

View File

@@ -10,11 +10,16 @@ import com.lingniu.ingest.core.pipeline.InterceptorChain;
import org.slf4j.Logger; import org.slf4j.Logger;
import org.slf4j.LoggerFactory; import org.slf4j.LoggerFactory;
import java.time.Duration;
import java.time.Instant; import java.time.Instant;
import java.util.ArrayList;
import java.util.LinkedHashMap; import java.util.LinkedHashMap;
import java.util.List; import java.util.List;
import java.util.Map; import java.util.Map;
import java.util.UUID; import java.util.UUID;
import java.util.concurrent.CompletableFuture;
import java.util.concurrent.CompletionException;
import java.util.concurrent.TimeUnit;
import java.util.concurrent.atomic.AtomicLong; import java.util.concurrent.atomic.AtomicLong;
/** /**
@@ -30,6 +35,7 @@ public final class Dispatcher {
private static final Logger log = LoggerFactory.getLogger(Dispatcher.class); private static final Logger log = LoggerFactory.getLogger(Dispatcher.class);
private static final AtomicLong RAW_ARCHIVE_SEQUENCE = new AtomicLong(); private static final AtomicLong RAW_ARCHIVE_SEQUENCE = new AtomicLong();
private static final String REQUIRED_DURABLE_SINK = "kafka";
private final HandlerRegistry registry; private final HandlerRegistry registry;
private final InterceptorChain interceptors; private final InterceptorChain interceptors;
@@ -52,13 +58,67 @@ public final class Dispatcher {
public void dispatch(RawFrame frame) { public void dispatch(RawFrame frame) {
IngestContext ctx = new IngestContext(UUID.randomUUID().toString()); IngestContext ctx = new IngestContext(UUID.randomUUID().toString());
try { try {
// 在 interceptor 之前发 RawArchive保证原始字节被无条件落盘dedup/rate-limit DispatchPlan plan = mapFrameToEvents(frame, ctx, false);
// 不会过滤它),满足"原始可回放"目标。archive 的写盘靠下游 ArchiveEventSink 消费。 for (VehicleEvent e : plan.events()) {
RawArchiveLookup rawArchive = emitRawArchive(frame, ctx); eventBus.publish(e);
}
if (plan.failure() != null) {
throw plan.failure();
}
} catch (Throwable t) {
log.error("dispatch failure traceId={}", ctx.traceId(), t);
interceptors.onError(t, ctx);
}
}
public CompletableFuture<Void> dispatchAndAwait(RawFrame frame) {
IngestContext ctx = new IngestContext(UUID.randomUUID().toString());
try {
DispatchPlan plan = mapFrameToEvents(frame, ctx, true);
if (plan.events().isEmpty() && plan.failure() == null) {
return CompletableFuture.failedFuture(new IllegalStateException(
"awaited dispatch produced no events; required sink '" + REQUIRED_DURABLE_SINK + "' was not reached"));
}
CompletableFuture<?>[] futures = plan.events().stream()
.map(event -> eventBus.publishAndAwait(event, REQUIRED_DURABLE_SINK))
.toArray(CompletableFuture[]::new);
CompletableFuture<Void> boundary = CompletableFuture.allOf(futures);
if (plan.failure() == null) {
return boundary;
}
return boundary.handle((ignored, publishFailure) -> {
log.error("dispatch failure traceId={}", ctx.traceId(), plan.failure());
interceptors.onError(plan.failure(), ctx);
if (publishFailure != null) {
throw new CompletionException(publishFailure);
}
throw new CompletionException(plan.failure());
});
} catch (Throwable t) {
log.error("dispatch failure traceId={}", ctx.traceId(), t);
interceptors.onError(t, ctx);
return CompletableFuture.failedFuture(t);
}
}
public CompletableFuture<Void> dispatchAndAwait(RawFrame frame, Duration timeout) {
CompletableFuture<Void> future = dispatchAndAwait(frame);
if (timeout == null || timeout.isZero() || timeout.isNegative()) {
return future;
}
return future.orTimeout(timeout.toMillis(), TimeUnit.MILLISECONDS);
}
private DispatchPlan mapFrameToEvents(RawFrame frame, IngestContext ctx, boolean awaitDurability) {
List<VehicleEvent> out = new ArrayList<>();
// 在 interceptor 之前发 RawArchive保证原始字节被无条件落盘dedup/rate-limit
// 不会过滤它),满足"原始可回放"目标。archive 的写盘靠下游 Sink 消费。
RawArchiveLookup rawArchive = appendRawArchive(frame, ctx, out);
try {
if (!interceptors.before(frame, ctx)) { if (!interceptors.before(frame, ctx)) {
log.debug("frame aborted: {}", ctx.abortReason()); log.debug("frame aborted: {}", ctx.abortReason());
return; return new DispatchPlan(out, null);
} }
List<HandlerDefinition> handlers = registry.resolve( List<HandlerDefinition> handlers = registry.resolve(
@@ -67,11 +127,16 @@ public final class Dispatcher {
log.debug("no handler for {} cmd=0x{} info=0x{}", log.debug("no handler for {} cmd=0x{} info=0x{}",
frame.protocolId(), Integer.toHexString(frame.command()), frame.protocolId(), Integer.toHexString(frame.command()),
Integer.toHexString(frame.infoType())); Integer.toHexString(frame.infoType()));
return; return new DispatchPlan(out, null);
} }
for (HandlerDefinition def : handlers) { for (HandlerDefinition def : handlers) {
if (def.asyncBatch() != null) { if (def.asyncBatch() != null) {
if (awaitDurability) {
throw new AwaitedAsyncBatchUnsupportedException(
"@AsyncBatch handler cannot be used with dispatchAndAwait until batch completion is awaited: "
+ def.method());
}
batchExecutor.submit(def, frame.payload(), event -> enrichWithRawArchive(event, rawArchive)); batchExecutor.submit(def, frame.payload(), event -> enrichWithRawArchive(event, rawArchive));
continue; continue;
} }
@@ -79,12 +144,15 @@ public final class Dispatcher {
for (VehicleEvent e : events) { for (VehicleEvent e : events) {
e = enrichWithRawArchive(e, rawArchive); e = enrichWithRawArchive(e, rawArchive);
interceptors.after(e, ctx); interceptors.after(e, ctx);
eventBus.publish(e); out.add(e);
} }
} }
return new DispatchPlan(out, null);
} catch (Throwable t) { } catch (Throwable t) {
log.error("dispatch failure traceId={}", ctx.traceId(), t); if (t instanceof AwaitedAsyncBatchUnsupportedException) {
interceptors.onError(t, ctx); return new DispatchPlan(List.of(), t);
}
return new DispatchPlan(out, t);
} }
} }
@@ -95,7 +163,7 @@ public final class Dispatcher {
* <p>VIN 取自 sourceMeta 里的 {@code vin} key由各入站适配器负责填充缺失时 * <p>VIN 取自 sourceMeta 里的 {@code vin} key由各入站适配器负责填充缺失时
* 留空字符串archive sink 会用 "unknown-vin" 占位保证 key 可解析。 * 留空字符串archive sink 会用 "unknown-vin" 占位保证 key 可解析。
*/ */
private RawArchiveLookup emitRawArchive(RawFrame frame, IngestContext ctx) { private RawArchiveLookup appendRawArchive(RawFrame frame, IngestContext ctx, List<VehicleEvent> out) {
byte[] bytes = frame.rawBytes(); byte[] bytes = frame.rawBytes();
if (bytes == null || bytes.length == 0) return RawArchiveLookup.empty(); if (bytes == null || bytes.length == 0) return RawArchiveLookup.empty();
@@ -117,7 +185,7 @@ public final class Dispatcher {
frame.command(), frame.command(),
frame.infoType(), frame.infoType(),
bytes); bytes);
eventBus.publish(raw); out.add(raw);
return new RawArchiveLookup(eventId, key, RawArchiveKeys.logicalUri(key)); return new RawArchiveLookup(eventId, key, RawArchiveKeys.logicalUri(key));
} }
@@ -182,4 +250,13 @@ public final class Dispatcher {
return key == null || key.isBlank(); return key == null || key.isBlank();
} }
} }
private record DispatchPlan(List<VehicleEvent> events, Throwable failure) {
}
private static final class AwaitedAsyncBatchUnsupportedException extends IllegalStateException {
private AwaitedAsyncBatchUnsupportedException(String message) {
super(message);
}
}
} }

View File

@@ -0,0 +1,81 @@
package com.lingniu.ingest.core.concurrency;
import com.lingniu.ingest.api.ProtocolId;
import com.lingniu.ingest.api.event.VehicleEvent;
import com.lingniu.ingest.api.sink.EventSink;
import org.junit.jupiter.api.Test;
import java.time.Instant;
import java.util.List;
import java.util.Map;
import java.util.concurrent.CompletableFuture;
import java.util.concurrent.CompletionException;
import java.util.concurrent.CountDownLatch;
import java.util.concurrent.TimeUnit;
import java.util.concurrent.atomic.AtomicLong;
import static org.assertj.core.api.Assertions.assertThat;
import static org.assertj.core.api.Assertions.assertThatThrownBy;
class DisruptorEventBusAwaitTest {
@Test
void publishAndAwaitDoesNotInvokeSinkPublishOnCallerThread() throws Exception {
long callerThreadId = Thread.currentThread().threadId();
CapturingSink sink = new CapturingSink("kafka");
try (DisruptorEventBus bus = new DisruptorEventBus(1024, "blocking", java.util.List.of(sink))) {
bus.publishAndAwait(event(), "kafka").get(3, TimeUnit.SECONDS);
}
assertThat(sink.publishThreadId.get()).isNotEqualTo(callerThreadId);
}
@Test
void publishAndAwaitDoesNotInvokeOptionalSinksWhenRequiredSinkIsAbsent() throws Exception {
CapturingSink optionalSink = new CapturingSink("event-file-store");
try (DisruptorEventBus bus = new DisruptorEventBus(1024, "blocking", List.of(optionalSink))) {
assertThatThrownBy(() -> bus.publishAndAwait(event(), "kafka").join())
.isInstanceOf(CompletionException.class)
.hasRootCauseInstanceOf(IllegalStateException.class)
.hasMessageContaining("kafka");
assertThat(optionalSink.published.await(200, TimeUnit.MILLISECONDS)).isFalse();
}
}
private static VehicleEvent.Heartbeat event() {
Instant now = Instant.parse("2026-06-23T10:00:00Z");
return new VehicleEvent.Heartbeat(
"heartbeat-1",
"VIN001",
ProtocolId.GB32960,
now,
now,
"trace-1",
Map.of());
}
private static final class CapturingSink implements EventSink {
private final String name;
private final AtomicLong publishThreadId = new AtomicLong(-1);
private final CountDownLatch published = new CountDownLatch(1);
private CapturingSink(String name) {
this.name = name;
}
@Override
public String name() {
return name;
}
@Override
public CompletableFuture<Void> publish(VehicleEvent event) {
publishThreadId.set(Thread.currentThread().threadId());
published.countDown();
return CompletableFuture.completedFuture(null);
}
}
}

View File

@@ -0,0 +1,217 @@
package com.lingniu.ingest.core.dispatcher;
import com.lingniu.ingest.api.ProtocolId;
import com.lingniu.ingest.api.annotation.AsyncBatch;
import com.lingniu.ingest.api.annotation.MessageMapping;
import com.lingniu.ingest.api.annotation.ProtocolHandler;
import com.lingniu.ingest.api.event.VehicleEvent;
import com.lingniu.ingest.api.pipeline.IngestContext;
import com.lingniu.ingest.api.pipeline.IngestInterceptor;
import com.lingniu.ingest.api.pipeline.RawFrame;
import com.lingniu.ingest.api.sink.EventSink;
import com.lingniu.ingest.core.concurrency.AsyncBatchExecutor;
import com.lingniu.ingest.core.concurrency.DisruptorEventBus;
import com.lingniu.ingest.core.pipeline.InterceptorChain;
import org.junit.jupiter.api.Test;
import java.lang.reflect.Method;
import java.time.Instant;
import java.util.List;
import java.util.Map;
import java.util.concurrent.CompletableFuture;
import java.util.concurrent.CompletionException;
import java.util.concurrent.CompletionStage;
import java.util.concurrent.atomic.AtomicInteger;
import java.util.concurrent.atomic.AtomicReference;
import static org.assertj.core.api.Assertions.assertThat;
import static org.assertj.core.api.Assertions.assertThatThrownBy;
class DispatcherDurableAckBoundaryTest {
@Test
void dispatchAndAwaitFailsWhenNoKafkaSinkAcceptsEvent() throws Exception {
EventSink fileSink = new NamedSink("event-file-store", true);
try (Harness harness = new Harness(registryWithHeartbeatHandler(), List.of(fileSink), List.of())) {
CompletionStage<Void> future = harness.dispatcher.dispatchAndAwait(frame(0x07, "payload"));
assertThatThrownBy(() -> future.toCompletableFuture().join())
.isInstanceOf(CompletionException.class)
.hasRootCauseInstanceOf(IllegalStateException.class)
.hasMessageContaining("kafka");
}
}
@Test
void dispatchAndAwaitNotifiesInterceptorWhenPlanCapturesFailure() throws Exception {
CapturingInterceptor interceptor = new CapturingInterceptor();
try (Harness harness = new Harness(registryWithThrowingHandler(), List.of(new NamedSink("kafka", true)),
List.of(interceptor))) {
CompletionStage<Void> future = harness.dispatcher.dispatchAndAwait(frame(0x08, "payload"));
assertThatThrownBy(() -> future.toCompletableFuture().join())
.isInstanceOf(CompletionException.class)
.hasRootCauseInstanceOf(IllegalStateException.class)
.rootCause()
.hasMessageContaining("handler failed");
assertThat(interceptor.error.get())
.isInstanceOf(RuntimeException.class)
.hasMessageContaining("Handler threw exception");
}
}
@Test
void dispatchAndAwaitFailsFastForAsyncBatchHandler() throws Exception {
NamedSink kafka = new NamedSink("kafka", true);
try (Harness harness = new Harness(registryWithAsyncBatchHandler(), List.of(kafka), List.of())) {
CompletionStage<Void> future = harness.dispatcher.dispatchAndAwait(
frame(0x09, "payload", new byte[]{0x23, 0x23, 0x09}));
assertThatThrownBy(() -> future.toCompletableFuture().join())
.isInstanceOf(CompletionException.class)
.hasRootCauseInstanceOf(IllegalStateException.class)
.hasMessageContaining("@AsyncBatch");
assertThat(kafka.published.get()).isZero();
}
}
private static HandlerRegistry registryWithHeartbeatHandler() throws NoSuchMethodException {
HeartbeatHandler bean = new HeartbeatHandler();
Method method = HeartbeatHandler.class.getDeclaredMethod("onHeartbeat", Object.class);
return registry(bean, method, 0x07, method.getAnnotation(AsyncBatch.class));
}
private static HandlerRegistry registryWithThrowingHandler() throws NoSuchMethodException {
ThrowingHandler bean = new ThrowingHandler();
Method method = ThrowingHandler.class.getDeclaredMethod("onFrame", Object.class);
return registry(bean, method, 0x08, method.getAnnotation(AsyncBatch.class));
}
private static HandlerRegistry registryWithAsyncBatchHandler() throws NoSuchMethodException {
AsyncHandler bean = new AsyncHandler();
Method method = AsyncHandler.class.getDeclaredMethod("onBatch", List.class);
return registry(bean, method, 0x09, method.getAnnotation(AsyncBatch.class));
}
private static HandlerRegistry registry(Object bean, Method method, int command, AsyncBatch asyncBatch) {
HandlerRegistry registry = new HandlerRegistry();
registry.register(new HandlerDefinition(
ProtocolId.GB32960,
command,
0,
method.getName(),
bean,
method,
Object.class,
null,
null,
asyncBatch));
return registry;
}
private static RawFrame frame(int command, Object payload) {
return frame(command, payload, null);
}
private static RawFrame frame(int command, Object payload, byte[] rawBytes) {
return new RawFrame(
ProtocolId.GB32960,
command,
0,
payload,
rawBytes,
Map.of("vin", "VIN001"),
Instant.parse("2026-06-23T10:00:00Z"));
}
private static VehicleEvent.Heartbeat event(String id) {
Instant now = Instant.parse("2026-06-23T10:00:00Z");
return new VehicleEvent.Heartbeat(id, "VIN001", ProtocolId.GB32960, now, now, "trace-" + id, Map.of());
}
private static final class Harness implements AutoCloseable {
private final DisruptorEventBus eventBus;
private final AsyncBatchExecutor batchExecutor;
private final Dispatcher dispatcher;
private Harness(HandlerRegistry registry, List<EventSink> sinks, List<IngestInterceptor> interceptors) {
this.eventBus = new DisruptorEventBus(1024, "blocking", sinks);
this.batchExecutor = new AsyncBatchExecutor(eventBus::publish);
this.dispatcher = new Dispatcher(
registry,
new InterceptorChain(interceptors),
new HandlerInvoker(),
eventBus,
batchExecutor);
}
@Override
public void close() {
batchExecutor.close();
eventBus.close();
}
}
private static final class NamedSink implements EventSink {
private final String name;
private final boolean accepts;
private final AtomicInteger published = new AtomicInteger();
private NamedSink(String name, boolean accepts) {
this.name = name;
this.accepts = accepts;
}
@Override
public String name() {
return name;
}
@Override
public CompletableFuture<Void> publish(VehicleEvent event) {
published.incrementAndGet();
return CompletableFuture.completedFuture(null);
}
@Override
public boolean accepts(VehicleEvent event) {
return accepts;
}
}
private static final class CapturingInterceptor implements IngestInterceptor {
private final AtomicReference<Throwable> error = new AtomicReference<>();
@Override
public void onError(Throwable error, IngestContext ctx) {
this.error.set(error);
}
}
@ProtocolHandler(protocol = ProtocolId.GB32960)
static final class HeartbeatHandler {
@MessageMapping(command = 0x07)
VehicleEvent.Heartbeat onHeartbeat(Object ignored) {
return event("heartbeat-1");
}
}
@ProtocolHandler(protocol = ProtocolId.GB32960)
static final class ThrowingHandler {
@MessageMapping(command = 0x08)
VehicleEvent.Heartbeat onFrame(Object ignored) {
throw new IllegalStateException("handler failed");
}
}
@ProtocolHandler(protocol = ProtocolId.GB32960)
public static final class AsyncHandler {
@MessageMapping(command = 0x09)
@AsyncBatch(size = 2, waitMs = 10, poolSize = 1)
public List<VehicleEvent> onBatch(List<Object> ignored) {
return List.of(event("async-1"));
}
}
}

View File

@@ -55,5 +55,10 @@
<artifactId>assertj-core</artifactId> <artifactId>assertj-core</artifactId>
<scope>test</scope> <scope>test</scope>
</dependency> </dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-test</artifactId>
<scope>test</scope>
</dependency>
</dependencies> </dependencies>
</project> </project>

View File

@@ -59,8 +59,8 @@ import java.util.List;
* <p>每个 Parser Bean 都会被 {@link InfoBlockParserRegistry} 按 * <p>每个 Parser Bean 都会被 {@link InfoBlockParserRegistry} 按
* {@code (ProtocolVersion, typeCode)} 注册。2016/2025 可同时加载。 * {@code (ProtocolVersion, typeCode)} 注册。2016/2025 可同时加载。
* *
* <p>只有 {@code lingniu.ingest.gb32960.enabled=true} 时才会启动 32960 TCP 服务 * <p>只有 {@code lingniu.ingest.gb32960.enabled=true} 时才会装配 32960 协议解析组件
* 解析、鉴权、ACK、RAW 入库都从 {@link Gb32960NettyServer} 建立的 Netty pipeline 进入 * TCP 监听由 {@code lingniu.ingest.gb32960.server.enabled} 单独控制
*/ */
@AutoConfiguration @AutoConfiguration
@EnableConfigurationProperties(Gb32960Properties.class) @EnableConfigurationProperties(Gb32960Properties.class)
@@ -222,6 +222,8 @@ public class Gb32960AutoConfiguration {
@Bean @Bean
@ConditionalOnMissingBean @ConditionalOnMissingBean
@ConditionalOnProperty(prefix = "lingniu.ingest.gb32960.server", name = "enabled",
havingValue = "true", matchIfMissing = true)
public Gb32960NettyServer gb32960NettyServer(Gb32960Properties props, public Gb32960NettyServer gb32960NettyServer(Gb32960Properties props,
Gb32960MessageDecoder decoder, Gb32960MessageDecoder decoder,
Dispatcher dispatcher, Dispatcher dispatcher,

View File

@@ -10,8 +10,10 @@ import java.util.Set;
@ConfigurationProperties(prefix = "lingniu.ingest.gb32960") @ConfigurationProperties(prefix = "lingniu.ingest.gb32960")
public class Gb32960Properties { public class Gb32960Properties {
/** 是否启动 32960 TCP 监听。生产 32960 线路打开后,此开关决定端口是否真正 bind。 */ /** 是否装配 32960 协议解析组件。TCP 监听由 {@link #server} 单独控制。 */
private boolean enabled = false; private boolean enabled = false;
/** 32960 TCP 监听开关。默认跟随协议启用后启动,查询类应用可单独关闭。 */
private Server server = new Server();
/** 32960 TCP 监听端口;当前生产验证线使用 32960。 */ /** 32960 TCP 监听端口;当前生产验证线使用 32960。 */
private int port = 9000; private int port = 9000;
/** Netty worker 线程数0 表示按 Netty/CPU 默认策略分配。 */ /** Netty worker 线程数0 表示按 Netty/CPU 默认策略分配。 */
@@ -63,6 +65,8 @@ public class Gb32960Properties {
public boolean isEnabled() { return enabled; } public boolean isEnabled() { return enabled; }
public void setEnabled(boolean enabled) { this.enabled = enabled; } public void setEnabled(boolean enabled) { this.enabled = enabled; }
public Server getServer() { return server; }
public void setServer(Server server) { this.server = server; }
public int getPort() { return port; } public int getPort() { return port; }
public void setPort(int port) { this.port = port; } public void setPort(int port) { this.port = port; }
public int getWorkerThreads() { return workerThreads; } public int getWorkerThreads() { return workerThreads; }
@@ -82,6 +86,13 @@ public class Gb32960Properties {
this.vendorExtensions = vendorExtensions; this.vendorExtensions = vendorExtensions;
} }
public static class Server {
private boolean enabled = true;
public boolean isEnabled() { return enabled; }
public void setEnabled(boolean enabled) { this.enabled = enabled; }
}
/** /**
* VIN 白名单认证。 * VIN 白名单认证。
* *

View File

@@ -20,6 +20,7 @@ import org.slf4j.LoggerFactory;
import java.net.InetSocketAddress; import java.net.InetSocketAddress;
import java.nio.ByteBuffer; import java.nio.ByteBuffer;
import java.time.Instant; import java.time.Instant;
import java.util.concurrent.CompletableFuture;
import java.util.HashMap; import java.util.HashMap;
import java.util.List; import java.util.List;
import java.util.Map; import java.util.Map;
@@ -55,6 +56,9 @@ public class Gb32960ChannelHandler extends SimpleChannelInboundHandler<byte[]> {
public static final AttributeKey<String> PLATFORM_ACCOUNT_ATTR = public static final AttributeKey<String> PLATFORM_ACCOUNT_ATTR =
Gb32960AccessService.PLATFORM_ACCOUNT_ATTR; Gb32960AccessService.PLATFORM_ACCOUNT_ATTR;
private static final AttributeKey<CompletableFuture<Void>> DURABLE_ACK_CHAIN_ATTR =
AttributeKey.valueOf("gb32960.durableAckChain");
private final Gb32960MessageDecoder decoder; private final Gb32960MessageDecoder decoder;
private final Dispatcher dispatcher; private final Dispatcher dispatcher;
private final Gb32960AccessService accessService; private final Gb32960AccessService accessService;
@@ -122,41 +126,137 @@ public class Gb32960ChannelHandler extends SimpleChannelInboundHandler<byte[]> {
return; return;
} }
// 鉴权通过:按命令类型回 ACK写成功后统一 dispatch 到通用管线。 // 鉴权通过:先完成命令侧状态变更/日志,再统一 dispatch 到通用管线。
// 需要成功 ACK 的已接收帧,等待配置的 Kafka dispatch 边界完成后再回 ACK。
// 平台登入鉴权失败会短路 return不进 dispatch。 // 平台登入鉴权失败会短路 return不进 dispatch。
switch (cmd) { switch (cmd) {
case VEHICLE_LOGIN -> handleVehicleLogin(ctx, msg, rawVin); case VEHICLE_LOGIN -> logVehicleLogin(ctx, msg);
case VEHICLE_LOGOUT -> handleVehicleLogout(ctx, msg, rawVin); case VEHICLE_LOGOUT -> logVehicleLogout(ctx, msg);
case PLATFORM_LOGIN -> { case PLATFORM_LOGIN -> {
if (!handlePlatformLogin(ctx, msg, rawVin)) return; if (!handlePlatformLogin(ctx, msg, rawVin)) return;
} }
case PLATFORM_LOGOUT -> handlePlatformLogout(ctx, msg, rawVin); case PLATFORM_LOGOUT -> logPlatformLogout(ctx);
case REALTIME_REPORT, RESEND_REPORT, HEARTBEAT -> handleReportOrHeartbeat(ctx, msg, rawVin); case REALTIME_REPORT, RESEND_REPORT, HEARTBEAT -> logReportOrHeartbeat(ctx, msg);
case TIME_CALIBRATION -> handleTimeCalibration(ctx, msg, rawVin); case TIME_CALIBRATION -> { }
default -> logOtherFrame(ctx, msg); default -> logOtherFrame(ctx, msg);
} }
RawFrame rf = rawFrame(ctx, msg, frame);
if (!requiresDurableAck(cmd)) {
dispatcher.dispatch(rf);
return;
}
enqueueDurableAck(ctx, msg, rawVin, vin, cmd, dispatcher.dispatchAndAwait(rf));
}
private void enqueueDurableAck(ChannelHandlerContext ctx,
Gb32960Message msg,
byte[] rawVin,
String vin,
CommandType cmd,
CompletableFuture<Void> dispatchFuture) {
CompletableFuture<Void> previous = ctx.channel().attr(DURABLE_ACK_CHAIN_ATTR).get();
if (previous == null) {
previous = CompletableFuture.completedFuture(null);
}
CompletableFuture<Void> next = previous
.handle((ignored, previousFailure) -> null)
.thenCompose(ignored -> dispatchFuture.handle((ok, failure) -> failure))
.thenCompose(failure -> runOnEventLoop(ctx,
() -> handleDurableAckResult(ctx, msg, rawVin, vin, cmd, failure)));
ctx.channel().attr(DURABLE_ACK_CHAIN_ATTR).set(next);
}
private CompletableFuture<Void> runOnEventLoop(ChannelHandlerContext ctx, Runnable task) {
CompletableFuture<Void> done = new CompletableFuture<>();
Runnable wrapped = () -> {
try {
task.run();
done.complete(null);
} catch (Throwable t) {
done.completeExceptionally(t);
}
};
if (ctx.executor().inEventLoop()) {
wrapped.run();
} else {
ctx.executor().execute(wrapped);
}
return done;
}
private void handleDurableAckResult(ChannelHandlerContext ctx,
Gb32960Message msg,
byte[] rawVin,
String vin,
CommandType cmd,
Throwable failure) {
if (!ctx.channel().isActive()) {
return;
}
if (failure != null) {
log.warn("[gb32960] dispatch failed before ack peer={} vin={} cmd=0x{}",
addr(ctx), vin, Integer.toHexString(cmd.code()), failure);
ctx.close();
return;
}
writeAckForCommand(ctx, msg, rawVin);
}
private RawFrame rawFrame(ChannelHandlerContext ctx, Gb32960Message msg, byte[] frame) {
Map<String, String> sourceMeta = new HashMap<>(4); Map<String, String> sourceMeta = new HashMap<>(4);
sourceMeta.put("vin", vin); sourceMeta.put("vin", msg.header().vin());
sourceMeta.put("peer", addr(ctx)); sourceMeta.put("peer", addr(ctx));
String platformAccount = accessService.platformAccount(ctx.channel()); String platformAccount = accessService.platformAccount(ctx.channel());
if (platformAccount != null && !platformAccount.isBlank()) { if (platformAccount != null && !platformAccount.isBlank()) {
sourceMeta.put("platformAccount", platformAccount); sourceMeta.put("platformAccount", platformAccount);
} }
// RawFrame.rawBytes 是后续 raw archive 和按 rawArchiveUri 回查的源头;不要在这里裁剪或重编码。 // RawFrame.rawBytes 是后续 raw archive 和按 rawArchiveUri 回查的源头;不要在这里裁剪或重编码。
RawFrame rf = new RawFrame( return new RawFrame(
ProtocolId.GB32960, ProtocolId.GB32960,
cmd.code(), msg.header().command().code(),
0, 0,
msg, msg,
frame, frame,
sourceMeta, sourceMeta,
Instant.now()); Instant.now());
dispatcher.dispatch(rf); }
private static boolean requiresDurableAck(CommandType cmd) {
return switch (cmd) {
case VEHICLE_LOGIN,
VEHICLE_LOGOUT,
PLATFORM_LOGIN,
PLATFORM_LOGOUT,
REALTIME_REPORT,
RESEND_REPORT,
HEARTBEAT,
TIME_CALIBRATION -> true;
default -> false;
};
}
private void writeAckForCommand(ChannelHandlerContext ctx, Gb32960Message msg, byte[] rawVin) {
switch (msg.header().command()) {
case VEHICLE_LOGIN -> writeVehicleLoginAck(ctx, msg, rawVin);
case VEHICLE_LOGOUT -> writeVehicleLogoutAck(ctx, msg, rawVin);
case PLATFORM_LOGIN -> writePlatformLoginAck(ctx, msg, rawVin);
case PLATFORM_LOGOUT -> writePlatformLogoutAck(ctx, msg, rawVin);
case REALTIME_REPORT, RESEND_REPORT, HEARTBEAT -> writeReportOrHeartbeatAck(ctx, msg, rawVin);
case TIME_CALIBRATION -> writeTimeCalibrationAck(ctx, msg, rawVin);
default -> { }
}
} }
/** 0x01 车辆登入:按 §6.3.2 带原采集时间回成功 ACK + INFO 日志。 */ /** 0x01 车辆登入:按 §6.3.2 带原采集时间回成功 ACK + INFO 日志。 */
private void handleVehicleLogin(ChannelHandlerContext ctx, Gb32960Message msg, byte[] rawVin) { private void logVehicleLogin(ChannelHandlerContext ctx, Gb32960Message msg) {
log.info("[gb32960] vehicle login peer={} vin={} protocolVersion={}",
addr(ctx), msg.header().vin(), msg.header().protocolVersion());
}
private void writeVehicleLoginAck(ChannelHandlerContext ctx, Gb32960Message msg, byte[] rawVin) {
ackService.writeResponse( ackService.writeResponse(
ctx, ctx,
msg.header().protocolVersion(), msg.header().protocolVersion(),
@@ -166,13 +266,14 @@ public class Gb32960ChannelHandler extends SimpleChannelInboundHandler<byte[]> {
eventOrNow(msg), eventOrNow(msg),
null, null,
"vehicle-login-ack"); "vehicle-login-ack");
log.info("[gb32960] vehicle login peer={} vin={} protocolVersion={}",
addr(ctx), msg.header().vin(), msg.header().protocolVersion());
} }
/** 0x04 车辆登出INFO 日志 + 带原采集时间回成功 ACK。 */ /** 0x04 车辆登出INFO 日志 + 带原采集时间回成功 ACK。 */
private void handleVehicleLogout(ChannelHandlerContext ctx, Gb32960Message msg, byte[] rawVin) { private void logVehicleLogout(ChannelHandlerContext ctx, Gb32960Message msg) {
log.info("[gb32960] vehicle logout peer={} vin={}", addr(ctx), msg.header().vin()); log.info("[gb32960] vehicle logout peer={} vin={}", addr(ctx), msg.header().vin());
}
private void writeVehicleLogoutAck(ChannelHandlerContext ctx, Gb32960Message msg, byte[] rawVin) {
ackService.writeResponse(ctx, ackService.writeResponse(ctx,
msg.header().protocolVersion(), msg.header().protocolVersion(),
CommandType.VEHICLE_LOGOUT, CommandType.VEHICLE_LOGOUT,
@@ -189,8 +290,8 @@ public class Gb32960ChannelHandler extends SimpleChannelInboundHandler<byte[]> {
* <ul> * <ul>
* <li>消息体解析缺失:关闭连接,返回 false 短路 * <li>消息体解析缺失:关闭连接,返回 false 短路
* <li>鉴权失败:写 {@code 0x02 OTHER_ERROR} NACK + 关闭连接,返回 false 短路 * <li>鉴权失败:写 {@code 0x02 OTHER_ERROR} NACK + 关闭连接,返回 false 短路
* <li>鉴权通过:把 username 钉到 channel attribute供后续 vendor profile 路由), * <li>鉴权通过:把 username 钉到 channel attribute供后续 vendor profile 路由),INFO 日志,
* 写成功 ACK + INFO 日志,返回 true 继续 dispatch * 返回 true 继续 dispatch;成功 ACK 等 dispatch 持久化边界完成后再写
* </ul> * </ul>
* *
* @return true 表示鉴权通过应继续 dispatchfalse 表示调用方应 short-circuit return * @return true 表示鉴权通过应继续 dispatchfalse 表示调用方应 short-circuit return
@@ -221,6 +322,10 @@ public class Gb32960ChannelHandler extends SimpleChannelInboundHandler<byte[]> {
addr(ctx), pl.username(), pl.encryptRule(), pl.serialNo(), pl.loginTime(), result); addr(ctx), pl.username(), pl.encryptRule(), pl.serialNo(), pl.loginTime(), result);
// 把 username 钉到 channel attribute供后续 0x02/0x03 帧的 vendor profile 路由 // 把 username 钉到 channel attribute供后续 0x02/0x03 帧的 vendor profile 路由
accessService.bindPlatformAccount(ctx.channel(), pl.username()); accessService.bindPlatformAccount(ctx.channel(), pl.username());
return true;
}
private void writePlatformLoginAck(ChannelHandlerContext ctx, Gb32960Message msg, byte[] rawVin) {
ackService.writeResponse( ackService.writeResponse(
ctx, ctx,
msg.header().protocolVersion(), msg.header().protocolVersion(),
@@ -230,12 +335,14 @@ public class Gb32960ChannelHandler extends SimpleChannelInboundHandler<byte[]> {
eventOrNow(msg), eventOrNow(msg),
null, null,
"platform-login-ack"); "platform-login-ack");
return true;
} }
/** 0x06 平台登出INFO 日志 + 带原采集时间回成功 ACK。 */ /** 0x06 平台登出INFO 日志 + 带原采集时间回成功 ACK。 */
private void handlePlatformLogout(ChannelHandlerContext ctx, Gb32960Message msg, byte[] rawVin) { private void logPlatformLogout(ChannelHandlerContext ctx) {
log.info("[gb32960] platform logout peer={}", addr(ctx)); log.info("[gb32960] platform logout peer={}", addr(ctx));
}
private void writePlatformLogoutAck(ChannelHandlerContext ctx, Gb32960Message msg, byte[] rawVin) {
ackService.writeResponse(ctx, ackService.writeResponse(ctx,
msg.header().protocolVersion(), msg.header().protocolVersion(),
CommandType.PLATFORM_LOGOUT, CommandType.PLATFORM_LOGOUT,
@@ -247,11 +354,11 @@ public class Gb32960ChannelHandler extends SimpleChannelInboundHandler<byte[]> {
} }
/** /**
* 0x02/0x03/0x07 实时 / 补发 / 心跳:按 §6.3.2 回应答帧,保留原帧采集时间 * 0x02/0x03/0x07 实时 / 补发 / 心跳:记录帧诊断;成功 ACK 在配置的 Kafka dispatch 边界后写回
* 部分车端实现严格按规范没收到应答会停止后续上报或重连。ack-tag 按 cmd.code 区分便于日志筛选。 * 部分车端实现严格按规范没收到应答会停止后续上报或重连。ack-tag 按 cmd.code 区分便于日志筛选。
* *
* <p>注意:应答成功不代表下游已完成持久化,只表示平台已接收并通过 Dispatcher 投递 * <p>注意:应答成功表示当前配置的 Kafka sink 已完成该帧映射事件的 dispatch 边界
* 生产排查存储问题时,应继续检查 raw archive sink、EventFileStoreSink 和 Kafka sink 的日志。 * 生产排查其他存储问题时,应继续检查 raw archive sink、EventFileStoreSink 等可选 sink 的日志。
* *
* <p>日志策略(按 (channel, vin, rawTypeSignature) 去重,避免高频帧刷屏): * <p>日志策略(按 (channel, vin, rawTypeSignature) 去重,避免高频帧刷屏):
* <ul> * <ul>
@@ -264,17 +371,9 @@ public class Gb32960ChannelHandler extends SimpleChannelInboundHandler<byte[]> {
* </ul> * </ul>
* 帧内字段全量 JSON 仍保留 DEBUG。 * 帧内字段全量 JSON 仍保留 DEBUG。
*/ */
private void handleReportOrHeartbeat(ChannelHandlerContext ctx, Gb32960Message msg, byte[] rawVin) { private void logReportOrHeartbeat(ChannelHandlerContext ctx, Gb32960Message msg) {
CommandType cmd = msg.header().command(); CommandType cmd = msg.header().command();
String platformAccount = accessService.platformAccount(ctx.channel()); String platformAccount = accessService.platformAccount(ctx.channel());
ackService.writeResponse(ctx,
msg.header().protocolVersion(),
cmd,
ResponseFlag.SUCCESS,
rawVin,
eventOrNow(msg),
null,
"report-ack-0x" + Integer.toHexString(cmd.code()));
if (cmd == CommandType.HEARTBEAT) { if (cmd == CommandType.HEARTBEAT) {
if (log.isDebugEnabled()) { if (log.isDebugEnabled()) {
log.debug("[gb32960] HEARTBEAT peer={} vin={} platformAccount={}", log.debug("[gb32960] HEARTBEAT peer={} vin={} platformAccount={}",
@@ -304,8 +403,20 @@ public class Gb32960ChannelHandler extends SimpleChannelInboundHandler<byte[]> {
} }
} }
private void writeReportOrHeartbeatAck(ChannelHandlerContext ctx, Gb32960Message msg, byte[] rawVin) {
CommandType cmd = msg.header().command();
ackService.writeResponse(ctx,
msg.header().protocolVersion(),
cmd,
ResponseFlag.SUCCESS,
rawVin,
eventOrNow(msg),
null,
"report-ack-0x" + Integer.toHexString(cmd.code()));
}
/** 0x08 终端校时:平台应答 data 段为平台当前时间 6B用 Instant.now())。 */ /** 0x08 终端校时:平台应答 data 段为平台当前时间 6B用 Instant.now())。 */
private void handleTimeCalibration(ChannelHandlerContext ctx, Gb32960Message msg, byte[] rawVin) { private void writeTimeCalibrationAck(ChannelHandlerContext ctx, Gb32960Message msg, byte[] rawVin) {
ackService.writeResponse(ctx, ackService.writeResponse(ctx,
msg.header().protocolVersion(), msg.header().protocolVersion(),
CommandType.TIME_CALIBRATION, CommandType.TIME_CALIBRATION,

View File

@@ -0,0 +1,37 @@
package com.lingniu.ingest.protocol.gb32960.config;
import com.lingniu.ingest.core.dispatcher.Dispatcher;
import com.lingniu.ingest.protocol.gb32960.codec.Gb32960MessageDecoder;
import com.lingniu.ingest.protocol.gb32960.inbound.Gb32960NettyServer;
import org.junit.jupiter.api.Test;
import org.springframework.boot.autoconfigure.AutoConfigurations;
import org.springframework.boot.test.context.runner.ApplicationContextRunner;
import static org.assertj.core.api.Assertions.assertThat;
class Gb32960AutoConfigurationTest {
private final ApplicationContextRunner runner = new ApplicationContextRunner()
.withConfiguration(AutoConfigurations.of(Gb32960AutoConfiguration.class))
.withBean(Dispatcher.class, () -> new Dispatcher(null, null, null, null, null))
.withPropertyValues(
"lingniu.ingest.gb32960.enabled=true",
"lingniu.ingest.gb32960.port=0");
@Test
void createsDecoderWithoutTcpServerWhenServerDisabled() {
runner.withPropertyValues("lingniu.ingest.gb32960.server.enabled=false")
.run(context -> {
assertThat(context).hasSingleBean(Gb32960MessageDecoder.class);
assertThat(context).doesNotHaveBean(Gb32960NettyServer.class);
});
}
@Test
void createsTcpServerByDefaultWhenProtocolEnabled() {
runner.run(context -> {
assertThat(context).hasSingleBean(Gb32960MessageDecoder.class);
assertThat(context).hasSingleBean(Gb32960NettyServer.class);
});
}
}

View File

@@ -0,0 +1,212 @@
package com.lingniu.ingest.protocol.gb32960.inbound;
import com.lingniu.ingest.api.event.VehicleEvent;
import com.lingniu.ingest.api.sink.EventSink;
import com.lingniu.ingest.codec.BccChecksum;
import com.lingniu.ingest.core.concurrency.AsyncBatchExecutor;
import com.lingniu.ingest.core.concurrency.DisruptorEventBus;
import com.lingniu.ingest.core.dispatcher.Dispatcher;
import com.lingniu.ingest.core.dispatcher.HandlerInvoker;
import com.lingniu.ingest.core.dispatcher.HandlerRegistry;
import com.lingniu.ingest.core.pipeline.InterceptorChain;
import com.lingniu.ingest.protocol.gb32960.auth.Gb32960PlatformAuthorizer;
import com.lingniu.ingest.protocol.gb32960.auth.Gb32960VinAuthorizer;
import com.lingniu.ingest.protocol.gb32960.codec.Gb32960BodyParser;
import com.lingniu.ingest.protocol.gb32960.codec.Gb32960DecoderTest;
import com.lingniu.ingest.protocol.gb32960.codec.Gb32960MessageDecoder;
import com.lingniu.ingest.protocol.gb32960.codec.InfoBlockParserRegistry;
import com.lingniu.ingest.protocol.gb32960.codec.parser.v2016.PositionV2016BlockParser;
import com.lingniu.ingest.protocol.gb32960.codec.parser.v2016.VehicleV2016BlockParser;
import com.lingniu.ingest.protocol.gb32960.config.Gb32960Properties;
import io.netty.buffer.ByteBuf;
import io.netty.channel.embedded.EmbeddedChannel;
import org.junit.jupiter.api.Test;
import java.io.ByteArrayOutputStream;
import java.nio.charset.StandardCharsets;
import java.util.ArrayList;
import java.util.List;
import java.util.concurrent.CompletableFuture;
import static org.assertj.core.api.Assertions.assertThat;
class Gb32960ChannelHandlerAckBoundaryTest {
@Test
void realtimeReportAckWaitsForDispatchFutureCompletion() {
try (DispatchHarness dispatch = new DispatchHarness()) {
EmbeddedChannel channel = new EmbeddedChannel(handlerWith(dispatch.dispatcher()));
assertThat(channel.writeInbound(Gb32960DecoderTest.buildRealtimeFrame("LTEST000000000001"))).isFalse();
assertThat((Object) channel.readOutbound()).isNull();
dispatch.awaitPublishCount(1);
dispatch.completeDurability();
channel.runPendingTasks();
ByteBuf ack = channel.readOutbound();
assertThat(ack).isNotNull();
assertThat(ack.getUnsignedByte(2)).isEqualTo((short) 0x02);
assertThat(ack.getUnsignedByte(3)).isEqualTo((short) 0x01);
ack.release();
channel.finishAndReleaseAll();
}
}
@Test
void realtimeReportAckIsNotWrittenWhenDispatchFutureFails() {
try (DispatchHarness dispatch = new DispatchHarness()) {
EmbeddedChannel channel = new EmbeddedChannel(handlerWith(dispatch.dispatcher()));
assertThat(channel.writeInbound(Gb32960DecoderTest.buildRealtimeFrame("LTEST000000000001"))).isFalse();
dispatch.awaitPublishCount(1);
dispatch.failDurability();
channel.runPendingTasks();
assertThat((Object) channel.readOutbound()).isNull();
assertThat(channel.isActive()).isFalse();
channel.finishAndReleaseAll();
}
}
@Test
void durableAcksAreWrittenInInboundOrderWhenLaterDispatchCompletesFirst() {
try (DispatchHarness dispatch = new DispatchHarness()) {
EmbeddedChannel channel = new EmbeddedChannel(handlerWith(dispatch.dispatcher()));
assertThat(channel.writeInbound(Gb32960DecoderTest.buildRealtimeFrame("LTEST000000000001"))).isFalse();
assertThat(channel.writeInbound(buildHeartbeatFrame("LTEST000000000001"))).isFalse();
assertThat((Object) channel.readOutbound()).isNull();
dispatch.awaitPublishCount(2);
dispatch.completeDurability(1);
channel.runPendingTasks();
assertThat((Object) channel.readOutbound()).isNull();
dispatch.completeDurability(0);
channel.runPendingTasks();
ByteBuf firstAck = channel.readOutbound();
ByteBuf secondAck = channel.readOutbound();
assertThat(firstAck).isNotNull();
assertThat(secondAck).isNotNull();
assertThat(firstAck.getUnsignedByte(2)).isEqualTo((short) 0x02);
assertThat(secondAck.getUnsignedByte(2)).isEqualTo((short) 0x07);
firstAck.release();
secondAck.release();
channel.finishAndReleaseAll();
}
}
private static Gb32960ChannelHandler handlerWith(Dispatcher dispatcher) {
return new Gb32960ChannelHandler(
decoder(),
dispatcher,
accessService(),
new Gb32960AckService(),
new Gb32960FrameDiagnostics(16));
}
private static Gb32960MessageDecoder decoder() {
return new Gb32960MessageDecoder(new Gb32960BodyParser(new InfoBlockParserRegistry(List.of(
new VehicleV2016BlockParser(),
new PositionV2016BlockParser()))));
}
private static Gb32960AccessService accessService() {
Gb32960Properties.Auth auth = new Gb32960Properties.Auth();
auth.setEnabled(false);
return new Gb32960AccessService(
new Gb32960VinAuthorizer(auth),
new Gb32960PlatformAuthorizer(auth.getPlatforms()));
}
private static byte[] buildHeartbeatFrame(String vin) {
ByteArrayOutputStream out = new ByteArrayOutputStream();
out.write(0x23);
out.write(0x23);
out.write(0x07);
out.write(0xFE);
byte[] vinBytes = vin.getBytes(StandardCharsets.US_ASCII);
out.write(vinBytes, 0, 17);
out.write(0x01);
out.write(0x00);
out.write(0x00);
byte[] almost = out.toByteArray();
out.write(BccChecksum.compute(almost, 2, almost.length - 2) & 0xFF);
return out.toByteArray();
}
private static final class DispatchHarness implements AutoCloseable {
private final ControlledSink sink = new ControlledSink();
private final DisruptorEventBus eventBus = new DisruptorEventBus(1024, "blocking", List.of(sink));
private final AsyncBatchExecutor batchExecutor = new AsyncBatchExecutor(eventBus::publish);
private final Dispatcher dispatcher = new Dispatcher(
new HandlerRegistry(),
new InterceptorChain(List.of()),
new HandlerInvoker(),
eventBus,
batchExecutor);
Dispatcher dispatcher() {
return dispatcher;
}
void completeDurability() {
completeDurability(0);
}
void completeDurability(int index) {
sink.future(index).complete(null);
}
void failDurability() {
sink.future(0).completeExceptionally(new IllegalStateException("durability failed"));
}
void awaitPublishCount(int count) {
long deadline = System.currentTimeMillis() + 3000;
while (System.currentTimeMillis() < deadline) {
synchronized (sink.futures) {
if (sink.futures.size() >= count) {
return;
}
}
Thread.onSpinWait();
}
throw new AssertionError("expected " + count + " sink publishes, actual=" + sink.futures.size());
}
@Override
public void close() {
batchExecutor.close();
eventBus.close();
}
}
private static final class ControlledSink implements EventSink {
private final List<CompletableFuture<Void>> futures = new ArrayList<>();
@Override
public String name() {
return "kafka";
}
@Override
public CompletableFuture<Void> publish(VehicleEvent event) {
CompletableFuture<Void> future = new CompletableFuture<>();
synchronized (futures) {
futures.add(future);
}
return future;
}
private CompletableFuture<Void> future(int index) {
synchronized (futures) {
return futures.get(index);
}
}
}
}

View File

@@ -1,7 +1,11 @@
package com.lingniu.ingest.eventhistory; package com.lingniu.ingest.eventhistory;
import com.fasterxml.jackson.core.JsonProcessingException;
import com.fasterxml.jackson.databind.ObjectMapper;
import com.lingniu.ingest.api.ProtocolId; import com.lingniu.ingest.api.ProtocolId;
import com.lingniu.ingest.api.event.RawArchiveKeys;
import com.lingniu.ingest.eventfilestore.EventFileRecord; import com.lingniu.ingest.eventfilestore.EventFileRecord;
import com.lingniu.ingest.sink.mq.proto.RawArchiveRef;
import com.lingniu.ingest.sink.mq.proto.VehicleEnvelope; import com.lingniu.ingest.sink.mq.proto.VehicleEnvelope;
import com.google.protobuf.InvalidProtocolBufferException; import com.google.protobuf.InvalidProtocolBufferException;
import com.google.protobuf.util.JsonFormat; import com.google.protobuf.util.JsonFormat;
@@ -21,20 +25,24 @@ import java.util.Map;
public final class TelemetryEnvelopeRecordMapper { public final class TelemetryEnvelopeRecordMapper {
private static final JsonFormat.Printer JSON_PRINTER = JsonFormat.printer(); private static final JsonFormat.Printer JSON_PRINTER = JsonFormat.printer();
private static final ObjectMapper OBJECT_MAPPER = new ObjectMapper();
public EventFileRecord toRecord(VehicleEnvelope envelope) { public EventFileRecord toRecord(VehicleEnvelope envelope) {
if (envelope == null) { if (envelope == null) {
throw new IllegalArgumentException("envelope must not be null"); throw new IllegalArgumentException("envelope must not be null");
} }
if (!envelope.hasTelemetrySnapshot()) { if (envelope.hasTelemetrySnapshot()) {
throw new IllegalArgumentException("envelope telemetry_snapshot is required"); return telemetryRecord(envelope);
} }
if (envelope.hasRawArchive()) {
return rawArchiveRecord(envelope);
}
throw new IllegalArgumentException("envelope telemetry_snapshot or raw_archive is required");
}
private static EventFileRecord telemetryRecord(VehicleEnvelope envelope) {
ProtocolId protocol = protocol(envelope.getSource()); ProtocolId protocol = protocol(envelope.getSource());
Map<String, String> metadata = new LinkedHashMap<>(envelope.getMetadataMap()); Map<String, String> metadata = metadata(envelope, protocol);
if (protocol == ProtocolId.UNKNOWN && !envelope.getSource().isBlank()) {
// 保留未知 source 原文,避免历史库标准化成 UNKNOWN 后丢失排障线索。
metadata.putIfAbsent("originalSource", envelope.getSource());
}
return new EventFileRecord( return new EventFileRecord(
envelope.getEventId(), envelope.getEventId(),
protocol, protocol,
@@ -48,6 +56,57 @@ public final class TelemetryEnvelopeRecordMapper {
); );
} }
private static EventFileRecord rawArchiveRecord(VehicleEnvelope envelope) {
ProtocolId protocol = protocol(envelope.getSource());
RawArchiveRef rawArchive = envelope.getRawArchive();
Map<String, String> metadata = metadata(envelope, protocol);
String rawArchiveKey = rawArchiveKey(rawArchive.getUri(), metadata);
String rawArchiveUri = rawArchive.getUri().isBlank() && !rawArchiveKey.isBlank()
? RawArchiveKeys.logicalUri(rawArchiveKey)
: rawArchive.getUri();
metadata.putIfAbsent(RawArchiveKeys.META_EVENT_ID, envelope.getEventId());
if (!rawArchiveKey.isBlank()) {
metadata.putIfAbsent(RawArchiveKeys.META_KEY, rawArchiveKey);
}
if (!rawArchiveUri.isBlank()) {
metadata.putIfAbsent(RawArchiveKeys.META_URI, rawArchiveUri);
}
return new EventFileRecord(
envelope.getEventId(),
protocol,
"RAW_ARCHIVE",
envelope.getVin(),
Instant.ofEpochMilli(envelope.getEventTimeMs()),
Instant.ofEpochMilli(envelope.getIngestTimeMs()),
rawArchiveUri,
metadata,
rawArchiveJson(envelope, metadata, rawArchiveKey, rawArchiveUri)
);
}
private static String rawArchiveKey(String rawArchiveUri, Map<String, String> metadata) {
String key = metadata.getOrDefault(RawArchiveKeys.META_KEY, "");
if (!key.isBlank()) {
return key;
}
if (rawArchiveUri != null && rawArchiveUri.startsWith("archive://")) {
return rawArchiveUri.substring("archive://".length());
}
return "";
}
private static Map<String, String> metadata(VehicleEnvelope envelope, ProtocolId protocol) {
Map<String, String> metadata = new LinkedHashMap<>(envelope.getMetadataMap());
if (!envelope.getProtocolVersion().isBlank()) {
metadata.putIfAbsent("protocolVersion", envelope.getProtocolVersion());
}
if (protocol == ProtocolId.UNKNOWN && !envelope.getSource().isBlank()) {
// 保留未知 source 原文,避免历史库标准化成 UNKNOWN 后丢失排障线索。
metadata.putIfAbsent("originalSource", envelope.getSource());
}
return metadata;
}
private static ProtocolId protocol(String source) { private static ProtocolId protocol(String source) {
if (source == null || source.isBlank()) { if (source == null || source.isBlank()) {
return ProtocolId.UNKNOWN; return ProtocolId.UNKNOWN;
@@ -66,4 +125,36 @@ public final class TelemetryEnvelopeRecordMapper {
throw new IllegalArgumentException("failed to serialize telemetry_snapshot", e); throw new IllegalArgumentException("failed to serialize telemetry_snapshot", e);
} }
} }
private static String rawArchiveJson(VehicleEnvelope envelope,
Map<String, String> metadata,
String rawArchiveKey,
String rawArchiveUri) {
Map<String, Object> payload = new LinkedHashMap<>();
RawArchiveRef rawArchive = envelope.getRawArchive();
payload.put("eventId", envelope.getEventId());
payload.put("vin", envelope.getVin());
payload.put("protocol", protocol(envelope.getSource()).name());
if (!envelope.getSource().isBlank()) {
payload.put("source", envelope.getSource());
}
payload.put("eventType", "RAW_ARCHIVE");
payload.put("eventTime", Instant.ofEpochMilli(envelope.getEventTimeMs()).toString());
payload.put("ingestTime", Instant.ofEpochMilli(envelope.getIngestTimeMs()).toString());
payload.put("rawArchiveUri", rawArchiveUri);
payload.put("rawArchiveKey", rawArchiveKey);
if (metadata.containsKey("command")) {
payload.put("command", metadata.get("command"));
}
if (metadata.containsKey("infoType")) {
payload.put("infoType", metadata.get("infoType"));
}
payload.put("rawSizeBytes", rawArchive.getSizeBytes());
payload.put("metadata", metadata);
try {
return OBJECT_MAPPER.writeValueAsString(payload);
} catch (JsonProcessingException e) {
throw new IllegalArgumentException("failed to serialize raw_archive", e);
}
}
} }

View File

@@ -1,16 +1,22 @@
package com.lingniu.ingest.eventhistory; package com.lingniu.ingest.eventhistory;
import com.fasterxml.jackson.databind.JsonNode;
import com.fasterxml.jackson.databind.ObjectMapper;
import com.lingniu.ingest.api.ProtocolId;
import com.lingniu.ingest.api.consumer.EnvelopeIngestResult; import com.lingniu.ingest.api.consumer.EnvelopeIngestResult;
import com.lingniu.ingest.api.consumer.EnvelopeIngestor; import com.lingniu.ingest.api.consumer.EnvelopeIngestor;
import com.lingniu.ingest.api.event.RawArchiveKeys;
import com.lingniu.ingest.eventfilestore.EventFileQuery; import com.lingniu.ingest.eventfilestore.EventFileQuery;
import com.lingniu.ingest.eventfilestore.EventFileRecord; import com.lingniu.ingest.eventfilestore.EventFileRecord;
import com.lingniu.ingest.eventfilestore.EventFileStore; import com.lingniu.ingest.eventfilestore.EventFileStore;
import com.lingniu.ingest.sink.mq.proto.RawArchiveRef;
import com.lingniu.ingest.sink.mq.proto.TelemetryField; import com.lingniu.ingest.sink.mq.proto.TelemetryField;
import com.lingniu.ingest.sink.mq.proto.TelemetrySnapshot; import com.lingniu.ingest.sink.mq.proto.TelemetrySnapshot;
import com.lingniu.ingest.sink.mq.proto.VehicleEnvelope; import com.lingniu.ingest.sink.mq.proto.VehicleEnvelope;
import org.junit.jupiter.api.Test; import org.junit.jupiter.api.Test;
import java.io.IOException; import java.io.IOException;
import java.time.Instant;
import java.util.ArrayList; import java.util.ArrayList;
import java.util.List; import java.util.List;
@@ -19,6 +25,8 @@ import static org.assertj.core.api.Assertions.assertThatThrownBy;
class EventHistoryEnvelopeIngestorTest { class EventHistoryEnvelopeIngestorTest {
private static final ObjectMapper OBJECT_MAPPER = new ObjectMapper();
@Test @Test
void parsesKafkaEnvelopeBytesAndAppendsRecord() throws Exception { void parsesKafkaEnvelopeBytesAndAppendsRecord() throws Exception {
CapturingStore store = new CapturingStore(); CapturingStore store = new CapturingStore();
@@ -55,6 +63,61 @@ class EventHistoryEnvelopeIngestorTest {
assertThat(store.records).isEmpty(); assertThat(store.records).isEmpty();
} }
@Test
void tryIngestRawArchiveEnvelopeAppendsRawArchiveRecord() throws Exception {
CapturingStore store = new CapturingStore();
EventHistoryEnvelopeIngestor ingestor =
new EventHistoryEnvelopeIngestor(store, new TelemetryEnvelopeRecordMapper());
String rawArchiveKey = "2026/06/23/GB32960/VINRAW001/raw-event-1.bin";
String rawArchiveUri = "archive://" + rawArchiveKey;
VehicleEnvelope envelope = VehicleEnvelope.newBuilder()
.setSchemaVersion("1.0")
.setEventId("raw-event-1")
.setVin("VINRAW001")
.setSource("GB32960")
.setProtocolVersion("V2016")
.setEventTimeMs(1_782_112_400_000L)
.setIngestTimeMs(1_782_112_401_000L)
.putMetadata(RawArchiveKeys.META_KEY, rawArchiveKey)
.putMetadata(RawArchiveKeys.META_URI, rawArchiveUri)
.putMetadata("command", "0x0002")
.setRawArchive(RawArchiveRef.newBuilder()
.setUri(rawArchiveUri)
.setSizeBytes(128)
.build())
.build();
EnvelopeIngestResult result = ingestor.tryIngest(envelope.toByteArray());
assertThat(result.status()).isEqualTo(EnvelopeIngestResult.Status.STORED);
assertThat(store.records).hasSize(1);
EventFileRecord record = store.records.getFirst();
assertThat(record.eventId()).isEqualTo("raw-event-1");
assertThat(record.protocol()).isEqualTo(ProtocolId.GB32960);
assertThat(record.eventType()).isEqualTo("RAW_ARCHIVE");
assertThat(record.vin()).isEqualTo("VINRAW001");
assertThat(record.eventTime()).isEqualTo(Instant.ofEpochMilli(1_782_112_400_000L));
assertThat(record.rawArchiveUri()).isEqualTo(rawArchiveUri);
assertThat(record.metadata())
.containsEntry("protocolVersion", "V2016")
.containsEntry(RawArchiveKeys.META_EVENT_ID, "raw-event-1")
.containsEntry(RawArchiveKeys.META_KEY, rawArchiveKey)
.containsEntry(RawArchiveKeys.META_URI, rawArchiveUri)
.containsEntry("command", "0x0002");
JsonNode payload = OBJECT_MAPPER.readTree(record.payloadJson());
assertThat(payload.get("eventId").asText()).isEqualTo("raw-event-1");
assertThat(payload.get("protocol").asText()).isEqualTo("GB32960");
assertThat(payload.get("eventType").asText()).isEqualTo("RAW_ARCHIVE");
assertThat(payload.get("vin").asText()).isEqualTo("VINRAW001");
assertThat(payload.get("eventTime").asText()).isEqualTo("2026-06-22T07:13:20Z");
assertThat(payload.get("rawArchiveUri").asText()).isEqualTo(rawArchiveUri);
assertThat(payload.get("rawArchiveKey").asText()).isEqualTo(rawArchiveKey);
assertThat(payload.get("rawSizeBytes").asLong()).isEqualTo(128);
assertThat(payload.get("metadata").get("command").asText()).isEqualTo("0x0002");
assertThat(payload.toString()).doesNotContain("rawBytes");
}
private static VehicleEnvelope envelope(String eventId) { private static VehicleEnvelope envelope(String eventId) {
return VehicleEnvelope.newBuilder() return VehicleEnvelope.newBuilder()
.setEventId(eventId) .setEventId(eventId)

View File

@@ -17,7 +17,11 @@ public final class VehicleStatEnvelopeIngestor implements EnvelopeIngestor {
} }
public void ingest(byte[] kafkaValue) { public void ingest(byte[] kafkaValue) {
processor.process(parse(kafkaValue)); VehicleEnvelope envelope = parse(kafkaValue);
if (!envelope.hasTelemetrySnapshot()) {
return;
}
processor.process(envelope);
} }
@Override @Override

View File

@@ -2,6 +2,7 @@ package com.lingniu.ingest.vehiclestat;
import com.lingniu.ingest.api.consumer.EnvelopeIngestResult; import com.lingniu.ingest.api.consumer.EnvelopeIngestResult;
import com.lingniu.ingest.api.consumer.EnvelopeIngestor; import com.lingniu.ingest.api.consumer.EnvelopeIngestor;
import com.lingniu.ingest.sink.mq.proto.RawArchiveRef;
import com.lingniu.ingest.sink.mq.proto.TelemetryField; import com.lingniu.ingest.sink.mq.proto.TelemetryField;
import com.lingniu.ingest.sink.mq.proto.TelemetrySnapshot; import com.lingniu.ingest.sink.mq.proto.TelemetrySnapshot;
import com.lingniu.ingest.sink.mq.proto.VehicleEnvelope; import com.lingniu.ingest.sink.mq.proto.VehicleEnvelope;
@@ -74,6 +75,23 @@ class VehicleStatEnvelopeIngestorTest {
assertThat(repository.mileagePoints("VIN001", LocalDate.of(2026, 6, 22))).isEmpty(); assertThat(repository.mileagePoints("VIN001", LocalDate.of(2026, 6, 22))).isEmpty();
} }
@Test
void rawArchiveEnvelopeIsIgnoredWithoutAppendingMileagePoint() {
InMemoryVehicleStatRepository repository = new InMemoryVehicleStatRepository();
VehicleStatEnvelopeIngestor ingestor =
new VehicleStatEnvelopeIngestor(new VehicleStatEventProcessor(repository));
VehicleEnvelope envelope = rawArchiveEnvelope();
ingestor.ingest(envelope.toByteArray());
var result = ingestor.tryIngest(envelope.toByteArray());
assertThat(result.status()).isEqualTo(EnvelopeIngestResult.Status.SKIPPED);
assertThat(result.eventId()).isEqualTo("raw-event-1");
assertThat(result.vin()).isEqualTo("VIN001");
assertThat(result.message()).contains("telemetry_snapshot");
assertThat(repository.mileagePoints("VIN001", LocalDate.of(2026, 6, 22))).isEmpty();
}
private static VehicleEnvelope envelope() { private static VehicleEnvelope envelope() {
return VehicleEnvelope.newBuilder() return VehicleEnvelope.newBuilder()
.setEventId("event-1") .setEventId("event-1")
@@ -91,6 +109,20 @@ class VehicleStatEnvelopeIngestorTest {
.build(); .build();
} }
private static VehicleEnvelope rawArchiveEnvelope() {
return VehicleEnvelope.newBuilder()
.setEventId("raw-event-1")
.setVin("VIN001")
.setSource("GB32960")
.setEventTimeMs(1_782_112_400_000L)
.setIngestTimeMs(1_782_112_401_000L)
.setRawArchive(RawArchiveRef.newBuilder()
.setUri("archive://2026/06/23/GB32960/VIN001/raw-event-1.bin")
.setSizeBytes(128)
.build())
.build();
}
private static final class InMemoryVehicleStatRepository implements VehicleStatRepository { private static final class InMemoryVehicleStatRepository implements VehicleStatRepository {
private final java.util.Map<String, java.util.List<MileagePoint>> points = new java.util.HashMap<>(); private final java.util.Map<String, java.util.List<MileagePoint>> points = new java.util.HashMap<>();

View File

@@ -17,7 +17,11 @@ public final class VehicleStateEnvelopeIngestor implements EnvelopeIngestor {
} }
public void ingest(byte[] kafkaValue) { public void ingest(byte[] kafkaValue) {
updater.update(parse(kafkaValue)); VehicleEnvelope envelope = parse(kafkaValue);
if (!envelope.hasTelemetrySnapshot()) {
return;
}
updater.update(envelope);
} }
@Override @Override
@@ -25,6 +29,10 @@ public final class VehicleStateEnvelopeIngestor implements EnvelopeIngestor {
VehicleEnvelope envelope = null; VehicleEnvelope envelope = null;
try { try {
envelope = parse(kafkaValue); envelope = parse(kafkaValue);
if (!envelope.hasTelemetrySnapshot()) {
return EnvelopeIngestResult.skipped(
envelope.getEventId(), envelope.getVin(), "envelope telemetry_snapshot is required");
}
// Kafka 消费路径只接受标准 VehicleEnvelope坏消息返回明确结果给处理器写 DLQ。 // Kafka 消费路径只接受标准 VehicleEnvelope坏消息返回明确结果给处理器写 DLQ。
updater.update(envelope); updater.update(envelope);
return EnvelopeIngestResult.processed(envelope.getEventId(), envelope.getVin()); return EnvelopeIngestResult.processed(envelope.getEventId(), envelope.getVin());

View File

@@ -2,6 +2,7 @@ package com.lingniu.ingest.vehiclestate;
import com.lingniu.ingest.api.consumer.EnvelopeIngestResult; import com.lingniu.ingest.api.consumer.EnvelopeIngestResult;
import com.lingniu.ingest.api.consumer.EnvelopeIngestor; import com.lingniu.ingest.api.consumer.EnvelopeIngestor;
import com.lingniu.ingest.sink.mq.proto.RawArchiveRef;
import com.lingniu.ingest.sink.mq.proto.TelemetryField; import com.lingniu.ingest.sink.mq.proto.TelemetryField;
import com.lingniu.ingest.sink.mq.proto.TelemetrySnapshot; import com.lingniu.ingest.sink.mq.proto.TelemetrySnapshot;
import com.lingniu.ingest.sink.mq.proto.VehicleEnvelope; import com.lingniu.ingest.sink.mq.proto.VehicleEnvelope;
@@ -47,6 +48,25 @@ class VehicleStateEnvelopeIngestorTest {
assertThat(repository.getState("VIN001")).isEmpty(); assertThat(repository.getState("VIN001")).isEmpty();
} }
@Test
void rawArchiveEnvelopeIsIgnoredWithoutUpdatingRepository() {
InMemoryVehicleStateRepository repository = new InMemoryVehicleStateRepository();
VehicleStateEnvelopeIngestor ingestor = new VehicleStateEnvelopeIngestor(new VehicleStateUpdater(repository));
VehicleEnvelope envelope = rawArchiveEnvelope();
ingestor.ingest(envelope.toByteArray());
var result = ingestor.tryIngest(envelope.toByteArray());
assertThat(result.status()).isEqualTo(EnvelopeIngestResult.Status.SKIPPED);
assertThat(result.eventId()).isEqualTo("raw-event-1");
assertThat(result.vin()).isEqualTo("VIN001");
assertThat(result.message()).contains("telemetry_snapshot");
assertThat(repository.getState("VIN001")).isEmpty();
assertThat(repository.getLocation("VIN001")).isEmpty();
assertThat(repository.getSafety("VIN001")).isEmpty();
assertThat(repository.getLastEvent("VIN001")).isEmpty();
}
private static VehicleEnvelope envelope() { private static VehicleEnvelope envelope() {
return VehicleEnvelope.newBuilder() return VehicleEnvelope.newBuilder()
.setEventId("event-1") .setEventId("event-1")
@@ -64,6 +84,20 @@ class VehicleStateEnvelopeIngestorTest {
.build(); .build();
} }
private static VehicleEnvelope rawArchiveEnvelope() {
return VehicleEnvelope.newBuilder()
.setEventId("raw-event-1")
.setVin("VIN001")
.setSource("GB32960")
.setEventTimeMs(1_782_112_400_000L)
.setIngestTimeMs(1_782_112_401_000L)
.setRawArchive(RawArchiveRef.newBuilder()
.setUri("archive://2026/06/23/GB32960/VIN001/raw-event-1.bin")
.setSizeBytes(128)
.build())
.build();
}
private static final class EmptyRepository implements VehicleStateRepository { private static final class EmptyRepository implements VehicleStateRepository {
@Override public void putState(String vin, String json) {} @Override public void putState(String vin, String json) {}
@Override public void putLocation(String vin, String json) {} @Override public void putLocation(String vin, String json) {}

View File

@@ -0,0 +1,17 @@
package com.lingniu.ingest.sink.archive;
import java.io.IOException;
import java.io.InputStream;
public interface ArchiveStore {
String put(String key, InputStream data, long length) throws IOException;
String append(String key, byte[] chunk) throws IOException;
InputStream get(String key) throws IOException;
boolean exists(String key);
long size(String key) throws IOException;
}

View File

@@ -0,0 +1,150 @@
package com.lingniu.ingest.sink.archive;
import java.io.IOException;
import java.io.InputStream;
import java.io.OutputStream;
import java.net.URI;
import java.nio.file.Files;
import java.nio.file.LinkOption;
import java.nio.file.Path;
import java.nio.file.StandardOpenOption;
import java.util.concurrent.ConcurrentHashMap;
import java.util.concurrent.ConcurrentMap;
public final class LocalArchiveStore implements ArchiveStore {
private final Path root;
private final ConcurrentMap<Path, Object> appendLocks = new ConcurrentHashMap<>();
public LocalArchiveStore(String root) {
this(rootPath(root));
}
public LocalArchiveStore(Path root) {
this.root = root.toAbsolutePath().normalize();
}
@Override
public String put(String key, InputStream data, long length) throws IOException {
if (data == null) {
throw new IllegalArgumentException("data must not be null");
}
Path target = resolve(key);
createDirectoriesInsideRoot(target.getParent());
rejectSymlinkPath(target);
try (OutputStream out = Files.newOutputStream(
target,
StandardOpenOption.CREATE,
StandardOpenOption.TRUNCATE_EXISTING,
StandardOpenOption.WRITE,
LinkOption.NOFOLLOW_LINKS)) {
data.transferTo(out);
}
return target.toUri().toString();
}
@Override
public String append(String key, byte[] chunk) throws IOException {
Path target = resolve(key);
synchronized (appendLocks.computeIfAbsent(target, ignored -> new Object())) {
createDirectoriesInsideRoot(target.getParent());
rejectSymlinkPath(target);
Files.write(
target,
chunk == null ? new byte[0] : chunk,
StandardOpenOption.CREATE,
StandardOpenOption.APPEND,
StandardOpenOption.WRITE,
LinkOption.NOFOLLOW_LINKS);
}
return target.toUri().toString();
}
@Override
public InputStream get(String key) throws IOException {
Path target = resolveExisting(key);
return Files.newInputStream(target, StandardOpenOption.READ, LinkOption.NOFOLLOW_LINKS);
}
@Override
public boolean exists(String key) {
try {
return Files.exists(resolveExisting(key), LinkOption.NOFOLLOW_LINKS);
} catch (IOException | IllegalArgumentException e) {
return false;
}
}
@Override
public long size(String key) throws IOException {
return Files.size(resolveExisting(key));
}
private Path resolve(String key) {
if (key == null || key.isBlank()) {
throw new IllegalArgumentException("archive key must not be blank");
}
Path target = root.resolve(key).normalize();
if (!target.startsWith(root)) {
throw new IllegalArgumentException("archive key escapes root: " + key);
}
return target;
}
private Path resolveExisting(String key) throws IOException {
Path target = resolve(key);
rejectSymlinkPath(target);
return target;
}
private void createDirectoriesInsideRoot(Path directory) throws IOException {
rejectEscapedPath(directory);
if (Files.notExists(root, LinkOption.NOFOLLOW_LINKS)) {
Files.createDirectories(root);
}
Path current = root;
Path relative = root.relativize(directory);
for (Path segment : relative) {
current = current.resolve(segment);
if (Files.isSymbolicLink(current)) {
throw new IOException("archive path traverses symlink: " + current);
}
if (Files.notExists(current, LinkOption.NOFOLLOW_LINKS)) {
Files.createDirectory(current);
} else if (!Files.isDirectory(current, LinkOption.NOFOLLOW_LINKS)) {
throw new IOException("archive path segment is not a directory: " + current);
}
}
}
private void rejectSymlinkPath(Path target) throws IOException {
rejectEscapedPath(target);
Path current = root;
Path relative = root.relativize(target);
for (Path segment : relative) {
current = current.resolve(segment);
if (Files.isSymbolicLink(current)) {
throw new IOException("archive path traverses symlink: " + current);
}
if (Files.notExists(current, LinkOption.NOFOLLOW_LINKS)) {
return;
}
}
}
private void rejectEscapedPath(Path path) {
if (!path.normalize().startsWith(root)) {
throw new IllegalArgumentException("archive path escapes root: " + path);
}
}
private static Path rootPath(String value) {
if (value == null || value.isBlank()) {
return Path.of(System.getProperty("java.io.tmpdir"), "lingniu-archive");
}
if (value.startsWith("file://")) {
return Path.of(URI.create(value));
}
return Path.of(value);
}
}

View File

@@ -0,0 +1,44 @@
package com.lingniu.ingest.sink.archive;
import com.lingniu.ingest.api.event.RawArchiveKeys;
import com.lingniu.ingest.api.event.VehicleEvent;
import com.lingniu.ingest.api.sink.EventSink;
import java.io.ByteArrayInputStream;
import java.util.concurrent.CompletableFuture;
public final class RawArchiveEventSink implements EventSink {
private final ArchiveStore store;
public RawArchiveEventSink(ArchiveStore store) {
this.store = store;
}
@Override
public String name() {
return "raw-archive";
}
@Override
public CompletableFuture<Void> publish(VehicleEvent event) {
if (!(event instanceof VehicleEvent.RawArchive raw)) {
return CompletableFuture.completedFuture(null);
}
try {
byte[] bytes = raw.rawBytes();
if (bytes == null) {
throw new IllegalArgumentException("raw archive bytes must not be null");
}
store.put(RawArchiveKeys.key(raw), new ByteArrayInputStream(bytes), bytes.length);
return CompletableFuture.completedFuture(null);
} catch (Exception e) {
return CompletableFuture.failedFuture(e);
}
}
@Override
public boolean accepts(VehicleEvent event) {
return event instanceof VehicleEvent.RawArchive;
}
}

View File

@@ -0,0 +1,35 @@
package com.lingniu.ingest.sink.archive.config;
import com.lingniu.ingest.sink.archive.ArchiveStore;
import com.lingniu.ingest.sink.archive.LocalArchiveStore;
import com.lingniu.ingest.sink.archive.RawArchiveEventSink;
import org.springframework.boot.autoconfigure.AutoConfiguration;
import org.springframework.boot.autoconfigure.condition.ConditionalOnMissingBean;
import org.springframework.boot.autoconfigure.condition.ConditionalOnBean;
import org.springframework.boot.autoconfigure.condition.ConditionalOnProperty;
import org.springframework.boot.context.properties.EnableConfigurationProperties;
import org.springframework.context.annotation.Bean;
@AutoConfiguration
@EnableConfigurationProperties(SinkArchiveProperties.class)
@ConditionalOnProperty(
prefix = "lingniu.ingest.sink.archive",
name = "enabled",
havingValue = "true",
matchIfMissing = true)
public class SinkArchiveAutoConfiguration {
@Bean
@ConditionalOnMissingBean
@ConditionalOnProperty(prefix = "lingniu.ingest.sink.archive", name = "type", havingValue = "local", matchIfMissing = true)
public ArchiveStore archiveStore(SinkArchiveProperties properties) {
return new LocalArchiveStore(properties.getPath());
}
@Bean
@ConditionalOnMissingBean
@ConditionalOnBean(ArchiveStore.class)
public RawArchiveEventSink rawArchiveEventSink(ArchiveStore store) {
return new RawArchiveEventSink(store);
}
}

View File

@@ -0,0 +1,35 @@
package com.lingniu.ingest.sink.archive.config;
import org.springframework.boot.context.properties.ConfigurationProperties;
@ConfigurationProperties(prefix = "lingniu.ingest.sink.archive")
public class SinkArchiveProperties {
private boolean enabled = true;
private String type = "local";
private String path = System.getProperty("java.io.tmpdir") + "/lingniu-archive";
public boolean isEnabled() {
return enabled;
}
public void setEnabled(boolean enabled) {
this.enabled = enabled;
}
public String getType() {
return type;
}
public void setType(String type) {
this.type = type;
}
public String getPath() {
return path;
}
public void setPath(String path) {
this.path = path;
}
}

View File

@@ -0,0 +1,74 @@
package com.lingniu.ingest.sink.archive;
import org.junit.jupiter.api.Test;
import org.junit.jupiter.api.io.TempDir;
import java.io.ByteArrayInputStream;
import java.nio.charset.StandardCharsets;
import java.nio.file.Files;
import java.nio.file.Path;
import java.util.ArrayList;
import java.util.List;
import java.util.concurrent.CompletableFuture;
import java.util.concurrent.Executors;
import static org.assertj.core.api.Assertions.assertThat;
import static org.assertj.core.api.Assertions.assertThatThrownBy;
class LocalArchiveStoreTest {
@TempDir
Path tempDir;
@Test
void rejectsSymlinkTraversalOutsideRootForWriteReadAndSize() throws Exception {
Path root = tempDir.resolve("archive");
Path outside = tempDir.resolve("outside");
Files.createDirectories(root);
Files.createDirectories(outside);
Files.writeString(outside.resolve("existing.bin"), "secret", StandardCharsets.UTF_8);
Files.createSymbolicLink(root.resolve("link"), outside);
LocalArchiveStore store = new LocalArchiveStore(root);
assertThatThrownBy(() -> store.put(
"link/new.bin",
new ByteArrayInputStream("payload".getBytes(StandardCharsets.UTF_8)),
7))
.isInstanceOfAny(IllegalArgumentException.class, java.io.IOException.class);
assertThat(outside.resolve("new.bin")).doesNotExist();
assertThatThrownBy(() -> store.get("link/existing.bin"))
.isInstanceOfAny(IllegalArgumentException.class, java.io.IOException.class);
assertThatThrownBy(() -> store.size("link/existing.bin"))
.isInstanceOfAny(IllegalArgumentException.class, java.io.IOException.class);
}
@Test
void appendsConcurrentChunksWithoutLosingData() throws Exception {
LocalArchiveStore store = new LocalArchiveStore(tempDir.resolve("archive"));
int chunks = 96;
try (var executor = Executors.newFixedThreadPool(12)) {
List<CompletableFuture<Void>> futures = new ArrayList<>();
for (int i = 0; i < chunks; i++) {
String chunk = "%03d\n".formatted(i);
futures.add(CompletableFuture.runAsync(() -> {
try {
store.append("same/key.bin", chunk.getBytes(StandardCharsets.UTF_8));
} catch (Exception e) {
throw new RuntimeException(e);
}
}, executor));
}
CompletableFuture.allOf(futures.toArray(CompletableFuture[]::new)).join();
}
String content = Files.readString(tempDir.resolve("archive/same/key.bin"), StandardCharsets.UTF_8);
assertThat(content).hasSize(chunks * 4);
assertThat(content.lines()).containsExactlyInAnyOrderElementsOf(
java.util.stream.IntStream.range(0, chunks)
.mapToObj("%03d"::formatted)
.toList());
}
}

View File

@@ -0,0 +1,63 @@
package com.lingniu.ingest.sink.archive;
import com.lingniu.ingest.api.ProtocolId;
import com.lingniu.ingest.api.event.VehicleEvent;
import org.junit.jupiter.api.Test;
import java.io.InputStream;
import java.time.Instant;
import java.util.Map;
import java.util.concurrent.CompletionException;
import static org.assertj.core.api.Assertions.assertThatThrownBy;
class RawArchiveEventSinkTest {
@Test
void failsFastWhenRawBytesAreNull() {
RawArchiveEventSink sink = new RawArchiveEventSink(new CapturingArchiveStore());
VehicleEvent.RawArchive raw = new VehicleEvent.RawArchive(
"event-1",
"VIN123",
ProtocolId.GB32960,
Instant.parse("2026-06-23T00:00:00Z"),
Instant.parse("2026-06-23T00:00:01Z"),
"trace-1",
Map.of(),
2,
1,
null);
assertThatThrownBy(() -> sink.publish(raw).join())
.isInstanceOf(CompletionException.class)
.hasCauseInstanceOf(IllegalArgumentException.class);
}
private static final class CapturingArchiveStore implements ArchiveStore {
@Override
public String put(String key, InputStream data, long length) {
return "archive://" + key;
}
@Override
public String append(String key, byte[] chunk) {
return "archive://" + key;
}
@Override
public InputStream get(String key) {
throw new UnsupportedOperationException();
}
@Override
public boolean exists(String key) {
return false;
}
@Override
public long size(String key) {
return 0;
}
}
}

View File

@@ -0,0 +1,40 @@
package com.lingniu.ingest.sink.archive.config;
import com.lingniu.ingest.sink.archive.ArchiveStore;
import com.lingniu.ingest.sink.archive.RawArchiveEventSink;
import org.junit.jupiter.api.Test;
import org.springframework.context.annotation.AnnotationConfigApplicationContext;
import org.springframework.core.env.MapPropertySource;
import static org.assertj.core.api.Assertions.assertThat;
class SinkArchiveAutoConfigurationTest {
@Test
void localArchiveTypeCreatesStoreAndSink() {
try (AnnotationConfigApplicationContext context = new AnnotationConfigApplicationContext()) {
context.getEnvironment().getPropertySources().addFirst(new MapPropertySource(
"test",
java.util.Map.of("lingniu.ingest.sink.archive.path", "target/test-archive")));
context.register(SinkArchiveAutoConfiguration.class);
context.refresh();
assertThat(context.getBeansOfType(ArchiveStore.class)).hasSize(1);
assertThat(context.getBeansOfType(RawArchiveEventSink.class)).hasSize(1);
}
}
@Test
void nonLocalArchiveTypeDoesNotCreateSinkWithoutStore() {
try (AnnotationConfigApplicationContext context = new AnnotationConfigApplicationContext()) {
context.getEnvironment().getPropertySources().addFirst(new MapPropertySource(
"test",
java.util.Map.of("lingniu.ingest.sink.archive.type", "oss")));
context.register(SinkArchiveAutoConfiguration.class);
context.refresh();
assertThat(context.getBeansOfType(ArchiveStore.class)).isEmpty();
assertThat(context.getBeansOfType(RawArchiveEventSink.class)).isEmpty();
}
}
}

View File

@@ -71,8 +71,7 @@ public final class EnvelopeMapper {
int size = ra.rawBytes() == null ? 0 : ra.rawBytes().length; int size = ra.rawBytes() == null ? 0 : ra.rawBytes().length;
String key = rawArchiveKey(ra); String key = rawArchiveKey(ra);
String uri = rawArchiveUri(ra, key); String uri = rawArchiveUri(ra, key);
// RAW envelope 只携带引用信息,不把完整原始字节塞进 Kafka避免 topic 膨胀。 // RAW envelope 只携带 URI/size 引用信息,不把完整原始字节塞进 Kafka避免 topic 膨胀。
// 注意 KafkaEventSink 当前仍过滤 RawArchive这段用于未来放开 raw 引用转发或消费者测试。
b.putMetadata(RawArchiveKeys.META_KEY, key); b.putMetadata(RawArchiveKeys.META_KEY, key);
b.putMetadata(RawArchiveKeys.META_URI, uri); b.putMetadata(RawArchiveKeys.META_URI, uri);
b.putMetadata(RawArchiveKeys.META_EVENT_ID, ra.eventId()); b.putMetadata(RawArchiveKeys.META_EVENT_ID, ra.eventId());

View File

@@ -43,13 +43,11 @@ public final class KafkaEventSink implements EventSink, AutoCloseable {
} }
/** /**
* 拒绝 {@link VehicleEvent.RawArchive} —— 当前生产 Kafka sink 只投递实时、会话、告警等轻量事件。 * Split-service mode uses Kafka for both normalized events and raw archive records.
* {@link EnvelopeMapper} 和 {@link TopicRouter} 已具备 RAW envelope/topic 的映射能力,但这里仍
* 显式过滤;如需把 32960 raw 引用转发到 {@code vehicle.raw.archive},应先移除此过滤并补充测试。
*/ */
@Override @Override
public boolean accepts(VehicleEvent event) { public boolean accepts(VehicleEvent event) {
return !(event instanceof VehicleEvent.RawArchive); return true;
} }
@Override @Override

View File

@@ -57,16 +57,16 @@ public class SinkMqProperties {
public static class Topics { public static class Topics {
/** 实时遥测事件 topicGB32960 RAW-only 架构下可逐步弱化该 topic。 */ /** 实时遥测事件 topicGB32960 RAW-only 架构下可逐步弱化该 topic。 */
private String realtime = "vehicle.realtime"; private String realtime = "vehicle.event.gb32960.v1";
/** 位置事件 topic通常可由 realtime/RAW 派生。 */ /** 位置事件 topic通常可由 realtime/RAW 派生。 */
private String location = "vehicle.location"; private String location = "vehicle.event.gb32960.v1";
private String alarm = "vehicle.alarm"; private String alarm = "vehicle.event.gb32960.v1";
private String session = "vehicle.session"; private String session = "vehicle.event.gb32960.v1";
private String mediaMeta = "vehicle.media.meta"; private String mediaMeta = "vehicle.media.meta.v1";
/** RAW 归档引用 topicpayload 应携带 archive URI/size不建议携带完整 raw bytes。 */ /** RAW 归档引用 topicpayload 应携带 archive URI/size不建议携带完整 raw bytes。 */
private String rawArchive = "vehicle.raw.archive"; private String rawArchive = "vehicle.raw.gb32960.v1";
/** producer 熔断或 consumer 处理失败时的死信 topic。 */ /** producer 熔断或 consumer 处理失败时的死信 topic。 */
private String dlq = "vehicle.dlq"; private String dlq = "vehicle.dlq.gb32960.v1";
public String getRealtime() { return realtime; } public String getRealtime() { return realtime; }
public void setRealtime(String realtime) { this.realtime = realtime; } public void setRealtime(String realtime) { this.realtime = realtime; }

View File

@@ -0,0 +1,40 @@
package com.lingniu.ingest.sink.mq;
import com.lingniu.ingest.api.ProtocolId;
import com.lingniu.ingest.api.event.VehicleEvent;
import io.github.resilience4j.circuitbreaker.CircuitBreaker;
import org.junit.jupiter.api.Test;
import java.time.Instant;
import java.util.Map;
import static org.assertj.core.api.Assertions.assertThat;
class KafkaEventSinkTest {
@Test
void acceptsRawArchiveRecordsForSplitServiceKafkaBoundary() {
KafkaEventSink sink = new KafkaEventSink(
null,
new EnvelopeMapper("node-1"),
new TopicRouter(new SinkMqProperties.Topics()),
"vehicle.dlq.gb32960.v1",
CircuitBreaker.ofDefaults("kafka-test"));
assertThat(sink.accepts(rawArchive())).isTrue();
}
private static VehicleEvent.RawArchive rawArchive() {
return new VehicleEvent.RawArchive(
"raw-1",
"VIN001",
ProtocolId.GB32960,
Instant.parse("2026-06-23T07:00:00Z"),
Instant.parse("2026-06-23T07:00:01Z"),
"trace-raw-1",
Map.of("platformAccount", "Hyundai"),
0x02,
0,
new byte[] {0x23, 0x23});
}
}

View File

@@ -0,0 +1,147 @@
package com.lingniu.ingest.sink.mq;
import com.lingniu.ingest.api.ProtocolId;
import com.lingniu.ingest.api.event.AlarmPayload;
import com.lingniu.ingest.api.event.LocationPayload;
import com.lingniu.ingest.api.event.RealtimePayload;
import com.lingniu.ingest.api.event.VehicleEvent;
import org.junit.jupiter.api.Test;
import java.time.Instant;
import java.util.List;
import java.util.Map;
import java.util.Set;
import static org.assertj.core.api.Assertions.assertThat;
class TopicRouterTest {
@Test
void rawArchiveRoutesToVersionedGb32960RawTopicByDefault() {
TopicRouter router = new TopicRouter(new SinkMqProperties.Topics());
assertThat(router.route(rawArchive())).isEqualTo("vehicle.raw.gb32960.v1");
}
@Test
void normalizedGb32960EventsRouteToVersionedEventTopicByDefault() {
TopicRouter router = new TopicRouter(new SinkMqProperties.Topics());
assertThat(router.route(realtime())).isEqualTo("vehicle.event.gb32960.v1");
assertThat(router.route(location())).isEqualTo("vehicle.event.gb32960.v1");
assertThat(router.route(alarm())).isEqualTo("vehicle.event.gb32960.v1");
assertThat(router.route(login())).isEqualTo("vehicle.event.gb32960.v1");
assertThat(router.route(logout())).isEqualTo("vehicle.event.gb32960.v1");
assertThat(router.route(heartbeat())).isEqualTo("vehicle.event.gb32960.v1");
}
@Test
void dlqDefaultsToVersionedGb32960DlqTopic() {
SinkMqProperties.Topics topics = new SinkMqProperties.Topics();
assertThat(topics.getDlq()).isEqualTo("vehicle.dlq.gb32960.v1");
}
private static VehicleEvent.RawArchive rawArchive() {
return new VehicleEvent.RawArchive(
"raw-1",
"VIN001",
ProtocolId.GB32960,
Instant.parse("2026-06-23T07:00:00Z"),
Instant.parse("2026-06-23T07:00:01Z"),
"trace-raw-1",
Map.of("platformAccount", "Hyundai"),
0x02,
0,
new byte[] {0x23, 0x23});
}
private static VehicleEvent.Realtime realtime() {
return new VehicleEvent.Realtime(
"event-1",
"VIN001",
ProtocolId.GB32960,
Instant.parse("2026-06-23T07:00:00Z"),
Instant.parse("2026-06-23T07:00:01Z"),
"trace-1",
Map.of("platformAccount", "Hyundai"),
new RealtimePayload(
42.1, 1234.5, 80.0, null, null,
null, null, null, null, null, null,
RealtimePayload.VehicleState.STARTED,
RealtimePayload.ChargingState.UNCHARGED,
RealtimePayload.RunningMode.ELECTRIC,
null, null, null,
113.12, 23.45, null, null, null, null));
}
private static VehicleEvent.Location location() {
return new VehicleEvent.Location(
"event-2",
"VIN001",
ProtocolId.GB32960,
Instant.parse("2026-06-23T07:00:00Z"),
Instant.parse("2026-06-23T07:00:01Z"),
"trace-2",
Map.of("platformAccount", "Hyundai"),
new LocationPayload(113.12, 23.45, 0, 42.1, 90, 0, 0));
}
private static VehicleEvent.Alarm alarm() {
return new VehicleEvent.Alarm(
"event-3",
"VIN001",
ProtocolId.GB32960,
Instant.parse("2026-06-23T07:00:00Z"),
Instant.parse("2026-06-23T07:00:01Z"),
"trace-3",
Map.of("platformAccount", "Hyundai"),
new AlarmPayload(
AlarmPayload.AlarmLevel.MINOR,
1,
"GB32960_ALARM",
List.of("1"),
Set.of("bit0"),
113.12,
23.45,
AlarmPayload.SafetyCategory.GENERAL,
false,
AlarmPayload.HydrogenLeakLevel.NONE,
false));
}
private static VehicleEvent.Login login() {
return new VehicleEvent.Login(
"event-4",
"VIN001",
ProtocolId.GB32960,
Instant.parse("2026-06-23T07:00:00Z"),
Instant.parse("2026-06-23T07:00:01Z"),
"trace-4",
Map.of("platformAccount", "Hyundai"),
"iccid-1",
"V2016");
}
private static VehicleEvent.Logout logout() {
return new VehicleEvent.Logout(
"event-5",
"VIN001",
ProtocolId.GB32960,
Instant.parse("2026-06-23T07:00:00Z"),
Instant.parse("2026-06-23T07:00:01Z"),
"trace-5",
Map.of("platformAccount", "Hyundai"));
}
private static VehicleEvent.Heartbeat heartbeat() {
return new VehicleEvent.Heartbeat(
"event-6",
"VIN001",
ProtocolId.GB32960,
Instant.parse("2026-06-23T07:00:00Z"),
Instant.parse("2026-06-23T07:00:01Z"),
"trace-6",
Map.of("platformAccount", "Hyundai"));
}
}

18
pom.xml
View File

@@ -36,6 +36,9 @@
<module>modules/inbound/inbound-mqtt</module> <module>modules/inbound/inbound-mqtt</module>
<module>modules/inbound/inbound-xinda-push</module> <module>modules/inbound/inbound-xinda-push</module>
<module>modules/apps/command-gateway</module> <module>modules/apps/command-gateway</module>
<module>modules/apps/gb32960-ingest-app</module>
<module>modules/apps/vehicle-history-app</module>
<module>modules/apps/vehicle-analytics-app</module>
<module>modules/apps/bootstrap-all</module> <module>modules/apps/bootstrap-all</module>
</modules> </modules>
@@ -222,6 +225,21 @@
<artifactId>command-gateway</artifactId> <artifactId>command-gateway</artifactId>
<version>${project.version}</version> <version>${project.version}</version>
</dependency> </dependency>
<dependency>
<groupId>com.lingniu.ingest</groupId>
<artifactId>gb32960-ingest-app</artifactId>
<version>${project.version}</version>
</dependency>
<dependency>
<groupId>com.lingniu.ingest</groupId>
<artifactId>vehicle-history-app</artifactId>
<version>${project.version}</version>
</dependency>
<dependency>
<groupId>com.lingniu.ingest</groupId>
<artifactId>vehicle-analytics-app</artifactId>
<version>${project.version}</version>
</dependency>
<!-- 第三方(非 BOM 管理) --> <!-- 第三方(非 BOM 管理) -->
<dependency> <dependency>