docs: add vehicle ingest redesign spec
This commit is contained in:
lingniu
613
docs/superpowers/specs/2026-06-29-vehicle-ingest-redesign.md
Normal file
613
docs/superpowers/specs/2026-06-29-vehicle-ingest-redesign.md
Normal file
@@ -0,0 +1,613 @@
|
||||
# Vehicle Ingest Redesign Design
|
||||
|
||||
## Background
|
||||
|
||||
Current GB32960 and JT808 ingestion can receive, decode, publish, and query data, but the responsibilities are split across several partially overlapping paths:
|
||||
|
||||
- Netty handlers produce `RawFrame` and the dispatcher emits both raw archive events and normalized vehicle events.
|
||||
- Kafka envelopes carry raw archive references, while raw bytes are stored separately by the archive sink.
|
||||
- GB32960 historical queries use raw archive records plus file replay decoding.
|
||||
- JT808 location history is materialized separately into TDengine.
|
||||
- DuckDB/Parquet, local raw archive, Kafka, and TDengine each own part of the story, so query and durability semantics are hard to reason about.
|
||||
|
||||
The new design intentionally keeps useful protocol parsers and app deployment knowledge, but replaces the ingestion, durability, storage, and query boundaries with a first-principles model.
|
||||
|
||||
## Goals
|
||||
|
||||
1. Store every original inbound protocol frame, including malformed frames when bytes are available.
|
||||
2. Query historical uploaded data for GB32960 and JT808 by vehicle, protocol, message type, and time range.
|
||||
3. Extend to new fields, protocol messages, and vendor profiles without rewriting the core pipeline.
|
||||
4. Sustain production load for 10,000 vehicles and 1,000 concurrent historical query requests per second.
|
||||
5. Make ACK and durability rules explicit enough that an acknowledged frame is not silently lost.
|
||||
|
||||
## Non-Goals
|
||||
|
||||
- Do not make Kafka the long-term historical query database.
|
||||
- Do not store large raw byte arrays directly in Kafka topics.
|
||||
- Do not keep DuckDB/Parquet as the production hot query store for GB32960 and JT808.
|
||||
- Do not build one generic API that hides protocol-specific semantics. Shared internals are preferred; external APIs may remain protocol-aware.
|
||||
|
||||
## Recommended Architecture
|
||||
|
||||
Use `Raw Archive + Kafka + TDengine` as the core architecture.
|
||||
|
||||
```mermaid
|
||||
flowchart LR
|
||||
GB["GB32960 TCP"] --> Edge["Protocol Edge"]
|
||||
JT["JT808 TCP"] --> Edge
|
||||
Edge --> RawStore["Raw Archive"]
|
||||
RawStore --> RawTopic["Kafka vehicle.raw-frame.v1"]
|
||||
Edge --> Decode["Protocol Decoder"]
|
||||
Decode --> FactTopic["Kafka vehicle.decoded-fact.v1"]
|
||||
RawTopic --> Writer["History Writer"]
|
||||
FactTopic --> Writer
|
||||
Writer --> TD["TDengine"]
|
||||
Writer --> Redis["Realtime State"]
|
||||
API["History API"] --> TD
|
||||
API --> RawStore
|
||||
RealtimeAPI["Realtime API"] --> Redis
|
||||
```
|
||||
|
||||
The architecture has three facts:
|
||||
|
||||
- `RawFrameFact`: one row per inbound frame, whether parsing succeeds or fails.
|
||||
- `DecodedFact`: zero or more normalized facts derived from a raw frame.
|
||||
- `RawArchiveObject`: immutable raw bytes addressed by `raw_uri`.
|
||||
|
||||
The raw archive is the forensic source of truth. TDengine is the query source of truth. Kafka is the decoupling and replay pipe.
|
||||
|
||||
## Module Boundaries
|
||||
|
||||
### Protocol Edge Apps
|
||||
|
||||
Apps:
|
||||
|
||||
- `gb32960-ingest-app`
|
||||
- `jt808-ingest-app`
|
||||
|
||||
Responsibilities:
|
||||
|
||||
- Accept TCP connections.
|
||||
- Split frames.
|
||||
- Perform protocol-level validation, authentication, session binding, and ACK.
|
||||
- Create `RawFrameFact`.
|
||||
- Persist raw bytes through `RawArchiveWriter`.
|
||||
- Publish raw frame fact to Kafka.
|
||||
- Decode frames and publish decoded facts.
|
||||
|
||||
They do not query history, export data, or write business tables directly.
|
||||
|
||||
### Ingest Core
|
||||
|
||||
New module:
|
||||
|
||||
- `modules/core/ingest-facts`
|
||||
|
||||
Core types:
|
||||
|
||||
```java
|
||||
public record RawFrameFact(
|
||||
String frameId,
|
||||
ProtocolId protocol,
|
||||
String vehicleKey,
|
||||
String vin,
|
||||
String phone,
|
||||
int messageId,
|
||||
int subType,
|
||||
Instant eventTime,
|
||||
Instant receivedAt,
|
||||
String peer,
|
||||
String rawUri,
|
||||
String checksum,
|
||||
long rawSizeBytes,
|
||||
ParseStatus parseStatus,
|
||||
String parseError,
|
||||
Map<String, String> metadata
|
||||
) {}
|
||||
```
|
||||
|
||||
```java
|
||||
public sealed interface DecodedFact permits
|
||||
DecodedFact.Location,
|
||||
DecodedFact.Realtime,
|
||||
DecodedFact.Alarm,
|
||||
DecodedFact.Session,
|
||||
DecodedFact.Register,
|
||||
DecodedFact.Extension {
|
||||
|
||||
String factId();
|
||||
String frameId();
|
||||
ProtocolId protocol();
|
||||
String vehicleKey();
|
||||
String vin();
|
||||
Instant eventTime();
|
||||
Instant receivedAt();
|
||||
String rawUri();
|
||||
Map<String, String> metadata();
|
||||
}
|
||||
```
|
||||
|
||||
`vehicleKey` is the stable partition key:
|
||||
|
||||
- GB32960: VIN.
|
||||
- JT808: resolved VIN when known, otherwise `jt808:<phone>`.
|
||||
- Unknown/malformed: `unknown:<protocol>:<hash>`.
|
||||
|
||||
### Raw Archive
|
||||
|
||||
New or refactored module:
|
||||
|
||||
- `modules/sinks/raw-archive-store`
|
||||
|
||||
Interface:
|
||||
|
||||
```java
|
||||
public interface RawArchiveWriter {
|
||||
RawArchiveReceipt write(RawArchiveWriteRequest request) throws IOException;
|
||||
}
|
||||
```
|
||||
|
||||
Receipt:
|
||||
|
||||
```java
|
||||
public record RawArchiveReceipt(
|
||||
String rawUri,
|
||||
String checksum,
|
||||
long sizeBytes
|
||||
) {}
|
||||
```
|
||||
|
||||
Rules:
|
||||
|
||||
- Raw bytes are immutable once written.
|
||||
- Write path is content-addressed or deterministic by `protocol/date/vehicleKey/frameId.bin`.
|
||||
- Write must be fsync-capable for local storage and pluggable for object storage.
|
||||
- `rawUri` must be resolvable by history API for single-frame replay.
|
||||
- Malformed frames are written with `parseStatus=FAILED` when bytes are available.
|
||||
|
||||
### Kafka Topics
|
||||
|
||||
Use stable versioned topics:
|
||||
|
||||
- `vehicle.raw-frame.v1`
|
||||
- `vehicle.decoded-fact.v1`
|
||||
- `vehicle.dlq.v1`
|
||||
|
||||
Kafka key:
|
||||
|
||||
```text
|
||||
<protocol>:<vehicleKey>
|
||||
```
|
||||
|
||||
`vehicle.raw-frame.v1` payload contains `RawFrameFact` without raw bytes. It carries `rawUri`, checksum, byte size, message id, parse status, and metadata.
|
||||
|
||||
`vehicle.decoded-fact.v1` payload contains one `DecodedFact`. It always references `frameId` and `rawUri`.
|
||||
|
||||
`vehicle.dlq.v1` carries failed store, publish, decode, and write attempts with enough metadata to replay from raw archive when possible.
|
||||
|
||||
### History Writer
|
||||
|
||||
New app or refactored app:
|
||||
|
||||
- `vehicle-history-app`
|
||||
|
||||
Responsibilities:
|
||||
|
||||
- Consume `vehicle.raw-frame.v1`.
|
||||
- Consume `vehicle.decoded-fact.v1`.
|
||||
- Batch write TDengine.
|
||||
- Update realtime state store for latest vehicle state.
|
||||
- Write DLQ on invalid payloads or storage failures.
|
||||
|
||||
The writer is idempotent by `(frame_id)` for raw facts and `(fact_id)` for decoded facts.
|
||||
|
||||
## TDengine Model
|
||||
|
||||
TDengine is the production historical query store.
|
||||
|
||||
### Raw Frames Super Table
|
||||
|
||||
```sql
|
||||
CREATE STABLE IF NOT EXISTS raw_frames (
|
||||
ts TIMESTAMP,
|
||||
frame_id NCHAR(64),
|
||||
received_at TIMESTAMP,
|
||||
message_id INT,
|
||||
sub_type INT,
|
||||
event_time TIMESTAMP,
|
||||
raw_uri NCHAR(512),
|
||||
checksum NCHAR(128),
|
||||
raw_size_bytes BIGINT,
|
||||
parse_status NCHAR(16),
|
||||
parse_error NCHAR(512),
|
||||
peer NCHAR(128),
|
||||
metadata_json NCHAR(4096)
|
||||
) TAGS (
|
||||
protocol NCHAR(16),
|
||||
vehicle_key NCHAR(128),
|
||||
vin NCHAR(64),
|
||||
phone NCHAR(32)
|
||||
);
|
||||
```
|
||||
|
||||
Child table:
|
||||
|
||||
```text
|
||||
raw_<protocol>_<hash(vehicleKey)>
|
||||
```
|
||||
|
||||
Primary query patterns:
|
||||
|
||||
- Raw frames by vehicle and time.
|
||||
- Raw frames by protocol, message id, parse status, and time.
|
||||
- Single raw frame by `frame_id` or `raw_uri`.
|
||||
|
||||
### Location Super Table
|
||||
|
||||
```sql
|
||||
CREATE STABLE IF NOT EXISTS vehicle_locations (
|
||||
ts TIMESTAMP,
|
||||
fact_id NCHAR(64),
|
||||
frame_id NCHAR(64),
|
||||
received_at TIMESTAMP,
|
||||
longitude DOUBLE,
|
||||
latitude DOUBLE,
|
||||
altitude_m DOUBLE,
|
||||
speed_kmh DOUBLE,
|
||||
direction_deg DOUBLE,
|
||||
alarm_flag BIGINT,
|
||||
status_flag BIGINT,
|
||||
total_mileage_km DOUBLE,
|
||||
raw_uri NCHAR(512),
|
||||
metadata_json NCHAR(4096)
|
||||
) TAGS (
|
||||
protocol NCHAR(16),
|
||||
vehicle_key NCHAR(128),
|
||||
vin NCHAR(64),
|
||||
phone NCHAR(32)
|
||||
);
|
||||
```
|
||||
|
||||
### Realtime Super Table
|
||||
|
||||
```sql
|
||||
CREATE STABLE IF NOT EXISTS vehicle_realtime (
|
||||
ts TIMESTAMP,
|
||||
fact_id NCHAR(64),
|
||||
frame_id NCHAR(64),
|
||||
received_at TIMESTAMP,
|
||||
speed_kmh DOUBLE,
|
||||
total_mileage_km DOUBLE,
|
||||
battery_soc DOUBLE,
|
||||
total_voltage_v DOUBLE,
|
||||
total_current_a DOUBLE,
|
||||
running_mode NCHAR(64),
|
||||
vehicle_state NCHAR(64),
|
||||
charging_state NCHAR(64),
|
||||
raw_uri NCHAR(512),
|
||||
fields_json NCHAR(8192),
|
||||
metadata_json NCHAR(4096)
|
||||
) TAGS (
|
||||
protocol NCHAR(16),
|
||||
vehicle_key NCHAR(128),
|
||||
vin NCHAR(64),
|
||||
phone NCHAR(32)
|
||||
);
|
||||
```
|
||||
|
||||
`fields_json` stores sparse full-field values for low-frequency inspection. High-frequency queried fields should be promoted to explicit columns or a dedicated extension table.
|
||||
|
||||
### Alarm Super Table
|
||||
|
||||
```sql
|
||||
CREATE STABLE IF NOT EXISTS vehicle_alarms (
|
||||
ts TIMESTAMP,
|
||||
fact_id NCHAR(64),
|
||||
frame_id NCHAR(64),
|
||||
received_at TIMESTAMP,
|
||||
alarm_level NCHAR(32),
|
||||
alarm_code BIGINT,
|
||||
alarm_name NCHAR(128),
|
||||
longitude DOUBLE,
|
||||
latitude DOUBLE,
|
||||
raw_uri NCHAR(512),
|
||||
fields_json NCHAR(8192),
|
||||
metadata_json NCHAR(4096)
|
||||
) TAGS (
|
||||
protocol NCHAR(16),
|
||||
vehicle_key NCHAR(128),
|
||||
vin NCHAR(64),
|
||||
phone NCHAR(32)
|
||||
);
|
||||
```
|
||||
|
||||
### Session and Registration Tables
|
||||
|
||||
```sql
|
||||
CREATE STABLE IF NOT EXISTS vehicle_sessions (
|
||||
ts TIMESTAMP,
|
||||
fact_id NCHAR(64),
|
||||
frame_id NCHAR(64),
|
||||
received_at TIMESTAMP,
|
||||
session_type NCHAR(32),
|
||||
result_code INT,
|
||||
raw_uri NCHAR(512),
|
||||
fields_json NCHAR(8192),
|
||||
metadata_json NCHAR(4096)
|
||||
) TAGS (
|
||||
protocol NCHAR(16),
|
||||
vehicle_key NCHAR(128),
|
||||
vin NCHAR(64),
|
||||
phone NCHAR(32)
|
||||
);
|
||||
```
|
||||
|
||||
JT808 registration details go into `vehicle_sessions` first and may later be projected into MySQL identity bindings for durable identity lookup.
|
||||
|
||||
## ACK and Durability Rules
|
||||
|
||||
For frames that require a protocol ACK:
|
||||
|
||||
1. Frame bytes are accepted from Netty.
|
||||
2. Raw bytes are written to raw archive.
|
||||
3. `RawFrameFact` is published to Kafka.
|
||||
4. Protocol-specific ACK is sent.
|
||||
5. Decoding and `DecodedFact` publication may run before or after ACK, but failures must emit a DLQ event and update raw frame parse status.
|
||||
|
||||
For GB32960 realtime/report frames, this prevents acknowledging a frame whose original bytes cannot be recovered.
|
||||
|
||||
For JT808 register/auth/heartbeat, ACK follows the same raw archive and raw fact boundary. Register ACK can still include token generation before raw fact publish, but must not be flushed to the channel until the durability boundary succeeds.
|
||||
|
||||
If raw archive write fails, close or keep the channel according to protocol tolerance, but do not ACK success.
|
||||
|
||||
If Kafka raw fact publish fails, do not ACK success for durable-ACK frames. Non-durable frames go to local DLQ when configured.
|
||||
|
||||
## Protocol Decoding and Extension
|
||||
|
||||
Use a plugin contract:
|
||||
|
||||
```java
|
||||
public interface ProtocolFrameDecoder {
|
||||
ProtocolId protocol();
|
||||
DecodeResult decode(RawFrameFact rawFact, byte[] rawBytes);
|
||||
}
|
||||
```
|
||||
|
||||
```java
|
||||
public record DecodeResult(
|
||||
ParseStatus status,
|
||||
List<DecodedFact> facts,
|
||||
String errorMessage,
|
||||
Map<String, String> metadata
|
||||
) {}
|
||||
```
|
||||
|
||||
GB32960:
|
||||
|
||||
- Frame command maps to session, realtime, location, alarm, and extension facts.
|
||||
- Vendor extensions are selected by platform account, VIN, or configured profile.
|
||||
- Full raw replay remains possible by reading `rawUri` and running the current decoder.
|
||||
|
||||
JT808:
|
||||
|
||||
- Message id maps to register, auth, heartbeat, location, batch location, media metadata, and extension facts.
|
||||
- Identity resolution updates `vehicleKey`.
|
||||
- Registration facts carry province, city, maker, device id, plate, plate color, and auth token metadata.
|
||||
|
||||
New protocol messages do not require TDengine schema changes unless they become high-frequency query dimensions. Unknown fields are stored as extension facts with `fields_json`.
|
||||
|
||||
## Query APIs
|
||||
|
||||
Keep API surfaces explicit.
|
||||
|
||||
Raw frame APIs:
|
||||
|
||||
- `GET /api/history/raw-frames`
|
||||
- Filters: `protocol`, `vin`, `phone`, `vehicleKey`, `messageId`, `parseStatus`, `dateFrom`, `dateTo`, `pageSize`, `cursor`.
|
||||
- Returns frame metadata, raw URI, checksum, parse status, and compact decoded summary when available.
|
||||
|
||||
Single raw frame API:
|
||||
|
||||
- `GET /api/history/raw-frames/{frameId}`
|
||||
- Returns raw metadata and decoded detail.
|
||||
- Supports `includeRawHex=true` with a strict size limit.
|
||||
|
||||
Location APIs:
|
||||
|
||||
- `GET /api/history/locations`
|
||||
- Filters: `protocol`, `vin`, `phone`, `vehicleKey`, time range, cursor.
|
||||
|
||||
Realtime APIs:
|
||||
|
||||
- `GET /api/history/realtime`
|
||||
- Historical time-series query.
|
||||
|
||||
Realtime state APIs:
|
||||
|
||||
- `GET /api/realtime/vehicles/{vehicleKey}`
|
||||
- Reads Redis or in-memory latest state, not TDengine.
|
||||
|
||||
Export:
|
||||
|
||||
- Export is asynchronous.
|
||||
- Request creates an export job.
|
||||
- Worker streams TDengine query results to object storage or local export files.
|
||||
- API returns job status and download URI.
|
||||
|
||||
## Pagination
|
||||
|
||||
Use cursor pagination, not deep offset, for high-QPS history queries.
|
||||
|
||||
Cursor fields:
|
||||
|
||||
```text
|
||||
ts, received_at, fact_id
|
||||
```
|
||||
|
||||
Descending query predicate:
|
||||
|
||||
```sql
|
||||
WHERE (
|
||||
ts < :cursorTs
|
||||
OR (ts = :cursorTs AND received_at < :cursorReceivedAt)
|
||||
OR (ts = :cursorTs AND received_at = :cursorReceivedAt AND fact_id < :cursorFactId)
|
||||
)
|
||||
```
|
||||
|
||||
Page size defaults to 100 and is capped at 1000.
|
||||
|
||||
## Capacity Design
|
||||
|
||||
Assumptions:
|
||||
|
||||
- 10,000 vehicles.
|
||||
- Normal report interval: 5 to 10 seconds.
|
||||
- Expected writes: 1,000 to 2,000 frames per second.
|
||||
- Peak burst factor: 3x.
|
||||
- Historical query target: 1,000 requests per second.
|
||||
|
||||
Write path:
|
||||
|
||||
- Netty worker threads stay CPU-light.
|
||||
- Raw archive writes use bounded async workers.
|
||||
- Kafka producer uses compression, batching, idempotence, and vehicle-key partitioning.
|
||||
- History writer batches TDengine inserts by super table and child table.
|
||||
- TDengine child tables are partitioned by vehicle key to make single-vehicle queries cheap.
|
||||
|
||||
Read path:
|
||||
|
||||
- Most high-QPS queries require vehicle key and time range.
|
||||
- Unbounded cross-vehicle queries are admin-only and capped.
|
||||
- Realtime latest state is served from Redis/in-memory state.
|
||||
- Raw detail replay reads one raw object and decodes on demand.
|
||||
- Common dictionary and metadata responses are cached in memory.
|
||||
|
||||
Backpressure:
|
||||
|
||||
- Raw archive worker queue has a fixed bound.
|
||||
- Kafka publish futures are tracked.
|
||||
- If durability boundary cannot complete within timeout, durable ACK fails and the channel is closed or throttled.
|
||||
- History writer lag is observable by Kafka consumer lag and TDengine batch latency.
|
||||
|
||||
## Migration Plan
|
||||
|
||||
Phase 1: Build new fact model beside existing code.
|
||||
|
||||
- Add `ingest-facts`.
|
||||
- Add raw archive receipt contract.
|
||||
- Add serializers for raw frame and decoded fact Kafka messages.
|
||||
- Add tests for stable ids, vehicle key, and raw URI generation.
|
||||
|
||||
Phase 2: Harden raw durability boundary.
|
||||
|
||||
- Refactor GB32960 and JT808 handlers so ACK waits for raw archive and raw fact publish.
|
||||
- Keep existing parsers.
|
||||
- Emit decoded facts to the new Kafka topic.
|
||||
|
||||
Phase 3: Build TDengine writer.
|
||||
|
||||
- Create TDengine schema manager.
|
||||
- Write raw frame facts.
|
||||
- Write location, realtime, alarm, session facts.
|
||||
- Add idempotent insert behavior by stable ids.
|
||||
|
||||
Phase 4: Build query APIs.
|
||||
|
||||
- Raw frame query.
|
||||
- Single frame replay.
|
||||
- Location history query.
|
||||
- Realtime history query.
|
||||
- Latest realtime state query.
|
||||
|
||||
Phase 5: Cut over and retire old hot paths.
|
||||
|
||||
- Stop using DuckDB/Parquet as production hot history for GB32960/JT808.
|
||||
- Keep raw replay compatibility while historical data migrates.
|
||||
- Remove protocol-specific ad hoc history storage once TDengine coverage is verified.
|
||||
|
||||
## Testing Strategy
|
||||
|
||||
Unit tests:
|
||||
|
||||
- `RawFrameFact` validation and vehicle key derivation.
|
||||
- Raw archive path generation and checksum.
|
||||
- GB32960 decoder facts from golden frames.
|
||||
- JT808 decoder facts from sample frames.
|
||||
- ACK boundary success and failure cases.
|
||||
- TDengine SQL generation.
|
||||
|
||||
Integration tests:
|
||||
|
||||
- Start app with embedded or mocked Kafka producer and fake raw archive.
|
||||
- Send GB32960 sample frame and verify raw fact, decoded fact, raw archive receipt.
|
||||
- Send JT808 register/auth/location frames and verify registration/session/location facts.
|
||||
- Simulate raw archive failure and verify no success ACK.
|
||||
- Simulate Kafka publish failure and verify durable ACK failure.
|
||||
|
||||
Storage tests:
|
||||
|
||||
- TDengine schema initialization.
|
||||
- Batch insert raw frames and decoded facts.
|
||||
- Query by vehicle key and time range.
|
||||
- Cursor pagination correctness.
|
||||
- Single raw frame replay by `rawUri`.
|
||||
|
||||
Load tests:
|
||||
|
||||
- 2,000 frames/s sustained write for at least 30 minutes.
|
||||
- 6,000 frames/s burst for 5 minutes.
|
||||
- 1,000 QPS location/realtime history queries with bounded p95 latency.
|
||||
- Verify no raw frame count gap between raw archive receipts, Kafka raw facts, and TDengine raw frame rows.
|
||||
|
||||
## Operational Requirements
|
||||
|
||||
Metrics:
|
||||
|
||||
- TCP connections.
|
||||
- Frame receive rate.
|
||||
- Raw archive write latency and failure count.
|
||||
- Kafka publish latency and failure count.
|
||||
- Decoder success/failure count by protocol and message id.
|
||||
- TDengine batch size, latency, and failure count.
|
||||
- Query QPS and latency by endpoint.
|
||||
- Kafka consumer lag.
|
||||
|
||||
Logs:
|
||||
|
||||
- One structured log per failed frame with `frameId`, protocol, vehicle key, message id, raw URI, and error.
|
||||
- No full raw bytes in normal logs.
|
||||
- Raw hex only in explicit debug tools with size limits.
|
||||
|
||||
Alerts:
|
||||
|
||||
- Raw archive write failure.
|
||||
- Durable ACK timeout.
|
||||
- Kafka producer failure.
|
||||
- History writer lag.
|
||||
- TDengine write failure.
|
||||
- Raw archive count and TDengine raw frame row count divergence.
|
||||
|
||||
## Acceptance Criteria
|
||||
|
||||
The redesign is complete only when all of the following are true:
|
||||
|
||||
1. Every accepted GB32960 and JT808 frame creates a raw archive object and a TDengine `raw_frames` row.
|
||||
2. Malformed frames with bytes are queryable from raw history with parse status and error reason.
|
||||
3. GB32960 and JT808 location history can be queried by vehicle and time range with cursor pagination.
|
||||
4. GB32960 realtime history can be queried by vehicle and time range.
|
||||
5. JT808 registration/session data is queryable and can update identity binding.
|
||||
6. Single-frame detail can reload raw bytes by `rawUri` and decode with the current decoder.
|
||||
7. ACK-required frames do not receive success ACK before raw archive and raw fact publication succeed.
|
||||
8. New protocol fields can be added by implementing a decoder/projector and, only when needed, a TDengine table migration.
|
||||
9. Load tests demonstrate sustained 2,000 frames/s writes and 1,000 QPS bounded historical queries.
|
||||
10. Old DuckDB/Parquet hot-history paths are no longer required for GB32960/JT808 production queries.
|
||||
|
||||
## Open Decisions
|
||||
|
||||
The following choices are intentionally fixed for the first implementation:
|
||||
|
||||
- Use local raw archive first, with the interface shaped for object storage later.
|
||||
- Use TDengine as the only production hot historical query store for GB32960/JT808.
|
||||
- Use explicit protocol-aware query APIs instead of a single generic query API.
|
||||
- Use cursor pagination for all high-QPS historical queries.
|
||||
- Keep Kafka payloads byte-light by sending raw references instead of raw bytes.
|
||||
Reference in New Issue
Block a user