refactor(go): make raw topics the realtime source
This commit is contained in:
lingniu
@@ -52,8 +52,8 @@ flowchart LR
|
||||
| MySQL | `vehicle_daily_mileage` | Queryable daily mileage | vin, date, protocol, daily mileage, source mileage | temporary vehicle keys, generic metric key/value rows, per-frame raw details |
|
||||
| MySQL | `vehicle_identity_binding` | Manual identity mapping | vin, plate, phone, device id | registration history |
|
||||
| MySQL | `jt808_registration` | JT808 registration and auth trace | phone, device id, plate, auth code, vin match state, first/latest seen | GB32960 or MQTT records |
|
||||
| Redis | `vehicle:latest:{vehicleKey}` | Latest merged realtime state | cross-protocol latest fields only | full parsed payloads and historical data |
|
||||
| Redis | `vehicle:realtime-raw:{protocol}:{vehicleKey}` | Latest full realtime protocol state | latest protocol parsed payload | historical data |
|
||||
| Redis | `vehicle:latest:{vin}` | Latest merged realtime state | cross-protocol latest fields only for VIN-bound vehicles | full parsed payloads, temporary identities, and historical data |
|
||||
| Redis | `vehicle:realtime-raw:{protocol}:{vin}` | Latest full realtime protocol state | latest protocol parsed payload for VIN-bound vehicles | historical data |
|
||||
|
||||
## Event Envelope Rules
|
||||
|
||||
@@ -91,7 +91,7 @@ Runtime metrics are exposed as Prometheus text from `/metrics`. They are operati
|
||||
Core counters:
|
||||
|
||||
- `vehicle_gateway_frames_total`: received protocol frames by protocol and parse status.
|
||||
- `vehicle_gateway_publish_total`: raw and unified publish results by protocol.
|
||||
- `vehicle_gateway_publish_total`: raw publish results by protocol. Unified publish metrics appear only when the explicit compatibility stream is enabled.
|
||||
- `vehicle_realtime_kafka_messages_total`: realtime consumer messages by topic and status.
|
||||
- `vehicle_realtime_updates_total`: Redis/MySQL realtime projector updates by topic and status.
|
||||
- `vehicle_history_writes_total`: TDengine history writes by topic and status.
|
||||
|
||||
@@ -25,9 +25,9 @@ NATS 部署在 Kafka ECS 内网 `172.17.111.56:4222`。
|
||||
| 项 | 当前值 |
|
||||
| --- | --- |
|
||||
| Stream | `VEHICLE_INGEST` |
|
||||
| Subjects | `vehicle.raw.go.gb32960.v1`、`vehicle.raw.go.jt808.v1`、`vehicle.raw.go.yutong-mqtt.v1`、`vehicle.event.go.unified.v1` |
|
||||
| Subjects | `vehicle.raw.go.gb32960.v1`、`vehicle.raw.go.jt808.v1`、`vehicle.raw.go.yutong-mqtt.v1` |
|
||||
| Durable consumer | `vehicle-kafka-bridge` |
|
||||
| 语义 | Gateway 先写 NATS；bridge 写 Kafka 成功后才 ACK NATS |
|
||||
| 语义 | Gateway 先写 NATS raw subject；bridge 写 Kafka 成功后才 ACK NATS |
|
||||
|
||||
### Kafka
|
||||
|
||||
@@ -38,7 +38,7 @@ NATS 部署在 Kafka ECS 内网 `172.17.111.56:4222`。
|
||||
| `vehicle.raw.go.gb32960.v1` | 12 | NATS Kafka bridge | history、stat、realtime |
|
||||
| `vehicle.raw.go.jt808.v1` | 12 | NATS Kafka bridge | history、stat、realtime |
|
||||
| `vehicle.raw.go.yutong-mqtt.v1` | 12 | NATS Kafka bridge | history、stat、realtime |
|
||||
| `vehicle.event.go.unified.v1` | 12 | NATS Kafka bridge | 预留给统一事件消费 |
|
||||
| `vehicle.event.go.unified.v1` | 12 | 显式打开 `PUBLISH_UNIFIED_ENABLED=true` 后才写入 | 兼容旧统一事件消费；当前最小链路不依赖 |
|
||||
|
||||
审计时 Kafka 上仍存在旧 Java/Xinda/telemetry topic，例如 `vehicle.raw.gb32960.v1`、`vehicle.raw.jt808.v1`、`vehicle.event.xinda.v1`、`vehicle.raw.telemetry-input.v1`。这些不属于当前 Go 最小链路，后续如果确认没有消费者依赖，应单独做 Kafka topic 下线清单，不在业务代码里继续引用。
|
||||
|
||||
|
||||
@@ -380,7 +380,7 @@
|
||||
</div>
|
||||
<div class="node bus">
|
||||
<strong>Subjects</strong>
|
||||
<p><code>vehicle.raw.go.gb32960.v1</code><br><code>vehicle.raw.go.jt808.v1</code><br><code>vehicle.raw.go.yutong-mqtt.v1</code><br><code>vehicle.event.go.unified.v1</code></p>
|
||||
<p><code>vehicle.raw.go.gb32960.v1</code><br><code>vehicle.raw.go.jt808.v1</code><br><code>vehicle.raw.go.yutong-mqtt.v1</code></p>
|
||||
</div>
|
||||
<div class="node warning">
|
||||
<strong>Async + Retry</strong>
|
||||
@@ -396,7 +396,7 @@
|
||||
</div>
|
||||
<div class="node bus">
|
||||
<strong>Kafka Topics</strong>
|
||||
<p>与 NATS subject 同名：raw 三个协议 + unified 一个实时主题。</p>
|
||||
<p>与 NATS subject 同名：三个协议 RAW topic。<code>vehicle.event.go.unified.v1</code> 仅作为显式兼容开关保留。</p>
|
||||
</div>
|
||||
<div class="node warning">
|
||||
<strong>失败语义</strong>
|
||||
@@ -416,7 +416,7 @@
|
||||
</div>
|
||||
<div class="node store">
|
||||
<strong>realtime-api 消费器</strong>
|
||||
<p>消费 unified topic，写 Redis 在线状态、实时合并快照、各协议 realtime-raw。</p>
|
||||
<p>消费三个 RAW topic，写 Redis 在线状态、实时合并快照、各协议 realtime-raw。</p>
|
||||
</div>
|
||||
<div class="node api">
|
||||
<strong>HTTP API</strong>
|
||||
@@ -433,7 +433,7 @@
|
||||
<div class="mini-step"><b>2. 拆包</b><span>TCP 按协议提取完整帧；MQTT 直接按消息处理。</span></div>
|
||||
<div class="mini-step"><b>3. 解析</b><span>生成 parsed 全量结构化数据和 fields 核心字段。</span></div>
|
||||
<div class="mini-step"><b>4. 绑定 VIN</b><span>根据 VIN/phone/device_id/plate 生成 vehicle_key。</span></div>
|
||||
<div class="mini-step"><b>5. 发 NATS</b><span>写 RAW subject；同时写 unified subject。</span></div>
|
||||
<div class="mini-step"><b>5. 发 NATS</b><span>写协议 RAW subject；unified 只在兼容开关打开时写。</span></div>
|
||||
<div class="mini-step"><b>6. 桥接 Kafka</b><span>bridge 写 Kafka 成功后 ACK NATS。</span></div>
|
||||
<div class="mini-step"><b>7. 多路消费</b><span>历史、实时、统计各自消费 Kafka，互不阻塞。</span></div>
|
||||
</div>
|
||||
|
||||
@@ -35,7 +35,7 @@ curl -fsS http://127.0.0.1:20200/readyz
|
||||
| --- | --- |
|
||||
| `vehicle_gateway_active_connections` | Current TCP connections by protocol. Labels: `protocol`. |
|
||||
| `vehicle_gateway_frames_total` | Protocol frames received and parsed by the gateway. Labels: `protocol`, `status`. |
|
||||
| `vehicle_gateway_publish_total` | Gateway publish result for raw and unified events. Labels: `protocol`, `kind`, `status`. |
|
||||
| `vehicle_gateway_publish_total` | Gateway publish result for raw events. Unified appears only when the compatibility stream is explicitly enabled. Labels: `protocol`, `kind`, `status`. |
|
||||
| `vehicle_bridge_messages_total` | NATS messages fetched by the bridge. Labels: `subject`, `status`. |
|
||||
| `vehicle_bridge_kafka_writes_total` | Bridge writes to Kafka. Labels: `topic`, `status`. |
|
||||
| `vehicle_bridge_nats_acks_total` | NATS ack results after Kafka write. Labels: `subject`, `status`. |
|
||||
|
||||
@@ -33,8 +33,8 @@
|
||||
| --- | --- |
|
||||
| `lingniu-go-history-writer.service` | `vehicle.raw.go.gb32960.v1`、`vehicle.raw.go.jt808.v1`、`vehicle.raw.go.yutong-mqtt.v1` |
|
||||
| `lingniu-go-stat-writer.service` | `vehicle.raw.go.gb32960.v1`、`vehicle.raw.go.jt808.v1`、`vehicle.raw.go.yutong-mqtt.v1` |
|
||||
| `lingniu-go-realtime-api.service` | `vehicle.event.go.unified.v1` |
|
||||
| `lingniu-go-nats-kafka-bridge.service` | NATS subjects 到同名 Kafka topic；包含三条 raw.go topic 和一条 unified topic |
|
||||
| `lingniu-go-realtime-api.service` | `vehicle.raw.go.gb32960.v1`、`vehicle.raw.go.jt808.v1`、`vehicle.raw.go.yutong-mqtt.v1` |
|
||||
| `lingniu-go-nats-kafka-bridge.service` | NATS raw subjects 到同名 Kafka topic；`vehicle.event.go.unified.v1` 只在兼容开关打开时使用 |
|
||||
|
||||
如果 stat-writer 少消费某个 raw topic，对应协议的每日里程不会进入 `vehicle_daily_mileage`。修正后可能出现短时间 Kafka lag，这是在追补历史 backlog；只要 `vehicle_stat_writes_total` 持续增长且 lag 下降，就是健康状态。
|
||||
|
||||
|
||||
Reference in New Issue
Block a user