shishengliang/lingniu-vehicle-ingest
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Compare commits

base: shishengliang:b452be3b94ba759d70d178f152f9b3697fb9d797
Branches Tags
shishengliang:main shishengliang:codex/vehicle-map-open-platform shishengliang:go
...
compare: shishengliang:3fabcf181aac79f67134493e47e85c91257ed24e
Branches Tags
shishengliang:codex/vehicle-map-open-platform shishengliang:go shishengliang:main

3 Commits
b452be3b94 ... 3fabcf181a

Author SHA1 Message Date
lingniu
3fabcf181a feat(platform): consolidate production vehicle data workflows 2026-07-15 23:26:29 +08:00
lingniu
6cddc0a43d feat: add vehicle mileage statistics 2026-07-14 13:18:52 +08:00
lingniu
bb59303a4b feat: build vehicle data platform and production pipeline 2026-07-14 12:35:33 +08:00
300 changed files with 94916 additions and 5037 deletions
Expand all files Collapse all files

5
.gitignore vendored
View File

@@ -40,3 +40,8 @@ hs_err_pid*
replay_pid*
.worktrees/
# Local frontend caches and generated review artifacts
.vite/
outputs/
vehicle-data-platform/apps/web/public/app-config.js

21
deploy/systemd/lingniu-go-fields-projector.service Normal file
View File

@@ -0,0 +1,21 @@
[Unit]
Description=Lingniu Go RAW Fields Projector
After=network-online.target
Wants=network-online.target
[Service]
Type=simple
WorkingDirectory=/opt/lingniu-go-native/current
EnvironmentFile=/opt/lingniu-go-native/env/base.env
EnvironmentFile=/opt/lingniu-go-native/env/fields-projector.env
Environment=GOMEMLIMIT=384MiB
ExecStart=/opt/lingniu-go-native/current/fields-projector
Restart=always
RestartSec=3
LimitNOFILE=1048576
MemoryLimit=512M
KillSignal=SIGTERM
TimeoutStopSec=30
[Install]
WantedBy=multi-user.target

86
docs/architecture/production-data-plane-inventory.md
View File

@@ -10,11 +10,14 @@
| 服务 | systemd 单元 | 端口 | 职责 |
| --- | --- | --- | --- |
| Gateway | `lingniu-go-gateway.service` | `0.0.0.0:32960`、`0.0.0.0:808`、`127.0.0.1:20211` | GB32960、JT808、宇通 MQTT 接入和协议解析 |
| NATS Kafka bridge | `lingniu-go-nats-kafka-bridge.service` | `127.0.0.1:20214` | NATS JetStream 到 Kafka 的可靠桥接 |
| History writer | `lingniu-go-history-writer.service` | `127.0.0.1:20212` | Kafka raw 消费，写 TDengine |
| Stat writer | `lingniu-go-stat-writer.service` | `127.0.0.1:20213` | Kafka raw 消费，写每日里程 |
| Realtime API/projector | `lingniu-go-realtime-api.service` | `0.0.0.0:20200` | Kafka raw 消费，写 Redis/MySQL 实时表，并提供 API |
| Gateway | `lingniu-go-gateway.service` | `0.0.0.0:32960`、`0.0.0.0:808`、`127.0.0.1:20211` | GB32960、JT808、宇通 MQTT 接入、协议解析、即时鉴权响应和内存身份解析 |
| NATS fast writer | `lingniu-go-nats-fast-writer.service` | `127.0.0.1:20215` | NATS raw 消费，写 Redis 实时投影；生产默认关闭 TDengine stage |
| NATS Kafka bridge | `lingniu-go-nats-kafka-bridge.service` | `127.0.0.1:20214` | NATS canonical raw 到 Kafka raw/fields 的可靠桥接和字段投影 |
| History writer | `lingniu-go-history-writer.service` | `127.0.0.1:20212` | 多 worker 消费 Kafka raw，写 TDengine |
| Stat writer | `lingniu-go-stat-writer.service` | `127.0.0.1:20213` | 多 worker 消费 Kafka fields，写每日里程 |
| Realtime writer | `lingniu-go-realtime-writer.service` | `127.0.0.1:20216` | 多 worker 消费 Kafka raw，写 MySQL 实时表；生产关闭 Redis projector |
| Identity writer | `lingniu-go-identity-writer.service` | `127.0.0.1:20217` | 多 worker 消费 Kafka JT808 raw，幂等写注册、鉴权和低频在线触达事实 |
| Realtime API | `lingniu-go-realtime-api.service` | `0.0.0.0:20200` | 读取 Redis/MySQL/TDengine 并提供查询 API，不消费 Kafka |
## 总线
@@ -25,24 +28,41 @@ NATS 部署在 Kafka ECS 内网 `172.17.111.56:4222`。
| 项 | 当前值 |
| --- | --- |
| Stream | `VEHICLE_INGEST` |
| Subjects | `vehicle.raw.go.gb32960.v1`、`vehicle.raw.go.jt808.v1`、`vehicle.raw.go.yutong-mqtt.v1` |
| Subjects | Gateway 只新增三类 `vehicle.raw.go.*`；三类 `vehicle.fields.go.*` 暂留在 Stream 合同中兼容切换前消息，不再由 Gateway 直接写入 |
| Durable consumer | `vehicle-kafka-bridge` |
| 语义 | Gateway 先写 NATS raw subject；bridge 写 Kafka 成功后才 ACK NATS |
| 语义 | Gateway 每帧只发布一份 canonical raw；bridge 复用其中已计算的 `parsed_fields` 生成 fields，Kafka raw 与 fields 都成功后才 ACK 同一条 NATS raw |
| 保留策略 | `NATS_STREAM_MAX_BYTES=21474836480`，`NATS_STREAM_MAX_AGE_HOURS=24`，`NATS_STREAM_ENSURE_TIMEOUT_SECONDS=60`，NATS 仅做入口缓冲 |
协议 parser 和字段扁平化都只在 Gateway 接入线程执行一次。TDengine、Redis、MySQL、bridge 和统计链路只能读取 envelope 中预计算的 `parsed_fields`；实时帧缺失该字段时 RAW 仍保留，但实时投影会以 `skipped_missing_fields` 跳过，禁止在存储层重新解析或误标在线。
Gateway 到 JetStream 的生产接受边界是本地分段 WAL：canonical RAW 先进入 1ms/256 条分组提交并完成 `fsync`，然后才异步发布 JetStream；只有收到 PubAck 才把记录标记完成，提交失败或 ACK 超时会释放给有界重放。进程崩溃后从 WAL 重建待发布记录，使用稳定 `Nats-Msg-Id=kind:subject:event_id` 让崩溃窗口内的重复重放由 JetStream 去重。生产基线为 16MiB/5s 分段、100000 append queue、10000 PubAck inflight；禁止和旧逐条 JSON spool 共用目录。Redis 快路径使用 `FAST_WRITER_WORKERS=16`、`FAST_WRITER_BATCH_SIZE=100`、`FAST_WRITER_FETCH_WAIT_MS=5`，仍在 Redis 成功后手工 ACK NATS。
### Kafka
当前 Go 链路正在使用的 topic：
| Topic | 分区 | 写入方 | 主要消费方 |
| --- | --- | --- | --- |
| `vehicle.raw.go.gb32960.v1` | 12 | NATS Kafka bridge | history、stat、realtime |
| `vehicle.raw.go.jt808.v1` | 12 | NATS Kafka bridge | history、stat、realtime |
| `vehicle.raw.go.yutong-mqtt.v1` | 12 | NATS Kafka bridge | history、stat、realtime |
| `vehicle.raw.go.gb32960.v1` | 12 | NATS Kafka bridge | history、realtime |
| `vehicle.raw.go.jt808.v1` | 12 | NATS Kafka bridge | history、realtime、identity |
| `vehicle.raw.go.yutong-mqtt.v1` | 12 | NATS Kafka bridge | history、realtime |
| `vehicle.fields.go.gb32960.v1` | 12 | NATS Kafka bridge 从 canonical raw 投影 | stat |
| `vehicle.fields.go.jt808.v1` | 12 | NATS Kafka bridge 从 canonical raw 投影 | stat |
| `vehicle.fields.go.yutong-mqtt.v1` | 12 | NATS Kafka bridge 从 canonical raw 投影 | stat |
| `vehicle.event.go.unified.v1` | 12 | 显式打开 `PUBLISH_UNIFIED_ENABLED=true` 后才写入 | 兼容旧统一事件消费；当前最小链路不依赖 |
审计时 Kafka 上仍存在旧 Java/Xinda/telemetry topic，例如 `vehicle.raw.gb32960.v1`、`vehicle.raw.jt808.v1`、`vehicle.event.xinda.v1`、`vehicle.raw.telemetry-input.v1`。这些不属于当前 Go 最小链路，后续如果确认没有消费者依赖，应单独做 Kafka topic 下线清单，不在业务代码里继续引用。
history、stat、realtime-writer、identity-writer 四个消费服务会以 `vehicle_kafka_consumer_info{service,group,topic}` 暴露当前运行时 Kafka 订阅配置；容量巡检将实时 writer 采集为 `realtime`，将身份 writer 采集为 `identity`，将查询服务采集为 `realtime-api`，以此区分“消费线程实际挂载 topic”和“API 可用性”。
history、stat、realtime-writer、identity-writer 的 Kafka 批消费采用 failure-closed 语义：存储失败后不抓取下一批；history/realtime/stat 只提交各分区连续成功前缀并保留失败后缀，identity 的 MySQL 批事务整体重试；commit 失败时四个 writer 都只重试 commit，不重复写存储。`vehicle_{history,realtime,stat}_retry_pending_messages` 和 `vehicle_identity_writer_retry_pending_messages` 表示当前卡在失败重试中的消息数，生产健康值必须为 `0`。
history、realtime、stat 和 identity writer 默认都在一个进程内启动 3 个同 group Kafka consumer，由 Kafka 分配 topic 分区。bridge 使用车辆键作为 Kafka key，因此同一车辆、同一协议的消息仍在单一分区内有序处理；不同分区分别并行写 TDengine、MySQL 实时投影、MySQL 统计和 JT808 身份事实。`HISTORY_WORKERS`、`REALTIME_WORKERS`、`STATS_WORKERS`、`IDENTITY_WRITER_WORKERS` 只能在 topic 分区数和后端连接容量允许的范围内调整，不能靠并发打乱单车状态、累计里程或注册鉴权顺序。
生产环境中 `nats-fast-writer` 只负责 Redis 快速当前态，`FAST_WRITER_TDENGINE_ENABLED=false`；TDengine RAW/位置历史由 `history-writer` 通过 Kafka raw topic 单路写入，避免 NATS fast path 与 Kafka history path 双写同一证据层。
生产环境中 Redis 当前态只允许 `nats-fast-writer` 写入。`realtime-writer` 的 `REALTIME_REDIS_PROJECTOR_ENABLED=false`，只保留 MySQL 当前态投影；`realtime-api` 只负责查询。指标 `vehicle_realtime_redis_projector_enabled` 必须为 `0`，否则 `capacity-check` 判为 degraded，避免 Kafka 慢链路覆盖 NATS 快链路的实时结果。
## TDengine
数据库：`lingniu_vehicle_ts`
@@ -51,7 +71,7 @@ NATS 部署在 Kafka ECS 内网 `172.17.111.56:4222`。
| --- | --- | --- | --- |
| `raw_frames` | `ts`、`frame_id`、`event_id`、`message_id`、`event_time`、`received_at`、`raw_size_bytes`、`raw_hex`、`raw_text`、`parsed_json`、`parse_status`、`parse_error`、`source_endpoint` | `protocol`、`vehicle_key`、`vin`、`phone`、`device_id` | RAW 证据层，保存完整解析 JSON |
| `raw_frame_payload_chunks` | 分片 payload 列 | 协议和车辆标识 tags | 保存超长 raw/parsed payload |
| `vehicle_locations` | `ts`、`event_id`、`frame_id`、`received_at`、`longitude`、`latitude`、`altitude_m`、`speed_kmh`、`direction_deg`、`alarm_flag`、`status_flag`、`total_mileage_km` | `protocol`、`vin` | 高频历史位置和总里程查询；只写 VIN 非空车辆 |
| `vehicle_locations` | `ts`、`event_id`、`frame_id`、`received_at`、`longitude`、`latitude`、`altitude_m`、`speed_kmh`、`direction_deg`、`alarm_flag`、`status_flag`、`total_mileage_km`、`soc_percent` | `protocol`、`vin` | 高频历史位置、SOC 和总里程查询；只写 VIN 非空车辆。writer 启动时幂等执行 `ADD COLUMN soc_percent`，兼容已有 stable |
不应重新出现：
@@ -64,15 +84,31 @@ NATS 部署在 Kafka ECS 内网 `172.17.111.56:4222`。
| 表 | 核心字段 | 写入方 | 职责 |
| --- | --- | --- | --- |
| `vehicle_identity_binding` | `vin`、`plate`、`phone`、`oem` | 导入/人工维护，Gateway 只读 | 通过车牌、手机号反查 VIN |
| `jt808_registration` | `phone`、`device_id`、`plate`、`vin`、`manufacturer`、`auth_token`、`source_endpoint`、首次/最新注册鉴权时间 | Gateway | JT808 注册、鉴权和 VIN 匹配状态 |
| `vehicle_realtime_snapshot` | `protocol`、`vin`、`plate`、`event_time`、`received_at`、`event_id` | Realtime API/projector | 每协议每 VIN 最新事件轻量快照 |
| `vehicle_realtime_location` | `protocol`、`vin`、`plate`、经纬度、速度、总里程、SOC、事件时间 | Realtime API/projector | 每协议每 VIN 最新位置业务缓存 |
| `vehicle_daily_mileage` | `vin`、`stat_date`、`protocol`、`daily_mileage_km`、首末总里程、样本数 | Stat writer | 基于总里程差值的每日里程；无 VIN 数据不进入正式统计表 |
| `vehicle` | `vin`、`plate`、`oem`、`enabled` | 导入/人工维护，Gateway 只读 | 车辆主实体，承载 VIN 到默认车牌/厂商的轻量索引 |
| `vehicle_identifier` | `protocol`、`source_code`、`identifier_type`、`identifier_value`、`vin`、`plate`、`oem` | 映射文件导入/人工维护，Gateway 只读 | 多平台、多标识反查 VIN；当前主要承载 JT808 手机号/车牌映射 |
| `vehicle_identity_binding` | `vin`、`plate`、`phone`、`oem` | 导入/人工维护，Gateway 兼容只读 | 旧版 VIN/车牌/手机号事实表；新映射导入会用它反查 VIN，运行期解析优先使用 `vehicle_identifier` |
| `jt808_registration` | `phone`、`device_id`、`plate`、`vin`、`manufacturer`、`auth_token`、`source_endpoint`、首次/最新注册鉴权时间 | Identity writer | JT808 注册、鉴权和 VIN 匹配状态；可从 Kafka JT808 raw 重放重建 |
| `vehicle_realtime_snapshot` | `protocol`、`vin`、`plate`、`platform_name`、`peer`、`parsed_json`、`event_time`、`received_at`、`event_id`；`access_first_seen_at/access_previous_received_at/access_latest_received_at/access_report_interval_ms/access_sample_count/access_latest_event_id/access_first_seen_source` | Realtime writer | 每协议每 VIN 最新合并扁平字段快照；同一条原子 upsert 还维护与设备事件时间解耦的接收证据。重复 event ID、相同或回退接收时间不推进连续间隔；旧行只做 `snapshot_backfill` 上线基线，不宣称历史首次接入 |
| `vehicle_realtime_location` | `protocol`、`vin`、`plate`、经纬度、速度、总里程、SOC、事件时间 | Realtime writer | 每协议每 VIN 最新位置业务缓存；由 Kafka raw 写入 MySQL，不参与 Redis 当前态 |
| `vehicle_data_source` | `protocol`、`source_ip`、`source_code`、`platform_name`、`trust_priority`、`enabled` | Stat writer / 映射导入 / 人工维护 | 车辆数据来源管理；`source_code` 是机器可用的稳定来源编码，`platform_name` 是展示和人工维护名称 |
| `vehicle_daily_mileage_source` | `vin`、`stat_date`、`protocol`、`source_key`、`source_ip`、首末总里程、样本数、质量状态、是否选中 | Stat writer | 每协议、每来源的每日里程事实层；高频更新，保留多源候选 |
| `vehicle_daily_mileage` | `vin`、`stat_date`、`protocol`、`source_id`、`daily_mileage_km`、`latest_total_mileage_km` | Stat writer | 对外查询的每日里程结果层；由 `vehicle_daily_mileage_source` 按来源优先级/质量选举投影 |
身份表使用业务主键：`vehicle_identity_binding` 以 `vin` 为主键，`jt808_registration` 以 `phone` 为主键；两者不保留代理自增主键和 `created_at`。`vehicle_identity_binding` 是外部维护的事实表，服务运行时不回写。
身份表使用业务主键：`vehicle` 以 `vin` 为主键，`vehicle_identifier` 以 `(protocol, source_code, identifier_type, identifier_value)` 为主键，`vehicle_identity_binding` 以 `vin` 为主键，`jt808_registration` 以 `phone` 为主键；这些核心身份表不保留代理自增主键和 `created_at`。`vehicle_identity_binding` 是外部维护的兼容事实表，服务运行时不回写。
日里程表使用 `(vin, stat_date, protocol)` 作为业务主键，不保留自增 `id` 和 `created_at`；首末总里程和样本数保留用于解释差值计算。
Gateway 默认每 60 秒把 `vehicle_identity_binding`、`vehicle_identifier`、`jt808_registration` 和 `vehicle_data_source` 原子刷新为本机只读身份快照。808 每帧只查内存，注册帧仍会立即更新当前 Gateway 的 phone 会话并返回鉴权码；持久化由 `identity-writer` 消费 Kafka JT808 raw 完成，Gateway 必须设置 `JT808_REGISTRATION_GATEWAY_WRITES_ENABLED=false`。注册、鉴权帧不节流，普通位置帧默认每个 phone 每 10 分钟触达一次；MySQL 写失败时不提交 Kafka offset，恢复后继续重放。快照刷新失败继续使用上一版，MySQL 不可用不会阻断 TCP 接入。
NATS→Kafka Bridge 只消费 canonical RAW，并从 RAW 中已有的 `parsed_fields` 派生 fields envelope，不重新解析协议。单批次按 Kafka topic 分组并发写入，默认 `BRIDGE_KAFKA_WRITE_CONCURRENCY=6`；同一个 RAW 对应的 RAW/fields topic 全部成功后才 ACK NATS，失败时保留源消息重放。因此并行化只缩短独立 topic 的网络确认等待，不改变 RAW 证据和统计字段的一致性边界。
日里程事实表 `vehicle_daily_mileage_source` 使用 `(vin, stat_date, protocol, source_key)` 作为业务主键，保留首末总里程和样本数用于解释差值计算；对外结果表 `vehicle_daily_mileage` 使用 `(vin, stat_date, protocol)` 作为业务主键，不保留自增 `id`、`created_at` 和候选来源明细。
Stat writer 每条有效里程样本都会更新 `vehicle_daily_mileage_source`，但 `vehicle_daily_mileage` 的选举投影默认按 `STATS_PROJECT_INTERVAL_SECONDS=15` 节流；新车辆/新协议/新日期/新来源会立即投影。这样保留每日里程事实的实时性，同时避免高频帧对 MySQL 结果表做多 SQL 放大。
每日里程只采用同来源累计里程边界差值：`当日最新累计总里程 - 最近历史日最后累计总里程`。优先取前一自然日；前一日没有数据时继续向更早日期查找，历史完全为空才以当天第一条为基线。不同协议、平台和终端来源分别计算候选，再由来源质量和优先级选举最终结果，不混用两个来源的累计总里程。
`vehicle_data_source` 以 `(protocol, source_ip)` 唯一识别来源。直连终端可能产生很多来源 IP，这些记录可以保留但默认不要求平台名；转发平台来源通过 `vehicle_identifier.source_code` 和 `jt808_registration.source_endpoint` 推断后补充 `source_code/platform_name`，后续统计优先使用 `source_id/trust_priority/enabled` 做可信源选择。
统计分日优先使用协议事件时间；如果事件时间明显晚于接收时间超过 10 分钟，则认为设备时间异常，使用接收时间归属统计日。RAW 历史仍保留原始事件时间，便于追溯。
两张实时当前态表均使用 `(protocol, vin)` 作为业务主键，不保留自增 `id` 和 `created_at`；最新更新时间使用 `updated_at`。
@@ -91,16 +127,22 @@ Redis 使用 DB 50，定位为实时缓存，不作为历史事实来源。
| `vehicle:latest:{vin}` | 跨协议合并后的最新核心字段快照，不重复保存完整 parsed；只写 VIN 非空车辆 |
| `vehicle:latest:{vin}:{protocol}` | 单协议最新轻量快照，保留核心 fields 和时间信息，不重复保存完整 parsed |
| `vehicle:realtime-raw:{protocol}:{vin}` | 单协议最新完整 parsed 状态，是实时完整协议字段的唯一 Redis 副本 |
| `vehicle:online:{vin}` | 在线状态和 TTL |
| `vehicle:rt-kv:{protocol}:{vin}:values` | 单协议扁平化实时字段值，字段名遵循协议字段映射 |
| `vehicle:rt-kv:{protocol}:{vin}:types` | `values` 中每个字段的值类型 |
| `vehicle:rt-kv:{protocol}:{vin}:times` | `values` 中每个字段最后写入的归一事件时间；快路径用它防止乱序旧帧覆盖新字段 |
| `vehicle:rt-kv:{protocol}:{vin}:meta` | 单协议 KV 投影的最新事件、接收时间和字段映射版本 |
| `vehicle:online:{protocol}:{vin}` | 在线状态和 TTL |
| `vehicle:online-state:{protocol}:{vin}` | 在线状态的 Hash 副本，便于分页查询 |
| `vehicle:protocols:{vin}` | 当前车辆最近出现过的协议集合 |
| `vehicle:last_seen` | 最近活跃车辆排序集合 |
审计时活跃 key 族主要是 `vehicle:latest:*`、`vehicle:realtime-raw:*`、`vehicle:online:*`、`vehicle:protocols:*`、`vehicle:last_seen`。旧文档中的 `vehicle:realtime:*`、`vehicle:merged:*` 不是当前活跃 key 族。
审计时活跃 key 族主要是 `vehicle:latest:*`、`vehicle:realtime-raw:*`、`vehicle:rt-kv:*`、`vehicle:online:*`、`vehicle:online-state:*`、`vehicle:protocols:*`、`vehicle:last_seen`。旧文档中的 `vehicle:realtime:*`、`vehicle:merged:*` 不是当前活跃 key 族。
## 后续优化约束
1. 接入层先写 NATS，Kafka 是持久回放层，下游消费者不能反向依赖 Gateway 内存状态。
1. 接入层每帧只解析和扁平化一次并写一份 NATS canonical raw，Kafka 是持久回放层；fields 必须由可重放消费者从 raw 中已有的 `parsed_fields` 投影，不能由 Gateway 双写或由存储层重算。
2. TDengine 只放高写入时序数据：RAW 证据和位置历史。
3. MySQL 只放低基数业务状态：身份、实时轻量快照、每日里程。
4. Redis 只放当前态，所有 key 都必须允许 TTL 过期后从 Kafka/TDengine/MySQL 重建。
5. 协议新增字段默认进入 `raw_frames.parsed_json`；VIN 非空车辆同步进入 Redis realtime-raw。只有稳定查询需求出现后才提升为 TDengine/MySQL 列。
5. 协议新增字段默认进入 `raw_frames.parsed_json` 物理列中的扁平 `parsed_fields` JSON，并同步进入 Redis KV；只有稳定查询需求出现后才提升为 TDengine/MySQL 独立列。
6. Gateway 不直接写业务数据库；JT808 即时会话留在内存，注册鉴权事实由 Kafka identity-writer 单路投影。

12
docs/architecture/storage-minimal-contract.md
View File

@@ -12,21 +12,22 @@
4. TDengine 只承担高写入时间序列：raw、位置历史。
5. MySQL 只承担低基数业务状态：身份映射、实时快照、日指标。
6. 同一个事实只落一个主表，其他表只存查询必要的投影。
7. 新字段先进入 `parsed_json`，只有稳定查询需求出现后才提升为列。
7. 新字段先由 Gateway 一次扁平化后进入 `parsed_fields`；TDengine 物理列仍名为 `parsed_json`，只有稳定查询需求出现后才提升为列。
## 目标表边界
| 存储 | 表/Key | 保留内容 | 不保留内容 |
| --- | --- | --- | --- |
| TDengine | `raw_frames` | raw hex/text、完整 `parsed_json`、解析状态、协议标签、车辆标识标签 | `fields_json` 这类可从 `parsed_json`/业务表再得到的重复字段 |
| TDengine | `raw_frames` | raw hex/text、扁平 `parsed_fields` JSON（物理列 `parsed_json`）、解析状态、协议标签、车辆标识标签 | 原始嵌套解析树、`fields_json` 等重复字段 |
| TDengine | `raw_frame_payload_chunks` | 超长 raw/parsed payload 分片 | 业务查询字段 |
| TDengine | `vehicle_locations` | 时间、VIN/协议标签、经纬度、速度、方向、SOC、总里程等位置核心字段 | 完整协议 JSON、注册鉴权信息、phone/device_id/vehicle_key 兜底标识 |
| MySQL | `vehicle_realtime_snapshot` | 每个协议+VIN 的最新事件时间、接收时间、车牌、事件 ID | phone、device、source endpoint、完整 JSON |
| MySQL | `vehicle_realtime_snapshot` | 每个协议+VIN 的最新事件态、合并扁平字段，以及独立的首次/前次/最新接收时间、连续间隔、样本数和证据来源 | phone、device、原始嵌套解析树、raw 报文、无限接收历史 |
| MySQL | `vehicle_realtime_location` | 每个协议+VIN 的最新位置核心字段 | raw、parsed JSON、消息头内部字段 |
| MySQL | `vehicle_daily_mileage` | 日期、VIN、协议、日里程、首末总里程、样本数 | 临时 vehicle key、泛化 metric key/value、自增 id、created_at、每帧细节、位置点列表 |
| MySQL | `vehicle_identity_binding` | 人工维护或导入的 VIN、车牌、phone、oem 映射 | 注册历史、协议状态、device_id、自增 id、created_at |
| MySQL | `jt808_registration` | JT808 phone 主键下的注册、鉴权、VIN 匹配状态、来源端点 | GB32960/MQTT 注册信息、位置历史、created_at |
| Redis | `vehicle:realtime-raw:{protocol}:{vin}` | 每协议每 VIN 最新完整 parsed 状态 | 无 VIN 临时身份、历史数据、统计结果 |
| Redis | `vehicle:rt-kv:{protocol}:{vin}:values/types/times/meta` | 每协议每 VIN 最新扁平实时字段、类型、字段级事件时间和投影元信息 | 历史字段值、统计结果、无 VIN 临时身份 |
## 当前应收敛的重复点
@@ -60,6 +61,7 @@
6. MySQL realtime 当前态表不保留代理主键和创建时间。
- 原因：`vehicle_realtime_snapshot`、`vehicle_realtime_location` 都是每协议每 VIN 一行的当前态投影，业务主键就是 `(protocol, vin)`。
- 接入证据按接收时间单调推进，不受设备事件时间乱序影响；重复事件 ID 或回退接收时间不能增加样本数。`access_first_seen_source=snapshot_backfill` 仅表示部署时基线，只有 `live_writer` 新行能证明 writer 上线后的首次观测。
- 状态：schema 使用 `PRIMARY KEY (protocol, vin)`，不保留自增 `id` 和 `created_at`，对外只暴露最新 `updated_at`。
- 索引：不在 `vehicle_realtime_location` 维护经纬度组合索引；实时表只回答当前态，地理范围和轨迹类查询走 TDengine 历史位置。
- 查询：`/api/realtime/snapshots`、`/api/realtime/locations` 默认不执行 `COUNT(*)`，`total` 表示本页返回条数；只有需要精确总数时传 `includeTotal=true`。
@@ -83,9 +85,9 @@
## 字段提升规则
协议字段进入系统后分三层：
协议字段只在 Gateway 解析、映射和扁平化一次，进入系统后分三层。`vehicle.fields.*` 事件中的所有字段名必须位于对应协议命名空间：`gb32960.*`、`jt808.*` 或 `yutong_mqtt.*`；统计层不读取内部标准化核心字段作为兼容兜底。
1. `parsed_json`：默认入口，保存完整结构化解析。
1. `parsed_fields`：默认入口，保存全量扁平协议字段；TDengine 继续使用兼容物理列名 `parsed_json`。
2. 核心列：只有跨协议稳定查询需要时才提升，例如经纬度、速度、SOC、总里程。
3. 指标表：只有聚合口径稳定、产品需要分页/排序/报表时才持久化。

2
docs/architecture/tdengine-batch-writer-design.md
View File

@@ -69,7 +69,7 @@
- `history.Writer.AppendAllBatch` 按 raw child table 和 location child table 生成多行 `INSERT ... VALUES (...),(...)`。
- oversized payload chunks 仍写入 `raw_frame_payload_chunks`，同样按 child table 批量写。
- `history-writer` Kafka 消费默认按 `HISTORY_BATCH_SIZE=200`、`HISTORY_BATCH_WAIT_MS=100` 收集消息。
- `history-writer` 默认启动 `HISTORY_WORKERS=3` 个同组消费者，每个 worker 按 `HISTORY_BATCH_SIZE=200`、`HISTORY_BATCH_WAIT_MS=20` 收集其已分配分区的消息。
- TDengine batch 成功后才批量提交 Kafka messages；batch 失败不提交 offset，让 Kafka 保留可重放语义。
- 保留 `processHistoryMessage` 和 `AppendAll` 单条路径，便于回退和测试。

51
docs/ops/100k-capacity-baseline.md
View File

@@ -1,6 +1,6 @@
# 100K 车辆接入容量基线
更新时间：2026-07-03
更新时间：2026-07-13
## 目标
@@ -40,6 +40,7 @@
- 已新增可重复的 TCP 连接压测工具，后续需要用它跑 10K/50K/100K 阶段测试并记录结果。
- 仍缺读超时按协议维度计数；Gateway frame duration 已提供 histogram，可用于 parse + enqueue + response 的 p95/p99 估算。
- TDengine history writer 和 NATS fast writer 已使用 batch 写入；后续需要用带帧率压测验证 batch size、flush latency 和存储端承载能力。
- History、realtime、stat、identity writer 均支持单进程多个同 group consumer，默认分别为 `HISTORY_WORKERS=3`、`REALTIME_WORKERS=3`、`STATS_WORKERS=3`、`IDENTITY_WRITER_WORKERS=3`。四者按 Kafka 分区并行且保持单车分区顺序；后续压测需联合观察 TDengine/MySQL 写延迟、连接数和各 consumer lag，再决定是否增加 worker。
- Kafka topic 当前 12 分区，100K 目标下需要结合实际 FPS 再评估分区数。
## ECS OS 参数建议
@@ -73,6 +74,8 @@ systemd gateway 已配置 `LimitNOFILE=1048576`，需要持续保持。
| History writer | `vehicle_history_batch_pending_messages` | 稳态接近 0，burst 后下降 |
| History writer | `vehicle_history_batch_pending_rows` | 稳态接近 0，burst 后下降 |
| History writer | `vehicle_history_batch_flush_duration_ms{status}` | flush 延迟不持续上升 |
| History writer | `vehicle_history_config{setting="workers"}` | 不低于 `3` |
| History writer | `vehicle_history_worker_active` | 每个配置 worker 均为 `1` |
| NATS bridge | `vehicle_bridge_nats_consumer_ack_pending` | 稳态为 0 |
| NATS bridge | `vehicle_bridge_nats_consumer_pending` | burst 后下降 |
| NATS bridge | `vehicle_bridge_batch_pending_messages` | 稳态接近 0，burst 后下降 |
@@ -86,7 +89,12 @@ systemd gateway 已配置 `LimitNOFILE=1048576`，需要持续保持。
| Realtime Redis | `vehicle_realtime_store_update_duration_ms_histogram_bucket{store="redis"}` | p99 毫秒级且不持续上升 |
| Realtime MySQL | `vehicle_realtime_store_update_duration_ms_histogram_bucket{store="mysql"}` | p99 不持续上升 |
| Realtime MySQL | async queue depth | 不持续增长 |
| Realtime MySQL | `vehicle_realtime_config{setting="workers"}` | 不低于 `3` |
| Realtime MySQL | `vehicle_realtime_worker_active` | 每个配置 worker 均为 `1` |
| Stat writer | `vehicle_stat_write_duration_ms_histogram_bucket` | p99 不持续上升 |
| Stat writer | `vehicle_stat_config{setting="workers"}` | 不低于 `3` |
| Stat writer | `vehicle_stat_worker_active` | 每个配置 worker 均为 `1` |
| Identity writer | `vehicle_identity_writer_worker_active` | 每个配置 worker 均为 `1` |
## Retention Guardrails
@@ -96,7 +104,8 @@ systemd gateway 已配置 `LimitNOFILE=1048576`，需要持续保持。
| --- | --- |
| NATS JetStream `VEHICLE_INGEST` | `NATS_STREAM_MAX_BYTES=21474836480`，`NATS_STREAM_MAX_AGE_HOURS=24`，`NATS_STREAM_ENSURE_TIMEOUT_SECONDS=60` |
| Kafka `vehicle.raw.go.*` / `vehicle.fields.go.*` | `retention.ms=21600000`、`segment.ms=600000`、`segment.bytes=268435456` |
| Fast writer | `FAST_WRITER_OPERATION_TIMEOUT_MS=1000`，避免 TDengine 批量写尾延迟导致整批重投 |
| Gateway NATS publisher | `NATS_ASYNC_RAW_WORKERS=128`，吸收平台按秒集中上报形成的瞬时突发；以 `vehicle_async_sink_queue_wait_recent_p99_ms` 校验，不按稳态 queue depth 猜测 |
| Fast writer | `FAST_WRITER_WORKERS=16`、`FAST_WRITER_BATCH_SIZE=100`、`FAST_WRITER_FETCH_WAIT_MS=5`、`FAST_WRITER_OPERATION_TIMEOUT_MS=1000`；继续使用手工 ACK，避免以可靠性换延迟 |
| Root disk | 使用率低于 `80%`，超过 `85%` 进入容量告警 |
如果 NATS `consumer_pending` 下降但仍高，说明系统在追历史积压；如果 `ack_pending=0` 且 Kafka lag 为 `0`，不要重启服务，重点观察 NATS data size 和 pending 下降斜率。
@@ -122,6 +131,8 @@ systemd gateway 已配置 `LimitNOFILE=1048576`，需要持续保持。
- JT808 会变更包头手机号、流水号和 `0x0200` 位置时间，并重新计算转义和校验码。
- GB32960 会变更 VIN 和实时数据时间，并重新计算 BCC。
- JT808 发送模式默认持续读取服务端通用应答；禁止关闭 `-drain-responses` 后用应答写阻塞产生的延迟评价服务端性能。
- JT808 使用 `-cleanup-registration` 时会在结束后只清理指定模拟手机号区间且来源为 loopback 的注册记录；RAW 仍保留作为压测证据。每轮发送压测必须使用未在 identity-writer 10 分钟 touch 节流窗口内使用过的新号段，否则注册不会重复写入，`rows_deleted` 可以为 `0`。
- 低 FPS 全链路压测必须使用该模式，避免重复静态帧导致 event id 冲突。
- 对生产端口执行 `-send=true` 会写入合成 raw 数据，只能在明确隔离标识和压测窗口后执行。
@@ -175,8 +186,42 @@ systemd gateway 已配置 `LimitNOFILE=1048576`，需要持续保持。
- `2026-07-03 18:15:36 CST` 5W 压测后 JT808 active 回落到约 `238`，`ss` 约 `239`。
- `2026-07-03 18:16:49 CST` NATS ack pending 回到 `0`，history/stat/realtime Kafka lag 总和均为 `0`。
- `2026-07-03 18:23:02 CST` 10W 总连接压测后 JT808 active 回落到 `236`，GB32960 active 回落到 `2`，NATS ack pending 为 `0`，history/stat/realtime Kafka lag 总和均为 `0`。
- `20211/20212/20213/20214/20200` readyz 均 OK。
- `20211/20212/20213/20214/20215/20216/20200` readyz 均 OK。
- 压测窗口未出现 gateway `error|failed|panic|fatal|rejected` 日志。
- `vehicle_gateway_connection_rejections_total` 未输出，表示本轮没有连接拒绝计数。
结论：单 ECS 已通过本机 10W 总连接 hold-only 基线测试。该结果证明当前内核参数、gateway 连接上限和 systemd 文件句柄设置可以承载 10W 空闲长连接；下一阶段必须进入带真实帧率的低 FPS 压测，验证解析、NATS、Kafka、Redis、TDengine、MySQL 全链路吞吐。
## 2026-07-13 1000 FPS 全链路压测记录
压测参数：JT808 `0x0200`，1000 个连接，每连接 1 FPS，持续 60 秒，手机号区间 `139000000000-139000000999`；开启服务端应答读取和注册自动清理。每轮均打开 1000 个连接、失败 0，发送 59,500 帧、写错误 0，读取应答约 1.19 MB、读错误 0，结束后清理 1000 条模拟注册。
未读取 JT808 应答的早期结果不作为延迟基线：客户端接收缓冲区会填满并反向阻塞 Gateway 写应答，测到的是压测器缺陷而非服务端真实链路。
| 版本 | 观测点 | Gateway 应答 p99 | Redis p99 | Bridge RAW p99 | TDengine history p99 | MySQL stat p99 |
| --- | --- | ---: | ---: | ---: | ---: | ---: |
| topic 串行写 Kafka | 压测中段 | 约 32ms | 约 212ms | 约 304ms | 约 466ms | 约 332ms |
| topic 串行写 Kafka | 压测结束 | 约 10ms | 约 162ms | 约 326ms | 约 427ms | 约 313ms |
| topic 并行写 Kafka，concurrency=6 | 压测中段 | 约 24ms | 约 207ms | 约 311ms | 约 409ms | 约 247ms |
| topic 并行写 Kafka，concurrency=6 | 压测结束 | 约 29ms | 约 184ms | 约 288ms | 约 368ms | 约 316ms |
并行版本在同一 NATS 批次内按 Kafka topic 分组并发写，RAW 与派生 fields 全部成功后才 ACK 原 NATS 消息；任一 topic 失败仍保留对应源消息待重放。低负载 Bridge p99 从约 110ms 降到约 65ms；1000 FPS 下 history p99 下降约 14%，Bridge RAW p99 下降约 12%。压测中段 NATS pending 短时峰值 71，Gateway async queue 短时 285，结束时均归零；Kafka lag 和各 writer retry 均为 0。主机为 4 核，瞬时 load average 约 9.68，但采样时 CPU 仍约 52% idle、无 D 状态进程、可用内存约 6.4 GiB。
增加队列等待指标后，使用五个全新 JT808 号段进行了 worker A/B；每轮仍是 59,500 帧、发送/读取错误为 0、清理注册 1000 条。以下均为压测中段最近 512 样本窗口，秒级集中发送会比均匀流量更严格：
| Gateway RAW worker | Fast writer worker / fetch | Gateway queue wait p99 | Gateway 应答 p99 | Redis p99 | 结论 |
| ---: | --- | ---: | ---: | ---: | --- |
| 32 | `8 / 20ms` | 约 95ms | 约 92ms | 约 234ms | 入口突发排队明显 |
| 64 | `8 / 20ms` | 约 84ms | 约 79ms | 约 254ms | Gateway 有小幅收益，下游未改善 |
| 128 | `16 / 5ms` | 约 42ms | 约 17ms | 约 151ms | 当前最佳组合 |
| 128 | `32 / 5ms` | 约 50ms | 约 32ms | 约 221ms | pull worker 过多产生调度竞争，回退 |
最终保留 `NATS_ASYNC_RAW_WORKERS=128`、`FAST_WRITER_WORKERS=16`、`FAST_WRITER_FETCH_WAIT_MS=5`。Redis 批写自身 p99 低于 `5ms`，约 151ms 的剩余尾延迟主要位于 JetStream 持久化发布和 consumer delivery；NATS/Kafka pending、Kafka lag 与 writer retry 在每轮结束后均归零。
## 2026-07-14 WAL Outbox 验证
Gateway 已切换为分段 WAL + 异步 JetStream PubAck。WAL 使用 16MiB/5s 分段、1ms/256 条 group commit、`fsync=true`、10000 最大 inflight；本地 Apple M4 并发基准约 1926 records/s（`519144 ns/op`），没有以关闭 `fsync` 换取吞吐。
ECS 生产使用 JT808 `0x0200`、1000 个连接、每连接 1 FPS、持续 60 秒验证：连接成功 1000、失败 0，发送 59500 帧、写错误 0、读错误 0，WAL `submitted=acked=66776`（含同期真实三协议流量），结束后 WAL backlog/inflight、NATS pending、Kafka lag 全部为 0。压测结束 Gateway JT808 响应 recent p99 约 80ms。
随后在约 1000 FPS 下对 Gateway 执行真实 `SIGKILL`。systemd 约 5 秒后自动拉起；启动后立即从崩溃前 WAL 重放并确认约 563 条记录，最终 WAL backlog/inflight、NATS pending 和 history/stat/realtime/identity Kafka lag 均回到 0，日志没有 CRC 损坏、publish error 或 panic。故障窗口内压测客户端的写/读错误是 TCP 连接被强制中断的预期结果，不代表已持久接受的服务端数据丢失。

96
docs/ops/go-service-observability.md
View File

@@ -14,6 +14,7 @@ The Go services expose local-only health and metrics endpoints. They are intende
| History writer | `lingniu-go-history-writer.service` | `127.0.0.1:20212` | `/healthz` | `/readyz` | `/metrics` |
| Stat writer | `lingniu-go-stat-writer.service` | `127.0.0.1:20213` | `/healthz` | `/readyz` | `/metrics` |
| NATS Kafka bridge | `lingniu-go-nats-kafka-bridge.service` | `127.0.0.1:20214` | `/healthz` | `/readyz` | `/metrics` |
| Identity writer | `lingniu-go-identity-writer.service` | `127.0.0.1:20217` | `/healthz` | `/readyz` | `/metrics` |
| Realtime API | `lingniu-go-realtime-api.service` | `0.0.0.0:20200` | `/healthz` | `/readyz` | `/metrics` |
## Quick Checks
@@ -26,6 +27,7 @@ curl -fsS http://127.0.0.1:20211/metrics
curl -fsS http://127.0.0.1:20212/readyz
curl -fsS http://127.0.0.1:20213/readyz
curl -fsS http://127.0.0.1:20214/readyz
curl -fsS http://127.0.0.1:20217/readyz
curl -fsS http://127.0.0.1:20200/readyz
```
@@ -35,7 +37,19 @@ curl -fsS http://127.0.0.1:20200/readyz
/opt/lingniu-go-native/current/capacity-check
```
它会抓取 Gateway、History writer、Stat writer、NATS bridge、Realtime API、NATS fast writer 的本地 `/metrics`，输出 JSON。退出码 `0` 表示当前关键 backlog 和拒绝计数正常，退出码 `2` 表示存在 pending、Kafka lag、连接拒绝或 metrics 抓取失败，适合接入 cron/告警。
它会抓取 Gateway、History writer、Stat writer、Identity writer、NATS bridge、Realtime writer/API、NATS fast writer 的本地 `/metrics`，输出 JSON。退出码 `0` 表示当前关键 backlog 和拒绝计数正常，退出码 `2` 表示存在超阈值 pending、Kafka lag、连接拒绝或 metrics 抓取失败，适合接入 cron/告警。
Identity writer 必须消费 `vehicle.raw.go.jt808.v1`，且 Gateway 指标 `vehicle_gateway_jt808_registration_gateway_writes_enabled` 必须为 `0`。两者同时写 `jt808_registration` 会被 `capacity-check` 判为重复 writer。
Gateway 的 NATS 接受边界由 WAL outbox 提供。`vehicle_durable_outbox_backlog_records` 表示已经 `fsync` 但尚未 PubAck 的记录，`vehicle_durable_outbox_inflight` 表示正在等待 PubAck 的记录；健康时两者最终回到 `0`。`vehicle_durable_outbox_publish_total` 的 `submit_error`、`ack_error`、`remove_error`、`close_timeout` 默认按 outbox 错误门禁处理。稳定 event ID/NATS MsgId 负责崩溃恢复期间的 JetStream 去重。
默认还会用 histogram 估算链路 p99：Gateway frame `250ms`、Gateway identity `100ms`、async sink `100ms`、bridge batch `5000ms`、fast-writer stage `100ms`、history flush `1000ms`、realtime store `100ms`、stat write `100ms`；history/stat/realtime/identity 的 Kafka lag 总和超过 `100` 时降级。Gateway async sink 队列深度默认超过 `10000` 降级，可用 `-async-sink-queue-depth-max` 调整；Gateway 身份快照必须 ready 且最近成功刷新时间不能超过 `180s`，分别用 `-gateway-identity-snapshot-required`、`-gateway-identity-snapshot-max-age-seconds` 调整。Gateway 身份解析缓存默认容量阈值是 `300000`，Stat writer 统计缓存默认容量阈值是 `1000000`，Realtime API 的 MySQL 写入节流缓存和 VIN 车牌缓存默认容量阈值分别是 `1000000` 和 `200000`，可分别用 `-gateway-identity-cache-entries-max`、`-stat-cache-entries-max`、`-realtime-throttle-cache-entries-max`、`-realtime-plate-cache-entries-max` 调整，设为 `0` 表示关闭该项容量检查。history、realtime、stat、identity writer 的 worker 数默认都不得低于 `3`，分别由 `-history-workers-min`、`-realtime-workers-min`、`-stat-workers-min`、`-identity-workers-min` 检查。低样本数会跳过 p99 判断，阈值设为 `0` 可临时关闭对应延迟检查。
批处理内部 in-flight pending 默认允许 `100` 条窗口：`-fast-writer-batch-pending-max=100`、`-history-batch-pending-max=100`、`-history-rows-pending-max=100`、`-stat-batch-pending-max=100`。这类指标用于发现批写卡住，不再因为高频流量下几十条正在 flush/ack 的瞬时批次直接降级；NATS consumer pending、ack pending 和 Kafka lag 仍按独立阈值判断真实积压。
topic 流转检查默认只抓完全断层：history/realtime 必须消费三类 raw topic，stat 必须消费三类 fields topic；bridge 未配置 subject 的 `route_error` 默认超过 `0` 条就降级，可用 `-bridge-route-error-max=-1` 临时关闭；fast-writer `invalid_json` 默认超过 `0` 条就降级，可用 `-fast-writer-invalid-json-max=-1` 临时关闭；history-writer `invalid_json` 默认超过 `0` 条就降级，可用 `-history-invalid-json-max=-1` 临时关闭；realtime projector `invalid_json` 默认超过 `0` 条就降级，可用 `-realtime-invalid-json-max=-1` 临时关闭；stat-writer `invalid_json` 默认超过 `0` 条就降级，可用 `-stat-invalid-json-max=-1` 临时关闭；某协议 gateway raw publish 达到 `100` 条，扣除 `skipped_non_realtime` 后仍然没有 fields publish 时降级；实时 fields 缺失或 publish error 比例超过 `20%` 时也会降级；某协议 gateway raw/fields publish 达到 `1000` 条但 bridge 没有写对应 Kafka topic 时降级；某协议 gateway raw publish 达到 `1000` 条但 fast-writer 没有消费同 raw subject 并成功写 Redis/ack 时降级；某 raw topic bridge 写 Kafka 达到 `1000` 条但 history 未收到/写入或 realtime 未收到/更新时降级；某 fields topic bridge 写 Kafka 达到 `100` 条但 stat-writer 未配置或未收到该 topic 时降级。解析质量检查默认在单协议 gateway frame 样本达到 `100` 条后启用，`OK` 以外的 `PARTIAL/BAD_FRAME/other` 比例超过 `5%` 时降级，用于发现协议解析器或上游报文质量突然恶化。响应质量检查默认在单协议 gateway response 尝试样本达到 `100` 条后启用，`build_error/write_error` 比例超过 `5%` 时降级；`skipped` 表示该消息类型无需响应，不计入分母。身份解析质量检查默认在单协议 identity 样本达到 `100` 条后启用，`resolved` 以外的 `unresolved/error/timeout` 比例超过 `20%` 时降级，用于发现 VIN/车牌/phone 映射突然失效。history-writer 会对实时数据帧检查 raw envelope 是否携带 `parsed_fields`，默认单 topic/protocol 达到 `100` 帧后，缺失比例超过 `5%` 降级，用于发现历史证据层丢失扁平化解析字段。Redis 快路径字段质量检查默认在单 raw subject 写入字段达到 `1000` 个后启用，`skipped_stale/seen` 超过 `20%` 时降级，用于发现大面积旧帧补发或设备时间漂移。stat-writer 还会在单个 fields topic 成功 append 达到 `100` 条但未抽取到任何里程样本时降级，或 VIN 缺失、里程缺失、非正里程、source 缺失这类可行动跳过比例超过 `60%` 时降级，用于发现字段映射或身份/source 关联断裂；单 topic 找到 `100` 条里程样本但没有 source tracking，或 source endpoint 缺失比例超过 `60%` 时也会降级，用于发现 `vehicle_data_source` 来源管理断层；单 topic 已写入 `100` 条里程样本但最终 `vehicle_daily_mileage` 投影写入仍为 `0` 时也会降级，用于发现 source 候选层到最终查询层的断层；`skipped_same_mileage` 是重复总里程去重，不计入可行动跳过比例。已出现过的成功/received `*_last_*_unix_seconds` 如果超过 `300s` 未刷新，也会降级；刚启动尚未出现过该指标时不会因缺失 last activity 降级。`capacity-check` JSON 会额外派生 `*_last_*_age_seconds`，便于直接读取距今秒数。特殊环境可用 `-required-consumer-topics=''` 临时关闭消费 topic 合同检查。
`capacity-check` 还会默认请求 Realtime API 的 `/api/stats/daily-metrics/diagnostics/reasons`，把当天每日里程诊断聚合放到 JSON 的 `daily_mileage_diagnostics`。这项默认只展示 `vehicle_total`、`actionable_issue_total` 和原因分布，不会因为存在业务缺口直接退出 `2`；需要把缺口纳入发布/巡检阻断时，显式传 `-daily-mileage-diagnostics-max-actionable=0` 或其他阈值。诊断 API 不可访问会被视为观测能力故障并降级；特殊环境可用 `-daily-mileage-diagnostics-url=''` 关闭。
ECS 上通过 systemd timer 每分钟执行一次：
@@ -56,32 +70,106 @@ systemctl start lingniu-go-capacity-check.service
| `vehicle_gateway_connection_closes_total` | TCP connection closes by protocol and reason. Labels: `protocol`, `reason`. Reasons include `eof`, `read_timeout`, `read_error`, `extract_error`, `context_cancelled`, `max_connections`. |
| `vehicle_gateway_connection_rejections_total` | TCP connection rejections by protocol and reason. Labels: `protocol`, `reason`. |
| `vehicle_gateway_frames_total` | Protocol frames received and parsed by the gateway. Labels: `protocol`, `status`. |
| `vehicle_gateway_last_frame_unix_seconds` | Last observed gateway frame time by protocol and parse status. Labels: `protocol`, `status`. |
| `vehicle_gateway_frame_duration_ms` | Last observed Gateway frame handling duration and histogram buckets for p95/p99. Labels: `protocol`, `status`. |
| `vehicle_gateway_publish_total` | Gateway publish result for raw events. Unified appears only when the compatibility stream is explicitly enabled. Labels: `protocol`, `kind`, `status`. |
| `vehicle_gateway_identity_total` | Gateway identity resolve results. Labels: `protocol`, `status`; status includes `resolved`, `unresolved`, `error`, `timeout`. |
| `vehicle_gateway_identity_duration_ms` | Last observed identity resolve duration and histogram buckets for p95/p99. Labels: `protocol`, `status`. |
| `vehicle_gateway_identity_cache_entries` | Gateway identity resolver cache entry count. Labels: `cache`; values include runtime `lookup`/`registration`/`source_code`/`location_touch` and `snapshot_binding`/`snapshot_identifier`/`snapshot_registration`/`snapshot_source`. |
| `vehicle_gateway_identity_cache_max_entries` | Gateway identity resolver cache entry cap per cache. Controlled by `IDENTITY_LOOKUP_CACHE_MAX_ENTRIES`, default `300000`. |
| `vehicle_gateway_identity_snapshot_ready` | `1` after at least one complete atomic identity snapshot refresh; `0` means frames continue ingesting but unknown JT808 identities cannot be resolved. |
| `vehicle_gateway_identity_snapshot_entries` | Current identity snapshot entry counts. Labels: `kind` = `binding`, `identifier`, `registration`, `source`. |
| `vehicle_gateway_identity_snapshot_refresh_total` | Snapshot refresh attempts. Labels: `status` = `ok` or `error`; an error keeps the last known good snapshot. |
| `vehicle_gateway_identity_snapshot_last_success_unix_seconds` | Last successful complete snapshot publication time. |
| `vehicle_gateway_publish_total` | Gateway publish/delegation result. In NATS mode raw is published and fields uses `status="delegated"`; unified appears only when explicitly enabled. Labels: `protocol`, `kind`, `status`. |
| `vehicle_gateway_last_publish_unix_seconds` | Last gateway publish time by protocol/kind/status, useful for protocol freshness checks. Labels: `protocol`, `kind`, `status`. |
| `vehicle_gateway_fields_total` | Gateway fields eligibility result. Labels: `protocol`, `status`; NATS mode uses `delegated_to_bridge`, while compatibility direct mode may use `published`/`publish_error`. |
| `vehicle_gateway_response_total` | Gateway protocol response build/write result. Labels: `protocol`, `message_id`, `status`; status includes `ok`, `skipped`, `build_error`, `write_error`. |
| `vehicle_gateway_last_response_unix_seconds` | Last gateway protocol response build/write result time. Labels: `protocol`, `message_id`, `status`. |
| `vehicle_gateway_response_e2e_recent_p99_ms` | Bounded recent p99 from frame receipt through successful protocol response. Labels: `protocol`; capacity target defaults to `100ms`. |
| `vehicle_gateway_response_e2e_recent_samples` | Samples in the bounded response-latency window. Labels: `protocol`. |
| `vehicle_gateway_authentication_total` | Protocol authentication decisions. Labels: `protocol`, `mode`, `source`, `status`; `source=configured` means a configured credential matched, `source=device` means the JT808 phone's snapshot token matched, and `source=none` means no credential matched. `observe` records mismatches without rejecting, while `enforce` returns a protocol failure and closes the connection. |
| `vehicle_gateway_authentication_mode` | Active authentication mode by protocol. Labels: `protocol`, `mode`. |
| `vehicle_gateway_authentication_credentials` | Configured account count, never credential values. Labels: `protocol`. |
| `vehicle_async_sink_publish_duration_ms_histogram` | Async sink publish duration histogram covering worker publish calls to the delegate sink. Labels: `sink`, `kind`, `status`. |
| `vehicle_async_sink_queue_capacity` | Gateway async sink queue capacity. Labels: `sink`. NATS uses `NATS_ASYNC_QUEUE_SIZE`; Kafka fallback uses `KAFKA_ASYNC_QUEUE_SIZE`. |
| `vehicle_bridge_messages_total` | NATS messages fetched by the bridge. Labels: `subject`, `status`. |
| `vehicle_bridge_fields_projection_total` | Fields projection result from canonical raw. Labels: `protocol`, `status`; `published` is healthy, error/skip labels explain why no fields event was emitted. |
| `vehicle_bridge_fields_projection_count` | Latest flattened field count emitted by bridge per protocol/status. |
| `vehicle_bridge_last_message_unix_seconds` | Last NATS message fetched by subject/status. Labels: `subject`, `status`. |
| `vehicle_bridge_kafka_writes_total` | Bridge writes to Kafka. Labels: `topic`, `status`. |
| `vehicle_bridge_nats_acks_total` | NATS ack results after Kafka write. Labels: `subject`, `status`. |
| `vehicle_bridge_last_kafka_write_unix_seconds` | Last bridge Kafka write by topic/status. Labels: `topic`, `status`. |
| `vehicle_bridge_nats_acks_total` | NATS ack results after Kafka write or unrouted-subject isolation. Labels: `subject`, `status`; status includes `ok`, `error`, and `dropped_route_error`. |
| `vehicle_bridge_last_ack_unix_seconds` | Last NATS ack by subject/status. Labels: `subject`, `status`. |
| `vehicle_bridge_nats_consumer_pending` | JetStream messages pending for the bridge durable consumer. |
| `vehicle_bridge_nats_consumer_ack_pending` | JetStream messages delivered to bridge but not yet acked. |
| `vehicle_bridge_nats_consumer_waiting` | Pull requests waiting on the bridge durable consumer. |
| `vehicle_bridge_batch_pending_messages` | NATS messages fetched by the bridge but not yet written to Kafka and acked. |
| `vehicle_bridge_batch_pending_kafka_messages` | Kafka messages prepared for the current bridge batch. |
| `vehicle_bridge_batch_duration_ms_histogram` | Bridge batch duration histogram covering Kafka write and NATS ack. Labels: `status`. |
| `vehicle_history_batch_flush_duration_ms_histogram` | History writer TDengine batch flush duration histogram. Labels: `status`. |
| `vehicle_realtime_store_update_duration_ms_histogram` | Realtime MySQL/Redis projection duration histogram. Labels: `store`, `protocol`, `status`. |
| `vehicle_stat_write_duration_ms_histogram` | Stat writer MySQL write duration histogram. Labels: `topic`, `status`. |
| `vehicle_stat_samples_total` | Stat writer mileage sample results. Labels: `topic`, `protocol`, `status`; status includes `found`, `written`, `skipped_missing_fields`, `skipped_missing_vin`, `skipped_missing_mileage`, `skipped_non_positive_mileage`, `skipped_same_mileage`, `skipped_missing_source`, `event_time_future_adjusted`. `skipped_missing_fields` means the fields topic carried an envelope without flattened fields and should be treated as a stream-contract issue. The last status means device event time was more than 10 minutes ahead of received time and stats used received time instead. |
| `vehicle_fast_writer_nats_consumer_pending` | JetStream messages pending for the fast writer durable consumer. |
| `vehicle_fast_writer_nats_consumer_ack_pending` | JetStream messages delivered to fast writer but not yet acked. |
| `vehicle_fast_writer_nats_consumer_waiting` | Pull requests waiting on the fast writer durable consumer. |
| `vehicle_fast_writer_batch_pending_messages` | NATS fast writer messages fetched but not yet written to TDengine/Redis and acked. |
| `vehicle_fast_writer_batch_pending_envelopes` | Valid parsed envelopes in the current NATS fast writer batch. |
| `vehicle_fast_writer_messages_total` | Fast writer message results by NATS subject. Labels: `subject`, `status`; status includes `ok`, `error`, `invalid_json`, and `ack_error`. `invalid_json` is acked and isolated so it will not block the raw fast path, but it indicates the stream contract is polluted. |
| `vehicle_fast_writer_stage_duration_ms_histogram` | Fast writer stage duration histogram for TDengine, Redis, and NATS ack. The Redis stage is one batch pipeline when the repository supports batch updates. Labels: `subject`, `stage`, `status`. |
| `vehicle_history_writes_total` | TDengine history writes. Labels: `topic`, `status`. |
| `vehicle_fast_writer_redis_envelopes_total` | Redis realtime projection envelope result from the NATS fast path. Labels: `subject`, `status`; status includes `seen`, `updated`, `skipped_non_realtime`, `skipped_missing_vin`, `skipped_missing_vehicle_key`, `skipped_missing_fields`. Missing fields is an ingress contract failure: RAW is retained, but Redis online/KV projection is skipped and never re-flattened downstream. |
| `vehicle_fast_writer_redis_fields_total` | Redis realtime KV field write result from the NATS fast path. Labels: `subject`, `status`; status includes `seen`, `written`, `skipped_stale`. `skipped_stale` means an older event-time frame was allowed to add missing fields but was blocked from overwriting newer field values. |
| `vehicle_fast_writer_last_message_unix_seconds` | Last fast-writer message processing result by subject/status. Labels: `subject`, `status`. |
| `vehicle_fast_writer_last_stage_unix_seconds` | Last fast-writer stage completion by subject/stage/status. Labels: `subject`, `stage`, `status`. |
| `vehicle_history_writes_total` | TDengine history writes for valid envelopes only. Labels: `topic`, `status`; invalid JSON messages are committed after `vehicle_history_kafka_messages_total{status="invalid_json"}` and are not counted as writes. |
| `vehicle_history_parsed_fields_total` | Realtime raw envelopes that did or did not carry precomputed `parsed_fields` before TDengine history write. Labels: `topic`, `protocol`, `status`; status includes `present`, `missing`. |
| `vehicle_history_last_message_unix_seconds` | Last history Kafka message by topic/status. Labels: `topic`, `status`. |
| `vehicle_history_last_write_unix_seconds` | Last history TDengine write result by topic/status. Labels: `topic`, `status`. |
| `vehicle_history_last_commit_unix_seconds` | Last history Kafka commit by topic/status. Labels: `topic`, `status`. |
| `vehicle_history_batch_pending_messages` | Messages fetched by history writer but not yet flushed and committed. |
| `vehicle_history_batch_pending_rows` | Parsed raw envelopes waiting in the current TDengine batch. |
| `vehicle_history_retry_pending_messages` | Fetched messages retained in memory behind a failed TDengine write or Kafka commit. Healthy value is `0`; the writer does not fetch a new batch while this is non-zero. |
| `vehicle_history_batch_retries_total` | Failure-closed history batch retries. Labels: `reason`; `write_error` retries only the uncommitted suffix and `commit_error` retries commit without replaying TDengine writes. |
| `vehicle_history_batch_flush_duration_ms` | Last TDengine batch flush duration. Labels: `status`. |
| `vehicle_history_config{setting="workers"}` | Configured in-process history Kafka consumer count; default and capacity minimum are `3`. |
| `vehicle_history_worker_active` | Active history consumer gauge by `worker`; all configured workers should be `1`. |
| `vehicle_stat_writes_total` | MySQL metric writes. Labels: `topic`, `status`. |
| `vehicle_stat_kafka_messages_total` | Stat writer Kafka message results by fields topic. Labels: `topic`, `status`; `invalid_json` messages are committed and isolated before MySQL mileage statistics. |
| `vehicle_stat_last_message_unix_seconds` | Last stat Kafka message by topic/status. Labels: `topic`, `status`. |
| `vehicle_stat_last_write_unix_seconds` | Last stat MySQL append result by topic/status. Labels: `topic`, `status`. |
| `vehicle_stat_last_commit_unix_seconds` | Last stat Kafka commit by topic/status. Labels: `topic`, `status`. |
| `vehicle_stat_write_duration_ms_histogram` | MySQL stat writer append duration histogram. Labels: `topic`, `status`. |
| `vehicle_stat_config{setting="workers"}` | Configured in-process Kafka consumer count. Default and capacity-check minimum are `3`. |
| `vehicle_stat_worker_active` | Active stat consumer gauge by `worker`; all configured workers should remain `1` while the service is running. |
| `vehicle_stat_batch_pending_messages` | Messages fetched by stat writer but not yet appended to MySQL and committed. Controlled by `STATS_BATCH_SIZE` and `STATS_BATCH_WAIT_MS`; capacity check default threshold is `1000`. |
| `vehicle_stat_retry_pending_messages` | Fetched messages retained in memory behind a failed MySQL write or Kafka commit. Healthy value is `0`. |
| `vehicle_stat_batch_retries_total` | Failure-closed stat batch retries. Labels: `reason`; commit-only retries do not append the mileage sample twice in the same process. |
| `vehicle_stat_cache_entries` | Stat writer in-memory cache entries by cache type. Labels: `cache`; cache includes `last_total_mileage`, `source_seen`, `projection`, and `baseline`. Runtime cap is controlled by `STATS_CACHE_MAX_ENTRIES` and defaults to `1000000`. |
| `vehicle_stat_cache_max_entries` | Stat writer configured per-cache entry cap. `0` means unlimited and should not be used in production without an external memory guard. |
| `vehicle_stat_cache_evictions_total` | Cumulative stat writer cache evictions caused by capacity pressure. Labels: `cache`. |
| `vehicle_stat_samples_total` | Mileage samples extracted and written by stat writer. Use this with `vehicle_stat_writes_total`: write `ok` only means the append call succeeded, while `status="written"` proves a sample reached `vehicle_daily_mileage_source`. |
| `vehicle_stat_sources_total` | Stat writer source tracking results for `vehicle_data_source`. Labels: `topic`, `protocol`, `status`; status includes `attempted`, `written`, `skipped_throttled`, `skipped_missing_endpoint`. This intentionally avoids source IP labels to keep metrics low-cardinality. |
| `vehicle_stat_projections_total` | Final daily mileage projection results from `vehicle_daily_mileage_source` to `vehicle_daily_mileage`. Labels: `topic`, `protocol`, `status`; status includes `attempted`, `written`, `skipped_throttled`. |
| `vehicle_realtime_updates_total` | Redis/MySQL realtime projector updates. Labels: `topic`, `status`. |
| `vehicle_realtime_kafka_messages_total` | Realtime projector Kafka message results by raw topic. Labels: `topic`, `status`; `invalid_json` messages are committed and isolated before Redis/MySQL projection. |
| `vehicle_realtime_last_message_unix_seconds` | Last realtime Kafka message by topic/status. Labels: `topic`, `status`. |
| `vehicle_realtime_last_update_unix_seconds` | Last realtime projection result by topic/status. Labels: `topic`, `status`. |
| `vehicle_realtime_last_commit_unix_seconds` | Last realtime Kafka commit by topic/status. Labels: `topic`, `status`. |
| `vehicle_realtime_store_update_duration_ms_histogram` | Redis/MySQL store update duration histogram. Labels: `store`, `protocol`, `status`. |
| `vehicle_realtime_config{setting="workers"}` | Configured in-process realtime Kafka consumer count; default and capacity minimum are `3`. |
| `vehicle_realtime_worker_active` | Active realtime consumer gauge by `worker`; all configured workers should be `1`. |
| `vehicle_realtime_async_queue_total` | Realtime API async secondary projection queue events for MySQL snapshot/location. Labels: `store`, `protocol`, `status`; status includes `queued`, `dropped`, `closed`. |
| `vehicle_realtime_async_queue_depth` | Realtime API async MySQL projection queue depth. Labels: `store`, `protocol`. |
| `vehicle_realtime_async_queue_capacity` | Realtime API async MySQL projection queue capacity. Labels: `store`; controlled by `MYSQL_REALTIME_ASYNC_QUEUE_SIZE`. |
| `vehicle_realtime_retry_pending_messages` | Fetched messages retained in memory behind a failed realtime projection or Kafka commit. Healthy value is `0`. |
| `vehicle_realtime_batch_retries_total` | Failure-closed realtime batch retries. Labels: `reason`; only the uncommitted suffix is projected again after a write failure. |
| `vehicle_identity_writer_retry_pending_messages` | JT808 raw messages retained behind a failed registration MySQL transaction or Kafka commit. Healthy value is `0`; identity writer does not fetch newer offsets while non-zero. |
| `vehicle_identity_writer_config{setting="workers"}` | Configured identity Kafka consumer count; default and capacity minimum are `3`. |
| `vehicle_identity_writer_worker_active` | Active identity consumer gauge by `worker`; every configured worker should remain `1`. |
| `vehicle_identity_writer_batch_retries_total` | Failure-closed identity retries. Labels: `reason`; `write_error` retries the transaction and `commit_error` retries only Kafka commit. |
| `vehicle_realtime_plate_cache_entries` | Realtime API VIN-to-plate binding cache entry count. Default runtime cap is controlled by `PLATE_CACHE_MAX_ENTRIES` and defaults to `200000`. |
| `vehicle_realtime_plate_cache_max_entries` | Realtime API configured VIN-to-plate cache entry cap. `0` means unlimited and should not be used in production without an external memory guard. |
| `vehicle_realtime_plate_cache_evictions_total` | Cumulative VIN-to-plate cache evictions caused by TTL expiry cleanup or capacity pressure. |
| `vehicle_history_kafka_lag` | Estimated Kafka lag for history writer by topic and partition. |
| `vehicle_stat_kafka_lag` | Estimated Kafka lag for stat writer by topic and partition. |
| `vehicle_realtime_kafka_lag` | Estimated Kafka lag for realtime projector by topic and partition. |

18
docs/ops/go-vehicle-ingest-memory.md
View File

@@ -58,8 +58,8 @@ Go 服务使用 systemd 裸机部署，不使用 Docker。
最近 release：
```text
/opt/lingniu-go-native/releases/production-track-telemetry-20260714082451
/opt/lingniu-go-native/releases/100k-hardening-20260703175730
/opt/lingniu-go-native/releases/parsed-fields-once-20260703163821
```
服务重启：
@@ -511,14 +511,28 @@ vehicle_gateway_frame_duration_ms_histogram_sum{protocol,status}
该耗时覆盖单帧从解析、identity resolve、parsed fields 生成、publish enqueue 到协议响应写入的整体入口处理路径。带帧率压测时用它按协议观察 p95/p99，不能只看 `vehicle_gateway_frame_duration_ms` 最后一条 gauge。
### Gateway identity resolve duration
2026-07-11 已为 Gateway 身份解析单独增加耗时 histogram 和超时分类：
```text
vehicle_gateway_identity_total{protocol,status}
vehicle_gateway_identity_duration_ms_histogram_bucket{protocol,status,le}
vehicle_gateway_identity_duration_ms_histogram_count{protocol,status}
vehicle_gateway_identity_duration_ms_histogram_sum{protocol,status}
```
生产默认使用周期身份快照，每帧只查本机内存；`status="timeout"` 主要用于兼容关闭 `IDENTITY_SNAPSHOT_ONLY_ENABLED` 后的回源查询模式。快照模式应重点观察 `vehicle_gateway_identity_snapshot_ready` 和最近成功刷新时间；RDS 故障时入口继续接收，上一版快照继续服务，未知 phone 暂时 unresolved。
### History writer TDengine 批写
2026-07-03 已实现 history-writer 第一阶段批写：
- `history.Writer.AppendAllBatch` 按 TDengine child table 聚合 raw/location 多行 `INSERT`。
- `cmd/history-writer` 默认 `HISTORY_BATCH_SIZE=200`、`HISTORY_BATCH_WAIT_MS=100`。
- `cmd/history-writer` 默认 `HISTORY_WORKERS=3`、`HISTORY_BATCH_SIZE=200`、`HISTORY_BATCH_WAIT_MS=20`。
- TDengine batch 成功后才批量提交 Kafka messages；失败不提交 offset。
- 保留单条 `AppendAll` 路径作为回退。
- `production-track-telemetry-20260714082451` 起，`vehicle_locations` stable 增加 `soc_percent`。history-writer 在建表后执行幂等 `ALTER STABLE ... ADD COLUMN soc_percent DOUBLE`，只忽略 TDengine 明确的重复列错误；其他迁移错误会阻止 ready，禁止带病继续写入。单条和批量位置 INSERT 都必须携带 SOC。
新增指标：

354
docs/ops/vehicle-ingest-runbook.md
View File

@@ -23,9 +23,12 @@
| --- | --- | --- | --- |
| 接入 | Gateway | `lingniu-go-gateway.service` | `127.0.0.1:20211` |
| 桥接 | NATS Kafka bridge | `lingniu-go-nats-kafka-bridge.service` | `127.0.0.1:20214` |
| 字段投影 | Kafka RAW fields projector | `lingniu-go-fields-projector.service` | `127.0.0.1:20218` |
| 历史 | TDengine writer | `lingniu-go-history-writer.service` | `127.0.0.1:20212` |
| 统计 | MySQL stat writer | `lingniu-go-stat-writer.service` | `127.0.0.1:20213` |
| 实时 | Realtime API/projector | `lingniu-go-realtime-api.service` | `127.0.0.1:20200` |
| 实时写入 | Realtime writer | `lingniu-go-realtime-writer.service` | `127.0.0.1:20216` |
| 身份事实 | JT808 identity writer | `lingniu-go-identity-writer.service` | `127.0.0.1:20217` |
| 实时/API | Realtime API | `lingniu-go-realtime-api.service` | `127.0.0.1:20200` |
## Topic 配置基线
@@ -34,20 +37,113 @@
| 服务 | 必要 topic |
| --- | --- |
| `lingniu-go-history-writer.service` | `vehicle.raw.go.gb32960.v1`、`vehicle.raw.go.jt808.v1`、`vehicle.raw.go.yutong-mqtt.v1` |
| `lingniu-go-stat-writer.service` | `vehicle.raw.go.gb32960.v1`、`vehicle.raw.go.jt808.v1`、`vehicle.raw.go.yutong-mqtt.v1` |
| `lingniu-go-realtime-api.service` | `vehicle.raw.go.gb32960.v1`、`vehicle.raw.go.jt808.v1`、`vehicle.raw.go.yutong-mqtt.v1` |
| `lingniu-go-nats-kafka-bridge.service` | NATS raw subjects 到同名 Kafka topic；`vehicle.event.go.unified.v1` 只在兼容开关打开时使用 |
| `lingniu-go-stat-writer.service` | `vehicle.fields.go.gb32960.v1`、`vehicle.fields.go.jt808.v1`、`vehicle.fields.go.yutong-mqtt.v1` |
| `lingniu-go-realtime-writer.service` | `vehicle.raw.go.gb32960.v1`、`vehicle.raw.go.jt808.v1`、`vehicle.raw.go.yutong-mqtt.v1` |
| `lingniu-go-identity-writer.service` | `vehicle.raw.go.jt808.v1` |
| `lingniu-go-fields-projector.service` | 消费三类 `vehicle.raw.go.*`，把 RAW 中已有的 `parsed_fields` 投影到对应 `vehicle.fields.go.*` |
| `lingniu-go-nats-kafka-bridge.service` | NATS canonical raw 到 Kafka raw，并从 raw 中已有的 `parsed_fields` 投影同协议 Kafka fields；`vehicle.event.go.unified.v1` 只在兼容开关打开时使用 |
如果 stat-writer 少消费某个 raw topic，对应协议的每日里程不会进入 `vehicle_daily_mileage`。修正后可能出现短时间 Kafka lag，这是在追补历史 backlog；只要 `vehicle_stat_writes_total` 持续增长且 lag 下降，就是健康状态。
Gateway 在 NATS 模式下设置 `FIELDS_DERIVE_FROM_RAW_ENABLED=true`，每帧只发布 canonical raw；bridge 设置 `BRIDGE_DERIVE_FIELDS_FROM_RAW_ENABLED=true`，只复用 raw envelope 中已计算一次的 `parsed_fields`，不会重新解析协议原文。bridge 对同一条 NATS raw 生成 Kafka raw 和可选 Kafka fields，两个输出都成功后才 ACK；任一输出失败会保留 NATS 消息等待重放。登录、注册、鉴权等非实时帧只写 raw，不生成 fields。
如果 stat-writer 少消费某个 fields topic，对应协议的每日里程不会进入 `vehicle_daily_mileage`。stat-writer 启动时会拒绝 `vehicle.raw.*` 这类 RAW topic 配置，避免统计链路重新从原始报文解析并破坏 RAW/fields 解耦。修正后可能出现短时间 Kafka lag，这是在追补历史 backlog；`vehicle_stat_writes_total{status="ok"}` 只表示 append 调用成功，真正证明里程样本入库的是 `vehicle_stat_samples_total{status="written"}` 持续增长且 lag 下降。
stat-writer 默认 `STATS_WORKERS=3`，每个 worker 是同一 consumer group 的独立 Kafka reader。Kafka 按车辆 key 固定分区，因此单车累计里程仍按分区顺序执行，不同分区并行。调整 worker 后同时核对 `vehicle_stat_config{setting="workers"}`、每个 `vehicle_stat_worker_active{worker}`、MySQL 连接数、write p99 和 stat lag；worker 不应超过 fields topic 的有效分区并行度。
history-writer 和 realtime-writer 同样默认分别使用 `HISTORY_WORKERS=3`、`REALTIME_WORKERS=3`。每个 worker 只顺序处理 Kafka 分配给自己的分区，TDengine 表缓存、MySQL 实时节流缓存和车牌缓存均支持并发访问。调整后必须同时核对对应的 `*_config{setting="workers"}`、`*_worker_active`、pending/retry、后端连接数和 Kafka lag。
history-writer 启动会对既有 `vehicle_locations` stable 幂等补齐 `soc_percent DOUBLE`。发布后除 `/readyz` 外，必须抽查一台支持 SOC 的车辆在 TDengine 历史位置中 `soc_percent IS NOT NULL`；如果迁移报错且不是明确的重复列错误，服务应保持未就绪，先处理 TDengine DDL/权限，不能跳过迁移后强启。
identity-writer 默认使用 `IDENTITY_WRITER_WORKERS=3`。每个 worker 持有独立的位置触达节流器并共享 MySQL 连接池；Kafka phone/vehicle key 保证同一终端在单一分区内有序，分区再平衡最多产生一次幂等触达。调整后核对 `vehicle_identity_writer_config{setting="workers"}`、`vehicle_identity_writer_worker_active`、pending/retry、MySQL 连接数和 identity lag。
Gateway 和 NATS bridge 启动时也会检查 raw/fields 合同：Kafka raw topic 必须是 `vehicle.raw.*`，fields topic 必须是 `vehicle.fields.*`；NATS raw subject 和 fields subject 不允许相同。若服务启动失败并提示 `raw and fields ... must be different` 或 `fields kafka topic ... must start with "vehicle.fields."`，应先修正 `/opt/lingniu-go-native/env/*.env` 的 topic/subject 配置。
Realtime API 是当前态投影，默认在没有已提交 offset 时从 latest 开始消费。切换 topic 或新建 consumer group 后，不应让 realtime 追扫历史 raw backlog；需要重建当前态时，应使用明确的回放任务或手动 reset offset。
history、stat、realtime、identity 四个 Kafka 消费服务都会暴露 `vehicle_kafka_consumer_info{service,group,topic}`。`capacity-check` 会把缺少该指标的消费服务判为 degraded，用来捕获“进程活着但没有挂上 Kafka topic”的配置错误。Identity writer 上线后 Gateway 必须设置 `JT808_REGISTRATION_GATEWAY_WRITES_ENABLED=false`，容量巡检会阻止双写 `jt808_registration`。
业务端口：
- GB32960 TCP：`0.0.0.0:32960`
- JT/T 808 TCP：`0.0.0.0:808`
- 实时 writer：`127.0.0.1:20216`
- 身份 writer：`127.0.0.1:20217`
- 实时/API：`0.0.0.0:20200`
## 协议鉴权配置
Gateway 的鉴权策略分为 `disabled`、`observe`、`enforce`。生产新增或变更账号时先使用 `observe`，确认 `vehicle_gateway_authentication_total` 中只有 `accepted` 后再切换 `enforce`。
GB32960 多平台账号使用仅 root 可读的 JSON 文件，不把密码提交到 Git 或写入 systemd unit：
```json
{
"platform-a": "current-password",
"platform-b": ["migration-old-password", "current-password"]
}
```
`gateway.env` 只配置文件地址和模式：
```bash
GB32960_PLATFORM_CREDENTIALS_FILE=/opt/lingniu-go-native/secrets/gb32960-platform-credentials.json
GB32960_AUTH_MODE=observe
JT808_AUTH_MODE=observe
JT808_REGISTER_AUTH_CODE=issued-auth-code
IDENTITY_RESOLVE_TIMEOUT_MS=50
```
Gateway 生产必须启用独立 WAL outbox，不与旧 `NATS_SPOOL_DIR` 混用：
```bash
NATS_DURABLE_OUTBOX_ENABLED=true
NATS_OUTBOX_DIR=/opt/lingniu-go-native/spool/nats-outbox-wal
NATS_OUTBOX_MAX_INFLIGHT=10000
NATS_OUTBOX_ACK_TIMEOUT_MS=3000
NATS_OUTBOX_REPLAY_BATCH_SIZE=1000
NATS_OUTBOX_REPLAY_INTERVAL_MS=1000
NATS_OUTBOX_FSYNC=true
NATS_OUTBOX_CLOSE_TIMEOUT_MS=5000
NATS_OUTBOX_WAL_SEGMENT_BYTES=16777216
NATS_OUTBOX_WAL_SEGMENT_AGE_MS=5000
NATS_OUTBOX_WAL_APPEND_QUEUE_SIZE=100000
NATS_OUTBOX_WAL_COMMIT_BATCH_SIZE=256
NATS_OUTBOX_WAL_COMMIT_INTERVAL_MS=1
```
WAL 在分组 `fsync` 成功后才向协议处理线程返回接受成功，网络发布不阻塞设备连接；PubAck 前记录始终可恢复。正常值为 `vehicle_durable_outbox_backlog_records=0`、`vehicle_durable_outbox_inflight=0`，且 `submitted` 与 `acked` 最终一致。`submit_error`、`ack_error`、`remove_error` 或 `close_timeout` 任一增长都必须排查 NATS、磁盘和进程关闭过程。WAL 的 CRC 损坏会拒绝 Gateway 启动，禁止跳过或删除文件后强行启动；应先保留目录副本再恢复。
GB32960 登录密码在鉴权完成后会从 `parsed_fields` 删除，只保留 `password_present`；协议原始帧仍可能包含凭据，因此 RAW 查询接口和 TDengine `raw_hex` 必须按敏感数据控制访问。JT808 的 `0x0102` 会先匹配配置的公共鉴权码，再按规范化手机号匹配 `jt808_registration.auth_token` 的只读内存快照，全程不查询 MySQL；指标 `source=device` 表示命中终端历史鉴权码。强制模式拒绝的鉴权帧不会覆盖该可信鉴权码和最近鉴权成功时间。JT808 的 `enforce` 不会拒绝未发注册/认证、直接上报 `0x0200` 的兼容平台；这类来源继续通过身份缺口指标治理。
## 标准发布流程
Go 原生服务发布必须使用 Linux amd64 构建产物，不要直接上传本机默认 `go build` 结果。仓库提供统一脚本：
```bash
cd go/vehicle-gateway
# 首次上线 identity-writer 前准备 systemd、env 和 Gateway 单写开关。
scripts/install-identity-writer-service.sh --host root@115.29.187.205
# 首次上线 fields-projector 前准备 systemd 和 env；该步骤不会自动关闭 bridge 的兼容字段投影。
scripts/install-fields-projector-service.sh --host root@115.29.187.205
# 只构建并校验 ELF x86-64 产物。
scripts/deploy-ecs-release.sh --build-only --release verify-$(date +%Y%m%d%H%M%S)
# 构建、上传并校验 release，但不切换 current、不重启服务。
scripts/deploy-ecs-release.sh \
--host root@115.29.187.205 \
--release staged-$(date +%Y%m%d%H%M%S) \
--stage-only
# 构建、上传新 release、切换 current、重启服务并执行 readyz/capacity-check。
scripts/deploy-ecs-release.sh \
--host root@115.29.187.205 \
--release production-$(date +%Y%m%d%H%M%S)
```
脚本会构建 `gateway`、`history-writer`、`stat-writer`、`identity-writer`、`realtime-api`、`nats-fast-writer`、`nats-kafka-bridge`、`fields-projector`、`capacity-check`、`load-sim`、`stats-backfill`、`identity-import`，并逐个校验 `file` 输出包含 `ELF 64-bit` 和 `x86-64`。正式发布按 bridge、fields projector、下游 writer、Gateway 的依赖顺序重启，确保 canonical raw 到 fields 的派生能力先于入口切换生效；发布窗口暂停 `lingniu-go-capacity-check.timer`，发布后恢复 timer、检查 9 个 systemd 服务、9 个 `/readyz` 和 `capacity-check`。`--stage-only` 只落盘 release，不改变运行态。生产发布如需把业务统计缺口纳入发布门禁，使用 `--capacity-args "-daily-mileage-diagnostics-max-actionable=50"` 这类参数，和 `lingniu-go-capacity-check.service` 的定时巡检口径保持一致。
## 五分钟排查顺序
先回答最核心的问题：数据有没有进来，有没有排队，有没有被消费，最后有没有被查询到。
@@ -56,30 +152,43 @@ Realtime API 是当前态投影，默认在没有已提交 offset 时从 latest
2. 检查 gateway 帧计数和 TCP 活跃连接。
3. 检查 NATS bridge 的 pending 和 ack-pending。
4. 检查 bridge 写 Kafka 与 NATS ack 是否同时增长。
5. 检查 history、stat、realtime 三类 Kafka consumer lag。
5. 检查 history、stat、realtime、identity 四类 Kafka consumer lag。
6. 检查各 writer 的写入、提交、更新计数。
7. 最后再查存储和业务查询 API。
```bash
for port in 20211 20212 20213 20214 20200; do
for port in 20211 20212 20213 20214 20215 20216 20217 20218 20200; do
curl -fsS "http://127.0.0.1:${port}/readyz"
echo
done
curl -fsS http://127.0.0.1:20211/metrics \
| grep -E 'vehicle_gateway_(active_connections|frames_total|publish_total)|vehicle_async_sink'
| grep -E 'vehicle_gateway_(active_connections|frames_total|last_frame|publish_total|last_publish)|vehicle_async_sink'
curl -fsS http://127.0.0.1:20214/metrics \
| grep -E 'vehicle_bridge_(nats_consumer|kafka_writes_total|nats_acks_total)'
| grep -E 'vehicle_bridge_(nats_consumer|fields_projection|kafka_writes_total|last_kafka_write|nats_acks_total|last_ack|last_message)'
curl -fsS http://127.0.0.1:20212/metrics | grep vehicle_history_kafka_lag
curl -fsS http://127.0.0.1:20212/metrics | grep vehicle_history_batch
curl -fsS http://127.0.0.1:20213/metrics | grep vehicle_stat_kafka_lag
curl -fsS http://127.0.0.1:20200/metrics | grep vehicle_realtime_kafka_lag
curl -fsS http://127.0.0.1:20213/metrics | grep vehicle_stat_samples_total
curl -fsS http://127.0.0.1:20213/metrics | grep vehicle_stat_sources_total
curl -fsS http://127.0.0.1:20213/metrics | grep vehicle_stat_last_
curl -fsS http://127.0.0.1:20216/metrics | grep vehicle_realtime_kafka_lag
curl -fsS http://127.0.0.1:20217/metrics | grep vehicle_identity_writer_kafka_lag
for port in 20212 20213 20216 20217 20200; do
curl -fsS "http://127.0.0.1:${port}/metrics" | grep vehicle_kafka_consumer_info
done
/opt/lingniu-go-native/current/capacity-check
```
`capacity-check` 会解析 histogram 并做链路耗时早期预警，默认 `-gateway-frame-p99-ms=250`、`-gateway-identity-p99-ms=100`、`-async-sink-p99-ms=100`、`-async-sink-queue-wait-recent-p99-ms=100`、`-bridge-batch-p99-ms=5000`、`-fast-writer-stage-p99-ms=100`、`-history-flush-p99-ms=1000`、`-realtime-batch-p99-ms=500`、`-realtime-store-p99-ms=100`、`-stat-write-p99-ms=100`、`-kafka-lag-max=1000`、`-histogram-min-samples=100`。其中 realtime batch 是 Kafka projector 的整批 flush 耗时，主要看吞吐是否吃紧；单车实时投影延迟仍看 fast-writer Redis stage、realtime store update、Kafka lag 和 pending。批处理内部 in-flight pending 默认允许 `-fast-writer-batch-pending-max=1000`、`-history-batch-pending-max=1000`、`-history-rows-pending-max=5000`、`-stat-batch-pending-max=1000`、`-realtime-batch-pending-max=1000`，用于避免高频流量里百条级正在 flush/ack 的批次误报；failure-closed 重试则由 `-consumer-retry-pending-max=0` 单独门禁，history/realtime/stat/identity 任一 `*_retry_pending_messages` 非零都会降级，设置为负数才关闭。bridge/fast-writer 的 unacked ratio 要达到 `-bridge-message-min-received=5000`、`-fast-writer-message-min-received=5000` 后才判断。刚重启样本不足时不判断 p99；少量 Kafka lag 可能只是高频流量里的 in-flight 消息，只有 history/stat/realtime/identity lag 总和超过阈值才降级；压测阶段可以临时调低阈值。
topic 流转检查默认只抓完全断层：`-required-consumer-topics` 要求 history/realtime 消费三类 raw topic、stat 消费三类 fields topic；`-gateway-field-min-raw=100` 检查某协议 raw publish 足够多，扣除登录、注册、鉴权等 `skipped_non_realtime` 后仍然没有 fields 直接发布或委托 bridge；`-gateway-field-max-missing-ratio=0.20` 检查实时帧缺少预计算 fields 或发布错误比例是否过高；`-gateway-bridge-min-publishes=1000` 检查某协议 raw/委托 fields 已经由 gateway 接受，但 bridge 没有写对应 Kafka topic；`-gateway-fast-writer-min-raw-publishes=1000` 检查某协议 raw 已经由 gateway 发布，但 fast-writer 没有消费同 raw subject 并成功写 Redis/ack；`-raw-fanout-min-bridge-writes=1000` 检查某 raw topic 已经由 bridge 写入 Kafka，但 history 未收到/写入或 realtime 未收到/更新；`-gateway-parse-min-frames=100` 和 `-gateway-parse-max-bad-ratio=0.05` 检查某协议入口帧解析质量是否突然恶化；`-gateway-response-min-samples=100` 和 `-gateway-response-max-error-ratio=0.05` 检查 32960 登录 ACK、808 注册/通用 ACK 等协议响应是否大量 build/write 失败，`skipped` 表示该消息类型无需响应，不计入比例；`-gateway-identity-snapshot-required=1` 和 `-gateway-identity-snapshot-max-age-seconds=180` 检查内存身份快照是否可用且持续刷新，`-gateway-identity-min-samples=100` 和 `-gateway-identity-max-unresolved-ratio=0.20` 检查某协议 VIN/车牌/phone 映射质量是否突然失效；`-history-parsed-field-min-frames=100` 和 `-history-parsed-field-max-missing-ratio=0.05` 检查 history-writer 收到的实时 raw envelope 是否缺失预计算 `parsed_fields`；`-fast-writer-redis-envelope-min-seen=100` 检查 Redis 快路径看到足够 raw envelope 后是否完全没有更新当前态，`-fast-writer-redis-envelope-max-bad-ratio=0.50` 检查缺 VIN/vehicle_key 的实时帧比例是否异常；`-fast-writer-redis-field-min-seen=1000` 和 `-fast-writer-redis-field-max-stale-ratio=0.20` 检查 Redis 快路径旧帧字段跳过比例是否异常升高；`-stat-topic-min-bridge-writes=1000` 检查某 fields topic 已经由 bridge 写入 Kafka，但 stat-writer 未配置或未收到该 topic。统计链路还会用 `-stat-sample-min-writes=100` 检查“某个 fields topic 已成功 append 但完全抽不到里程样本”的配置/字段映射故障，并用 `-stat-sample-max-actionable-skip-ratio=0.60` 检查 fields 缺失、VIN 缺失、里程缺失、非正里程、source 缺失这类可行动跳过是否占比过高；`-stat-source-min-samples=100` 和 `-stat-source-max-missing-ratio=0.60` 检查已找到里程样本但来源端点缺失或 `vehicle_data_source` tracking 没有工作；`-stat-projection-min-sample-writes=100` 检查某 fields topic 已写入 source 候选里程样本但最终 `vehicle_daily_mileage` 投影完全没有写入；`skipped_same_mileage` 只是重复总里程去重，不计入该比例。`-last-activity-stale-seconds=300` 检查已出现过的成功/received `*_last_*_unix_seconds` 是否超过 5 分钟未刷新。上述阈值设为 `0` 可临时关闭；特殊环境可用 `-required-consumer-topics=''` 关闭消费 topic 合同检查。
每日里程业务诊断也会进入 `capacity-check` 的 JSON：默认通过 `-daily-mileage-diagnostics-url=http://127.0.0.1:20200/api/stats/daily-metrics/diagnostics/reasons` 查询当天原因汇总，输出 `daily_mileage_diagnostics.vehicle_total`、`actionable_issue_total` 和 `items`。二进制默认 `-daily-mileage-diagnostics-max-actionable=-1` 表示只展示不阻断；生产定时巡检和发布门禁应显式设置一个阈值，例如当前已知 MQTT 稀疏总里程期间可先用 `50`，待问题清理后收紧到 `0`。诊断接口不可访问会降级，因为这表示 MySQL/API 观测链路不可用；临时环境可用 `-daily-mileage-diagnostics-url=''` 关闭。
## 健康基线
当前生产环境的健康特征：
@@ -87,10 +196,13 @@ curl -fsS http://127.0.0.1:20200/metrics | grep vehicle_realtime_kafka_lag
- 所有 `/readyz` 都返回 `status=ok`。
- `vehicle_bridge_nats_consumer_ack_pending` 为 `0`。
- history、stat、realtime 的 Kafka lag 为 `0` 或短时间小幅波动后归零。
- history、stat、realtime 都能看到 `vehicle_kafka_consumer_info`。
- `capacity-check` 返回 `status=ok` 且退出码为 `0`。
- gateway 的帧计数持续增长。
- bridge 的 Kafka write 和 NATS ack 计数同时增长。
- writer 的成功计数增长，同时 Kafka lag 不持续扩大。
- writer 的成功计数增长，同时 Kafka lag 不持续扩大；统计链路还应看到 `vehicle_stat_samples_total{status="written"}` 随有效里程字段流量增长。
生产默认日志级别保持 `LOG_LEVEL=info`。Gateway 的 TCP 连接打开、关闭、空闲超时属于高频事件，只保留在 debug 级别；排查单车连接问题时可以临时设置 `LOG_LEVEL=debug`，排查结束后应恢复 info，常态监控以 `vehicle_gateway_active_connections` 和 `vehicle_gateway_connection_closes_total` 为准。
定时容量巡检由 systemd timer 触发：
@@ -103,6 +215,8 @@ journalctl -u lingniu-go-capacity-check.service --since '10 minutes ago' --no-pa
GB32960 和 JT/T 808 的活跃连接数受上游平台连接方式影响，不能直接等同于车辆数。突然归零或持续异常下降才是信号。
计数器只说明服务启动后累计发生过，不代表当前仍有流量。排查断流时同时看 `*_last_*_unix_seconds`：gateway 看 `vehicle_gateway_last_frame_unix_seconds` 和 `vehicle_gateway_last_publish_unix_seconds`，bridge 看 `vehicle_bridge_last_kafka_write_unix_seconds`，fast-writer 看 `vehicle_fast_writer_last_message_unix_seconds` 和 `vehicle_fast_writer_last_stage_unix_seconds`，history/stat/realtime 分别看对应 `last_message`、`last_write/update`、`last_commit`。如果 counter 很高但 last 时间长时间不变，说明对应协议或 topic 已经停止活动。`capacity-check` JSON 会额外派生 `*_last_*_age_seconds`，现场排查优先看 age 是否持续扩大。
## 压测入口
Go 版本提供 `cmd/load-sim` 用于阶段性连接和帧写入压测。压测生产入口前必须先确认上游真实数据窗口，避免和业务流量混淆。
@@ -121,6 +235,26 @@ go run ./cmd/load-sim \
-send=false
```
带帧压测 JT808 时必须读取服务端应答，并使用隔离手机号区间和自动清理：
```bash
export MYSQL_DSN="$(sed -n 's/^MYSQL_DSN=//p' /opt/lingniu-go-native/env/base.env | tail -n 1)"
/opt/lingniu-go-native/current/load-sim \
-protocol jt808 \
-addr 127.0.0.1:808 \
-connections 1000 \
-connect-rate 500 \
-send-interval 1s \
-duration 60s \
-template 0200 \
-jt808-phone-base 139000012000 \
-drain-responses=true \
-cleanup-registration
```
输出必须满足 `connections_failed=0`、`write_errors=0`、`response_bytes>0`、`read_errors=0`。每轮使用未在最近 10 分钟压测过的新手机号区间时，结束后的 `rows_deleted` 应等于模拟连接数；重复号段可能被 identity-writer 的 location touch 节流跳过写入，因此清理数为 `0` 也不表示删除失败。`-cleanup-only` 可用于异常中断后的补清理；删除条件固定受手机号区间和 loopback 来源约束，不清理真实注册。Bridge 默认 `BRIDGE_KAFKA_WRITE_CONCURRENCY=6`，同一批次按 topic 并行写 Kafka；RAW 和 fields 均成功后才 ACK NATS 源消息。当前生产压测基线使用 `NATS_ASYNC_RAW_WORKERS=128`、`FAST_WRITER_WORKERS=16`、`FAST_WRITER_FETCH_WAIT_MS=5`。
## 告警阈值建议
| 信号 | 建议阈值 | 含义 |
@@ -130,28 +264,82 @@ go run ./cmd/load-sim \
| `vehicle_gateway_connection_closes_total{reason="read_error"}` | 连续增长 | 入口 TCP 读失败，优先查网络、客户端断连和内核连接状态。 |
| `vehicle_gateway_connection_closes_total{reason="extract_error"}` | 连续增长 | 报文边界或协议提取异常，优先抽查 raw 日志和协议 extractor。 |
| `vehicle_gateway_connection_closes_total{reason="read_timeout"}` | 突然高于历史基线 | 车辆长时间无上报或链路空闲超时，需结合在线数和上游平台状态判断。 |
| `vehicle_gateway_frames_total{status!="OK"}` | 连续 2 分钟增长 | 解析器或上游报文质量异常。 |
| `vehicle_gateway_frame_duration_ms_histogram_bucket` | p99 连续 5 分钟超过容量目标 | Gateway parse、identity resolve、publish enqueue 或响应链路变慢。 |
| `vehicle_async_sink_queue_depth{sink="nats"}` | 持续增长且不回落 | Gateway 到 NATS/Kafka 的异步 publish 队列开始积压。 |
| `vehicle_async_sink_enqueue_total{status="timeout"}` | 任意增长 | Gateway publish 队列已满或 worker 长时间阻塞，入口可能开始丢实时性。 |
| `vehicle_gateway_frames_total{status!="OK"}` | 单协议样本达到 `capacity-check -gateway-parse-min-frames` 后，异常比例超过 `capacity-check -gateway-parse-max-bad-ratio` | 解析器或上游报文质量异常。优先抽查 raw 日志、协议 extractor、最近部署的解析规则和上游转发内容。 |
| `vehicle_gateway_frame_duration_ms_histogram_bucket` | p99 超过 `capacity-check -gateway-frame-p99-ms` 或连续 5 分钟超过容量目标 | Gateway parse、identity resolve、publish enqueue 或响应链路变慢。 |
| `vehicle_gateway_response_total{status="build_error"}` / `status="write_error"` | 单协议响应尝试样本达到 `capacity-check -gateway-response-min-samples` 后，错误比例超过 `capacity-check -gateway-response-max-error-ratio` | 协议响应构造或 TCP 写回失败。优先查 32960 登录/实时 ACK、808 注册/通用 ACK、连接断开、响应编码和上游是否提前断链。 |
| `vehicle_gateway_response_e2e_recent_p99_ms` | 单协议样本达到 `capacity-check -histogram-min-samples` 后超过 `capacity-check -gateway-response-recent-p99-ms`，默认 `100ms` | 收帧、解析、内存身份映射、RAW 入队或协议写回变慢。结合 identity、async queue 和 NATS 指标定位具体阶段。 |
| `vehicle_gateway_authentication_total{mode="observe",status!="accepted"}` | 任意增长 | 当前登录/认证与配置不一致但仍放行。先核对账号覆盖、上游迁移密码和 JT808 认证码，确认无误后才能切换 `enforce`。 |
| `vehicle_gateway_authentication_total{mode="enforce",status!="accepted"}` | 任意增长 | 网关已返回协议失败并关闭该连接；检查是否为未授权来源、过期账号或配置错误。 |
| `vehicle_gateway_publish_total{kind="fields"}` | 某协议 `raw` 达到 `capacity-check -gateway-field-min-raw` 但 fields 直接发布或 `delegated` 都为 0 | Gateway 收到实时帧但没有可供 bridge 派生的 `parsed_fields`，优先查协议解析和字段映射。 |
| `vehicle_gateway_fields_total{status="skipped_non_realtime"}` | 持续增长但 fields publish 正常 | Gateway 正确跳过登录、注册、鉴权、心跳等非实时帧，不会进入统计 topic；如果实时数据也断流，再查上游是否只在发控制帧。 |
| `vehicle_gateway_fields_total{status="skipped_missing_fields"}` / `publish_error` | 单协议达到 `capacity-check -gateway-field-min-raw` 后，缺失比例超过 `capacity-check -gateway-field-max-missing-ratio` | 实时帧没有生成扁平字段，或兼容直发模式发布失败。NATS canonical raw 模式优先查协议字段映射和 `parsed_fields`。 |
| `vehicle_gateway_publish_total{kind="raw"}` / `vehicle_gateway_publish_total{kind="fields",status="delegated"}` | 某协议某 kind 达到 `capacity-check -gateway-bridge-min-publishes` 但 bridge 无对应 Kafka topic 写入 | Gateway 已接受 raw/fields 派生任务，但 bridge 没有写 Kafka。优先查 bridge projection、route、NATS subject、Kafka write 和 bridge 日志。 |
| `vehicle_gateway_publish_total{kind="raw"}` | 某协议 raw 达到 `capacity-check -gateway-fast-writer-min-raw-publishes` 但 fast-writer 无同 raw subject `ok` | Gateway 已经发布到 NATS，但 Redis 快路径未消费或未成功写入/ack。优先查 `vehicle_fast_writer_messages_total`、`vehicle_fast_writer_stage_duration_ms_histogram_bucket{stage="redis"}`、`stage="ack"`、NATS durable pending 和 Redis 健康。 |
| `vehicle_gateway_last_frame_unix_seconds` / `vehicle_gateway_last_publish_unix_seconds` | 已出现过的成功 activity 超过 `capacity-check -last-activity-stale-seconds` 未刷新 | 对应协议入口或 publish 已停止活动，先对比连接数、上游转发状态和 bridge last 指标。 |
| `vehicle_gateway_identity_total{status!="resolved"}` | 单协议样本达到 `capacity-check -gateway-identity-min-samples` 后，异常比例超过 `capacity-check -gateway-identity-max-unresolved-ratio` | VIN/车牌/phone 映射质量下降。优先查 `vehicle_identifier` 导入、`vehicle_identity_binding` 基础车牌/VIN、`jt808_registration` phone 状态和 resolver 缓存 TTL。 |
| `vehicle_gateway_identity_total{status="timeout"}` | 任意持续增长 | Gateway 身份解析超过 `IDENTITY_RESOLVE_TIMEOUT_MS`，帧会继续 partial/unresolved 下发，但 VIN/车牌映射可能滞后。 |
| `vehicle_gateway_identity_duration_ms_histogram_bucket` | p99 超过 `capacity-check -gateway-identity-p99-ms` 或接近 `IDENTITY_RESOLVE_TIMEOUT_MS` | MySQL/缓存身份解析开始拖慢入口链路，优先查 `vehicle_identifier`、`vehicle_identity_binding`、`jt808_registration` 索引和 RDS 状态。 |
| `vehicle_gateway_identity_cache_entries` | 超过 `capacity-check -gateway-identity-cache-entries-max`，默认 `300000` | Gateway 身份解析缓存超过容量预期。优先确认 `IDENTITY_LOOKUP_CACHE_MAX_ENTRIES`、`IDENTITY_LOOKUP_CACHE_TTL_SECONDS`、`IDENTITY_LOOKUP_CACHE_CLEANUP_INTERVAL_SECONDS`，并检查上游是否出现大量异常 phone、source IP 或历史回放。 |
| `vehicle_gateway_identity_snapshot_ready` / `vehicle_gateway_identity_snapshot_last_success_unix_seconds` | ready 不是 `1`，或最近成功刷新超过 `180s` | Gateway 会继续接入并保留 raw，但新 808 phone 可能暂时无法映射 VIN。检查 RDS 连通性和四张身份来源表；恢复后周期刷新会自动发布新快照，不需要重启。 |
| `vehicle_durable_outbox_backlog_records` / `vehicle_durable_outbox_inflight` | 持续非 0 或超过容量门禁 | Gateway 已持久接受但尚未收到 JetStream PubAck。短时突发可自动回落；持续增长时检查 NATS 可用性、磁盘延迟、ACK timeout 和 inflight 上限。 |
| `vehicle_durable_outbox_publish_total{status=~"submit_error|ack_error|remove_error|close_timeout"}` | 任意增长 | WAL 后的异步发布、PubAck、分段删除或关闭等待失败。记录不会因提交/ACK 失败丢失；修复依赖后由 replay 自动恢复。 |
| `vehicle_async_sink_queue_depth{sink="nats"}` | 持续增长且不回落，或超过 `capacity-check -async-sink-queue-depth-max`，默认 `10000` | Gateway 到 NATS/Kafka 的异步 publish 队列开始积压。结合 `vehicle_async_sink_queue_capacity` 判断是否接近满队列；NATS 入队等待由 `NATS_ASYNC_ENQUEUE_TIMEOUT_MS` 控制，Kafka fallback 由 `KAFKA_ASYNC_ENQUEUE_TIMEOUT_MS` 控制。 |
| `vehicle_async_sink_queue_wait_recent_p99_ms` | 样本达到 `capacity-check -histogram-min-samples` 后超过 `-async-sink-queue-wait-recent-p99-ms`，默认 `100ms` | 秒级突发曾造成排队，即使当前 queue depth 已回到 0 也会被最近 512 样本窗口捕获。结合 `vehicle_async_sink_workers`、publish duration 和 NATS pending 判断是 worker 不足还是持久化确认变慢。 |
| `vehicle_async_sink_enqueue_total{status="timeout"}` | 任意增长 | Gateway publish 队列已满或 worker 长时间阻塞，入口会快速失败并记录 publish error，避免连接处理 goroutine 长时间堆积。优先查 NATS/Kafka 写入延迟、async worker 数、队列容量和 bridge/fast-writer pending。 |
| `vehicle_async_sink_publish_total{status="error"}` | 连续增长 | NATS/Kafka publish 失败，需要先查中间件连接和日志。 |
| `vehicle_async_sink_publish_duration_ms_histogram_bucket` | p99 连续 5 分钟上升 | Gateway async worker 写 NATS/Kafka 变慢，通常会带动 queue depth 增长。 |
| `vehicle_async_sink_publish_duration_ms_histogram_bucket` | p99 超过 `capacity-check -async-sink-p99-ms` 或连续 5 分钟上升 | Gateway async worker 写 NATS/Kafka 变慢，通常会带动 queue depth 增长。 |
| `vehicle_history_batch_flush_total{status="error"}` | 任意增长 | TDengine 批写失败；Kafka offset 不会提交，应先查 TDengine 和 SQL 错误。 |
| `vehicle_history_batch_pending_messages` | 持续非 0 或 burst 后不回落 | history-writer 已拉取但未完成写入/提交，可能卡在 TDengine 或 Kafka commit。 |
| `vehicle_history_batch_pending_rows` | 持续非 0 或 burst 后不回落 | TDengine 有批量写入积压，通常早于 Kafka lag 放大。 |
| `vehicle_history_batch_flush_duration_ms{status="ok"}` | 持续上升 | TDengine 写入延迟增加，可能需要降低 batch size 或扩容 TDengine。 |
| `vehicle_history_parsed_fields_total{status="missing"}` | 单 topic/protocol 实时帧样本达到 `capacity-check -history-parsed-field-min-frames` 后，缺失比例超过 `capacity-check -history-parsed-field-max-missing-ratio` | history-writer 收到的实时 raw envelope 没有携带预计算 `parsed_fields`，TDengine RAW 证据层会缺少扁平化解析字段。优先查 gateway `BuildFieldsEnvelope` / raw envelope 构造、协议字段映射和最近部署。 |
| `vehicle_history_batch_pending_messages` | 超过 `capacity-check -history-batch-pending-max` 或 burst 后不回落 | history-writer 已拉取但未完成写入/提交，可能卡在 TDengine 或 Kafka commit。 |
| `vehicle_history_batch_pending_rows` | 超过 `capacity-check -history-rows-pending-max` 或 burst 后不回落 | TDengine 有批量写入积压，通常早于 Kafka lag 放大。 |
| `vehicle_history_batch_flush_duration_ms_histogram_bucket{status="ok"}` | p99 超过 `capacity-check -history-flush-p99-ms` 或持续上升 | TDengine 写入延迟增加，可能需要降低 batch size 或扩容 TDengine。 |
| `vehicle_history_kafka_messages_total{status="invalid_json"}` | 超过 `capacity-check -history-invalid-json-max`，默认 `0` | Kafka raw topic 中出现无法反序列化的 envelope。history-writer 会 commit 并隔离该消息，避免阻塞历史落库；必须检查 NATS bridge 写入 Kafka 的 payload、raw topic 是否混入非 envelope 数据，以及 gateway 发布合同。 |
| `vehicle_bridge_nats_consumer_ack_pending` | 连续 2 分钟 `> 0` | 消息已投递给 bridge，但 Kafka 写入后未完成 ack。 |
| `vehicle_bridge_nats_consumer_pending` | 持续增长且 `> 10000` | bridge 消费 NATS 的速度跟不上生产速度。 |
| `vehicle_bridge_batch_pending_messages` | 持续非 0 或 burst 后不回落 | bridge 已拉取 NATS 消息但尚未完成 Kafka 写入和 NATS ack。 |
| `vehicle_bridge_batch_duration_ms_histogram_bucket` | p99 连续 5 分钟上升 | Kafka 写入或 NATS ack 开始变慢，通常会先于 ack-pending 扩大。 |
| `vehicle_fast_writer_nats_consumer_ack_pending` | 连续 2 分钟 `> 0` | 消息已投递给 fast-writer，但 TDengine/Redis 写入后未完成 ack。 |
| `vehicle_bridge_batch_duration_ms_histogram_bucket` | p99 超过 `capacity-check -bridge-batch-p99-ms` 或连续 5 分钟上升 | Kafka 写入或 NATS ack 开始变慢，通常会先于 ack-pending 扩大。 |
| `vehicle_bridge_fields_projection_total` | `invalid_json`、`invalid_envelope`、`protocol_mismatch`、`skipped_missing_fields` 或 `marshal_error` 增长 | canonical raw 无法派生统计 fields。raw 仍进入历史链路，但每日里程可能缺样本；按协议检查 Gateway `parsed_fields` 和字段合同。 |
| `vehicle_bridge_messages_total{status="route_error"}` / `vehicle_bridge_nats_acks_total{status="dropped_route_error"}` | route_error 超过 `capacity-check -bridge-route-error-max`，默认 `0` | NATS stream 中出现 bridge 未配置的 subject。bridge 会 ACK 并丢弃该消息，避免 poison message 阻塞正常 subject；必须检查 `NATS_STREAM_SUBJECTS`、`NATS_FILTER` 和 raw/fields subject 到 Kafka topic 的路由配置。 |
| `vehicle_bridge_kafka_writes_total{topic="vehicle.fields.go.*"}` | 某 fields topic 达到 `capacity-check -stat-topic-min-bridge-writes` 但 stat-writer 无同 topic consumer 或 received | fields 已进 Kafka，但统计消费配置或消费循环断层。 |
| `vehicle_bridge_kafka_writes_total{topic="vehicle.raw.go.*"}` | 某 raw topic 达到 `capacity-check -raw-fanout-min-bridge-writes` 但 history/realtime 无同 topic received/write/update | raw 已进 Kafka，但历史落库或当前态投影断层。先查对应服务 `vehicle_kafka_consumer_info`，再查 `vehicle_history_kafka_messages_total` / `vehicle_history_writes_total` / `vehicle_realtime_kafka_messages_total` / `vehicle_realtime_updates_total`。 |
| `vehicle_bridge_last_message_unix_seconds` / `vehicle_bridge_last_kafka_write_unix_seconds` / `vehicle_bridge_last_ack_unix_seconds` | 已出现过的成功 activity 超过 `capacity-check -last-activity-stale-seconds` 未刷新 | NATS 到 Kafka 桥接某 subject/topic 停止流动；先查 NATS pending、Kafka 写入和 ack。 |
| `vehicle_fast_writer_nats_consumer_ack_pending` | 超过 `100` 或持续不回落 | 消息已投递给 fast-writer，但 Redis 写入后未完成 ack；短暂个位数/十几条通常只是高频流量里的 NATS ack 飞行窗口，如果显式启用 TDengine stage，也可能卡在 TDengine。 |
| `vehicle_fast_writer_nats_consumer_pending` | 持续增长且 `> 10000` | fast-writer 消费 NATS 的速度跟不上入口写入速度。 |
| `vehicle_fast_writer_batch_pending_messages` | 持续非 0 或 burst 后不回落 | fast-writer 已拉取 NATS 消息但尚未完成 TDengine/Redis 写入和 ack。 |
| `vehicle_fast_writer_batch_pending_messages` | 超过 `capacity-check -fast-writer-batch-pending-max` 或 burst 后不回落 | fast-writer 已拉取 NATS 消息但尚未完成 Redis 写入和 ack；如果显式启用 TDengine stage，也可能卡在 TDengine。 |
| `vehicle_fast_writer_batch_pending_envelopes` | 持续非 0 或 burst 后不回落 | fast-writer 当前批次已有有效 envelope 在等待落库或 ack。 |
| `vehicle_fast_writer_stage_duration_ms_histogram_bucket` | 某个 stage 的 p99 连续 5 分钟上升 | NATS 快速写链路在 TDengine、Redis 或 NATS ack 某一阶段变慢；TDengine 阶段可小步调整 `FAST_WRITER_TDENGINE_MAX_OPEN_CONNS`，Redis 阶段是整批 pipeline 写入耗时，调整后必须观察是否出现选库错误和 ack-pending 增长。 |
| `vehicle_realtime_store_update_duration_ms_histogram_bucket{store="redis"}` | p99 连续 5 分钟上升 | Redis 实时投影变慢，会直接影响 realtime consumer 追平能力。 |
| `vehicle_realtime_store_update_duration_ms_histogram_bucket{store="mysql"}` | p99 连续 5 分钟上升 | MySQL 当前态/位置投影变慢，需结合 async queue depth 和 dropped 计数判断。 |
| `vehicle_stat_write_duration_ms_histogram_bucket` | p99 连续 5 分钟上升 | MySQL 每日里程统计写入变慢，可能导致 stat Kafka lag 增长。 |
| `vehicle_fast_writer_stage_duration_ms_histogram_bucket` | 某个 stage 的 p99 超过 `capacity-check -fast-writer-stage-p99-ms` 或连续 5 分钟上升 | NATS 快速写链路在 Redis 或 NATS ack 某一阶段变慢；生产默认 `FAST_WRITER_TDENGINE_ENABLED=false`，如果临时启用 TDengine stage，再观察 `stage="tdengine"`。 |
| `vehicle_fast_writer_messages_total{status="invalid_json"}` | 超过 `capacity-check -fast-writer-invalid-json-max`，默认 `0` | NATS raw subject 中出现无法反序列化的 envelope。fast-writer 会 ACK 并隔离该消息，避免阻塞 Redis 当前态；必须检查 gateway 发布 payload、NATS stream 是否混入非 envelope 数据，以及 bridge/fast-writer 的 subject 过滤配置。 |
| `vehicle_fast_writer_redis_envelopes_total{status="skipped_non_realtime"}` | 持续增长但 `updated` 正常 | raw topic 中存在登录、注册、鉴权等非实时帧，Redis 当前态会跳过它们，不代表污染当前态；如果 `updated=0` 且只有 skip 增长，应检查 gateway 是否停止发送实时帧。 |
| `vehicle_fast_writer_redis_envelopes_total{status="skipped_missing_vin"}` / `skipped_missing_vehicle_key` | 持续增长 | 实时帧进入 Redis 快路径但身份解析不完整，当前态不会写入。优先查 identity resolver、`vehicle_identity_binding`、`jt808_registration`、MQTT VIN 映射和 GB32960 VIN 字段。 |
| `vehicle_fast_writer_redis_envelopes_total{status="skipped_missing_fields"}` | 任意持续增长 | Gateway 发布了实时 RAW，但没有附带一次计算后的 `parsed_fields`。RAW 仍写历史，Redis 在线/KV 和 MySQL snapshot/location 均跳过；检查协议 parser、字段映射和 Gateway envelope 构造，禁止在 writer 中增加重新解析兜底。 |
| `vehicle_fast_writer_redis_fields_total{status="skipped_stale"}` | 某协议持续增长且占 `status="seen"` 比例异常升高 | Redis 当前态收到大量旧事件时间字段，旧帧不会覆盖新字段，但会影响上游时序质量判断。优先按 `subject` 查上游平台补发、设备时间漂移、NATS backlog 和 gateway event-time 归一化记录。 |
| `vehicle_fast_writer_last_message_unix_seconds` / `vehicle_fast_writer_last_stage_unix_seconds` | 已出现过的成功 activity 超过 `capacity-check -last-activity-stale-seconds` 未刷新 | Redis 快路径某 raw subject 或具体 stage 停止活动。优先对比 gateway last publish、NATS pending、`vehicle_fast_writer_messages_total`、Redis 健康和 `stage="redis"` / `stage="ack"` 的最近时间。 |
| `vehicle_fast_writer_tdengine_enabled` | 生产非 `0` | fast-writer 正在写 TDengine，可能与 history-writer 双写 RAW/位置证据层；`capacity-check` 会降级，除压测或故障切换外应保持 `0`。 |
| `vehicle_kafka_consumer_info` | history/stat/realtime 缺失 | 服务进程可能启动了，但 Kafka broker、group 或 topic 配置没有生效。 |
| `vehicle_history_config{setting="workers"}` / `vehicle_history_worker_active` | 配置低于 `-history-workers-min=3` 或任一 worker 不活跃 | 检查 `HISTORY_WORKERS`、Kafka group 分区分配、进程日志和 TDengine 连接承载。 |
| `capacity-check` finding: `required kafka consumer topic missing` | history/realtime/stat 缺少生产合同 topic | 对照 `/opt/lingniu-go-native/env/*.env` 的 `KAFKA_TOPICS` 或 `KAFKA_TOPIC` 配置。history/realtime 应有三类 raw topic，stat 应有三类 fields topic。 |
| `vehicle_realtime_redis_projector_enabled` | 生产非 `0` | realtime-api 正在从 Kafka 重写 Redis，可能与 NATS fast-writer 双写当前态；`capacity-check` 会降级，生产应保持 `0`。 |
| `vehicle_realtime_kafka_messages_total{status="invalid_json"}` | 超过 `capacity-check -realtime-invalid-json-max`，默认 `0` | realtime-api 的 Kafka 当前态投影收到无法反序列化的 raw envelope。服务会 commit 并隔离该消息，避免阻塞 Redis/MySQL 当前态；必须检查 bridge 写入 Kafka 的 payload、raw topic 污染和 gateway 发布合同。 |
| `vehicle_realtime_store_update_duration_ms_histogram_bucket{store="redis"}` | 生产出现或 p99 连续 5 分钟上升 | 仅临时启用 realtime Redis projector 时使用；常规 Redis 写入延迟看 `vehicle_fast_writer_stage_duration_ms_histogram_bucket{stage="redis"}`。 |
| `vehicle_realtime_store_update_duration_ms_histogram_bucket{store="mysql"}` | p99 超过 `capacity-check -realtime-store-p99-ms` 或连续 5 分钟上升 | MySQL 当前态/位置投影变慢，需结合 async queue depth 和 dropped 计数判断。 |
| `vehicle_realtime_config{setting="workers"}` / `vehicle_realtime_worker_active` | 配置低于 `-realtime-workers-min=3` 或任一 worker 不活跃 | 检查 `REALTIME_WORKERS`、Kafka group 分区分配和 MySQL 连接/锁等待。 |
| `vehicle_identity_writer_config{setting="workers"}` / `vehicle_identity_writer_worker_active` | 配置低于 `-identity-workers-min=3` 或任一 worker 不活跃 | 检查 `IDENTITY_WRITER_WORKERS`、JT808 raw 分区分配和 MySQL 连接/锁等待。 |
| `vehicle_realtime_async_queue_total{status="dropped"}` / `status="closed"` | 任意增长 | MySQL 当前态/位置投影的异步二级队列没有接收该帧。`dropped` 表示队列满，优先查 MySQL 写入延迟和 `MYSQL_REALTIME_ASYNC_WORKERS`；`closed` 只应出现在服务退出窗口，退出时会先 drain 已排队数据再关闭 MySQL 连接。 |
| `vehicle_realtime_plate_cache_entries` | 超过 `capacity-check -realtime-plate-cache-entries-max`，默认 `200000` | Realtime API 的 VIN 到车牌缓存超过容量预期。优先确认 `PLATE_CACHE_MAX_ENTRIES`、`PLATE_CACHE_TTL_SECONDS` 和 `vehicle_identity_binding` 的 VIN 质量；若短时间快速增长，通常说明上游出现大量异常 VIN 或历史回放范围过大。 |
| `vehicle_stat_write_duration_ms_histogram_bucket` | p99 超过 `capacity-check -stat-write-p99-ms` 或连续 5 分钟上升 | MySQL 每日里程统计写入变慢，可能导致 stat Kafka lag 增长。 |
| `vehicle_stat_kafka_messages_total{status="invalid_json"}` | 超过 `capacity-check -stat-invalid-json-max`，默认 `0` | Kafka fields topic 中出现无法反序列化的 envelope。stat-writer 会 commit 并隔离该消息，避免阻塞每日里程统计；必须检查 gateway fields 发布、NATS bridge 路由以及是否误把 RAW/非 envelope 数据写入 fields topic。 |
| `vehicle_stat_batch_pending_messages` | 超过 `capacity-check -stat-batch-pending-max`，默认 `1000` | stat-writer 已从 Kafka 拉取一批 fields 消息，但还没完成 MySQL append 和 Kafka commit。短暂非 0 是批处理窗口；持续升高优先查 MySQL 写入延迟、`vehicle_stat_write_duration_ms_histogram_bucket` 和 stat Kafka lag。 |
| `vehicle_stat_cache_entries` / `vehicle_stat_cache_evictions_total` | entries 超过 `capacity-check -stat-cache-entries-max`，默认 `1000000`；evictions 持续增长 | stat-writer 的去重、source touch、投影节流、历史基线缓存达到运行时上限。优先确认 `STATS_CACHE_MAX_ENTRIES`、`STATS_CACHE_RETENTION_HOURS`、`STATS_CACHE_CLEANUP_INTERVAL_SECONDS`，并检查是否有大范围历史回放、异常 VIN/source 或 Kafka 积压恢复。 |
| `vehicle_stat_samples_total{status="found"}` | 单 topic `vehicle_stat_writes_total{status="ok"}` 达到 `capacity-check -stat-sample-min-writes` 但 `found=0` | fields topic 有消费但完全抽不到里程样本，优先检查 VIN 映射、协议字段映射和总里程字段是否进入 fields；该指标带 `protocol` 标签，可直接按 GB32960/JT808/YUTONG_MQTT 聚合。 |
| `vehicle_stat_samples_total{status="skipped_missing_fields"}` | 任意持续增长，或单 topic 可行动跳过数 / `vehicle_stat_writes_total{status="ok"}` 超过 `capacity-check -stat-sample-max-actionable-skip-ratio` | fields topic 收到了 envelope 但没有扁平化 `fields`。优先查 gateway `BuildFieldsEnvelope`、NATS/Kafka bridge topic 映射，以及是否误把 RAW envelope 写进 fields topic。 |
| `vehicle_stat_samples_total{status="skipped_missing_vin"}` / `skipped_missing_mileage` / `skipped_non_positive_mileage` / `skipped_missing_source` | 单 topic 可行动跳过数 / `vehicle_stat_writes_total{status="ok"}` 超过 `capacity-check -stat-sample-max-actionable-skip-ratio` | fields topic 有消费但大量样本无法进入每日里程。按协议和跳过原因分别检查 VIN 映射、总里程字段映射、里程单位/异常值、`vehicle_data_source` 关联。 |
| `vehicle_stat_samples_total{status="event_time_future_adjusted"}` | 连续增长或集中在某协议/source | 设备事件时间比接收时间未来超过 10 分钟，统计已回退到接收时间。优先检查上游平台或终端时钟；RAW 帧仍保留设备原始时间，业务 location、Redis 当前态和每日里程使用归一后的时间。 |
| `vehicle_stat_samples_total{status="written"}` | stat-writer `ok` 增长但 `written` 不增长 | fields topic 有消费但没有有效里程样本入库，按 `skipped_missing_vin`、`skipped_missing_mileage`、`skipped_non_positive_mileage`、`skipped_missing_source`、`skipped_same_mileage` 定位原因；如果主要是 `skipped_same_mileage`，通常表示上游总里程未变化，不等同故障。 |
| `vehicle_stat_sources_total{status="written"}` / `skipped_throttled` | `vehicle_stat_samples_total{status="found"}` 增长但 source tracking 无增长 | 来源管理链路没有维护 `vehicle_data_source`。优先查 source endpoint 是否为空、`vehicle_data_source` 表结构、MySQL 写入错误和 `STATS_SOURCE_TOUCH_INTERVAL_SECONDS`。 |
| `vehicle_stat_sources_total{status="skipped_missing_endpoint"}` | 单 topic 缺 endpoint / found 超过 `capacity-check -stat-source-max-missing-ratio` | fields envelope 缺 `source_endpoint`，无法区分平台来源和维护 source。优先查 Gateway 是否填充 TCP remote/MQTT endpoint，以及 NATS/Kafka bridge 是否保留 envelope 字段。 |
| `vehicle_stat_projections_total{status="written"}` / `skipped_throttled` | source 候选持续写入但最终日里程更新频率低 | `vehicle_daily_mileage_source` 是每样本候选事实层，`vehicle_daily_mileage` 是最终投影层。`skipped_throttled` 增长通常表示受 `STATS_PROJECT_INTERVAL_SECONDS` 节流，不代表样本丢失；如需强一致核验可临时设投影间隔为 `0`。 |
| `vehicle_history_last_*_unix_seconds` / `vehicle_stat_last_*_unix_seconds` / `vehicle_realtime_last_*_unix_seconds` | 已出现过的成功 activity 超过 `capacity-check -last-activity-stale-seconds` 未刷新 | 下游消费、写入或 commit 某 topic 停止活动；结合 Kafka lag 和对应存储健康判断。 |
| `vehicle_stat_project_interval_seconds` | 生产异常为 `0` 或被调得过小 | `vehicle_daily_mileage_source` 仍每样本更新，但 `vehicle_daily_mileage` 选举投影会被节流；过小会放大 MySQL 压力。 |
| Kafka lag | 连续 5 分钟增长或 `> 10000` | 下游 consumer 或存储存在瓶颈。 |
| Writer 成功计数 | 入口增长但 writer 不增长 | bridge、Kafka、consumer 或存储链路断开。 |
@@ -162,7 +350,7 @@ go run ./cmd/load-sim \
1. 先确认监听端口。
```bash
ss -lntp | grep -E ':(808|32960|20200|20211|20212|20213|20214) '
ss -lntp | grep -E ':(808|32960|20200|20211|20212|20213|20214|20215|20216) '
```
2. 检查 gateway readiness 和日志。
@@ -218,21 +406,129 @@ df -h /
当前建议值：`NATS_STREAM_MAX_BYTES=21474836480`，即 `20GiB`；`NATS_STREAM_ENSURE_TIMEOUT_SECONDS=60`，避免大 stream 元数据更新时被 NATS 客户端默认 5s 超时误杀。Kafka topic 只作为短期缓冲，当前建议保留 `6h`，不要把 Kafka 或 NATS 当长期历史存储；长期历史和 RAW 查询以 TDengine/MySQL 投影为准。
fast-writer 的 `FAST_WRITER_OPERATION_TIMEOUT_MS` 建议为 `1000`。实时链路仍以 100ms 级为目标，但 TDengine 批量写存在尾延迟，过小的超时会造成 NATS 消息反复重投和重复写压力。
fast-writer 的 `FAST_WRITER_OPERATION_TIMEOUT_MS` 建议为 `1000`。生产默认设置 `FAST_WRITER_TDENGINE_ENABLED=false`，让 fast-writer 只写 Redis 实时投影，TDengine RAW/位置由 history-writer 通过 Kafka raw topic 单路写入；如果临时启用 TDengine stage，过小的超时会造成 NATS 消息反复重投和重复写压力。
生产 Redis 当前态只由 `nats-fast-writer` 写入；`realtime-writer` 应设置 `REALTIME_ROLE=writer` 和 `REALTIME_REDIS_PROJECTOR_ENABLED=false`，继续从 Kafka raw 写 MySQL snapshot/location；`realtime-api` 应设置 `REALTIME_ROLE=api`，只提供 Redis/MySQL/TDengine 查询 API。若 `capacity-check` 提示 `realtime redis projector enabled`，应先检查 `/opt/lingniu-go-native/env/realtime-writer.env`。
stat-writer 默认 `STATS_WORKERS=3`、`STATS_PROJECT_INTERVAL_SECONDS=15`。`vehicle_daily_mileage_source` 每条有效里程样本都会更新，用于保留多源事实和最新总里程；`vehicle_daily_mileage` 是对外查询结果层，会按来源优先级/质量从 source 表投影，投影被节流以降低高频帧下的 MySQL 写放大。需要故障回放或强一致核验时可临时设为 `0` 恢复每样本投影，处理完成后应调回正常值。
`vehicle_data_source.latest_seen_at` 表示平台来源最后一次被本系统接收到的时间，stat-writer 优先使用 `received_at_ms` 更新，且 SQL 层禁止该字段被补发、乱序或设备时间异常的帧回退。里程统计日期仍按设备事件时间计算，只有事件时间超出接收时间 `10min` 以上时才回退到接收时间。
`vehicle_realtime_snapshot.access_*` 是车辆/协议级接入证据，由 realtime writer 在快照同一条原子 upsert 中维护。`access_latest_received_at` 只按更大的 `received_at_ms` 且不同 event ID 推进，同时把旧值写入 `access_previous_received_at`，计算 `access_report_interval_ms` 并增加 `access_sample_count`；设备事件时间乱序不影响此路径。迁移 `008` 将存量行标为 `snapshot_backfill`，该时间只是上线基线，不得对外描述为历史首次接入。验证上线时至少观察同一 VIN 的样本数递增、间隔非负、首次时间不变，并确认重复/补发帧没有回退最新接收时间。
### Raw 有数据但实时查不到
1. 查 realtime Kafka lag。
1. 查 realtime writer Kafka lag。
2. 查 realtime update 计数。
3. 查 realtime API readiness。
4. 查最新 snapshot/location API。
```bash
curl -fsS http://127.0.0.1:20216/readyz
curl -fsS http://127.0.0.1:20200/readyz
curl -fsS 'http://127.0.0.1:20200/api/realtime/locations?limit=1'
curl -fsS 'http://127.0.0.1:20200/api/realtime/snapshots?limit=1'
```
### 更新 JT808 平台映射
多平台 808 车牌/手机号映射统一进入 `vehicle_identifier`，不要再直接把不同平台的手机号硬塞进 `vehicle_identity_binding` 的唯一列。导入器会先读取映射文件，再用旧 `vehicle_identity_binding` 的车牌/手机号反查 VIN，只写入可确定 VIN 的记录；冲突和未匹配记录留在 JSON 报告里人工处理。
```bash
release=$(basename "$(readlink -f /opt/lingniu-go-native/current)")
input=/opt/lingniu-go-native/imports/${release}/jt808_mapping
# 先 dry-run，确认 scan.sources、resolved / unresolved / conflicts。
/opt/lingniu-go-native/current/identity-import \
-input "${input}" \
-ensure-schema \
-unresolved-out "/tmp/${release}-identity-unresolved.csv" \
-conflicts-out "/tmp/${release}-identity-conflicts.csv" \
-timeout 5m > /tmp/${release}-identity-dryrun.json
jq '.scan.sources' /tmp/${release}-identity-dryrun.json
jq '.scan.unsupported_items // []' /tmp/${release}-identity-dryrun.json
jq '.source_results' /tmp/${release}-identity-dryrun.json
# dry-run 无冲突后再 apply。
/opt/lingniu-go-native/current/identity-import \
-input "${input}" \
-apply \
-sync-data-sources \
-timeout 5m > /tmp/${release}-identity-apply.json
# 只同步来源表时可以不传 input；不带 -apply 只输出候选/跳过数量，不写库。
/opt/lingniu-go-native/current/identity-import \
-sync-data-sources \
-timeout 1m > /tmp/${release}-source-sync-dryrun.json
/opt/lingniu-go-native/current/identity-import \
-sync-data-sources \
-apply \
-timeout 1m > /tmp/${release}-source-sync-apply.json
```
导入后检查：
- `scan.sources` 是否覆盖全部平台目录；重点看每个 `source_code` 的 `phone_records`、`plate_records`、`skipped`，如果某个平台记录数异常低，先处理源文件表头/列位置，不要直接 apply。
- `scan.unsupported_items` 必须为空；如果出现 `.xls` 或 `.xlsb`，先转成 `.xlsx` 再导入，避免某个平台文件被跳过后造成 VIN 映射缺口。`identity-import -apply` 会在连接 MySQL/建表前拒绝带 unsupported 文件的目录。
- `source_results` 是否逐平台呈现合理的 `resolved/unresolved/conflicts`；如果某个平台 `unresolved` 很高，先补 `vehicle_identity_binding` 的 VIN/车牌基础事实；如果 `conflicts` 不为 `0`，先人工确认同一平台下手机号/车牌是否重复指向不同车辆。
- 同一平台同一手机号/车牌重复出现但不冲突时，导入器会合并非空字段，优先保留可用于 VIN 解析的车牌；如果同一标识对应多个不同车牌，会进入 conflict，不会静默覆盖。
- `identity-import -apply` 会跳过 unresolved/conflict 项，只写入可确定 VIN 且无冲突的标识；实际写入包在单个 MySQL 事务里，任一写入错误都会回滚本次导入。
- `vehicle_identifier` 总数和各 `source_code` 分布是否符合文件规模。
- `vehicle_data_source.source_code/platform_name/source_kind` 会从 `jt808_registration.phone/source_ip` 与 `vehicle_identifier` 推断；同一来源 IP 推断出多个平台时会跳过，且不会覆盖人工维护的平台名。
- 来源维护 API 支持 `sourceCodeMissing=true` 快速列出未绑定平台编码的来源，例如：`/api/stats/data-sources?protocol=JT808&sourceCodeMissing=true&includeTotal=true`。
- 来源类型使用 `source_kind` 维护：`PLATFORM` 表示稳定平台源，`DIRECT` 表示车辆直连或动态 IP，`UNKNOWN` 表示未分类。每日里程最终投影在同等质量下按 `PLATFORM -> UNKNOWN -> DIRECT` 选择来源，然后再比较 `trust_priority`、样本数和最新时间。
- 每日里程 API 会通过 `source_id` 关联 `vehicle_data_source` 返回选中来源证据，例如：`/api/stats/daily-metrics?protocol=JT808&dateFrom=2026-07-12&dateTo=2026-07-12&limit=1`。排查异常里程时同时核对 `source_id`、`source_ip`、`platform_name`、`source_code`、`source_kind` 和 `latest_total_mileage_km`，不要只看 `daily_mileage_km`。
- 每日里程候选来源 API 用于审计多源选举，例如：`/api/stats/daily-metrics/sources?protocol=JT808&dateFrom=2026-07-12&dateTo=2026-07-12&selected=true&limit=1`。它返回 `source_key`、`phone`、`sample_count`、`first_total_mileage_km`、`latest_total_mileage_km`、`quality_status`、`quality_reason`、`is_selected`，用于确认每个来源独立算出的里程以及最终是否被选中。`first_event_time` 可能是跨日前推得到的历史基线时间，排查时应结合 `quality_reason` 理解。
- 每日里程诊断 API 用于排查“实时在线但统计缺失”，例如：`/api/stats/daily-metrics/diagnostics?protocol=YUTONG_MQTT&date=2026-07-12&diagnosis=NO_SOURCE_SAMPLE&includeTotal=true`。它会从 `vehicle_realtime_snapshot/location` 找当天活跃车辆，再关联最终表和候选来源表输出 `diagnosis`：`OK` 表示最终日里程已存在；`MISSING_DAILY` 表示候选来源已有样本但最终投影缺失；`NO_SOURCE_SAMPLE` 表示当天确实收到总里程字段但 stat-writer 没抽到候选样本；`NO_TOTAL_MILEAGE` 表示当天实时活跃但没有收到总里程字段。宇通 MQTT 可能稀疏上报，`vehicle_realtime_location.total_mileage_km` 会保留旧值，判断当天是否真实上报要看 `total_mileage_event_time`。
- `NO_TOTAL_MILEAGE` 的 `reason` 会进一步说明源头问题：`realtime_total_mileage_missing` 表示当前态没有总里程；`realtime_total_mileage_non_positive` 表示源头总里程为 0 或负数；`realtime_total_mileage_time_missing` 表示当前态有总里程旧值但缺少该字段的上报时间证据；`realtime_total_mileage_not_reported_on_stat_date` 表示当前态保留了旧总里程，但指定业务日期没有重新上报总里程字段。
- `vehicle_realtime_snapshot.parsed_json` 是稀疏帧合并后的当前态，字段存在只证明历史上曾上报，不能证明当前帧或统计日上报。字段诊断返回 `mapped_protocol_field_without_fresh_evidence` 时应按 `raw_frame_query_path` 核对原始帧；只有 `vehicle_realtime_location.total_mileage_event_time` 落在统计日内，才把标准总里程视为新鲜证据。
- 每日里程诊断汇总 API 用于大屏和告警入口，例如：`/api/stats/daily-metrics/diagnostics/summary?date=2026-07-12`。它按协议返回 `active_count`、`ok_count`、`missing_daily_count`、`no_source_sample_count`、`no_total_mileage_count` 和 `actionable_issue_count`；健康状态下应优先关注 `actionable_issue_count=0`，异常时再下钻到诊断明细接口。
- 每日里程诊断原因汇总 API 用于告警聚合和运营看板，例如：`/api/stats/daily-metrics/diagnostics/reasons?date=2026-07-12`。它按 `protocol + diagnosis + reason` 返回车辆数；需要只看某类问题时可传 `diagnosis=NO_TOTAL_MILEAGE`，比拉取逐车明细更适合高频轮询。
- `stats-backfill` 用于从 TDengine RAW 证据层补算 MySQL 每日里程。ECS 上直接执行会默认加载 `/opt/lingniu-go-native/env/base.env` 和 `/opt/lingniu-go-native/env/stat-writer.env`；默认 `BACKFILL_DRY_RUN=true`，不会写库。补算采用 `BACKFILL_METHOD=last_diff`：对每个协议、VIN、来源分别取当日最新累计总里程和同一来源在目标日前最近一次累计总里程，按 `当日最新 - 最近历史总里程` 计算。补算窗口首日会聚合目标日前的历史并取 `LAST(event_time)`；因此前一日缺失时会继续向更早日期寻找，历史完全为空时才使用当天第一条作基线。JT808 历史补算结束后会按 `source_code` 归并平台来源键，保证历史与实时统计使用同一来源。正式写入前先 dry-run，例如：
```bash
BACKFILL_PROTOCOLS=YUTONG_MQTT \
BACKFILL_DATE_FROM=2026-07-12 \
BACKFILL_DATE_TO=2026-07-12 \
/opt/lingniu-go-native/current/stats-backfill
BACKFILL_DRY_RUN=false \
BACKFILL_PROTOCOLS=GB32960,JT808,YUTONG_MQTT \
BACKFILL_DATE_FROM=2026-07-12 \
BACKFILL_DATE_TO=2026-07-12 \
/opt/lingniu-go-native/current/stats-backfill
```
生产定时补算使用 systemd timer，默认每天 01:30 补算昨天往前 3 天，覆盖上游晚到或断传后恢复的总里程字段：
```bash
cd go/vehicle-gateway
scripts/install-stats-backfill-timer.sh \
--host root@115.29.187.205 \
--days-back 1 \
--window-days 3 \
--on-calendar '*-*-* 01:30:00'
systemctl list-timers lingniu-go-stats-backfill.timer --no-pager
systemctl cat lingniu-go-stats-backfill.service lingniu-go-stats-backfill.timer --no-pager
```
如果只想验证定时任务配置不写库，可以加 `--dry-run` 安装；正式生产不加 `--dry-run`，服务内会设置 `BACKFILL_DRY_RUN=false`。timer 设置 `Persistent=false`，避免首次安装时立刻执行当天已错过的计划；即使 ECS 短暂停机，下一次 3 天窗口也会补到遗漏日期。
- 诊断 API 判断“指定日期活跃”时会同时看 `event_time`、`received_at`、`updated_at`，任一时间落在业务日期内即纳入；这样设备时间漂移或未来时间不会遮蔽服务端当天实际收到的数据。
- 每日里程质量规则会把 1km 内的小幅负漂移按 0km 处理，`quality_reason=negative_jitter_clamped`。优先用前一自然日同来源末值；前一日缺失时继续向前查找最近历史日末值，并按“2500km × 两个累计里程端点相隔自然日数”放大物理合理性上限，但仍把完整差值记在当前统计日，不平均分摊或伪造中间日期。该上限覆盖车辆以约 100km/h 连续运行 24 小时的极端场景，同时继续拒绝约 4000km/日的异常跳变。超过窗口上限的正向突增或超过 1km 的倒退是 `INVALID_DELTA`，不会进入最终 `vehicle_daily_mileage`；历史完全为空时使用当天第一条。如果设备事件时间比接收时间未来超过 10 分钟，业务层会统一使用接收时间；TDengine RAW 帧仍保留设备原始时间作为证据。
- 来源诊断 API 支持按来源 IP 汇总注册手机号和 `vehicle_identifier` 匹配情况，例如：`/api/stats/data-sources/diagnostics?protocol=JT808&sourceCodeMissing=true&includeTotal=true`。`reason=no_identifier_match` 表示该来源下手机号还没有进入 `vehicle_identifier`；`reason=ambiguous_source_code` 表示同一来源 IP 匹配到多个平台编码，需要人工确认；`reason=candidate_available` 表示可以同步或确认候选 `source_code`。
- 来源类型建议 API 是只读辅助，例如：`/api/stats/data-sources/kind-suggestions?protocol=JT808&sourceKind=UNKNOWN&includeTotal=true`。它会输出 `suggested_source_kind`、`suggestion_confidence` 和 `suggestion_reason`，用于人工确认后再 PATCH `source_kind`；不要把建议结果无审核地批量写回。
- `source_code` 用于程序稳定识别来源，`platform_name` 用于展示和人工修正；如果两者看起来不一致，优先核对该来源 IP 下的 `jt808_registration.phone` 是否来自同一平台，再决定是否人工修正平台名。
- Gateway 默认启用 `IDENTITY_SOURCE_CODE_LOOKUP_ENABLED=true`。808 VIN 解析会先用 `source_endpoint -> vehicle_data_source.source_code` 做 `vehicle_identifier` 精确匹配，精确未命中时再回退到全局 `vehicle_identifier`、旧 `vehicle_identity_binding` 和 `jt808_registration`，因此多平台同一手机号/车牌映射不同 VIN 时，应优先维护正确的 `vehicle_data_source.source_code`。
- `jt808_registration` 中 `vin = 'unknown'` 的手机号是否仍能在 `vehicle_identifier` 中匹配；如果能匹配，等待下一帧 808 位置或注册/鉴权帧触发自动更新。
- Gateway `vehicle_gateway_identity_total{protocol="JT808",status="resolved"}` 应持续增长。
Gateway 默认启用 `IDENTITY_SNAPSHOT_ONLY_ENABLED=true`，启动时并每隔 `IDENTITY_SNAPSHOT_REFRESH_INTERVAL_SECONDS=60` 秒把 `vehicle_data_source`、`vehicle_identifier`、旧 `vehicle_identity_binding` 和 `jt808_registration` 原子载入内存。快照包含 JT808 的 VIN、设备、车牌和最近可信鉴权码；每帧身份解析及鉴权均不查询 MySQL。注册/鉴权/位置触达由独立 identity-writer 投影写入 `jt808_registration`，人工改库或新鉴权码最多等待一个刷新周期即可生效。快照刷新失败保留上一版，启动时 RDS 不可用也不阻止接入；未知 phone 会暂时 unresolved，RDS 恢复并完成刷新后自动解析。
## 重启顺序
优先重启最小故障层：

231
docs/vehicle-data-platform-analysis.md Normal file
View File

@@ -0,0 +1,231 @@
# 车辆数据中台一期现状分析与总体方案
> 结论日期：2026-07-14
> 范围：只完成现状分析和方案设计，不进入 V2 功能开发或 ECS 部署。
> 代码依据：`/Users/lingniu/project/ai-coding/ln-bi`、`vehicle-data-platform`、`go/vehicle-gateway`，以及 `docs/architecture`、`docs/ops` 中的生产说明。
## 1. 结论摘要
当前数据接入链路已经具备可用的生产底座：GB32960、JT808、YUTONG_MQTT 经 NATS/Kafka 分流后，分别形成 Redis 当前态、TDengine RAW/位置历史、MySQL 身份/实时/日里程投影；现有平台 BFF 也已提供车辆、实时位置、轨迹点、RAW、里程、质量和运维健康等只读接口。
但现有前端不是可直接扩展的一期产品：生产入口加载的是约 6000 行的 `PrototypeApp.tsx`，仍混有 mock 数据和“接口缺失时展示能力边界”的原型逻辑；另一个模块化 `App.tsx` 并非当前入口。现有地图每次最多创建 500 个普通 Marker，没有聚合、MassMarks/Canvas 图层和视口查询，无法满足 1 万辆级监控。所谓告警接口实际上是质量问题接口的别名，没有规则、事件、处理记录和通知持久化。
建议丢弃现有原型页面的产品实现，但保留经验证的数据适配、领域工具、测试、运行时配置和部署脚手架，建立独立 V2。前端沿用 React + TypeScript + Vite，吸收 `ln-bi` 的视觉语言、壳层、鉴权思想和基础组件，但采用正式路由、查询缓存、虚拟表格和独立地图 SDK 层。后端保留现有接入链路，通过平台 BFF 和新增的告警/导出 worker 补齐业务能力，不让浏览器直接访问 Redis、TDengine 或 MySQL。
## 2. 当前前端技术栈与目录
### 2.1 `ln-bi` 参考项目
| 项目 | 当前实现 | 评价 |
| --- | --- | --- |
| 框架 | React 19、React DOM 19、TypeScript 5.8、Vite 6.2 | 可作为 V2 目标栈参考；不建议仅为对齐版本立刻升级已有平台依赖 |
| 样式 | Tailwind CSS 4、少量 CSS 变量 | 视觉语言清晰，但业务组件 class 较分散，需要提炼 token 和组件配方 |
| 图标/动效 | lucide-react、Motion | 可复用设计语言；监控页面动效应克制并支持 reduced motion |
| 图表 | Recharts 3.8 | 适合常规 BI；百万点、多 Y 轴时序更推荐现平台已有 ECharts 5 |
| 地图 | `@amap/amap-jsapi-loader` + AMap JS API 2.0 | 加载和实例生命周期可借鉴；目前只是热力图业务组件，不是通用地图 SDK |
| API | 原生 `fetch` + JWT 注入 | 可借鉴鉴权流程；错误类型、超时、重试、取消、缓存仍不足 |
| 服务端 | Hono、MySQL/Postgres、JWT、XLSX | 属于 `ln-bi` 自身 BFF，不应直接搬入车辆接入 Go 数据面 |
| 路由 | pathname + hash + `replaceState` 手写 | 适合少量模块，不适合可分享查询、详情层级和权限路由 |
| 状态 | React 本地状态/Context | 未使用全局状态库或服务端查询缓存 |
主要目录：
```text
ln-bi/src/
├── App.tsx # 模块注册、权限门禁、懒加载
├── auth/ # jumpToken -> JWT、fetch token 注入
├── components/
│ ├── Shell.tsx # 桌面侧栏、移动底栏、模块切换、水印
│ ├── SearchSelect.tsx
│ ├── MultiSearchSelect.tsx
│ └── ui/surface.tsx # PageFrame、SurfaceCard、MetricTile、状态组件
├── modules/
│ ├── vehicle-heatmap/ # AMap 热力图和筛选/详情面板
│ ├── hydrogen-heatmap/
│ ├── mileage/ # 表格、详情弹窗、XLSX 导出
│ └── ...
├── server/ # Hono BFF、认证、DB 查询
├── shared/auth/roles.ts # 角色判断
└── index.css # Tailwind、主题变量、地图控件样式
```
### 2.2 当前车辆平台前端
目录为 `vehicle-data-platform/apps/web`，使用 React 18.3、TypeScript 5.7、Vite 6、Semi UI 2.71、ECharts 5.6。当前入口 `src/main.tsx` 加载 `PrototypeApp`，不是 `src/App.tsx`。
```text
vehicle-data-platform/apps/web/src/
├── main.tsx # 当前生产入口 -> PrototypeApp
├── PrototypeApp.tsx # 大型原型单体，真实接口与 mock 逻辑混合
├── App.tsx # 未作为入口的模块化旧实现
├── api/ # 统一响应信封、类型和调用方法
├── components/ # VehicleMap、状态标签、空状态等
├── config/ # API/高德运行时配置
├── domain/ # 路由、导出、车辆查询等纯函数
├── integrations/amap.ts # AMap loader、坐标校验、逆地理编码
├── layout/AppShell.tsx
├── pages/ # 旧模块化页面
├── prototype/ # mock、真实数据适配、view model、字段映射
└── styles/ # token、全局样式、原型样式
```
V2 的原则是“页面重建、能力甄别复用”：不复制 `PrototypeApp.tsx` 和 `prototype.css`；保留可独立测试的 `api`、`domain`、`integrations`、真实数据归一化逻辑及其测试，再按新接口契约重构。
## 3. 可复用能力清单
| 能力 | 来源 | 复用决策 | 说明 |
| --- | --- | --- | --- |
| 80px 深色侧栏、顶部面包屑、水印、移动底栏 | `ln-bi/components/Shell.tsx` | 设计复用、代码重构 | V2 桌面优先，应改为正式路由和可折叠侧栏 |
| PageFrame、SurfaceCard、MetricTile、SegmentedNav | `ln-bi/components/ui/surface.tsx` | 可迁移后组件化 | 统一 token、尺寸和可访问性后复用 |
| SearchSelect、MultiSearchSelect | `ln-bi/components` | 交互参考 | 当前仅接收字符串数组，缺少远程搜索、虚拟列表、label/value 和键盘完整性 |
| 加载、空、错误状态 | 两项目 | 合并重构 | 建立统一 `AsyncState`，禁止页面各自拼接 |
| JWT/jumpToken 门禁、角色判断 | `ln-bi/auth`、`shared/auth` | 复用认证协议思想 | token 仍应保存在 session；权限改为路由/操作级声明，后端必须二次校验 |
| AMap loader、实例销毁、运行时密钥 | 两项目 | 合并为通用 SDK | 统一 loader promise、插件注册、安全代理和错误状态 |
| 热力图数据归一化 | `ln-bi/vehicle-heatmap` | 可复用算法 | `log1p/sqrt` 强度变换适合密度层，不等于车辆状态点图层 |
| 表格/详情弹窗/XLSX | `ln-bi/mileage` | 交互与导出格式参考 | 一期大数据导出必须后端异步，不能沿用浏览器全量 XLSX |
| ECharts | 当前车辆平台 | 保留 | 更适合多 Y 轴、dataZoom、断点、阶梯线和大数据采样 |
| `api/client.ts` 响应信封和 traceId | 当前车辆平台 | 扩展复用 | 增加鉴权、AbortSignal、超时、错误码、幂等键和查询缓存 |
| 车辆 lookup、字段归一化、CSV 纯函数及测试 | 当前车辆平台 | 审核后复用 | 去掉 mock fallback，统一到 V2 DTO/领域模型 |
| Semi UI 页面组件 | 当前车辆平台 | 暂不作为 V2 默认 | 与 `ln-bi` Tailwind 视觉体系并存会形成双设计系统；V2 启动前只选一套基础组件策略 |
不存在可直接复用的“通用图表组件”或“通用高性能表格组件”。`ln-bi` 的 Recharts 与页面耦合，当前平台的 ECharts 也需要建立 `TimeSeriesChart`、轴/单位/空值策略和采样契约；表格需新增服务端分页、列配置、固定列和虚拟滚动封装。
## 4. 高德地图当前封装方式
### 4.1 `ln-bi`
`AmapHeatmapCanvas.tsx` 动态导入 `@amap/amap-jsapi-loader`，写入 `_AMapSecurityConfig.securityJsCode`，加载 AMap 2.0 的 HeatMap、ToolBar、Scale 插件；创建白色地图和 HeatMap，点击地图回传经纬度，数据变化时调用 `setDataSet`，焦点变化时计算 bounds，卸载时销毁实例。
优点是 loader 官方、生命周期完整、地图与侧面板分屏清楚。限制是车辆和加氢模块各有近似实现，密钥直接下发前端，未封装 Marker/聚合/信息窗/轨迹/播放，也没有实例共享和视口事件节流。
### 4.2 当前车辆平台
`integrations/amap.ts` 自行加载 `https://webapi.amap.com/loader.js`，按插件集合缓存 Promise，支持 Scale、Geocoder、坐标合法性检查、URI Marker 链接和浏览器逆地理编码；`appConfig.ts` 从 `window.__LINGNIU_APP_CONFIG__` 或 Vite 环境变量读取 Web JS key、安全代码/代理和 API base URL。
`VehicleMap.tsx` 在组件内创建 Map、Marker、Polyline，重绘时清除 overlay，最多截取 500 个点，并在未配置地图时提供坐标预览。它适合验证和小规模页面，不适合一期：每点 HTML button、没有聚合、状态图层、InfoWindow 管理、车辆朝向、轨迹抽稀/播放或视口查询。
### 4.3 V2 建议封装
建立 `features/map-sdk`，分为：
- `AMapProvider/useAMap`：唯一 loader、插件按需加载、实例注册、resize/destroy。
- `BaseMap`：纯地图容器，只接收中心、缩放、样式和事件。
- `VehiclePointLayer`：小规模用 Marker，规模化优先 MarkerCluster/LabelsLayer/MassMarks 或 Canvas/WebGL 能力；详情卡只保留一个 InfoWindow。
- `TrackLayer`：Polyline、起终点、停车/告警节点和移动标记；播放状态在业务 hook，不绑入地图实例。
- `HeatmapLayer`、`GeocoderService`、`MapControls`：独立能力。
- 视口查询使用 `bounds + zoom + filterHash`；移动结束后防抖请求服务器聚合结果，前端不一次拉取全量 1 万点 DOM。
高德 Web JS key 可由运行时配置下发，但安全密钥优先使用服务端安全代理；逆地理编码优先走现有 `/api/map/reverse-geocode`，避免泄露 REST key并便于限流/缓存。
## 5. 当前车辆相关接口
现有平台 BFF 的有效能力如下；详细缺口见 `vehicle-data-platform-api-gap.md`。
| 领域 | 接口 | 当前用途 |
| --- | --- | --- |
| 总览 | `GET /api/dashboard/summary` | 在线、今日活跃、帧数、问题数、协议/服务/链路统计 |
| 车辆 | `GET /api/vehicles`、`/api/vehicles/resolve`、`/api/vehicles/coverage`、`/summary` | 车辆列表、VIN/车牌/手机号解析、来源覆盖 |
| 单车 | `GET /api/vehicle-service`、`/summary`、`/overview`；`POST /overviews` | 聚合身份、实时、历史、RAW、里程、质量证据 |
| 实时 | `GET /api/realtime/vehicles`、`/api/realtime/locations` | VIN 聚合实时状态和每协议最新位置 |
| 历史 | `GET /api/history/locations`、`GET/POST /api/history/raw-frames` | TDengine 位置点和 RAW 证据分页 |
| 里程 | `GET /api/mileage/summary`、`/api/mileage/daily` | MySQL 日里程统计 |
| 在线 | `GET /api/statistics/online-summary`、`/online-vehicles` | 当前固定口径的在线统计和车辆状态 |
| 数据质量 | `GET /api/quality/summary`、`/issues`、`/notification-plan` | 由当前数据推导的质量信号和静态通知方案 |
| 告警兼容 | `GET /api/alert-events/summary`、`/alert-events`、`/notification-plan` | 实际是上述质量接口别名，不是业务告警事件 |
| 地图 | `GET /api/map/reverse-geocode` | 服务端高德逆地理编码 |
| 运维 | `GET /api/ops/health`、`/source-readiness` | 数据源可写性、Kafka lag、连接/Redis key、发布版本和接入准备度 |
同时，Go `realtime-api` 还提供面向底层表的查询接口和 OpenAPI，但 V2 浏览器应统一访问平台 BFF，避免前端绑定多个端口与底层存储模型。
## 6. 数据职责概览
- TDengine `lingniu_vehicle_ts`：`raw_frames` 是原始接收与扁平解析证据，`raw_frame_payload_chunks` 保存超长 payload，`vehicle_locations` 保存 VIN 级位置、速度、方向、告警标志和总里程历史。它不是车辆档案或告警业务库。
- Redis DB 50：`vehicle:latest:*`、`vehicle:realtime-raw:*`、`vehicle:rt-kv:*`、`vehicle:online:*`、`vehicle:protocols:*`、`vehicle:last_seen` 提供可过期、可重建的当前态。它不能作为历史、告警或导出任务事实源。
- MySQL `lingniu_vehicle_data`：保存 `vehicle`、多标识映射、JT808 注册鉴权、每协议实时快照/位置、来源配置和日里程事实/结果。现有 `vehicle` 档案很轻，只含 VIN、车牌、OEM、启用状态，尚不足以支撑车型、公司、车辆类型等产品字段。
统一模型及新增模型详见 `vehicle-data-platform-data-model.md`。
## 7. 一期缺口结论
必须新增或升级的核心能力：
1. 完整车辆档案、组织/车型/协议/接入厂家字典与首次接入事实。
2. 地图聚合查询、组合筛选、状态统计和可配置在线口径。
3. 包含方向、SOC、告警标志、停车点、异常点过滤和抽稀元数据的轨迹接口。
4. 动态指标目录，以及多车、多指标、聚合/抽稀、分页的通用时序查询。
5. MySQL 持久化告警规则、告警事件、处理记录、站内通知和告警计算 worker。
6. MySQL 异步导出任务、对象/本地文件存储、worker、进度与下载授权。
7. 接入状态字段：首次/最近上报、事件时间与接收时间延迟、最近消息类型、最近错误、动态在线阈值。
8. 真正的认证、菜单/操作权限审计；现车辆平台 BFF 尚未呈现完整权限体系。
## 8. 推荐一期路由与菜单
```text
/
├── /monitor 全局监控（默认首页）
├── /vehicles 车辆查询
│ └── /vehicles/:vin 单车数字档案
├── /tracks 轨迹查询与回放
├── /history 历史数据分析
│ └── /history/exports 导出任务
├── /alerts
│ ├── /alerts/events 告警事件
│ ├── /alerts/rules 告警规则
│ └── /alerts/inbox 站内通知
├── /access 车辆接入状态
└── /operations 运维质量（管理员）
```
侧栏一级菜单保持 6 个业务入口：全局监控、车辆中心、轨迹回放、历史分析、告警中心、接入管理；导出任务挂在历史分析二级菜单，运维质量按权限置底。详情、告警、轨迹之间用带 `returnTo` 的上下文跳转，查询条件同步 URL，浏览器返回可恢复筛选、时间和选中车辆。
## 9. 推荐前后端整体架构
```mermaid
flowchart LR
UI["V2 React Web"] --> BFF["Platform API / BFF"]
BFF --> RT["Redis 当前态"]
BFF --> TS["TDengine 时序与 RAW"]
BFF --> DB["MySQL 业务事实"]
BFF --> MAP["高德 REST / 安全代理"]
ING["Gateway + NATS + Kafka Writers"] --> RT
ING --> TS
ING --> DB
K["Kafka fields/raw"] --> AW["Alert evaluator"]
AW --> DB
BFF --> EQ["Export queue"]
EQ --> EW["Export worker"]
EW --> TS
EW --> DB
EW --> FS["文件/对象存储"]
```
前端按 `app/router`、`shared`、`entities/vehicle`、`features`、`pages` 分层。服务端状态使用查询缓存库统一取消、去重、失效和后台刷新；少量 UI 状态用 React Context/局部 store。地图、图表、表格均只接收领域 DTO，不直接处理协议字段名。
平台 BFF 负责统一鉴权、参数校验、分页、在线状态计算、跨源聚合、字段目录、限流、traceId 和 DTO；禁止让前端分别拼 Redis/MySQL/TDengine。已有接入 writer 保持不变，新告警 worker 消费可重放 fields/raw 流，异步导出 worker 读取 TDengine/MySQL 并把任务状态写 MySQL。
## 10. 运维与最终 ECS 部署边界
已查阅 `docs/ops/vehicle-ingest-runbook.md`、`docs/ops/go-vehicle-ingest-memory.md`、`docs/architecture/production-data-plane-inventory.md` 和 `vehicle-data-platform/docs/deployment.md`。当前接入数据面采用 systemd 裸机 release 目录与 `current` 软链，平台使用 `/opt/lingniu-vehicle-platform/{releases,current,env}`，HTTP 端口 20300；部署后应验证 `/api/ops/health`、车辆查询和运行时配置。
V2 最终仍部署到当前 ECS，但不应在分析阶段执行。实施完成后的发布顺序应是：数据库向前兼容迁移 -> 新 worker（默认禁用规则）-> BFF -> 静态 Web -> 小流量健康验证 -> 启用告警/导出 worker。每一步保留旧 release 和可回滚软链；密钥只进 ECS 环境文件，不进仓库或构建产物。完整发布验收纳入路线图最后阶段。
## 11. 风险与待确认事项
| 优先级 | 风险/待确认 | 影响与建议 |
| --- | --- | --- |
| P0 | `vehicle` 档案缺车型、类型、公司、接入厂家等 | 先确定主数据来源和维护责任，不要从实时 JSON 猜档案 |
| P0 | 现有告警接口名与真实语义不符 | V2 使用 `/api/v2/alerts/*` 新契约，旧别名标记 deprecated，避免误把质量信号当已处理事件 |
| P0 | 动态指标没有目录、单位、类型和协议映射 | 先落指标元数据；否则图表、规则和导出会各自硬编码 |
| P0 | 用户/角色来源尚未确认 | 明确复用 `ln-bi` jumpToken/JWT，还是接入统一 SSO；后端权限不可只靠菜单隐藏 |
| P1 | 在线阈值是全局、协议级还是车辆级 | 建议全局默认 + 协议覆盖；一期暂不做单车覆盖 |
| P1 | 轨迹异常过滤可能隐藏原始证据 | API 同时返回 raw/filtered 计数和算法版本，允许关闭过滤 |
| P1 | TDengine `raw_frames.parsed_json` 是动态字段唯一历史来源 | 通用指标查询先从 JSON 提取会有成本；高频稳定指标应按使用量逐步物化，而非一期全列化 |
| P1 | 单 ECS 同时承担接入、查询、告警和导出 | 导出/重查询必须有并发、时间范围、行数和资源配额；压测后再定 worker 并发 |
| P1 | 地图 1 万辆的插件/授权与浏览器性能 | 先完成 1k/10k 基准，确定 MarkerCluster、MassMarks 或 Canvas 方案和高德配额 |
| P1 | 站内通知实时方式 | 一期可 SSE + 轮询降级；若 ECS 反代不适合长连接，先使用增量轮询 |
| P2 | Semi UI 与 Tailwind 双设计系统 | V2 启动前确认唯一基础组件策略；建议 Tailwind token + 无样式/轻量基础组件，ECharts 保留 |
| P2 | 导出文件存储位置与保留期 | 确认 OSS 或 ECS 本地盘；推荐 OSS/兼容对象存储，任务和文件设过期清理策略 |
## 12. 本阶段完成定义
本阶段仅交付本分析、路线图、API 缺口和数据模型四份文档。V2 代码、数据库迁移、告警/导出 worker、ECS 发布均须在确认 P0 项后按路线图进入下一阶段。

153
docs/vehicle-data-platform-api-gap.md Normal file
View File

@@ -0,0 +1,153 @@
# 车辆数据中台一期 API 能力与缺口
## 1. 判定规则
- “已有”表示当前平台 BFF 已有路由和生产 Store 实现。
- “部分”表示可作为底层数据来源，但 DTO、筛选、性能或业务语义不足。
- “缺失”表示当前没有可支撑验收的持久化/API 链路。
- 新接口建议统一放 `/api/v2`；旧 `/api/alert-events*` 是质量接口别名，不得作为新告警契约继续扩展。
## 2. 现有接口清单
| 方法与路径 | 状态 | 数据源/说明 |
| --- | --- | --- |
| `GET /api/dashboard/summary` | 已有 | MySQL/运维探针聚合；口径偏现有服务健康 |
| `GET /api/vehicles` | 已有 | 车辆身份与实时快照集合，支持关键词/协议等基础过滤 |
| `GET /api/vehicles/resolve` | 已有 | 通过 VIN、车牌、手机号解析车辆 |
| `GET /api/vehicles/coverage`、`/summary` | 已有 | 多协议来源覆盖、绑定和在线来源数 |
| `GET /api/vehicle-service`、`/summary`、`/overview` | 已有 | 单车/车队证据聚合 |
| `POST /api/vehicle-service/overviews` | 已有 | 批量关键词概览 |
| `GET /api/realtime/vehicles` | 已有 | VIN 级聚合实时状态 |
| `GET /api/realtime/locations` | 已有 | MySQL 每协议最新位置 |
| `GET /api/history/locations` | 已有 | TDengine 位置历史；基础分页和时间过滤 |
| `GET /api/history/raw-frames` | 已有 | TDengine RAW；默认 100 条，可选解析字段 |
| `POST /api/history/raw-frames/query` | 已有 | 复杂 RAW 查询 |
| `GET /api/mileage/summary`、`/daily` | 已有 | MySQL 日里程 |
| `GET /api/statistics/online-summary`、`/online-vehicles` | 已有 | 固定口径在线统计 |
| `GET /api/quality/summary`、`/issues`、`/notification-plan` | 已有 | 推导质量信号和静态方案，不持久化 |
| `GET /api/alert-events*` | 语义错误 | 上述质量接口别名，不是告警规则/事件/通知 |
| `GET /api/map/reverse-geocode` | 已有 | 服务端高德逆地理编码 |
| `GET /api/ops/health`、`/source-readiness` | 已有 | 链路、release、lag、连接、可写性和来源准备度 |
## 3. 按一期功能的缺口矩阵
| 功能 | 当前可复用 | 仍缺少 | 优先级 |
| --- | --- | --- | --- |
| 全局统计 | V2 monitor summary 已实现接入/在线/离线/行驶/静止/未知/今日上报，以及活跃业务告警 VIN 与同筛选车辆集合交集后的权威告警车辆数；页面仅在服务端明确可用时展示数值 | 更复杂的车型/企业/接入商组合字典可在车辆规模增长时从接入管理筛选下沉复用 | P0（一期完成） |
| 地图车辆 | V2 monitor map 已实现 bounds/zoom、关键词/协议/状态筛选、低缩放聚合、高缩放轻量点、2,000 点自适应降级；前端独立缓存、防抖请求和 MassMarks，列表上限 200 | 差量 cursor/SSE 作为车辆规模或刷新频率继续增长后的扩展项 | P0（一期完成） |
| 车辆信息卡 | 选中车辆按需并发组合 realtime、vehicle service、daily mileage、服务端逆地理编码和 `status=active` 告警；展示当日里程、地址、全部真实来源、接入厂家、当前告警及三处深链 | 权威档案未提供接入厂家时明确显示“待补充”，不从协议猜测 | P0（一期完成） |
| 单车档案 | vehicle service 已合并 `vehicle_profile`；管理员可单车维护或用 CSV/API 批量同步，外部同步具备来源/版本幂等、人工及异源保护、显式接管、dry-run、身份校验和版本审计 | 仍需为具体车厂/GPS 厂商配置定时连接器；首次接入/运行时长的自动投影仍待网关权威源 | P0（部分完成） |
| 最新遥测 | `GET /api/v2/vehicles/:vin/telemetry/latest` 已服务端选择每个统一指标/源字段最新值，动态分组，保留厂家扩展，并返回中文名、单位、类型、设备/接收时间、帧、协议、端点、freshness/delay 与值级质量原因；页面独立缓存刷新且不再猜测 RAW 元数据 | 更多动态 RAW 指标进入统一目录仍需受控发现和管理端审计，不影响一期最新值取证 | P0（一期完成） |
| 轨迹 | V2 playback 已有覆盖边界、最多 7 天校验、GPS 质量过滤、推断停车、活动分段、关键点保留抽稀、方向/SOC/告警同步字段和统计元数据 | ignition 可信怠速、持久 queryId/游标分段仍缺少 | P0（部分完成） |
| 回放地址 | 暂停点按坐标缓存逆地理编码，播放时暂停解析，避免逐点请求 | 服务端批量地址目录仅在未来需要轨迹点地址表时再建设 | P1（一期完成） |
| 历史列表/曲线 | V2 动态指标目录、多车多指标列表、服务端分页、按单位拆轴的时间序列聚合、空窗/覆盖率/异常证据和 60–600 点受控曲线已上线 | 更多可聚合遥测指标需随统一指标目录逐项开放；RAW 保持离散证据 | P0（一期完成） |
| 导出 | 单 ECS 持久异步 CSV 队列已上线：进度/状态/完成时间/下载、单并发、31 天/5 车/32 指标/100 万行/30 分钟配额、游标流式写入、重启恢复和原子文件发布 | XLSX、取消和多实例对象存储属于扩容项，不阻塞一期 CSV 验收 | P0（一期完成） |
| 告警规则 | V2 MySQL 规则、版本审计、CRUD/启停/校验、统一指标目录动态校验、数值区间内外、布尔状态变化、协议/VIN/OEM/车型/企业范围、持续/恢复/重复抑制已实现 | 软删除、复杂范围组合预估和批量规则导入仍待补齐 | P0（部分完成） |
| 告警事件 | V2 持久事件、事件时间持续候选、活跃指纹去重、恢复/处置时间线、通知、Kafka fields active consumer、规则副作用与 checkpoint 同事务、数据库权威重放抑制、迟到门禁、动态/墙钟所有权拆分、真实规则灰度、锁竞争及连续运行门禁均已上线 | 跨窗口历史重算属于后续分析能力；短信/邮件/企微供应商明确不在一期范围 | P0（一期完成） |
| 站内通知 | V2 列表、未读数、批量已读和页面轮询已实现；外部通道明确为 reserved | SSE 可在通知规模增长时替换轮询；外部供应商不在一期范围 | P1（一期完成） |
| 接入状态 | V2 summary/vehicles/thresholds 已实现动态阈值、事件/接收延迟、状态区分、协议/车辆厂家/车型/接入厂家/首次接入/最新上报筛选、同口径统计和版本审计；网关同一快照 upsert 已维护首次/前次/最新接收、连续间隔、样本数及回填边界；缺 VIN 的 JT808 终端已进入脱敏处置队列 | 仍需运维核对来源并维护权威 phone→VIN 绑定；历史首次接入只能由未来权威台账补齐，不能把上线回填当历史真值 | P0（部分完成） |
| 权限审计 | ECS 强制 Bearer 鉴权；viewer/operator/admin 累积权限、前端会话门禁、后端逐操作校验、规则/阈值/告警动作版本审计及越权拒绝均已验证 | 企业 SSO 与复杂多租户不在一期范围 | P0（一期完成） |
## 4. 推荐 V2 API
### 4.1 元数据与车辆
| 方法与路径 | 用途 | 关键参数/响应 |
| --- | --- | --- |
| `GET /api/v2/meta/vehicle-filters` | 全局筛选字典 | OEM、车型、公司、协议、接入厂家、状态；带版本/ETag |
| `GET /api/v2/metrics` | 动态指标目录 | `protocol, category, valueType, searchable, chartable, alertable` |
| `GET /api/v2/vehicles/search` | 远程车辆选择 | `q, cursor, limit`；返回 VIN、车牌、车辆编号 |
| `GET /api/v2/vehicles/:vin` | 单车数字档案 | 档案、实时摘要、接入摘要、当前告警摘要 |
| `GET /api/v2/vehicles/:vin/telemetry/latest` | 动态最新遥测 | 按 category 分组，值含时间、单位、质量、来源 |
### 4.2 全局监控
| 方法与路径 | 用途 | 关键参数/响应 |
| --- | --- | --- |
| `POST /api/v2/monitor/summary` | 与筛选完全一致的状态统计 | 组合过滤对象、`asOf`、在线阈值版本 |
| `POST /api/v2/monitor/map` | 视口聚合/车辆点 | `bounds, zoom, filters, cursor`；返回 `mode=clusters|points`、聚合数或轻量点 |
| `GET /api/v2/monitor/vehicles/:vin/card` | 点位详情卡 | 核心实时、当日里程、来源、告警、地址缓存 |
| `GET /api/v2/monitor/changes` | 可选差量刷新 | `since` 或 SSE；返回变更车辆和统计版本 |
`monitor/map` 必须有最大点数和聚合降级；低缩放级别绝不返回全部明细。轻量点不携带完整 telemetry JSON。
### 4.3 轨迹
| 方法与路径 | 用途 | 关键参数/响应 |
| --- | --- | --- |
| `POST /api/v2/tracks/query` | 轨迹分段查询 | VIN、时间、协议、filter、targetPoints、cursor；返回点、起终点、过滤/抽稀计数和算法版本 |
| `GET /api/v2/tracks/:queryId/segments` | 大查询后续分段 | segment/cursor；支持取消/过期 |
| `GET /api/v2/tracks/:queryId/stops` | 停车点 | 起止时间、时长、坐标、地址缓存状态 |
轨迹点至少包含 `eventTime, receivedAt, lng, lat, speedKmh, directionDeg, socPercent, totalMileageKm, alarmFlag, statusFlag, state`。抽稀必须保留起终点、停车边界和告警点，并返回原始/有效/返回点数。
### 4.4 历史分析
| 方法与路径 | 用途 | 关键参数/响应 |
| --- | --- | --- |
| `POST /api/v2/timeseries/query` | 多车多指标曲线 | VINs、metricKeys、时间、granularity、aggregation、targetPoints、fillPolicy |
| `POST /api/v2/timeseries/rows` | 动态列列表 | 同上 + cursor、limit、sort、filters |
| `POST /api/v2/timeseries/estimate` | 查询/导出预估 | 估算点数、行数、扫描范围并给出建议粒度 |
服务端只允许指标目录白名单，不能把客户端 metricKey 直接拼入 SQL。响应带指标显示名、单位、类型、来源、采样方式和空值策略。
### 4.5 导出任务
| 方法与路径 | 用途 |
| --- | --- |
| `POST /api/v2/exports` | 创建任务；携带查询快照、CSV/XLSX、时区；使用幂等键 |
| `GET /api/v2/exports` | 当前用户任务分页 |
| `GET /api/v2/exports/:id` | 状态、进度、行数、完成/过期时间、错误 |
| `POST /api/v2/exports/:id/cancel` | 取消排队/运行任务 |
| `GET /api/v2/exports/:id/download` | 权限检查后短期下载或签名 URL |
状态：`queued, running, succeeded, failed, canceled, expired`。同步 API 不生成大文件。
### 4.6 告警与通知
| 方法与路径 | 用途 |
| --- | --- |
| `GET/POST /api/v2/alert-rules` | 规则列表/创建 |
| `GET/PUT/DELETE /api/v2/alert-rules/:id` | 详情/修改/软删除 |
| `POST /api/v2/alert-rules/:id/enable|disable` | 明确启停操作 |
| `POST /api/v2/alert-rules/validate` | 校验指标类型、操作符、阈值和范围 |
| `GET /api/v2/alert-events`、`/summary` | 事件列表和统计 |
| `GET /api/v2/alert-events/:id` | 事件详情与完整处理时间线 |
| `POST /api/v2/alert-events/:id/acknowledge` | 确认/进入处理中 |
| `POST /api/v2/alert-events/:id/close` | 关闭，要求备注 |
| `POST /api/v2/alert-events/:id/ignore` | 忽略，要求原因 |
| `GET /api/v2/notifications`、`/unread-count` | 站内通知/未读数 |
| `POST /api/v2/notifications/read` | 批量已读 |
所有变更接口记录操作者、时间、前后状态、备注和 traceId；规则修改采用 `version` 乐观锁。
实现记录（2026-07-14）：实际落地路由统一在 `/api/v2/alerts/*`，列表/汇总使用 POST 同构筛选体，处置统一为 `POST /events/:id/actions`，规则启停为版本化 `PUT /rules/:id/enabled`。迁移为 `002_alert_center.sql`、`003_alert_rule_advanced.sql` 与 `004_alert_repeat_index.sql`，独立 `alert-evaluator` systemd 服务以当前 MySQL realtime location 作为一期评估输入。旧 `/api/alert-events*` 继续只是质量投影兼容接口，新页面不依赖它们。Kafka 可重放消费、迟到窗口和 traceId 写入通用审计仍是生产退出缺口。
指标目录实现记录（2026-07-14）：`GET /api/v2/metrics` 已返回统一 key、中文名、单位、类别、值类型、协议及源字段映射和三类能力标记；生产权威数据已落入 `vehicle_metric_definition/vehicle_metric_protocol_mapping`，`005_metric_catalog.sql` 仅补齐初始缺失项，不覆盖后续配置。告警编辑器只消费其中 `alertable=true` 且类型匹配的指标，规则写接口再次校验白名单、能力、值类型和 evaluator 支持。`GET /api/v2/vehicles/:vin/telemetry/latest` 现消费同一目录并为未入目录的厂家字段保留源字段、动态分类和质量证据；历史页仍保留 `/api/v2/history/metrics` 兼容目录。后续仅剩动态 RAW 指标受控发现与管理端版本审计。
### 4.7 接入管理
| 方法与路径 | 用途 |
| --- | --- |
| `POST /api/v2/access/summary` | 按同一筛选口径统计协议、厂家、在线率、今日上报、长离线、从未上报、延迟异常 |
| `POST /api/v2/access/vehicles` | 接入状态分页；支持阈值、时间、厂家/车型/协议过滤 |
| `POST /api/v2/access/unresolved-identities` | 无权威 VIN 的真实终端处置队列；只返回哈希 ID、脱敏标识和登记/上报证据 |
| `GET /api/v2/access/thresholds` | 获取全局默认和协议覆盖阈值 |
| `PUT /api/v2/access/thresholds` | 管理员更新阈值并记录版本/审计 |
接入行应返回 `firstSeenAt, latestEventAt, latestReceivedAt, reportIntervalSec, dataDelaySec, onlineState, thresholdSec, latestMessageType, latestError`，并区分 `online/offline/never_reported/unknown`。
实现记录（2026-07-14）：状态/阈值接口、同口径筛选、动态状态、阈值乐观锁和 MySQL 审计已落地；gateway access projection 已维护 first/previous/latest received、latest error、样本数和连续间隔。`platform-v2-20260714080619` 新增身份待绑定接口和紧凑处置队列：生产当前识别 1 个真实 JT808 缺 VIN 终端，响应和复制证据均仅含脱敏标识，20 次顺序查询 p50 2.02ms、p95 2.44ms、最大 4.66ms。该队列用于推动权威绑定，不会把手机号猜成 VIN，也不会让未绑定身份参与车辆告警。
## 5. 横切契约
- 时间统一使用 RFC 3339/UTC 传输，UI 按 Asia/Shanghai 展示；同时保留事件时间和接收时间。
- 列表固定 `items + pageInfo`；大时序优先 cursor，管理列表可 offset；总数默认可选，避免高成本 COUNT。
- 所有筛选/排序字段白名单化；请求体大小、VIN 数、指标数、时间范围、返回点数均设上限。
- API 返回 `traceId`、数据 `asOf`、口径/算法版本；可取消请求使用 `AbortSignal`。
- 查询 GET 可 ETag/短缓存；任务/规则 POST 使用幂等键；状态修改使用版本号。
- 前端只访问平台 BFF。Redis/TDengine/MySQL 连接与高德 REST key 均不暴露。
## 6. 兼容与下线
旧接口在 V2 联调期保留；新页面不依赖旧 `alert-events` 别名。完成迁移后先增加 deprecation 响应头和调用监控，再按版本窗口下线。现有 `/api/history/locations` 和 RAW 查询可继续作为运维证据接口，不强行替换为业务时序 DTO。

224
docs/vehicle-data-platform-data-model.md Normal file
View File

@@ -0,0 +1,224 @@
# 车辆数据中台一期数据模型
## 1. 数据职责原则
| 存储 | 负责 | 不负责 |
| --- | --- | --- |
| Redis DB 50 | 最新状态、在线 TTL、字段级当前值、热查询缓存 | 历史事实、告警事件、规则、导出任务 |
| TDengine `lingniu_vehicle_ts` | 高频 RAW 证据、位置/稳定时序历史 | 车辆档案、权限、工作流状态 |
| MySQL `lingniu_vehicle_data` | 车辆主数据、身份映射、接入配置、当前业务投影、统计、规则、事件、通知、导出任务 | 每帧完整历史和无限增长的遥测明细 |
统一业务主键为 VIN。车牌、JT808 手机号、设备号、平台标识只是可变标识，通过映射解析到 VIN；无法解析 VIN 的数据仍留 RAW/接入排障，不进入正式车辆统计。
## 2. 当前 TDengine 模型
### 2.1 `raw_frames` stable
列：`ts, frame_id, event_id, message_id, event_time, received_at, raw_size_bytes, raw_hex, raw_text, parsed_json, parse_status, parse_error, source_endpoint`。
Tags：`protocol, vehicle_key, vin, phone, device_id`。
用途：证明收到什么、何时收到、解析成什么。`parsed_json` 物理列保存 Gateway 已一次扁平化的 `parsed_fields`，是动态协议字段的历史证据。新字段默认先进入这里，不为每个厂家字段立即新增物理列。
### 2.2 `raw_frame_payload_chunks` stable
列：事件/帧、接收时间、payload 类型、分片序号/总数、分片文本；Tags 与 RAW 身份类似。
用途：保存超过主表列长度的 raw/parsed payload，不参与常规业务查询。
### 2.3 `vehicle_locations` stable
列：`ts, event_id, received_at, longitude, latitude, altitude_m, speed_kmh, direction_deg, alarm_flag, status_flag, total_mileage_km`。
Tags：`protocol, vin`。
用途：VIN 非空车辆的高频位置、速度、方向、告警/状态位、总里程历史。当前平台 DTO 尚未完整透出方向、告警和状态位，也没有 SOC 列；轨迹一期需先补 DTO/查询，SOC 可从 RAW 指标取值或按验证后的需求物化。
数据库当前配置 `KEEP 7300 DURATION 10 BUFFER 256`。保留期和磁盘容量应纳入运维监控，不在业务页面查询时临时改变。
## 3. 当前 Redis 模型
| Key | 值/用途 | 约束 |
| --- | --- | --- |
| `vehicle:latest:{vin}` | 跨协议最新核心字段 | 仅 VIN 非空；轻量快照 |
| `vehicle:latest:{vin}:{protocol}` | 单协议最新核心字段 | 可过期/重建 |
| `vehicle:realtime-raw:{protocol}:{vin}` | 单协议最新完整 parsed 状态 | Redis 中完整协议字段唯一副本 |
| `vehicle:rt-kv:{protocol}:{vin}:values` | 扁平字段当前值 | 与类型/时间/meta 配套 |
| `...:types` | 字段值类型 | 支持动态展示和规则类型校验的实时侧依据 |
| `...:times` | 每字段最新归一事件时间 | 防止乱序旧帧覆盖 |
| `...:meta` | 最新事件/接收时间、映射版本 | 用于新鲜度和追踪 |
| `vehicle:online:{protocol}:{vin}` | TTL 在线状态 | 在线事实的快速判断，不应永久保存 |
| `vehicle:online-state:{protocol}:{vin}` | 可分页的 Hash 副本 | 用于在线列表 |
| `vehicle:protocols:{vin}` | 最近出现协议集合 | 车辆来源发现 |
| `vehicle:last_seen` | 最近活跃车辆 ZSET | 分页/排序候选 |
生产只允许 `nats-fast-writer` 写 Redis 当前态；Kafka 慢链路的 realtime writer 不应覆盖快路径。在线产品状态应由“最新时间 + 配置阈值”计算并返回阈值/依据，不能只信一个固定 Boolean。
## 4. 当前 MySQL 模型
### 4.1 身份与档案
| 表 | 主键/关键字段 | 用途 |
| --- | --- | --- |
| `vehicle` | `vin`；`plate, oem, enabled, updated_at` | 轻量车辆主实体；当前字段不足一期完整档案 |
| `vehicle_identifier` | `(protocol, source_code, identifier_type, identifier_value)`；VIN/车牌/OEM | 多协议、多来源、多标识解析到 VIN |
| `vehicle_identity_binding` | VIN；车牌、phone、OEM | 外部维护的兼容映射事实，运行期只读 |
| `jt808_registration` | `phone`；device、plate、VIN、厂家、鉴权、来源、首次/最近注册鉴权/出现时间 | JT808 注册鉴权与绑定证据 |
### 4.2 当前态与来源
| 表 | 主键/关键字段 | 用途 |
| --- | --- | --- |
| `vehicle_realtime_snapshot` | `(protocol, vin)`；plate、platform、peer、扁平 `parsed_json`、event/received time | 每协议每 VIN 最新合并遥测快照 |
| `vehicle_realtime_location` | `(protocol, vin)`；经纬度、速度、总里程、SOC、海拔、方向、alarm/status、event/received time | 每协议每 VIN 最新位置业务缓存 |
| `vehicle_data_source` | 自增 ID，唯一 `(protocol, source_ip)`；source code/kind、platform、priority、enabled、first/latest seen | 数据来源配置和可信源选择 |
### 4.3 日里程
| 表 | 主键/关键字段 | 用途 |
| --- | --- | --- |
| `vehicle_daily_mileage_source` | `(vin, stat_date, protocol, source_key)`；首末里程、样本、质量、是否选中 | 多来源日里程事实和质量解释 |
| `vehicle_daily_mileage` | `(vin, stat_date, protocol)`；source ID、日里程、最新总里程 | 对外查询结果投影 |
当前明确不存在/不应恢复的旧模型包括泛化 `vehicle_daily_metric`、TDengine `vehicle_mileage_points` 和 `raw_frames.fields_json` 重复列。
## 5. 一期统一领域模型
### 5.1 车辆标识
```text
VehicleId = VIN（内部规范化大写、去空格）
VehicleIdentifier = protocol + sourceCode + type + value -> VIN
Plate = 可变展示标识，不作为跨表唯一主键
Source = protocol + sourceCode/sourceId + endpoint
```
所有 API 都以 VIN 作为详情路径主键；搜索接口可接受车牌、VIN、车辆编号/手机号并返回解析结果。多协议数据保留 `protocol` 和 `sourceId` 做归因，不能在合并时丢失来源。
### 5.2 时间模型
- `eventTime`：车端/协议事件时间，是轨迹、曲线和告警判断的主要时间。
- `receivedAt`：平台接收时间，用于延迟、迟到数据和链路健康。
- `storedAt/updatedAt`：投影或业务记录更新时间，不冒充车辆上报时间。
- `dataDelaySec = receivedAt - eventTime`；`silenceSec = now - max(receivedAt/eventTime 的已确认口径)`。
### 5.3 在线状态
```text
never_reported: 无任何 latestReceivedAt
online: silenceSec <= thresholdSec
offline: silenceSec > thresholdSec
unknown: 时间非法、未来时间过大或来源状态无法判断
```
行驶/静止是独立 motion 状态，建议以速度阈值 + 最近更新时间判断；告警也是独立状态。不要把 `online/running/alarm` 压成一个互斥枚举，UI 可基于多维状态决定主色和徽标。
### 5.4 指标值
```text
MetricValue {
vehicleId, metricKey, value, valueType, unit,
eventTime, receivedAt, protocol, sourceId,
quality, mappingVersion
}
```
`metricKey` 是协议无关的规范键；协议原字段通过映射表关联。厂家扩展字段允许命名空间，但必须有显示名、类型、分类和单位后才可进入图表/告警。
## 6. 建议新增 MySQL 模型
### 6.1 档案与组织
建议优先用关联表而非不断扩宽 `vehicle`：
- `vehicle_profile(vin PK, vehicle_no, model_id, vehicle_type, company_id, operation_status, access_vendor_id, first_access_at, ...)`
- `vehicle_model(id, oem_id, code, name, vehicle_type, ...)`
- `organization(id, parent_id, type, code, name, enabled, ...)`
- `access_vendor(id, code, name, enabled, ...)`
外部主数据通过受控同步写入 `source_system/source_version/synced_at`。默认策略保护 `manual` 和其他外部来源；只有管理员显式选择 takeover 才能改变所有权。同一来源版本相同载荷幂等，不同载荷冲突，避免上游版本不可变性被破坏；每次创建/更新都进入 `vehicle_profile_audit`。
### 6.2 指标目录
- `metric_definition(metric_key PK, display_name, category, value_type, unit, precision, chartable, alertable, enabled, sort_order, version)`
- `metric_protocol_mapping(metric_key, protocol, source_field, transform, mapping_version, enabled)`
`transform` 不能存任意可执行代码；使用有限转换类型或由版本化 Go 映射实现。目录服务可缓存，变更有版本和审计。
### 6.3 告警域
- `alert_rule(id, name, description, metric_key, operator, threshold_json, severity, duration_sec, recovery_json, repeat_interval_sec, enabled, version, created_by, updated_by, timestamps)`
- `alert_rule_scope(rule_id, scope_type, scope_value)`：vehicle/model/oem/protocol/company；一期可限制组合复杂度。
- `alert_event(id, rule_id, rule_version, vin, protocol, source_id, status, trigger_value_json, threshold_json, first_triggered_at, last_triggered_at, recovered_at, closed_at, location_json, dedupe_key, assignee, timestamps)`
- `alert_event_action(id, event_id, action, from_status, to_status, operator_id, note, created_at)`
- `notification(id, user_id, event_id, title, body, read_at, created_at)`
`alert_event.dedupe_key` 建唯一约束或等价幂等机制；阈值快照和 rule version 必须保存在事件上，避免规则修改后历史无法解释。
当前实现使用带业务前缀的实体表，完整 DDL 位于 `vehicle-data-platform/deploy/migrations/002_alert_center.sql`：
- `vehicle_alert_rule` + `vehicle_alert_rule_audit`：当前规则与每版本不可变快照。
- `vehicle_alert_candidate`：`(rule_id, vin, protocol)` 持续命中计时，避免单帧达到阈值就误开事件。
- `vehicle_alert_stream_checkpoint`：`(consumer_group, topic, partition_id)` 保存数据库权威 `next_offset`、Kafka high watermark、处理/非法/迟到/回放计数、最近事件证据以及最近非法代码/时间。流 worker 先提交该事务，再提交 Kafka group offset；崩溃后重复抓取的 offset 会被数据库 checkpoint 跳过。累计非法数用于审计，健康状态只对五分钟内仍在发生的非法消息告警。
- `vehicle_alert_event`：保存规则名/版本、触发阈值快照、证据时间、位置和乐观锁版本；`fingerprint + active status` 用于评估器幂等检查。
- `vehicle_alert_event_action`：触发、恢复和人工处置的不可变时间线。
- `vehicle_alert_notification`：站内通知真实已读状态；外部通道只有 `reserved`，没有配置供应商时绝不标记 `sent`。
- `vehicle_alert_rule_state`：为 Boolean `changed` 规则保存每个 rule/VIN/protocol 的最近观测值；`003` 同时增加区间上限和 OEM 范围，`004` 为 fingerprint 重复间隔查询增加索引。
一期 evaluator 读取 `vehicle_realtime_location` 的当前快照并用 candidate 表证明持续时长。除 `freshness_sec` 明确按平台墙钟增长外，候选只允许由不同 `source_event_id` 且事件时间不回退的新观测推进；重复快照不能制造持续时长，迟到观测不能回退 candidate 或 Boolean 状态。规则行会在评估事务内加锁，停用规则、清理 candidate/Boolean 状态和写版本审计在同一事务提交，从而避免 evaluator 与管理员停用并发后留下幽灵候选。它仍不是 Kafka event-time 可重放评估器，因此跨当前快照的迟到数据重算、历史窗口规则和消费 offset 仍是阶段 8 的后续能力。
`alert-stream-evaluator` 已以 active 模式消费三类 canonical fields topic。它复用 Kafka consumer group 的分区顺序，校验 `event_kind/field_mapping/protocol/VIN/source_event_id/字段命名空间`，用接收时间识别超过可配置窗口的迟到观测。动态规则锁、当前批次范围内的 candidate/Boolean/活跃事件/重复窗口、事件/动作/通知副作用和 checkpoint 在同一个 MySQL 事务内完成，成功后才 commit Kafka；数据库 ahead 时的重复 offset 由 checkpoint 跳过。超过迟到窗口的观测只允许驱动 `data_delay_sec`，不能开关速度/SOC/告警位事件。快照 evaluator 在 active 模式只负责按墙钟增长的 `freshness_sec`，不再读取动态规则。
### 6.4 导出域
- `export_job(id, user_id, type, format, query_json, status, progress, estimated_rows, exported_rows, file_uri, file_size, checksum, error_code, error_message, started_at, completed_at, expires_at, created_at)`
索引至少覆盖 `(user_id, created_at)`、`(status, created_at)` 和过期清理。`query_json` 保存规范化查询快照，不保存数据库密码或签名下载 URL。
### 6.5 在线阈值与审计
- `online_threshold(scope_type, scope_value, threshold_sec, version, updated_by, updated_at)`，一期建议 scope 仅 global/protocol。
- `audit_log(id, actor, action, resource_type, resource_id, before_json, after_json, trace_id, created_at)`。
当前 V2 接入管理已先落地等价的窄表：
- `vehicle_access_threshold_config(id=1, version, default_threshold_sec, delay_threshold_sec, long_offline_sec, protocol_overrides_json, updated_by, updated_at)`。
- `vehicle_access_threshold_audit(id, version, actor, summary, config_json, changed_at)`。
表仅向前新增，不改变 gateway 的 `vehicle_realtime_snapshot(protocol, vin)` 主键与写入语义。后续统一审计域上线时可迁移到通用 `audit_log`，但必须保留版本历史。
## 7. 查询与物化策略
1. 实时页面优先 Redis，MySQL realtime 作为业务查询/降级投影；返回值统一由 BFF 归一化。
2. 轨迹只查 TDengine `vehicle_locations`，必要的 SOC/告警补充应避免对每个点逐条回查 RAW；先验证是否需要将 SOC 提升为位置列或建立专用指标 stable。
3. 通用历史指标首版可从 `raw_frames.parsed_json` 按白名单提取，但必须限制车辆数、指标数、时间范围并做基准；高频稳定指标按实际查询热度物化为专用时序模型。
4. 档案和字典在 BFF/Redis 做版本化短缓存，避免每次实时请求重复查不变数据。
5. 地图聚合可从 MySQL realtime/Redis 快照构建短周期服务端缓存；不要把地理历史查询压到实时表，也不要把全量点放浏览器聚合。
6. 告警 worker 使用 Kafka 可重放流和 eventTime，MySQL 保存工作流事实；Redis 可保存短期去重/规则缓存但不是唯一事实。
## 8. 数据质量与治理要求
- 数值同时记录单位和精度；未知单位不得参与跨协议比较或规则判断。
- 原始值、规范值和转换版本可追踪；异常值标记不直接篡改 RAW。
- 未来时间、经纬度越界、速度突变、里程回退、重复事件均定义质量码。
- 轨迹过滤返回算法版本、原始/过滤/抽稀数量，可选择查看未过滤证据。
- 协议新增字段先进入扁平 RAW + Redis KV，经目录登记后才进入 UI；形成稳定高频查询后再物化。
- 所有新增表、字段、索引、保留期和清理任务随 migration 与数据字典发布。
## 9. 容量和保留
- 约 1,000 到 10,000 车辆、5～30 秒上报意味着时序写入持续增长；TDengine 查询必须带 VIN/时间，禁止开放无界扫描。
- RAW 保留期、位置保留期、导出文件保留期和告警/审计保留期分别配置，不能共用一个默认值。
- MySQL 告警事件和审计按时间索引并规划归档；导出任务定时过期，文件删除和 DB 状态更新需幂等。
- 单 ECS 上告警和导出 worker 设置独立并发、内存、CPU 和 systemd 限制，避免影响接入写链路。
## 10. 仍需确认的数据问题
1. `vehicle_identity_binding` 的实际 DDL、权威维护方和未来是否继续兼容。
2. 车型、车辆类型、运营公司、车辆编号、运营状态、接入厂家的权威来源与更新频率。
3. 各协议 SOC、方向、启动、告警等级、电机/燃料电池等规范字段映射和单位。
4. “累计运行时长”是已有车端字段、由状态积分计算，还是一期可不提供。
5. 在线时间基准使用 eventTime 还是 receivedAt，以及不同协议的默认阈值。
6. 轨迹停车阈值、漂移判定、迟到点容忍和跨协议主轨迹选择规则。
7. 导出存储与保留期、单用户并发/最大行数。
8. 告警规则适用范围组合、迟到窗口、重复/恢复语义和事件归属人规则。

162
docs/vehicle-data-platform-roadmap.md Normal file
View File

@@ -0,0 +1,162 @@
# 车辆数据中台一期实施路线图
> 本路线图从分析完成后的下一阶段开始。任何阶段都以“契约、实现、测试、运行检查、文档”全部完成为退出条件，不并行铺开所有页面。
## 1. 阶段总览
| 阶段 | 目标 | 主要交付 | 退出条件 |
| --- | --- | --- | --- |
| 0 分析与决策 | 冻结一期边界和关键决策 | 本次四份文档、P0 决策记录 | 档案来源、认证、指标元数据、在线阈值、文件存储有负责人和结论 |
| 1 数据与 API 基座 | 建立统一模型和可演进契约 | MySQL 向前迁移、指标目录、V2 OpenAPI、鉴权/审计、统一错误/分页 | 契约测试通过；旧链路不受影响；迁移可回滚 |
| 2 V2 前端基础 | 建立独立、无 mock 的前端壳 | 路由/菜单、权限、token、API client、AsyncState、地图/图表/表格基础组件 | 构建和组件测试通过；1440×900 基础布局可用；无 mock fallback |
| 3 全局监控 | 先打通实时主路径 | 地图聚合、组合筛选、统计、车辆信息卡、详情跳转 | 1k/10k 数据基准达标；视口移动不卡顿；状态口径一致 |
| 4 单车数字档案 | 形成统一车辆入口 | 档案、实时状态、地图、动态遥测分组、来源证据 | 车牌/VIN/车辆编号均可解析；缺失字段不渲染空大表 |
| 5 轨迹回放 | 完成历史位置产品能力 | 分段/抽稀查询、过滤、停车点、播放控制、列表联动 | 长时段不会全量阻塞；抽稀/过滤可解释；播放状态正确 |
| 6 历史分析与导出 | 建立通用指标分析 | 指标目录、多车多指标列表/曲线、异步 CSV/XLSX | 百万点查询经服务端聚合；导出有任务进度和受控下载 |
| 7 接入管理 | 产品化接入质量 | 动态在线阈值、延迟、最近消息/错误、协议/厂家统计 | 从未上报与离线可区分；事件/接收时间延迟口径明确 |
| 8 告警中心 | 建立可持久业务告警 | 规则 CRUD、评估 worker、事件、处理记录、站内通知 | 数值/Boolean 规则可验证；去重、恢复、确认、关闭链路完整 |
| 9 联调与生产发布 | 达到一期验收并部署当前 ECS | 性能/异常/安全测试、运维文档、release、监控和回滚 | 验收清单通过，ECS 健康、无 lag、旧 release 可回滚 |
进度注记（2026-07-14）：阶段 7 的 V2 接入汇总、车辆状态、动态阈值、协议覆盖、版本审计及桌面/移动端页面已实现。网关 realtime snapshot 同一条原子 upsert 现维护独立于设备事件时间的首次/前次/最新接收时间、连续间隔、样本数和 event ID 去重；平台读取该投影并合并车型/企业，明确区分 `live_writer` 真首次观测和 `snapshot_backfill` 上线基线。`platform-v2-20260714080619` 又将缺少权威 VIN 的真实 JT808 终端纳入“身份待绑定”队列，只暴露哈希 ID、脱敏终端号和登记/上报证据；当前生产 1 条，页面与复制内容均未泄漏原始手机号，查询 p95 2.44ms。实际 phone→VIN 绑定仍必须由运维核验权威来源，不能猜测。
阶段 8 进度注记（2026-07-14）：durable 规则/审计、持续候选、事件/动作、站内通知、独立 evaluator、V2 API 与告警中心三个工作区已上线；确认、规则创建、通知已读和权限通过真实接口联调。`platform-v2-20260714070507` 将持续时间改为由不同 source event 的单调事件时间推进，重复快照和迟到观测不再制造持续异常，`freshness_sec` 作为明确的墙钟规则单独处理；后续 Kafka fields active evaluator 又补齐分区 checkpoint、迟到窗口和数据库权威重放抑制。真实规则灰度、规则编辑锁竞争及 31 分钟连续运行门禁均通过，阶段 8 一期退出项已关闭。
Kafka 告警流进度注记（2026-07-14）：`platform-v2-20260714080619` 当前生产仍运行 active event-time evaluator。动态规则锁、批次范围 candidate/Boolean/活跃事件/重复窗口、事件/动作/通知和 `(group,topic,partition)` checkpoint 在同一个 MySQL 事务内完成，随后才 commit Kafka；失败事务不会越过 offset。快照 evaluator 在 active 模式只保留 `freshness_sec`，空动态规则扫描由约 13ms 降至 0.5–1ms。迁移 `010` 已用真实 RAW 样本修正国标告警位和宇通速度/SOC 映射。受限 JT808 `speed>=0`、持续 20 秒、重复 1 小时灰度只打开 1 条事件；关闭后又有 2 批 candidate 推进但未重开，随后规则停用并清理状态。914 个带活跃规则的生产批次 p50 7.80ms、p95 12.07ms、最大 18.68ms，候选推进 11、重复观测 9、迟到观测 1、批次失败 0。迁移 `011` 进一步持久化最近非法原因/时间；当前已定位为单个未绑定 VIN 的 JT808 终端，保留 `missing_vin_jt808` 运维告警并在接入页生成脱敏处置队列。当前 9 分区 lag 0，跨窗口历史重算和代表性峰值长稳仍待阶段 9 门禁。
规则编辑锁竞争门禁（2026-07-14）：生产创建了不可能命中业务车辆的启用规则，在 active stream evaluator 持续消费时完成 60 次顺序版本化编辑，HTTP 200 为 60/60，冲突和 5xx 均为 0；编辑延迟 p50 3.991ms、p95 7.220ms、p99 8.584ms、最大 10.875ms。并发窗口 41 个活跃规则批次处理 303 条消息，失败、重放、误开和恢复均为 0；规则随后停用且事件总数为 0。锁竞争退出项已关闭，仍保留代表性峰值长稳与 RDS 峰值观测。
阶段 5 进度注记（2026-07-14）：生产真实轨迹基线显示活跃车辆总点数约 9,500–71,361，旧接口仅加工最新 5,000 点但摘要未充分表达覆盖边界。`platform-v2-20260714060112` 已在当前 ECS 上线默认今天、最长 7 天校验、完整/最新切片证据、同协议无效坐标/重复/疑似漂移过滤、多源主来源选择、GPS 推断停车、行驶/停车/数据间隔分段、质量摘要，以及保留事件和分段边界的抽稀映射。8 辆生产 canary 今天窗口为 5.7–34.22ms；正式切流抽样 2,254 个源点完整读取、1,200 个地图点、33 段和 5 次停车为 32.83ms。并行协议不会被拼成一条路线，停车也不冒充 ignition 状态；服务端持久 queryId/游标分段和长周期观测仍待后续数据合同。
轨迹与接入补齐注记（2026-07-14）：`platform-v2-20260714083640` 已在当前 ECS 上线。`vehicle_locations` 向前增加 SOC，轨迹返回 SOC 可用性、方向和告警并使用绝对时间戳；页面以 `Asia/Shanghai` 展示，同步卡片只为暂停点做一小时坐标缓存的地址解析。生产宇通 5 分钟 300 点全部有 SOC，JT808 5 分钟 10 点全部有方向/告警；修复了 TDengine 时间窗二次偏移及亚秒采样被截断成零秒后误判 gap 的问题，5,000 点页面分段从 3,203 降到 14，仅保留 1 个真实 28 分钟间隔，无横向溢出或控制台错误。接入页新增车型、接入厂家、首次接入和最新上报范围；生产 `G7s + latestSeenFrom` 深链筛选当前页 50/50 匹配。小窗位置导出 120/120 行完成，Kafka lag 0，MySQL/TDengine 可写。
全局监控性能闭环（2026-07-14）：`platform-v2-20260714090040` 已在当前 ECS 上线。此前页面虽有后端聚合接口，实际仍把 1,000 行普通车辆列表直接送入地图；现改为 200 行服务端筛选列表与独立地图查询，`moveend/zoomend` 300ms 防抖并由 React Query 去重缓存。低缩放返回聚合，高缩放按 bounds 返回轻量 MassMarks，超过 2,000 个点时自适应扩大网格，10,000 合成车辆不会退化成 10,000 个点或聚合标记。10k 纯聚合基准 30 次为 0.976ms/op；当前 ECS 727 辆真实数据 30 次低缩放聚合为 29 组，API p50 57.31ms、p95 63.44ms、最大 70.65ms，城市视口 168 点 p50 94.89ms、p95 108.56ms、最大 114.54ms。生产浏览器首屏为 15 个聚合/200 行，无横向溢出；行驶筛选所有加载行均匹配，唯一车牌查询自动从聚合切到 1 个点并聚焦到 2km 比例尺。
全局监控产品闭环（2026-07-14）：`platform-v2-20260714093533` 将活跃告警 VIN 与当前关键词/协议/状态筛选后的车辆集合在服务端求交，`alertDataAvailable=true` 后页面展示权威告警车辆数，不再使用占位。选中车辆卡片按需并发读取档案/日里程、当前业务告警和地址，并展示真实来源、接入供应商、今日里程、解析位置及当前告警；无权威供应商时明确“待补充”。生产 727 车显示告警车辆 0，抽样车辆今日里程 59.4 km、三协议来源、嘉兴解析地址和 0 条当前告警；1280×720 无页面溢出。
单车最新遥测闭环（2026-07-14）：`platform-v2-20260714092531` 已在当前 ECS 上线 `GET /api/v2/vehicles/:vin/telemetry/latest`。服务端复用权威指标目录统一 key/中文名/单位/类型/分类，同时保留真实源字段、厂家扩展、协议、端点、帧、设备/接收时间、freshness/delay 和 `good|stale|warning` 原因；前端不再从 RAW key 猜测标签或质量。身份只解析一次，三个协议表固定并行各取最多 5 帧且跳过分页计数，真实三源车辆扫描 10 帧返回 155 值/7 分组；生产 30 次 p50 174ms、p95 179ms、最大 179ms，本地 100 帧×50 字段纯归并基准 0.389ms/op。页面使用独立 10 秒缓存、20 秒刷新，一次遍历建立分组与来源索引；1280 桌面和 390×844 移动端均无横向溢出，行内时间为紧凑时分秒，控制台无错误或警告。
连续运行门禁（2026-07-14）：alert stream evaluator 自 08:31:50 起连续 active 运行超过 31 分钟且未重启；该窗口 16,226 批、216,118 条消息全部完成处理，valid 216,072、历史未绑定身份导致 invalid 46、replay skipped 0、失败日志 0，批次延迟 p50 6.511ms、p95 10.481ms、p99 12.806ms、最大 29.697ms。结合此前 929 个带活跃规则批次和 60 次并发版本编辑均无失败/误开，已覆盖正常流量长稳及规则锁竞争。窗口结束时 checkpoint 累计 processed 563,598、9 分区 lag 0、MySQL/TDengine 可写；最近非法身份时间已超过 20 分钟且脱敏处置队列仍保留。阶段 9 的代表性连续运行门禁关闭，云数据库 Performance Insights 仍作为容量扩容观察项，不再阻塞单 ECS 一期上线。
阶段 6 进度注记（2026-07-14）：`platform-v2-20260714063725` 已在当前 ECS 上线历史时间序列聚合与百万行受控导出。位置历史按车辆与协议在 TDengine 使用 `PARTITION BY protocol INTERVAL(...)` 服务端聚合，速度使用桶均值、总里程使用桶末值并保留 min/max/样本数；最长 31 天、每序列目标 60–600 点、空窗不 `FILL`，前端按单位拆成速度/总里程小多图并明确粒度、覆盖率、缺失桶、原始点数和查询耗时。真实 canary 单车双协议 1,392 个原始点聚合为 127 桶、254 浏览器点，20.67ms；5 台车 8,197 个原始点聚合为 376 桶、752 浏览器点，86.53ms。导出采用单并发队列、5,000 行 TDengine 复合游标、30 分钟超时、100 万行双重配额、流式 CSV、`.part` + `fsync` + 原子发布，并持久化总行数、已处理行数、文件大小和完成时间。预发布门禁发现并修复 8 小时时区偏移和移动端面板压缩；最终桌面与 390×844 验收无横向溢出。RAW 保持离散证据而不伪造连续趋势；日里程趋势、多实例数据库队列/对象存储与更多可聚合指标仍待后续数据合同。
上线进度注记（2026-07-14）：`platform-v2-20260714033929` 已发布到当前 ECS `115.29.187.205:20300`。API 与告警 evaluator 均为 active，数据模式强制 `production`，MySQL/TDengine/Redis/容量检查已接入，Kafka lag 为 0；viewer/operator/admin 鉴权、越权拒绝、核心真实查询、根页面与 hashed 静态资源均已冒烟。告警 evaluator 当前扫描 1159 条车辆来源快照约 8–9ms，规则为 0，未产生生产告警；10k 车辆 × 20 规则的本地合成判定与批次规划约 20ms，候选写入已改为每 500 条一次批处理、活跃事件不重复写、超过 5 分钟的动态遥测不继续累积。首轮发布发现 Web 归档遗漏并在 API 冒烟后修复，runbook 已增加根页和主资源门禁。仍需补齐 ECS 10k 真实数据库写入基准、长时间稳定性、真实规则灰度及导出文件生产存储后，阶段 8/9 才可完全退出。
一期最终发布注记（2026-07-14）：当前 release 为 `platform-v2-20260714093533`，前一 `platform-v2-20260714092531` 保留可回滚。全部 11 项迁移 journal 校验通过；平台 API、快照 evaluator、流 evaluator 均 active，MySQL/TDengine 可写，9 个 Kafka 分区 lag 0、replay skipped 0，三单元最近发布窗口 error 日志均为 0。未鉴权会话返回 401，管理员会话、monitor、单车、active 告警、最新遥测、根页面与 hashed 主资源均通过真实生产冒烟；Web 22 个测试文件 272 项、API `go test ./...` 和 `go vet ./...` 全部通过。阶段 9 的单 ECS 一期退出项关闭；云数据库峰值容量和多实例对象存储作为后续扩容观察项。
导出进度注记（2026-07-14）：CSV 异步导出位于 `/opt/lingniu-vehicle-platform/data/exports` 持久目录，任务索引使用原子 `jobs.json`；完成文件和任务在 API 重启后仍可列出、下载，中断任务会显式失败而非永久 running。`platform-v2-20260714063725` 将旧的 1,000 行内存聚合/OFFSET/2 分钟实现升级为 100 万行流式游标任务。本地真实写入 1,000,000 行为 0.45 秒；ECS 候选位置 1,451 行与 RAW 3,538 行在单并发队列中 0.41 秒完成。正式生产 1,457 行、162,533 字节文件在 API 重启前后 SHA-256 完全一致，桌面和 390×844 任务卡无溢出、控制台无错误。当前满足单 ECS 一期受控导出；多实例扩容前仍需迁移到数据库队列与 OSS/兼容对象存储。
高级告警与指标目录进度注记（2026-07-14）：数值区间/区间外、布尔状态变化、OEM 范围、重复间隔、状态表和索引迁移已上线；ECS 以不存在厂家范围的两条规则完成启用灰度，evaluator 扫描 1159 条快照约 9ms、未误开事件，随后停用并保留版本审计。规则编辑器已通过生产浏览器检查。统一 `/api/v2/metrics` 目录现提供协议字段映射及可检索/绘图/告警能力，编辑器和后端写入均按目录校验。事件 INSERT 占位符、状态批写和重复边界已有自动化契约测试；布尔真实翻转、重复抑制与活跃规则长稳仍需独立生产灰度。
事件时间告警灰度注记（2026-07-14）：当前 ECS 对单车 `JT808` 建立 `speed_kmh >= 0`、持续 20 秒、重复间隔 1 小时的受限规则。真实新观测形成 candidate 后只打开 1 条事件，事件证据时间 `06:58:57.000`、接收时间 `06:58:57.934`、平台触发时间 `06:59:06.871`；事件随后关闭、通知设为已读、规则停用并保留审计。再次启用时 evaluator 记录 `candidates_advanced` 与 `duplicate_observations`，重复窗口内 `opened=0`。该门禁同时发现 MySQL `UNIX_TIMESTAMP(DATETIME(3))` 返回小数字符串导致候选及最近触发时间整数扫描失败，最终改为直接扫描 `DATETIME(3)` 并增加毫秒精度 SQL mock 契约。发布期间两次失败校验均由软链回滚陷阱恢复，最终 release 为 `platform-v2-20260714070507`。
指标配置持久化进度注记（2026-07-14）：`005_metric_catalog.sql` 已建立指标定义与协议源字段映射权威表；生产目录读取和告警规则校验共享该数据源，关闭或修改能力标记会立即约束后续规则写入。种子数据使用 `INSERT IGNORE`，发布不会回写覆盖运维配置；尚缺管理端 CRUD、版本审计以及把动态 RAW 指标纳入统一目录的受控发现流程。
告警写入性能注记（2026-07-14）：`alert-benchmark` 默认使用连接级 MySQL 临时表，同时提供必须显式确认且带 advisory lock 的独立物理表持久模式，两者都复制候选表主键与持续时间索引且绝不写业务候选/事件。当前 ECS 到生产 RDS 的持久单事务基准：10,000 行/500 批次为 195ms（约 51,234 行/秒），100,000 行为 1,971ms（约 50,713 行/秒）；两档行数和物理表清理均独立校验。10 万行窗口观察到实例全局 redo 增量约 31MB，但全局计数包含并发业务流量，不能当作本事务精确归因。当前结果证明真实提交、网络和索引路径具备明显余量，最终容量退出仍需 evaluator 活跃规则长稳、锁竞争和 RDS Performance Insights 峰值窗口证据。
车辆主数据进度注记（2026-07-14）：采用“网关权威身份 + 平台补充主档”的混合边界。`vehicle_profile` 不覆盖 VIN/车牌/厂家，专门保存车型、车辆类型、所属企业、运营状态、接入服务商、首次接入和累计运行时长；`vehicle_profile_audit`、乐观版本和 admin-only 写入保证可追溯。单车详情已合并完整度与缺失字段，管理员可在档案卡原位维护，只读角色仅查看。外部主数据同步 API 与 CSV 管理入口已实现：单批 500 辆、dry-run、来源/版本幂等、同版本变更拒绝、人工及异源默认保护、显式接管、网关身份校验和逐 VIN 结果；写入持久化 `source_system/source_version/synced_at` 并生成不可变同步审计。具体车厂/GPS 厂商的定时连接器仍需按其认证和字段合同配置。
主数据同步上线注记（2026-07-14）：`platform-v2-20260714051915` 已部署当前 ECS，API 与 evaluator 均为 active。管理员 dry-run、operator 403、页面分包和外网根页均已验证；生产可查询身份集合的 410 个唯一 VIN 在同一事务预演中耗时 175.44ms，重复预演 170.08ms，缺失身份为 0，抽样档案前后完全一致。上线验证未向生产写入虚构车型/企业数据；正式写入需由权威车厂/GPS 数据文件或连接器触发。
主数据告警范围进度注记（2026-07-14）：规则新增 `scopeModels/scopeCompanies`，评估器以 VIN 主键左联 `vehicle_profile`，未维护对应维度的车辆不会误匹配；协议、VIN、厂家、车型、企业范围均执行去重和大小上限，避免配置膨胀拖慢 10 秒评估周期。规则编辑器、版本快照和数据库持久化共享同一字段合同。
## 2. 阶段 0：必须先确认的决策
1. 车辆档案采用两者结合：网关身份表维护 VIN/车牌/厂家，平台 `vehicle_profile` 承载补充业务属性并预留外部同步来源元数据。
2. 认证是否复用 `ln-bi` 的 jumpToken/JWT 交换流程；角色和操作权限的权威来源是什么。
3. 动态指标目录的首批范围、中文名、单位、类型、协议字段映射及负责人。
4. 在线阈值是否采用“全局默认 + 协议覆盖”，建议默认 5 分钟并同时展示实际延迟。
5. 导出文件使用 OSS/兼容对象存储还是 ECS 本地盘，保留天数和单任务配额。
6. 告警时间基准使用事件时间，迟到数据容忍窗口和重复告警间隔的产品定义。
决策记录写 ADR，不把未确认事项固化进页面常量。
## 3. 阶段 1：数据与 API 基座
### 3.1 MySQL 迁移
- 扩充或关联车辆档案：车型、车辆类型、运营公司、运营状态、接入厂家、首次接入时间。
- 新建 `metric_definition`、`metric_protocol_mapping`。
- 新建告警域：`alert_rule`、`alert_rule_scope`、`alert_event`、`alert_event_action`、`notification`。
- 新建导出域：`export_job`，文件本体放对象/文件存储。
- 为所有业务表增加必要的状态、时间和组合索引；迁移只向前新增，不改写接入 writer 的现有主键语义。
### 3.2 BFF 契约
- 建立 `/api/v2`，保留旧只读接口作为过渡。
- 统一游标/offset 分页规则、排序白名单、时间格式、错误码、traceId 和最大查询范围。
- 生成 OpenAPI 与前端类型，避免 Go/TypeScript 手工重复漂移。
- 接入鉴权、操作权限、审计日志、限流、请求超时和幂等键。
### 3.3 验证
- 在生产结构副本/测试库运行 migration up/down 或等价回滚演练。
- 契约测试覆盖空值、未知协议、时区、分页边界和非法指标。
- 回归现有 Gateway、writers、realtime API 的写入与查询。
## 4. 阶段 2：V2 前端基础
建议新建独立入口/目录，而不是在 `PrototypeApp.tsx` 内渐进堆叠：
```text
src-v2/
├── app/ # router、providers、权限和菜单
├── shared/ # ui、api、map、chart、table、utils
├── entities/vehicle/ # 车辆领域类型、状态、展示组件
├── features/ # 筛选、车辆选择、时间范围、指标选择
├── pages/ # 路由页面
└── widgets/ # 地图工作区、详情面板、事件时间线
```
基础能力包括：正式路由、URL 查询恢复、认证门禁、统一 API client、查询缓存、错误边界、加载/空/错误状态、车辆状态 token、远程搜索选择器、服务端表格、ECharts 时序封装、AMap SDK。V2 默认禁止导入 `prototype/mockData.ts`。
## 5. 阶段 3 至 8：业务模块次序
每个模块遵循相同步骤：
1. 冻结页面信息架构和 DTO。
2. 先实现/验证后端接口和样本响应。
3. 实现页面及深链跳转。
4. 增加单元、契约和交互测试。
5. 用真实接口实际运行，检查加载、空、错误、权限和不同屏幕。
6. 更新已完成/未完成、接口和运维文档后再进入下一模块。
告警中心虽然菜单重要，但排在数据查询能力之后：规则依赖统一指标目录、车辆范围和时间语义。可在阶段 1 先建表/契约，阶段 8 再启用评估与通知。
## 6. 性能验收门槛
以下是建议基线，开发前应在目标 ECS/浏览器上校准：
| 场景 | 门槛 |
| --- | --- |
| 全局监控首屏 | 1,000 车 P95 API < 1s；10,000 车聚合结果而非 10,000 DOM Marker；主要交互保持流畅 |
| 地图拖动/缩放 | 只在 moveend/zoomend 防抖查询；旧请求可取消；无请求风暴 |
| 实时刷新 | 差量或低频批量刷新，不按车轮询；页面隐藏时降频/暂停 |
| 轨迹 | 单次返回受控；长时间段自动抽稀/分段；保留起终点、停车和告警关键点 |
| 历史曲线 | 浏览器展示点数设上限；TDengine 服务端聚合/抽稀；缩放后按窗口补查 |
| 表格 | 全部服务端分页/排序/筛选；长列表虚拟化 |
| 导出 | 异步任务；并发、行数、时间范围和文件大小有限额；不占用同步 API worker |
| 告警 | 消费失败不丢 offset；事件按规则+车辆+窗口幂等；可观测积压和评估延迟 |
## 7. 阶段 9：当前 ECS 发布计划
现有运维说明表明接入链路和平台都使用 systemd 裸机 release + `current` 软链。最终发布应继续该模式，但平台、告警 worker、导出 worker使用独立 unit 和健康检查。
发布顺序：
1. 备份并执行 MySQL 向前兼容迁移，确认现有服务继续健康。
2. 上传新 release，先启动告警/导出 worker 的 disabled 或 idle 模式。
3. 启动/切换平台 BFF，验证 `/readyz`、`/api/ops/health`、核心查询和运行时配置。
4. 切换 V2 静态 Web，进行全局监控、单车、轨迹、历史、接入、告警 smoke。
5. 开启 worker，小范围规则验证后再开放全部规则。
6. 检查 Kafka lag、Redis 在线 key、TDengine/MySQL 可写性、CPU/内存/磁盘和错误日志。
7. 保留前一 release 与数据库兼容窗口；失败时先回切软链并停新 worker，不做破坏性数据库回滚。
密钥放 `/opt/lingniu-vehicle-platform/env` 或对应 worker 环境文件，禁止写入 git、静态 JS 或部署日志。具体命令以实施时更新后的 runbook 为准，不能直接复制过时 release 名。
## 8. 一期完成清单
- 六大业务模块和导出任务满足目标文件的功能验收。
- 页面在 1440×900 正常，常用操作三次点击内，真实数据无 mock 残留。
- 1k/10k 地图、百万点历史、异步导出、告警积压均有测试记录。
- OpenAPI、数据字典、环境变量、启动、部署、回滚、监控与未完成事项完整。
- ECS 上全部新旧服务健康，Kafka lag 回零，运行版本可从健康接口确认。

1762
go/vehicle-gateway/cmd/capacity-check/main.go
View File

File diff suppressed because it is too large Load Diff

1119
go/vehicle-gateway/cmd/capacity-check/main_test.go
View File

File diff suppressed because it is too large Load Diff

490
go/vehicle-gateway/cmd/fields-projector/main.go Normal file
View File

@@ -0,0 +1,490 @@
package main
import (
"context"
"encoding/json"
"errors"
"fmt"
"log/slog"
"os"
"os/signal"
"strconv"
"strings"
"sync"
"syscall"
"time"
"github.com/segmentio/kafka-go"
"lingniu-vehicle-ingest/go/vehicle-gateway/internal/envelope"
"lingniu-vehicle-ingest/go/vehicle-gateway/internal/health"
"lingniu-vehicle-ingest/go/vehicle-gateway/internal/metrics"
"lingniu-vehicle-ingest/go/vehicle-gateway/internal/observability"
"lingniu-vehicle-ingest/go/vehicle-gateway/internal/realtime"
"lingniu-vehicle-ingest/go/vehicle-gateway/internal/topics"
)
func main() {
logger := observability.NewLogger("vehicle-fields-projector")
ctx, stop := signal.NotifyContext(context.Background(), os.Interrupt, syscall.SIGTERM)
defer stop()
cfg := loadConfig()
if err := cfg.Validate(); err != nil {
logger.Error("invalid fields projector config", "error", err)
os.Exit(1)
}
if err := pingKafka(ctx, cfg.KafkaBrokers); err != nil {
logger.Error("kafka connectivity check failed", "error", err)
os.Exit(1)
}
registry := metrics.NewRegistry()
recordConfigMetrics(registry, cfg)
for _, route := range cfg.Routes {
metrics.RegisterKafkaConsumerInfo(registry, "vehicle-fields-projector", route.GroupID, []string{route.RawTopic})
}
health.Start(ctx, logger, health.NewServer(env("HEALTH_ADDR", ""), "vehicle-fields-projector", nil, registry))
logger.Info("fields projector started",
"kafka_brokers", strings.Join(cfg.KafkaBrokers, ","),
"group_prefix", cfg.GroupPrefix,
"workers_per_protocol", cfg.WorkersPerProtocol,
"batch_size", cfg.BatchSize,
"batch_wait_ms", cfg.BatchWait.Milliseconds(),
"operation_timeout_ms", cfg.OperationTimeout.Milliseconds(),
"retry_delay_ms", cfg.RetryDelay.Milliseconds(),
"start_offset", cfg.StartOffsetName)
var workers sync.WaitGroup
for _, route := range cfg.Routes {
for workerID := 1; workerID <= cfg.WorkersPerProtocol; workerID++ {
workers.Add(1)
go func(route projectionRoute, workerID int) {
defer workers.Done()
runProjector(ctx, logger.With("protocol", route.Protocol, "worker", workerID), registry, cfg, route, workerID)
}(route, workerID)
}
}
workers.Wait()
}
type config struct {
KafkaBrokers []string
GroupPrefix string
Routes []projectionRoute
WorkersPerProtocol int
BatchSize int
BatchWait time.Duration
OperationTimeout time.Duration
RetryDelay time.Duration
StartOffset int64
StartOffsetName string
}
type projectionRoute struct {
Protocol envelope.Protocol
RawTopic string
FieldsTopic string
GroupID string
}
func loadConfig() config {
groupPrefix := env("FIELDS_PROJECTOR_GROUP_PREFIX", "vehicle-fields-projector-v1")
startOffsetName := strings.ToLower(env("KAFKA_START_OFFSET", "last"))
startOffset := int64(kafka.LastOffset)
if startOffsetName == "first" {
startOffset = kafka.FirstOffset
} else {
startOffsetName = "last"
}
routes := []projectionRoute{
{Protocol: envelope.ProtocolGB32960, RawTopic: env("KAFKA_TOPIC_GB32960_RAW", topics.RawGB32960), FieldsTopic: env("KAFKA_TOPIC_GB32960_FIELDS", topics.FieldsGB32960)},
{Protocol: envelope.ProtocolJT808, RawTopic: env("KAFKA_TOPIC_JT808_RAW", topics.RawJT808), FieldsTopic: env("KAFKA_TOPIC_JT808_FIELDS", topics.FieldsJT808)},
{Protocol: envelope.ProtocolYutongMQTT, RawTopic: env("KAFKA_TOPIC_YUTONG_MQTT_RAW", topics.RawYutongMQTT), FieldsTopic: env("KAFKA_TOPIC_YUTONG_MQTT_FIELDS", topics.FieldsYutongMQTT)},
}
for index := range routes {
routes[index].GroupID = groupPrefix + "-" + strings.ToLower(strings.ReplaceAll(string(routes[index].Protocol), "_", "-"))
}
return config{
KafkaBrokers: splitCSV(env("KAFKA_BROKERS", "127.0.0.1:9092")),
GroupPrefix: groupPrefix,
Routes: routes,
WorkersPerProtocol: envInt("FIELDS_PROJECTOR_WORKERS_PER_PROTOCOL", 1),
BatchSize: envInt("FIELDS_PROJECTOR_BATCH_SIZE", 500),
BatchWait: time.Duration(envInt("FIELDS_PROJECTOR_BATCH_WAIT_MS", 20)) * time.Millisecond,
OperationTimeout: time.Duration(envInt("FIELDS_PROJECTOR_OPERATION_TIMEOUT_MS", 30000)) * time.Millisecond,
RetryDelay: time.Duration(envInt("FIELDS_PROJECTOR_RETRY_DELAY_MS", 500)) * time.Millisecond,
StartOffset: startOffset,
StartOffsetName: startOffsetName,
}
}
func (c config) Validate() error {
if len(c.KafkaBrokers) == 0 {
return errors.New("kafka brokers are required")
}
if strings.TrimSpace(c.GroupPrefix) == "" {
return errors.New("fields projector group prefix is required")
}
if c.WorkersPerProtocol < 1 || c.BatchSize < 1 || c.BatchWait <= 0 || c.OperationTimeout <= 0 || c.RetryDelay <= 0 {
return errors.New("fields projector worker, batch and timeout settings must be positive")
}
raw := make(map[string]string, len(c.Routes))
fields := make(map[string]string, len(c.Routes))
groups := map[string]struct{}{}
for _, route := range c.Routes {
protocol := string(route.Protocol)
raw[protocol] = route.RawTopic
fields[protocol] = route.FieldsTopic
if strings.TrimSpace(route.GroupID) == "" {
return fmt.Errorf("consumer group is required for protocol %s", route.Protocol)
}
if _, exists := groups[route.GroupID]; exists {
return fmt.Errorf("duplicate consumer group %q", route.GroupID)
}
groups[route.GroupID] = struct{}{}
}
return topics.ValidateKafkaRawFields(raw, fields)
}
type kafkaBatchWriter interface {
WriteMessages(context.Context, ...kafka.Message) error
}
type kafkaMessageFetcher interface {
FetchMessage(context.Context) (kafka.Message, error)
}
type kafkaMessageCommitter interface {
CommitMessages(context.Context, ...kafka.Message) error
}
func runProjector(ctx context.Context, logger *slog.Logger, registry *metrics.Registry, cfg config, route projectionRoute, workerID int) {
reader := kafka.NewReader(kafka.ReaderConfig{
Brokers: cfg.KafkaBrokers,
GroupID: route.GroupID,
GroupTopics: []string{route.RawTopic},
StartOffset: cfg.StartOffset,
MinBytes: 1,
MaxBytes: 10e6,
})
defer reader.Close()
writer := &kafka.Writer{
Addr: kafka.TCP(cfg.KafkaBrokers...),
Balancer: &kafka.Hash{},
RequiredAcks: kafka.RequireAll,
AllowAutoTopicCreation: false,
BatchTimeout: cfg.BatchWait,
Async: false,
}
defer writer.Close()
labels := metrics.Labels{"protocol": string(route.Protocol), "worker": strconv.Itoa(workerID)}
registry.SetGauge("vehicle_fields_projector_worker_active", labels, 1)
defer registry.SetGauge("vehicle_fields_projector_worker_active", labels, 0)
for {
first, err := reader.FetchMessage(ctx)
if err != nil {
if ctx.Err() != nil {
return
}
logger.Error("kafka fetch failed", "error", err)
continue
}
batch := collectBatch(ctx, reader, first, cfg.BatchSize, cfg.BatchWait)
processBatchReliably(ctx, logger, registry, writer, reader, route, batch, cfg.OperationTimeout, cfg.RetryDelay)
}
}
func collectBatch(ctx context.Context, fetcher kafkaMessageFetcher, first kafka.Message, maxSize int, maxWait time.Duration) []kafka.Message {
if maxSize <= 1 {
return []kafka.Message{first}
}
batch := []kafka.Message{first}
deadline := time.Now().Add(maxWait)
for len(batch) < maxSize {
remaining := time.Until(deadline)
if remaining <= 0 {
break
}
fetchCtx, cancel := context.WithTimeout(ctx, remaining)
message, err := fetcher.FetchMessage(fetchCtx)
cancel()
if err != nil {
break
}
batch = append(batch, message)
}
return batch
}
func processBatchReliably(ctx context.Context, logger interface {
Error(string, ...any)
Warn(string, ...any)
}, registry *metrics.Registry, writer kafkaBatchWriter, committer kafkaMessageCommitter, route projectionRoute, messages []kafka.Message, operationTimeout time.Duration, retryDelay time.Duration) {
labels := metrics.Labels{"protocol": string(route.Protocol)}
defer registry.SetGauge("vehicle_fields_projector_retry_pending_messages", labels, 0)
for len(messages) > 0 {
operationCtx, cancel := context.WithTimeout(context.WithoutCancel(ctx), operationTimeout)
outputs, err := projectBatch(operationCtx, logger, registry, writer, route, messages)
cancel()
if err != nil {
registry.SetGauge("vehicle_fields_projector_retry_pending_messages", labels, float64(len(messages)))
registry.IncCounter("vehicle_fields_projector_batch_retries_total", metrics.Labels{"protocol": string(route.Protocol), "reason": "write_error"})
if !waitForRetry(ctx, retryDelay) {
return
}
continue
}
commitCtx, commitCancel := context.WithTimeout(context.WithoutCancel(ctx), operationTimeout)
err = committer.CommitMessages(commitCtx, messages...)
commitCancel()
if err == nil {
for _, message := range messages {
recordMessageMetric(registry, "vehicle_fields_projector_kafka_commits_total", message.Topic, "ok")
}
_ = outputs
return
}
for _, message := range messages {
recordMessageMetric(registry, "vehicle_fields_projector_kafka_commits_total", message.Topic, "error")
}
registry.SetGauge("vehicle_fields_projector_retry_pending_messages", labels, float64(len(messages)))
registry.IncCounter("vehicle_fields_projector_batch_retries_total", metrics.Labels{"protocol": string(route.Protocol), "reason": "commit_error"})
logger.Error("kafka source offset commit failed", "topic", route.RawTopic, "messages", len(messages), "error", err)
if !retryCommit(ctx, logger, registry, committer, messages, route.Protocol, operationTimeout, retryDelay) {
return
}
return
}
}
func projectBatch(ctx context.Context, logger interface {
Error(string, ...any)
Warn(string, ...any)
}, registry *metrics.Registry, writer kafkaBatchWriter, route projectionRoute, messages []kafka.Message) (int, error) {
if len(messages) == 0 {
return 0, nil
}
outputs := make([]kafka.Message, 0, len(messages))
setBatchPending(registry, route.Protocol, len(messages), 0)
defer setBatchPending(registry, route.Protocol, -len(messages), -len(outputs))
for _, message := range messages {
recordMessageMetric(registry, "vehicle_fields_projector_kafka_messages_total", message.Topic, "received")
recordKafkaLag(registry, message)
var raw envelope.FrameEnvelope
if err := json.Unmarshal(message.Value, &raw); err != nil {
recordMessageMetric(registry, "vehicle_fields_projector_kafka_messages_total", message.Topic, "invalid_json")
logger.Warn("skip invalid raw envelope json", "topic", message.Topic, "partition", message.Partition, "offset", message.Offset, "error", err)
continue
}
if status, err := topics.ValidateRawEnvelope(route.RawTopic, raw); err != nil {
recordMessageMetric(registry, "vehicle_fields_projector_kafka_messages_total", message.Topic, status)
logger.Warn("skip mismatched raw envelope", "topic", message.Topic, "partition", message.Partition, "offset", message.Offset, "protocol", raw.Protocol, "event_id", raw.StableEventID(), "error", err)
continue
}
fields, ok := realtime.BuildFieldsEnvelope(raw)
if !ok {
status := "skipped_missing_fields"
if !envelope.IsRealtimeTelemetryFrame(raw) {
status = "skipped_non_realtime"
}
recordProjectionMetric(registry, route.Protocol, status, 0)
continue
}
if status, err := topics.ValidateFieldsEnvelope(route.FieldsTopic, fields); err != nil {
recordProjectionMetric(registry, route.Protocol, status, len(fields.Fields))
logger.Warn("skip invalid projected fields envelope", "topic", route.FieldsTopic, "protocol", fields.Protocol, "event_id", fields.StableEventID(), "error", err)
continue
}
payload, err := fields.MarshalJSONBytes()
if err != nil {
recordProjectionMetric(registry, route.Protocol, "marshal_error", len(fields.Fields))
logger.Warn("skip fields envelope marshal error", "topic", route.FieldsTopic, "event_id", fields.StableEventID(), "error", err)
continue
}
outputs = append(outputs, kafka.Message{
Topic: route.FieldsTopic,
Key: fields.KafkaKey(),
Value: payload,
Time: message.Time,
})
recordProjectionMetric(registry, route.Protocol, "projected", len(fields.Fields))
}
setBatchPending(registry, route.Protocol, 0, len(outputs))
defer setBatchPending(registry, route.Protocol, 0, -len(outputs))
if len(outputs) == 0 {
return 0, nil
}
started := time.Now()
err := writer.WriteMessages(ctx, outputs...)
status := "ok"
if err != nil {
status = "error"
}
recordWriteDuration(registry, route.FieldsTopic, status, time.Since(started))
for range outputs {
recordMessageMetric(registry, "vehicle_fields_projector_kafka_writes_total", route.FieldsTopic, status)
}
if err != nil {
logger.Error("fields kafka write failed", "topic", route.FieldsTopic, "messages", len(outputs), "error", err)
return len(outputs), err
}
return len(outputs), nil
}
func retryCommit(ctx context.Context, logger interface {
Error(string, ...any)
}, registry *metrics.Registry, committer kafkaMessageCommitter, messages []kafka.Message, protocol envelope.Protocol, operationTimeout, retryDelay time.Duration) bool {
for {
if !waitForRetry(ctx, retryDelay) {
return false
}
operationCtx, cancel := context.WithTimeout(context.WithoutCancel(ctx), operationTimeout)
err := committer.CommitMessages(operationCtx, messages...)
cancel()
if err == nil {
for _, message := range messages {
recordMessageMetric(registry, "vehicle_fields_projector_kafka_commits_total", message.Topic, "ok")
}
return true
}
for _, message := range messages {
recordMessageMetric(registry, "vehicle_fields_projector_kafka_commits_total", message.Topic, "error")
}
registry.IncCounter("vehicle_fields_projector_batch_retries_total", metrics.Labels{"protocol": string(protocol), "reason": "commit_error"})
logger.Error("kafka source offset commit retry failed", "messages", len(messages), "error", err)
}
}
func waitForRetry(ctx context.Context, delay time.Duration) bool {
timer := time.NewTimer(delay)
defer timer.Stop()
select {
case <-ctx.Done():
return false
case <-timer.C:
return true
}
}
func pingKafka(ctx context.Context, brokers []string) error {
if len(brokers) == 0 {
return errors.New("kafka broker is required")
}
checkCtx, cancel := context.WithTimeout(ctx, 5*time.Second)
defer cancel()
conn, err := kafka.DialContext(checkCtx, "tcp", brokers[0])
if err != nil {
return err
}
return conn.Close()
}
func recordConfigMetrics(registry *metrics.Registry, cfg config) {
registry.SetGauge("vehicle_fields_projector_config", metrics.Labels{"setting": "workers_per_protocol"}, float64(cfg.WorkersPerProtocol))
registry.SetGauge("vehicle_fields_projector_config", metrics.Labels{"setting": "batch_size"}, float64(cfg.BatchSize))
registry.SetGauge("vehicle_fields_projector_config", metrics.Labels{"setting": "batch_wait_ms"}, float64(cfg.BatchWait.Milliseconds()))
registry.SetGauge("vehicle_fields_projector_config", metrics.Labels{"setting": "operation_timeout_ms"}, float64(cfg.OperationTimeout.Milliseconds()))
}
func recordMessageMetric(registry *metrics.Registry, name, topic, status string) {
if registry == nil {
return
}
labels := metrics.Labels{"topic": topic, "status": status}
registry.IncCounter(name, labels)
switch name {
case "vehicle_fields_projector_kafka_messages_total":
metrics.RecordLastActivity(registry, "vehicle_fields_projector_last_message_unix_seconds", labels)
case "vehicle_fields_projector_kafka_writes_total":
metrics.RecordLastActivity(registry, "vehicle_fields_projector_last_write_unix_seconds", labels)
case "vehicle_fields_projector_kafka_commits_total":
metrics.RecordLastActivity(registry, "vehicle_fields_projector_last_commit_unix_seconds", labels)
}
}
func recordProjectionMetric(registry *metrics.Registry, protocol envelope.Protocol, status string, fieldCount int) {
if registry == nil {
return
}
labels := metrics.Labels{"protocol": string(protocol), "status": status}
registry.IncCounter("vehicle_fields_projector_projections_total", labels)
if fieldCount > 0 {
registry.SetGauge("vehicle_fields_projector_field_count", labels, float64(fieldCount))
}
}
func recordKafkaLag(registry *metrics.Registry, message kafka.Message) {
if registry == nil {
return
}
lag := message.HighWaterMark - message.Offset - 1
if lag < 0 {
lag = 0
}
registry.SetGauge("vehicle_fields_projector_kafka_lag", metrics.Labels{
"topic": message.Topic, "partition": strconv.Itoa(message.Partition),
}, float64(lag))
}
var projectorPendingMu sync.Mutex
var projectorPendingMessages = map[envelope.Protocol]int{}
var projectorPendingFields = map[envelope.Protocol]int{}
func setBatchPending(registry *metrics.Registry, protocol envelope.Protocol, messagesDelta, fieldsDelta int) {
if registry == nil {
return
}
projectorPendingMu.Lock()
projectorPendingMessages[protocol] += messagesDelta
projectorPendingFields[protocol] += fieldsDelta
messages := projectorPendingMessages[protocol]
fields := projectorPendingFields[protocol]
projectorPendingMu.Unlock()
labels := metrics.Labels{"protocol": string(protocol)}
registry.SetGauge("vehicle_fields_projector_batch_pending_messages", labels, float64(messages))
registry.SetGauge("vehicle_fields_projector_batch_pending_fields", labels, float64(fields))
}
var projectorWriteBucketsMS = []float64{1, 5, 10, 25, 50, 100, 250, 500, 1000, 5000}
func recordWriteDuration(registry *metrics.Registry, topic, status string, elapsed time.Duration) {
if registry == nil {
return
}
registry.ObserveHistogram("vehicle_fields_projector_write_duration_ms_histogram", metrics.Labels{
"topic": topic, "status": status,
}, projectorWriteBucketsMS, float64(elapsed.Milliseconds()))
}
func env(key, fallback string) string {
value := strings.TrimSpace(os.Getenv(key))
if value == "" {
return fallback
}
return value
}
func envInt(key string, fallback int) int {
value := strings.TrimSpace(os.Getenv(key))
if value == "" {
return fallback
}
parsed, err := strconv.Atoi(value)
if err != nil {
return fallback
}
return parsed
}
func splitCSV(value string) []string {
var out []string
for _, item := range strings.Split(value, ",") {
item = strings.TrimSpace(item)
if item != "" {
out = append(out, item)
}
}
return out
}

242
go/vehicle-gateway/cmd/fields-projector/main_test.go Normal file
View File

@@ -0,0 +1,242 @@
package main
import (
"context"
"encoding/json"
"errors"
"strings"
"sync"
"testing"
"time"
"github.com/segmentio/kafka-go"
"lingniu-vehicle-ingest/go/vehicle-gateway/internal/envelope"
"lingniu-vehicle-ingest/go/vehicle-gateway/internal/metrics"
"lingniu-vehicle-ingest/go/vehicle-gateway/internal/topics"
)
func TestLoadConfigCreatesProtocolIsolatedConsumerGroups(t *testing.T) {
t.Setenv("FIELDS_PROJECTOR_GROUP_PREFIX", "projector-test")
t.Setenv("KAFKA_START_OFFSET", "first")
cfg := loadConfig()
if cfg.StartOffset != kafka.FirstOffset || cfg.StartOffsetName != "first" {
t.Fatalf("start offset = %d/%q", cfg.StartOffset, cfg.StartOffsetName)
}
wantGroups := map[envelope.Protocol]string{
envelope.ProtocolGB32960: "projector-test-gb32960",
envelope.ProtocolJT808: "projector-test-jt808",
envelope.ProtocolYutongMQTT: "projector-test-yutong-mqtt",
}
for _, route := range cfg.Routes {
if route.GroupID != wantGroups[route.Protocol] {
t.Fatalf("group for %s = %q", route.Protocol, route.GroupID)
}
}
if err := cfg.Validate(); err != nil {
t.Fatalf("Validate() error = %v", err)
}
}
func TestProjectBatchUsesPrecomputedFieldsAndPreservesSourceMetadata(t *testing.T) {
raw := projectorRawEnvelope()
payload, err := raw.MarshalJSONBytes()
if err != nil {
t.Fatalf("marshal raw: %v", err)
}
writer := &recordingProjectorWriter{}
registry := metrics.NewRegistry()
route := jt808ProjectionRoute()
count, err := projectBatch(context.Background(), discardProjectorLogger{}, registry, writer, route, []kafka.Message{{
Topic: route.RawTopic, Key: raw.KafkaKey(), Value: payload, Partition: 2, Offset: 10, HighWaterMark: 11,
}})
if err != nil {
t.Fatalf("projectBatch() error = %v", err)
}
if count != 1 || writer.callCount() != 1 || len(writer.messages) != 1 {
t.Fatalf("projected=%d calls=%d messages=%d", count, writer.callCount(), len(writer.messages))
}
var fields envelope.FrameEnvelope
if err := json.Unmarshal(writer.messages[0].Value, &fields); err != nil {
t.Fatalf("decode fields: %v", err)
}
if fields.EventKind != envelope.EventKindFields || fields.SourceEventID != raw.EventID || fields.EventID != raw.EventID+":fields" {
t.Fatalf("fields identity = %#v", fields)
}
if fields.SourceCode != raw.SourceCode || fields.SourceKind != raw.SourceKind || fields.SourceEndpoint != raw.SourceEndpoint {
t.Fatalf("source metadata not preserved: %#v", fields)
}
if got := fields.Fields["jt808.location.total_mileage_km"]; got != 1234.5 {
t.Fatalf("total mileage = %#v", got)
}
if len(fields.Parsed) != 0 || len(fields.ParsedFields) != 0 {
t.Fatalf("fields projection must not duplicate raw payload: parsed=%v parsed_fields=%v", fields.Parsed, fields.ParsedFields)
}
text := registry.Render()
for _, want := range []string{
`vehicle_fields_projector_projections_total{protocol="JT808",status="projected"} 1`,
`vehicle_fields_projector_kafka_writes_total{status="ok",topic="vehicle.fields.go.jt808.v1"} 1`,
`vehicle_fields_projector_kafka_lag{partition="2",topic="vehicle.raw.go.jt808.v1"} 0`,
} {
if !strings.Contains(text, want) {
t.Fatalf("metric missing %q:\n%s", want, text)
}
}
}
func TestProcessBatchReliablySkipsNonRealtimeAndInvalidWithoutWriting(t *testing.T) {
route := jt808ProjectionRoute()
nonRealtime := projectorRawEnvelope()
nonRealtime.MessageID = "0x0002"
nonRealtime.ParsedFields = map[string]any{"jt808.header.message_id": "0x0002"}
payload, _ := nonRealtime.MarshalJSONBytes()
messages := []kafka.Message{
{Topic: route.RawTopic, Value: []byte("{bad"), Partition: 0, Offset: 1, HighWaterMark: 3},
{Topic: route.RawTopic, Value: payload, Partition: 0, Offset: 2, HighWaterMark: 3},
}
writer := &recordingProjectorWriter{}
committer := &recordingProjectorCommitter{}
registry := metrics.NewRegistry()
processBatchReliably(context.Background(), discardProjectorLogger{}, registry, writer, committer, route, messages, time.Second, time.Millisecond)
if writer.callCount() != 0 {
t.Fatalf("writer calls = %d, want 0", writer.callCount())
}
if committer.callCount() != 1 || committer.messageCount != 2 {
t.Fatalf("commit calls/messages = %d/%d", committer.callCount(), committer.messageCount)
}
text := registry.Render()
if !strings.Contains(text, `vehicle_fields_projector_kafka_messages_total{status="invalid_json",topic="vehicle.raw.go.jt808.v1"} 1`) ||
!strings.Contains(text, `vehicle_fields_projector_projections_total{protocol="JT808",status="skipped_non_realtime"} 1`) {
t.Fatalf("skip metrics missing:\n%s", text)
}
}
func TestProcessBatchReliablyRetriesWriteBeforeCommitting(t *testing.T) {
route := jt808ProjectionRoute()
payload, _ := projectorRawEnvelope().MarshalJSONBytes()
messages := []kafka.Message{{Topic: route.RawTopic, Value: payload, Partition: 1, Offset: 7, HighWaterMark: 8}}
writer := &recordingProjectorWriter{errors: []error{errors.New("kafka unavailable"), nil}}
committer := &recordingProjectorCommitter{}
registry := metrics.NewRegistry()
processBatchReliably(context.Background(), discardProjectorLogger{}, registry, writer, committer, route, messages, time.Second, time.Millisecond)
if writer.callCount() != 2 {
t.Fatalf("writer calls = %d, want 2", writer.callCount())
}
if committer.callCount() != 1 {
t.Fatalf("commit calls = %d, want 1 after successful write", committer.callCount())
}
if !strings.Contains(registry.Render(), `vehicle_fields_projector_batch_retries_total{protocol="JT808",reason="write_error"} 1`) {
t.Fatalf("write retry metric missing:\n%s", registry.Render())
}
}
func TestProcessBatchReliablyRetriesOnlyCommitAfterSuccessfulWrite(t *testing.T) {
route := jt808ProjectionRoute()
payload, _ := projectorRawEnvelope().MarshalJSONBytes()
messages := []kafka.Message{{Topic: route.RawTopic, Value: payload, Partition: 1, Offset: 7, HighWaterMark: 8}}
writer := &recordingProjectorWriter{}
committer := &recordingProjectorCommitter{errors: []error{errors.New("commit timeout"), nil}}
processBatchReliably(context.Background(), discardProjectorLogger{}, metrics.NewRegistry(), writer, committer, route, messages, time.Second, time.Millisecond)
if writer.callCount() != 1 {
t.Fatalf("writer calls = %d, want 1", writer.callCount())
}
if committer.callCount() != 2 {
t.Fatalf("commit calls = %d, want 2", committer.callCount())
}
}
func TestProjectBatchWriteFailureDoesNotCommitByItself(t *testing.T) {
route := jt808ProjectionRoute()
payload, _ := projectorRawEnvelope().MarshalJSONBytes()
writer := &recordingProjectorWriter{errors: []error{errors.New("write failed")}}
count, err := projectBatch(context.Background(), discardProjectorLogger{}, nil, writer, route, []kafka.Message{{Topic: route.RawTopic, Value: payload}})
if err == nil || count != 1 {
t.Fatalf("projectBatch() count/error = %d/%v", count, err)
}
}
func projectorRawEnvelope() envelope.FrameEnvelope {
return envelope.FrameEnvelope{
EventID: "raw-event-1",
EventKind: envelope.EventKindRaw,
Protocol: envelope.ProtocolJT808,
MessageID: "0x0200",
VIN: "VIN001",
Phone: "13307795425",
SourceEndpoint: "115.231.168.135:43625",
SourceCode: "g7s",
SourceKind: "PLATFORM",
PlatformName: "G7s",
EventTimeMS: 1783960000000,
ReceivedAtMS: 1783960000010,
ParsedFields: map[string]any{
"jt808.location.latitude": 30.1,
"jt808.location.longitude": 121.2,
"jt808.location.total_mileage_km": 1234.5,
},
ParseStatus: envelope.ParseOK,
}
}
func jt808ProjectionRoute() projectionRoute {
return projectionRoute{
Protocol: envelope.ProtocolJT808, RawTopic: topics.RawJT808, FieldsTopic: topics.FieldsJT808, GroupID: "projector-jt808",
}
}
type recordingProjectorWriter struct {
mu sync.Mutex
errors []error
calls int
messages []kafka.Message
}
func (w *recordingProjectorWriter) WriteMessages(_ context.Context, messages ...kafka.Message) error {
w.mu.Lock()
defer w.mu.Unlock()
w.calls++
w.messages = append(w.messages, messages...)
if len(w.errors) == 0 {
return nil
}
err := w.errors[0]
w.errors = w.errors[1:]
return err
}
func (w *recordingProjectorWriter) callCount() int {
w.mu.Lock()
defer w.mu.Unlock()
return w.calls
}
type recordingProjectorCommitter struct {
mu sync.Mutex
errors []error
calls int
messageCount int
}
func (c *recordingProjectorCommitter) CommitMessages(_ context.Context, messages ...kafka.Message) error {
c.mu.Lock()
defer c.mu.Unlock()
c.calls++
c.messageCount += len(messages)
if len(c.errors) == 0 {
return nil
}
err := c.errors[0]
c.errors = c.errors[1:]
return err
}
func (c *recordingProjectorCommitter) callCount() int {
c.mu.Lock()
defer c.mu.Unlock()
return c.calls
}
type discardProjectorLogger struct{}
func (discardProjectorLogger) Error(string, ...any) {}
func (discardProjectorLogger) Warn(string, ...any) {}

589
go/vehicle-gateway/cmd/gateway/main.go
View File

@@ -3,6 +3,8 @@ package main
import (
"context"
"database/sql"
"encoding/json"
"fmt"
"log/slog"
"os"
"os/signal"
@@ -13,6 +15,7 @@ import (
_ "github.com/go-sql-driver/mysql"
"lingniu-vehicle-ingest/go/vehicle-gateway/internal/authentication"
"lingniu-vehicle-ingest/go/vehicle-gateway/internal/envelope"
"lingniu-vehicle-ingest/go/vehicle-gateway/internal/eventbus"
"lingniu-vehicle-ingest/go/vehicle-gateway/internal/gateway"
@@ -37,7 +40,7 @@ func main() {
os.Exit(1)
}
defer sink.Close()
resolver, closeResolver, err := buildIdentityResolver(ctx, logger)
resolver, jt808DeviceTokens, closeResolver, err := buildIdentityResolver(ctx, logger, registry)
if err != nil {
logger.Error("build identity resolver failed", "error", err)
os.Exit(1)
@@ -45,21 +48,38 @@ func main() {
defer closeResolver()
health.Start(ctx, logger, health.NewServer(env("HEALTH_ADDR", ""), "vehicle-gateway", nil, registry))
publishUnified := envBool("PUBLISH_UNIFIED_ENABLED", false)
delegateFields := envBool("FIELDS_DERIVE_FROM_RAW_ENABLED", strings.TrimSpace(os.Getenv("NATS_URL")) != "")
gb32960Authenticator, gb32960AuthMode, gb32960CredentialCount, err := buildGB32960Authenticator()
if err != nil {
logger.Error("build gb32960 authenticator failed", "error", err)
os.Exit(1)
}
jt808AuthCode := env("JT808_REGISTER_AUTH_CODE", "g7gps")
jt808Authenticator, jt808AuthMode, err := buildJT808Authenticator(jt808AuthCode, jt808DeviceTokens)
if err != nil {
logger.Error("build jt808 authenticator failed", "error", err)
os.Exit(1)
}
recordAuthenticationConfig(registry, envelope.ProtocolGB32960, gb32960AuthMode, gb32960CredentialCount)
recordAuthenticationConfig(registry, envelope.ProtocolJT808, jt808AuthMode, int(boolMetric(strings.TrimSpace(jt808AuthCode) != "")))
logger.Info("protocol authentication configured", "gb32960_mode", gb32960AuthMode, "gb32960_accounts", gb32960CredentialCount, "jt808_mode", jt808AuthMode)
protocols := []gateway.TCPProtocol{
{
Protocol: envelope.ProtocolGB32960,
Addr: env("GB32960_TCP_ADDR", ":32960"),
Extract: gb32960.ExtractFrames,
Parse: gb32960.ParseFrame,
Respond: gb32960.AutoResponse,
Protocol: envelope.ProtocolGB32960,
Addr: env("GB32960_TCP_ADDR", ":32960"),
Extract: gb32960.ExtractFrames,
Parse: gb32960.ParseFrame,
Authenticate: gb32960Authenticator,
Respond: gb32960.AutoResponse,
},
{
Protocol: envelope.ProtocolJT808,
Addr: env("JT808_TCP_ADDR", ":808"),
Extract: jt808.ExtractFrames,
Parse: jt808.ParseFrame,
Respond: jt808.NewAutoResponder(env("JT808_REGISTER_AUTH_CODE", "g7gps")).Respond,
Protocol: envelope.ProtocolJT808,
Addr: env("JT808_TCP_ADDR", ":808"),
Extract: jt808.ExtractFrames,
Parse: jt808.ParseFrame,
Authenticate: jt808Authenticator,
Respond: jt808.NewAutoResponder(jt808AuthCode).Respond,
},
}
@@ -76,6 +96,7 @@ func main() {
IdleTimeout: time.Duration(envInt("TCP_IDLE_TIMEOUT_SECONDS", 180)) * time.Second,
MaxConnections: envInt("TCP_MAX_CONNECTIONS", 120_000),
PublishUnified: publishUnified,
DelegateFields: delegateFields,
})
if err != nil {
logger.Error("build tcp server failed", "protocol", protocol.Protocol, "error", err)
@@ -110,6 +131,7 @@ func main() {
Logger: logger,
Metrics: registry,
PublishUnified: publishUnified,
DelegateFields: delegateFields,
})
if err != nil {
logger.Error("build yutong mqtt client failed", "error", err)
@@ -122,7 +144,7 @@ func main() {
logger.Info("yutong mqtt client started")
}
logger.Info("vehicle gateway started")
logger.Info("vehicle gateway started", "fields_derive_from_raw_enabled", delegateFields)
select {
case <-ctx.Done():
case err := <-errs:
@@ -132,63 +154,433 @@ func main() {
wg.Wait()
}
func buildIdentityResolver(ctx context.Context, logger *slog.Logger) (identity.Resolver, func(), error) {
func buildGB32960Authenticator() (authentication.Authenticator, authentication.Mode, int, error) {
mode, err := authentication.ParseMode(os.Getenv("GB32960_AUTH_MODE"), authentication.ModeObserve)
if err != nil {
return nil, "", 0, err
}
credentials, err := loadGB32960Credentials()
if err != nil {
return nil, "", 0, err
}
if mode == authentication.ModeEnforce && len(credentials) == 0 {
return nil, "", 0, fmt.Errorf("GB32960_AUTH_MODE=enforce requires configured platform credentials")
}
return authentication.NewGB32960PlatformAuthenticator(mode, credentials), mode, len(credentials), nil
}
func buildJT808Authenticator(authCode string, deviceTokens authentication.JT808DeviceTokenProvider) (authentication.Authenticator, authentication.Mode, error) {
mode, err := authentication.ParseMode(os.Getenv("JT808_AUTH_MODE"), authentication.ModeObserve)
if err != nil {
return nil, "", err
}
if mode == authentication.ModeEnforce && strings.TrimSpace(authCode) == "" && deviceTokens == nil {
return nil, "", fmt.Errorf("JT808_AUTH_MODE=enforce requires a configured or device token provider")
}
return authentication.NewJT808Authenticator(mode, authCode, deviceTokens), mode, nil
}
func loadGB32960Credentials() (map[string][]string, error) {
payload := strings.TrimSpace(os.Getenv("GB32960_PLATFORM_CREDENTIALS_JSON"))
if path := strings.TrimSpace(os.Getenv("GB32960_PLATFORM_CREDENTIALS_FILE")); path != "" {
contents, err := os.ReadFile(path)
if err != nil {
return nil, fmt.Errorf("read gb32960 credentials file: %w", err)
}
payload = strings.TrimSpace(string(contents))
}
if payload == "" {
return map[string][]string{}, nil
}
rawCredentials := map[string]json.RawMessage{}
if err := json.Unmarshal([]byte(payload), &rawCredentials); err != nil {
return nil, fmt.Errorf("parse gb32960 platform credentials: %w", err)
}
credentials := make(map[string][]string, len(rawCredentials))
for username, rawPassword := range rawCredentials {
trimmed := strings.TrimSpace(username)
if trimmed == "" {
continue
}
var passwords []string
var single string
if err := json.Unmarshal(rawPassword, &single); err == nil {
passwords = []string{single}
} else if err := json.Unmarshal(rawPassword, &passwords); err != nil {
return nil, fmt.Errorf("parse gb32960 credentials for account %q: expected string or string array", trimmed)
}
for _, password := range passwords {
if password != "" {
credentials[trimmed] = append(credentials[trimmed], password)
}
}
if len(credentials[trimmed]) == 0 {
delete(credentials, trimmed)
}
}
return credentials, nil
}
func recordAuthenticationConfig(registry *metrics.Registry, protocol envelope.Protocol, mode authentication.Mode, credentialCount int) {
if registry == nil {
return
}
registry.SetGauge("vehicle_gateway_authentication_mode", metrics.Labels{
"protocol": string(protocol),
"mode": string(mode),
}, 1)
registry.SetGauge("vehicle_gateway_authentication_credentials", metrics.Labels{
"protocol": string(protocol),
}, float64(credentialCount))
}
func buildIdentityResolver(ctx context.Context, logger *slog.Logger, registry *metrics.Registry) (identity.Resolver, authentication.JT808DeviceTokenProvider, func(), error) {
dsn := env("IDENTITY_MYSQL_DSN", strings.TrimSpace(os.Getenv("MYSQL_DSN")))
if strings.TrimSpace(dsn) == "" {
logger.Warn("identity mysql dsn is empty; using noop identity resolver")
return identity.NoopResolver{}, func() {}, nil
return identity.NoopResolver{}, nil, func() {}, nil
}
db, err := sql.Open("mysql", dsn)
if err != nil {
return nil, nil, err
}
if err := db.PingContext(ctx); err != nil {
_ = db.Close()
return nil, nil, err
logger.Error("identity mysql configuration failed; continuing with unresolved identities", "error", err)
return identity.NoopResolver{}, nil, func() {}, nil
}
db.SetMaxOpenConns(envInt("IDENTITY_MYSQL_MAX_OPEN_CONNS", 16))
db.SetMaxIdleConns(envInt("IDENTITY_MYSQL_MAX_IDLE_CONNS", 8))
db.SetConnMaxLifetime(time.Duration(envInt("IDENTITY_MYSQL_CONN_MAX_LIFETIME_SECONDS", 300)) * time.Second)
db.SetConnMaxIdleTime(time.Duration(envInt("IDENTITY_MYSQL_CONN_MAX_IDLE_SECONDS", 60)) * time.Second)
table := env("VEHICLE_IDENTITY_TABLE", "vehicle_identity_binding")
cacheMaxEntries := envInt("IDENTITY_LOOKUP_CACHE_MAX_ENTRIES", 300000)
cacheCleanupInterval := time.Duration(envInt("IDENTITY_LOOKUP_CACHE_CLEANUP_INTERVAL_SECONDS", 60)) * time.Second
staleLookupTTLSeconds := envInt("IDENTITY_STALE_LOOKUP_TTL_SECONDS", 3600)
registrationGatewayWritesEnabled := envBool("JT808_REGISTRATION_GATEWAY_WRITES_ENABLED", false)
resolver := identity.NewMySQLResolverWithOptions(db, table, identity.MySQLResolverOptions{
LocationTouchInterval: time.Duration(envInt("JT808_REGISTRATION_LOCATION_TOUCH_INTERVAL_SECONDS", 600)) * time.Second,
LookupCacheTTL: time.Duration(envInt("IDENTITY_LOOKUP_CACHE_TTL_SECONDS", 600)) * time.Second,
SnapshotOnlyLookups: envBool("IDENTITY_SNAPSHOT_ONLY_ENABLED", true),
LocationTouchInterval: time.Duration(envInt("JT808_REGISTRATION_LOCATION_TOUCH_INTERVAL_SECONDS", 600)) * time.Second,
LocationTouchRetryInterval: time.Duration(envInt("JT808_REGISTRATION_LOCATION_TOUCH_RETRY_INTERVAL_SECONDS", 5)) * time.Second,
RegistrationWriteAttempts: envInt("JT808_REGISTRATION_WRITE_RETRY_ATTEMPTS", 2),
RegistrationWriteRetryDelay: time.Duration(envInt("JT808_REGISTRATION_WRITE_RETRY_DELAY_MS", 20)) * time.Millisecond,
LookupCacheTTL: time.Duration(envInt("IDENTITY_LOOKUP_CACHE_TTL_SECONDS", 600)) * time.Second,
StaleLookupTTL: time.Duration(staleLookupTTLSeconds) * time.Second,
CacheCleanupInterval: cacheCleanupInterval,
MaxCacheEntries: cacheMaxEntries,
SourceCodeLookup: envBool("IDENTITY_SOURCE_CODE_LOOKUP_ENABLED", true),
AsyncRegistrationWrites: envBool("JT808_REGISTRATION_ASYNC_WRITE_ENABLED", true),
RegistrationWriteQueueSize: envInt("JT808_REGISTRATION_WRITE_QUEUE_SIZE", 100000),
RegistrationWriteWorkers: envInt("JT808_REGISTRATION_WRITE_WORKERS", 4),
RegistrationWriteTimeout: time.Duration(envInt("JT808_REGISTRATION_WRITE_TIMEOUT_MS", 5000)) * time.Millisecond,
RegistrationEnqueueTimeout: time.Duration(envInt("JT808_REGISTRATION_WRITE_ENQUEUE_TIMEOUT_MS", 50)) * time.Millisecond,
DisableRegistrationWrites: !registrationGatewayWritesEnabled,
OnRegistrationWriteResult: func(result identity.RegistrationWriteResult) {
recordJT808RegistrationWriteResult(registry, result)
},
OnRegistrationWriteError: func(err error) {
logger.Warn("jt808 registration async write failed", "error", err)
},
})
if envBool("IDENTITY_MYSQL_ENSURE_SCHEMA", true) {
if err := resolver.EnsureSchema(ctx); err != nil {
_ = db.Close()
return nil, nil, err
}
startIdentityDatabaseMaintenance(
ctx,
logger,
db,
resolver,
envBool("IDENTITY_MYSQL_ENSURE_SCHEMA", false),
time.Duration(envInt("IDENTITY_MYSQL_PING_TIMEOUT_MS", 3000))*time.Millisecond,
time.Duration(envInt("IDENTITY_MYSQL_SCHEMA_TIMEOUT_SECONDS", 10))*time.Second,
)
startIdentitySnapshotRefresh(
ctx,
logger,
registry,
resolver,
time.Duration(envInt("IDENTITY_SNAPSHOT_REFRESH_INTERVAL_SECONDS", 60))*time.Second,
time.Duration(envInt("IDENTITY_SNAPSHOT_REFRESH_TIMEOUT_SECONDS", 10))*time.Second,
)
startIdentityCacheMetrics(ctx, registry, resolver, time.Duration(envInt("IDENTITY_CACHE_METRICS_INTERVAL_SECONDS", 30))*time.Second)
resolveTimeout := time.Duration(envInt("IDENTITY_RESOLVE_TIMEOUT_MS", 50)) * time.Millisecond
registry.SetGauge("vehicle_gateway_jt808_registration_gateway_writes_enabled", nil, boolMetric(registrationGatewayWritesEnabled))
logger.Info("identity mysql resolver enabled", "table", table, "snapshot_only_enabled", envBool("IDENTITY_SNAPSHOT_ONLY_ENABLED", true), "snapshot_refresh_interval_seconds", envInt("IDENTITY_SNAPSHOT_REFRESH_INTERVAL_SECONDS", 60), "lookup_cache_ttl_seconds", envInt("IDENTITY_LOOKUP_CACHE_TTL_SECONDS", 600), "stale_lookup_ttl_seconds", staleLookupTTLSeconds, "lookup_cache_max_entries", cacheMaxEntries, "lookup_cache_cleanup_interval_seconds", cacheCleanupInterval.Seconds(), "source_code_lookup_enabled", envBool("IDENTITY_SOURCE_CODE_LOOKUP_ENABLED", true), "resolve_timeout_ms", resolveTimeout.Milliseconds(), "jt808_registration_gateway_writes_enabled", registrationGatewayWritesEnabled, "jt808_registration_location_touch_retry_interval_seconds", envInt("JT808_REGISTRATION_LOCATION_TOUCH_RETRY_INTERVAL_SECONDS", 5), "jt808_registration_write_retry_attempts", envInt("JT808_REGISTRATION_WRITE_RETRY_ATTEMPTS", 2), "jt808_registration_write_retry_delay_ms", envInt("JT808_REGISTRATION_WRITE_RETRY_DELAY_MS", 20), "jt808_registration_async_write_enabled", envBool("JT808_REGISTRATION_ASYNC_WRITE_ENABLED", true), "jt808_registration_write_queue_size", envInt("JT808_REGISTRATION_WRITE_QUEUE_SIZE", 100000), "jt808_registration_write_workers", envInt("JT808_REGISTRATION_WRITE_WORKERS", 4), "jt808_registration_write_timeout_ms", envInt("JT808_REGISTRATION_WRITE_TIMEOUT_MS", 5000), "jt808_registration_write_enqueue_timeout_ms", envInt("JT808_REGISTRATION_WRITE_ENQUEUE_TIMEOUT_MS", 50))
return identity.TimeoutResolver{Delegate: resolver, Timeout: resolveTimeout}, resolver, func() {
_ = resolver.Close()
_ = db.Close()
}, nil
}
type identitySnapshotRefresher interface {
RefreshSnapshot(context.Context) (identity.SnapshotRefreshResult, error)
}
type identityDatabasePinger interface {
PingContext(context.Context) error
}
type identitySchemaEnsurer interface {
EnsureSchema(context.Context) error
}
func startIdentityDatabaseMaintenance(ctx context.Context, logger *slog.Logger, db identityDatabasePinger, schema identitySchemaEnsurer, ensureSchema bool, pingTimeout time.Duration, schemaTimeout time.Duration) {
if db == nil {
return
}
logger.Info("identity mysql resolver enabled", "table", table, "lookup_cache_ttl_seconds", envInt("IDENTITY_LOOKUP_CACHE_TTL_SECONDS", 600))
return resolver, func() { _ = db.Close() }, nil
if pingTimeout <= 0 {
pingTimeout = 3 * time.Second
}
if schemaTimeout <= 0 {
schemaTimeout = 10 * time.Second
}
go func() {
pingCtx, cancelPing := context.WithTimeout(ctx, pingTimeout)
err := db.PingContext(pingCtx)
cancelPing()
if err != nil {
logger.Warn("identity mysql unavailable; ingress remains enabled and snapshot refresh will retry", "error", err)
return
}
if !ensureSchema || schema == nil {
return
}
schemaCtx, cancelSchema := context.WithTimeout(ctx, schemaTimeout)
err = schema.EnsureSchema(schemaCtx)
cancelSchema()
if err != nil {
logger.Warn("identity mysql schema check failed; ingress remains enabled", "error", err)
}
}()
}
func startIdentitySnapshotRefresh(ctx context.Context, logger *slog.Logger, registry *metrics.Registry, refresher identitySnapshotRefresher, interval time.Duration, timeout time.Duration) {
if refresher == nil {
return
}
if interval <= 0 {
interval = time.Minute
}
if timeout <= 0 {
timeout = 10 * time.Second
}
refresh := func() {
refreshCtx, cancel := context.WithTimeout(ctx, timeout)
result, err := refresher.RefreshSnapshot(refreshCtx)
cancel()
if err != nil {
recordIdentitySnapshotRefresh(registry, identity.SnapshotRefreshResult{}, "error")
logger.Warn("identity snapshot refresh failed; keeping last known good snapshot", "error", err)
return
}
recordIdentitySnapshotRefresh(registry, result, "ok")
logger.Info("identity snapshot refreshed", "binding_entries", result.BindingEntries, "identifier_entries", result.IdentifierEntries, "registration_entries", result.RegistrationEntries, "source_entries", result.SourceEntries)
}
go func() {
refresh()
ticker := time.NewTicker(interval)
defer ticker.Stop()
for {
select {
case <-ctx.Done():
return
case <-ticker.C:
refresh()
}
}
}()
}
func recordIdentitySnapshotRefresh(registry *metrics.Registry, result identity.SnapshotRefreshResult, status string) {
if registry == nil {
return
}
registry.IncCounter("vehicle_gateway_identity_snapshot_refresh_total", metrics.Labels{"status": status})
if status != "ok" {
return
}
for _, item := range []struct {
kind string
value int
}{
{kind: "binding", value: result.BindingEntries},
{kind: "identifier", value: result.IdentifierEntries},
{kind: "registration", value: result.RegistrationEntries},
{kind: "source", value: result.SourceEntries},
} {
registry.SetGauge("vehicle_gateway_identity_snapshot_entries", metrics.Labels{"kind": item.kind}, float64(item.value))
}
registry.SetGauge("vehicle_gateway_identity_snapshot_last_success_unix_seconds", nil, float64(result.RefreshedAt.Unix()))
}
type identityCacheStatsReporter interface {
CacheStats() identity.CacheStats
}
func startIdentityCacheMetrics(ctx context.Context, registry *metrics.Registry, reporter identityCacheStatsReporter, interval time.Duration) {
if registry == nil || reporter == nil {
return
}
if interval <= 0 {
interval = 30 * time.Second
}
recordIdentityCacheStats(registry, reporter.CacheStats())
go func() {
ticker := time.NewTicker(interval)
defer ticker.Stop()
for {
select {
case <-ctx.Done():
return
case <-ticker.C:
recordIdentityCacheStats(registry, reporter.CacheStats())
}
}
}()
}
func recordIdentityCacheStats(registry *metrics.Registry, stats identity.CacheStats) {
if registry == nil {
return
}
for _, item := range []struct {
name string
value int
max int
}{
{name: "lookup", value: stats.LookupEntries, max: stats.MaxEntries},
{name: "registration", value: stats.RegistrationEntries, max: stats.MaxEntries},
{name: "source_code", value: stats.SourceCodeEntries, max: stats.MaxEntries},
{name: "location_touch", value: stats.LocationTouchEntries, max: stats.MaxEntries},
{name: "location_touch_failure", value: stats.LocationTouchFailureEntries, max: stats.MaxEntries},
{name: "registration_write_queue", value: stats.RegistrationWriteQueueDepth, max: stats.RegistrationWriteQueueCap},
{name: "snapshot_binding", value: stats.SnapshotBindingEntries, max: stats.MaxEntries},
{name: "snapshot_identifier", value: stats.SnapshotIdentifierEntries, max: stats.MaxEntries},
{name: "snapshot_registration", value: stats.SnapshotRegistrationEntries, max: stats.MaxEntries},
{name: "snapshot_source", value: stats.SnapshotSourceEntries, max: stats.MaxEntries},
} {
registry.SetGauge("vehicle_gateway_identity_cache_entries", metrics.Labels{"cache": item.name}, float64(item.value))
registry.SetGauge("vehicle_gateway_identity_cache_max_entries", metrics.Labels{"cache": item.name}, float64(item.max))
}
ready := 0.0
if stats.SnapshotReady {
ready = 1
}
registry.SetGauge("vehicle_gateway_identity_snapshot_ready", nil, ready)
if !stats.SnapshotRefreshedAt.IsZero() {
registry.SetGauge("vehicle_gateway_identity_snapshot_last_success_unix_seconds", nil, float64(stats.SnapshotRefreshedAt.Unix()))
}
}
func recordJT808RegistrationWriteResult(registry *metrics.Registry, result identity.RegistrationWriteResult) {
if registry == nil {
return
}
mode := strings.TrimSpace(result.Mode)
if mode == "" {
mode = "unknown"
}
status := strings.TrimSpace(result.Status)
if status == "" {
status = "unknown"
}
registry.IncCounter("vehicle_gateway_jt808_registration_write_total", metrics.Labels{
"mode": mode,
"status": status,
})
metrics.RecordLastActivity(registry, "vehicle_gateway_last_jt808_registration_write_unix_seconds", metrics.Labels{
"mode": mode,
"status": status,
})
}
func buildSink(ctx context.Context, logger *slog.Logger, registry *metrics.Registry) (eventbus.Sink, error) {
if strings.TrimSpace(os.Getenv("NATS_URL")) != "" {
sink, err := eventbus.NewNATSSink(natsSinkConfigFromEnv())
natsConfig := natsSinkConfigFromEnv()
sink, err := eventbus.NewNATSSink(natsConfig)
if err != nil {
return nil, err
}
outboxRuntime := natsOutboxConfigFromEnv(registry, func(err error) {
logger.Warn("nats durable outbox publish failed", "error", err)
})
if outboxRuntime.Enabled {
outbox, err := eventbus.NewDurableOutboxSink(sink, outboxRuntime.Config)
if err != nil {
_ = sink.Close()
return nil, err
}
go outbox.ReplayLoop(ctx, outboxRuntime.ReplayInterval)
logger.Info("nats durable outbox enabled",
"dir", outboxRuntime.Config.Directory,
"fsync", outboxRuntime.Config.SyncWrites,
"replay_interval_ms", outboxRuntime.ReplayInterval.Milliseconds(),
"replay_batch_size", outboxRuntime.Config.ReplayBatchSize,
"close_timeout_ms", outboxRuntime.Config.CloseTimeout.Milliseconds(),
"wal_segment_bytes", outboxRuntime.Config.WALSegmentBytes,
"wal_segment_age_ms", outboxRuntime.Config.WALSegmentAge.Milliseconds(),
"wal_append_queue_size", outboxRuntime.Config.WALAppendQueue,
"wal_commit_batch_size", outboxRuntime.Config.WALCommitBatch,
"wal_commit_interval_ms", outboxRuntime.Config.WALCommitWait.Milliseconds(),
"max_inflight", natsConfig.AsyncMaxPending,
"ack_timeout_ms", natsConfig.AsyncAckTimeout.Milliseconds(),
)
return outbox, nil
}
var out eventbus.Sink = eventbus.NewRetryingSink(sink, eventbus.RetryConfig{
Attempts: envInt("NATS_PUBLISH_ATTEMPTS", 3),
Backoff: time.Duration(envInt("NATS_PUBLISH_BACKOFF_MS", 100)) * time.Millisecond,
AttemptTimeout: time.Duration(envInt("NATS_PUBLISH_TIMEOUT_MS", 3000)) * time.Millisecond,
})
spoolDir := strings.TrimSpace(os.Getenv("NATS_SPOOL_DIR"))
if spoolDir != "" {
replayBatchSize := envInt("NATS_SPOOL_REPLAY_BATCH_SIZE", 200)
durable := eventbus.NewDurableSink(out, eventbus.DurableConfig{
Directory: spoolDir,
ReplayBatchSize: replayBatchSize,
Metrics: registry,
Name: "nats",
})
interval := time.Duration(envInt("NATS_SPOOL_REPLAY_INTERVAL_MS", 1000)) * time.Millisecond
go durable.ReplayLoop(ctx, interval, func(err error) {
logger.Warn("nats spool replay failed", "error", err)
})
logger.Info("nats durable spool enabled", "dir", spoolDir, "replay_interval_ms", interval.Milliseconds(), "replay_batch_size", replayBatchSize)
out = durable
}
if envBool("NATS_ASYNC_ENABLED", true) {
queueSize := envInt("NATS_ASYNC_QUEUE_SIZE", envInt("KAFKA_ASYNC_QUEUE_SIZE", 100000))
workers := envInt("NATS_ASYNC_WORKERS", envInt("KAFKA_ASYNC_WORKERS", 8))
enqueueTimeout := time.Duration(envInt("NATS_ASYNC_ENQUEUE_TIMEOUT_MS", envInt("KAFKA_ASYNC_ENQUEUE_TIMEOUT_MS", 1000))) * time.Millisecond
timeout := time.Duration(envInt("NATS_ASYNC_PUBLISH_TIMEOUT_MS", envInt("KAFKA_ASYNC_PUBLISH_TIMEOUT_MS", 30000))) * time.Millisecond
out = eventbus.NewAsyncSink(out, eventbus.AsyncConfig{
QueueSize: queueSize,
Workers: workers,
OperationTimeout: timeout,
Metrics: registry,
Name: "nats",
OnError: func(err error) {
logger.Warn("nats async publish failed", "error", err)
},
})
logger.Info("nats async publish enabled", "queue_size", queueSize, "workers", workers, "publish_timeout_ms", timeout.Milliseconds())
if envBool("NATS_PARTITIONED_ASYNC_ENABLED", true) {
rawQueueSize, derivedQueueSize, rawWorkers, derivedWorkers := partitionedAsyncConfigFromEnv("NATS", queueSize, workers)
rawEnqueueTimeout, derivedEnqueueTimeout := partitionedAsyncEnqueueTimeoutsFromEnv("NATS", enqueueTimeout)
out = eventbus.NewPartitionedAsyncSink(out, eventbus.PartitionedAsyncConfig{
RawQueueSize: rawQueueSize,
DerivedQueueSize: derivedQueueSize,
RawWorkers: rawWorkers,
DerivedWorkers: derivedWorkers,
RawEnqueueTimeout: rawEnqueueTimeout,
DerivedEnqueueTimeout: derivedEnqueueTimeout,
EnqueueTimeout: enqueueTimeout,
OperationTimeout: timeout,
Metrics: registry,
Name: "nats",
OnError: func(err error) {
logger.Warn("nats partitioned async publish failed", "error", err)
},
})
logger.Info("nats partitioned async publish enabled", "raw_queue_size", rawQueueSize, "derived_queue_size", derivedQueueSize, "raw_workers", rawWorkers, "derived_workers", derivedWorkers, "raw_enqueue_timeout_ms", rawEnqueueTimeout.Milliseconds(), "derived_enqueue_timeout_ms", derivedEnqueueTimeout.Milliseconds(), "publish_timeout_ms", timeout.Milliseconds())
} else {
out = eventbus.NewAsyncSink(out, eventbus.AsyncConfig{
QueueSize: queueSize,
Workers: workers,
EnqueueTimeout: enqueueTimeout,
OperationTimeout: timeout,
Metrics: registry,
Name: "nats",
OnError: func(err error) {
logger.Warn("nats async publish failed", "error", err)
},
})
logger.Info("nats async publish enabled", "queue_size", queueSize, "workers", workers, "enqueue_timeout_ms", enqueueTimeout.Milliseconds(), "publish_timeout_ms", timeout.Milliseconds())
}
}
logger.Info("nats jetstream sink enabled", "url", env("NATS_URL", ""), "unified_subject", natsSinkConfigFromEnv().UnifiedSubject, "publish_unified_enabled", envBool("PUBLISH_UNIFIED_ENABLED", false), "publish_fields_enabled", true)
logger.Info("nats jetstream sink enabled", "url", env("NATS_URL", ""), "unified_subject", natsSinkConfigFromEnv().UnifiedSubject, "publish_unified_enabled", envBool("PUBLISH_UNIFIED_ENABLED", false), "publish_fields_enabled", !envBool("FIELDS_DERIVE_FROM_RAW_ENABLED", true))
return out, nil
}
brokers := splitCSV(os.Getenv("KAFKA_BROKERS"))
@@ -221,7 +613,12 @@ func buildSink(ctx context.Context, logger *slog.Logger, registry *metrics.Regis
spoolDir := strings.TrimSpace(os.Getenv("KAFKA_SPOOL_DIR"))
if spoolDir != "" {
replayBatchSize := envInt("KAFKA_SPOOL_REPLAY_BATCH_SIZE", 200)
durable := eventbus.NewDurableSink(out, eventbus.DurableConfig{Directory: spoolDir, ReplayBatchSize: replayBatchSize})
durable := eventbus.NewDurableSink(out, eventbus.DurableConfig{
Directory: spoolDir,
ReplayBatchSize: replayBatchSize,
Metrics: registry,
Name: "kafka",
})
interval := time.Duration(envInt("KAFKA_SPOOL_REPLAY_INTERVAL_MS", 1000)) * time.Millisecond
go durable.ReplayLoop(ctx, interval, func(err error) {
logger.Warn("kafka spool replay failed", "error", err)
@@ -232,26 +629,73 @@ func buildSink(ctx context.Context, logger *slog.Logger, registry *metrics.Regis
if envBool("KAFKA_ASYNC_ENABLED", true) {
queueSize := envInt("KAFKA_ASYNC_QUEUE_SIZE", 100000)
workers := envInt("KAFKA_ASYNC_WORKERS", 8)
enqueueTimeout := time.Duration(envInt("KAFKA_ASYNC_ENQUEUE_TIMEOUT_MS", 1000)) * time.Millisecond
timeout := time.Duration(envInt("KAFKA_ASYNC_PUBLISH_TIMEOUT_MS", 30000)) * time.Millisecond
out = eventbus.NewAsyncSink(out, eventbus.AsyncConfig{
QueueSize: queueSize,
Workers: workers,
OperationTimeout: timeout,
Metrics: registry,
Name: "kafka",
OnError: func(err error) {
logger.Warn("kafka async publish failed", "error", err)
},
})
logger.Info("kafka async publish enabled", "queue_size", queueSize, "workers", workers, "publish_timeout_ms", timeout.Milliseconds())
if envBool("KAFKA_PARTITIONED_ASYNC_ENABLED", true) {
rawQueueSize, derivedQueueSize, rawWorkers, derivedWorkers := partitionedAsyncConfigFromEnv("KAFKA", queueSize, workers)
rawEnqueueTimeout, derivedEnqueueTimeout := partitionedAsyncEnqueueTimeoutsFromEnv("KAFKA", enqueueTimeout)
out = eventbus.NewPartitionedAsyncSink(out, eventbus.PartitionedAsyncConfig{
RawQueueSize: rawQueueSize,
DerivedQueueSize: derivedQueueSize,
RawWorkers: rawWorkers,
DerivedWorkers: derivedWorkers,
RawEnqueueTimeout: rawEnqueueTimeout,
DerivedEnqueueTimeout: derivedEnqueueTimeout,
EnqueueTimeout: enqueueTimeout,
OperationTimeout: timeout,
Metrics: registry,
Name: "kafka",
OnError: func(err error) {
logger.Warn("kafka partitioned async publish failed", "error", err)
},
})
logger.Info("kafka partitioned async publish enabled", "raw_queue_size", rawQueueSize, "derived_queue_size", derivedQueueSize, "raw_workers", rawWorkers, "derived_workers", derivedWorkers, "raw_enqueue_timeout_ms", rawEnqueueTimeout.Milliseconds(), "derived_enqueue_timeout_ms", derivedEnqueueTimeout.Milliseconds(), "publish_timeout_ms", timeout.Milliseconds())
} else {
out = eventbus.NewAsyncSink(out, eventbus.AsyncConfig{
QueueSize: queueSize,
Workers: workers,
EnqueueTimeout: enqueueTimeout,
OperationTimeout: timeout,
Metrics: registry,
Name: "kafka",
OnError: func(err error) {
logger.Warn("kafka async publish failed", "error", err)
},
})
logger.Info("kafka async publish enabled", "queue_size", queueSize, "workers", workers, "enqueue_timeout_ms", enqueueTimeout.Milliseconds(), "publish_timeout_ms", timeout.Milliseconds())
}
}
return out, nil
}
func partitionedAsyncConfigFromEnv(prefix string, queueSize int, workers int) (rawQueueSize int, derivedQueueSize int, rawWorkers int, derivedWorkers int) {
derivedQueueDefault := queueSize / 2
if derivedQueueDefault <= 0 {
derivedQueueDefault = 1
}
derivedWorkersDefault := workers / 2
if derivedWorkersDefault <= 0 {
derivedWorkersDefault = 1
}
rawQueueSize = envInt(prefix+"_ASYNC_RAW_QUEUE_SIZE", queueSize)
derivedQueueSize = envInt(prefix+"_ASYNC_DERIVED_QUEUE_SIZE", derivedQueueDefault)
rawWorkers = envInt(prefix+"_ASYNC_RAW_WORKERS", workers)
derivedWorkers = envInt(prefix+"_ASYNC_DERIVED_WORKERS", derivedWorkersDefault)
return rawQueueSize, derivedQueueSize, rawWorkers, derivedWorkers
}
func partitionedAsyncEnqueueTimeoutsFromEnv(prefix string, enqueueTimeout time.Duration) (rawEnqueueTimeout time.Duration, derivedEnqueueTimeout time.Duration) {
rawEnqueueTimeout = time.Duration(envInt(prefix+"_ASYNC_RAW_ENQUEUE_TIMEOUT_MS", int(enqueueTimeout/time.Millisecond))) * time.Millisecond
derivedEnqueueTimeout = time.Duration(envInt(prefix+"_ASYNC_DERIVED_ENQUEUE_TIMEOUT_MS", 50)) * time.Millisecond
return rawEnqueueTimeout, derivedEnqueueTimeout
}
func natsSinkConfigFromEnv() eventbus.NATSConfig {
return eventbus.NATSConfig{
URL: env("NATS_URL", ""),
Name: env("NATS_CLIENT_NAME", "lingniu-vehicle-gateway"),
URL: env("NATS_URL", ""),
Name: env("NATS_CLIENT_NAME", "lingniu-vehicle-gateway"),
AsyncMaxPending: envInt("NATS_OUTBOX_MAX_INFLIGHT", 10_000),
AsyncAckTimeout: time.Duration(envInt("NATS_OUTBOX_ACK_TIMEOUT_MS", 3000)) * time.Millisecond,
RawSubjects: map[envelope.Protocol]string{
envelope.ProtocolGB32960: env("NATS_SUBJECT_GB32960_RAW", env("KAFKA_TOPIC_GB32960_RAW", topics.RawGB32960)),
envelope.ProtocolJT808: env("NATS_SUBJECT_JT808_RAW", env("KAFKA_TOPIC_JT808_RAW", topics.RawJT808)),
@@ -266,6 +710,34 @@ func natsSinkConfigFromEnv() eventbus.NATSConfig {
}
}
type natsOutboxRuntimeConfig struct {
Enabled bool
Config eventbus.DurableOutboxConfig
ReplayInterval time.Duration
}
func natsOutboxConfigFromEnv(registry *metrics.Registry, onError func(error)) natsOutboxRuntimeConfig {
directory := strings.TrimSpace(os.Getenv("NATS_OUTBOX_DIR"))
return natsOutboxRuntimeConfig{
Enabled: envBool("NATS_DURABLE_OUTBOX_ENABLED", false),
Config: eventbus.DurableOutboxConfig{
Directory: directory,
ReplayBatchSize: envInt("NATS_OUTBOX_REPLAY_BATCH_SIZE", 1000),
SyncWrites: envBool("NATS_OUTBOX_FSYNC", true),
CloseTimeout: time.Duration(envInt("NATS_OUTBOX_CLOSE_TIMEOUT_MS", 5000)) * time.Millisecond,
WALSegmentBytes: int64(envInt("NATS_OUTBOX_WAL_SEGMENT_BYTES", 16<<20)),
WALSegmentAge: time.Duration(envInt("NATS_OUTBOX_WAL_SEGMENT_AGE_MS", 5000)) * time.Millisecond,
WALAppendQueue: envInt("NATS_OUTBOX_WAL_APPEND_QUEUE_SIZE", 100_000),
WALCommitBatch: envInt("NATS_OUTBOX_WAL_COMMIT_BATCH_SIZE", 256),
WALCommitWait: time.Duration(envInt("NATS_OUTBOX_WAL_COMMIT_INTERVAL_MS", 1)) * time.Millisecond,
Metrics: registry,
Name: "nats-outbox",
OnError: onError,
},
ReplayInterval: time.Duration(envInt("NATS_OUTBOX_REPLAY_INTERVAL_MS", 1000)) * time.Millisecond,
}
}
func env(key string, fallback string) string {
value := strings.TrimSpace(os.Getenv(key))
if value == "" {
@@ -300,6 +772,13 @@ func envBool(key string, fallback bool) bool {
return value == "1" || value == "true" || value == "yes" || value == "on"
}
func boolMetric(value bool) float64 {
if value {
return 1
}
return 0
}
func splitCSV(value string) []string {
var out []string
for _, item := range strings.Split(value, ",") {

340
go/vehicle-gateway/cmd/gateway/main_test.go
View File

@@ -1,11 +1,19 @@
package main
import (
"context"
"errors"
"io"
"log/slog"
"os"
"strings"
"testing"
"time"
"lingniu-vehicle-ingest/go/vehicle-gateway/internal/authentication"
"lingniu-vehicle-ingest/go/vehicle-gateway/internal/envelope"
"lingniu-vehicle-ingest/go/vehicle-gateway/internal/identity"
"lingniu-vehicle-ingest/go/vehicle-gateway/internal/metrics"
)
func TestNATSSinkConfigFromEnvUsesExplicitSubjects(t *testing.T) {
@@ -13,7 +21,12 @@ func TestNATSSinkConfigFromEnvUsesExplicitSubjects(t *testing.T) {
t.Setenv("NATS_SUBJECT_GB32960_RAW", "custom.raw.gb32960")
t.Setenv("NATS_SUBJECT_JT808_RAW", "custom.raw.jt808")
t.Setenv("NATS_SUBJECT_YUTONG_MQTT_RAW", "custom.raw.yutong")
t.Setenv("NATS_SUBJECT_GB32960_FIELDS", "custom.fields.gb32960")
t.Setenv("NATS_SUBJECT_JT808_FIELDS", "custom.fields.jt808")
t.Setenv("NATS_SUBJECT_YUTONG_MQTT_FIELDS", "custom.fields.yutong")
t.Setenv("NATS_SUBJECT_UNIFIED", "custom.unified")
t.Setenv("NATS_OUTBOX_MAX_INFLIGHT", "12000")
t.Setenv("NATS_OUTBOX_ACK_TIMEOUT_MS", "4500")
cfg := natsSinkConfigFromEnv()
@@ -29,9 +42,131 @@ func TestNATSSinkConfigFromEnvUsesExplicitSubjects(t *testing.T) {
if got, want := cfg.RawSubjects[envelope.ProtocolYutongMQTT], "custom.raw.yutong"; got != want {
t.Fatalf("yutong subject = %q, want %q", got, want)
}
if got, want := cfg.FieldsSubjects[envelope.ProtocolGB32960], "custom.fields.gb32960"; got != want {
t.Fatalf("gb32960 fields subject = %q, want %q", got, want)
}
if got, want := cfg.FieldsSubjects[envelope.ProtocolJT808], "custom.fields.jt808"; got != want {
t.Fatalf("jt808 fields subject = %q, want %q", got, want)
}
if got, want := cfg.FieldsSubjects[envelope.ProtocolYutongMQTT], "custom.fields.yutong"; got != want {
t.Fatalf("yutong fields subject = %q, want %q", got, want)
}
if got, want := cfg.UnifiedSubject, "custom.unified"; got != want {
t.Fatalf("unified subject = %q, want %q", got, want)
}
if got, want := cfg.AsyncMaxPending, 12000; got != want {
t.Fatalf("async max pending = %d, want %d", got, want)
}
if got, want := cfg.AsyncAckTimeout, 4500*time.Millisecond; got != want {
t.Fatalf("async ack timeout = %s, want %s", got, want)
}
}
func TestNATSOutboxConfigFromEnvIsExplicitAndDefaultsToSafeFsync(t *testing.T) {
t.Setenv("NATS_DURABLE_OUTBOX_ENABLED", "true")
t.Setenv("NATS_OUTBOX_DIR", "/var/lib/lingniu-go/nats-outbox")
t.Setenv("NATS_OUTBOX_REPLAY_BATCH_SIZE", "750")
t.Setenv("NATS_OUTBOX_REPLAY_INTERVAL_MS", "250")
t.Setenv("NATS_OUTBOX_CLOSE_TIMEOUT_MS", "7000")
t.Setenv("NATS_OUTBOX_WAL_SEGMENT_BYTES", "8388608")
t.Setenv("NATS_OUTBOX_WAL_SEGMENT_AGE_MS", "4000")
t.Setenv("NATS_OUTBOX_WAL_APPEND_QUEUE_SIZE", "50000")
t.Setenv("NATS_OUTBOX_WAL_COMMIT_BATCH_SIZE", "128")
t.Setenv("NATS_OUTBOX_WAL_COMMIT_INTERVAL_MS", "2")
t.Setenv("NATS_OUTBOX_FSYNC", "")
registry := metrics.NewRegistry()
onError := func(error) {}
runtime := natsOutboxConfigFromEnv(registry, onError)
if !runtime.Enabled {
t.Fatal("outbox should be enabled")
}
if got, want := runtime.Config.Directory, "/var/lib/lingniu-go/nats-outbox"; got != want {
t.Fatalf("directory = %q, want %q", got, want)
}
if got, want := runtime.Config.ReplayBatchSize, 750; got != want {
t.Fatalf("replay batch = %d, want %d", got, want)
}
if !runtime.Config.SyncWrites {
t.Fatal("outbox fsync should default to true")
}
if got, want := runtime.Config.CloseTimeout, 7*time.Second; got != want {
t.Fatalf("close timeout = %s, want %s", got, want)
}
if got, want := runtime.Config.WALSegmentBytes, int64(8<<20); got != want {
t.Fatalf("WAL segment bytes = %d, want %d", got, want)
}
if got, want := runtime.Config.WALSegmentAge, 4*time.Second; got != want {
t.Fatalf("WAL segment age = %s, want %s", got, want)
}
if got, want := runtime.Config.WALAppendQueue, 50_000; got != want {
t.Fatalf("WAL append queue = %d, want %d", got, want)
}
if got, want := runtime.Config.WALCommitBatch, 128; got != want {
t.Fatalf("WAL commit batch = %d, want %d", got, want)
}
if got, want := runtime.Config.WALCommitWait, 2*time.Millisecond; got != want {
t.Fatalf("WAL commit wait = %s, want %s", got, want)
}
if got, want := runtime.ReplayInterval, 250*time.Millisecond; got != want {
t.Fatalf("replay interval = %s, want %s", got, want)
}
if runtime.Config.Metrics != registry || runtime.Config.OnError == nil {
t.Fatal("metrics and error callback should be wired")
}
}
func TestNATSOutboxConfigDoesNotReuseLegacySpoolDirectory(t *testing.T) {
t.Setenv("NATS_DURABLE_OUTBOX_ENABLED", "")
t.Setenv("NATS_OUTBOX_DIR", "")
t.Setenv("NATS_SPOOL_DIR", "/var/lib/lingniu-go/nats-spool")
t.Setenv("NATS_OUTBOX_FSYNC", "false")
runtime := natsOutboxConfigFromEnv(nil, nil)
if runtime.Enabled {
t.Fatal("outbox must remain opt-in")
}
if runtime.Config.Directory != "" {
t.Fatalf("WAL directory must be explicit, got %q", runtime.Config.Directory)
}
if runtime.Config.SyncWrites {
t.Fatal("explicit false should disable fsync")
}
}
func TestBuildGB32960AuthenticatorUsesConfiguredCredentials(t *testing.T) {
t.Setenv("GB32960_AUTH_MODE", "enforce")
t.Setenv("GB32960_PLATFORM_CREDENTIALS_FILE", "")
t.Setenv("GB32960_PLATFORM_CREDENTIALS_JSON", `{"platform-a":"secret-a","platform-b":["secret-b-old","secret-b"]}`)
authenticator, mode, count, err := buildGB32960Authenticator()
if err != nil {
t.Fatalf("buildGB32960Authenticator() error = %v", err)
}
if mode != authentication.ModeEnforce || count != 2 {
t.Fatalf("mode=%q count=%d", mode, count)
}
result := authenticator.Authenticate(envelope.FrameEnvelope{
Protocol: envelope.ProtocolGB32960,
MessageID: "0x05",
Parsed: map[string]any{
"platform_login": map[string]any{"username": "platform-b", "password": "secret-b"},
},
})
if !result.Allowed || result.Status != authentication.StatusAccepted {
t.Fatalf("authentication result = %#v", result)
}
}
func TestBuildGB32960AuthenticatorRejectsEnforceWithoutCredentials(t *testing.T) {
t.Setenv("GB32960_AUTH_MODE", "enforce")
t.Setenv("GB32960_PLATFORM_CREDENTIALS_FILE", "")
t.Setenv("GB32960_PLATFORM_CREDENTIALS_JSON", "")
if _, _, _, err := buildGB32960Authenticator(); err == nil {
t.Fatal("expected enforce mode configuration error")
}
}
func TestGatewayConfiguresIdentityLookupCacheTTL(t *testing.T) {
@@ -39,13 +174,158 @@ func TestGatewayConfiguresIdentityLookupCacheTTL(t *testing.T) {
if err != nil {
t.Fatalf("read main.go: %v", err)
}
for _, want := range []string{"LookupCacheTTL", "IDENTITY_LOOKUP_CACHE_TTL_SECONDS"} {
for _, want := range []string{"LookupCacheTTL", "StaleLookupTTL", "IDENTITY_LOOKUP_CACHE_TTL_SECONDS", "IDENTITY_STALE_LOOKUP_TTL_SECONDS", "IDENTITY_LOOKUP_CACHE_MAX_ENTRIES", "IDENTITY_LOOKUP_CACHE_CLEANUP_INTERVAL_SECONDS", "vehicle_gateway_identity_cache_entries", "vehicle_gateway_jt808_registration_write_total", "OnRegistrationWriteResult", "IDENTITY_SOURCE_CODE_LOOKUP_ENABLED", "IDENTITY_RESOLVE_TIMEOUT_MS", "IDENTITY_SNAPSHOT_ONLY_ENABLED", "IDENTITY_SNAPSHOT_REFRESH_INTERVAL_SECONDS", "startIdentitySnapshotRefresh", "JT808_REGISTRATION_LOCATION_TOUCH_RETRY_INTERVAL_SECONDS", "JT808_REGISTRATION_WRITE_RETRY_ATTEMPTS", "JT808_REGISTRATION_WRITE_RETRY_DELAY_MS", "JT808_REGISTRATION_ASYNC_WRITE_ENABLED", "JT808_REGISTRATION_WRITE_QUEUE_SIZE", "JT808_REGISTRATION_WRITE_WORKERS", "JT808_REGISTRATION_WRITE_ENQUEUE_TIMEOUT_MS", "TimeoutResolver"} {
if !strings.Contains(string(source), want) {
t.Fatalf("gateway should expose identity lookup cache ttl, missing %s", want)
}
}
}
func TestRecordIdentityCacheStatsIncludesLocationTouchFailures(t *testing.T) {
registry := metrics.NewRegistry()
recordIdentityCacheStats(registry, identity.CacheStats{
LookupEntries: 1,
RegistrationEntries: 2,
SourceCodeEntries: 3,
LocationTouchEntries: 4,
LocationTouchFailureEntries: 5,
SnapshotBindingEntries: 6,
SnapshotIdentifierEntries: 7,
SnapshotRegistrationEntries: 8,
SnapshotSourceEntries: 9,
SnapshotReady: true,
SnapshotRefreshedAt: time.Unix(1234, 0),
MaxEntries: 9,
})
rendered := registry.Render()
for _, want := range []string{
`vehicle_gateway_identity_cache_entries{cache="location_touch_failure"} 5`,
`vehicle_gateway_identity_cache_entries{cache="registration_write_queue"} 0`,
`vehicle_gateway_identity_cache_entries{cache="snapshot_binding"} 6`,
`vehicle_gateway_identity_cache_entries{cache="snapshot_identifier"} 7`,
`vehicle_gateway_identity_cache_entries{cache="snapshot_registration"} 8`,
`vehicle_gateway_identity_cache_entries{cache="snapshot_source"} 9`,
`vehicle_gateway_identity_cache_max_entries{cache="location_touch_failure"} 9`,
`vehicle_gateway_identity_snapshot_ready 1`,
`vehicle_gateway_identity_snapshot_last_success_unix_seconds 1234`,
} {
if !strings.Contains(rendered, want) {
t.Fatalf("identity cache metrics missing %s in:\n%s", want, rendered)
}
}
}
func TestRecordIdentitySnapshotRefresh(t *testing.T) {
registry := metrics.NewRegistry()
recordIdentitySnapshotRefresh(registry, identity.SnapshotRefreshResult{
BindingEntries: 10,
IdentifierEntries: 20,
RegistrationEntries: 30,
SourceEntries: 40,
RefreshedAt: time.Unix(5678, 0),
}, "ok")
rendered := registry.Render()
for _, want := range []string{
`vehicle_gateway_identity_snapshot_refresh_total{status="ok"} 1`,
`vehicle_gateway_identity_snapshot_entries{kind="binding"} 10`,
`vehicle_gateway_identity_snapshot_entries{kind="identifier"} 20`,
`vehicle_gateway_identity_snapshot_entries{kind="registration"} 30`,
`vehicle_gateway_identity_snapshot_entries{kind="source"} 40`,
`vehicle_gateway_identity_snapshot_last_success_unix_seconds 5678`,
} {
if !strings.Contains(rendered, want) {
t.Fatalf("identity snapshot metrics missing %s in:\n%s", want, rendered)
}
}
}
func TestIdentityDatabaseMaintenanceDoesNotBlockIngressStartup(t *testing.T) {
ctx, cancel := context.WithCancel(context.Background())
defer cancel()
db := &blockingIdentityDatabase{
started: make(chan struct{}),
release: make(chan struct{}),
}
logger := slog.New(slog.NewTextHandler(io.Discard, nil))
startedAt := time.Now()
startIdentityDatabaseMaintenance(ctx, logger, db, nil, false, time.Second, time.Second)
if elapsed := time.Since(startedAt); elapsed > 100*time.Millisecond {
t.Fatalf("database maintenance blocked gateway startup for %s", elapsed)
}
select {
case <-db.started:
case <-time.After(time.Second):
t.Fatal("database maintenance did not start in background")
}
close(db.release)
}
func TestIdentitySnapshotRefreshDoesNotBlockIngressStartup(t *testing.T) {
ctx, cancel := context.WithCancel(context.Background())
defer cancel()
refresher := &blockingIdentitySnapshotRefresher{started: make(chan struct{})}
logger := slog.New(slog.NewTextHandler(io.Discard, nil))
startedAt := time.Now()
startIdentitySnapshotRefresh(ctx, logger, metrics.NewRegistry(), refresher, time.Hour, time.Second)
if elapsed := time.Since(startedAt); elapsed > 100*time.Millisecond {
t.Fatalf("snapshot refresh blocked gateway startup for %s", elapsed)
}
select {
case <-refresher.started:
case <-time.After(time.Second):
t.Fatal("snapshot refresh did not start in background")
}
}
func TestRecordJT808RegistrationWriteResult(t *testing.T) {
registry := metrics.NewRegistry()
recordJT808RegistrationWriteResult(registry, identity.RegistrationWriteResult{
Mode: "async_background",
Status: "error",
})
rendered := registry.Render()
for _, want := range []string{
`vehicle_gateway_jt808_registration_write_total{mode="async_background",status="error"} 1`,
`vehicle_gateway_last_jt808_registration_write_unix_seconds{mode="async_background",status="error"}`,
} {
if !strings.Contains(rendered, want) {
t.Fatalf("registration write metric missing %s in:\n%s", want, rendered)
}
}
}
type blockingIdentityDatabase struct {
started chan struct{}
release chan struct{}
}
func (d *blockingIdentityDatabase) PingContext(ctx context.Context) error {
close(d.started)
select {
case <-ctx.Done():
return ctx.Err()
case <-d.release:
return nil
}
}
type blockingIdentitySnapshotRefresher struct {
started chan struct{}
}
func (r *blockingIdentitySnapshotRefresher) RefreshSnapshot(ctx context.Context) (identity.SnapshotRefreshResult, error) {
close(r.started)
<-ctx.Done()
return identity.SnapshotRefreshResult{}, errors.New("test refresh stopped")
}
func TestGatewayDefaultsTo100KConnectionCeiling(t *testing.T) {
source, err := os.ReadFile("main.go")
if err != nil {
@@ -64,6 +344,18 @@ func TestGatewayPassesMetricsRegistryToAsyncSink(t *testing.T) {
for _, want := range []string{
"buildSink(ctx, logger, registry)",
"Metrics: registry",
"EnqueueTimeout: enqueueTimeout",
"NewPartitionedAsyncSink",
"NATS_PARTITIONED_ASYNC_ENABLED",
"KAFKA_PARTITIONED_ASYNC_ENABLED",
"_ASYNC_RAW_QUEUE_SIZE",
"_ASYNC_DERIVED_QUEUE_SIZE",
"_ASYNC_RAW_ENQUEUE_TIMEOUT_MS",
"_ASYNC_DERIVED_ENQUEUE_TIMEOUT_MS",
"RawEnqueueTimeout",
"DerivedEnqueueTimeout",
"NATS_ASYNC_ENQUEUE_TIMEOUT_MS",
"KAFKA_ASYNC_ENQUEUE_TIMEOUT_MS",
`Name: "nats"`,
`Name: "kafka"`,
} {
@@ -73,6 +365,43 @@ func TestGatewayPassesMetricsRegistryToAsyncSink(t *testing.T) {
}
}
func TestGatewayExposesNATSDurableSpoolConfig(t *testing.T) {
source, err := os.ReadFile("main.go")
if err != nil {
t.Fatalf("read main.go: %v", err)
}
for _, want := range []string{
"NATS_DURABLE_OUTBOX_ENABLED",
"NATS_OUTBOX_DIR",
"NATS_OUTBOX_MAX_INFLIGHT",
"NATS_OUTBOX_ACK_TIMEOUT_MS",
"NATS_OUTBOX_FSYNC",
"NATS_OUTBOX_CLOSE_TIMEOUT_MS",
"NATS_OUTBOX_REPLAY_BATCH_SIZE",
"NATS_OUTBOX_REPLAY_INTERVAL_MS",
"NATS_OUTBOX_WAL_SEGMENT_BYTES",
"NATS_OUTBOX_WAL_SEGMENT_AGE_MS",
"NATS_OUTBOX_WAL_APPEND_QUEUE_SIZE",
"NATS_OUTBOX_WAL_COMMIT_BATCH_SIZE",
"NATS_OUTBOX_WAL_COMMIT_INTERVAL_MS",
"eventbus.NewDurableOutboxSink",
"nats durable outbox enabled",
"NATS_SPOOL_DIR",
"NATS_SPOOL_REPLAY_BATCH_SIZE",
"NATS_SPOOL_REPLAY_INTERVAL_MS",
"nats durable spool enabled",
"nats spool replay failed",
"eventbus.NewDurableSink",
"Metrics: registry",
`Name: "nats"`,
`Name: "kafka"`,
} {
if !strings.Contains(string(source), want) {
t.Fatalf("gateway nats spool wiring missing %s", want)
}
}
}
func TestNATSSinkConfigFromEnvDefaultsToGoSubjects(t *testing.T) {
t.Setenv("NATS_URL", "nats://172.17.111.56:4222")
@@ -87,6 +416,15 @@ func TestNATSSinkConfigFromEnvDefaultsToGoSubjects(t *testing.T) {
if got, want := cfg.RawSubjects[envelope.ProtocolYutongMQTT], "vehicle.raw.go.yutong-mqtt.v1"; got != want {
t.Fatalf("yutong subject = %q, want %q", got, want)
}
if got, want := cfg.FieldsSubjects[envelope.ProtocolGB32960], "vehicle.fields.go.gb32960.v1"; got != want {
t.Fatalf("gb32960 fields subject = %q, want %q", got, want)
}
if got, want := cfg.FieldsSubjects[envelope.ProtocolJT808], "vehicle.fields.go.jt808.v1"; got != want {
t.Fatalf("jt808 fields subject = %q, want %q", got, want)
}
if got, want := cfg.FieldsSubjects[envelope.ProtocolYutongMQTT], "vehicle.fields.go.yutong-mqtt.v1"; got != want {
t.Fatalf("yutong fields subject = %q, want %q", got, want)
}
if got, want := cfg.UnifiedSubject, "vehicle.event.go.unified.v1"; got != want {
t.Fatalf("unified subject = %q, want %q", got, want)
}

612
go/vehicle-gateway/cmd/history-writer/main.go
View File

@@ -4,10 +4,14 @@ import (
"context"
"database/sql"
"encoding/json"
"errors"
"fmt"
"os"
"os/signal"
"sort"
"strconv"
"strings"
"sync"
"syscall"
"time"
@@ -15,6 +19,7 @@ import (
_ "github.com/taosdata/driver-go/v3/taosWS"
"lingniu-vehicle-ingest/go/vehicle-gateway/internal/envelope"
"lingniu-vehicle-ingest/go/vehicle-gateway/internal/eventbus"
"lingniu-vehicle-ingest/go/vehicle-gateway/internal/health"
"lingniu-vehicle-ingest/go/vehicle-gateway/internal/history"
"lingniu-vehicle-ingest/go/vehicle-gateway/internal/metrics"
@@ -28,6 +33,10 @@ func main() {
defer stop()
cfg := loadConfig()
if err := cfg.Validate(); err != nil {
logger.Error("invalid history writer config", "error", err)
os.Exit(1)
}
db, err := sql.Open(cfg.TDengineDriver, cfg.TDengineDSN)
if err != nil {
logger.Error("tdengine open failed", "error", err)
@@ -39,18 +48,51 @@ func main() {
os.Exit(1)
}
registry := metrics.NewRegistry()
metrics.RegisterKafkaConsumerInfo(registry, "vehicle-history-writer", cfg.KafkaGroup, cfg.KafkaTopics)
registry.SetGauge("vehicle_history_config", metrics.Labels{"setting": "workers"}, float64(cfg.Workers))
health.Start(ctx, logger, health.NewServer(env("HEALTH_ADDR", ""), "vehicle-history-writer", []health.Check{
{Name: "tdengine", Check: db.PingContext},
}, registry))
writer := history.NewWriter(db)
writer := history.NewWriterWithDatabase(db, cfg.TDengineDatabase)
if cfg.EnsureSchema {
if err := writer.EnsureSchema(ctx, cfg.TDengineDatabase); err != nil {
logger.Error("tdengine schema bootstrap failed", "error", err)
os.Exit(1)
}
}
var appender historyAppender = retryHistoryAppender{
delegate: writer,
attempts: cfg.RetryAttempts,
delay: cfg.RetryDelay,
registry: registry,
}
logger.Info("history writer started",
"driver", cfg.TDengineDriver,
"group", cfg.KafkaGroup,
"topics", strings.Join(cfg.KafkaTopics, ","),
"workers", cfg.Workers,
"batch_size", cfg.BatchSize,
"batch_wait_ms", cfg.BatchWait,
"retry_attempts", cfg.RetryAttempts,
"retry_delay_ms", cfg.RetryDelay.Milliseconds())
var workers sync.WaitGroup
for workerID := 1; workerID <= cfg.Workers; workerID++ {
workers.Add(1)
go func(id int) {
defer workers.Done()
runHistoryConsumer(ctx, logger, registry, appender, cfg, id)
}(workerID)
}
workers.Wait()
}
func runHistoryConsumer(ctx context.Context, logger interface {
Error(string, ...any)
Warn(string, ...any)
}, registry *metrics.Registry, appender historyAppender, cfg config, workerID int) {
reader := kafka.NewReader(kafka.ReaderConfig{
Brokers: cfg.KafkaBrokers,
GroupID: cfg.KafkaGroup,
@@ -60,12 +102,9 @@ func main() {
})
defer reader.Close()
logger.Info("history writer started",
"driver", cfg.TDengineDriver,
"group", cfg.KafkaGroup,
"topics", strings.Join(cfg.KafkaTopics, ","),
"batch_size", cfg.BatchSize,
"batch_wait_ms", cfg.BatchWait)
workerLabels := metrics.Labels{"worker": strconv.Itoa(workerID)}
registry.SetGauge("vehicle_history_worker_active", workerLabels, 1)
defer registry.SetGauge("vehicle_history_worker_active", workerLabels, 0)
for {
message, err := reader.FetchMessage(ctx)
@@ -77,7 +116,7 @@ func main() {
continue
}
batch := collectHistoryBatch(ctx, reader, message, cfg.BatchSize, time.Duration(cfg.BatchWait)*time.Millisecond)
processHistoryBatch(ctx, logger, registry, writer, reader, batch)
processHistoryBatchReliablyForWorker(ctx, logger, registry, appender, reader, batch, cfg.RetryDelay, workerLabels)
}
}
@@ -88,6 +127,11 @@ type historyAppender interface {
AppendAllBatch(context.Context, []envelope.FrameEnvelope) error
}
type historyResultAppender interface {
AppendAllWithResult(context.Context, envelope.FrameEnvelope) (history.AppendResult, error)
AppendAllBatchWithResult(context.Context, []envelope.FrameEnvelope) (history.AppendResult, error)
}
type kafkaMessageCommitter interface {
CommitMessages(context.Context, ...kafka.Message) error
}
@@ -96,6 +140,17 @@ type kafkaMessageFetcher interface {
FetchMessage(context.Context) (kafka.Message, error)
}
type historyBatchItem struct {
message kafka.Message
valid bool
processed bool
}
type historyBatchOutcome struct {
commitMessages []kafka.Message
retryMessages []kafka.Message
}
func collectHistoryBatch(ctx context.Context, fetcher kafkaMessageFetcher, first kafka.Message, maxSize int, maxWait time.Duration) []kafka.Message {
if maxSize <= 1 {
return []kafka.Message{first}
@@ -134,15 +189,38 @@ func processHistoryMessage(ctx context.Context, logger interface {
if err := json.Unmarshal(message.Value, &env); err != nil {
addWriterMetric(registry, "vehicle_history_kafka_messages_total", message, "invalid_json")
logger.Warn("skip invalid envelope json", "topic", message.Topic, "partition", message.Partition, "offset", message.Offset, "error", err)
_ = committer.CommitMessages(messageCtx, message)
if err := committer.CommitMessages(messageCtx, message); err != nil {
addWriterMetric(registry, "vehicle_history_kafka_commits_total", message, "error")
logger.Error("kafka commit invalid envelope failed", "topic", message.Topic, "partition", message.Partition, "offset", message.Offset, "error", err)
return
}
addWriterMetric(registry, "vehicle_history_kafka_commits_total", message, "ok")
return
}
if err := appender.AppendAll(messageCtx, env); err != nil {
addWriterMetric(registry, "vehicle_history_writes_total", message, "error")
logger.Error("tdengine append failed", "topic", message.Topic, "partition", message.Partition, "offset", message.Offset, "event_id", env.StableEventID(), "error", err)
if status, err := topics.ValidateRawEnvelope(message.Topic, env); err != nil {
addWriterMetric(registry, "vehicle_history_kafka_messages_total", message, status)
logger.Warn("skip mismatched history raw envelope", "topic", message.Topic, "partition", message.Partition, "offset", message.Offset, "protocol", env.Protocol, "event_id", env.StableEventID(), "event_kind", env.EventKind, "error", err)
if err := committer.CommitMessages(messageCtx, message); err != nil {
addWriterMetric(registry, "vehicle_history_kafka_commits_total", message, "error")
logger.Error("kafka commit mismatched raw envelope failed", "topic", message.Topic, "partition", message.Partition, "offset", message.Offset, "event_id", env.StableEventID(), "status", status, "error", err)
return
}
addWriterMetric(registry, "vehicle_history_kafka_commits_total", message, "ok")
return
}
recordHistoryParsedFieldMetrics(registry, message, env)
result, err := appendHistoryEnvelope(messageCtx, appender, env)
if err != nil {
addWriterMetric(registry, "vehicle_history_writes_total", message, "error")
logger.Error("tdengine raw append failed", "topic", message.Topic, "partition", message.Partition, "offset", message.Offset, "event_id", env.StableEventID(), "error", err)
return
}
recordHistoryDerivedMetrics(registry, []kafka.Message{message}, []envelope.FrameEnvelope{env}, result)
if result.LocationError != nil {
logger.Warn("tdengine location append failed after raw append", "topic", message.Topic, "partition", message.Partition, "offset", message.Offset, "event_id", env.StableEventID(), "error", result.LocationError)
}
addWriterMetric(registry, "vehicle_history_writes_total", message, "ok")
recordHistoryWriteE2EDuration(registry, message, env)
if err := committer.CommitMessages(messageCtx, message); err != nil {
addWriterMetric(registry, "vehicle_history_kafka_commits_total", message, "error")
logger.Error("kafka commit failed", "topic", message.Topic, "partition", message.Partition, "offset", message.Offset, "error", err)
@@ -154,30 +232,52 @@ func processHistoryMessage(ctx context.Context, logger interface {
func processHistoryBatch(ctx context.Context, logger interface {
Error(string, ...any)
Warn(string, ...any)
}, registry *metrics.Registry, appender historyAppender, committer kafkaMessageCommitter, messages []kafka.Message) {
}, registry *metrics.Registry, appender historyAppender, committer kafkaMessageCommitter, messages []kafka.Message) historyBatchOutcome {
return processHistoryBatchForWorker(ctx, logger, registry, appender, committer, messages, nil)
}
func processHistoryBatchForWorker(ctx context.Context, logger interface {
Error(string, ...any)
Warn(string, ...any)
}, registry *metrics.Registry, appender historyAppender, committer kafkaMessageCommitter, messages []kafka.Message, workerLabels metrics.Labels) historyBatchOutcome {
if len(messages) == 0 {
return
return historyBatchOutcome{}
}
messageCtx, cancel := context.WithTimeout(context.WithoutCancel(ctx), kafkaMessageOperationTimeout)
defer cancel()
envelopes := make([]envelope.FrameEnvelope, 0, len(messages))
validMessages := make([]kafka.Message, 0, len(messages))
validItemIndexes := make([]int, 0, len(messages))
items := make([]historyBatchItem, 0, len(messages))
for _, message := range messages {
itemIndex := len(items)
addWriterMetric(registry, "vehicle_history_kafka_messages_total", message, "received")
addWriterLagMetric(registry, message)
var env envelope.FrameEnvelope
if err := json.Unmarshal(message.Value, &env); err != nil {
addWriterMetric(registry, "vehicle_history_kafka_messages_total", message, "invalid_json")
logger.Warn("skip invalid envelope json", "topic", message.Topic, "partition", message.Partition, "offset", message.Offset, "error", err)
items = append(items, historyBatchItem{message: message, processed: true})
continue
}
if status, err := topics.ValidateRawEnvelope(message.Topic, env); err != nil {
addWriterMetric(registry, "vehicle_history_kafka_messages_total", message, status)
logger.Warn("skip mismatched history raw envelope", "topic", message.Topic, "partition", message.Partition, "offset", message.Offset, "protocol", env.Protocol, "event_id", env.StableEventID(), "event_kind", env.EventKind, "error", err)
items = append(items, historyBatchItem{message: message, processed: true})
continue
}
recordHistoryParsedFieldMetrics(registry, message, env)
envelopes = append(envelopes, env)
validMessages = append(validMessages, message)
validItemIndexes = append(validItemIndexes, itemIndex)
items = append(items, historyBatchItem{message: message, valid: true})
}
if len(envelopes) > 0 {
setBatchPending(registry, len(messages), len(envelopes))
defer setBatchPending(registry, 0, 0)
setBatchPendingForWorker(registry, workerLabels, len(messages), len(envelopes))
defer setBatchPendingForWorker(registry, workerLabels, 0, 0)
started := time.Now()
err := appender.AppendAllBatch(messageCtx, envelopes)
result, err := appendHistoryBatch(messageCtx, appender, envelopes)
elapsed := time.Since(started)
status := "ok"
if err != nil {
@@ -187,15 +287,33 @@ func processHistoryBatch(ctx context.Context, logger interface {
addBatchMetric(registry, "vehicle_history_batch_rows_total", status, float64(len(envelopes)))
setBatchDuration(registry, status, elapsed)
if err != nil {
for _, message := range messages {
addWriterMetric(registry, "vehicle_history_writes_total", message, "error")
}
first := messages[0]
logger.Error("tdengine batch append failed", "topic", first.Topic, "partition", first.Partition, "offset", first.Offset, "rows", len(envelopes), "error", err)
return
logger.Error("tdengine raw batch append failed", "topic", first.Topic, "partition", first.Partition, "offset", first.Offset, "rows", len(envelopes), "error", err)
if isTransientTDengineHistoryError(err) {
for _, message := range validMessages {
addWriterMetric(registry, "vehicle_history_writes_total", message, "error")
}
addBatchMetric(registry, "vehicle_history_batch_fallback_total", "skipped_transient", 1)
committed, commitErr := commitHistoryProcessedPrefixAfterFailure(messageCtx, logger, registry, committer, items)
return historyFailureOutcome(messages, committed, commitErr)
}
if fallbackErr := fallbackHistoryBatchToSingles(messageCtx, logger, registry, appender, validMessages, envelopes, validItemIndexes, items); fallbackErr != nil {
addBatchMetric(registry, "vehicle_history_batch_fallback_total", "error", 1)
committed, commitErr := commitHistoryProcessedPrefixAfterFailure(messageCtx, logger, registry, committer, items)
return historyFailureOutcome(messages, committed, commitErr)
}
addBatchMetric(registry, "vehicle_history_batch_fallback_total", "ok", 1)
committed, commitErr := commitHistoryProcessedPrefixAfterFailure(messageCtx, logger, registry, committer, items)
return historyFailureOutcome(messages, committed, commitErr)
}
for _, message := range messages {
recordHistoryDerivedMetrics(registry, validMessages, envelopes, result)
if result.LocationError != nil {
first := messages[0]
logger.Warn("tdengine location batch append failed after raw append", "topic", first.Topic, "partition", first.Partition, "offset", first.Offset, "rows", len(envelopes), "location_rows", result.LocationRows, "error", result.LocationError)
}
for index, message := range validMessages {
addWriterMetric(registry, "vehicle_history_writes_total", message, "ok")
recordHistoryWriteE2EDuration(registry, message, envelopes[index])
}
}
if err := committer.CommitMessages(messageCtx, messages...); err != nil {
@@ -204,18 +322,337 @@ func processHistoryBatch(ctx context.Context, logger interface {
}
first := messages[0]
logger.Error("kafka batch commit failed", "topic", first.Topic, "partition", first.Partition, "offset", first.Offset, "messages", len(messages), "error", err)
return
return historyBatchOutcome{commitMessages: messages}
}
for _, message := range messages {
addWriterMetric(registry, "vehicle_history_kafka_commits_total", message, "ok")
}
return historyBatchOutcome{}
}
func historyFailureOutcome(messages []kafka.Message, committed []kafka.Message, commitErr error) historyBatchOutcome {
outcome := historyBatchOutcome{
retryMessages: eventbus.MessagesAfterCommittedPrefixes(messages, committed),
}
if commitErr != nil {
outcome.commitMessages = committed
}
return outcome
}
func processHistoryBatchReliably(ctx context.Context, logger interface {
Error(string, ...any)
Warn(string, ...any)
}, registry *metrics.Registry, appender historyAppender, committer kafkaMessageCommitter, messages []kafka.Message, retryDelay time.Duration) {
processHistoryBatchReliablyForWorker(ctx, logger, registry, appender, committer, messages, retryDelay, nil)
}
func processHistoryBatchReliablyForWorker(ctx context.Context, logger interface {
Error(string, ...any)
Warn(string, ...any)
}, registry *metrics.Registry, appender historyAppender, committer kafkaMessageCommitter, messages []kafka.Message, retryDelay time.Duration, workerLabels metrics.Labels) {
defer registry.SetGauge("vehicle_history_retry_pending_messages", workerLabels, 0)
pending := messages
for len(pending) > 0 {
outcome := processHistoryBatchForWorker(ctx, logger, registry, appender, committer, pending, workerLabels)
if len(outcome.commitMessages) > 0 {
registry.IncCounter("vehicle_history_batch_retries_total", metrics.Labels{"reason": "commit_error"})
if !retryHistoryCommit(ctx, logger, registry, committer, outcome.commitMessages, retryDelay) {
return
}
}
pending = outcome.retryMessages
registry.SetGauge("vehicle_history_retry_pending_messages", workerLabels, float64(len(pending)))
if len(pending) == 0 {
return
}
registry.IncCounter("vehicle_history_batch_retries_total", metrics.Labels{"reason": "write_error"})
if !waitForHistoryRetry(ctx, retryDelay) {
return
}
}
}
func retryHistoryCommit(ctx context.Context, logger interface {
Error(string, ...any)
Warn(string, ...any)
}, registry *metrics.Registry, committer kafkaMessageCommitter, messages []kafka.Message, retryDelay time.Duration) bool {
for len(messages) > 0 {
operationCtx, cancel := context.WithTimeout(context.WithoutCancel(ctx), kafkaMessageOperationTimeout)
err := committer.CommitMessages(operationCtx, messages...)
cancel()
if err == nil {
for _, message := range messages {
addWriterMetric(registry, "vehicle_history_kafka_commits_total", message, "ok")
}
return true
}
for _, message := range messages {
addWriterMetric(registry, "vehicle_history_kafka_commits_total", message, "error")
}
first := messages[0]
logger.Error("kafka commit retry failed", "topic", first.Topic, "partition", first.Partition, "offset", first.Offset, "messages", len(messages), "error", err)
registry.IncCounter("vehicle_history_batch_retries_total", metrics.Labels{"reason": "commit_error"})
if !waitForHistoryRetry(ctx, retryDelay) {
return false
}
}
return true
}
func waitForHistoryRetry(ctx context.Context, retryDelay time.Duration) bool {
if retryDelay <= 0 {
retryDelay = 100 * time.Millisecond
}
timer := time.NewTimer(retryDelay)
defer timer.Stop()
select {
case <-ctx.Done():
return false
case <-timer.C:
return true
}
}
func fallbackHistoryBatchToSingles(ctx context.Context, logger interface {
Error(string, ...any)
Warn(string, ...any)
}, registry *metrics.Registry, appender historyAppender, messages []kafka.Message, envelopes []envelope.FrameEnvelope, itemIndexes []int, items []historyBatchItem) error {
for index, env := range envelopes {
message := messages[index]
started := time.Now()
result, err := appendHistoryEnvelope(ctx, appender, env)
setFallbackDuration(registry, statusFromError(err), time.Since(started))
if err != nil {
addWriterMetric(registry, "vehicle_history_writes_total", message, "error")
logger.Error("tdengine raw single append failed after batch fallback", "topic", message.Topic, "partition", message.Partition, "offset", message.Offset, "event_id", env.StableEventID(), "error", err)
return err
}
recordHistoryDerivedMetrics(registry, []kafka.Message{message}, []envelope.FrameEnvelope{env}, result)
if result.LocationError != nil {
logger.Warn("tdengine location append failed after raw single fallback", "topic", message.Topic, "partition", message.Partition, "offset", message.Offset, "event_id", env.StableEventID(), "error", result.LocationError)
}
addWriterMetric(registry, "vehicle_history_writes_total", message, "ok")
recordHistoryWriteE2EDuration(registry, message, env)
items[itemIndexes[index]].processed = true
}
return nil
}
func commitHistoryProcessedPrefixAfterFailure(ctx context.Context, logger interface {
Error(string, ...any)
Warn(string, ...any)
}, registry *metrics.Registry, committer kafkaMessageCommitter, items []historyBatchItem) ([]kafka.Message, error) {
committable := processedHistoryPrefixMessagesByPartition(items)
if len(committable) == 0 {
return nil, nil
}
if err := committer.CommitMessages(ctx, committable...); err != nil {
for _, message := range committable {
addWriterMetric(registry, "vehicle_history_kafka_commits_total", message, "error")
}
first := committable[0]
logger.Error("kafka processed-prefix commit failed after raw batch append failure", "topic", first.Topic, "partition", first.Partition, "offset", first.Offset, "messages", len(committable), "error", err)
return committable, err
}
for _, message := range committable {
addWriterMetric(registry, "vehicle_history_kafka_commits_total", message, "ok")
}
return committable, nil
}
func processedHistoryPrefixMessagesByPartition(items []historyBatchItem) []kafka.Message {
type partitionKey struct {
topic string
partition int
}
groups := map[partitionKey][]historyBatchItem{}
for _, item := range items {
key := partitionKey{topic: item.message.Topic, partition: item.message.Partition}
groups[key] = append(groups[key], item)
}
var keys []partitionKey
for key := range groups {
keys = append(keys, key)
}
sort.Slice(keys, func(i, j int) bool {
if keys[i].topic != keys[j].topic {
return keys[i].topic < keys[j].topic
}
return keys[i].partition < keys[j].partition
})
var out []kafka.Message
for _, key := range keys {
group := groups[key]
sort.Slice(group, func(i, j int) bool {
return group[i].message.Offset < group[j].message.Offset
})
var previousOffset int64
for index, item := range group {
if index > 0 && item.message.Offset != previousOffset+1 {
break
}
if !item.processed {
break
}
out = append(out, item.message)
previousOffset = item.message.Offset
}
}
return out
}
func appendHistoryEnvelope(ctx context.Context, appender historyAppender, env envelope.FrameEnvelope) (history.AppendResult, error) {
if resultAppender, ok := appender.(historyResultAppender); ok {
return resultAppender.AppendAllWithResult(ctx, env)
}
if err := appender.AppendAll(ctx, env); err != nil {
return history.AppendResult{}, err
}
return history.AppendResult{RawRows: 1}, nil
}
func appendHistoryBatch(ctx context.Context, appender historyAppender, envelopes []envelope.FrameEnvelope) (history.AppendResult, error) {
if resultAppender, ok := appender.(historyResultAppender); ok {
return resultAppender.AppendAllBatchWithResult(ctx, envelopes)
}
if err := appender.AppendAllBatch(ctx, envelopes); err != nil {
return history.AppendResult{}, err
}
return history.AppendResult{RawRows: len(envelopes)}, nil
}
type retryHistoryAppender struct {
delegate historyAppender
attempts int
delay time.Duration
registry *metrics.Registry
}
func (a retryHistoryAppender) AppendAll(ctx context.Context, env envelope.FrameEnvelope) error {
_, err := a.AppendAllWithResult(ctx, env)
return err
}
func (a retryHistoryAppender) AppendAllWithResult(ctx context.Context, env envelope.FrameEnvelope) (history.AppendResult, error) {
if a.delegate == nil {
return history.AppendResult{}, nil
}
attempts := a.attempts
if attempts <= 0 {
attempts = 1
}
var result history.AppendResult
var err error
for attempt := 1; attempt <= attempts; attempt++ {
result, err = appendHistoryEnvelope(ctx, a.delegate, env)
if err == nil || !isTransientTDengineHistoryError(err) {
return result, err
}
if attempt == attempts {
a.recordRetry("single", "exhausted")
return result, err
}
a.recordRetry("single", "retry")
if waitErr := sleepBeforeRetry(ctx, a.delay); waitErr != nil {
return result, waitErr
}
}
return result, err
}
func (a retryHistoryAppender) AppendAllBatch(ctx context.Context, envelopes []envelope.FrameEnvelope) error {
_, err := a.AppendAllBatchWithResult(ctx, envelopes)
return err
}
func (a retryHistoryAppender) AppendAllBatchWithResult(ctx context.Context, envelopes []envelope.FrameEnvelope) (history.AppendResult, error) {
if a.delegate == nil {
return history.AppendResult{}, nil
}
attempts := a.attempts
if attempts <= 0 {
attempts = 1
}
var result history.AppendResult
var err error
for attempt := 1; attempt <= attempts; attempt++ {
result, err = appendHistoryBatch(ctx, a.delegate, envelopes)
if err == nil || !isTransientTDengineHistoryError(err) {
return result, err
}
if attempt == attempts {
a.recordRetry("batch", "exhausted")
return result, err
}
a.recordRetry("batch", "retry")
if waitErr := sleepBeforeRetry(ctx, a.delay); waitErr != nil {
return result, waitErr
}
}
return result, err
}
func (a retryHistoryAppender) recordRetry(operation string, status string) {
if a.registry == nil {
return
}
labels := metrics.Labels{"operation": operation, "status": status}
a.registry.IncCounter("vehicle_history_write_retries_total", labels)
metrics.RecordLastActivity(a.registry, "vehicle_history_last_write_retry_unix_seconds", labels)
}
func sleepBeforeRetry(ctx context.Context, delay time.Duration) error {
if delay <= 0 {
return nil
}
timer := time.NewTimer(delay)
defer timer.Stop()
select {
case <-ctx.Done():
return ctx.Err()
case <-timer.C:
return nil
}
}
func isTransientTDengineHistoryError(err error) bool {
if err == nil || errors.Is(err, context.Canceled) || errors.Is(err, context.DeadlineExceeded) {
return false
}
text := strings.ToLower(strings.TrimSpace(err.Error()))
return strings.Contains(text, "timeout") ||
strings.Contains(text, "temporary") ||
strings.Contains(text, "temporarily") ||
strings.Contains(text, "connection refused") ||
strings.Contains(text, "connection reset") ||
strings.Contains(text, "connection closed") ||
strings.Contains(text, "broken pipe") ||
strings.Contains(text, "bad connection") ||
strings.Contains(text, "i/o timeout") ||
text == "eof" ||
strings.Contains(text, "unexpected eof") ||
strings.Contains(text, "server is down") ||
strings.Contains(text, "network is unreachable") ||
strings.Contains(text, "no route to host")
}
func addWriterMetric(registry *metrics.Registry, name string, message kafka.Message, status string) {
if registry == nil {
return
}
registry.IncCounter(name, metrics.Labels{"topic": message.Topic, "status": status})
labels := metrics.Labels{"topic": message.Topic, "status": status}
registry.IncCounter(name, labels)
switch name {
case "vehicle_history_kafka_messages_total":
metrics.RecordLastActivity(registry, "vehicle_history_last_message_unix_seconds", labels)
case "vehicle_history_writes_total":
metrics.RecordLastActivity(registry, "vehicle_history_last_write_unix_seconds", labels)
case "vehicle_history_kafka_commits_total":
metrics.RecordLastActivity(registry, "vehicle_history_last_commit_unix_seconds", labels)
case "vehicle_history_location_writes_total":
metrics.RecordLastActivity(registry, "vehicle_history_last_location_write_unix_seconds", labels)
}
}
func addBatchMetric(registry *metrics.Registry, name string, status string, value float64) {
@@ -225,19 +662,113 @@ func addBatchMetric(registry *metrics.Registry, name string, status string, valu
registry.AddCounter(name, metrics.Labels{"status": status}, value)
}
func recordHistoryParsedFieldMetrics(registry *metrics.Registry, message kafka.Message, env envelope.FrameEnvelope) {
if registry == nil || !envelope.IsRealtimeTelemetryFrame(env) {
return
}
status := "present"
if len(env.ParsedFields) == 0 {
status = "missing"
}
registry.IncCounter("vehicle_history_parsed_fields_total", metrics.Labels{
"topic": message.Topic,
"protocol": historyProtocolLabel(env.Protocol),
"status": status,
})
}
func recordHistoryDerivedMetrics(registry *metrics.Registry, messages []kafka.Message, envelopes []envelope.FrameEnvelope, result history.AppendResult) {
if registry == nil || len(messages) == 0 {
return
}
batchStatus := "skipped"
if result.LocationError != nil {
batchStatus = "error"
} else if result.LocationRows > 0 {
batchStatus = "ok"
}
for index, message := range messages {
status := batchStatus
if index < len(envelopes) {
status = history.LocationStatus(envelopes[index])
if status == history.LocationStatusOK && result.LocationError != nil {
status = "error"
}
}
addWriterMetric(registry, "vehicle_history_location_writes_total", message, status)
}
addBatchMetric(registry, "vehicle_history_location_batch_flush_total", batchStatus, 1)
addBatchMetric(registry, "vehicle_history_location_rows_total", batchStatus, float64(result.LocationRows))
}
func historyProtocolLabel(protocol envelope.Protocol) string {
protocolLabel := strings.TrimSpace(string(protocol))
if protocolLabel == "" {
return "UNKNOWN"
}
return protocolLabel
}
func setBatchDuration(registry *metrics.Registry, status string, elapsed time.Duration) {
if registry == nil {
return
}
registry.SetGauge("vehicle_history_batch_flush_duration_ms", metrics.Labels{"status": status}, float64(elapsed.Milliseconds()))
elapsedMS := float64(elapsed.Milliseconds())
labels := metrics.Labels{"status": status}
registry.SetGauge("vehicle_history_batch_flush_duration_ms", labels, elapsedMS)
registry.ObserveHistogram("vehicle_history_batch_flush_duration_ms_histogram", labels, historyBatchFlushDurationBucketsMS, elapsedMS)
}
func setBatchPending(registry *metrics.Registry, messages int, rows int) {
var historyBatchFlushDurationBucketsMS = []float64{1, 5, 10, 25, 50, 100, 250, 500, 1000, 5000}
func setFallbackDuration(registry *metrics.Registry, status string, elapsed time.Duration) {
if registry == nil {
return
}
registry.SetGauge("vehicle_history_batch_pending_messages", nil, float64(messages))
registry.SetGauge("vehicle_history_batch_pending_rows", nil, float64(rows))
elapsedMS := float64(elapsed.Milliseconds())
labels := metrics.Labels{"status": status}
registry.SetGauge("vehicle_history_batch_fallback_duration_ms", labels, elapsedMS)
registry.ObserveHistogram("vehicle_history_batch_fallback_duration_ms_histogram", labels, historyBatchFallbackDurationBucketsMS, elapsedMS)
}
var historyBatchFallbackDurationBucketsMS = []float64{1, 5, 10, 25, 50, 100, 250, 500, 1000, 5000}
var historyWriteE2EDurationBucketsMS = []float64{10, 25, 50, 100, 250, 500, 1000, 2500, 5000, 10000}
var historyWriteE2ERecent = metrics.NewRecentLatencyByKey(512)
func recordHistoryWriteE2EDuration(registry *metrics.Registry, message kafka.Message, env envelope.FrameEnvelope) {
if registry == nil || env.ReceivedAtMS <= 0 {
return
}
elapsed := time.Since(time.UnixMilli(env.ReceivedAtMS)).Milliseconds()
if elapsed < 0 {
elapsed = 0
}
labels := metrics.Labels{"topic": message.Topic}
registry.ObserveHistogram("vehicle_history_write_e2e_duration_ms_histogram", labels, historyWriteE2EDurationBucketsMS, float64(elapsed))
p99, samples := historyWriteE2ERecent.Observe(message.Topic, float64(elapsed))
registry.SetGauge("vehicle_history_write_e2e_recent_p99_ms", labels, p99)
registry.SetGauge("vehicle_history_write_e2e_recent_samples", labels, float64(samples))
metrics.RecordLastActivity(registry, "vehicle_history_last_write_e2e_unix_seconds", labels)
}
func statusFromError(err error) string {
if err != nil {
return "error"
}
return "ok"
}
func setBatchPending(registry *metrics.Registry, messages int, rows int) {
setBatchPendingForWorker(registry, nil, messages, rows)
}
func setBatchPendingForWorker(registry *metrics.Registry, workerLabels metrics.Labels, messages int, rows int) {
if registry == nil {
return
}
registry.SetGauge("vehicle_history_batch_pending_messages", workerLabels, float64(messages))
registry.SetGauge("vehicle_history_batch_pending_rows", workerLabels, float64(rows))
}
func addWriterLagMetric(registry *metrics.Registry, message kafka.Message) {
@@ -255,8 +786,26 @@ type config struct {
TDengineDSN string
TDengineDatabase string
EnsureSchema bool
Workers int
BatchSize int
BatchWait int
RetryAttempts int
RetryDelay time.Duration
}
func (c config) Validate() error {
if len(c.KafkaTopics) == 0 {
return errors.New("KAFKA_TOPICS must include raw topics")
}
for _, topic := range c.KafkaTopics {
if _, ok := topics.ProtocolForKnownRawTopic(topic); !ok {
return fmt.Errorf("history-writer consumes known raw topics only, got %q", topic)
}
}
if c.Workers <= 0 {
return errors.New("HISTORY_WORKERS must be greater than zero")
}
return nil
}
func loadConfig() config {
@@ -268,8 +817,11 @@ func loadConfig() config {
TDengineDSN: env("TDENGINE_DSN", ""),
TDengineDatabase: env("TDENGINE_DATABASE", history.DefaultDatabase),
EnsureSchema: env("TDENGINE_ENSURE_SCHEMA", "true") != "false",
Workers: envInt("HISTORY_WORKERS", 3),
BatchSize: envInt("HISTORY_BATCH_SIZE", 200),
BatchWait: envInt("HISTORY_BATCH_WAIT_MS", 100),
BatchWait: envInt("HISTORY_BATCH_WAIT_MS", 20),
RetryAttempts: envInt("HISTORY_RETRY_ATTEMPTS", 3),
RetryDelay: time.Duration(envInt("HISTORY_RETRY_DELAY_MS", 20)) * time.Millisecond,
}
}

924
go/vehicle-gateway/cmd/history-writer/main_test.go
View File

File diff suppressed because it is too large Load Diff

695
go/vehicle-gateway/cmd/identity-import/main.go Normal file
View File

@@ -0,0 +1,695 @@
package main
import (
"context"
"database/sql"
"encoding/csv"
"encoding/json"
"errors"
"flag"
"fmt"
"os"
"strings"
"time"
_ "github.com/go-sql-driver/mysql"
"lingniu-vehicle-ingest/go/vehicle-gateway/internal/envelope"
"lingniu-vehicle-ingest/go/vehicle-gateway/internal/identity"
"lingniu-vehicle-ingest/go/vehicle-gateway/internal/stats"
)
func main() {
var input string
var dsn string
var legacyTable string
var apply bool
var ensureSchema bool
var reportLimit int
var unresolvedOut string
var conflictsOut string
var syncDataSources bool
var pruneUnmanagedDataSources bool
var pruneDataSourceMinAge time.Duration
var retireStaleUnmanagedDataSources bool
var retireDataSourceMinAge time.Duration
var timeout time.Duration
flag.StringVar(&input, "input", env("IDENTITY_MAPPING_INPUT", ""), "directory that contains 808 plate/phone mapping workbooks")
flag.StringVar(&dsn, "mysql-dsn", env("IDENTITY_MYSQL_DSN", env("MYSQL_DSN", "")), "MySQL DSN; empty means scan only")
flag.StringVar(&legacyTable, "legacy-table", env("VEHICLE_IDENTITY_TABLE", "vehicle_identity_binding"), "legacy VIN/plate binding table used to resolve VIN")
flag.BoolVar(&apply, "apply", false, "write resolved mappings to vehicle and vehicle_identifier")
flag.BoolVar(&ensureSchema, "ensure-schema", false, "create vehicle and vehicle_identifier tables before import")
flag.IntVar(&reportLimit, "report-limit", 50, "max unresolved/conflict items embedded in JSON; use -1 for all")
flag.StringVar(&unresolvedOut, "unresolved-out", "", "optional CSV path for all unresolved mappings")
flag.StringVar(&conflictsOut, "conflicts-out", "", "optional CSV path for all conflict mappings")
flag.BoolVar(&syncDataSources, "sync-data-sources", false, "infer JT808 vehicle_data_source platform names from jt808_registration and vehicle_identifier")
flag.BoolVar(&pruneUnmanagedDataSources, "prune-unmanaged-data-sources", false, "delete stale unclassified vehicle_data_source rows that have no registration evidence and are not referenced by final mileage")
flag.DurationVar(&pruneDataSourceMinAge, "prune-data-source-min-age", time.Hour, "minimum latest_seen/updated age before -prune-unmanaged-data-sources can delete rows")
flag.BoolVar(&retireStaleUnmanagedDataSources, "retire-stale-unmanaged-data-sources", false, "disable stale unclassified vehicle_data_source rows that have no registration evidence but may be referenced by historical mileage")
flag.DurationVar(&retireDataSourceMinAge, "retire-data-source-min-age", 24*time.Hour, "minimum latest_seen/updated age before -retire-stale-unmanaged-data-sources can disable rows")
flag.DurationVar(&timeout, "timeout", 2*time.Minute, "import timeout")
flag.Parse()
ctx, cancel := context.WithTimeout(context.Background(), timeout)
defer cancel()
input = strings.TrimSpace(input)
output := map[string]any{}
var records []identity.MappingRecord
var scan identity.MappingScanReport
if input != "" {
var err error
records, scan, err = identity.ReadMappingDirectory(input)
if err != nil {
fail(err)
}
output["input"] = input
output["scan"] = scan
} else if !syncDataSources && !pruneUnmanagedDataSources && !retireStaleUnmanagedDataSources {
fail(errors.New("-input is required unless -sync-data-sources, -prune-unmanaged-data-sources or -retire-stale-unmanaged-data-sources is set"))
}
if err := validateMappingApplyInput(apply, scan); err != nil {
fail(err)
}
if strings.TrimSpace(dsn) == "" {
if syncDataSources || pruneUnmanagedDataSources || retireStaleUnmanagedDataSources {
fail(errors.New("-mysql-dsn is required when -sync-data-sources, -prune-unmanaged-data-sources or -retire-stale-unmanaged-data-sources is set"))
}
output["mode"] = "scan_only"
writeJSON(output)
return
}
db, err := sql.Open("mysql", dsn)
if err != nil {
fail(err)
}
defer func() { _ = db.Close() }()
if err := db.PingContext(ctx); err != nil {
fail(err)
}
if input != "" && (ensureSchema || apply) {
if err := identity.EnsureVehicleIdentifierSchema(ctx, db); err != nil {
fail(err)
}
}
if input != "" {
report, err := identity.ImportMappingRecords(ctx, db, records, scan, identity.MappingImportOptions{
Apply: apply,
LegacyTable: legacyTable,
ReportItemLimit: reportItemLimit(reportLimit, unresolvedOut, conflictsOut),
})
if err != nil {
fail(err)
}
if strings.TrimSpace(unresolvedOut) != "" {
if err := writeUnresolvedCSV(unresolvedOut, report.UnresolvedItems); err != nil {
fail(err)
}
}
if strings.TrimSpace(conflictsOut) != "" {
if err := writeConflictsCSV(conflictsOut, report.ConflictItems); err != nil {
fail(err)
}
}
output["report"] = report
output["mode"] = map[bool]string{true: "apply", false: "dry_run"}[apply]
}
if syncDataSources {
syncReport, err := syncJT808DataSourcesFromIdentifiers(ctx, db, apply)
if err != nil {
fail(err)
}
output["data_source_sync"] = syncReport
}
if pruneUnmanagedDataSources {
pruneReport, err := pruneUnmanagedDataSourcesWithoutEvidence(ctx, db, apply, pruneDataSourceMinAge)
if err != nil {
fail(err)
}
output["data_source_prune"] = pruneReport
}
if retireStaleUnmanagedDataSources {
retireReport, err := retireStaleUnmanagedDataSourcesWithoutEvidence(ctx, db, apply, retireDataSourceMinAge)
if err != nil {
fail(err)
}
output["data_source_retire"] = retireReport
}
if input == "" {
output["mode"] = maintenanceMode(apply, syncDataSources, pruneUnmanagedDataSources, retireStaleUnmanagedDataSources)
}
writeJSON(output)
}
func maintenanceMode(apply bool, syncDataSources bool, pruneUnmanagedDataSources bool, retireStaleUnmanagedDataSources bool) string {
var parts []string
if syncDataSources {
parts = append(parts, "sync")
}
if pruneUnmanagedDataSources {
parts = append(parts, "prune")
}
if retireStaleUnmanagedDataSources {
parts = append(parts, "retire")
}
if len(parts) == 0 {
return map[bool]string{true: "apply", false: "dry_run"}[apply]
}
parts = append(parts, map[bool]string{true: "apply", false: "dry_run"}[apply])
return strings.Join(parts, "_")
}
func reportItemLimit(limit int, unresolvedOut string, conflictsOut string) int {
if strings.TrimSpace(unresolvedOut) != "" || strings.TrimSpace(conflictsOut) != "" {
return -1
}
return limit
}
func validateMappingApplyInput(apply bool, scan identity.MappingScanReport) error {
if !apply {
return nil
}
return unsupportedMappingFilesError(scan)
}
func unsupportedMappingFilesError(scan identity.MappingScanReport) error {
if scan.UnsupportedFiles <= 0 {
return nil
}
items := make([]string, 0, len(scan.UnsupportedItems))
for _, item := range scan.UnsupportedItems {
path := strings.TrimSpace(item.File)
if path == "" {
path = strings.TrimSpace(item.Ext)
}
if path != "" {
items = append(items, path)
}
if len(items) >= 3 {
break
}
}
suffix := ""
if len(items) > 0 {
suffix = ": " + strings.Join(items, ", ")
}
return fmt.Errorf("identity mapping import has %d unsupported workbook(s); convert .xls/.xlsb to .xlsx before -apply%s", scan.UnsupportedFiles, suffix)
}
func env(key string, fallback string) string {
value := strings.TrimSpace(os.Getenv(key))
if value == "" {
return fallback
}
return value
}
func writeJSON(value any) {
encoder := json.NewEncoder(os.Stdout)
encoder.SetIndent("", " ")
if err := encoder.Encode(value); err != nil {
fail(err)
}
}
func writeUnresolvedCSV(path string, items []identity.MappingRecord) error {
file, err := os.Create(path)
if err != nil {
return err
}
defer file.Close()
writer := csv.NewWriter(file)
defer writer.Flush()
if err := writer.Write([]string{
"file", "sheet", "row", "source_code", "source_name", "protocol",
"identifier_type", "identifier_value", "raw_value", "plate", "oem", "reason",
}); err != nil {
return err
}
for _, item := range items {
if err := writer.Write([]string{
item.File,
item.Sheet,
fmt.Sprint(item.Row),
item.SourceCode,
item.SourceName,
item.Protocol,
item.IdentifierType,
item.IdentifierValue,
item.RawValue,
item.Plate,
item.OEM,
"vin_not_found_in_legacy_binding",
}); err != nil {
return err
}
}
return writer.Error()
}
func writeConflictsCSV(path string, items []identity.MappingConflict) error {
file, err := os.Create(path)
if err != nil {
return err
}
defer file.Close()
writer := csv.NewWriter(file)
defer writer.Flush()
if err := writer.Write([]string{
"file", "sheet", "row", "source_code", "source_name", "protocol",
"identifier_type", "identifier_value", "raw_value", "plate", "oem",
"existing_vin", "new_vin", "reason",
}); err != nil {
return err
}
for _, item := range items {
record := item.Record
if err := writer.Write([]string{
record.File,
record.Sheet,
fmt.Sprint(record.Row),
record.SourceCode,
record.SourceName,
record.Protocol,
record.IdentifierType,
record.IdentifierValue,
record.RawValue,
record.Plate,
record.OEM,
item.ExistingVIN,
item.NewVIN,
item.Reason,
}); err != nil {
return err
}
}
return writer.Error()
}
type dataSourceSyncReport struct {
Apply bool `json:"apply"`
CandidateSources int64 `json:"candidate_sources"`
SkippedSources int64 `json:"skipped_sources"`
ConflictingSources int64 `json:"conflicting_sources"`
PlatformKindCandidates int64 `json:"platform_kind_candidates"`
PlatformKindClassified int64 `json:"platform_kind_classified,omitempty"`
ReprojectDailyMileageTargets int64 `json:"reproject_daily_mileage_targets,omitempty"`
ReprojectedDailyMileageTargets int64 `json:"reprojected_daily_mileage_targets,omitempty"`
Synced int64 `json:"synced,omitempty"`
}
type dailyMileageProjectionTarget struct {
VIN string
StatDate string
Protocol envelope.Protocol
}
type dataSourcePruneReport struct {
Apply bool `json:"apply"`
MinAgeSeconds int64 `json:"min_age_seconds"`
CandidateSources int64 `json:"candidate_sources"`
Pruned int64 `json:"pruned,omitempty"`
}
type dataSourceRetireReport struct {
Apply bool `json:"apply"`
MinAgeSeconds int64 `json:"min_age_seconds"`
CandidateSources int64 `json:"candidate_sources"`
Retired int64 `json:"retired,omitempty"`
}
func syncJT808DataSourcesFromIdentifiers(ctx context.Context, db *sql.DB, apply bool) (dataSourceSyncReport, error) {
report := dataSourceSyncReport{Apply: apply}
var err error
report.CandidateSources, err = queryInt64(ctx, db, syncJT808DataSourcesCandidateCountSQL)
if err != nil {
return report, err
}
report.SkippedSources, err = queryInt64(ctx, db, syncJT808DataSourcesSkippedCountSQL)
if err != nil {
return report, err
}
report.ConflictingSources, err = queryInt64(ctx, db, syncJT808DataSourcesConflictCountSQL)
if err != nil {
return report, err
}
if apply {
result, err := db.ExecContext(ctx, syncJT808DataSourcesSQL)
if err != nil {
return report, err
}
rowsAffected, err := result.RowsAffected()
if err == nil {
report.Synced = rowsAffected
}
}
report.PlatformKindCandidates, err = queryInt64(ctx, db, classifyConfiguredDataSourcesCountSQL)
if err != nil {
return report, err
}
if apply {
result, err := db.ExecContext(ctx, classifyConfiguredDataSourcesSQL)
if err != nil {
return report, err
}
rowsAffected, err := result.RowsAffected()
if err == nil {
report.PlatformKindClassified = rowsAffected
}
}
targets, err := queryDailyMileageProjectionTargets(ctx, db)
if err != nil {
return report, err
}
report.ReprojectDailyMileageTargets = int64(len(targets))
if apply {
for _, target := range targets {
if err := stats.ProjectDailyMileage(ctx, db, target.VIN, target.StatDate, target.Protocol); err != nil {
return report, err
}
report.ReprojectedDailyMileageTargets++
}
}
return report, nil
}
func queryDailyMileageProjectionTargets(ctx context.Context, db *sql.DB) ([]dailyMileageProjectionTarget, error) {
rows, err := db.QueryContext(ctx, reprojectSelectableDailyMileageSourcesSQL)
if err != nil {
return nil, err
}
defer rows.Close()
var targets []dailyMileageProjectionTarget
for rows.Next() {
var target dailyMileageProjectionTarget
var protocol string
if err := rows.Scan(&target.VIN, &target.StatDate, &protocol); err != nil {
return nil, err
}
target.Protocol = envelope.Protocol(strings.TrimSpace(protocol))
if strings.TrimSpace(target.VIN) != "" && strings.TrimSpace(target.StatDate) != "" && strings.TrimSpace(string(target.Protocol)) != "" {
targets = append(targets, target)
}
}
return targets, rows.Err()
}
func pruneUnmanagedDataSourcesWithoutEvidence(ctx context.Context, db *sql.DB, apply bool, minAge time.Duration) (dataSourcePruneReport, error) {
if minAge < 0 {
minAge = 0
}
minAgeSeconds := int64(minAge.Seconds())
report := dataSourcePruneReport{Apply: apply, MinAgeSeconds: minAgeSeconds}
var err error
report.CandidateSources, err = queryInt64WithArgs(ctx, db, pruneUnmanagedDataSourcesCountSQL, minAgeSeconds)
if err != nil {
return report, err
}
if !apply {
return report, nil
}
result, err := db.ExecContext(ctx, pruneUnmanagedDataSourcesSQL, minAgeSeconds)
if err != nil {
return report, err
}
rowsAffected, err := result.RowsAffected()
if err == nil {
report.Pruned = rowsAffected
}
return report, nil
}
func retireStaleUnmanagedDataSourcesWithoutEvidence(ctx context.Context, db *sql.DB, apply bool, minAge time.Duration) (dataSourceRetireReport, error) {
if minAge < 0 {
minAge = 0
}
minAgeSeconds := int64(minAge.Seconds())
report := dataSourceRetireReport{Apply: apply, MinAgeSeconds: minAgeSeconds}
var err error
report.CandidateSources, err = queryInt64WithArgs(ctx, db, retireStaleUnmanagedDataSourcesCountSQL, minAgeSeconds)
if err != nil {
return report, err
}
if !apply {
return report, nil
}
result, err := db.ExecContext(ctx, retireStaleUnmanagedDataSourcesSQL, minAgeSeconds)
if err != nil {
return report, err
}
rowsAffected, err := result.RowsAffected()
if err == nil {
report.Retired = rowsAffected
}
return report, nil
}
func queryInt64(ctx context.Context, db *sql.DB, query string) (int64, error) {
return queryInt64WithArgs(ctx, db, query)
}
func queryInt64WithArgs(ctx context.Context, db *sql.DB, query string, args ...any) (int64, error) {
var count int64
err := db.QueryRowContext(ctx, query, args...).Scan(&count)
return count, err
}
const jt808DataSourceInferenceSQL = `
SELECT
'JT808' AS protocol,
TRIM(r.source_ip) AS source_ip,
MAX(TRIM(r.source_endpoint)) AS latest_source_endpoint,
MIN(COALESCE(NULLIF(TRIM(vi.oem), ''), NULLIF(TRIM(vi.source_code), ''))) AS platform_name,
MIN(NULLIF(TRIM(vi.source_code), '')) AS source_code,
MIN(COALESCE(r.first_registered_at, r.latest_registered_at, r.latest_authenticated_at, r.latest_seen_at, r.updated_at, CURRENT_TIMESTAMP)) AS first_seen_at,
MAX(COALESCE(r.latest_seen_at, r.latest_authenticated_at, r.latest_registered_at, r.updated_at, CURRENT_TIMESTAMP)) AS latest_seen_at,
COUNT(DISTINCT COALESCE(NULLIF(TRIM(vi.oem), ''), NULLIF(TRIM(vi.source_code), ''))) AS platform_count,
COUNT(DISTINCT NULLIF(TRIM(vi.source_code), '')) AS source_code_count
FROM jt808_registration r
JOIN vehicle_identifier vi
ON vi.protocol = 'JT808'
AND vi.identifier_type = 'JT808_PHONE'
AND vi.identifier_value = r.phone
AND vi.enabled = 1
WHERE r.source_endpoint IS NOT NULL
AND TRIM(r.source_endpoint) <> ''
AND r.source_ip IS NOT NULL
AND TRIM(r.source_ip) <> ''
GROUP BY TRIM(r.source_ip)
`
const jt808DataSourceCandidateWhereSQL = `inferred.source_ip IS NOT NULL
AND inferred.source_ip <> ''
AND inferred.platform_name IS NOT NULL
AND inferred.platform_name <> ''
AND inferred.platform_count = 1
AND inferred.source_code_count = 1`
const jt808DataSourceNeedsSyncWhereSQL = `(ds.id IS NULL
OR ds.platform_name IS NULL OR TRIM(ds.platform_name) = ''
OR ds.source_code IS NULL OR TRIM(ds.source_code) = ''
OR ds.source_kind IS NULL OR TRIM(ds.source_kind) = '' OR ds.source_kind = 'UNKNOWN'
OR (ds.enabled = 0 AND (ds.remark LIKE 'auto-retired:%' OR ds.remark = 'auto-reenabled: source evidence restored')))`
const syncJT808DataSourcesSQL = `
INSERT INTO vehicle_data_source
(protocol, source_ip, latest_source_endpoint, platform_name, source_code, source_kind, first_seen_at, latest_seen_at)
SELECT
inferred.protocol,
inferred.source_ip,
inferred.latest_source_endpoint,
inferred.platform_name,
inferred.source_code,
'PLATFORM',
inferred.first_seen_at,
inferred.latest_seen_at
FROM (` + jt808DataSourceInferenceSQL + `) inferred
LEFT JOIN vehicle_data_source ds
ON ds.protocol = inferred.protocol
AND ds.source_ip = inferred.source_ip
WHERE ` + jt808DataSourceCandidateWhereSQL + `
AND ` + jt808DataSourceNeedsSyncWhereSQL + `
ON DUPLICATE KEY UPDATE
latest_source_endpoint = VALUES(latest_source_endpoint),
platform_name = CASE
WHEN vehicle_data_source.platform_name IS NULL OR TRIM(vehicle_data_source.platform_name) = ''
THEN VALUES(platform_name)
ELSE vehicle_data_source.platform_name
END,
source_code = CASE
WHEN vehicle_data_source.source_code IS NULL OR TRIM(vehicle_data_source.source_code) = ''
THEN VALUES(source_code)
ELSE vehicle_data_source.source_code
END,
source_kind = CASE
WHEN vehicle_data_source.source_kind IS NULL OR TRIM(vehicle_data_source.source_kind) = '' OR vehicle_data_source.source_kind = 'UNKNOWN'
THEN VALUES(source_kind)
ELSE vehicle_data_source.source_kind
END,
enabled = CASE
WHEN vehicle_data_source.enabled = 0
AND (vehicle_data_source.remark LIKE 'auto-retired:%' OR vehicle_data_source.remark = 'auto-reenabled: source evidence restored')
AND (
(VALUES(platform_name) IS NOT NULL AND TRIM(VALUES(platform_name)) <> '')
OR (VALUES(source_code) IS NOT NULL AND TRIM(VALUES(source_code)) <> '')
OR VALUES(source_kind) <> 'UNKNOWN'
)
THEN 1
ELSE vehicle_data_source.enabled
END,
remark = CASE
WHEN vehicle_data_source.enabled = 0
AND (vehicle_data_source.remark LIKE 'auto-retired:%' OR vehicle_data_source.remark = 'auto-reenabled: source evidence restored')
AND (
(VALUES(platform_name) IS NOT NULL AND TRIM(VALUES(platform_name)) <> '')
OR (VALUES(source_code) IS NOT NULL AND TRIM(VALUES(source_code)) <> '')
OR VALUES(source_kind) <> 'UNKNOWN'
)
THEN 'auto-reenabled: source evidence restored'
ELSE vehicle_data_source.remark
END,
first_seen_at = COALESCE(vehicle_data_source.first_seen_at, VALUES(first_seen_at)),
latest_seen_at = GREATEST(COALESCE(vehicle_data_source.latest_seen_at, VALUES(latest_seen_at)), VALUES(latest_seen_at)),
updated_at = CURRENT_TIMESTAMP
`
const reprojectSelectableDailyMileageSourcesSQL = `
SELECT DISTINCT s.vin, s.stat_date, s.protocol
FROM vehicle_daily_mileage_source s
LEFT JOIN vehicle_data_source ds
ON ds.protocol = s.protocol AND ds.source_ip = s.source_ip
LEFT JOIN vehicle_daily_mileage m
ON m.vin = s.vin AND m.stat_date = s.stat_date AND m.protocol = s.protocol
WHERE s.protocol = 'JT808'
AND s.quality_status = 'OK'
AND (ds.id IS NULL OR ds.enabled = 1 OR (
COALESCE(NULLIF(TRIM(ds.source_kind), ''), 'UNKNOWN') = 'UNKNOWN'
AND (ds.source_code IS NULL OR TRIM(ds.source_code) = '')
AND (ds.platform_name IS NULL OR TRIM(ds.platform_name) = '')
))
AND (
m.vin IS NULL
OR NOT EXISTS (
SELECT 1
FROM vehicle_daily_mileage_source selected
WHERE selected.vin = s.vin
AND selected.stat_date = s.stat_date
AND selected.protocol = s.protocol
AND selected.is_selected = 1
)
)
ORDER BY s.stat_date DESC, s.vin ASC
`
const syncJT808DataSourcesCandidateCountSQL = `
SELECT COUNT(*)
FROM (` + jt808DataSourceInferenceSQL + `) inferred
LEFT JOIN vehicle_data_source ds
ON ds.protocol = inferred.protocol
AND ds.source_ip = inferred.source_ip
WHERE ` + jt808DataSourceCandidateWhereSQL + `
AND ` + jt808DataSourceNeedsSyncWhereSQL
const syncJT808DataSourcesSkippedCountSQL = `
SELECT COUNT(*)
FROM (` + jt808DataSourceInferenceSQL + `) inferred
WHERE inferred.source_ip IS NOT NULL
AND inferred.source_ip <> ''
AND inferred.platform_name IS NOT NULL
AND inferred.platform_name <> ''
AND NOT (` + jt808DataSourceCandidateWhereSQL + `)`
const syncJT808DataSourcesConflictCountSQL = `
SELECT COUNT(*)
FROM (` + jt808DataSourceInferenceSQL + `) inferred
JOIN vehicle_data_source ds
ON ds.protocol = inferred.protocol
AND ds.source_ip = inferred.source_ip
WHERE ` + jt808DataSourceCandidateWhereSQL + `
AND ds.source_code IS NOT NULL
AND TRIM(ds.source_code) <> ''
AND ds.source_code <> inferred.source_code`
const classifyConfiguredDataSourcesWhereSQL = `
(source_kind IS NULL OR TRIM(source_kind) = '' OR source_kind = 'UNKNOWN')
AND (
(source_code IS NOT NULL AND TRIM(source_code) <> '')
OR (platform_name IS NOT NULL AND TRIM(platform_name) <> '')
)`
const classifyConfiguredDataSourcesCountSQL = `
SELECT COUNT(*)
FROM vehicle_data_source
WHERE ` + classifyConfiguredDataSourcesWhereSQL
const classifyConfiguredDataSourcesSQL = `
UPDATE vehicle_data_source
SET source_kind = 'PLATFORM',
updated_at = CURRENT_TIMESTAMP
WHERE ` + classifyConfiguredDataSourcesWhereSQL
const pruneUnmanagedDataSourcesWhereSQL = `
(ds.platform_name IS NULL OR TRIM(ds.platform_name) = '')
AND (ds.source_code IS NULL OR TRIM(ds.source_code) = '')
AND (ds.source_kind IS NULL OR TRIM(ds.source_kind) = '' OR ds.source_kind = 'UNKNOWN')
AND TIMESTAMPDIFF(SECOND, COALESCE(ds.latest_seen_at, ds.updated_at, ds.created_at), CURRENT_TIMESTAMP) >= ?
AND NOT EXISTS (
SELECT 1
FROM vehicle_daily_mileage m
WHERE m.source_id = ds.id
LIMIT 1
)
AND NOT EXISTS (
SELECT 1
FROM jt808_registration r
WHERE ds.protocol = 'JT808'
AND r.source_ip = ds.source_ip
LIMIT 1
)`
const pruneUnmanagedDataSourcesCountSQL = `
SELECT COUNT(*)
FROM vehicle_data_source ds
WHERE ` + pruneUnmanagedDataSourcesWhereSQL
const pruneUnmanagedDataSourcesSQL = `
DELETE ds
FROM vehicle_data_source ds
WHERE ` + pruneUnmanagedDataSourcesWhereSQL
const retireStaleUnmanagedDataSourcesWhereSQL = `
ds.enabled = 1
AND (ds.platform_name IS NULL OR TRIM(ds.platform_name) = '')
AND (ds.source_code IS NULL OR TRIM(ds.source_code) = '')
AND (ds.source_kind IS NULL OR TRIM(ds.source_kind) = '' OR ds.source_kind = 'UNKNOWN')
AND TIMESTAMPDIFF(SECOND, COALESCE(ds.latest_seen_at, ds.updated_at, ds.created_at), CURRENT_TIMESTAMP) >= ?
AND NOT EXISTS (
SELECT 1
FROM jt808_registration r
WHERE ds.protocol = 'JT808'
AND r.source_ip = ds.source_ip
LIMIT 1
)`
const retireStaleUnmanagedDataSourcesCountSQL = `
SELECT COUNT(*)
FROM vehicle_data_source ds
WHERE ` + retireStaleUnmanagedDataSourcesWhereSQL
const retireStaleUnmanagedDataSourcesSQL = `
UPDATE vehicle_data_source ds
SET enabled = 0,
remark = CASE
WHEN remark IS NULL OR TRIM(remark) = ''
THEN 'auto-retired: stale unmanaged source without registration evidence'
ELSE remark
END,
updated_at = CURRENT_TIMESTAMP
WHERE ` + retireStaleUnmanagedDataSourcesWhereSQL
func fail(err error) {
_, _ = fmt.Fprintln(os.Stderr, err)
os.Exit(1)
}

426
go/vehicle-gateway/cmd/identity-import/main_test.go Normal file
View File

@@ -0,0 +1,426 @@
package main
import (
"context"
"database/sql/driver"
"os"
"path/filepath"
"strings"
"testing"
"time"
"github.com/DATA-DOG/go-sqlmock"
"lingniu-vehicle-ingest/go/vehicle-gateway/internal/identity"
)
func TestReportItemLimitExpandsWhenCSVOutputIsRequested(t *testing.T) {
if got := reportItemLimit(50, "", ""); got != 50 {
t.Fatalf("reportItemLimit without csv = %d, want 50", got)
}
if got := reportItemLimit(50, "unresolved.csv", ""); got != -1 {
t.Fatalf("reportItemLimit with unresolved csv = %d, want -1", got)
}
if got := reportItemLimit(50, "", "conflicts.csv"); got != -1 {
t.Fatalf("reportItemLimit with conflicts csv = %d, want -1", got)
}
}
func TestUnsupportedMappingFilesError(t *testing.T) {
err := unsupportedMappingFilesError(identity.MappingScanReport{})
if err != nil {
t.Fatalf("unsupportedMappingFilesError(empty) = %v", err)
}
err = unsupportedMappingFilesError(identity.MappingScanReport{
UnsupportedFiles: 2,
UnsupportedItems: []identity.MappingUnsupportedFileReport{
{File: "G7s/legacy.xls", Ext: ".xls"},
{File: "信达/legacy.xlsb", Ext: ".xlsb"},
},
})
if err == nil {
t.Fatal("unsupportedMappingFilesError() nil, want error")
}
text := err.Error()
for _, want := range []string{"2 unsupported workbook", "G7s/legacy.xls", "信达/legacy.xlsb", "before -apply"} {
if !strings.Contains(text, want) {
t.Fatalf("error missing %q: %s", want, text)
}
}
}
func TestValidateMappingApplyInputAllowsDryRunButBlocksApply(t *testing.T) {
scan := identity.MappingScanReport{
UnsupportedFiles: 1,
UnsupportedItems: []identity.MappingUnsupportedFileReport{
{File: "G7s/legacy.xls", Ext: ".xls"},
},
}
if err := validateMappingApplyInput(false, scan); err != nil {
t.Fatalf("validateMappingApplyInput(dry-run) = %v", err)
}
if err := validateMappingApplyInput(true, scan); err == nil {
t.Fatal("validateMappingApplyInput(apply) nil, want error")
}
}
func TestWriteUnresolvedCSV(t *testing.T) {
path := filepath.Join(t.TempDir(), "unresolved.csv")
err := writeUnresolvedCSV(path, []identity.MappingRecord{
{
File: "G7s/example.xlsx",
Sheet: "Sheet1",
Row: 2,
SourceCode: "g7s",
SourceName: "G7s",
Protocol: "JT808",
IdentifierType: identity.IdentifierTypeJT808Phone,
IdentifierValue: "13307795425",
RawValue: "013307795425",
Plate: "粤AG18312",
OEM: "G7s",
},
})
if err != nil {
t.Fatalf("writeUnresolvedCSV() error = %v", err)
}
data, err := os.ReadFile(path)
if err != nil {
t.Fatalf("ReadFile() error = %v", err)
}
text := string(data)
for _, want := range []string{"identifier_type", "JT808_PHONE", "13307795425", "vin_not_found_in_legacy_binding"} {
if !strings.Contains(text, want) {
t.Fatalf("csv missing %q:\n%s", want, text)
}
}
}
func TestWriteConflictsCSV(t *testing.T) {
path := filepath.Join(t.TempDir(), "conflicts.csv")
err := writeConflictsCSV(path, []identity.MappingConflict{
{
Record: identity.MappingRecord{
File: "source.xlsx",
Sheet: "Sheet1",
Row: 3,
SourceCode: "xinda",
Protocol: "JT808",
IdentifierType: identity.IdentifierTypePlate,
IdentifierValue: "粤AG18312",
Plate: "粤AG18312",
},
ExistingVIN: "VIN001",
NewVIN: "VIN002",
Reason: "identifier already points to another vin",
},
})
if err != nil {
t.Fatalf("writeConflictsCSV() error = %v", err)
}
data, err := os.ReadFile(path)
if err != nil {
t.Fatalf("ReadFile() error = %v", err)
}
text := string(data)
for _, want := range []string{"existing_vin", "VIN001", "VIN002", "identifier already points to another vin"} {
if !strings.Contains(text, want) {
t.Fatalf("csv missing %q:\n%s", want, text)
}
}
}
func TestSyncJT808DataSourcesFromIdentifiersPreservesManualPlatformNames(t *testing.T) {
db, mock, err := sqlmock.New()
if err != nil {
t.Fatalf("sqlmock.New() error = %v", err)
}
defer db.Close()
mock.ExpectQuery("SELECT COUNT\\(\\*\\)").
WillReturnRows(sqlmock.NewRows([]string{"count"}).AddRow(3))
mock.ExpectQuery("SELECT COUNT\\(\\*\\)").
WillReturnRows(sqlmock.NewRows([]string{"count"}).AddRow(1))
mock.ExpectQuery("SELECT COUNT\\(\\*\\)").
WillReturnRows(sqlmock.NewRows([]string{"count"}).AddRow(2))
mock.ExpectExec("INSERT INTO vehicle_data_source").
WillReturnResult(driver.RowsAffected(3))
mock.ExpectQuery("SELECT COUNT\\(\\*\\)").
WillReturnRows(sqlmock.NewRows([]string{"count"}).AddRow(5))
mock.ExpectExec("UPDATE vehicle_data_source").
WillReturnResult(driver.RowsAffected(5))
mock.ExpectQuery("SELECT DISTINCT s.vin, s.stat_date, s.protocol").
WillReturnRows(sqlmock.NewRows([]string{"vin", "stat_date", "protocol"}).
AddRow("LNXNEGRRXSR319449", "2026-07-13", "JT808"))
mock.ExpectBegin()
mock.ExpectExec("INSERT INTO vehicle_daily_mileage").
WillReturnResult(driver.RowsAffected(1))
mock.ExpectExec("UPDATE vehicle_daily_mileage_source s").
WillReturnResult(driver.RowsAffected(1))
mock.ExpectExec("DELETE FROM vehicle_daily_mileage").
WillReturnResult(driver.RowsAffected(0))
mock.ExpectCommit()
report, err := syncJT808DataSourcesFromIdentifiers(context.Background(), db, true)
if err != nil {
t.Fatalf("syncJT808DataSourcesFromIdentifiers() error = %v", err)
}
if !report.Apply || report.CandidateSources != 3 || report.SkippedSources != 1 || report.ConflictingSources != 2 || report.Synced != 3 || report.PlatformKindCandidates != 5 || report.PlatformKindClassified != 5 || report.ReprojectDailyMileageTargets != 1 || report.ReprojectedDailyMileageTargets != 1 {
t.Fatalf("report = %#v", report)
}
for _, want := range []string{
"COUNT(DISTINCT",
"vehicle_identifier vi",
"TRIM(r.source_ip) AS source_ip",
"GROUP BY TRIM(r.source_ip)",
"inferred.source_code",
"'PLATFORM'",
"inferred.source_code_count = 1",
"vehicle_data_source.source_code IS NULL OR TRIM(vehicle_data_source.source_code) = ''",
"ELSE vehicle_data_source.source_code",
"vehicle_data_source.source_kind IS NULL OR TRIM(vehicle_data_source.source_kind) = '' OR vehicle_data_source.source_kind = 'UNKNOWN'",
"ELSE vehicle_data_source.source_kind",
"vehicle_data_source.enabled = 0",
"vehicle_data_source.remark LIKE 'auto-retired:%'",
"vehicle_data_source.remark = 'auto-reenabled: source evidence restored'",
"THEN 'auto-reenabled: source evidence restored'",
"THEN 1",
"LEFT JOIN vehicle_data_source ds",
"ds.id IS NULL",
} {
if !strings.Contains(syncJT808DataSourcesSQL, want) {
t.Fatalf("sync sql missing %q:\n%s", want, syncJT808DataSourcesSQL)
}
}
if strings.Index(syncJT808DataSourcesSQL, "enabled = CASE") < 0 ||
strings.Index(syncJT808DataSourcesSQL, "remark = CASE") < 0 ||
strings.Index(syncJT808DataSourcesSQL, "enabled = CASE") > strings.Index(syncJT808DataSourcesSQL, "remark = CASE") {
t.Fatalf("sync sql must restore enabled before updating remark because MySQL evaluates assignments in order:\n%s", syncJT808DataSourcesSQL)
}
for _, want := range []string{
"LEFT JOIN vehicle_data_source ds",
"ds.id IS NULL",
"ds.source_code IS NULL OR TRIM(ds.source_code) = ''",
"ds.enabled = 0 AND (ds.remark LIKE 'auto-retired:%'",
"ds.remark = 'auto-reenabled: source evidence restored'",
} {
if !strings.Contains(syncJT808DataSourcesCandidateCountSQL, want) {
t.Fatalf("candidate count sql missing %q:\n%s", want, syncJT808DataSourcesCandidateCountSQL)
}
}
for _, want := range []string{
"JOIN vehicle_data_source ds",
"ds.source_code <> inferred.source_code",
} {
if !strings.Contains(syncJT808DataSourcesConflictCountSQL, want) {
t.Fatalf("conflict count sql missing %q:\n%s", want, syncJT808DataSourcesConflictCountSQL)
}
}
for _, want := range []string{
"vehicle_data_source.platform_name IS NULL OR TRIM(vehicle_data_source.platform_name) = ''",
"ELSE vehicle_data_source.platform_name",
} {
if !strings.Contains(syncJT808DataSourcesSQL, want) {
t.Fatalf("sync sql missing %q:\n%s", want, syncJT808DataSourcesSQL)
}
}
for _, want := range []string{
"source_kind IS NULL OR TRIM(source_kind) = '' OR source_kind = 'UNKNOWN'",
"source_code IS NOT NULL AND TRIM(source_code) <> ''",
"platform_name IS NOT NULL AND TRIM(platform_name) <> ''",
"SET source_kind = 'PLATFORM'",
} {
if !strings.Contains(classifyConfiguredDataSourcesSQL, want) && !strings.Contains(classifyConfiguredDataSourcesCountSQL, want) {
t.Fatalf("configured-source classification sql missing %q:\n%s\n%s", want, classifyConfiguredDataSourcesSQL, classifyConfiguredDataSourcesCountSQL)
}
}
for _, want := range []string{
"SELECT DISTINCT s.vin, s.stat_date, s.protocol",
"vehicle_daily_mileage_source s",
"LEFT JOIN vehicle_data_source ds",
"LEFT JOIN vehicle_daily_mileage m",
"s.protocol = 'JT808'",
"s.quality_status = 'OK'",
"ds.enabled = 1",
"m.vin IS NULL",
"selected.is_selected = 1",
} {
if !strings.Contains(reprojectSelectableDailyMileageSourcesSQL, want) {
t.Fatalf("reproject sql missing %q:\n%s", want, reprojectSelectableDailyMileageSourcesSQL)
}
}
if err := mock.ExpectationsWereMet(); err != nil {
t.Fatalf("sql expectations: %v", err)
}
}
func TestSyncJT808DataSourcesDryRunDoesNotWrite(t *testing.T) {
db, mock, err := sqlmock.New()
if err != nil {
t.Fatalf("sqlmock.New() error = %v", err)
}
defer db.Close()
mock.ExpectQuery("SELECT COUNT\\(\\*\\)").
WillReturnRows(sqlmock.NewRows([]string{"count"}).AddRow(4))
mock.ExpectQuery("SELECT COUNT\\(\\*\\)").
WillReturnRows(sqlmock.NewRows([]string{"count"}).AddRow(2))
mock.ExpectQuery("SELECT COUNT\\(\\*\\)").
WillReturnRows(sqlmock.NewRows([]string{"count"}).AddRow(1))
mock.ExpectQuery("SELECT COUNT\\(\\*\\)").
WillReturnRows(sqlmock.NewRows([]string{"count"}).AddRow(6))
mock.ExpectQuery("SELECT DISTINCT s.vin, s.stat_date, s.protocol").
WillReturnRows(sqlmock.NewRows([]string{"vin", "stat_date", "protocol"}).
AddRow("LNXNEGRRXSR319449", "2026-07-13", "JT808"))
report, err := syncJT808DataSourcesFromIdentifiers(context.Background(), db, false)
if err != nil {
t.Fatalf("syncJT808DataSourcesFromIdentifiers() error = %v", err)
}
if report.Apply || report.CandidateSources != 4 || report.SkippedSources != 2 || report.ConflictingSources != 1 || report.PlatformKindCandidates != 6 || report.PlatformKindClassified != 0 || report.Synced != 0 || report.ReprojectDailyMileageTargets != 1 || report.ReprojectedDailyMileageTargets != 0 {
t.Fatalf("report = %#v", report)
}
if err := mock.ExpectationsWereMet(); err != nil {
t.Fatalf("sql expectations: %v", err)
}
}
func TestMaintenanceModeNamesAllSourceMaintenanceSteps(t *testing.T) {
tests := []struct {
name string
apply bool
sync bool
prune bool
retire bool
want string
}{
{name: "dry run", want: "dry_run"},
{name: "apply", apply: true, want: "apply"},
{name: "sync dry run", sync: true, want: "sync_dry_run"},
{name: "sync prune retire apply", apply: true, sync: true, prune: true, retire: true, want: "sync_prune_retire_apply"},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
if got := maintenanceMode(tt.apply, tt.sync, tt.prune, tt.retire); got != tt.want {
t.Fatalf("maintenanceMode() = %q, want %q", got, tt.want)
}
})
}
}
func TestPruneUnmanagedDataSourcesDryRunDoesNotDelete(t *testing.T) {
db, mock, err := sqlmock.New()
if err != nil {
t.Fatalf("sqlmock.New() error = %v", err)
}
defer db.Close()
mock.ExpectQuery("SELECT COUNT\\(\\*\\)").
WithArgs(int64(3600)).
WillReturnRows(sqlmock.NewRows([]string{"count"}).AddRow(339))
report, err := pruneUnmanagedDataSourcesWithoutEvidence(context.Background(), db, false, time.Hour)
if err != nil {
t.Fatalf("pruneUnmanagedDataSourcesWithoutEvidence() error = %v", err)
}
if report.Apply || report.MinAgeSeconds != 3600 || report.CandidateSources != 339 || report.Pruned != 0 {
t.Fatalf("report = %#v", report)
}
for _, want := range []string{
"vehicle_data_source ds",
"vehicle_daily_mileage m",
"m.source_id = ds.id",
"jt808_registration r",
"r.source_ip = ds.source_ip",
"ds.source_kind IS NULL OR TRIM(ds.source_kind) = '' OR ds.source_kind = 'UNKNOWN'",
"TIMESTAMPDIFF(SECOND",
} {
if !strings.Contains(pruneUnmanagedDataSourcesCountSQL, want) {
t.Fatalf("prune count sql missing %q:\n%s", want, pruneUnmanagedDataSourcesCountSQL)
}
}
if err := mock.ExpectationsWereMet(); err != nil {
t.Fatalf("sql expectations: %v", err)
}
}
func TestRetireStaleUnmanagedDataSourcesDryRunDoesNotDisable(t *testing.T) {
db, mock, err := sqlmock.New()
if err != nil {
t.Fatalf("sqlmock.New() error = %v", err)
}
defer db.Close()
mock.ExpectQuery("SELECT COUNT\\(\\*\\)").
WithArgs(int64(86400)).
WillReturnRows(sqlmock.NewRows([]string{"count"}).AddRow(83))
report, err := retireStaleUnmanagedDataSourcesWithoutEvidence(context.Background(), db, false, 24*time.Hour)
if err != nil {
t.Fatalf("retireStaleUnmanagedDataSourcesWithoutEvidence() error = %v", err)
}
if report.Apply || report.MinAgeSeconds != 86400 || report.CandidateSources != 83 || report.Retired != 0 {
t.Fatalf("report = %#v", report)
}
for _, want := range []string{
"vehicle_data_source ds",
"ds.enabled = 1",
"jt808_registration r",
"r.source_ip = ds.source_ip",
"ds.source_kind IS NULL OR TRIM(ds.source_kind) = '' OR ds.source_kind = 'UNKNOWN'",
"TIMESTAMPDIFF(SECOND",
} {
if !strings.Contains(retireStaleUnmanagedDataSourcesCountSQL, want) {
t.Fatalf("retire count sql missing %q:\n%s", want, retireStaleUnmanagedDataSourcesCountSQL)
}
}
if err := mock.ExpectationsWereMet(); err != nil {
t.Fatalf("sql expectations: %v", err)
}
}
func TestRetireStaleUnmanagedDataSourcesApplyDisablesOnlyCandidates(t *testing.T) {
db, mock, err := sqlmock.New()
if err != nil {
t.Fatalf("sqlmock.New() error = %v", err)
}
defer db.Close()
mock.ExpectQuery("SELECT COUNT\\(\\*\\)").
WithArgs(int64(7200)).
WillReturnRows(sqlmock.NewRows([]string{"count"}).AddRow(7))
mock.ExpectExec("UPDATE vehicle_data_source ds").
WithArgs(int64(7200)).
WillReturnResult(driver.RowsAffected(7))
report, err := retireStaleUnmanagedDataSourcesWithoutEvidence(context.Background(), db, true, 2*time.Hour)
if err != nil {
t.Fatalf("retireStaleUnmanagedDataSourcesWithoutEvidence() error = %v", err)
}
if !report.Apply || report.MinAgeSeconds != 7200 || report.CandidateSources != 7 || report.Retired != 7 {
t.Fatalf("report = %#v", report)
}
if err := mock.ExpectationsWereMet(); err != nil {
t.Fatalf("sql expectations: %v", err)
}
}
func TestPruneUnmanagedDataSourcesApplyDeletesOnlyCandidates(t *testing.T) {
db, mock, err := sqlmock.New()
if err != nil {
t.Fatalf("sqlmock.New() error = %v", err)
}
defer db.Close()
mock.ExpectQuery("SELECT COUNT\\(\\*\\)").
WithArgs(int64(1800)).
WillReturnRows(sqlmock.NewRows([]string{"count"}).AddRow(12))
mock.ExpectExec("DELETE ds\\s+FROM vehicle_data_source ds").
WithArgs(int64(1800)).
WillReturnResult(driver.RowsAffected(12))
report, err := pruneUnmanagedDataSourcesWithoutEvidence(context.Background(), db, true, 30*time.Minute)
if err != nil {
t.Fatalf("pruneUnmanagedDataSourcesWithoutEvidence() error = %v", err)
}
if !report.Apply || report.MinAgeSeconds != 1800 || report.CandidateSources != 12 || report.Pruned != 12 {
t.Fatalf("report = %#v", report)
}
if err := mock.ExpectationsWereMet(); err != nil {
t.Fatalf("sql expectations: %v", err)
}
}

512
go/vehicle-gateway/cmd/identity-writer/main.go Normal file
View File

@@ -0,0 +1,512 @@
package main
import (
"context"
"database/sql"
"encoding/json"
"errors"
"fmt"
"log/slog"
"os"
"os/signal"
"strconv"
"strings"
"sync"
"syscall"
"time"
_ "github.com/go-sql-driver/mysql"
"github.com/segmentio/kafka-go"
"lingniu-vehicle-ingest/go/vehicle-gateway/internal/envelope"
"lingniu-vehicle-ingest/go/vehicle-gateway/internal/health"
"lingniu-vehicle-ingest/go/vehicle-gateway/internal/identity"
"lingniu-vehicle-ingest/go/vehicle-gateway/internal/metrics"
"lingniu-vehicle-ingest/go/vehicle-gateway/internal/observability"
"lingniu-vehicle-ingest/go/vehicle-gateway/internal/topics"
)
const identityBatchOperationTimeout = 30 * time.Second
func main() {
logger := observability.NewLogger("vehicle-identity-writer")
ctx, stop := signal.NotifyContext(context.Background(), os.Interrupt, syscall.SIGTERM)
defer stop()
cfg := loadConfig()
if err := cfg.Validate(); err != nil {
logger.Error("invalid identity writer config", "error", err)
os.Exit(1)
}
db, err := sql.Open("mysql", cfg.MySQLDSN)
if err != nil {
logger.Error("mysql open failed", "error", err)
os.Exit(1)
}
defer db.Close()
db.SetMaxOpenConns(cfg.MySQLMaxOpenConns)
db.SetMaxIdleConns(cfg.MySQLMaxIdleConns)
db.SetConnMaxLifetime(cfg.MySQLConnMaxLifetime)
if err := db.PingContext(ctx); err != nil {
logger.Error("mysql ping failed", "error", err)
os.Exit(1)
}
if cfg.EnsureSchema {
if err := identity.EnsureJT808RegistrationSchema(ctx, db); err != nil {
logger.Error("jt808 registration schema bootstrap failed", "error", err)
os.Exit(1)
}
}
registry := metrics.NewRegistry()
metrics.RegisterKafkaConsumerInfo(registry, "vehicle-identity-writer", cfg.KafkaGroup, []string{cfg.KafkaTopic})
registry.SetGauge("vehicle_identity_writer_config", metrics.Labels{"setting": "batch_size"}, float64(cfg.BatchSize))
registry.SetGauge("vehicle_identity_writer_config", metrics.Labels{"setting": "batch_wait_ms"}, float64(cfg.BatchWait.Milliseconds()))
registry.SetGauge("vehicle_identity_writer_config", metrics.Labels{"setting": "location_touch_interval_seconds"}, cfg.LocationTouchInterval.Seconds())
registry.SetGauge("vehicle_identity_writer_config", metrics.Labels{"setting": "workers"}, float64(cfg.Workers))
health.Start(ctx, logger, health.NewServer(cfg.HealthAddr, "vehicle-identity-writer", []health.Check{
{Name: "mysql", Check: db.PingContext},
}, registry))
store := identity.NewJT808RegistrationStore(db)
logger.Info("identity writer started",
"group", cfg.KafkaGroup,
"topic", cfg.KafkaTopic,
"workers", cfg.Workers,
"batch_size", cfg.BatchSize,
"batch_wait_ms", cfg.BatchWait.Milliseconds(),
"location_touch_interval_seconds", cfg.LocationTouchInterval.Seconds())
var workers sync.WaitGroup
for workerID := 1; workerID <= cfg.Workers; workerID++ {
workers.Add(1)
go func(id int) {
defer workers.Done()
runIdentityConsumer(ctx, logger, registry, store, cfg, id)
}(workerID)
}
workers.Wait()
}
func runIdentityConsumer(
ctx context.Context,
logger *slog.Logger,
registry *metrics.Registry,
store registrationBatchStore,
cfg config,
workerID int,
) {
reader := kafka.NewReader(kafka.ReaderConfig{
Brokers: cfg.KafkaBrokers,
GroupID: cfg.KafkaGroup,
GroupTopics: []string{cfg.KafkaTopic},
MinBytes: 1,
MaxBytes: 10e6,
StartOffset: cfg.StartOffset,
})
defer reader.Close()
// Kafka keeps one phone on one partition. Per-worker throttling avoids a
// global hot lock; a rebalance can only cause one harmless idempotent touch.
projector := identity.NewJT808RegistrationProjector(cfg.Location, cfg.LocationTouchInterval)
workerLabels := metrics.Labels{"worker": strconv.Itoa(workerID)}
registry.SetGauge("vehicle_identity_writer_worker_active", workerLabels, 1)
defer registry.SetGauge("vehicle_identity_writer_worker_active", workerLabels, 0)
logger.Info("identity kafka consumer started", "worker", workerID, "topic", cfg.KafkaTopic)
for {
first, err := reader.FetchMessage(ctx)
if err != nil {
if ctx.Err() != nil {
return
}
logger.Error("kafka fetch failed", "error", err)
if !waitForRetry(ctx, cfg.RetryDelay) {
return
}
continue
}
batch := collectIdentityBatch(ctx, reader, first, cfg.BatchSize, cfg.BatchWait)
if !processIdentityBatchReliablyForWorker(ctx, logger, registry, projector, store, reader, batch, cfg.RetryDelay, workerLabels) {
return
}
}
}
type registrationBatchStore interface {
UpsertBatch(context.Context, []identity.JT808RegistrationFact) error
}
type registrationProjector interface {
ProjectBatch([]envelope.FrameEnvelope) []identity.JT808RegistrationFact
MarkPersisted([]identity.JT808RegistrationFact)
}
type kafkaMessageFetcher interface {
FetchMessage(context.Context) (kafka.Message, error)
}
type kafkaMessageCommitter interface {
CommitMessages(context.Context, ...kafka.Message) error
}
const (
identityFailureWrite = "write_error"
identityFailureCommit = "commit_error"
)
type identityBatchFailure struct {
reason string
err error
}
func (e *identityBatchFailure) Error() string { return e.err.Error() }
func (e *identityBatchFailure) Unwrap() error { return e.err }
func collectIdentityBatch(ctx context.Context, fetcher kafkaMessageFetcher, first kafka.Message, maxSize int, maxWait time.Duration) []kafka.Message {
if maxSize <= 1 {
return []kafka.Message{first}
}
if maxWait <= 0 {
maxWait = 20 * time.Millisecond
}
batch := []kafka.Message{first}
deadline := time.Now().Add(maxWait)
for len(batch) < maxSize {
remaining := time.Until(deadline)
if remaining <= 0 {
break
}
fetchCtx, cancel := context.WithTimeout(ctx, remaining)
message, err := fetcher.FetchMessage(fetchCtx)
cancel()
if err != nil {
break
}
batch = append(batch, message)
}
return batch
}
func processIdentityBatch(
ctx context.Context,
logger *slog.Logger,
registry *metrics.Registry,
projector registrationProjector,
store registrationBatchStore,
committer kafkaMessageCommitter,
messages []kafka.Message,
) error {
return processIdentityBatchAttemptForWorker(ctx, logger, registry, projector, store, committer, messages, true, nil)
}
func processIdentityBatchAttempt(
ctx context.Context,
logger *slog.Logger,
registry *metrics.Registry,
projector registrationProjector,
store registrationBatchStore,
committer kafkaMessageCommitter,
messages []kafka.Message,
recordReceived bool,
) error {
return processIdentityBatchAttemptForWorker(ctx, logger, registry, projector, store, committer, messages, recordReceived, nil)
}
func processIdentityBatchAttemptForWorker(
ctx context.Context,
logger *slog.Logger,
registry *metrics.Registry,
projector registrationProjector,
store registrationBatchStore,
committer kafkaMessageCommitter,
messages []kafka.Message,
recordReceived bool,
workerLabels metrics.Labels,
) error {
if len(messages) == 0 {
return nil
}
operationCtx, cancel := context.WithTimeout(context.WithoutCancel(ctx), identityBatchOperationTimeout)
defer cancel()
registry.SetGauge("vehicle_identity_writer_batch_pending_messages", workerLabels, float64(len(messages)))
defer registry.SetGauge("vehicle_identity_writer_batch_pending_messages", workerLabels, 0)
envelopes := make([]envelope.FrameEnvelope, 0, len(messages))
for _, message := range messages {
if recordReceived {
recordIdentityMessage(registry, message, "received")
registry.SetKafkaLag("vehicle_identity_writer_kafka_lag", message.Topic, message.Partition, message.Offset, message.HighWaterMark)
}
var env envelope.FrameEnvelope
if err := json.Unmarshal(message.Value, &env); err != nil {
if recordReceived {
recordIdentityMessage(registry, message, "invalid_json")
logger.Warn("skip invalid identity raw envelope", "topic", message.Topic, "partition", message.Partition, "offset", message.Offset, "error", err)
}
continue
}
if status, err := topics.ValidateRawEnvelope(message.Topic, env); err != nil {
if recordReceived {
recordIdentityMessage(registry, message, status)
logger.Warn("skip mismatched identity raw envelope", "topic", message.Topic, "partition", message.Partition, "offset", message.Offset, "protocol", env.Protocol, "event_id", env.StableEventID(), "error", err)
}
continue
}
envelopes = append(envelopes, env)
}
facts := projector.ProjectBatch(envelopes)
registry.SetGauge("vehicle_identity_writer_batch_pending_facts", workerLabels, float64(len(facts)))
defer registry.SetGauge("vehicle_identity_writer_batch_pending_facts", workerLabels, 0)
if len(facts) > 0 {
started := time.Now()
if err := store.UpsertBatch(operationCtx, facts); err != nil {
recordIdentityWrite(registry, "error", len(facts), time.Since(started))
return &identityBatchFailure{reason: identityFailureWrite, err: fmt.Errorf("upsert jt808 registration facts: %w", err)}
}
projector.MarkPersisted(facts)
recordIdentityWrite(registry, "ok", len(facts), time.Since(started))
}
if err := committer.CommitMessages(operationCtx, messages...); err != nil {
for _, message := range messages {
recordIdentityCommit(registry, message, "error")
}
return &identityBatchFailure{reason: identityFailureCommit, err: fmt.Errorf("commit identity kafka batch: %w", err)}
}
for _, message := range messages {
recordIdentityCommit(registry, message, "ok")
}
return nil
}
func processIdentityBatchReliably(
ctx context.Context,
logger *slog.Logger,
registry *metrics.Registry,
projector registrationProjector,
store registrationBatchStore,
committer kafkaMessageCommitter,
messages []kafka.Message,
retryDelay time.Duration,
) bool {
return processIdentityBatchReliablyForWorker(ctx, logger, registry, projector, store, committer, messages, retryDelay, nil)
}
func processIdentityBatchReliablyForWorker(
ctx context.Context,
logger *slog.Logger,
registry *metrics.Registry,
projector registrationProjector,
store registrationBatchStore,
committer kafkaMessageCommitter,
messages []kafka.Message,
retryDelay time.Duration,
workerLabels metrics.Labels,
) bool {
defer registry.SetGauge("vehicle_identity_writer_retry_pending_messages", workerLabels, 0)
recordReceived := true
for {
err := processIdentityBatchAttemptForWorker(ctx, logger, registry, projector, store, committer, messages, recordReceived, workerLabels)
if err == nil {
return true
}
if ctx.Err() != nil {
return false
}
reason := identityFailureReason(err)
registry.SetGauge("vehicle_identity_writer_retry_pending_messages", workerLabels, float64(len(messages)))
registry.IncCounter("vehicle_identity_writer_batch_retries_total", metrics.Labels{"reason": reason})
logger.Error("identity batch failed; retrying without fetching newer offsets", "messages", len(messages), "reason", reason, "error", err)
if reason == identityFailureCommit {
return retryIdentityCommit(ctx, logger, registry, committer, messages, retryDelay)
}
if !waitForRetry(ctx, retryDelay) {
return false
}
recordReceived = false
}
}
func retryIdentityCommit(
ctx context.Context,
logger *slog.Logger,
registry *metrics.Registry,
committer kafkaMessageCommitter,
messages []kafka.Message,
retryDelay time.Duration,
) bool {
for {
if !waitForRetry(ctx, retryDelay) {
return false
}
operationCtx, cancel := context.WithTimeout(context.WithoutCancel(ctx), identityBatchOperationTimeout)
err := committer.CommitMessages(operationCtx, messages...)
cancel()
if err == nil {
for _, message := range messages {
recordIdentityCommit(registry, message, "ok")
}
return true
}
for _, message := range messages {
recordIdentityCommit(registry, message, "error")
}
registry.IncCounter("vehicle_identity_writer_batch_retries_total", metrics.Labels{"reason": identityFailureCommit})
logger.Error("identity kafka commit retry failed", "messages", len(messages), "error", err)
}
}
func identityFailureReason(err error) string {
var failure *identityBatchFailure
if errors.As(err, &failure) && failure.reason != "" {
return failure.reason
}
return identityFailureWrite
}
var identityWriteDurationBucketsMS = []float64{1, 5, 10, 25, 50, 100, 250, 500, 1000, 5000}
func recordIdentityMessage(registry *metrics.Registry, message kafka.Message, status string) {
labels := metrics.Labels{"topic": message.Topic, "status": status}
registry.IncCounter("vehicle_identity_writer_kafka_messages_total", labels)
metrics.RecordLastActivity(registry, "vehicle_identity_writer_last_message_unix_seconds", labels)
}
func recordIdentityWrite(registry *metrics.Registry, status string, facts int, elapsed time.Duration) {
labels := metrics.Labels{"status": status}
registry.IncCounter("vehicle_identity_writer_batches_total", labels)
registry.AddCounter("vehicle_identity_writer_facts_total", labels, float64(facts))
registry.ObserveHistogram("vehicle_identity_writer_write_duration_ms_histogram", labels, identityWriteDurationBucketsMS, float64(elapsed.Milliseconds()))
metrics.RecordLastActivity(registry, "vehicle_identity_writer_last_write_unix_seconds", labels)
}
func recordIdentityCommit(registry *metrics.Registry, message kafka.Message, status string) {
labels := metrics.Labels{"topic": message.Topic, "status": status}
registry.IncCounter("vehicle_identity_writer_kafka_commits_total", labels)
metrics.RecordLastActivity(registry, "vehicle_identity_writer_last_commit_unix_seconds", labels)
}
func waitForRetry(ctx context.Context, delay time.Duration) bool {
if delay <= 0 {
delay = time.Second
}
timer := time.NewTimer(delay)
defer timer.Stop()
select {
case <-ctx.Done():
return false
case <-timer.C:
return true
}
}
type config struct {
KafkaBrokers []string
KafkaTopic string
KafkaGroup string
MySQLDSN string
MySQLMaxOpenConns int
MySQLMaxIdleConns int
MySQLConnMaxLifetime time.Duration
EnsureSchema bool
HealthAddr string
Location *time.Location
LocationTouchInterval time.Duration
BatchSize int
BatchWait time.Duration
RetryDelay time.Duration
StartOffset int64
Workers int
}
func loadConfig() config {
location, err := time.LoadLocation(env("LOCAL_TZ", "Asia/Shanghai"))
if err != nil {
location = time.FixedZone("Asia/Shanghai", 8*60*60)
}
startOffset := kafka.LastOffset
if strings.EqualFold(env("KAFKA_START_OFFSET", "last"), "first") {
startOffset = kafka.FirstOffset
}
return config{
KafkaBrokers: splitCSV(env("KAFKA_BROKERS", "127.0.0.1:9092")),
KafkaTopic: env("KAFKA_TOPIC", topics.RawJT808),
KafkaGroup: env("KAFKA_GROUP", "go-identity-writer"),
MySQLDSN: env("MYSQL_DSN", ""),
MySQLMaxOpenConns: envInt("MYSQL_MAX_OPEN_CONNS", 8),
MySQLMaxIdleConns: envInt("MYSQL_MAX_IDLE_CONNS", 4),
MySQLConnMaxLifetime: time.Duration(envInt("MYSQL_CONN_MAX_LIFETIME_SECONDS", 300)) * time.Second,
EnsureSchema: envBool("MYSQL_ENSURE_SCHEMA", true),
HealthAddr: env("HEALTH_ADDR", "127.0.0.1:20217"),
Location: location,
LocationTouchInterval: time.Duration(envInt("JT808_REGISTRATION_LOCATION_TOUCH_INTERVAL_SECONDS", 600)) * time.Second,
BatchSize: envInt("IDENTITY_WRITER_BATCH_SIZE", 500),
BatchWait: time.Duration(envInt("IDENTITY_WRITER_BATCH_WAIT_MS", 20)) * time.Millisecond,
RetryDelay: time.Duration(envInt("IDENTITY_WRITER_RETRY_DELAY_MS", 1000)) * time.Millisecond,
StartOffset: startOffset,
Workers: envInt("IDENTITY_WRITER_WORKERS", 3),
}
}
func (c config) Validate() error {
if len(c.KafkaBrokers) == 0 {
return fmt.Errorf("KAFKA_BROKERS is required")
}
if strings.TrimSpace(c.MySQLDSN) == "" {
return fmt.Errorf("MYSQL_DSN is required")
}
protocol, ok := topics.ProtocolForKnownRawTopic(c.KafkaTopic)
if !ok || protocol != string(envelope.ProtocolJT808) {
return fmt.Errorf("identity writer consumes JT808 raw topic only, got %q", c.KafkaTopic)
}
if c.BatchSize <= 0 {
return fmt.Errorf("IDENTITY_WRITER_BATCH_SIZE must be positive")
}
if c.Workers <= 0 {
return fmt.Errorf("IDENTITY_WRITER_WORKERS must be positive")
}
return nil
}
func env(key string, fallback string) string {
value := strings.TrimSpace(os.Getenv(key))
if value == "" {
return fallback
}
return value
}
func envInt(key string, fallback int) int {
value := strings.TrimSpace(os.Getenv(key))
if value == "" {
return fallback
}
parsed, err := strconv.Atoi(value)
if err != nil {
return fallback
}
return parsed
}
func envBool(key string, fallback bool) bool {
value := strings.TrimSpace(os.Getenv(key))
if value == "" {
return fallback
}
parsed, err := strconv.ParseBool(value)
if err != nil {
return fallback
}
return parsed
}
func splitCSV(value string) []string {
parts := strings.Split(value, ",")
out := make([]string, 0, len(parts))
for _, item := range parts {
if trimmed := strings.TrimSpace(item); trimmed != "" {
out = append(out, trimmed)
}
}
return out
}

314
go/vehicle-gateway/cmd/identity-writer/main_test.go Normal file
View File

@@ -0,0 +1,314 @@
package main
import (
"context"
"errors"
"io"
"log/slog"
"strings"
"testing"
"time"
"github.com/segmentio/kafka-go"
"lingniu-vehicle-ingest/go/vehicle-gateway/internal/envelope"
"lingniu-vehicle-ingest/go/vehicle-gateway/internal/identity"
"lingniu-vehicle-ingest/go/vehicle-gateway/internal/metrics"
"lingniu-vehicle-ingest/go/vehicle-gateway/internal/topics"
)
type fakeRegistrationProjector struct {
facts []identity.JT808RegistrationFact
envelopes []envelope.FrameEnvelope
markedFacts []identity.JT808RegistrationFact
}
func (p *fakeRegistrationProjector) ProjectBatch(envs []envelope.FrameEnvelope) []identity.JT808RegistrationFact {
p.envelopes = append(p.envelopes, envs...)
return p.facts
}
func (p *fakeRegistrationProjector) MarkPersisted(facts []identity.JT808RegistrationFact) {
p.markedFacts = append(p.markedFacts, facts...)
}
type fakeRegistrationStore struct {
facts []identity.JT808RegistrationFact
err error
count int
failOnCount int
}
func (s *fakeRegistrationStore) UpsertBatch(_ context.Context, facts []identity.JT808RegistrationFact) error {
s.count++
s.facts = append(s.facts, facts...)
if s.err != nil && (s.failOnCount == 0 || s.count == s.failOnCount) {
return s.err
}
return nil
}
type fakeCommitter struct {
messages []kafka.Message
err error
count int
failOnCount int
}
func (c *fakeCommitter) CommitMessages(_ context.Context, messages ...kafka.Message) error {
c.count++
c.messages = append(c.messages, messages...)
if c.err != nil && (c.failOnCount == 0 || c.count == c.failOnCount) {
return c.err
}
return nil
}
func TestProcessIdentityBatchDoesNotCommitOrThrottleOnStoreFailure(t *testing.T) {
wantErr := errors.New("mysql unavailable")
projector := &fakeRegistrationProjector{facts: []identity.JT808RegistrationFact{{
Phone: "13307795425",
SeenAt: time.Now(),
}}}
store := &fakeRegistrationStore{err: wantErr}
committer := &fakeCommitter{}
err := processIdentityBatch(
context.Background(),
slog.New(slog.NewTextHandler(io.Discard, nil)),
metrics.NewRegistry(),
projector,
store,
committer,
[]kafka.Message{validIdentityMessage(t, 1)},
)
if !errors.Is(err, wantErr) {
t.Fatalf("processIdentityBatch() error = %v, want %v", err, wantErr)
}
if len(committer.messages) != 0 {
t.Fatalf("committed messages = %d, want 0", len(committer.messages))
}
if len(projector.markedFacts) != 0 {
t.Fatalf("marked facts = %d, want 0", len(projector.markedFacts))
}
}
func TestProcessIdentityBatchMarksOnlyAfterStoreAndCommits(t *testing.T) {
fact := identity.JT808RegistrationFact{Phone: "13307795425", SeenAt: time.Now()}
projector := &fakeRegistrationProjector{facts: []identity.JT808RegistrationFact{fact}}
store := &fakeRegistrationStore{}
committer := &fakeCommitter{}
messages := []kafka.Message{validIdentityMessage(t, 1), validIdentityMessage(t, 2)}
err := processIdentityBatch(
context.Background(),
slog.New(slog.NewTextHandler(io.Discard, nil)),
metrics.NewRegistry(),
projector,
store,
committer,
messages,
)
if err != nil {
t.Fatalf("processIdentityBatch() error = %v", err)
}
if len(store.facts) != 1 || len(projector.markedFacts) != 1 {
t.Fatalf("store facts = %d marked facts = %d, want 1/1", len(store.facts), len(projector.markedFacts))
}
if len(committer.messages) != len(messages) {
t.Fatalf("committed messages = %d, want %d", len(committer.messages), len(messages))
}
}
func TestProcessIdentityBatchCommitsPoisonEnvelopeWithoutProjection(t *testing.T) {
projector := &fakeRegistrationProjector{}
store := &fakeRegistrationStore{}
committer := &fakeCommitter{}
message := kafka.Message{Topic: topics.RawJT808, Partition: 1, Offset: 7, Value: []byte("not-json")}
err := processIdentityBatch(
context.Background(),
slog.New(slog.NewTextHandler(io.Discard, nil)),
metrics.NewRegistry(),
projector,
store,
committer,
[]kafka.Message{message},
)
if err != nil {
t.Fatalf("processIdentityBatch() error = %v", err)
}
if len(store.facts) != 0 || len(projector.envelopes) != 0 {
t.Fatalf("poison message reached projector/store: envs=%d facts=%d", len(projector.envelopes), len(store.facts))
}
if len(committer.messages) != 1 {
t.Fatalf("committed messages = %d, want 1", len(committer.messages))
}
}
func TestProcessIdentityBatchReliablyRetriesCommitWithoutRewritingMySQL(t *testing.T) {
fact := identity.JT808RegistrationFact{Phone: "13307795425", SeenAt: time.Now()}
projector := &fakeRegistrationProjector{facts: []identity.JT808RegistrationFact{fact}}
store := &fakeRegistrationStore{}
committer := &fakeCommitter{err: errors.New("commit failed"), failOnCount: 1}
registry := metrics.NewRegistry()
messages := []kafka.Message{validIdentityMessage(t, 1), validIdentityMessage(t, 2)}
ok := processIdentityBatchReliably(
context.Background(),
slog.New(slog.NewTextHandler(io.Discard, nil)),
registry,
projector,
store,
committer,
messages,
time.Nanosecond,
)
if !ok {
t.Fatal("reliable identity batch returned false")
}
if store.count != 1 {
t.Fatalf("mysql writes = %d, want 1 after commit-only retry", store.count)
}
if committer.count != 2 {
t.Fatalf("commit attempts = %d, want initial failure and one retry", committer.count)
}
text := registry.Render()
for _, want := range []string{
`vehicle_identity_writer_batch_retries_total{reason="commit_error"} 1`,
`vehicle_identity_writer_retry_pending_messages 0`,
} {
if !strings.Contains(text, want) {
t.Fatalf("missing metric %s:\n%s", want, text)
}
}
}
func TestProcessIdentityBatchReliablyRetriesStoreBeforeCommit(t *testing.T) {
fact := identity.JT808RegistrationFact{Phone: "13307795425", SeenAt: time.Now()}
projector := &fakeRegistrationProjector{facts: []identity.JT808RegistrationFact{fact}}
store := &fakeRegistrationStore{err: errors.New("mysql unavailable"), failOnCount: 1}
committer := &fakeCommitter{}
registry := metrics.NewRegistry()
messages := []kafka.Message{validIdentityMessage(t, 1)}
ok := processIdentityBatchReliably(
context.Background(),
slog.New(slog.NewTextHandler(io.Discard, nil)),
registry,
projector,
store,
committer,
messages,
time.Nanosecond,
)
if !ok {
t.Fatal("reliable identity batch returned false")
}
if store.count != 2 || committer.count != 1 {
t.Fatalf("mysql writes=%d commits=%d, want 2/1", store.count, committer.count)
}
text := registry.Render()
for _, want := range []string{
`vehicle_identity_writer_batch_retries_total{reason="write_error"} 1`,
`vehicle_identity_writer_kafka_messages_total{status="received",topic="vehicle.raw.go.jt808.v1"} 1`,
} {
if !strings.Contains(text, want) {
t.Fatalf("missing metric %s:\n%s", want, text)
}
}
}
func TestConfigRejectsNonJT808Topic(t *testing.T) {
cfg := config{
KafkaBrokers: []string{"127.0.0.1:9092"},
KafkaTopic: topics.RawGB32960,
MySQLDSN: "user:pass@tcp(localhost:3306)/db",
BatchSize: 100,
Workers: 3,
}
if err := cfg.Validate(); err == nil {
t.Fatal("Validate() error = nil, want non-JT808 topic error")
}
cfg.KafkaTopic = topics.RawJT808
if err := cfg.Validate(); err != nil {
t.Fatalf("Validate() error = %v", err)
}
}
func TestLoadConfigDefaultsAndOverridesWorkers(t *testing.T) {
t.Setenv("IDENTITY_WRITER_WORKERS", "")
if got := loadConfig().Workers; got != 3 {
t.Fatalf("default workers = %d, want 3", got)
}
t.Setenv("IDENTITY_WRITER_WORKERS", "5")
if got := loadConfig().Workers; got != 5 {
t.Fatalf("configured workers = %d, want 5", got)
}
}
func TestConfigRejectsNonPositiveWorkers(t *testing.T) {
cfg := config{
KafkaBrokers: []string{"127.0.0.1:9092"},
KafkaTopic: topics.RawJT808,
MySQLDSN: "user:pass@tcp(localhost:3306)/db",
BatchSize: 100,
}
if err := cfg.Validate(); err == nil || !strings.Contains(err.Error(), "IDENTITY_WRITER_WORKERS") {
t.Fatalf("Validate() error = %v, want workers error", err)
}
}
func TestProcessIdentityBatchForWorkerLabelsPendingMetrics(t *testing.T) {
registry := metrics.NewRegistry()
workerLabels := metrics.Labels{"worker": "2"}
err := processIdentityBatchAttemptForWorker(
context.Background(),
slog.New(slog.NewTextHandler(io.Discard, nil)),
registry,
&fakeRegistrationProjector{},
&fakeRegistrationStore{},
&fakeCommitter{},
[]kafka.Message{validIdentityMessage(t, 1)},
true,
workerLabels,
)
if err != nil {
t.Fatalf("processIdentityBatchAttemptForWorker() error = %v", err)
}
text := registry.Render()
for _, want := range []string{
`vehicle_identity_writer_batch_pending_messages{worker="2"} 0`,
`vehicle_identity_writer_batch_pending_facts{worker="2"} 0`,
} {
if !strings.Contains(text, want) {
t.Fatalf("missing metric %s:\n%s", want, text)
}
}
}
func validIdentityMessage(t *testing.T, offset int64) kafka.Message {
t.Helper()
env := envelope.FrameEnvelope{
Protocol: envelope.ProtocolJT808,
MessageID: identity.JT808LocationMessageID,
EventKind: envelope.EventKindRaw,
Phone: "13307795425",
VIN: "LTESTVIN000000001",
ReceivedAtMS: time.Now().UnixMilli(),
ParseStatus: envelope.ParseOK,
}
payload, err := env.MarshalJSONBytes()
if err != nil {
t.Fatal(err)
}
return kafka.Message{
Topic: topics.RawJT808,
Partition: 1,
Offset: offset,
HighWaterMark: offset + 1,
Value: payload,
}
}

61
go/vehicle-gateway/cmd/load-sim/main.go
View File

@@ -2,19 +2,26 @@ package main
import (
"context"
"database/sql"
"flag"
"fmt"
"log"
"os"
"os/signal"
"strings"
"syscall"
_ "github.com/go-sql-driver/mysql"
"lingniu-vehicle-ingest/go/vehicle-gateway/internal/loadsim"
)
func main() {
log.SetFlags(log.LstdFlags | log.Lmicroseconds)
flagCfg := loadsim.RegisterFlags(flag.CommandLine)
cleanupRegistration := flag.Bool("cleanup-registration", false, "delete this run's loopback JT808 registration rows after the simulation")
cleanupOnly := flag.Bool("cleanup-only", false, "skip the simulation and only clean the configured JT808 phone range")
mysqlDSN := flag.String("mysql-dsn", strings.TrimSpace(os.Getenv("MYSQL_DSN")), "MySQL DSN used only by JT808 registration cleanup")
if err := flag.CommandLine.Parse(os.Args[1:]); err != nil {
log.Fatal(err)
}
@@ -25,6 +32,14 @@ func main() {
ctx, stop := signal.NotifyContext(context.Background(), os.Interrupt, syscall.SIGTERM)
defer stop()
if *cleanupOnly {
deleted, err := cleanupJT808Registrations(ctx, cfg, *mysqlDSN)
if err != nil {
log.Fatal(err)
}
log.Printf("jt808 synthetic registration cleanup completed rows_deleted=%d", deleted)
return
}
log.Printf("load simulation started protocol=%s addr=%s connections=%d connect_rate=%d send_interval=%s duration=%s template=%s",
cfg.Protocol, cfg.Addr, cfg.Connections, cfg.ConnectRatePerSecond, cfg.SendInterval, cfg.Duration, cfg.Template)
@@ -33,13 +48,57 @@ func main() {
log.Fatal(err)
}
log.Print(formatStats(stats))
if *cleanupRegistration {
deleted, err := cleanupJT808Registrations(ctx, cfg, *mysqlDSN)
if err != nil {
log.Fatal(err)
}
log.Printf("jt808 synthetic registration cleanup completed rows_deleted=%d", deleted)
}
}
func cleanupJT808Registrations(ctx context.Context, cfg loadsim.Config, dsn string) (int64, error) {
if cfg.Protocol != loadsim.ProtocolJT808 {
return 0, fmt.Errorf("registration cleanup only supports jt808")
}
if strings.TrimSpace(dsn) == "" {
return 0, fmt.Errorf("mysql-dsn or MYSQL_DSN is required for registration cleanup")
}
db, err := sql.Open("mysql", dsn)
if err != nil {
return 0, fmt.Errorf("open mysql for registration cleanup: %w", err)
}
defer db.Close()
return cleanupJT808RegistrationsWithDB(ctx, db, cfg)
}
type cleanupExecer interface {
ExecContext(context.Context, string, ...any) (sql.Result, error)
}
func cleanupJT808RegistrationsWithDB(ctx context.Context, exec cleanupExecer, cfg loadsim.Config) (int64, error) {
firstPhone := fmt.Sprintf("%012d", cfg.JT808PhoneBase)
lastPhone := fmt.Sprintf("%012d", cfg.JT808PhoneBase+int64(cfg.Connections)-1)
result, err := exec.ExecContext(ctx, `DELETE FROM jt808_registration
WHERE phone BETWEEN ? AND ?
AND source_ip IN ('127.0.0.1', '::1')`, firstPhone, lastPhone)
if err != nil {
return 0, fmt.Errorf("delete loopback jt808 registrations: %w", err)
}
deleted, err := result.RowsAffected()
if err != nil {
return 0, fmt.Errorf("read registration cleanup result: %w", err)
}
return deleted, nil
}
func formatStats(stats loadsim.Stats) string {
return fmt.Sprintf("connections_opened=%d connections_failed=%d frames_written=%d write_errors=%d",
return fmt.Sprintf("connections_opened=%d connections_failed=%d frames_written=%d write_errors=%d response_bytes=%d read_errors=%d",
stats.ConnectionsOpened,
stats.ConnectionsFailed,
stats.FramesWritten,
stats.WriteErrors,
stats.ResponseBytes,
stats.ReadErrors,
)
}

33
go/vehicle-gateway/cmd/load-sim/main_test.go
View File

@@ -1,9 +1,12 @@
package main
import (
"context"
"strings"
"testing"
"github.com/DATA-DOG/go-sqlmock"
"lingniu-vehicle-ingest/go/vehicle-gateway/internal/loadsim"
)
@@ -13,6 +16,8 @@ func TestFormatStatsIncludesCapacityCounters(t *testing.T) {
ConnectionsFailed: 2,
FramesWritten: 300,
WriteErrors: 1,
ResponseBytes: 2048,
ReadErrors: 0,
})
for _, want := range []string{
@@ -20,9 +25,37 @@ func TestFormatStatsIncludesCapacityCounters(t *testing.T) {
"connections_failed=2",
"frames_written=300",
"write_errors=1",
"response_bytes=2048",
"read_errors=0",
} {
if !strings.Contains(out, want) {
t.Fatalf("formatStats() = %q, missing %q", out, want)
}
}
}
func TestCleanupJT808RegistrationsUsesBoundedLoopbackDelete(t *testing.T) {
db, mock, err := sqlmock.New()
if err != nil {
t.Fatal(err)
}
defer db.Close()
mock.ExpectExec(`DELETE FROM jt808_registration`).
WithArgs("139000000000", "139000000999").
WillReturnResult(sqlmock.NewResult(0, 1000))
deleted, err := cleanupJT808RegistrationsWithDB(context.Background(), db, loadsim.Config{
Protocol: loadsim.ProtocolJT808,
Connections: 1000,
JT808PhoneBase: loadsim.DefaultJT808PhoneBase,
})
if err != nil {
t.Fatalf("cleanupJT808RegistrationsWithDB() error = %v", err)
}
if deleted != 1000 {
t.Fatalf("deleted = %d, want 1000", deleted)
}
if err := mock.ExpectationsWereMet(); err != nil {
t.Fatal(err)
}
}

579
go/vehicle-gateway/cmd/nats-fast-writer/main.go
View File

@@ -60,24 +60,32 @@ func main() {
os.Exit(1)
}
tdDB, err := sql.Open(cfg.TDengineDriver, cfg.TDengineDSN)
if err != nil {
logger.Error("tdengine open failed", "error", err)
os.Exit(1)
}
defer tdDB.Close()
tdDB.SetMaxOpenConns(cfg.TDengineMaxOpenConns)
tdDB.SetMaxIdleConns(cfg.TDengineMaxIdleConns)
if err := tdDB.PingContext(ctx); err != nil {
logger.Error("tdengine ping failed", "error", err)
os.Exit(1)
}
historyWriter := history.NewWriterWithDatabase(tdDB, cfg.TDengineDatabase)
if cfg.TDengineEnsureSchema {
if err := historyWriter.EnsureSchema(ctx, cfg.TDengineDatabase); err != nil {
logger.Error("tdengine schema bootstrap failed", "error", err)
var historyWriter fastAppender
var tdCheck health.Check
if cfg.TDengineEnabled {
tdDB, err := sql.Open(cfg.TDengineDriver, cfg.TDengineDSN)
if err != nil {
logger.Error("tdengine open failed", "error", err)
os.Exit(1)
}
defer tdDB.Close()
tdDB.SetMaxOpenConns(cfg.TDengineMaxOpenConns)
tdDB.SetMaxIdleConns(cfg.TDengineMaxIdleConns)
if err := tdDB.PingContext(ctx); err != nil {
logger.Error("tdengine ping failed", "error", err)
os.Exit(1)
}
writer := history.NewWriterWithDatabase(tdDB, cfg.TDengineDatabase)
if cfg.TDengineEnsureSchema {
if err := writer.EnsureSchema(ctx, cfg.TDengineDatabase); err != nil {
logger.Error("tdengine schema bootstrap failed", "error", err)
os.Exit(1)
}
}
historyWriter = writer
tdCheck = health.Check{Name: "tdengine", Check: tdDB.PingContext}
} else {
logger.Info("nats fast writer tdengine stage disabled")
}
redisClient := redis.NewClient(&redis.Options{
@@ -94,23 +102,30 @@ func main() {
realtimeRepo := realtime.NewRepository(redisClient, realtime.Config{OnlineTTL: cfg.OnlineTTL})
registry := metrics.NewRegistry()
health.Start(ctx, logger, health.NewServer(env("HEALTH_ADDR", ""), "nats-fast-writer", []health.Check{
recordFastWriterConfigMetrics(registry, cfg)
registry.SetGauge("vehicle_fast_writer_tdengine_enabled", nil, boolGauge(cfg.TDengineEnabled))
healthChecks := []health.Check{
{Name: "nats", Check: func(context.Context) error {
if conn.Status() != nats.CONNECTED {
return fmt.Errorf("nats status is %s", conn.Status().String())
}
return nil
}},
{Name: "tdengine", Check: tdDB.PingContext},
{Name: "redis", Check: func(ctx context.Context) error { return redisClient.Ping(ctx).Err() }},
}, registry))
}
if tdCheck.Name != "" {
healthChecks = append(healthChecks, tdCheck)
}
health.Start(ctx, logger, health.NewServer(env("HEALTH_ADDR", ""), "nats-fast-writer", healthChecks, registry))
logger.Info("nats fast writer started",
"stream", cfg.NATSStream,
"durable", cfg.NATSDurable,
"filter", cfg.NATSFilter,
"batch_size", cfg.BatchSize,
"fetch_wait_ms", cfg.FetchWait.Milliseconds(),
"workers", cfg.Workers,
"tdengine_enabled", cfg.TDengineEnabled,
"tdengine_max_open_conns", cfg.TDengineMaxOpenConns,
"tdengine_max_idle_conns", cfg.TDengineMaxIdleConns,
"operation_timeout_ms", cfg.OperationWait.Milliseconds())
@@ -135,6 +150,7 @@ type config struct {
StreamMaxBytes int64
StreamEnsureWait time.Duration
Workers int
TDengineEnabled bool
TDengineDriver string
TDengineDSN string
TDengineDatabase string
@@ -168,13 +184,14 @@ func loadConfig() config {
NATSFilter: env("NATS_FILTER", "vehicle.raw.go.>"),
NATSSubjects: subjects,
BatchSize: envInt("FAST_WRITER_BATCH_SIZE", 100),
FetchWait: time.Duration(envInt("FAST_WRITER_FETCH_WAIT_MS", 100)) * time.Millisecond,
FetchWait: time.Duration(envInt("FAST_WRITER_FETCH_WAIT_MS", 20)) * time.Millisecond,
OperationWait: time.Duration(envInt("FAST_WRITER_OPERATION_TIMEOUT_MS", 1000)) * time.Millisecond,
AckWait: time.Duration(envInt("NATS_ACK_WAIT_SECONDS", 30)) * time.Second,
StreamMaxAge: time.Duration(envInt("NATS_STREAM_MAX_AGE_HOURS", 24)) * time.Hour,
StreamMaxBytes: envInt64("NATS_STREAM_MAX_BYTES", 20*1024*1024*1024),
StreamEnsureWait: time.Duration(envInt("NATS_STREAM_ENSURE_TIMEOUT_SECONDS", 60)) * time.Second,
Workers: workers,
TDengineEnabled: envBool("FAST_WRITER_TDENGINE_ENABLED", false),
TDengineDriver: env("TDENGINE_DRIVER", "taosWS"),
TDengineDSN: env("TDENGINE_DSN", ""),
TDengineDatabase: env("TDENGINE_DATABASE", history.DefaultDatabase),
@@ -198,10 +215,18 @@ type fastUpdater interface {
FastUpdate(context.Context, envelope.FrameEnvelope) error
}
type fastResultUpdater interface {
FastUpdateWithResult(context.Context, envelope.FrameEnvelope) (realtime.FastUpdateResult, error)
}
type fastBatchUpdater interface {
FastUpdateBatch(context.Context, []envelope.FrameEnvelope) error
}
type fastBatchResultUpdater interface {
FastUpdateBatchWithResult(context.Context, []envelope.FrameEnvelope) (realtime.FastUpdateResult, error)
}
type fastMessage struct {
subject string
data []byte
@@ -227,9 +252,17 @@ func runFastWorker(ctx context.Context, logger *slog.Logger, registry *metrics.R
}
msgs, err := sub.Fetch(cfg.BatchSize, nats.MaxWait(cfg.FetchWait))
if err != nil {
if isFastWorkerShutdownFetchError(ctx, err) {
return
}
if errors.Is(err, nats.ErrTimeout) {
continue
}
if isTransientFastFetchError(err) {
logger.Warn("nats fetch interrupted", "error", err)
time.Sleep(time.Second)
continue
}
logger.Error("nats fetch failed", "error", err)
time.Sleep(time.Second)
continue
@@ -253,12 +286,33 @@ func runFastWorker(ctx context.Context, logger *slog.Logger, registry *metrics.R
logger.Error("fast write batch failed", "messages", len(fastMessages), "error", err)
continue
}
for _, msg := range fastMessages {
addFastMetric(registry, msg.subject, "ok")
}
}
}
func isFastWorkerShutdownFetchError(ctx context.Context, err error) bool {
if err == nil {
return false
}
if ctx.Err() != nil {
return true
}
return errors.Is(err, nats.ErrConnectionClosed)
}
func isTransientFastFetchError(err error) bool {
if err == nil {
return false
}
text := strings.ToLower(strings.TrimSpace(err.Error()))
return strings.Contains(text, "disconnected during fetch") ||
strings.Contains(text, "connection closed") ||
strings.Contains(text, "connection reset") ||
strings.Contains(text, "broken pipe") ||
strings.Contains(text, "temporary") ||
strings.Contains(text, "temporarily") ||
strings.Contains(text, "timeout")
}
type natsPullSubscription interface {
Fetch(int, ...nats.PullOpt) ([]*nats.Msg, error)
}
@@ -268,18 +322,44 @@ type natsConsumerInfoReader interface {
}
var fastWriterStageDurationBucketsMS = []float64{1, 5, 10, 25, 50, 100, 250, 500, 1000, 5000}
var fastWriterRedisE2EDurationBucketsMS = []float64{10, 25, 50, 100, 250, 500, 1000, 2500, 5000, 10000}
var fastWriterRedisE2ERecent = metrics.NewRecentLatencyByKey(512)
var fastBatchPending = metrics.PendingPairGauge{}
func processFastBatch(ctx context.Context, registry *metrics.Registry, appender fastAppender, updater fastUpdater, messages []*fastMessage) error {
if len(messages) == 0 {
return nil
}
for _, msg := range messages {
addFastMetric(registry, msg.subject, "received")
}
envelopes := make([]envelope.FrameEnvelope, 0, len(messages))
validMessages := make([]*fastMessage, 0, len(messages))
for _, msg := range messages {
var env envelope.FrameEnvelope
if err := json.Unmarshal(msg.data, &env); err != nil {
addFastMetric(registry, msg.subject, "invalid_json")
if msg.ack != nil {
_ = msg.ack()
started := time.Now()
err := msg.ack()
recordFastWriterStageDuration(registry, msg.subject, "ack", statusFromError(err), time.Since(started))
if err != nil {
addFastMetric(registry, msg.subject, "ack_error")
return fmt.Errorf("nats ack invalid json: %w", err)
}
}
continue
}
if status, err := topics.ValidateRawEnvelope(msg.subject, env); err != nil {
addFastMetric(registry, msg.subject, status)
if msg.ack != nil {
started := time.Now()
err := msg.ack()
recordFastWriterStageDuration(registry, msg.subject, "ack", statusFromError(err), time.Since(started))
if err != nil {
addFastMetric(registry, msg.subject, "ack_error")
return fmt.Errorf("nats ack mismatched raw envelope: %w", err)
}
}
continue
}
@@ -289,85 +369,358 @@ func processFastBatch(ctx context.Context, registry *metrics.Registry, appender
if len(envelopes) == 0 {
return nil
}
setFastBatchPending(registry, len(messages), len(envelopes))
defer setFastBatchPending(registry, 0, 0)
addFastBatchPending(registry, len(messages), len(envelopes))
defer addFastBatchPending(registry, -len(messages), -len(envelopes))
subject := fastBatchSubject(validMessages)
started := time.Now()
err := appender.AppendAllBatch(ctx, envelopes)
recordFastWriterStageDuration(registry, subject, "tdengine", statusFromError(err), time.Since(started))
if err != nil {
return fmt.Errorf("tdengine batch append: %w", err)
}
if batchUpdater, ok := updater.(fastBatchUpdater); ok {
started = time.Now()
err = batchUpdater.FastUpdateBatch(ctx, envelopes)
recordFastWriterStageDuration(registry, subject, "redis", statusFromError(err), time.Since(started))
if appender != nil {
started := time.Now()
err := appender.AppendAllBatch(ctx, envelopes)
recordFastWriterStageDuration(registry, subject, "tdengine", statusFromError(err), time.Since(started))
if err != nil {
return fmt.Errorf("redis fast batch update: %w", err)
if shouldFallbackFastBatchError(err) {
if fallbackErr := processFastMessagesIndividually(ctx, registry, appender, updater, validMessages); fallbackErr != nil {
addFastBatchFallbackMetric(registry, "tdengine", "error")
return fmt.Errorf("tdengine batch fallback: %w", fallbackErr)
}
addFastBatchFallbackMetric(registry, "tdengine", "ok")
return nil
}
addFastBatchFallbackMetric(registry, "tdengine", "skipped_transient")
if catchupErr := updateFastRedisBatchWithoutAck(ctx, registry, updater, validMessages, envelopes); catchupErr != nil {
addFastDecoupledUpdateMetric(registry, "tdengine_transient", "error")
return fmt.Errorf("tdengine batch append: %w; redis catchup: %v", err, catchupErr)
}
addFastDecoupledUpdateMetric(registry, "tdengine_transient", "ok")
return fmt.Errorf("tdengine batch append: %w", err)
}
}
if batchUpdater, ok := updater.(fastBatchResultUpdater); ok {
for _, group := range fastSubjectGroups(validMessages, envelopes) {
started := time.Now()
result, err := batchUpdater.FastUpdateBatchWithResult(ctx, group.envelopes)
recordFastWriterStageDuration(registry, group.subject, "redis", statusFromError(err), time.Since(started))
if err != nil {
if shouldFallbackFastBatchError(err) {
if fallbackErr := processFastMessagesIndividually(ctx, registry, nil, updater, validMessages); fallbackErr != nil {
addFastBatchFallbackMetric(registry, "redis", "error")
return fmt.Errorf("redis fast batch fallback: %w", fallbackErr)
}
addFastBatchFallbackMetric(registry, "redis", "ok")
return nil
}
addFastBatchFallbackMetric(registry, "redis", "skipped_transient")
return fmt.Errorf("redis fast batch update: %w", err)
}
recordFastWriterRedisEnvelopeMetrics(registry, group.subject, result)
recordFastWriterRedisFieldMetrics(registry, group.subject, result)
recordFastWriterRedisE2EDurationMessages(registry, group.messages, group.envelopes, group.subject)
}
for _, msg := range validMessages {
if msg.ack != nil {
started = time.Now()
err = msg.ack()
started := time.Now()
err := msg.ack()
recordFastWriterStageDuration(registry, msg.subject, "ack", statusFromError(err), time.Since(started))
if err != nil {
addFastMetric(registry, msg.subject, "ack_error")
return fmt.Errorf("nats ack: %w", err)
}
}
addFastMetric(registry, msg.subject, "ok")
}
return nil
}
if batchUpdater, ok := updater.(fastBatchUpdater); ok {
for _, group := range fastSubjectGroups(validMessages, envelopes) {
started := time.Now()
err := batchUpdater.FastUpdateBatch(ctx, group.envelopes)
recordFastWriterStageDuration(registry, group.subject, "redis", statusFromError(err), time.Since(started))
if err != nil {
if shouldFallbackFastBatchError(err) {
if fallbackErr := processFastMessagesIndividually(ctx, registry, nil, updater, validMessages); fallbackErr != nil {
addFastBatchFallbackMetric(registry, "redis", "error")
return fmt.Errorf("redis fast batch fallback: %w", fallbackErr)
}
addFastBatchFallbackMetric(registry, "redis", "ok")
return nil
}
addFastBatchFallbackMetric(registry, "redis", "skipped_transient")
return fmt.Errorf("redis fast batch update: %w", err)
}
recordFastWriterRedisE2EDurationMessages(registry, group.messages, group.envelopes, group.subject)
}
for _, msg := range validMessages {
if msg.ack != nil {
started := time.Now()
err := msg.ack()
recordFastWriterStageDuration(registry, msg.subject, "ack", statusFromError(err), time.Since(started))
if err != nil {
addFastMetric(registry, msg.subject, "ack_error")
return fmt.Errorf("nats ack: %w", err)
}
}
addFastMetric(registry, msg.subject, "ok")
}
return nil
}
for i, env := range envelopes {
msg := validMessages[i]
started = time.Now()
err = updater.FastUpdate(ctx, env)
started := time.Now()
var result realtime.FastUpdateResult
var err error
if resultUpdater, ok := updater.(fastResultUpdater); ok {
result, err = resultUpdater.FastUpdateWithResult(ctx, env)
} else {
err = updater.FastUpdate(ctx, env)
}
recordFastWriterStageDuration(registry, msg.subject, "redis", statusFromError(err), time.Since(started))
if err != nil {
return fmt.Errorf("redis fast update: %w", err)
}
recordFastWriterRedisEnvelopeMetrics(registry, msg.subject, result)
recordFastWriterRedisFieldMetrics(registry, msg.subject, result)
recordFastWriterRedisE2EDuration(registry, msg.subject, env)
if msg.ack != nil {
started = time.Now()
err = msg.ack()
started := time.Now()
err := msg.ack()
recordFastWriterStageDuration(registry, msg.subject, "ack", statusFromError(err), time.Since(started))
if err != nil {
addFastMetric(registry, msg.subject, "ack_error")
return fmt.Errorf("nats ack: %w", err)
}
}
addFastMetric(registry, msg.subject, "ok")
}
return nil
}
func processFastMessagesIndividually(ctx context.Context, registry *metrics.Registry, appender fastAppender, updater fastUpdater, messages []*fastMessage) error {
for _, msg := range messages {
if err := processFastMessageWithReceived(ctx, registry, appender, updater, msg, false); err != nil {
return err
}
}
return nil
}
func processFastMessage(ctx context.Context, registry *metrics.Registry, appender fastAppender, updater fastUpdater, msg *fastMessage) error {
return processFastMessageWithReceived(ctx, registry, appender, updater, msg, true)
}
func processFastMessageWithReceived(ctx context.Context, registry *metrics.Registry, appender fastAppender, updater fastUpdater, msg *fastMessage, recordReceived bool) error {
if recordReceived {
addFastMetric(registry, msg.subject, "received")
}
var env envelope.FrameEnvelope
if err := json.Unmarshal(msg.data, &env); err != nil {
addFastMetric(registry, msg.subject, "invalid_json")
if msg.ack != nil {
_ = msg.ack()
started := time.Now()
err := msg.ack()
recordFastWriterStageDuration(registry, msg.subject, "ack", statusFromError(err), time.Since(started))
if err != nil {
addFastMetric(registry, msg.subject, "ack_error")
return fmt.Errorf("nats ack invalid json: %w", err)
}
}
return nil
}
started := time.Now()
err := appender.AppendAll(ctx, env)
recordFastWriterStageDuration(registry, msg.subject, "tdengine", statusFromError(err), time.Since(started))
if err != nil {
return fmt.Errorf("tdengine append: %w", err)
if status, err := topics.ValidateRawEnvelope(msg.subject, env); err != nil {
addFastMetric(registry, msg.subject, status)
if msg.ack != nil {
started := time.Now()
err := msg.ack()
recordFastWriterStageDuration(registry, msg.subject, "ack", statusFromError(err), time.Since(started))
if err != nil {
addFastMetric(registry, msg.subject, "ack_error")
return fmt.Errorf("nats ack mismatched raw envelope: %w", err)
}
}
return nil
}
if appender != nil {
started := time.Now()
err := appender.AppendAll(ctx, env)
recordFastWriterStageDuration(registry, msg.subject, "tdengine", statusFromError(err), time.Since(started))
if err != nil {
if isTransientFastBatchError(err) {
if catchupErr := updateFastRedisSingleWithoutAck(ctx, registry, updater, msg.subject, env); catchupErr != nil {
addFastDecoupledUpdateMetric(registry, "tdengine_transient", "error")
return fmt.Errorf("tdengine append: %w; redis catchup: %v", err, catchupErr)
}
addFastDecoupledUpdateMetric(registry, "tdengine_transient", "ok")
}
return fmt.Errorf("tdengine append: %w", err)
}
}
started := time.Now()
var result realtime.FastUpdateResult
var err error
if resultUpdater, ok := updater.(fastResultUpdater); ok {
result, err = resultUpdater.FastUpdateWithResult(ctx, env)
} else {
err = updater.FastUpdate(ctx, env)
}
started = time.Now()
err = updater.FastUpdate(ctx, env)
recordFastWriterStageDuration(registry, msg.subject, "redis", statusFromError(err), time.Since(started))
if err != nil {
return fmt.Errorf("redis fast update: %w", err)
}
recordFastWriterRedisEnvelopeMetrics(registry, msg.subject, result)
recordFastWriterRedisFieldMetrics(registry, msg.subject, result)
recordFastWriterRedisE2EDuration(registry, msg.subject, env)
if msg.ack != nil {
started = time.Now()
err = msg.ack()
recordFastWriterStageDuration(registry, msg.subject, "ack", statusFromError(err), time.Since(started))
if err != nil {
addFastMetric(registry, msg.subject, "ack_error")
return fmt.Errorf("nats ack: %w", err)
}
}
addFastMetric(registry, msg.subject, "ok")
return nil
}
func updateFastRedisBatchWithoutAck(ctx context.Context, registry *metrics.Registry, updater fastUpdater, messages []*fastMessage, envelopes []envelope.FrameEnvelope) error {
if len(envelopes) == 0 {
return nil
}
subject := fastBatchSubject(messages)
if batchUpdater, ok := updater.(fastBatchResultUpdater); ok {
started := time.Now()
result, err := batchUpdater.FastUpdateBatchWithResult(ctx, envelopes)
recordFastWriterStageDuration(registry, subject, "redis", statusFromError(err), time.Since(started))
if err == nil {
recordFastWriterRedisEnvelopeMetrics(registry, subject, result)
recordFastWriterRedisFieldMetrics(registry, subject, result)
recordFastWriterRedisE2EDurationMessages(registry, messages, envelopes, subject)
return nil
}
if shouldFallbackFastBatchError(err) {
addFastBatchFallbackMetric(registry, "redis_catchup", "attempted")
return updateFastRedisSinglesWithoutAck(ctx, registry, updater, messages, envelopes)
}
return err
}
if batchUpdater, ok := updater.(fastBatchUpdater); ok {
started := time.Now()
err := batchUpdater.FastUpdateBatch(ctx, envelopes)
recordFastWriterStageDuration(registry, subject, "redis", statusFromError(err), time.Since(started))
if err == nil {
recordFastWriterRedisE2EDurationMessages(registry, messages, envelopes, subject)
return nil
}
if shouldFallbackFastBatchError(err) {
addFastBatchFallbackMetric(registry, "redis_catchup", "attempted")
return updateFastRedisSinglesWithoutAck(ctx, registry, updater, messages, envelopes)
}
return err
}
return updateFastRedisSinglesWithoutAck(ctx, registry, updater, messages, envelopes)
}
func updateFastRedisSinglesWithoutAck(ctx context.Context, registry *metrics.Registry, updater fastUpdater, messages []*fastMessage, envelopes []envelope.FrameEnvelope) error {
for index, env := range envelopes {
subject := fastMessageSubject(messages, index, fastBatchSubject(messages))
if err := updateFastRedisSingleWithoutAck(ctx, registry, updater, subject, env); err != nil {
return err
}
}
return nil
}
func updateFastRedisSingleWithoutAck(ctx context.Context, registry *metrics.Registry, updater fastUpdater, subject string, env envelope.FrameEnvelope) error {
started := time.Now()
var result realtime.FastUpdateResult
var err error
if resultUpdater, ok := updater.(fastResultUpdater); ok {
result, err = resultUpdater.FastUpdateWithResult(ctx, env)
} else {
err = updater.FastUpdate(ctx, env)
}
recordFastWriterStageDuration(registry, subject, "redis", statusFromError(err), time.Since(started))
if err != nil {
return err
}
recordFastWriterRedisEnvelopeMetrics(registry, subject, result)
recordFastWriterRedisFieldMetrics(registry, subject, result)
recordFastWriterRedisE2EDuration(registry, subject, env)
return nil
}
func shouldFallbackFastBatchError(err error) bool {
if err == nil || errors.Is(err, context.Canceled) || errors.Is(err, context.DeadlineExceeded) {
return false
}
return !isTransientFastBatchError(err)
}
func isTransientFastBatchError(err error) bool {
if err == nil {
return false
}
text := strings.ToLower(strings.TrimSpace(err.Error()))
return strings.Contains(text, "timeout") ||
strings.Contains(text, "temporary") ||
strings.Contains(text, "temporarily") ||
strings.Contains(text, "connection refused") ||
strings.Contains(text, "connection reset") ||
strings.Contains(text, "connection closed") ||
strings.Contains(text, "broken pipe") ||
strings.Contains(text, "bad connection") ||
strings.Contains(text, "i/o timeout") ||
text == "eof" ||
strings.Contains(text, "unexpected eof") ||
strings.Contains(text, "server is down") ||
strings.Contains(text, "network is unreachable") ||
strings.Contains(text, "no route to host")
}
func recordFastWriterConfigMetrics(registry *metrics.Registry, cfg config) {
if registry == nil {
return
}
registry.SetGauge("vehicle_fast_writer_config", metrics.Labels{"setting": "workers"}, float64(cfg.Workers))
registry.SetGauge("vehicle_fast_writer_config", metrics.Labels{"setting": "batch_size"}, float64(cfg.BatchSize))
registry.SetGauge("vehicle_fast_writer_config", metrics.Labels{"setting": "fetch_wait_ms"}, float64(cfg.FetchWait.Milliseconds()))
registry.SetGauge("vehicle_fast_writer_config", metrics.Labels{"setting": "operation_timeout_ms"}, float64(cfg.OperationWait.Milliseconds()))
}
func boolGauge(value bool) float64 {
if value {
return 1
}
return 0
}
type fastSubjectGroup struct {
subject string
messages []*fastMessage
envelopes []envelope.FrameEnvelope
}
func fastSubjectGroups(messages []*fastMessage, envelopes []envelope.FrameEnvelope) []fastSubjectGroup {
if len(envelopes) == 0 {
return nil
}
fallback := fastBatchSubject(messages)
groups := make([]fastSubjectGroup, 0, len(envelopes))
indexBySubject := make(map[string]int, len(envelopes))
for index, env := range envelopes {
subject := fastMessageSubject(messages, index, fallback)
groupIndex, exists := indexBySubject[subject]
if !exists {
groupIndex = len(groups)
indexBySubject[subject] = groupIndex
groups = append(groups, fastSubjectGroup{subject: subject})
}
groups[groupIndex].envelopes = append(groups[groupIndex].envelopes, env)
if index >= 0 && index < len(messages) {
groups[groupIndex].messages = append(groups[groupIndex].messages, messages[index])
}
}
return groups
}
func fastBatchSubject(messages []*fastMessage) string {
if len(messages) == 0 {
return "unknown"
@@ -384,6 +737,16 @@ func fastBatchSubject(messages []*fastMessage) string {
return subject
}
func fastMessageSubject(messages []*fastMessage, index int, fallback string) string {
if index >= 0 && index < len(messages) && strings.TrimSpace(messages[index].subject) != "" {
return messages[index].subject
}
if strings.TrimSpace(fallback) != "" {
return fallback
}
return "unknown"
}
func ensureStream(js nats.JetStreamContext, cfg config) error {
ctx, cancel := context.WithTimeout(context.Background(), cfg.StreamEnsureWait)
defer cancel()
@@ -416,15 +779,36 @@ func addFastMetric(registry *metrics.Registry, subject string, status string) {
if registry == nil {
return
}
registry.IncCounter("vehicle_fast_writer_messages_total", metrics.Labels{"subject": subject, "status": status})
labels := metrics.Labels{"subject": subject, "status": status}
registry.IncCounter("vehicle_fast_writer_messages_total", labels)
metrics.RecordLastActivity(registry, "vehicle_fast_writer_last_message_unix_seconds", labels)
}
func setFastBatchPending(registry *metrics.Registry, messages int, envelopes int) {
func addFastBatchFallbackMetric(registry *metrics.Registry, stage string, status string) {
if registry == nil {
return
}
registry.SetGauge("vehicle_fast_writer_batch_pending_messages", nil, float64(messages))
registry.SetGauge("vehicle_fast_writer_batch_pending_envelopes", nil, float64(envelopes))
registry.IncCounter("vehicle_fast_writer_batch_fallback_total", metrics.Labels{
"stage": stage,
"status": status,
})
}
func addFastDecoupledUpdateMetric(registry *metrics.Registry, reason string, status string) {
if registry == nil {
return
}
registry.IncCounter("vehicle_fast_writer_decoupled_updates_total", metrics.Labels{
"reason": reason,
"status": status,
})
}
func addFastBatchPending(registry *metrics.Registry, messages int, envelopes int) {
if registry == nil {
return
}
fastBatchPending.Add(registry, "vehicle_fast_writer_batch_pending_messages", "vehicle_fast_writer_batch_pending_envelopes", messages, envelopes)
}
func recordFastNATSConsumerInfoMetrics(registry *metrics.Registry, cfg config, info *nats.ConsumerInfo) {
@@ -441,11 +825,78 @@ func recordFastWriterStageDuration(registry *metrics.Registry, subject string, s
if registry == nil {
return
}
registry.ObserveHistogram("vehicle_fast_writer_stage_duration_ms_histogram", metrics.Labels{
labels := metrics.Labels{
"subject": subject,
"stage": stage,
"status": status,
}, fastWriterStageDurationBucketsMS, float64(elapsed.Milliseconds()))
}
registry.ObserveHistogram("vehicle_fast_writer_stage_duration_ms_histogram", labels, fastWriterStageDurationBucketsMS, float64(elapsed.Milliseconds()))
metrics.RecordLastActivity(registry, "vehicle_fast_writer_last_stage_unix_seconds", labels)
}
func recordFastWriterRedisE2EDurationMessages(registry *metrics.Registry, messages []*fastMessage, envelopes []envelope.FrameEnvelope, fallback string) {
for index, env := range envelopes {
recordFastWriterRedisE2EDuration(registry, fastMessageSubject(messages, index, fallback), env)
}
}
func recordFastWriterRedisE2EDuration(registry *metrics.Registry, subject string, env envelope.FrameEnvelope) {
if registry == nil || env.ReceivedAtMS <= 0 {
return
}
elapsedMS := time.Since(time.UnixMilli(env.ReceivedAtMS)).Milliseconds()
if elapsedMS < 0 {
elapsedMS = 0
}
labels := metrics.Labels{
"subject": subject,
}
registry.ObserveHistogram("vehicle_fast_writer_redis_e2e_duration_ms_histogram", labels, fastWriterRedisE2EDurationBucketsMS, float64(elapsedMS))
p99, samples := fastWriterRedisE2ERecent.Observe(subject, float64(elapsedMS))
registry.SetGauge("vehicle_fast_writer_redis_e2e_recent_p99_ms", labels, p99)
registry.SetGauge("vehicle_fast_writer_redis_e2e_recent_samples", labels, float64(samples))
metrics.RecordLastActivity(registry, "vehicle_fast_writer_last_redis_e2e_unix_seconds", labels)
}
func recordFastWriterRedisFieldMetrics(registry *metrics.Registry, subject string, result realtime.FastUpdateResult) {
if registry == nil || result.FieldsSeen == 0 {
return
}
addFastWriterRedisFieldMetric(registry, subject, "seen", result.FieldsSeen)
addFastWriterRedisFieldMetric(registry, subject, "written", result.FieldsWritten)
addFastWriterRedisFieldMetric(registry, subject, "skipped_stale", result.FieldsSkippedStale)
}
func recordFastWriterRedisEnvelopeMetrics(registry *metrics.Registry, subject string, result realtime.FastUpdateResult) {
if registry == nil || result.EnvelopesSeen == 0 {
return
}
addFastWriterRedisEnvelopeMetric(registry, subject, "seen", result.EnvelopesSeen)
addFastWriterRedisEnvelopeMetric(registry, subject, "updated", result.EnvelopesUpdated)
addFastWriterRedisEnvelopeMetric(registry, subject, "skipped_non_realtime", result.EnvelopesSkippedNonRealtime)
addFastWriterRedisEnvelopeMetric(registry, subject, "skipped_missing_vin", result.EnvelopesSkippedMissingVIN)
addFastWriterRedisEnvelopeMetric(registry, subject, "skipped_missing_vehicle_key", result.EnvelopesSkippedMissingVehicleKey)
addFastWriterRedisEnvelopeMetric(registry, subject, "skipped_missing_fields", result.EnvelopesSkippedMissingFields)
}
func addFastWriterRedisEnvelopeMetric(registry *metrics.Registry, subject string, status string, value int) {
if value <= 0 {
return
}
registry.AddCounter("vehicle_fast_writer_redis_envelopes_total", metrics.Labels{
"subject": subject,
"status": status,
}, float64(value))
}
func addFastWriterRedisFieldMetric(registry *metrics.Registry, subject string, status string, value int) {
if value <= 0 {
return
}
registry.AddCounter("vehicle_fast_writer_redis_fields_total", metrics.Labels{
"subject": subject,
"status": status,
}, float64(value))
}
func statusFromError(err error) string {
@@ -487,6 +938,20 @@ func envInt64(key string, fallback int64) int64 {
return parsed
}
func envBool(key string, fallback bool) bool {
value := strings.ToLower(strings.TrimSpace(os.Getenv(key)))
switch value {
case "":
return fallback
case "1", "true", "yes", "y", "on":
return true
case "0", "false", "no", "n", "off":
return false
default:
return fallback
}
}
func splitCSV(value string) []string {
var out []string
for _, item := range strings.Split(value, ",") {

876
go/vehicle-gateway/cmd/nats-fast-writer/main_test.go
View File

@@ -12,6 +12,7 @@ import (
"lingniu-vehicle-ingest/go/vehicle-gateway/internal/envelope"
"lingniu-vehicle-ingest/go/vehicle-gateway/internal/metrics"
"lingniu-vehicle-ingest/go/vehicle-gateway/internal/realtime"
)
func TestProcessFastMessageWritesTDengineAndRedisBeforeAck(t *testing.T) {
@@ -58,8 +59,45 @@ func TestProcessFastMessageDoesNotAckWhenRedisFails(t *testing.T) {
}
}
func TestProcessFastMessageUpdatesRedisButDoesNotAckOnTransientTDengineFailure(t *testing.T) {
env := envelope.FrameEnvelope{Protocol: envelope.ProtocolJT808, VIN: "VIN001", EventID: "evt-td-transient-single"}
payload, err := env.MarshalJSONBytes()
if err != nil {
t.Fatal(err)
}
registry := metrics.NewRegistry()
appender := &recordingFastAppender{err: errors.New("connection reset by peer")}
updater := &recordingFastSingleUpdater{}
ackCount := 0
msg := &fastMessage{subject: "vehicle.raw.go.jt808.v1", data: payload, ack: func() error {
ackCount++
return nil
}}
if err := processFastMessage(context.Background(), registry, appender, updater, msg); err == nil {
t.Fatal("processFastMessage() error = nil, want tdengine transient failure")
}
if updater.count != 1 || ackCount != 0 {
t.Fatalf("updates=%d acks=%d, want 1/0", updater.count, ackCount)
}
text := registry.Render()
for _, want := range []string{
`vehicle_fast_writer_messages_total{status="received",subject="vehicle.raw.go.jt808.v1"} 1`,
`vehicle_fast_writer_last_message_unix_seconds{status="received",subject="vehicle.raw.go.jt808.v1"} `,
`vehicle_fast_writer_decoupled_updates_total{reason="tdengine_transient",status="ok"} 1`,
`vehicle_fast_writer_stage_duration_ms_histogram_bucket{le="+Inf",stage="redis",status="ok",subject="vehicle.raw.go.jt808.v1"} 1`,
} {
if !strings.Contains(text, want) {
t.Fatalf("tdengine transient redis catchup metric missing %s:\n%s", want, text)
}
}
if strings.Contains(text, `vehicle_fast_writer_messages_total{status="ok",subject="vehicle.raw.go.jt808.v1"}`) {
t.Fatalf("message should not be counted ok because it was not acked:\n%s", text)
}
}
func TestProcessFastMessageRecordsStageDurationMetrics(t *testing.T) {
env := envelope.FrameEnvelope{Protocol: envelope.ProtocolJT808, VIN: "VIN001", EventID: "evt-3"}
env := envelope.FrameEnvelope{Protocol: envelope.ProtocolJT808, VIN: "VIN001", EventID: "evt-3", ReceivedAtMS: time.Now().Add(-20 * time.Millisecond).UnixMilli()}
payload, err := env.MarshalJSONBytes()
if err != nil {
t.Fatal(err)
@@ -77,6 +115,14 @@ func TestProcessFastMessageRecordsStageDurationMetrics(t *testing.T) {
`vehicle_fast_writer_stage_duration_ms_histogram_count{stage="tdengine",status="ok",subject="vehicle.raw.go.jt808.v1"} 1`,
`vehicle_fast_writer_stage_duration_ms_histogram_bucket{le="+Inf",stage="redis",status="ok",subject="vehicle.raw.go.jt808.v1"} 1`,
`vehicle_fast_writer_stage_duration_ms_histogram_bucket{le="+Inf",stage="ack",status="ok",subject="vehicle.raw.go.jt808.v1"} 1`,
`vehicle_fast_writer_last_stage_unix_seconds{stage="tdengine",status="ok",subject="vehicle.raw.go.jt808.v1"} `,
`vehicle_fast_writer_last_stage_unix_seconds{stage="redis",status="ok",subject="vehicle.raw.go.jt808.v1"} `,
`vehicle_fast_writer_last_stage_unix_seconds{stage="ack",status="ok",subject="vehicle.raw.go.jt808.v1"} `,
`vehicle_fast_writer_redis_e2e_duration_ms_histogram_bucket{le="+Inf",subject="vehicle.raw.go.jt808.v1"} 1`,
`vehicle_fast_writer_redis_e2e_duration_ms_histogram_count{subject="vehicle.raw.go.jt808.v1"} 1`,
`vehicle_fast_writer_redis_e2e_recent_p99_ms{subject="vehicle.raw.go.jt808.v1"} `,
`vehicle_fast_writer_redis_e2e_recent_samples{subject="vehicle.raw.go.jt808.v1"} `,
`vehicle_fast_writer_last_redis_e2e_unix_seconds{subject="vehicle.raw.go.jt808.v1"} `,
} {
if !strings.Contains(text, want) {
t.Fatalf("fast writer stage metric missing %s:\n%s", want, text)
@@ -84,6 +130,151 @@ func TestProcessFastMessageRecordsStageDurationMetrics(t *testing.T) {
}
}
func TestRecordFastWriterRedisE2EDurationSkipsMissingReceiveTime(t *testing.T) {
registry := metrics.NewRegistry()
recordFastWriterRedisE2EDuration(registry, "vehicle.raw.go.jt808.v1", envelope.FrameEnvelope{})
if text := registry.Render(); strings.Contains(text, "vehicle_fast_writer_redis_e2e_duration_ms_histogram") || strings.Contains(text, "vehicle_fast_writer_redis_e2e_recent") {
t.Fatalf("e2e metric should be skipped when received_at_ms is missing:\n%s", text)
}
}
func TestProcessFastMessageAcksInvalidJSONAndRecordsMetric(t *testing.T) {
registry := metrics.NewRegistry()
ackCount := 0
msg := &fastMessage{subject: "vehicle.raw.go.jt808.v1", data: []byte(`{bad json`), ack: func() error {
ackCount++
return nil
}}
updater := &recordingFastUpdater{}
if err := processFastMessage(context.Background(), registry, nil, updater, msg); err != nil {
t.Fatalf("processFastMessage() error = %v", err)
}
if ackCount != 1 {
t.Fatalf("ack count = %d, want 1", ackCount)
}
if updater.count != 0 || updater.batchCount != 0 {
t.Fatalf("updater counts = single %d batch %d, want 0", updater.count, updater.batchCount)
}
text := registry.Render()
for _, want := range []string{
`vehicle_fast_writer_messages_total{status="invalid_json",subject="vehicle.raw.go.jt808.v1"} 1`,
`vehicle_fast_writer_stage_duration_ms_histogram_bucket{le="+Inf",stage="ack",status="ok",subject="vehicle.raw.go.jt808.v1"} 1`,
} {
if !strings.Contains(text, want) {
t.Fatalf("invalid json metric missing %s:\n%s", want, text)
}
}
if strings.Contains(text, `vehicle_fast_writer_messages_total{status="ok",subject="vehicle.raw.go.jt808.v1"}`) {
t.Fatalf("invalid json should not be counted as ok:\n%s", text)
}
}
func TestProcessFastMessageAcksExplicitNonRawEventKindWithoutWriting(t *testing.T) {
env := envelope.FrameEnvelope{
Protocol: envelope.ProtocolJT808,
EventKind: envelope.EventKindFields,
MessageID: "0x0200",
VIN: "VIN001",
}
payload, err := env.MarshalJSONBytes()
if err != nil {
t.Fatal(err)
}
registry := metrics.NewRegistry()
appender := &recordingFastAppender{}
updater := &recordingFastUpdater{}
ackCount := 0
msg := &fastMessage{subject: "vehicle.raw.go.jt808.v1", data: payload, ack: func() error {
ackCount++
return nil
}}
if err := processFastMessage(context.Background(), registry, appender, updater, msg); err != nil {
t.Fatalf("processFastMessage() error = %v", err)
}
if appender.count != 0 || updater.count != 0 || updater.batchCount != 0 {
t.Fatalf("appends=%d updates=%d batch_updates=%d, want no writes", appender.count, updater.count, updater.batchCount)
}
if ackCount != 1 {
t.Fatalf("ack count=%d, want 1", ackCount)
}
text := registry.Render()
for _, want := range []string{
`vehicle_fast_writer_messages_total{status="event_kind_mismatch",subject="vehicle.raw.go.jt808.v1"} 1`,
`vehicle_fast_writer_stage_duration_ms_histogram_count{stage="ack",status="ok",subject="vehicle.raw.go.jt808.v1"} 1`,
} {
if !strings.Contains(text, want) {
t.Fatalf("mismatch metric missing %s:\n%s", want, text)
}
}
if strings.Contains(text, `vehicle_fast_writer_messages_total{status="ok",subject="vehicle.raw.go.jt808.v1"}`) {
t.Fatalf("mismatched event should not be counted ok:\n%s", text)
}
}
func TestProcessFastMessageRecordsRedisFieldMetrics(t *testing.T) {
env := envelope.FrameEnvelope{Protocol: envelope.ProtocolJT808, VIN: "VIN001", EventID: "evt-fields"}
payload, err := env.MarshalJSONBytes()
if err != nil {
t.Fatal(err)
}
registry := metrics.NewRegistry()
updater := &recordingFastResultUpdater{result: realtime.FastUpdateResult{
EnvelopesSeen: 4,
EnvelopesUpdated: 1,
EnvelopesSkippedNonRealtime: 1,
EnvelopesSkippedMissingVIN: 1,
EnvelopesSkippedMissingVehicleKey: 0,
EnvelopesSkippedMissingFields: 1,
FieldsSeen: 3,
FieldsWritten: 2,
FieldsSkippedStale: 1,
}}
msg := &fastMessage{subject: "vehicle.raw.go.jt808.v1", data: payload, ack: func() error { return nil }}
if err := processFastMessage(context.Background(), registry, nil, updater, msg); err != nil {
t.Fatalf("processFastMessage() error = %v", err)
}
if updater.resultCount != 1 {
t.Fatalf("FastUpdateWithResult count=%d, want 1", updater.resultCount)
}
text := registry.Render()
for _, want := range []string{
`vehicle_fast_writer_redis_envelopes_total{status="seen",subject="vehicle.raw.go.jt808.v1"} 4`,
`vehicle_fast_writer_redis_envelopes_total{status="updated",subject="vehicle.raw.go.jt808.v1"} 1`,
`vehicle_fast_writer_redis_envelopes_total{status="skipped_missing_vin",subject="vehicle.raw.go.jt808.v1"} 1`,
`vehicle_fast_writer_redis_envelopes_total{status="skipped_non_realtime",subject="vehicle.raw.go.jt808.v1"} 1`,
`vehicle_fast_writer_redis_envelopes_total{status="skipped_missing_fields",subject="vehicle.raw.go.jt808.v1"} 1`,
`vehicle_fast_writer_redis_fields_total{status="seen",subject="vehicle.raw.go.jt808.v1"} 3`,
`vehicle_fast_writer_redis_fields_total{status="skipped_stale",subject="vehicle.raw.go.jt808.v1"} 1`,
`vehicle_fast_writer_redis_fields_total{status="written",subject="vehicle.raw.go.jt808.v1"} 2`,
} {
if !strings.Contains(text, want) {
t.Fatalf("redis field metric missing %s:\n%s", want, text)
}
}
}
func TestAddFastMetricRecordsLastMessage(t *testing.T) {
registry := metrics.NewRegistry()
addFastMetric(registry, "vehicle.raw.go.gb32960.v1", "ok")
text := registry.Render()
for _, want := range []string{
`vehicle_fast_writer_messages_total{status="ok",subject="vehicle.raw.go.gb32960.v1"} 1`,
`vehicle_fast_writer_last_message_unix_seconds{status="ok",subject="vehicle.raw.go.gb32960.v1"} `,
} {
if !strings.Contains(text, want) {
t.Fatalf("fast writer message metric missing %s:\n%s", want, text)
}
}
}
func TestProcessFastBatchAppendsTDengineBatchBeforeRedisAndAck(t *testing.T) {
first := envelope.FrameEnvelope{Protocol: envelope.ProtocolJT808, VIN: "VIN001", EventID: "evt-4"}
second := envelope.FrameEnvelope{Protocol: envelope.ProtocolJT808, VIN: "VIN002", EventID: "evt-5"}
@@ -149,6 +340,407 @@ func TestProcessFastBatchUsesRedisBatchUpdaterWhenAvailable(t *testing.T) {
}
}
func TestProcessFastBatchFallsBackToSinglesAfterNonTransientTDengineBatchFailure(t *testing.T) {
first := envelope.FrameEnvelope{Protocol: envelope.ProtocolJT808, VIN: "VIN001", EventID: "evt-fallback-td-1"}
second := envelope.FrameEnvelope{Protocol: envelope.ProtocolJT808, VIN: "VIN002", EventID: "evt-fallback-td-2"}
firstPayload, err := first.MarshalJSONBytes()
if err != nil {
t.Fatal(err)
}
secondPayload, err := second.MarshalJSONBytes()
if err != nil {
t.Fatal(err)
}
registry := metrics.NewRegistry()
appender := &recordingFastAppender{batchErr: errors.New("syntax error in batch insert")}
updater := &recordingFastSingleUpdater{}
ackCount := 0
msgs := []*fastMessage{
{subject: "vehicle.raw.go.jt808.v1", data: firstPayload, ack: func() error { ackCount++; return nil }},
{subject: "vehicle.raw.go.jt808.v1", data: secondPayload, ack: func() error { ackCount++; return nil }},
}
if err := processFastBatch(context.Background(), registry, appender, updater, msgs); err != nil {
t.Fatalf("processFastBatch() error = %v", err)
}
if appender.batchCount != 1 || appender.count != 2 {
t.Fatalf("batch appends=%d single appends=%d, want 1/2", appender.batchCount, appender.count)
}
if updater.count != 2 || ackCount != 2 {
t.Fatalf("updates=%d acks=%d, want 2/2", updater.count, ackCount)
}
text := registry.Render()
if !strings.Contains(text, `vehicle_fast_writer_batch_fallback_total{stage="tdengine",status="ok"} 1`) {
t.Fatalf("tdengine fallback metric missing:\n%s", text)
}
}
func TestProcessFastBatchUpdatesRedisButDoesNotAckOnTransientTDengineBatchFailure(t *testing.T) {
env := envelope.FrameEnvelope{Protocol: envelope.ProtocolJT808, VIN: "VIN001", EventID: "evt-fallback-td-transient"}
payload, err := env.MarshalJSONBytes()
if err != nil {
t.Fatal(err)
}
registry := metrics.NewRegistry()
appender := &recordingFastAppender{batchErr: errors.New("i/o timeout")}
updater := &recordingFastSingleUpdater{}
ackCount := 0
msgs := []*fastMessage{
{subject: "vehicle.raw.go.jt808.v1", data: payload, ack: func() error { ackCount++; return nil }},
}
if err := processFastBatch(context.Background(), registry, appender, updater, msgs); err == nil {
t.Fatal("processFastBatch() error = nil, want transient tdengine failure")
}
if appender.batchCount != 1 || appender.count != 0 {
t.Fatalf("batch appends=%d single appends=%d, want 1/0", appender.batchCount, appender.count)
}
if updater.count != 1 || ackCount != 0 {
t.Fatalf("updates=%d acks=%d, want 1/0", updater.count, ackCount)
}
text := registry.Render()
for _, want := range []string{
`vehicle_fast_writer_messages_total{status="received",subject="vehicle.raw.go.jt808.v1"} 1`,
`vehicle_fast_writer_batch_fallback_total{stage="tdengine",status="skipped_transient"} 1`,
`vehicle_fast_writer_decoupled_updates_total{reason="tdengine_transient",status="ok"} 1`,
`vehicle_fast_writer_stage_duration_ms_histogram_bucket{le="+Inf",stage="redis",status="ok",subject="vehicle.raw.go.jt808.v1"} 1`,
} {
if !strings.Contains(text, want) {
t.Fatalf("tdengine transient catchup metric missing %s:\n%s", want, text)
}
}
}
func TestProcessFastBatchFallsBackToRedisSinglesAfterNonTransientRedisBatchFailure(t *testing.T) {
first := envelope.FrameEnvelope{Protocol: envelope.ProtocolJT808, VIN: "VIN001", EventID: "evt-fallback-redis-1"}
second := envelope.FrameEnvelope{Protocol: envelope.ProtocolJT808, VIN: "VIN002", EventID: "evt-fallback-redis-2"}
firstPayload, err := first.MarshalJSONBytes()
if err != nil {
t.Fatal(err)
}
secondPayload, err := second.MarshalJSONBytes()
if err != nil {
t.Fatal(err)
}
registry := metrics.NewRegistry()
appender := &recordingFastAppender{}
updater := &recordingFastUpdater{batchErr: errors.New("WRONGTYPE operation against a key holding the wrong kind of value")}
ackCount := 0
msgs := []*fastMessage{
{subject: "vehicle.raw.go.jt808.v1", data: firstPayload, ack: func() error { ackCount++; return nil }},
{subject: "vehicle.raw.go.jt808.v1", data: secondPayload, ack: func() error { ackCount++; return nil }},
}
if err := processFastBatch(context.Background(), registry, appender, updater, msgs); err != nil {
t.Fatalf("processFastBatch() error = %v", err)
}
if appender.batchCount != 1 || appender.count != 0 {
t.Fatalf("tdengine batch appends=%d single appends=%d, want 1/0", appender.batchCount, appender.count)
}
if updater.batchCount != 1 || updater.count != 2 {
t.Fatalf("redis batch updates=%d single updates=%d, want 1/2", updater.batchCount, updater.count)
}
if ackCount != 2 {
t.Fatalf("ack count=%d, want 2", ackCount)
}
text := registry.Render()
if !strings.Contains(text, `vehicle_fast_writer_batch_fallback_total{stage="redis",status="ok"} 1`) {
t.Fatalf("redis fallback metric missing:\n%s", text)
}
}
func TestProcessFastBatchFallbackLeavesFailedAndRemainingMessagesUnacked(t *testing.T) {
first := envelope.FrameEnvelope{Protocol: envelope.ProtocolJT808, VIN: "VIN001", EventID: "evt-fallback-prefix-1"}
second := envelope.FrameEnvelope{Protocol: envelope.ProtocolJT808, VIN: "VIN002", EventID: "evt-fallback-prefix-2"}
firstPayload, err := first.MarshalJSONBytes()
if err != nil {
t.Fatal(err)
}
secondPayload, err := second.MarshalJSONBytes()
if err != nil {
t.Fatal(err)
}
registry := metrics.NewRegistry()
appender := &recordingFastAppender{
batchErr: errors.New("syntax error in batch insert"),
singleErr: errors.New("single insert failed"),
failSingleOnCount: 2,
}
updater := &recordingFastSingleUpdater{}
ackCount := 0
msgs := []*fastMessage{
{subject: "vehicle.raw.go.jt808.v1", data: firstPayload, ack: func() error { ackCount++; return nil }},
{subject: "vehicle.raw.go.jt808.v1", data: secondPayload, ack: func() error { ackCount++; return nil }},
}
if err := processFastBatch(context.Background(), registry, appender, updater, msgs); err == nil {
t.Fatal("processFastBatch() error = nil, want fallback single write failure")
}
if appender.batchCount != 1 || appender.count != 2 {
t.Fatalf("tdengine batch appends=%d single appends=%d, want 1/2", appender.batchCount, appender.count)
}
if updater.count != 1 || ackCount != 1 {
t.Fatalf("updates=%d acks=%d, want successful prefix 1/1", updater.count, ackCount)
}
text := registry.Render()
if !strings.Contains(text, `vehicle_fast_writer_batch_fallback_total{stage="tdengine",status="error"} 1`) {
t.Fatalf("tdengine fallback error metric missing:\n%s", text)
}
}
func TestProcessFastBatchRecordsRedisFieldMetrics(t *testing.T) {
first := envelope.FrameEnvelope{Protocol: envelope.ProtocolJT808, VIN: "VIN001", EventID: "evt-batch-fields-1"}
second := envelope.FrameEnvelope{Protocol: envelope.ProtocolJT808, VIN: "VIN002", EventID: "evt-batch-fields-2"}
firstPayload, err := first.MarshalJSONBytes()
if err != nil {
t.Fatal(err)
}
secondPayload, err := second.MarshalJSONBytes()
if err != nil {
t.Fatal(err)
}
registry := metrics.NewRegistry()
updater := &recordingFastResultUpdater{result: realtime.FastUpdateResult{
EnvelopesSeen: 2,
EnvelopesUpdated: 2,
FieldsSeen: 10,
FieldsWritten: 7,
FieldsSkippedStale: 3,
}}
ackCount := 0
msgs := []*fastMessage{
{subject: "vehicle.raw.go.jt808.v1", data: firstPayload, ack: func() error { ackCount++; return nil }},
{subject: "vehicle.raw.go.jt808.v1", data: secondPayload, ack: func() error { ackCount++; return nil }},
}
if err := processFastBatch(context.Background(), registry, nil, updater, msgs); err != nil {
t.Fatalf("processFastBatch() error = %v", err)
}
if updater.batchResultCount != 1 || updater.batchCount != 0 {
t.Fatalf("batch result count=%d legacy batch count=%d, want 1/0", updater.batchResultCount, updater.batchCount)
}
if ackCount != 2 {
t.Fatalf("ack count=%d, want 2", ackCount)
}
text := registry.Render()
for _, want := range []string{
`vehicle_fast_writer_redis_envelopes_total{status="seen",subject="vehicle.raw.go.jt808.v1"} 2`,
`vehicle_fast_writer_redis_envelopes_total{status="updated",subject="vehicle.raw.go.jt808.v1"} 2`,
`vehicle_fast_writer_redis_fields_total{status="seen",subject="vehicle.raw.go.jt808.v1"} 10`,
`vehicle_fast_writer_redis_fields_total{status="skipped_stale",subject="vehicle.raw.go.jt808.v1"} 3`,
`vehicle_fast_writer_redis_fields_total{status="written",subject="vehicle.raw.go.jt808.v1"} 7`,
} {
if !strings.Contains(text, want) {
t.Fatalf("redis field metric missing %s:\n%s", want, text)
}
}
}
func TestProcessFastBatchRecordsRedisE2EByOriginalSubject(t *testing.T) {
first := envelope.FrameEnvelope{
Protocol: envelope.ProtocolJT808,
VIN: "VIN001",
EventID: "evt-batch-e2e-1",
ReceivedAtMS: time.Now().Add(-20 * time.Millisecond).UnixMilli(),
}
second := envelope.FrameEnvelope{
Protocol: envelope.ProtocolGB32960,
VIN: "VIN002",
EventID: "evt-batch-e2e-2",
ReceivedAtMS: time.Now().Add(-25 * time.Millisecond).UnixMilli(),
}
firstPayload, err := first.MarshalJSONBytes()
if err != nil {
t.Fatal(err)
}
secondPayload, err := second.MarshalJSONBytes()
if err != nil {
t.Fatal(err)
}
registry := metrics.NewRegistry()
updater := &recordingFastResultUpdater{result: realtime.FastUpdateResult{
EnvelopesSeen: 2,
EnvelopesUpdated: 2,
}}
msgs := []*fastMessage{
{subject: "vehicle.raw.go.jt808.v1", data: firstPayload},
{subject: "vehicle.raw.go.gb32960.v1", data: secondPayload},
}
if err := processFastBatch(context.Background(), registry, nil, updater, msgs); err != nil {
t.Fatalf("processFastBatch() error = %v", err)
}
text := registry.Render()
for _, want := range []string{
`vehicle_fast_writer_redis_e2e_duration_ms_histogram_count{subject="vehicle.raw.go.jt808.v1"} 1`,
`vehicle_fast_writer_redis_e2e_duration_ms_histogram_count{subject="vehicle.raw.go.gb32960.v1"} 1`,
`vehicle_fast_writer_redis_e2e_recent_p99_ms{subject="vehicle.raw.go.jt808.v1"} `,
`vehicle_fast_writer_redis_e2e_recent_p99_ms{subject="vehicle.raw.go.gb32960.v1"} `,
} {
if !strings.Contains(text, want) {
t.Fatalf("redis e2e metric missing %s:\n%s", want, text)
}
}
if strings.Contains(text, `vehicle_fast_writer_redis_e2e_duration_ms_histogram_count{subject="mixed"}`) {
t.Fatalf("redis e2e metric should keep original subjects, not mixed:\n%s", text)
}
}
func TestProcessFastBatchRecordsRedisFieldMetricsByOriginalSubjectForMixedBatch(t *testing.T) {
first := envelope.FrameEnvelope{Protocol: envelope.ProtocolJT808, VIN: "VIN001", EventID: "evt-batch-mixed-fields-1"}
second := envelope.FrameEnvelope{Protocol: envelope.ProtocolGB32960, VIN: "VIN002", EventID: "evt-batch-mixed-fields-2"}
firstPayload, err := first.MarshalJSONBytes()
if err != nil {
t.Fatal(err)
}
secondPayload, err := second.MarshalJSONBytes()
if err != nil {
t.Fatal(err)
}
registry := metrics.NewRegistry()
updater := &recordingFastDynamicResultUpdater{}
msgs := []*fastMessage{
{subject: "vehicle.raw.go.jt808.v1", data: firstPayload},
{subject: "vehicle.raw.go.gb32960.v1", data: secondPayload},
}
if err := processFastBatch(context.Background(), registry, nil, updater, msgs); err != nil {
t.Fatalf("processFastBatch() error = %v", err)
}
if updater.batchResultCount != 2 {
t.Fatalf("batch result count=%d, want one batch per subject", updater.batchResultCount)
}
text := registry.Render()
for _, want := range []string{
`vehicle_fast_writer_redis_envelopes_total{status="seen",subject="vehicle.raw.go.jt808.v1"} 1`,
`vehicle_fast_writer_redis_envelopes_total{status="seen",subject="vehicle.raw.go.gb32960.v1"} 1`,
`vehicle_fast_writer_redis_fields_total{status="seen",subject="vehicle.raw.go.jt808.v1"} 1`,
`vehicle_fast_writer_redis_fields_total{status="seen",subject="vehicle.raw.go.gb32960.v1"} 1`,
`vehicle_fast_writer_redis_fields_total{status="written",subject="vehicle.raw.go.jt808.v1"} 1`,
`vehicle_fast_writer_redis_fields_total{status="written",subject="vehicle.raw.go.gb32960.v1"} 1`,
} {
if !strings.Contains(text, want) {
t.Fatalf("redis field metric missing %s:\n%s", want, text)
}
}
if strings.Contains(text, `subject="mixed"`) {
t.Fatalf("mixed subject should not be used for per-subject Redis metrics:\n%s", text)
}
}
func TestProcessFastBatchAcksInvalidJSONWithoutMarkingItOK(t *testing.T) {
valid := envelope.FrameEnvelope{Protocol: envelope.ProtocolJT808, VIN: "VIN001", EventID: "evt-valid"}
validPayload, err := valid.MarshalJSONBytes()
if err != nil {
t.Fatal(err)
}
registry := metrics.NewRegistry()
updater := &recordingFastUpdater{}
ackCount := 0
msgs := []*fastMessage{
{subject: "vehicle.raw.go.jt808.v1", data: []byte(`{bad json`), ack: func() error { ackCount++; return nil }},
{subject: "vehicle.raw.go.jt808.v1", data: validPayload, ack: func() error { ackCount++; return nil }},
}
if err := processFastBatch(context.Background(), registry, nil, updater, msgs); err != nil {
t.Fatalf("processFastBatch() error = %v", err)
}
if ackCount != 2 {
t.Fatalf("ack count = %d, want 2", ackCount)
}
if updater.batchCount != 1 || updater.batchRows != 1 {
t.Fatalf("redis batch count=%d rows=%d, want one valid row", updater.batchCount, updater.batchRows)
}
text := registry.Render()
for _, want := range []string{
`vehicle_fast_writer_messages_total{status="received",subject="vehicle.raw.go.jt808.v1"} 2`,
`vehicle_fast_writer_messages_total{status="invalid_json",subject="vehicle.raw.go.jt808.v1"} 1`,
`vehicle_fast_writer_messages_total{status="ok",subject="vehicle.raw.go.jt808.v1"} 1`,
`vehicle_fast_writer_batch_pending_messages 0`,
`vehicle_fast_writer_batch_pending_envelopes 0`,
} {
if !strings.Contains(text, want) {
t.Fatalf("batch invalid json metric missing %s:\n%s", want, text)
}
}
}
func TestProcessFastBatchSkipsExplicitNonRawEventKindAndWritesValidRows(t *testing.T) {
mismatched := envelope.FrameEnvelope{Protocol: envelope.ProtocolJT808, EventKind: envelope.EventKindFields, VIN: "VIN_BAD", EventID: "evt-fields-on-raw"}
valid := envelope.FrameEnvelope{Protocol: envelope.ProtocolJT808, EventKind: envelope.EventKindRaw, VIN: "VIN001", EventID: "evt-valid-raw"}
mismatchedPayload, err := mismatched.MarshalJSONBytes()
if err != nil {
t.Fatal(err)
}
validPayload, err := valid.MarshalJSONBytes()
if err != nil {
t.Fatal(err)
}
registry := metrics.NewRegistry()
updater := &recordingFastUpdater{}
ackCount := 0
msgs := []*fastMessage{
{subject: "vehicle.raw.go.jt808.v1", data: mismatchedPayload, ack: func() error { ackCount++; return nil }},
{subject: "vehicle.raw.go.jt808.v1", data: validPayload, ack: func() error { ackCount++; return nil }},
}
if err := processFastBatch(context.Background(), registry, nil, updater, msgs); err != nil {
t.Fatalf("processFastBatch() error = %v", err)
}
if ackCount != 2 {
t.Fatalf("ack count = %d, want 2", ackCount)
}
if updater.batchCount != 1 || updater.batchRows != 1 {
t.Fatalf("redis batch count=%d rows=%d, want only valid row", updater.batchCount, updater.batchRows)
}
text := registry.Render()
for _, want := range []string{
`vehicle_fast_writer_messages_total{status="received",subject="vehicle.raw.go.jt808.v1"} 2`,
`vehicle_fast_writer_messages_total{status="event_kind_mismatch",subject="vehicle.raw.go.jt808.v1"} 1`,
`vehicle_fast_writer_messages_total{status="ok",subject="vehicle.raw.go.jt808.v1"} 1`,
`vehicle_fast_writer_batch_pending_messages 0`,
`vehicle_fast_writer_batch_pending_envelopes 0`,
} {
if !strings.Contains(text, want) {
t.Fatalf("batch mismatch metric missing %s:\n%s", want, text)
}
}
}
func TestProcessFastBatchCanSkipTDengineAppender(t *testing.T) {
env := envelope.FrameEnvelope{Protocol: envelope.ProtocolJT808, VIN: "VIN001", EventID: "evt-no-td"}
payload, err := env.MarshalJSONBytes()
if err != nil {
t.Fatal(err)
}
registry := metrics.NewRegistry()
updater := &recordingFastUpdater{}
ackCount := 0
msgs := []*fastMessage{
{subject: "vehicle.raw.go.jt808.v1", data: payload, ack: func() error { ackCount++; return nil }},
}
if err := processFastBatch(context.Background(), registry, nil, updater, msgs); err != nil {
t.Fatalf("processFastBatch() error = %v", err)
}
if updater.batchCount != 1 || updater.batchRows != 1 {
t.Fatalf("redis batch count=%d rows=%d, want 1/1", updater.batchCount, updater.batchRows)
}
if ackCount != 1 {
t.Fatalf("ack count=%d, want 1", ackCount)
}
text := registry.Render()
if strings.Contains(text, `stage="tdengine"`) {
t.Fatalf("tdengine stage should not be recorded when appender is nil:\n%s", text)
}
if !strings.Contains(text, `stage="redis"`) || !strings.Contains(text, `stage="ack"`) {
t.Fatalf("redis and ack stages should still be recorded:\n%s", text)
}
}
func TestProcessFastBatchExposesPendingMetricsDuringAppend(t *testing.T) {
first := envelope.FrameEnvelope{Protocol: envelope.ProtocolJT808, VIN: "VIN001", EventID: "evt-6"}
second := envelope.FrameEnvelope{Protocol: envelope.ProtocolJT808, VIN: "VIN002", EventID: "evt-7"}
@@ -208,6 +800,77 @@ func TestProcessFastBatchExposesPendingMetricsDuringAppend(t *testing.T) {
}
}
func TestProcessFastBatchPendingAggregatesConcurrentWorkers(t *testing.T) {
first := envelope.FrameEnvelope{Protocol: envelope.ProtocolJT808, VIN: "VIN001", EventID: "evt-fast-pending-concurrent-1"}
second := envelope.FrameEnvelope{Protocol: envelope.ProtocolJT808, VIN: "VIN002", EventID: "evt-fast-pending-concurrent-2"}
firstPayload, err := first.MarshalJSONBytes()
if err != nil {
t.Fatal(err)
}
secondPayload, err := second.MarshalJSONBytes()
if err != nil {
t.Fatal(err)
}
registry := metrics.NewRegistry()
firstStarted := make(chan struct{})
secondStarted := make(chan struct{})
release := make(chan struct{})
appenderFor := func(started chan struct{}) *recordingFastAppender {
return &recordingFastAppender{
onAppendBatch: func() {
close(started)
<-release
},
}
}
messages := func(payload []byte, count int) []*fastMessage {
out := make([]*fastMessage, 0, count)
for i := 0; i < count; i++ {
out = append(out, &fastMessage{subject: "vehicle.raw.go.jt808.v1", data: payload, ack: func() error { return nil }})
}
return out
}
var wg sync.WaitGroup
wg.Add(2)
var firstErr error
var secondErr error
go func() {
defer wg.Done()
firstErr = processFastBatch(context.Background(), registry, appenderFor(firstStarted), &recordingFastUpdater{}, messages(firstPayload, 2))
}()
go func() {
defer wg.Done()
secondErr = processFastBatch(context.Background(), registry, appenderFor(secondStarted), &recordingFastUpdater{}, messages(secondPayload, 3))
}()
<-firstStarted
<-secondStarted
text := registry.Render()
for _, want := range []string{
`vehicle_fast_writer_batch_pending_messages 5`,
`vehicle_fast_writer_batch_pending_envelopes 5`,
} {
if !strings.Contains(text, want) {
t.Fatalf("aggregate pending metric missing %s during concurrent append:\n%s", want, text)
}
}
close(release)
wg.Wait()
if firstErr != nil || secondErr != nil {
t.Fatalf("processFastBatch errors = %v / %v", firstErr, secondErr)
}
text = registry.Render()
for _, want := range []string{
`vehicle_fast_writer_batch_pending_messages 0`,
`vehicle_fast_writer_batch_pending_envelopes 0`,
} {
if !strings.Contains(text, want) {
t.Fatalf("aggregate pending metric should reset after concurrent append, missing %s:\n%s", want, text)
}
}
}
func TestRecordFastNATSConsumerInfoMetrics(t *testing.T) {
registry := metrics.NewRegistry()
cfg := config{NATSStream: "VEHICLE_INGEST", NATSDurable: "vehicle-fast-writer"}
@@ -230,6 +893,41 @@ func TestRecordFastNATSConsumerInfoMetrics(t *testing.T) {
}
}
func TestIsFastWorkerShutdownFetchError(t *testing.T) {
if isFastWorkerShutdownFetchError(context.Background(), errors.New("temporary nats failure")) {
t.Fatal("temporary failure should not be treated as shutdown")
}
if !isFastWorkerShutdownFetchError(context.Background(), nats.ErrConnectionClosed) {
t.Fatal("connection closed should stop worker without noisy error log")
}
ctx, cancel := context.WithCancel(context.Background())
cancel()
if !isFastWorkerShutdownFetchError(ctx, errors.New("any fetch error after cancellation")) {
t.Fatal("cancelled context should stop worker without noisy error log")
}
}
func TestIsTransientFastFetchError(t *testing.T) {
for _, err := range []error{
errors.New("nats: disconnected during fetch"),
errors.New("nats: connection closed"),
errors.New("read tcp: connection reset by peer"),
errors.New("write tcp: broken pipe"),
errors.New("temporary network unavailable"),
errors.New("i/o timeout"),
} {
if !isTransientFastFetchError(err) {
t.Fatalf("isTransientFastFetchError(%q) = false, want true", err.Error())
}
}
if isTransientFastFetchError(errors.New("permission denied")) {
t.Fatal("permission denied should remain an unexpected fetch error")
}
if isTransientFastFetchError(nil) {
t.Fatal("nil should not be treated as transient")
}
}
func TestLoadConfigDefaultsTDenginePoolToSingleConnection(t *testing.T) {
t.Setenv("FAST_WRITER_WORKERS", "12")
t.Setenv("FAST_WRITER_TDENGINE_MAX_OPEN_CONNS", "")
@@ -244,6 +942,28 @@ func TestLoadConfigDefaultsTDenginePoolToSingleConnection(t *testing.T) {
}
}
func TestLoadConfigDefaultsTDengineStageDisabled(t *testing.T) {
cfg := loadConfig()
if cfg.TDengineEnabled {
t.Fatal("TDengineEnabled = true, want production default false")
}
}
func TestLoadConfigCanEnableTDengineStageExplicitly(t *testing.T) {
for _, value := range []string{"true", "1", "yes", "on"} {
t.Run(value, func(t *testing.T) {
t.Setenv("FAST_WRITER_TDENGINE_ENABLED", value)
cfg := loadConfig()
if !cfg.TDengineEnabled {
t.Fatalf("TDengineEnabled = false for %q, want explicit true", value)
}
})
}
}
func TestLoadConfigDefaultsStreamMaxBytes(t *testing.T) {
cfg := loadConfig()
@@ -256,12 +976,16 @@ func TestLoadConfigDefaultsStreamMaxBytes(t *testing.T) {
if got, want := cfg.OperationWait, time.Second; got != want {
t.Fatalf("OperationWait = %v, want %v", got, want)
}
if got, want := cfg.FetchWait, 20*time.Millisecond; got != want {
t.Fatalf("FetchWait = %v, want %v", got, want)
}
}
func TestLoadConfigReadsStreamMaxBytesOverride(t *testing.T) {
t.Setenv("NATS_STREAM_MAX_BYTES", "1073741824")
t.Setenv("NATS_STREAM_ENSURE_TIMEOUT_SECONDS", "90")
t.Setenv("FAST_WRITER_OPERATION_TIMEOUT_MS", "1500")
t.Setenv("FAST_WRITER_FETCH_WAIT_MS", "45")
cfg := loadConfig()
@@ -274,6 +998,32 @@ func TestLoadConfigReadsStreamMaxBytesOverride(t *testing.T) {
if got, want := cfg.OperationWait, 1500*time.Millisecond; got != want {
t.Fatalf("OperationWait = %v, want %v", got, want)
}
if got, want := cfg.FetchWait, 45*time.Millisecond; got != want {
t.Fatalf("FetchWait = %v, want %v", got, want)
}
}
func TestRecordFastWriterConfigMetrics(t *testing.T) {
registry := metrics.NewRegistry()
recordFastWriterConfigMetrics(registry, config{
BatchSize: 120,
FetchWait: 20 * time.Millisecond,
OperationWait: 1500 * time.Millisecond,
Workers: 8,
})
text := registry.Render()
for _, want := range []string{
`vehicle_fast_writer_config{setting="batch_size"} 120`,
`vehicle_fast_writer_config{setting="fetch_wait_ms"} 20`,
`vehicle_fast_writer_config{setting="operation_timeout_ms"} 1500`,
`vehicle_fast_writer_config{setting="workers"} 8`,
} {
if !strings.Contains(text, want) {
t.Fatalf("fast writer config metric missing %s:\n%s", want, text)
}
}
}
func TestLoadConfigReadsTDenginePoolOverride(t *testing.T) {
@@ -292,15 +1042,30 @@ func TestLoadConfigReadsTDenginePoolOverride(t *testing.T) {
}
type recordingFastAppender struct {
count int
batchCount int
batchRows int
err error
onAppendBatch func()
count int
batchCount int
batchRows int
err error
batchErr error
singleErr error
failSingleOnCount int
onAppendBatch func()
}
func (a *recordingFastAppender) AppendAll(context.Context, envelope.FrameEnvelope) error {
a.count++
if a.failSingleOnCount > 0 && a.count == a.failSingleOnCount {
if a.singleErr != nil {
return a.singleErr
}
return a.err
}
if a.failSingleOnCount > 0 {
return nil
}
if a.singleErr != nil {
return a.singleErr
}
return a.err
}
@@ -310,24 +1075,45 @@ func (a *recordingFastAppender) AppendAllBatch(_ context.Context, envs []envelop
if a.onAppendBatch != nil {
a.onAppendBatch()
}
if a.batchErr != nil {
return a.batchErr
}
return a.err
}
type recordingFastUpdater struct {
count int
batchCount int
batchRows int
err error
count int
batchCount int
batchRows int
err error
batchErr error
singleErr error
failSingleOnCount int
}
func (u *recordingFastUpdater) FastUpdate(context.Context, envelope.FrameEnvelope) error {
u.count++
if u.failSingleOnCount > 0 && u.count == u.failSingleOnCount {
if u.singleErr != nil {
return u.singleErr
}
return u.err
}
if u.failSingleOnCount > 0 {
return nil
}
if u.singleErr != nil {
return u.singleErr
}
return u.err
}
func (u *recordingFastUpdater) FastUpdateBatch(_ context.Context, envs []envelope.FrameEnvelope) error {
u.batchCount++
u.batchRows += len(envs)
if u.batchErr != nil {
return u.batchErr
}
return u.err
}
@@ -340,3 +1126,73 @@ func (u *recordingFastSingleUpdater) FastUpdate(context.Context, envelope.FrameE
u.count++
return u.err
}
type recordingFastResultUpdater struct {
count int
resultCount int
batchCount int
batchResultCount int
batchRows int
result realtime.FastUpdateResult
err error
batchErr error
singleErr error
}
func (u *recordingFastResultUpdater) FastUpdate(context.Context, envelope.FrameEnvelope) error {
u.count++
if u.singleErr != nil {
return u.singleErr
}
return u.err
}
func (u *recordingFastResultUpdater) FastUpdateWithResult(context.Context, envelope.FrameEnvelope) (realtime.FastUpdateResult, error) {
u.resultCount++
if u.singleErr != nil {
return u.result, u.singleErr
}
return u.result, u.err
}
type recordingFastDynamicResultUpdater struct {
batchResultCount int
batchRows int
err error
}
func (u *recordingFastDynamicResultUpdater) FastUpdate(context.Context, envelope.FrameEnvelope) error {
return u.err
}
func (u *recordingFastDynamicResultUpdater) FastUpdateBatchWithResult(_ context.Context, envs []envelope.FrameEnvelope) (realtime.FastUpdateResult, error) {
u.batchResultCount++
u.batchRows += len(envs)
if u.err != nil {
return realtime.FastUpdateResult{}, u.err
}
return realtime.FastUpdateResult{
EnvelopesSeen: len(envs),
EnvelopesUpdated: len(envs),
FieldsSeen: len(envs),
FieldsWritten: len(envs),
}, nil
}
func (u *recordingFastResultUpdater) FastUpdateBatch(_ context.Context, envs []envelope.FrameEnvelope) error {
u.batchCount++
u.batchRows += len(envs)
if u.batchErr != nil {
return u.batchErr
}
return u.err
}
func (u *recordingFastResultUpdater) FastUpdateBatchWithResult(_ context.Context, envs []envelope.FrameEnvelope) (realtime.FastUpdateResult, error) {
u.batchResultCount++
u.batchRows += len(envs)
if u.batchErr != nil {
return u.result, u.batchErr
}
return u.result, u.err
}

615
go/vehicle-gateway/cmd/nats-kafka-bridge/main.go
View File

@@ -8,6 +8,7 @@ import (
"log/slog"
"os"
"os/signal"
"sort"
"strconv"
"strings"
"syscall"
@@ -20,6 +21,7 @@ import (
"lingniu-vehicle-ingest/go/vehicle-gateway/internal/health"
"lingniu-vehicle-ingest/go/vehicle-gateway/internal/metrics"
"lingniu-vehicle-ingest/go/vehicle-gateway/internal/observability"
"lingniu-vehicle-ingest/go/vehicle-gateway/internal/realtime"
"lingniu-vehicle-ingest/go/vehicle-gateway/internal/topics"
)
@@ -29,6 +31,10 @@ func main() {
defer stop()
cfg := loadConfig()
if err := cfg.Validate(); err != nil {
logger.Error("invalid bridge config", "error", err)
os.Exit(1)
}
conn, err := nats.Connect(cfg.NATSURL, nats.Name(cfg.NATSClientName), nats.Timeout(5*time.Second))
if err != nil {
logger.Error("nats connect failed", "error", err)
@@ -36,6 +42,7 @@ func main() {
}
defer conn.Close()
registry := metrics.NewRegistry()
recordBridgeConfigMetrics(registry, cfg)
health.Start(ctx, logger, health.NewServer(env("HEALTH_ADDR", ""), "nats-kafka-bridge", []health.Check{
{Name: "nats", Check: func(context.Context) error {
if conn.Status() != nats.CONNECTED {
@@ -66,71 +73,205 @@ func main() {
os.Exit(1)
}
writer := &kafka.Writer{
Addr: kafka.TCP(cfg.KafkaBrokers...),
Balancer: &kafka.Hash{},
AllowAutoTopicCreation: false,
RequiredAcks: kafka.RequireAll,
Async: false,
}
defer writer.Close()
logger.Info("nats kafka bridge started",
"nats_url", cfg.NATSURL,
"stream", cfg.NATSStream,
"durable", cfg.NATSDurable,
"filter", cfg.NATSFilter,
"kafka_brokers", strings.Join(cfg.KafkaBrokers, ","))
runBridge(ctx, logger, registry, js, sub, writer, cfg)
"kafka_brokers", strings.Join(cfg.KafkaBrokers, ","),
"kafka_batch_timeout_ms", cfg.KafkaBatchTimeout.Milliseconds(),
"kafka_write_concurrency", cfg.KafkaWriteConcurrency,
"fetch_wait_ms", cfg.FetchWait.Milliseconds(),
"batch_size", cfg.BatchSize,
"workers", cfg.Workers,
"derive_fields_from_raw_enabled", cfg.DeriveFieldsFromRaw)
for i := 0; i < cfg.Workers; i++ {
writer := newKafkaWriter(cfg)
defer writer.Close()
go runBridge(ctx, logger.With("worker", i), registry, js, sub, writer, cfg)
}
<-ctx.Done()
}
type config struct {
NATSURL string
NATSClientName string
NATSStream string
NATSDurable string
NATSFilter string
NATSSubjects []string
KafkaBrokers []string
Route map[string]string
BatchSize int
FetchWait time.Duration
OperationWait time.Duration
AckWait time.Duration
StreamMaxAge time.Duration
StreamMaxBytes int64
StreamEnsureWait time.Duration
NATSURL string
NATSClientName string
NATSStream string
NATSDurable string
NATSFilter string
NATSSubjects []string
KafkaBrokers []string
RawRoutes []subjectRoute
FieldsRoutes []subjectRoute
Route map[string]string
RawFieldRoutes map[string]fieldsProjectionRoute
DeriveFieldsFromRaw bool
KafkaBatchTimeout time.Duration
KafkaWriteConcurrency int
BatchSize int
FetchWait time.Duration
OperationWait time.Duration
AckWait time.Duration
StreamMaxAge time.Duration
StreamMaxBytes int64
StreamEnsureWait time.Duration
Workers int
}
type subjectRoute struct {
Protocol envelope.Protocol
Subject string
Topic string
}
type fieldsProjectionRoute struct {
Protocol envelope.Protocol
Topic string
}
func loadConfig() config {
route := map[string]string{
env("NATS_SUBJECT_GB32960_RAW", env("KAFKA_TOPIC_GB32960_RAW", topics.RawGB32960)): env("KAFKA_TOPIC_GB32960_RAW", topics.RawGB32960),
env("NATS_SUBJECT_JT808_RAW", env("KAFKA_TOPIC_JT808_RAW", topics.RawJT808)): env("KAFKA_TOPIC_JT808_RAW", topics.RawJT808),
env("NATS_SUBJECT_YUTONG_MQTT_RAW", env("KAFKA_TOPIC_YUTONG_MQTT_RAW", topics.RawYutongMQTT)): env("KAFKA_TOPIC_YUTONG_MQTT_RAW", topics.RawYutongMQTT),
env("NATS_SUBJECT_GB32960_FIELDS", env("KAFKA_TOPIC_GB32960_FIELDS", topics.FieldsGB32960)): env("KAFKA_TOPIC_GB32960_FIELDS", topics.FieldsGB32960),
env("NATS_SUBJECT_JT808_FIELDS", env("KAFKA_TOPIC_JT808_FIELDS", topics.FieldsJT808)): env("KAFKA_TOPIC_JT808_FIELDS", topics.FieldsJT808),
env("NATS_SUBJECT_YUTONG_MQTT_FIELDS", env("KAFKA_TOPIC_YUTONG_MQTT_FIELDS", topics.FieldsYutongMQTT)): env("KAFKA_TOPIC_YUTONG_MQTT_FIELDS", topics.FieldsYutongMQTT),
rawRoutes := []subjectRoute{
{Protocol: envelope.ProtocolGB32960, Subject: env("NATS_SUBJECT_GB32960_RAW", env("KAFKA_TOPIC_GB32960_RAW", topics.RawGB32960)), Topic: env("KAFKA_TOPIC_GB32960_RAW", topics.RawGB32960)},
{Protocol: envelope.ProtocolJT808, Subject: env("NATS_SUBJECT_JT808_RAW", env("KAFKA_TOPIC_JT808_RAW", topics.RawJT808)), Topic: env("KAFKA_TOPIC_JT808_RAW", topics.RawJT808)},
{Protocol: envelope.ProtocolYutongMQTT, Subject: env("NATS_SUBJECT_YUTONG_MQTT_RAW", env("KAFKA_TOPIC_YUTONG_MQTT_RAW", topics.RawYutongMQTT)), Topic: env("KAFKA_TOPIC_YUTONG_MQTT_RAW", topics.RawYutongMQTT)},
}
fieldsRoutes := []subjectRoute{
{Protocol: envelope.ProtocolGB32960, Subject: env("NATS_SUBJECT_GB32960_FIELDS", env("KAFKA_TOPIC_GB32960_FIELDS", topics.FieldsGB32960)), Topic: env("KAFKA_TOPIC_GB32960_FIELDS", topics.FieldsGB32960)},
{Protocol: envelope.ProtocolJT808, Subject: env("NATS_SUBJECT_JT808_FIELDS", env("KAFKA_TOPIC_JT808_FIELDS", topics.FieldsJT808)), Topic: env("KAFKA_TOPIC_JT808_FIELDS", topics.FieldsJT808)},
{Protocol: envelope.ProtocolYutongMQTT, Subject: env("NATS_SUBJECT_YUTONG_MQTT_FIELDS", env("KAFKA_TOPIC_YUTONG_MQTT_FIELDS", topics.FieldsYutongMQTT)), Topic: env("KAFKA_TOPIC_YUTONG_MQTT_FIELDS", topics.FieldsYutongMQTT)},
}
route := routeMap(rawRoutes, fieldsRoutes)
if unifiedSubject, unifiedTopic, ok := unifiedRouteFromEnv(); ok {
route[unifiedSubject] = unifiedTopic
}
return config{
NATSURL: env("NATS_URL", "nats://127.0.0.1:4222"),
NATSClientName: env("NATS_CLIENT_NAME", "lingniu-nats-kafka-bridge"),
NATSStream: env("NATS_STREAM", "VEHICLE_INGEST"),
NATSDurable: env("NATS_DURABLE", "vehicle-kafka-bridge"),
NATSFilter: env("NATS_FILTER", "vehicle.>"),
NATSSubjects: splitCSV(env("NATS_STREAM_SUBJECTS", strings.Join(mapKeys(route), ","))),
KafkaBrokers: splitCSV(env("KAFKA_BROKERS", "127.0.0.1:9092")),
Route: route,
BatchSize: envInt("BRIDGE_BATCH_SIZE", 500),
FetchWait: time.Duration(envInt("BRIDGE_FETCH_WAIT_MS", 1000)) * time.Millisecond,
OperationWait: time.Duration(envInt("BRIDGE_OPERATION_TIMEOUT_MS", 30000)) * time.Millisecond,
AckWait: time.Duration(envInt("NATS_ACK_WAIT_SECONDS", 60)) * time.Second,
StreamMaxAge: time.Duration(envInt("NATS_STREAM_MAX_AGE_HOURS", 24)) * time.Hour,
StreamMaxBytes: envInt64("NATS_STREAM_MAX_BYTES", 20*1024*1024*1024),
StreamEnsureWait: time.Duration(envInt("NATS_STREAM_ENSURE_TIMEOUT_SECONDS", 60)) * time.Second,
workers := envInt("BRIDGE_WORKERS", 4)
if workers < 1 {
workers = 1
}
return config{
NATSURL: env("NATS_URL", "nats://127.0.0.1:4222"),
NATSClientName: env("NATS_CLIENT_NAME", "lingniu-nats-kafka-bridge"),
NATSStream: env("NATS_STREAM", "VEHICLE_INGEST"),
NATSDurable: env("NATS_DURABLE", "vehicle-kafka-bridge"),
NATSFilter: env("NATS_FILTER", "vehicle.>"),
NATSSubjects: splitCSV(env("NATS_STREAM_SUBJECTS", strings.Join(mapKeys(route), ","))),
KafkaBrokers: splitCSV(env("KAFKA_BROKERS", "127.0.0.1:9092")),
RawRoutes: rawRoutes,
FieldsRoutes: fieldsRoutes,
Route: route,
RawFieldRoutes: rawFieldProjectionRoutes(rawRoutes, fieldsRoutes),
DeriveFieldsFromRaw: envBool("BRIDGE_DERIVE_FIELDS_FROM_RAW_ENABLED", true),
KafkaBatchTimeout: time.Duration(envInt("BRIDGE_KAFKA_BATCH_TIMEOUT_MS", 20)) * time.Millisecond,
KafkaWriteConcurrency: envInt("BRIDGE_KAFKA_WRITE_CONCURRENCY", 6),
BatchSize: envInt("BRIDGE_BATCH_SIZE", 500),
FetchWait: time.Duration(envInt("BRIDGE_FETCH_WAIT_MS", 20)) * time.Millisecond,
OperationWait: time.Duration(envInt("BRIDGE_OPERATION_TIMEOUT_MS", 30000)) * time.Millisecond,
AckWait: time.Duration(envInt("NATS_ACK_WAIT_SECONDS", 60)) * time.Second,
StreamMaxAge: time.Duration(envInt("NATS_STREAM_MAX_AGE_HOURS", 24)) * time.Hour,
StreamMaxBytes: envInt64("NATS_STREAM_MAX_BYTES", 20*1024*1024*1024),
StreamEnsureWait: time.Duration(envInt("NATS_STREAM_ENSURE_TIMEOUT_SECONDS", 60)) * time.Second,
Workers: workers,
}
}
func recordBridgeConfigMetrics(registry *metrics.Registry, cfg config) {
if registry == nil {
return
}
registry.SetGauge("vehicle_bridge_config", metrics.Labels{"setting": "workers"}, float64(cfg.Workers))
registry.SetGauge("vehicle_bridge_config", metrics.Labels{"setting": "batch_size"}, float64(cfg.BatchSize))
registry.SetGauge("vehicle_bridge_config", metrics.Labels{"setting": "fetch_wait_ms"}, float64(cfg.FetchWait.Milliseconds()))
registry.SetGauge("vehicle_bridge_config", metrics.Labels{"setting": "kafka_batch_timeout_ms"}, float64(cfg.KafkaBatchTimeout.Milliseconds()))
registry.SetGauge("vehicle_bridge_config", metrics.Labels{"setting": "kafka_write_concurrency"}, float64(cfg.KafkaWriteConcurrency))
registry.SetGauge("vehicle_bridge_config", metrics.Labels{"setting": "derive_fields_from_raw_enabled"}, boolMetric(cfg.DeriveFieldsFromRaw))
}
func newKafkaWriter(cfg config) *kafka.Writer {
return &kafka.Writer{
Addr: kafka.TCP(cfg.KafkaBrokers...),
Balancer: &kafka.Hash{},
AllowAutoTopicCreation: false,
RequiredAcks: kafka.RequireAll,
BatchTimeout: cfg.KafkaBatchTimeout,
Async: false,
}
}
func routeMap(rawRoutes []subjectRoute, fieldsRoutes []subjectRoute) map[string]string {
route := make(map[string]string, len(rawRoutes)+len(fieldsRoutes))
for _, item := range rawRoutes {
if item.Subject != "" {
route[item.Subject] = item.Topic
}
}
for _, item := range fieldsRoutes {
if item.Subject != "" {
route[item.Subject] = item.Topic
}
}
return route
}
func rawFieldProjectionRoutes(rawRoutes []subjectRoute, fieldsRoutes []subjectRoute) map[string]fieldsProjectionRoute {
fieldsByProtocol := make(map[envelope.Protocol]string, len(fieldsRoutes))
for _, route := range fieldsRoutes {
if route.Protocol != "" && strings.TrimSpace(route.Topic) != "" {
fieldsByProtocol[route.Protocol] = strings.TrimSpace(route.Topic)
}
}
out := make(map[string]fieldsProjectionRoute, len(rawRoutes))
for _, route := range rawRoutes {
subject := strings.TrimSpace(route.Subject)
topic := fieldsByProtocol[route.Protocol]
if subject == "" || topic == "" {
continue
}
out[subject] = fieldsProjectionRoute{Protocol: route.Protocol, Topic: topic}
}
return out
}
func (c config) Validate() error {
rawSubjects, rawTopics := routeSubjectsAndTopics(c.RawRoutes)
fieldsSubjects, fieldsTopics := routeSubjectsAndTopics(c.FieldsRoutes)
if err := topics.ValidateKafkaRawFields(rawTopics, fieldsTopics); err != nil {
return err
}
return topics.ValidateRawFieldsDisjoint(rawSubjects, fieldsSubjects, "nats subject")
}
func routeSubjectsAndTopics(routes []subjectRoute) (map[string]string, map[string]string) {
subjects := make(map[string]string, len(routes))
kafkaTopics := make(map[string]string, len(routes))
for _, route := range routes {
name := strings.TrimSpace(route.Subject)
if name == "" {
name = strings.TrimSpace(route.Topic)
}
if name == "" {
name = "unknown"
}
subjects[name] = strings.TrimSpace(route.Subject)
topicName := routeProtocolName(route)
if topicName == "" {
topicName = name
}
kafkaTopics[topicName] = strings.TrimSpace(route.Topic)
}
return subjects, kafkaTopics
}
func routeProtocolName(route subjectRoute) string {
if protocol := strings.TrimSpace(string(route.Protocol)); protocol != "" {
return protocol
}
if protocol, ok := topics.ProtocolForKnownRawTopic(route.Topic); ok {
return protocol
}
if protocol, ok := topics.ProtocolForKnownFieldsTopic(route.Topic); ok {
return protocol
}
return ""
}
func unifiedRouteFromEnv() (string, string, bool) {
@@ -169,6 +310,18 @@ type bridgeMessage struct {
ack func() error
}
type routedBridgeMessage struct {
sourceIndex int
kafka kafka.Message
receivedAtMS int64
}
type bridgeSourceState struct {
source bridgeMessage
routed bool
failed bool
}
func runBridge(ctx context.Context, logger *slog.Logger, registry *metrics.Registry, infoReader natsConsumerInfoReader, sub natsPullSubscription, writer kafkaBatchWriter, cfg config) {
var lastConsumerInfoAt time.Time
for {
@@ -188,9 +341,17 @@ func runBridge(ctx context.Context, logger *slog.Logger, registry *metrics.Regis
}
msgs, err := sub.Fetch(cfg.BatchSize, nats.MaxWait(cfg.FetchWait))
if err != nil {
if isBridgeShutdownFetchError(ctx, err) {
return
}
if errors.Is(err, nats.ErrTimeout) {
continue
}
if isTransientBridgeFetchError(err) {
logger.Warn("nats fetch interrupted", "error", err)
time.Sleep(time.Second)
continue
}
logger.Error("nats fetch failed", "error", err)
time.Sleep(time.Second)
continue
@@ -207,7 +368,7 @@ func runBridge(ctx context.Context, logger *slog.Logger, registry *metrics.Regis
})
}
operationCtx, cancel := context.WithTimeout(context.WithoutCancel(ctx), cfg.OperationWait)
err = bridgeBatch(operationCtx, registry, writer, bridgeMessages, cfg.Route)
err = bridgeBatchWithProjectionConcurrency(operationCtx, registry, writer, bridgeMessages, cfg.Route, cfg.RawFieldRoutes, cfg.DeriveFieldsFromRaw, cfg.KafkaWriteConcurrency)
cancel()
if err != nil {
logger.Error("bridge batch failed", "count", len(bridgeMessages), "error", err)
@@ -216,6 +377,30 @@ func runBridge(ctx context.Context, logger *slog.Logger, registry *metrics.Regis
}
}
func isBridgeShutdownFetchError(ctx context.Context, err error) bool {
if err == nil {
return false
}
if ctx.Err() != nil {
return true
}
return errors.Is(err, nats.ErrConnectionClosed)
}
func isTransientBridgeFetchError(err error) bool {
if err == nil {
return false
}
text := strings.ToLower(strings.TrimSpace(err.Error()))
return strings.Contains(text, "disconnected during fetch") ||
strings.Contains(text, "connection closed") ||
strings.Contains(text, "connection reset") ||
strings.Contains(text, "broken pipe") ||
strings.Contains(text, "temporary") ||
strings.Contains(text, "temporarily") ||
strings.Contains(text, "timeout")
}
func ensureStream(js nats.JetStreamContext, cfg config) error {
ctx, cancel := context.WithTimeout(context.Background(), cfg.StreamEnsureWait)
defer cancel()
@@ -245,6 +430,31 @@ func ensureStream(js nats.JetStreamContext, cfg config) error {
}
func bridgeBatch(ctx context.Context, registry *metrics.Registry, writer kafkaBatchWriter, messages []bridgeMessage, route map[string]string) error {
return bridgeBatchWithProjectionConcurrency(ctx, registry, writer, messages, route, nil, false, 6)
}
func bridgeBatchWithProjection(
ctx context.Context,
registry *metrics.Registry,
writer kafkaBatchWriter,
messages []bridgeMessage,
route map[string]string,
rawFieldRoutes map[string]fieldsProjectionRoute,
deriveFieldsFromRaw bool,
) error {
return bridgeBatchWithProjectionConcurrency(ctx, registry, writer, messages, route, rawFieldRoutes, deriveFieldsFromRaw, 6)
}
func bridgeBatchWithProjectionConcurrency(
ctx context.Context,
registry *metrics.Registry,
writer kafkaBatchWriter,
messages []bridgeMessage,
route map[string]string,
rawFieldRoutes map[string]fieldsProjectionRoute,
deriveFieldsFromRaw bool,
kafkaWriteConcurrency int,
) error {
if len(messages) == 0 {
return nil
}
@@ -252,66 +462,240 @@ func bridgeBatch(ctx context.Context, registry *metrics.Registry, writer kafkaBa
status := "ok"
defer func() {
recordBridgeBatchDuration(registry, status, time.Since(started))
recordBridgeBatchPending(registry, 0, 0)
}()
kafkaMessages := make([]kafka.Message, 0, len(messages))
for _, message := range messages {
routed := make([]routedBridgeMessage, 0, len(messages)*2)
sources := make([]bridgeSourceState, len(messages))
for sourceIndex, message := range messages {
sources[sourceIndex].source = message
addBridgeSubjectMetric(registry, "vehicle_bridge_messages_total", message.subject, "received")
topic, ok := route[message.subject]
if !ok || topic == "" {
addBridgeSubjectMetric(registry, "vehicle_bridge_messages_total", message.subject, "route_error")
status = "error"
return fmt.Errorf("kafka topic not configured for nats subject %q", message.subject)
}
kafkaMessages = append(kafkaMessages, kafkaMessage(topic, message.data))
}
recordBridgeBatchPending(registry, len(messages), len(kafkaMessages))
if err := writer.WriteMessages(ctx, kafkaMessages...); err != nil {
for _, message := range kafkaMessages {
addBridgeTopicMetric(registry, "vehicle_bridge_kafka_writes_total", message.Topic, "error")
}
status = "error"
return err
}
for _, message := range kafkaMessages {
addBridgeTopicMetric(registry, "vehicle_bridge_kafka_writes_total", message.Topic, "ok")
}
for _, message := range messages {
if message.ack == nil {
if message.ack != nil {
if err := message.ack(); err != nil {
addBridgeSubjectMetric(registry, "vehicle_bridge_nats_acks_total", message.subject, "error")
status = "error"
return fmt.Errorf("ack unrouted nats subject %q: %w", message.subject, err)
}
addBridgeSubjectMetric(registry, "vehicle_bridge_nats_acks_total", message.subject, "dropped_route_error")
}
continue
}
if err := message.ack(); err != nil {
addBridgeSubjectMetric(registry, "vehicle_bridge_nats_acks_total", message.subject, "error")
kafkaMessage, receivedAtMS, decoded, decodeErr := kafkaMessageWithEnvelope(topic, message.data)
sources[sourceIndex].routed = true
routed = append(routed, routedBridgeMessage{
sourceIndex: sourceIndex,
kafka: kafkaMessage,
receivedAtMS: receivedAtMS,
})
if !deriveFieldsFromRaw {
continue
}
projection, ok := rawFieldRoutes[message.subject]
if !ok {
continue
}
fieldsMessage, fieldsReceivedAtMS, fieldCount, projectionStatus, projected := projectRawFields(message.subject, decoded, decodeErr, projection)
recordBridgeFieldsProjection(registry, projection.Protocol, projectionStatus, fieldCount)
if !projected {
continue
}
routed = append(routed, routedBridgeMessage{
sourceIndex: sourceIndex,
kafka: fieldsMessage,
receivedAtMS: fieldsReceivedAtMS,
})
}
if len(routed) == 0 {
addBridgeBatchPending(registry, 0, 0)
return nil
}
addBridgeBatchPending(registry, len(messages), len(routed))
defer addBridgeBatchPending(registry, -len(messages), -len(routed))
groups := groupRoutedBridgeMessagesByTopic(routed)
results := writeBridgeKafkaGroups(ctx, writer, groups, kafkaWriteConcurrency)
var firstErr error
for _, result := range results {
group := result.group
err := result.err
recordBridgeKafkaWriteDuration(registry, group[0].kafka.Topic, statusFromError(err), result.elapsed)
if err != nil {
for _, item := range group {
addBridgeTopicMetric(registry, "vehicle_bridge_kafka_writes_total", item.kafka.Topic, "error")
sources[item.sourceIndex].failed = true
}
status = "error"
if firstErr == nil {
firstErr = fmt.Errorf("kafka write topic %s: %w", group[0].kafka.Topic, err)
}
continue
}
for _, item := range group {
addBridgeTopicMetric(registry, "vehicle_bridge_kafka_writes_total", item.kafka.Topic, "ok")
recordBridgeKafkaE2EDuration(registry, item.kafka.Topic, item.receivedAtMS)
}
}
for i := range sources {
source := &sources[i]
if !source.routed || source.failed || source.source.ack == nil {
continue
}
if err := source.source.ack(); err != nil {
addBridgeSubjectMetric(registry, "vehicle_bridge_nats_acks_total", source.source.subject, "error")
status = "error"
return err
}
addBridgeSubjectMetric(registry, "vehicle_bridge_nats_acks_total", message.subject, "ok")
addBridgeSubjectMetric(registry, "vehicle_bridge_nats_acks_total", source.source.subject, "ok")
}
return nil
return firstErr
}
type bridgeKafkaWriteResult struct {
group []routedBridgeMessage
elapsed time.Duration
err error
}
func writeBridgeKafkaGroups(ctx context.Context, writer kafkaBatchWriter, groups [][]routedBridgeMessage, concurrency int) []bridgeKafkaWriteResult {
if len(groups) == 0 {
return nil
}
if concurrency < 1 {
concurrency = 1
}
if concurrency > len(groups) {
concurrency = len(groups)
}
semaphore := make(chan struct{}, concurrency)
results := make(chan bridgeKafkaWriteResult, len(groups))
for _, group := range groups {
group := group
semaphore <- struct{}{}
go func() {
defer func() { <-semaphore }()
messages := make([]kafka.Message, 0, len(group))
for _, item := range group {
messages = append(messages, item.kafka)
}
started := time.Now()
err := writer.WriteMessages(ctx, messages...)
results <- bridgeKafkaWriteResult{group: group, elapsed: time.Since(started), err: err}
}()
}
out := make([]bridgeKafkaWriteResult, 0, len(groups))
for range groups {
out = append(out, <-results)
}
return out
}
func projectRawFields(subject string, raw envelope.FrameEnvelope, decodeErr error, projection fieldsProjectionRoute) (kafka.Message, int64, int, string, bool) {
if decodeErr != nil {
return kafka.Message{}, 0, 0, "invalid_json", false
}
if _, err := topics.ValidateRawEnvelope(subject, raw); err != nil {
return kafka.Message{}, 0, 0, "invalid_envelope", false
}
if raw.Protocol != projection.Protocol {
return kafka.Message{}, 0, 0, "protocol_mismatch", false
}
if !envelope.IsRealtimeTelemetryFrame(raw) {
return kafka.Message{}, 0, 0, "skipped_non_realtime", false
}
if len(raw.ParsedFields) == 0 {
return kafka.Message{}, 0, 0, "skipped_missing_fields", false
}
fields, ok := realtime.BuildFieldsEnvelope(raw)
if !ok || len(fields.Fields) == 0 {
return kafka.Message{}, 0, 0, "skipped_missing_fields", false
}
if _, err := topics.ValidateFieldsEnvelope(projection.Topic, fields); err != nil {
return kafka.Message{}, 0, 0, "invalid_fields_envelope", false
}
payload, err := fields.MarshalJSONBytes()
if err != nil {
return kafka.Message{}, 0, 0, "marshal_error", false
}
result := kafka.Message{Topic: projection.Topic, Key: fields.KafkaKey(), Value: payload}
return result, fields.ReceivedAtMS, len(fields.Fields), "published", true
}
func groupRoutedBridgeMessagesByTopic(messages []routedBridgeMessage) [][]routedBridgeMessage {
if len(messages) == 0 {
return nil
}
groupsByTopic := make(map[string][]routedBridgeMessage)
for _, message := range messages {
groupsByTopic[message.kafka.Topic] = append(groupsByTopic[message.kafka.Topic], message)
}
topics := make([]string, 0, len(groupsByTopic))
for topic := range groupsByTopic {
topics = append(topics, topic)
}
sort.Strings(topics)
groups := make([][]routedBridgeMessage, 0, len(topics))
for _, topic := range topics {
groups = append(groups, groupsByTopic[topic])
}
return groups
}
var bridgeBatchDurationBucketsMS = []float64{1, 5, 10, 25, 50, 100, 250, 500, 1000, 5000}
var bridgeKafkaWriteDurationBucketsMS = []float64{1, 5, 10, 25, 50, 100, 250, 500, 1000, 5000}
var bridgeKafkaE2EDurationBucketsMS = []float64{10, 25, 50, 100, 250, 500, 1000, 2500, 5000, 10000}
var bridgeKafkaE2ERecent = metrics.NewRecentLatencyByKey(512)
var bridgeBatchPending = metrics.PendingPairGauge{}
func addBridgeSubjectMetric(registry *metrics.Registry, name string, subject string, status string) {
if registry == nil {
return
}
registry.IncCounter(name, metrics.Labels{"subject": subject, "status": status})
labels := metrics.Labels{"subject": subject, "status": status}
registry.IncCounter(name, labels)
if name == "vehicle_bridge_messages_total" {
metrics.RecordLastActivity(registry, "vehicle_bridge_last_message_unix_seconds", labels)
return
}
if name == "vehicle_bridge_nats_acks_total" {
metrics.RecordLastActivity(registry, "vehicle_bridge_last_ack_unix_seconds", labels)
}
}
func addBridgeTopicMetric(registry *metrics.Registry, name string, topic string, status string) {
if registry == nil {
return
}
registry.IncCounter(name, metrics.Labels{"topic": topic, "status": status})
labels := metrics.Labels{"topic": topic, "status": status}
registry.IncCounter(name, labels)
if name == "vehicle_bridge_kafka_writes_total" {
metrics.RecordLastActivity(registry, "vehicle_bridge_last_kafka_write_unix_seconds", labels)
}
}
func recordBridgeBatchPending(registry *metrics.Registry, messages int, kafkaMessages int) {
func recordBridgeFieldsProjection(registry *metrics.Registry, protocol envelope.Protocol, status string, fieldCount int) {
if registry == nil {
return
}
registry.SetGauge("vehicle_bridge_batch_pending_messages", nil, float64(messages))
registry.SetGauge("vehicle_bridge_batch_pending_kafka_messages", nil, float64(kafkaMessages))
protocolLabel := strings.TrimSpace(string(protocol))
if protocolLabel == "" {
protocolLabel = "unknown"
}
if strings.TrimSpace(status) == "" {
status = "unknown"
}
labels := metrics.Labels{"protocol": protocolLabel, "status": status}
registry.IncCounter("vehicle_bridge_fields_projection_total", labels)
metrics.RecordLastActivity(registry, "vehicle_bridge_last_fields_projection_unix_seconds", labels)
if status == "published" {
registry.SetGauge("vehicle_bridge_fields_projection_count", labels, float64(fieldCount))
}
}
func addBridgeBatchPending(registry *metrics.Registry, messages int, kafkaMessages int) {
if registry == nil {
return
}
bridgeBatchPending.Add(registry, "vehicle_bridge_batch_pending_messages", "vehicle_bridge_batch_pending_kafka_messages", messages, kafkaMessages)
}
func recordBridgeBatchDuration(registry *metrics.Registry, status string, elapsed time.Duration) {
@@ -323,6 +707,41 @@ func recordBridgeBatchDuration(registry *metrics.Registry, status string, elapse
}, bridgeBatchDurationBucketsMS, float64(elapsed.Milliseconds()))
}
func recordBridgeKafkaWriteDuration(registry *metrics.Registry, topic string, status string, elapsed time.Duration) {
if registry == nil {
return
}
registry.ObserveHistogram("vehicle_bridge_kafka_write_duration_ms_histogram", metrics.Labels{
"topic": topic,
"status": status,
}, bridgeKafkaWriteDurationBucketsMS, float64(elapsed.Milliseconds()))
}
func recordBridgeKafkaE2EDuration(registry *metrics.Registry, topic string, receivedAtMS int64) {
if registry == nil || receivedAtMS <= 0 {
return
}
elapsedMS := time.Since(time.UnixMilli(receivedAtMS)).Milliseconds()
if elapsedMS < 0 {
elapsedMS = 0
}
labels := metrics.Labels{
"topic": topic,
}
registry.ObserveHistogram("vehicle_bridge_kafka_e2e_duration_ms_histogram", labels, bridgeKafkaE2EDurationBucketsMS, float64(elapsedMS))
p99, samples := bridgeKafkaE2ERecent.Observe(topic, float64(elapsedMS))
registry.SetGauge("vehicle_bridge_kafka_e2e_recent_p99_ms", labels, p99)
registry.SetGauge("vehicle_bridge_kafka_e2e_recent_samples", labels, float64(samples))
metrics.RecordLastActivity(registry, "vehicle_bridge_last_kafka_e2e_unix_seconds", labels)
}
func statusFromError(err error) string {
if err != nil {
return "error"
}
return "ok"
}
func recordNATSConsumerInfoMetrics(registry *metrics.Registry, cfg config, info *nats.ConsumerInfo) {
if registry == nil || info == nil {
return
@@ -334,12 +753,23 @@ func recordNATSConsumerInfoMetrics(registry *metrics.Registry, cfg config, info
}
func kafkaMessage(topic string, data []byte) kafka.Message {
message, _ := kafkaMessageWithReceivedAt(topic, data)
return message
}
func kafkaMessageWithReceivedAt(topic string, data []byte) (kafka.Message, int64) {
message, receivedAtMS, _, _ := kafkaMessageWithEnvelope(topic, data)
return message, receivedAtMS
}
func kafkaMessageWithEnvelope(topic string, data []byte) (kafka.Message, int64, envelope.FrameEnvelope, error) {
var env envelope.FrameEnvelope
message := kafka.Message{Topic: topic, Value: data}
if err := json.Unmarshal(data, &env); err == nil {
message.Key = env.KafkaKey()
if err := json.Unmarshal(data, &env); err != nil {
return message, 0, envelope.FrameEnvelope{}, err
}
return message
message.Key = env.KafkaKey()
return message, env.ReceivedAtMS, env, nil
}
func env(key string, fallback string) string {
@@ -350,6 +780,25 @@ func env(key string, fallback string) string {
return value
}
func envBool(key string, fallback bool) bool {
value := strings.TrimSpace(os.Getenv(key))
if value == "" {
return fallback
}
parsed, err := strconv.ParseBool(value)
if err != nil {
return fallback
}
return parsed
}
func boolMetric(value bool) float64 {
if value {
return 1
}
return 0
}
func envOptional(key string) (string, bool) {
value, ok := os.LookupEnv(key)
if !ok {

620
go/vehicle-gateway/cmd/nats-kafka-bridge/main_test.go
View File

@@ -5,6 +5,8 @@ import (
"encoding/json"
"errors"
"strings"
"sync"
"sync/atomic"
"testing"
"time"
@@ -13,6 +15,7 @@ import (
"lingniu-vehicle-ingest/go/vehicle-gateway/internal/envelope"
"lingniu-vehicle-ingest/go/vehicle-gateway/internal/metrics"
"lingniu-vehicle-ingest/go/vehicle-gateway/internal/topics"
)
func TestBridgeBatchWritesKafkaThenAcks(t *testing.T) {
@@ -65,12 +68,303 @@ func TestBridgeBatchDoesNotAckWhenKafkaFails(t *testing.T) {
}
}
func TestBridgeBatchRecordsMetrics(t *testing.T) {
func TestBridgeBatchProjectsFieldsFromCanonicalRawAndAcksOnce(t *testing.T) {
raw := envelope.FrameEnvelope{
EventID: "raw-event-1",
EventKind: envelope.EventKindRaw,
Protocol: envelope.ProtocolJT808,
MessageID: "0x0200",
VIN: "LTEST000000000001",
EventTimeMS: 1_700_000_000_000,
ReceivedAtMS: 1_700_000_000_100,
ParseStatus: envelope.ParseOK,
ParsedFields: map[string]any{
"jt808.location.speed_kmh": 52.3,
"jt808.location.total_mileage_km": 12345.6,
},
}
payload, err := raw.MarshalJSONBytes()
if err != nil {
t.Fatal(err)
}
acked := 0
writer := &recordingBridgeWriter{}
registry := metrics.NewRegistry()
err = bridgeBatchWithProjection(
context.Background(),
registry,
writer,
[]bridgeMessage{{subject: topics.RawJT808, data: payload, ack: func() error { acked++; return nil }}},
map[string]string{topics.RawJT808: topics.RawJT808},
map[string]fieldsProjectionRoute{topics.RawJT808: {Protocol: envelope.ProtocolJT808, Topic: topics.FieldsJT808}},
true,
)
if err != nil {
t.Fatalf("bridgeBatchWithProjection() error = %v", err)
}
if acked != 1 {
t.Fatalf("acks = %d, want exactly one ack for raw plus derived fields", acked)
}
if len(writer.messages) != 2 {
t.Fatalf("kafka messages = %d, want raw and fields", len(writer.messages))
}
byTopic := map[string]kafka.Message{}
for _, message := range writer.messages {
byTopic[message.Topic] = message
}
if len(byTopic[topics.RawJT808].Value) == 0 || len(byTopic[topics.FieldsJT808].Value) == 0 {
t.Fatalf("projected topics = %#v", byTopic)
}
var fields envelope.FrameEnvelope
if err := json.Unmarshal(byTopic[topics.FieldsJT808].Value, &fields); err != nil {
t.Fatal(err)
}
if fields.EventKind != envelope.EventKindFields || fields.SourceEventID != raw.EventID {
t.Fatalf("fields envelope = %#v", fields)
}
if got := fields.Fields["jt808.location.total_mileage_km"]; got != 12345.6 {
t.Fatalf("projected mileage = %#v", got)
}
metricText := registry.Render()
for _, want := range []string{
`vehicle_bridge_fields_projection_total{protocol="JT808",status="published"} 1`,
`vehicle_bridge_fields_projection_count{protocol="JT808",status="published"} 2`,
`vehicle_bridge_nats_acks_total{status="ok",subject="vehicle.raw.go.jt808.v1"} 1`,
} {
if !strings.Contains(metricText, want) {
t.Fatalf("projection metric missing %s:\n%s", want, metricText)
}
}
}
func TestBridgeBatchDoesNotAckRawWhenDerivedFieldsWriteFails(t *testing.T) {
raw := envelope.FrameEnvelope{
EventKind: envelope.EventKindRaw,
Protocol: envelope.ProtocolJT808,
MessageID: "0x0200",
VIN: "LTEST000000000001",
EventTimeMS: 1_700_000_000_000,
ReceivedAtMS: 1_700_000_000_100,
ParseStatus: envelope.ParseOK,
ParsedFields: map[string]any{"jt808.location.total_mileage_km": 12345.6},
}
payload, err := raw.MarshalJSONBytes()
if err != nil {
t.Fatal(err)
}
acked := 0
writer := &recordingBridgeWriter{topicErr: map[string]error{topics.FieldsJT808: errors.New("fields unavailable")}}
err = bridgeBatchWithProjection(
context.Background(),
nil,
writer,
[]bridgeMessage{{subject: topics.RawJT808, data: payload, ack: func() error { acked++; return nil }}},
map[string]string{topics.RawJT808: topics.RawJT808},
map[string]fieldsProjectionRoute{topics.RawJT808: {Protocol: envelope.ProtocolJT808, Topic: topics.FieldsJT808}},
true,
)
if err == nil {
t.Fatal("bridgeBatchWithProjection() error = nil, want fields Kafka error")
}
if acked != 0 {
t.Fatalf("acks = %d, want raw left pending until both Kafka outputs succeed", acked)
}
if len(writer.messages) != 1 || writer.messages[0].Topic != topics.RawJT808 {
t.Fatalf("successful kafka messages = %#v, want only raw before replay", writer.messages)
}
}
func TestBridgeBatchSkipsFieldsProjectionForNonRealtimeRaw(t *testing.T) {
raw := envelope.FrameEnvelope{
EventKind: envelope.EventKindRaw,
Protocol: envelope.ProtocolJT808,
MessageID: "0x0100",
Phone: "13307795425",
EventTimeMS: 1_700_000_000_000,
ReceivedAtMS: 1_700_000_000_100,
ParseStatus: envelope.ParseOK,
}
payload, err := raw.MarshalJSONBytes()
if err != nil {
t.Fatal(err)
}
acked := 0
writer := &recordingBridgeWriter{}
registry := metrics.NewRegistry()
err = bridgeBatchWithProjection(
context.Background(),
registry,
writer,
[]bridgeMessage{{subject: topics.RawJT808, data: payload, ack: func() error { acked++; return nil }}},
map[string]string{topics.RawJT808: topics.RawJT808},
map[string]fieldsProjectionRoute{topics.RawJT808: {Protocol: envelope.ProtocolJT808, Topic: topics.FieldsJT808}},
true,
)
if err != nil {
t.Fatalf("bridgeBatchWithProjection() error = %v", err)
}
if acked != 1 || len(writer.messages) != 1 || writer.messages[0].Topic != topics.RawJT808 {
t.Fatalf("acks=%d messages=%#v", acked, writer.messages)
}
if text := registry.Render(); !strings.Contains(text, `vehicle_bridge_fields_projection_total{protocol="JT808",status="skipped_non_realtime"} 1`) {
t.Fatalf("non-realtime projection metric missing:\n%s", text)
}
}
func TestBridgeBatchAcksSuccessfulTopicWhenAnotherTopicFails(t *testing.T) {
rawEnv := envelope.FrameEnvelope{Protocol: envelope.ProtocolJT808, Phone: "13307795425", MessageID: "0x0200"}
fieldsEnv := envelope.FrameEnvelope{Protocol: envelope.ProtocolJT808, Phone: "13307795425", MessageID: "0x0200"}
rawPayload, err := json.Marshal(rawEnv)
if err != nil {
t.Fatal(err)
}
fieldsPayload, err := json.Marshal(fieldsEnv)
if err != nil {
t.Fatal(err)
}
registry := metrics.NewRegistry()
writer := &recordingBridgeWriter{
topicErr: map[string]error{
"vehicle.fields.go.jt808.v1": errors.New("fields topic unavailable"),
},
}
acks := map[string]int{}
err = bridgeBatch(context.Background(), registry, writer, []bridgeMessage{
{subject: "vehicle.fields.go.jt808.v1", data: fieldsPayload, ack: func() error {
acks["fields"]++
return nil
}},
{subject: "vehicle.raw.go.jt808.v1", data: rawPayload, ack: func() error {
acks["raw"]++
return nil
}},
}, map[string]string{
"vehicle.fields.go.jt808.v1": "vehicle.fields.go.jt808.v1",
"vehicle.raw.go.jt808.v1": "vehicle.raw.go.jt808.v1",
})
if err == nil {
t.Fatal("bridgeBatch() error = nil, want failed fields topic")
}
if len(writer.calls) != 2 {
t.Fatalf("writer calls = %d, want one call per topic", len(writer.calls))
}
if len(writer.messages) != 1 || writer.messages[0].Topic != "vehicle.raw.go.jt808.v1" {
t.Fatalf("successful kafka messages = %#v, want only raw topic", writer.messages)
}
if acks["raw"] != 1 || acks["fields"] != 0 {
t.Fatalf("acks = %#v, want raw acked and fields left unacked", acks)
}
text := registry.Render()
for _, want := range []string{
`vehicle_bridge_kafka_writes_total{status="error",topic="vehicle.fields.go.jt808.v1"} 1`,
`vehicle_bridge_kafka_writes_total{status="ok",topic="vehicle.raw.go.jt808.v1"} 1`,
`vehicle_bridge_nats_acks_total{status="ok",subject="vehicle.raw.go.jt808.v1"} 1`,
} {
if !strings.Contains(text, want) {
t.Fatalf("partial bridge metric missing %s:\n%s", want, text)
}
}
if strings.Contains(text, `vehicle_bridge_nats_acks_total{status="ok",subject="vehicle.fields.go.jt808.v1"}`) {
t.Fatalf("failed topic should not be acked:\n%s", text)
}
}
func TestBridgeBatchAcksAndDropsUnroutedSubject(t *testing.T) {
writer := &recordingBridgeWriter{}
acked := 0
registry := metrics.NewRegistry()
err := bridgeBatch(context.Background(), registry, writer, []bridgeMessage{
{subject: "vehicle.unconfigured.v1", data: []byte(`{"protocol":"JT808"}`), ack: func() error {
acked++
return nil
}},
}, map[string]string{"vehicle.raw.go.jt808.v1": "vehicle.raw.go.jt808.v1"})
if err != nil {
t.Fatalf("bridgeBatch() error = %v", err)
}
if acked != 1 {
t.Fatalf("acks = %d, want unrouted message acked", acked)
}
if len(writer.messages) != 0 {
t.Fatalf("kafka writes = %d, want 0", len(writer.messages))
}
text := registry.Render()
for _, want := range []string{
`vehicle_bridge_messages_total{status="route_error",subject="vehicle.unconfigured.v1"} 1`,
`vehicle_bridge_nats_acks_total{status="dropped_route_error",subject="vehicle.unconfigured.v1"} 1`,
`vehicle_bridge_batch_pending_kafka_messages 0`,
} {
if !strings.Contains(text, want) {
t.Fatalf("metrics missing %s:\n%s", want, text)
}
}
}
func TestBridgeBatchKeepsValidMessagesWhenUnroutedSubjectIsPresent(t *testing.T) {
env := envelope.FrameEnvelope{Protocol: envelope.ProtocolJT808, Phone: "13307795425", MessageID: "0x0200"}
payload, err := json.Marshal(env)
if err != nil {
t.Fatal(err)
}
writer := &recordingBridgeWriter{}
acked := map[string]int{}
err = bridgeBatch(context.Background(), nil, writer, []bridgeMessage{
{subject: "vehicle.unconfigured.v1", data: []byte(`{"protocol":"JT808"}`), ack: func() error {
acked["unknown"]++
return nil
}},
{subject: "vehicle.raw.go.jt808.v1", data: payload, ack: func() error {
acked["valid"]++
return nil
}},
}, map[string]string{"vehicle.raw.go.jt808.v1": "vehicle.raw.go.jt808.v1"})
if err != nil {
t.Fatalf("bridgeBatch() error = %v", err)
}
if len(writer.messages) != 1 {
t.Fatalf("kafka writes = %d, want 1 valid message", len(writer.messages))
}
if writer.messages[0].Topic != "vehicle.raw.go.jt808.v1" {
t.Fatalf("topic = %q", writer.messages[0].Topic)
}
if acked["unknown"] != 1 || acked["valid"] != 1 {
t.Fatalf("acks = %#v, want both messages acked", acked)
}
}
func TestBridgeBatchReturnsErrorWhenUnroutedAckFails(t *testing.T) {
writer := &recordingBridgeWriter{}
err := bridgeBatch(context.Background(), nil, writer, []bridgeMessage{
{subject: "vehicle.unconfigured.v1", data: []byte(`{"protocol":"JT808"}`), ack: func() error {
return errors.New("nats ack unavailable")
}},
}, map[string]string{"vehicle.raw.go.jt808.v1": "vehicle.raw.go.jt808.v1"})
if err == nil {
t.Fatal("bridgeBatch() error = nil, want ack error")
}
if len(writer.messages) != 0 {
t.Fatalf("kafka writes = %d, want 0", len(writer.messages))
}
}
func TestBridgeBatchRecordsMetrics(t *testing.T) {
env := envelope.FrameEnvelope{
Protocol: envelope.ProtocolJT808,
Phone: "13307795425",
MessageID: "0x0200",
ReceivedAtMS: time.Now().Add(-20 * time.Millisecond).UnixMilli(),
}
payload, err := json.Marshal(env)
if err != nil {
t.Fatal(err)
}
registry := metrics.NewRegistry()
writer := &recordingBridgeWriter{}
@@ -83,8 +377,16 @@ func TestBridgeBatchRecordsMetrics(t *testing.T) {
text := registry.Render()
for _, want := range []string{
`vehicle_bridge_messages_total{status="received",subject="vehicle.raw.jt808.v1"} 1`,
`vehicle_bridge_last_message_unix_seconds{status="received",subject="vehicle.raw.jt808.v1"} `,
`vehicle_bridge_kafka_writes_total{status="ok",topic="vehicle.raw.jt808.v1"} 1`,
`vehicle_bridge_last_kafka_write_unix_seconds{status="ok",topic="vehicle.raw.jt808.v1"} `,
`vehicle_bridge_kafka_write_duration_ms_histogram_count{status="ok",topic="vehicle.raw.jt808.v1"} 1`,
`vehicle_bridge_kafka_e2e_duration_ms_histogram_count{topic="vehicle.raw.jt808.v1"} 1`,
`vehicle_bridge_kafka_e2e_recent_p99_ms{topic="vehicle.raw.jt808.v1"} `,
`vehicle_bridge_kafka_e2e_recent_samples{topic="vehicle.raw.jt808.v1"} `,
`vehicle_bridge_last_kafka_e2e_unix_seconds{topic="vehicle.raw.jt808.v1"} `,
`vehicle_bridge_nats_acks_total{status="ok",subject="vehicle.raw.jt808.v1"} 1`,
`vehicle_bridge_last_ack_unix_seconds{status="ok",subject="vehicle.raw.jt808.v1"} `,
} {
if !strings.Contains(text, want) {
t.Fatalf("metrics missing %s:\n%s", want, text)
@@ -92,6 +394,16 @@ func TestBridgeBatchRecordsMetrics(t *testing.T) {
}
}
func TestRecordBridgeKafkaE2EDurationSkipsMissingReceiveTime(t *testing.T) {
registry := metrics.NewRegistry()
recordBridgeKafkaE2EDuration(registry, "vehicle.raw.go.jt808.v1", 0)
if text := registry.Render(); strings.Contains(text, "vehicle_bridge_kafka_e2e_duration_ms_histogram") || strings.Contains(text, "vehicle_bridge_kafka_e2e_recent") {
t.Fatalf("e2e metric should be skipped when received_at_ms is missing:\n%s", text)
}
}
func TestBridgeBatchExposesPendingAndDurationMetrics(t *testing.T) {
env := envelope.FrameEnvelope{Protocol: envelope.ProtocolGB32960, VIN: "VIN001", MessageID: "0x02"}
payload, err := json.Marshal(env)
@@ -135,6 +447,73 @@ func TestBridgeBatchExposesPendingAndDurationMetrics(t *testing.T) {
}
}
func TestBridgeBatchPendingAggregatesConcurrentWorkers(t *testing.T) {
env := envelope.FrameEnvelope{Protocol: envelope.ProtocolGB32960, VIN: "VIN001", MessageID: "0x02"}
payload, err := json.Marshal(env)
if err != nil {
t.Fatal(err)
}
registry := metrics.NewRegistry()
firstStarted := make(chan struct{})
secondStarted := make(chan struct{})
release := make(chan struct{})
writerFor := func(started chan struct{}) *recordingBridgeWriter {
return &recordingBridgeWriter{
onWrite: func() {
close(started)
<-release
},
}
}
messages := func(count int) []bridgeMessage {
out := make([]bridgeMessage, 0, count)
for i := 0; i < count; i++ {
out = append(out, bridgeMessage{subject: "vehicle.raw.go.gb32960.v1", data: payload, ack: func() error { return nil }})
}
return out
}
route := map[string]string{"vehicle.raw.go.gb32960.v1": "vehicle.raw.go.gb32960.v1"}
var wg sync.WaitGroup
wg.Add(2)
var firstErr error
var secondErr error
go func() {
defer wg.Done()
firstErr = bridgeBatch(context.Background(), registry, writerFor(firstStarted), messages(2), route)
}()
go func() {
defer wg.Done()
secondErr = bridgeBatch(context.Background(), registry, writerFor(secondStarted), messages(3), route)
}()
<-firstStarted
<-secondStarted
text := registry.Render()
for _, want := range []string{
`vehicle_bridge_batch_pending_messages 5`,
`vehicle_bridge_batch_pending_kafka_messages 5`,
} {
if !strings.Contains(text, want) {
t.Fatalf("aggregate pending bridge metric missing %s during concurrent writes:\n%s", want, text)
}
}
close(release)
wg.Wait()
if firstErr != nil || secondErr != nil {
t.Fatalf("bridgeBatch errors = %v / %v", firstErr, secondErr)
}
text = registry.Render()
for _, want := range []string{
`vehicle_bridge_batch_pending_messages 0`,
`vehicle_bridge_batch_pending_kafka_messages 0`,
} {
if !strings.Contains(text, want) {
t.Fatalf("aggregate pending bridge metric should reset after concurrent writes, missing %s:\n%s", want, text)
}
}
}
func TestRecordNATSConsumerInfoMetrics(t *testing.T) {
registry := metrics.NewRegistry()
cfg := config{NATSStream: "VEHICLE_INGEST", NATSDurable: "vehicle-kafka-bridge"}
@@ -157,8 +536,45 @@ func TestRecordNATSConsumerInfoMetrics(t *testing.T) {
}
}
func TestIsBridgeShutdownFetchError(t *testing.T) {
if isBridgeShutdownFetchError(context.Background(), errors.New("temporary nats failure")) {
t.Fatal("temporary failure should not be treated as shutdown")
}
if !isBridgeShutdownFetchError(context.Background(), nats.ErrConnectionClosed) {
t.Fatal("connection closed should stop worker without noisy error log")
}
ctx, cancel := context.WithCancel(context.Background())
cancel()
if !isBridgeShutdownFetchError(ctx, errors.New("any fetch error after cancellation")) {
t.Fatal("cancelled context should stop worker without noisy error log")
}
}
func TestIsTransientBridgeFetchError(t *testing.T) {
for _, err := range []error{
errors.New("nats: disconnected during fetch"),
errors.New("nats: connection closed"),
errors.New("read tcp: connection reset by peer"),
errors.New("temporary network unavailable"),
errors.New("i/o timeout"),
} {
if !isTransientBridgeFetchError(err) {
t.Fatalf("isTransientBridgeFetchError(%v) = false, want true", err)
}
}
if isTransientBridgeFetchError(errors.New("permission denied")) {
t.Fatal("non-transient bridge fetch error should stay non-transient")
}
if isTransientBridgeFetchError(nil) {
t.Fatal("nil should not be transient")
}
}
func TestLoadConfigDefaultsToGoSubjectRoutes(t *testing.T) {
cfg := loadConfig()
if err := cfg.Validate(); err != nil {
t.Fatalf("default config Validate() error = %v", err)
}
want := map[string]string{
"vehicle.raw.go.gb32960.v1": "vehicle.raw.go.gb32960.v1",
@@ -176,6 +592,75 @@ func TestLoadConfigDefaultsToGoSubjectRoutes(t *testing.T) {
t.Fatalf("route[%q] = %q, want %q; route=%#v", subject, got, topic, cfg.Route)
}
}
if !cfg.DeriveFieldsFromRaw {
t.Fatal("DeriveFieldsFromRaw = false, want canonical raw projection enabled by default")
}
for subject, wantTopic := range map[string]string{
topics.RawGB32960: topics.FieldsGB32960,
topics.RawJT808: topics.FieldsJT808,
topics.RawYutongMQTT: topics.FieldsYutongMQTT,
} {
if got := cfg.RawFieldRoutes[subject].Topic; got != wantTopic {
t.Fatalf("RawFieldRoutes[%q].Topic = %q, want %q", subject, got, wantTopic)
}
}
}
func TestConfigValidateRejectsRawFieldsSubjectOverlap(t *testing.T) {
cfg := config{
RawRoutes: []subjectRoute{
{Subject: "vehicle.same.jt808", Topic: "vehicle.raw.go.jt808.v1"},
},
FieldsRoutes: []subjectRoute{
{Subject: "vehicle.same.jt808", Topic: "vehicle.fields.go.jt808.v1"},
},
}
err := cfg.Validate()
if err == nil {
t.Fatal("Validate() error = nil, want subject overlap rejection")
}
if !strings.Contains(err.Error(), "nats subject") {
t.Fatalf("Validate() error = %q, want nats subject hint", err)
}
}
func TestConfigValidateRejectsKnownProtocolTopicMismatch(t *testing.T) {
cfg := config{
RawRoutes: []subjectRoute{
{Protocol: envelope.ProtocolJT808, Subject: "vehicle.raw.go.jt808.v1", Topic: "vehicle.raw.go.gb32960.v1"},
},
}
err := cfg.Validate()
if err == nil {
t.Fatal("Validate() error = nil, want known protocol topic mismatch")
}
if !strings.Contains(err.Error(), "must match protocol") {
t.Fatalf("Validate() error = %q, want protocol mismatch hint", err)
}
}
func TestConfigValidateRejectsFieldsRouteToRawKafkaTopic(t *testing.T) {
cfg := config{
RawRoutes: []subjectRoute{
{Subject: "vehicle.raw.go.jt808.v1", Topic: "vehicle.raw.go.jt808.v1"},
},
FieldsRoutes: []subjectRoute{
{Subject: "vehicle.fields.go.jt808.v1", Topic: "vehicle.raw.go.jt808.v1"},
},
}
err := cfg.Validate()
if err == nil {
t.Fatal("Validate() error = nil, want fields topic family rejection")
}
if !strings.Contains(err.Error(), "fields kafka topic") {
t.Fatalf("Validate() error = %q, want fields kafka topic hint", err)
}
}
func TestLoadConfigDefaultsStreamMaxBytes(t *testing.T) {
@@ -187,11 +672,27 @@ func TestLoadConfigDefaultsStreamMaxBytes(t *testing.T) {
if got, want := cfg.StreamEnsureWait, 60*time.Second; got != want {
t.Fatalf("StreamEnsureWait = %v, want %v", got, want)
}
if got, want := cfg.KafkaBatchTimeout, 20*time.Millisecond; got != want {
t.Fatalf("KafkaBatchTimeout = %v, want %v", got, want)
}
if got, want := cfg.KafkaWriteConcurrency, 6; got != want {
t.Fatalf("KafkaWriteConcurrency = %d, want %d", got, want)
}
if got, want := cfg.FetchWait, 20*time.Millisecond; got != want {
t.Fatalf("FetchWait = %v, want %v", got, want)
}
if got, want := cfg.Workers, 4; got != want {
t.Fatalf("Workers = %d, want %d", got, want)
}
}
func TestLoadConfigReadsStreamMaxBytesOverride(t *testing.T) {
t.Setenv("NATS_STREAM_MAX_BYTES", "1073741824")
t.Setenv("NATS_STREAM_ENSURE_TIMEOUT_SECONDS", "90")
t.Setenv("BRIDGE_KAFKA_BATCH_TIMEOUT_MS", "35")
t.Setenv("BRIDGE_KAFKA_WRITE_CONCURRENCY", "4")
t.Setenv("BRIDGE_FETCH_WAIT_MS", "45")
t.Setenv("BRIDGE_WORKERS", "6")
cfg := loadConfig()
@@ -201,6 +702,57 @@ func TestLoadConfigReadsStreamMaxBytesOverride(t *testing.T) {
if got, want := cfg.StreamEnsureWait, 90*time.Second; got != want {
t.Fatalf("StreamEnsureWait = %v, want %v", got, want)
}
if got, want := cfg.KafkaBatchTimeout, 35*time.Millisecond; got != want {
t.Fatalf("KafkaBatchTimeout = %v, want %v", got, want)
}
if got, want := cfg.KafkaWriteConcurrency, 4; got != want {
t.Fatalf("KafkaWriteConcurrency = %d, want %d", got, want)
}
if got, want := cfg.FetchWait, 45*time.Millisecond; got != want {
t.Fatalf("FetchWait = %v, want %v", got, want)
}
if got, want := cfg.Workers, 6; got != want {
t.Fatalf("Workers = %d, want %d", got, want)
}
}
func TestRecordBridgeConfigMetrics(t *testing.T) {
registry := metrics.NewRegistry()
recordBridgeConfigMetrics(registry, config{
KafkaBatchTimeout: 35 * time.Millisecond,
KafkaWriteConcurrency: 4,
BatchSize: 600,
FetchWait: 20 * time.Millisecond,
Workers: 6,
DeriveFieldsFromRaw: true,
})
text := registry.Render()
for _, want := range []string{
`vehicle_bridge_config{setting="batch_size"} 600`,
`vehicle_bridge_config{setting="fetch_wait_ms"} 20`,
`vehicle_bridge_config{setting="kafka_batch_timeout_ms"} 35`,
`vehicle_bridge_config{setting="kafka_write_concurrency"} 4`,
`vehicle_bridge_config{setting="workers"} 6`,
`vehicle_bridge_config{setting="derive_fields_from_raw_enabled"} 1`,
} {
if !strings.Contains(text, want) {
t.Fatalf("bridge config metric missing %s:\n%s", want, text)
}
}
}
func TestNewKafkaWriterUsesConfiguredBatchTimeout(t *testing.T) {
writer := newKafkaWriter(config{
KafkaBrokers: []string{"127.0.0.1:9092"},
KafkaBatchTimeout: 17 * time.Millisecond,
})
defer writer.Close()
if got, want := writer.BatchTimeout, 17*time.Millisecond; got != want {
t.Fatalf("BatchTimeout = %v, want %v", got, want)
}
}
func TestLoadConfigIncludesUnifiedOnlyWhenExplicitlyConfigured(t *testing.T) {
@@ -215,8 +767,11 @@ func TestLoadConfigIncludesUnifiedOnlyWhenExplicitlyConfigured(t *testing.T) {
}
type recordingBridgeWriter struct {
mu sync.Mutex
messages []kafka.Message
calls [][]kafka.Message
err error
topicErr map[string]error
onWrite func()
}
@@ -224,9 +779,72 @@ func (w *recordingBridgeWriter) WriteMessages(_ context.Context, messages ...kaf
if w.onWrite != nil {
w.onWrite()
}
w.mu.Lock()
defer w.mu.Unlock()
call := append([]kafka.Message(nil), messages...)
w.calls = append(w.calls, call)
if len(messages) > 0 && w.topicErr != nil {
if err := w.topicErr[messages[0].Topic]; err != nil {
return err
}
}
if w.err != nil {
return w.err
}
w.messages = append(w.messages, messages...)
return nil
}
type concurrentBridgeWriter struct {
started chan string
release chan struct{}
active atomic.Int32
max atomic.Int32
}
func (w *concurrentBridgeWriter) WriteMessages(_ context.Context, messages ...kafka.Message) error {
active := w.active.Add(1)
defer w.active.Add(-1)
for {
current := w.max.Load()
if active <= current || w.max.CompareAndSwap(current, active) {
break
}
}
w.started <- messages[0].Topic
<-w.release
return nil
}
func TestBridgeWritesIndependentKafkaTopicsConcurrently(t *testing.T) {
writer := &concurrentBridgeWriter{
started: make(chan string, 2),
release: make(chan struct{}),
}
messages := []bridgeMessage{
{subject: topics.RawJT808, data: []byte(`{"protocol":"JT808"}`), ack: func() error { return nil }},
{subject: topics.RawGB32960, data: []byte(`{"protocol":"GB32960"}`), ack: func() error { return nil }},
}
route := map[string]string{
topics.RawJT808: topics.RawJT808,
topics.RawGB32960: topics.RawGB32960,
}
done := make(chan error, 1)
go func() {
done <- bridgeBatchWithProjectionConcurrency(context.Background(), nil, writer, messages, route, nil, false, 2)
}()
for i := 0; i < 2; i++ {
select {
case <-writer.started:
case <-time.After(time.Second):
t.Fatal("independent Kafka topic writes did not start concurrently")
}
}
if got := writer.max.Load(); got != 2 {
t.Fatalf("max concurrent Kafka writes = %d, want 2", got)
}
close(writer.release)
if err := <-done; err != nil {
t.Fatalf("bridgeBatchWithProjectionConcurrency() error = %v", err)
}
}

1157
go/vehicle-gateway/cmd/realtime-api/main.go
View File

File diff suppressed because it is too large Load Diff

998
go/vehicle-gateway/cmd/realtime-api/main_test.go
View File

File diff suppressed because it is too large Load Diff

648
go/vehicle-gateway/cmd/stat-writer/main.go
View File

@@ -4,9 +4,14 @@ import (
"context"
"database/sql"
"encoding/json"
"errors"
"fmt"
"os"
"os/signal"
"sort"
"strconv"
"strings"
"sync"
"syscall"
"time"
@@ -14,6 +19,7 @@ import (
"github.com/segmentio/kafka-go"
"lingniu-vehicle-ingest/go/vehicle-gateway/internal/envelope"
"lingniu-vehicle-ingest/go/vehicle-gateway/internal/eventbus"
"lingniu-vehicle-ingest/go/vehicle-gateway/internal/health"
"lingniu-vehicle-ingest/go/vehicle-gateway/internal/metrics"
"lingniu-vehicle-ingest/go/vehicle-gateway/internal/observability"
@@ -27,6 +33,10 @@ func main() {
defer stop()
cfg := loadConfig()
if err := cfg.Validate(); err != nil {
logger.Error("invalid stat writer config", "error", err)
os.Exit(1)
}
db, err := sql.Open("mysql", cfg.MySQLDSN)
if err != nil {
logger.Error("mysql open failed", "error", err)
@@ -38,18 +48,64 @@ func main() {
os.Exit(1)
}
registry := metrics.NewRegistry()
metrics.RegisterKafkaConsumerInfo(registry, "vehicle-stat-writer", cfg.KafkaGroup, cfg.KafkaTopics)
registry.SetGauge("vehicle_stat_project_interval_seconds", nil, cfg.ProjectInterval.Seconds())
registry.SetGauge("vehicle_stat_source_touch_interval_seconds", nil, cfg.SourceTouchInterval.Seconds())
registry.SetGauge("vehicle_stat_cache_retention_seconds", nil, cfg.CacheRetention.Seconds())
registry.SetGauge("vehicle_stat_cache_cleanup_interval_seconds", nil, cfg.CacheCleanupInterval.Seconds())
registry.SetGauge("vehicle_stat_baseline_miss_ttl_seconds", nil, cfg.BaselineMissTTL.Seconds())
registry.SetGauge("vehicle_stat_config", metrics.Labels{"setting": "workers"}, float64(cfg.Workers))
health.Start(ctx, logger, health.NewServer(env("HEALTH_ADDR", ""), "vehicle-stat-writer", []health.Check{
{Name: "mysql", Check: db.PingContext},
}, registry))
writer := stats.NewWriter(db, cfg.Location)
writer.SetProjectionInterval(cfg.ProjectInterval)
writer.SetSourceTouchInterval(cfg.SourceTouchInterval)
writer.SetCacheRetention(cfg.CacheRetention)
writer.SetCacheCleanupInterval(cfg.CacheCleanupInterval)
writer.SetBaselineMissTTL(cfg.BaselineMissTTL)
writer.SetMaxCacheEntries(cfg.CacheMaxEntries)
if cfg.EnsureSchema {
if err := writer.EnsureSchema(ctx); err != nil {
logger.Error("mysql schema bootstrap failed", "error", err)
os.Exit(1)
}
}
if cfg.NormalizePlatformSourcesOnStart {
statDate := time.Now().In(cfg.Location).Format("2006-01-02")
normalizeCtx, cancel := context.WithTimeout(ctx, cfg.NormalizePlatformSourcesTimeout)
normalized, err := stats.NormalizePlatformSourceMileageForDate(normalizeCtx, db, statDate, envelope.ProtocolJT808)
cancel()
if err != nil {
logger.Warn("platform source startup normalization failed", "stat_date", statDate, "protocol", envelope.ProtocolJT808, "normalized", normalized, "error", err)
} else if normalized > 0 {
logger.Info("platform source startup normalization finished", "stat_date", statDate, "protocol", envelope.ProtocolJT808, "normalized", normalized)
}
}
var appender statAppender = retryStatAppender{
delegate: writer,
attempts: cfg.RetryAttempts,
delay: cfg.RetryDelay,
registry: registry,
}
logger.Info("stat writer started", "group", cfg.KafkaGroup, "topics", strings.Join(cfg.KafkaTopics, ","), "workers", cfg.Workers, "project_interval_seconds", cfg.ProjectInterval.Seconds(), "source_touch_interval_seconds", cfg.SourceTouchInterval.Seconds(), "cache_retention_seconds", cfg.CacheRetention.Seconds(), "cache_cleanup_interval_seconds", cfg.CacheCleanupInterval.Seconds(), "baseline_miss_ttl_seconds", cfg.BaselineMissTTL.Seconds(), "cache_max_entries", cfg.CacheMaxEntries, "batch_size", cfg.BatchSize, "batch_wait_ms", cfg.BatchWait, "retry_attempts", cfg.RetryAttempts, "retry_delay_ms", cfg.RetryDelay.Milliseconds())
var workers sync.WaitGroup
for workerID := 1; workerID <= cfg.Workers; workerID++ {
workers.Add(1)
go func(id int) {
defer workers.Done()
runStatConsumer(ctx, logger, registry, appender, cfg, id)
}(workerID)
}
workers.Wait()
}
func runStatConsumer(ctx context.Context, logger interface {
Error(string, ...any)
Warn(string, ...any)
}, registry *metrics.Registry, appender statAppender, cfg config, workerID int) {
reader := kafka.NewReader(kafka.ReaderConfig{
Brokers: cfg.KafkaBrokers,
GroupID: cfg.KafkaGroup,
@@ -59,7 +115,10 @@ func main() {
})
defer reader.Close()
logger.Info("stat writer started", "group", cfg.KafkaGroup, "topics", strings.Join(cfg.KafkaTopics, ","))
workerLabels := metrics.Labels{"worker": strconv.Itoa(workerID)}
registry.SetGauge("vehicle_stat_worker_active", workerLabels, 1)
defer registry.SetGauge("vehicle_stat_worker_active", workerLabels, 0)
for {
message, err := reader.FetchMessage(ctx)
if err != nil {
@@ -69,7 +128,8 @@ func main() {
logger.Error("kafka fetch failed", "error", err)
continue
}
processStatMessage(ctx, logger, registry, writer, reader, message)
batch := collectStatBatch(ctx, reader, message, cfg.BatchSize, time.Duration(cfg.BatchWait)*time.Millisecond)
processStatBatchReliablyForWorker(ctx, logger, registry, appender, reader, batch, cfg.RetryDelay, workerLabels)
}
}
@@ -79,50 +139,488 @@ type statAppender interface {
Append(context.Context, envelope.FrameEnvelope) error
}
type statResultAppender interface {
AppendWithResult(context.Context, envelope.FrameEnvelope) (stats.AppendResult, error)
}
type statCacheReporter interface {
CacheStats() stats.CacheStats
}
type kafkaMessageCommitter interface {
CommitMessages(context.Context, ...kafka.Message) error
}
type kafkaMessageFetcher interface {
FetchMessage(context.Context) (kafka.Message, error)
}
type statBatchItem struct {
message kafka.Message
processed bool
}
type statBatchOutcome struct {
commitMessages []kafka.Message
retryMessages []kafka.Message
}
var statWriteDurationBucketsMS = []float64{1, 5, 10, 25, 50, 100, 250, 500, 1000, 5000}
var statWriteE2EDurationBucketsMS = []float64{10, 25, 50, 100, 250, 500, 1000, 2500, 5000, 10000}
var statWriteE2ERecent = metrics.NewRecentLatencyByKey(512)
func collectStatBatch(ctx context.Context, fetcher kafkaMessageFetcher, first kafka.Message, maxSize int, maxWait time.Duration) []kafka.Message {
if maxSize <= 1 {
return []kafka.Message{first}
}
if maxWait <= 0 {
maxWait = 100 * time.Millisecond
}
batch := []kafka.Message{first}
deadline := time.Now().Add(maxWait)
for len(batch) < maxSize {
remaining := time.Until(deadline)
if remaining <= 0 {
break
}
fetchCtx, cancel := context.WithTimeout(ctx, remaining)
message, err := fetcher.FetchMessage(fetchCtx)
cancel()
if err != nil {
break
}
batch = append(batch, message)
}
return batch
}
func processStatMessage(ctx context.Context, logger interface {
Error(string, ...any)
Warn(string, ...any)
}, registry *metrics.Registry, appender statAppender, committer kafkaMessageCommitter, message kafka.Message) {
processStatBatch(ctx, logger, registry, appender, committer, []kafka.Message{message})
}
func processStatBatch(ctx context.Context, logger interface {
Error(string, ...any)
Warn(string, ...any)
}, registry *metrics.Registry, appender statAppender, committer kafkaMessageCommitter, messages []kafka.Message) statBatchOutcome {
return processStatBatchForWorker(ctx, logger, registry, appender, committer, messages, nil)
}
func processStatBatchForWorker(ctx context.Context, logger interface {
Error(string, ...any)
Warn(string, ...any)
}, registry *metrics.Registry, appender statAppender, committer kafkaMessageCommitter, messages []kafka.Message, workerLabels metrics.Labels) statBatchOutcome {
if len(messages) == 0 {
return statBatchOutcome{}
}
messageCtx, cancel := context.WithTimeout(context.WithoutCancel(ctx), kafkaMessageOperationTimeout)
defer cancel()
addStatMetric(registry, "vehicle_stat_kafka_messages_total", message, "received")
addStatLagMetric(registry, message)
var env envelope.FrameEnvelope
if err := json.Unmarshal(message.Value, &env); err != nil {
addStatMetric(registry, "vehicle_stat_kafka_messages_total", message, "invalid_json")
logger.Warn("skip invalid envelope json", "topic", message.Topic, "partition", message.Partition, "offset", message.Offset, "error", err)
_ = committer.CommitMessages(messageCtx, message)
setStatBatchPendingForWorker(registry, workerLabels, len(messages))
defer setStatBatchPendingForWorker(registry, workerLabels, 0)
defer recordStatCacheMetrics(registry, appender)
items := make([]statBatchItem, len(messages))
for index, message := range messages {
items[index].message = message
}
for itemIndex, message := range messages {
addStatMetric(registry, "vehicle_stat_kafka_messages_total", message, "received")
addStatLagMetric(registry, message)
var env envelope.FrameEnvelope
if err := json.Unmarshal(message.Value, &env); err != nil {
addStatMetric(registry, "vehicle_stat_kafka_messages_total", message, "invalid_json")
logger.Warn("skip invalid envelope json", "topic", message.Topic, "partition", message.Partition, "offset", message.Offset, "error", err)
items[itemIndex].processed = true
continue
}
if status, err := validateStatFieldsEnvelope(message.Topic, env); err != nil {
addStatMetric(registry, "vehicle_stat_kafka_messages_total", message, status)
logger.Warn("skip mismatched stat fields envelope", "topic", message.Topic, "partition", message.Partition, "offset", message.Offset, "protocol", env.Protocol, "event_id", env.StableEventID(), "event_kind", env.EventKind, "error", err)
items[itemIndex].processed = true
continue
}
started := time.Now()
result, err := appendStatEnvelope(messageCtx, appender, env)
recordStatWriteDuration(registry, message, statusFromError(err), time.Since(started))
recordStatSampleMetrics(registry, message, env.Protocol, result)
recordStatSourceMetrics(registry, message, env.Protocol, result)
recordStatProjectionMetrics(registry, message, env.Protocol, result)
if err != nil {
addStatMetric(registry, "vehicle_stat_writes_total", message, "error")
logger.Error("mysql append failed", "topic", message.Topic, "partition", message.Partition, "offset", message.Offset, "event_id", env.StableEventID(), "error", err)
committed, commitErr := commitStatProcessedPrefixAfterFailure(messageCtx, logger, registry, committer, items)
outcome := statBatchOutcome{retryMessages: eventbus.MessagesAfterCommittedPrefixes(messages, committed)}
if commitErr != nil {
outcome.commitMessages = committed
}
return outcome
}
addStatMetric(registry, "vehicle_stat_writes_total", message, "ok")
recordStatWriteE2EDuration(registry, message, env)
items[itemIndex].processed = true
}
if err := committer.CommitMessages(messageCtx, messages...); err != nil {
for _, message := range messages {
addStatMetric(registry, "vehicle_stat_kafka_commits_total", message, "error")
}
first := messages[0]
logger.Error("kafka commit failed", "topic", first.Topic, "partition", first.Partition, "offset", first.Offset, "messages", len(messages), "error", err)
return statBatchOutcome{commitMessages: messages}
}
for _, message := range messages {
addStatMetric(registry, "vehicle_stat_kafka_commits_total", message, "ok")
}
return statBatchOutcome{}
}
func processStatBatchReliably(ctx context.Context, logger interface {
Error(string, ...any)
Warn(string, ...any)
}, registry *metrics.Registry, appender statAppender, committer kafkaMessageCommitter, messages []kafka.Message, retryDelay time.Duration) {
processStatBatchReliablyForWorker(ctx, logger, registry, appender, committer, messages, retryDelay, nil)
}
func processStatBatchReliablyForWorker(ctx context.Context, logger interface {
Error(string, ...any)
Warn(string, ...any)
}, registry *metrics.Registry, appender statAppender, committer kafkaMessageCommitter, messages []kafka.Message, retryDelay time.Duration, workerLabels metrics.Labels) {
defer registry.SetGauge("vehicle_stat_retry_pending_messages", workerLabels, 0)
pending := messages
for len(pending) > 0 {
outcome := processStatBatchForWorker(ctx, logger, registry, appender, committer, pending, workerLabels)
if len(outcome.commitMessages) > 0 {
registry.IncCounter("vehicle_stat_batch_retries_total", metrics.Labels{"reason": "commit_error"})
if !retryStatCommit(ctx, logger, registry, committer, outcome.commitMessages, retryDelay) {
return
}
}
pending = outcome.retryMessages
registry.SetGauge("vehicle_stat_retry_pending_messages", workerLabels, float64(len(pending)))
if len(pending) == 0 {
return
}
registry.IncCounter("vehicle_stat_batch_retries_total", metrics.Labels{"reason": "write_error"})
if !waitForStatRetry(ctx, retryDelay) {
return
}
}
}
func retryStatCommit(ctx context.Context, logger interface {
Error(string, ...any)
Warn(string, ...any)
}, registry *metrics.Registry, committer kafkaMessageCommitter, messages []kafka.Message, retryDelay time.Duration) bool {
for len(messages) > 0 {
operationCtx, cancel := context.WithTimeout(context.WithoutCancel(ctx), kafkaMessageOperationTimeout)
err := committer.CommitMessages(operationCtx, messages...)
cancel()
if err == nil {
for _, message := range messages {
addStatMetric(registry, "vehicle_stat_kafka_commits_total", message, "ok")
}
return true
}
for _, message := range messages {
addStatMetric(registry, "vehicle_stat_kafka_commits_total", message, "error")
}
first := messages[0]
logger.Error("kafka commit retry failed", "topic", first.Topic, "partition", first.Partition, "offset", first.Offset, "messages", len(messages), "error", err)
registry.IncCounter("vehicle_stat_batch_retries_total", metrics.Labels{"reason": "commit_error"})
if !waitForStatRetry(ctx, retryDelay) {
return false
}
}
return true
}
func waitForStatRetry(ctx context.Context, retryDelay time.Duration) bool {
if retryDelay <= 0 {
retryDelay = 100 * time.Millisecond
}
timer := time.NewTimer(retryDelay)
defer timer.Stop()
select {
case <-ctx.Done():
return false
case <-timer.C:
return true
}
}
func validateStatFieldsEnvelope(topic string, env envelope.FrameEnvelope) (string, error) {
return topics.ValidateFieldsEnvelope(topic, env)
}
func commitStatProcessedPrefixAfterFailure(ctx context.Context, logger interface {
Error(string, ...any)
Warn(string, ...any)
}, registry *metrics.Registry, committer kafkaMessageCommitter, items []statBatchItem) ([]kafka.Message, error) {
committable := processedPrefixMessagesByPartition(items)
if len(committable) == 0 {
return nil, nil
}
if err := committer.CommitMessages(ctx, committable...); err != nil {
for _, message := range committable {
addStatMetric(registry, "vehicle_stat_kafka_commits_total", message, "error")
}
first := committable[0]
logger.Error("kafka processed-prefix commit failed after mysql append failure", "topic", first.Topic, "partition", first.Partition, "offset", first.Offset, "messages", len(committable), "error", err)
return committable, err
}
for _, message := range committable {
addStatMetric(registry, "vehicle_stat_kafka_commits_total", message, "ok")
}
return committable, nil
}
func processedPrefixMessagesByPartition(items []statBatchItem) []kafka.Message {
type partitionKey struct {
topic string
partition int
}
groups := map[partitionKey][]statBatchItem{}
for _, item := range items {
key := partitionKey{topic: item.message.Topic, partition: item.message.Partition}
groups[key] = append(groups[key], item)
}
var keys []partitionKey
for key := range groups {
keys = append(keys, key)
}
sort.Slice(keys, func(i, j int) bool {
if keys[i].topic != keys[j].topic {
return keys[i].topic < keys[j].topic
}
return keys[i].partition < keys[j].partition
})
var out []kafka.Message
for _, key := range keys {
group := groups[key]
sort.Slice(group, func(i, j int) bool {
return group[i].message.Offset < group[j].message.Offset
})
var previousOffset int64
for index, item := range group {
if index > 0 && item.message.Offset != previousOffset+1 {
break
}
if !item.processed {
break
}
out = append(out, item.message)
previousOffset = item.message.Offset
}
}
return out
}
func appendStatEnvelope(ctx context.Context, appender statAppender, env envelope.FrameEnvelope) (stats.AppendResult, error) {
if resultAppender, ok := appender.(statResultAppender); ok {
return resultAppender.AppendWithResult(ctx, env)
}
return stats.AppendResult{}, appender.Append(ctx, env)
}
type retryStatAppender struct {
delegate statAppender
attempts int
delay time.Duration
registry *metrics.Registry
}
func (a retryStatAppender) Append(ctx context.Context, env envelope.FrameEnvelope) error {
_, err := a.AppendWithResult(ctx, env)
return err
}
func (a retryStatAppender) AppendWithResult(ctx context.Context, env envelope.FrameEnvelope) (stats.AppendResult, error) {
if a.delegate == nil {
return stats.AppendResult{}, nil
}
attempts := a.attempts
if attempts <= 0 {
attempts = 1
}
var result stats.AppendResult
var err error
for attempt := 1; attempt <= attempts; attempt++ {
result, err = appendStatEnvelope(ctx, a.delegate, env)
if err == nil || !isTransientMySQLStatError(err) {
return result, err
}
if attempt == attempts {
a.recordRetry("single", "exhausted")
return result, err
}
a.recordRetry("single", "retry")
if a.delay <= 0 {
continue
}
timer := time.NewTimer(a.delay)
select {
case <-ctx.Done():
timer.Stop()
return result, ctx.Err()
case <-timer.C:
}
}
return result, err
}
func (a retryStatAppender) recordRetry(operation string, status string) {
if a.registry == nil {
return
}
started := time.Now()
err := appender.Append(messageCtx, env)
recordStatWriteDuration(registry, message, statusFromError(err), time.Since(started))
if err != nil {
addStatMetric(registry, "vehicle_stat_writes_total", message, "error")
logger.Error("mysql append failed", "topic", message.Topic, "partition", message.Partition, "offset", message.Offset, "event_id", env.StableEventID(), "error", err)
return
labels := metrics.Labels{"operation": operation, "status": status}
a.registry.IncCounter("vehicle_stat_write_retries_total", labels)
metrics.RecordLastActivity(a.registry, "vehicle_stat_last_write_retry_unix_seconds", labels)
}
func (a retryStatAppender) CacheStats() stats.CacheStats {
if reporter, ok := a.delegate.(statCacheReporter); ok {
return reporter.CacheStats()
}
addStatMetric(registry, "vehicle_stat_writes_total", message, "ok")
if err := committer.CommitMessages(messageCtx, message); err != nil {
addStatMetric(registry, "vehicle_stat_kafka_commits_total", message, "error")
logger.Error("kafka commit failed", "topic", message.Topic, "partition", message.Partition, "offset", message.Offset, "error", err)
return
return stats.CacheStats{}
}
func isTransientMySQLStatError(err error) bool {
if err == nil || errors.Is(err, context.Canceled) || errors.Is(err, context.DeadlineExceeded) {
return false
}
addStatMetric(registry, "vehicle_stat_kafka_commits_total", message, "ok")
text := strings.ToLower(strings.TrimSpace(err.Error()))
return strings.Contains(text, "deadlock") ||
strings.Contains(text, "error 1213") ||
strings.Contains(text, "40001") ||
strings.Contains(text, "lock wait timeout") ||
strings.Contains(text, "error 1205") ||
strings.Contains(text, "timeout") ||
strings.Contains(text, "temporary") ||
strings.Contains(text, "temporarily") ||
strings.Contains(text, "connection refused") ||
strings.Contains(text, "connection reset") ||
strings.Contains(text, "connection closed") ||
strings.Contains(text, "broken pipe") ||
strings.Contains(text, "bad connection") ||
strings.Contains(text, "invalid connection") ||
strings.Contains(text, "i/o timeout") ||
text == "eof" ||
strings.Contains(text, "unexpected eof") ||
strings.Contains(text, "server is down") ||
strings.Contains(text, "network is unreachable") ||
strings.Contains(text, "no route to host")
}
func addStatMetric(registry *metrics.Registry, name string, message kafka.Message, status string) {
if registry == nil {
return
}
registry.IncCounter(name, metrics.Labels{"topic": message.Topic, "status": status})
labels := metrics.Labels{"topic": message.Topic, "status": status}
registry.IncCounter(name, labels)
switch name {
case "vehicle_stat_kafka_messages_total":
metrics.RecordLastActivity(registry, "vehicle_stat_last_message_unix_seconds", labels)
case "vehicle_stat_writes_total":
metrics.RecordLastActivity(registry, "vehicle_stat_last_write_unix_seconds", labels)
case "vehicle_stat_kafka_commits_total":
metrics.RecordLastActivity(registry, "vehicle_stat_last_commit_unix_seconds", labels)
}
}
func recordStatSampleMetrics(registry *metrics.Registry, message kafka.Message, protocol envelope.Protocol, result stats.AppendResult) {
addStatSampleMetric(registry, message, protocol, "found", result.SamplesFound)
addStatSampleMetric(registry, message, protocol, "written", result.SamplesWritten)
addStatSampleMetric(registry, message, protocol, "skipped_missing_fields", result.SamplesSkippedMissingFields)
addStatSampleMetric(registry, message, protocol, "skipped_missing_vin", result.SamplesSkippedMissingVIN)
addStatSampleMetric(registry, message, protocol, "skipped_missing_mileage", result.SamplesSkippedMissingMileage)
addStatSampleMetric(registry, message, protocol, "skipped_non_mileage_frame", result.SamplesSkippedNonMileageFrame)
addStatSampleMetric(registry, message, protocol, "skipped_non_positive_mileage", result.SamplesSkippedNonPositiveMileage)
addStatSampleMetric(registry, message, protocol, "skipped_missing_time", result.SamplesSkippedMissingTime)
addStatSampleMetric(registry, message, protocol, "skipped_same_mileage", result.SamplesSkippedSameMileage)
addStatSampleMetric(registry, message, protocol, "skipped_missing_source", result.SamplesSkippedMissingSource)
addStatSampleMetric(registry, message, protocol, "event_time_future_adjusted", result.SamplesAdjustedFutureEventTime)
}
func addStatSampleMetric(registry *metrics.Registry, message kafka.Message, protocol envelope.Protocol, status string, count int) {
if registry == nil || count == 0 {
return
}
registry.AddCounter("vehicle_stat_samples_total", metrics.Labels{
"topic": message.Topic,
"protocol": statProtocolLabel(protocol),
"status": status,
}, float64(count))
}
func recordStatSourceMetrics(registry *metrics.Registry, message kafka.Message, protocol envelope.Protocol, result stats.AppendResult) {
addStatSourceMetric(registry, message, protocol, "attempted", result.SourceTouchesAttempted)
addStatSourceMetric(registry, message, protocol, "written", result.SourceTouchesWritten)
addStatSourceMetric(registry, message, protocol, "skipped_throttled", result.SourceTouchesSkippedThrottled)
addStatSourceMetric(registry, message, protocol, "skipped_missing_endpoint", result.SourceTouchesSkippedMissing)
addStatSourceMetric(registry, message, protocol, "skipped_unmanaged", result.SourceTouchesSkippedUnmanaged)
}
func addStatSourceMetric(registry *metrics.Registry, message kafka.Message, protocol envelope.Protocol, status string, count int) {
if registry == nil || count == 0 {
return
}
registry.AddCounter("vehicle_stat_sources_total", metrics.Labels{
"topic": message.Topic,
"protocol": statProtocolLabel(protocol),
"status": status,
}, float64(count))
}
func recordStatProjectionMetrics(registry *metrics.Registry, message kafka.Message, protocol envelope.Protocol, result stats.AppendResult) {
addStatProjectionMetric(registry, message, protocol, "attempted", result.ProjectionsAttempted)
addStatProjectionMetric(registry, message, protocol, "written", result.ProjectionsWritten)
addStatProjectionMetric(registry, message, protocol, "skipped_throttled", result.ProjectionsSkippedThrottled)
}
func addStatProjectionMetric(registry *metrics.Registry, message kafka.Message, protocol envelope.Protocol, status string, count int) {
if registry == nil || count == 0 {
return
}
registry.AddCounter("vehicle_stat_projections_total", metrics.Labels{
"topic": message.Topic,
"protocol": statProtocolLabel(protocol),
"status": status,
}, float64(count))
}
func statProtocolLabel(protocol envelope.Protocol) string {
protocolLabel := strings.TrimSpace(string(protocol))
if protocolLabel == "" {
return "UNKNOWN"
}
return protocolLabel
}
func recordStatCacheMetrics(registry *metrics.Registry, appender statAppender) {
if registry == nil {
return
}
reporter, ok := appender.(statCacheReporter)
if !ok {
return
}
stats := reporter.CacheStats()
setStatCacheGauge(registry, "last_total_mileage", stats.LastTotalMileageEntries, stats.MaxEntries, stats.LastCleanupTotalMileage, stats.TotalMileageEvictions)
setStatCacheGauge(registry, "source_seen", stats.LastSourceSeenEntries, stats.MaxEntries, stats.LastCleanupSourceSeen, stats.SourceSeenEvictions)
setStatCacheGauge(registry, "projection", stats.LastProjectionEntries, stats.MaxEntries, stats.LastCleanupProjection, stats.ProjectionEvictions)
setStatCacheGauge(registry, "baseline", stats.BaselineEntries, stats.MaxEntries, stats.LastCleanupBaseline, stats.BaselineEvictions)
if !stats.LastCleanupAt.IsZero() {
registry.SetGauge("vehicle_stat_cache_last_cleanup_unix_seconds", nil, float64(stats.LastCleanupAt.Unix()))
}
}
func setStatCacheGauge(registry *metrics.Registry, cache string, entries int, maxEntries int, lastCleanupDeleted int, evictions int) {
labels := metrics.Labels{"cache": cache}
registry.SetGauge("vehicle_stat_cache_entries", labels, float64(entries))
registry.SetGauge("vehicle_stat_cache_max_entries", labels, float64(maxEntries))
registry.SetGauge("vehicle_stat_cache_last_cleanup_deleted", labels, float64(lastCleanupDeleted))
registry.SetGauge("vehicle_stat_cache_evictions_total", labels, float64(evictions))
}
func addStatLagMetric(registry *metrics.Registry, message kafka.Message) {
@@ -142,6 +640,33 @@ func recordStatWriteDuration(registry *metrics.Registry, message kafka.Message,
}, statWriteDurationBucketsMS, float64(elapsed.Milliseconds()))
}
func recordStatWriteE2EDuration(registry *metrics.Registry, message kafka.Message, env envelope.FrameEnvelope) {
if registry == nil || env.ReceivedAtMS <= 0 {
return
}
elapsed := time.Since(time.UnixMilli(env.ReceivedAtMS)).Milliseconds()
if elapsed < 0 {
elapsed = 0
}
labels := metrics.Labels{"topic": message.Topic}
registry.ObserveHistogram("vehicle_stat_write_e2e_duration_ms_histogram", labels, statWriteE2EDurationBucketsMS, float64(elapsed))
p99, samples := statWriteE2ERecent.Observe(message.Topic, float64(elapsed))
registry.SetGauge("vehicle_stat_write_e2e_recent_p99_ms", labels, p99)
registry.SetGauge("vehicle_stat_write_e2e_recent_samples", labels, float64(samples))
metrics.RecordLastActivity(registry, "vehicle_stat_last_write_e2e_unix_seconds", labels)
}
func setStatBatchPending(registry *metrics.Registry, messages int) {
setStatBatchPendingForWorker(registry, nil, messages)
}
func setStatBatchPendingForWorker(registry *metrics.Registry, workerLabels metrics.Labels, messages int) {
if registry == nil {
return
}
registry.SetGauge("vehicle_stat_batch_pending_messages", workerLabels, float64(messages))
}
func statusFromError(err error) string {
if err != nil {
return "error"
@@ -150,12 +675,40 @@ func statusFromError(err error) string {
}
type config struct {
KafkaBrokers []string
KafkaTopics []string
KafkaGroup string
MySQLDSN string
EnsureSchema bool
Location *time.Location
KafkaBrokers []string
KafkaTopics []string
KafkaGroup string
MySQLDSN string
EnsureSchema bool
Location *time.Location
ProjectInterval time.Duration
SourceTouchInterval time.Duration
CacheRetention time.Duration
CacheCleanupInterval time.Duration
BaselineMissTTL time.Duration
CacheMaxEntries int
Workers int
BatchSize int
BatchWait int
RetryAttempts int
RetryDelay time.Duration
NormalizePlatformSourcesOnStart bool
NormalizePlatformSourcesTimeout time.Duration
}
func (c config) Validate() error {
if len(c.KafkaTopics) == 0 {
return fmt.Errorf("KAFKA_TOPICS must include fields topics")
}
for _, topic := range c.KafkaTopics {
if !strings.HasPrefix(topic, "vehicle.fields.") {
return fmt.Errorf("stat-writer consumes fields topics only, got %q", topic)
}
}
if c.Workers <= 0 {
return fmt.Errorf("STATS_WORKERS must be greater than zero")
}
return nil
}
func loadConfig() config {
@@ -164,12 +717,25 @@ func loadConfig() config {
loc = time.FixedZone("Asia/Shanghai", 8*3600)
}
return config{
KafkaBrokers: splitCSV(env("KAFKA_BROKERS", "127.0.0.1:9092")),
KafkaTopics: splitCSV(env("KAFKA_TOPICS", strings.Join([]string{topics.FieldsGB32960, topics.FieldsJT808, topics.FieldsYutongMQTT}, ","))),
KafkaGroup: env("KAFKA_GROUP", "go-stat-writer"),
MySQLDSN: env("MYSQL_DSN", ""),
EnsureSchema: env("MYSQL_ENSURE_SCHEMA", "true") != "false",
Location: loc,
KafkaBrokers: splitCSV(env("KAFKA_BROKERS", "127.0.0.1:9092")),
KafkaTopics: splitCSV(env("KAFKA_TOPICS", strings.Join([]string{topics.FieldsGB32960, topics.FieldsJT808, topics.FieldsYutongMQTT}, ","))),
KafkaGroup: env("KAFKA_GROUP", "go-stat-writer"),
MySQLDSN: env("MYSQL_DSN", ""),
EnsureSchema: env("MYSQL_ENSURE_SCHEMA", "true") != "false",
Location: loc,
ProjectInterval: time.Duration(envInt("STATS_PROJECT_INTERVAL_SECONDS", 15)) * time.Second,
SourceTouchInterval: time.Duration(envInt("STATS_SOURCE_TOUCH_INTERVAL_SECONDS", 60)) * time.Second,
CacheRetention: time.Duration(envInt("STATS_CACHE_RETENTION_HOURS", 72)) * time.Hour,
CacheCleanupInterval: time.Duration(envInt("STATS_CACHE_CLEANUP_INTERVAL_SECONDS", 600)) * time.Second,
BaselineMissTTL: time.Duration(envInt("STATS_BASELINE_MISS_TTL_SECONDS", 60)) * time.Second,
CacheMaxEntries: envInt("STATS_CACHE_MAX_ENTRIES", 1000000),
Workers: envInt("STATS_WORKERS", 3),
BatchSize: envInt("STATS_BATCH_SIZE", 200),
BatchWait: envInt("STATS_BATCH_WAIT_MS", 20),
RetryAttempts: envInt("STATS_RETRY_ATTEMPTS", 3),
RetryDelay: time.Duration(envInt("STATS_RETRY_DELAY_MS", 20)) * time.Millisecond,
NormalizePlatformSourcesOnStart: env("STATS_NORMALIZE_PLATFORM_SOURCES_ON_START", "true") != "false",
NormalizePlatformSourcesTimeout: time.Duration(envInt("STATS_NORMALIZE_PLATFORM_SOURCES_TIMEOUT_SECONDS", 30)) * time.Second,
}
}
@@ -181,6 +747,18 @@ func env(key string, fallback string) string {
return value
}
func envInt(key string, fallback int) int {
value := strings.TrimSpace(os.Getenv(key))
if value == "" {
return fallback
}
parsed, err := strconv.Atoi(value)
if err != nil {
return fallback
}
return parsed
}
func splitCSV(value string) []string {
var out []string
for _, item := range strings.Split(value, ",") {

1029
go/vehicle-gateway/cmd/stat-writer/main_test.go
View File

File diff suppressed because it is too large Load Diff

559
go/vehicle-gateway/cmd/stats-backfill/main.go
View File

@@ -22,20 +22,21 @@ import (
)
type config struct {
MySQLDSN string
TDengineDriver string
TDengineDSN string
TDengineDatabase string
DateFrom string
DateTo string
Protocols []envelope.Protocol
Method string
Limit int
DryRun bool
Reset bool
Debug bool
ProgressEvery int64
Location *time.Location
MySQLDSN string
TDengineDriver string
TDengineDSN string
TDengineDatabase string
DateFrom string
DateTo string
Protocols []envelope.Protocol
Method string
Limit int
DryRun bool
Reset bool
Debug bool
EventTimeFullScan bool
ProgressEvery int64
Location *time.Location
}
type rawFrameRow struct {
@@ -62,6 +63,9 @@ type metricAgg struct {
Phone string
DeviceID string
SourceEndpoint string
SourceCode string
PlatformName string
SourceKind string
FirstEventTime time.Time
LatestEventTime time.Time
QualityStatus string
@@ -74,6 +78,9 @@ type dailySourceLast struct {
Phone string
DeviceID string
SourceEndpoint string
SourceCode string
PlatformName string
SourceKind string
FirstTS time.Time
TS time.Time
FirstTotalKM float64
@@ -81,8 +88,19 @@ type dailySourceLast struct {
RawSampleCount int64
}
type sourceHistoryID struct {
VIN string
SourceKey string
}
var defaultBackfillEnvFiles = []string{
"/opt/lingniu-go-native/env/base.env",
"/opt/lingniu-go-native/env/stat-writer.env",
}
func main() {
if err := loadEnvFiles(env("BACKFILL_ENV_FILES", os.Getenv("BACKFILL_ENV_FILE"))); err != nil {
envFiles := backfillEnvFiles(os.Getenv("BACKFILL_ENV_FILES"), os.Getenv("BACKFILL_ENV_FILE"), defaultBackfillEnvFiles)
if err := loadEnvFiles(envFiles); err != nil {
fail("load env file", err)
}
cfg, err := loadConfig()
@@ -121,18 +139,27 @@ func main() {
slog.Info("reset stats rows", "deleted", deleted)
}
if cfg.Method == "last_diff" {
aggregates, err := buildLastDiffAggregates(ctx, td, cfg)
aggregates, err := buildLastDiffAggregates(ctx, mysqlDB, td, cfg)
if err != nil {
fail("build last-diff aggregates", err)
}
fallbacks, err := addRealtimeLocationFallbackAggregates(ctx, mysqlDB, td, cfg, aggregates)
if err != nil {
fail("build realtime-location fallback aggregates", err)
}
var written int64
var normalized int
if !cfg.DryRun {
written, err = writeAggregates(ctx, mysqlDB, aggregates, 500)
if err != nil {
fail("write aggregates", err)
}
normalized, err = normalizeHistoricalPlatformSources(ctx, mysqlDB, cfg)
if err != nil {
fail("normalize historical platform sources", err)
}
}
slog.Info("stats backfill complete", "method", cfg.Method, "dry_run", cfg.DryRun, "aggregates", len(aggregates), "written", written)
slog.Info("stats backfill complete", "method", cfg.Method, "dry_run", cfg.DryRun, "aggregates", len(aggregates), "realtimeLocationFallbacks", fallbacks, "written", written, "platformSourcesNormalized", normalized)
return
}
@@ -234,6 +261,7 @@ func addSamples(aggregates map[string]*metricAgg, samples []stats.MetricSample)
Phone: sample.Phone,
DeviceID: sample.DeviceID,
SourceEndpoint: sample.SourceEndpoint,
PlatformName: sample.PlatformName,
FirstEventTime: sample.EventTime,
LatestEventTime: sample.EventTime,
QualityStatus: stats.QualityOK,
@@ -255,6 +283,9 @@ func addSamples(aggregates map[string]*metricAgg, samples []stats.MetricSample)
if strings.TrimSpace(sample.DeviceID) != "" {
agg.DeviceID = sample.DeviceID
}
if strings.TrimSpace(sample.PlatformName) != "" {
agg.PlatformName = sample.PlatformName
}
}
agg.Count++
}
@@ -271,6 +302,9 @@ func writeAggregates(ctx context.Context, db *sql.DB, aggregates map[string]*met
Protocol: agg.Protocol,
SourceIP: stats.NormalizeSourceIP(agg.SourceEndpoint),
SourceEndpoint: agg.SourceEndpoint,
SourceCode: agg.SourceCode,
PlatformName: agg.PlatformName,
SourceKind: agg.SourceKind,
}
if strings.TrimSpace(identity.SourceIP) == "" {
continue
@@ -285,7 +319,7 @@ func writeAggregates(ctx context.Context, db *sql.DB, aggregates map[string]*met
if err := stats.UpsertDataSource(ctx, db, identity, agg.LatestEventTime); err != nil {
return written, err
}
dailyKM := agg.LatestKM - agg.FirstKM
dailyKM := stats.DailyMileageFromDayBoundary(agg.FirstKM, agg.LatestKM)
candidate := stats.SourceMileageSample{
VIN: agg.VIN,
StatDate: agg.Date,
@@ -295,6 +329,7 @@ func writeAggregates(ctx context.Context, db *sql.DB, aggregates map[string]*met
SourceEndpoint: agg.SourceEndpoint,
Phone: agg.Phone,
DeviceID: agg.DeviceID,
PlatformName: agg.PlatformName,
FirstTotalKM: agg.FirstKM,
LatestTotalKM: agg.LatestKM,
DailyKM: dailyKM,
@@ -310,10 +345,7 @@ func writeAggregates(ctx context.Context, db *sql.DB, aggregates map[string]*met
if candidate.QualityReason == "" {
candidate.QualityReason = "same_source_previous_day"
}
if candidate.QualityStatus == stats.QualityOK && (candidate.DailyKM < 0 || candidate.DailyKM > maxTrustedDailyMileageKM) {
candidate.QualityStatus = stats.QualityInvalidDelta
candidate.QualityReason = "outside_daily_range"
}
stats.ApplyMileageQualityRules(&candidate)
if err := stats.UpsertSourceMileage(ctx, db, candidate); err != nil {
return written, err
}
@@ -325,6 +357,27 @@ func writeAggregates(ctx context.Context, db *sql.DB, aggregates map[string]*met
return written, nil
}
func normalizeHistoricalPlatformSources(ctx context.Context, db *sql.DB, cfg config) (int, error) {
dates, err := dateRange(cfg.DateFrom, cfg.DateTo)
if err != nil {
return 0, err
}
normalized := 0
for _, protocol := range cfg.Protocols {
if protocol != envelope.ProtocolJT808 {
continue
}
for _, date := range dates {
count, err := stats.NormalizePlatformSourceMileageForDate(ctx, db, date, protocol)
if err != nil {
return normalized, err
}
normalized += count
}
}
return normalized, nil
}
func clearBackfillTargetMileage(ctx context.Context, db *sql.DB, vin string, statDate string, protocol envelope.Protocol) error {
if db == nil || strings.TrimSpace(vin) == "" || strings.TrimSpace(statDate) == "" || strings.TrimSpace(string(protocol)) == "" {
return nil
@@ -337,47 +390,173 @@ func clearBackfillTargetMileage(ctx context.Context, db *sql.DB, vin string, sta
return err
}
func buildLastDiffAggregates(ctx context.Context, db *sql.DB, cfg config) (map[string]*metricAgg, error) {
func buildLastDiffAggregates(ctx context.Context, mysqlDB *sql.DB, tdDB *sql.DB, cfg config) (map[string]*metricAgg, error) {
targetDates, err := dateRange(cfg.DateFrom, cfg.DateTo)
if err != nil {
return nil, err
}
aggregates := map[string]*metricAgg{}
for _, protocol := range cfg.Protocols {
latestHistory := map[sourceHistoryID]dailySourceLast{}
preWindow, err := queryPreviousLastSourceRows(ctx, tdDB, cfg, protocol, targetDates[0])
if err != nil {
return nil, err
}
rememberLatestSourceRows(latestHistory, preWindow)
slog.Info("pre-window baseline loaded", "protocol", protocol, "before_date", targetDates[0], "vehicles", len(preWindow))
for _, date := range targetDates {
current, err := queryDailyLastSourceRows(ctx, db, cfg, protocol, date)
current, err := queryDailyLastSourceRows(ctx, tdDB, cfg, protocol, date)
if err != nil {
return nil, err
}
previous, err := queryPreviousLastSourceRows(ctx, db, cfg, protocol, date)
if err != nil {
return nil, err
}
slog.Info("daily last loaded", "protocol", protocol, "date", date, "vehicles", len(current), "previousVehicles", len(previous))
slog.Info("daily last loaded", "protocol", protocol, "date", date, "vehicles", len(current), "historicalSources", len(latestHistory))
for vin, currentSources := range current {
previousBySource := map[string]dailySourceLast{}
for _, previousRow := range previous[vin] {
previousBySource[previousRow.SourceKey] = previousRow
}
for _, currentRow := range currentSources {
key := vin + "|" + date + "|" + string(protocol) + "|" + currentRow.SourceKey
previousRow, hasPrevious := previousBySource[currentRow.SourceKey]
previousRow, hasPrevious, err := resolvePreviousSourceRow(ctx, mysqlDB, date, protocol, latestHistory, currentRow)
if err != nil {
return nil, err
}
aggregates[key] = aggregateFromDailySource(date, protocol, currentRow, previousRow, hasPrevious)
}
}
// Current-day last values become eligible only for later target dates.
rememberLatestSourceRows(latestHistory, current)
}
}
return aggregates, nil
}
func addRealtimeLocationFallbackAggregates(ctx context.Context, mysqlDB *sql.DB, tdDB *sql.DB, cfg config, aggregates map[string]*metricAgg) (int, error) {
if aggregates == nil {
return 0, fmt.Errorf("aggregates map is nil")
}
targetDates, err := dateRange(cfg.DateFrom, cfg.DateTo)
if err != nil {
return 0, err
}
var added int
for _, protocol := range cfg.Protocols {
if protocol != envelope.ProtocolYutongMQTT {
continue
}
latestHistory := map[sourceHistoryID]dailySourceLast{}
preWindow, err := queryPreviousLastSourceRows(ctx, tdDB, cfg, protocol, targetDates[0])
if err != nil {
return added, err
}
rememberLatestSourceRows(latestHistory, preWindow)
aggregateRowsByDate := indexAggregateSourceRowsByDate(aggregates, protocol)
for _, date := range targetDates {
current, err := queryRealtimeLocationLastRows(ctx, mysqlDB, cfg, protocol, date)
if err != nil {
return added, err
}
for vin, currentSources := range current {
for _, currentRow := range currentSources {
key := vin + "|" + date + "|" + string(protocol) + "|" + currentRow.SourceKey
if _, exists := aggregates[key]; exists {
continue
}
previousRow, hasPrevious, err := resolvePreviousSourceRow(ctx, mysqlDB, date, protocol, latestHistory, currentRow)
if err != nil {
return added, err
}
agg := aggregateFromDailySource(date, protocol, currentRow, previousRow, hasPrevious)
if hasPrevious {
agg.QualityReason = "realtime_location_fallback_historical_baseline"
} else {
agg.QualityReason = "realtime_location_fallback_current_day_first_baseline"
}
aggregates[key] = agg
added++
}
}
rememberLatestSourceRows(latestHistory, aggregateRowsByDate[date])
rememberLatestSourceRows(latestHistory, current)
if len(current) > 0 {
slog.Info("realtime location fallback loaded", "protocol", protocol, "date", date, "vehicles", len(current), "added", added)
}
}
}
return added, nil
}
func indexAggregateSourceRowsByDate(aggregates map[string]*metricAgg, protocol envelope.Protocol) map[string]map[string][]dailySourceLast {
indexed := map[string]map[string][]dailySourceLast{}
for _, agg := range aggregates {
if agg == nil || agg.Protocol != protocol || strings.TrimSpace(agg.Date) == "" {
continue
}
rows := indexed[agg.Date]
if rows == nil {
rows = map[string][]dailySourceLast{}
indexed[agg.Date] = rows
}
rows[agg.VIN] = append(rows[agg.VIN], dailySourceLast{
VIN: agg.VIN,
SourceKey: agg.SourceKey,
Phone: agg.Phone,
DeviceID: agg.DeviceID,
SourceEndpoint: agg.SourceEndpoint,
SourceCode: agg.SourceCode,
PlatformName: agg.PlatformName,
SourceKind: agg.SourceKind,
FirstTS: agg.LatestEventTime,
TS: agg.LatestEventTime,
FirstTotalKM: agg.LatestKM,
TotalKM: agg.LatestKM,
RawSampleCount: agg.Count,
})
}
return indexed
}
func rememberLatestSourceRows(history map[sourceHistoryID]dailySourceLast, rows map[string][]dailySourceLast) {
for vin, sourceRows := range rows {
for _, row := range sourceRows {
rowVIN := strings.TrimSpace(row.VIN)
if rowVIN == "" {
rowVIN = strings.TrimSpace(vin)
row.VIN = rowVIN
}
id := sourceHistoryID{VIN: rowVIN, SourceKey: strings.TrimSpace(row.SourceKey)}
if id.VIN == "" || id.SourceKey == "" || row.TS.IsZero() {
continue
}
if existing, ok := history[id]; !ok || row.TS.After(existing.TS) {
history[id] = row
}
}
}
}
func latestHistoricalSourceRow(history map[sourceHistoryID]dailySourceLast, current dailySourceLast) (dailySourceLast, bool) {
id := sourceHistoryID{
VIN: strings.TrimSpace(current.VIN),
SourceKey: strings.TrimSpace(current.SourceKey),
}
previous, ok := history[id]
return previous, ok && previous.TS.Before(current.TS)
}
func resolvePreviousSourceRow(ctx context.Context, db *sql.DB, date string, protocol envelope.Protocol, history map[sourceHistoryID]dailySourceLast, current dailySourceLast) (dailySourceLast, bool, error) {
if previous, ok := latestHistoricalSourceRow(history, current); ok {
return previous, true, nil
}
return queryDurablePreviousSourceRow(ctx, db, date, protocol, current)
}
func aggregateFromDailySource(date string, protocol envelope.Protocol, current dailySourceLast, previous dailySourceLast, hasPrevious bool) *metricAgg {
firstKM := current.FirstTotalKM
firstEventTime := current.FirstTS
qualityReason := "current_day_first_sample"
qualityStatus := stats.QualityOK
qualityReason := stats.QualityReasonCurrentDayFirst
if hasPrevious {
firstKM = previous.TotalKM
firstEventTime = previous.TS
qualityReason = "historical_source_baseline"
qualityStatus = stats.QualityOK
qualityReason = stats.QualityReasonHistorical
}
if firstEventTime.IsZero() {
firstEventTime = current.TS
@@ -386,7 +565,7 @@ func aggregateFromDailySource(date string, protocol envelope.Protocol, current d
if count <= 0 {
count = 1
}
return &metricAgg{
agg := &metricAgg{
VIN: current.VIN,
Date: date,
Protocol: protocol,
@@ -397,11 +576,43 @@ func aggregateFromDailySource(date string, protocol envelope.Protocol, current d
Phone: current.Phone,
DeviceID: current.DeviceID,
SourceEndpoint: current.SourceEndpoint,
SourceCode: current.SourceCode,
PlatformName: current.PlatformName,
SourceKind: current.SourceKind,
FirstEventTime: firstEventTime,
LatestEventTime: current.TS,
QualityStatus: stats.QualityOK,
QualityStatus: qualityStatus,
QualityReason: qualityReason,
}
candidate := stats.SourceMileageSample{
FirstTotalKM: agg.FirstKM,
LatestTotalKM: agg.LatestKM,
DailyKM: stats.DailyMileageFromDayBoundary(agg.FirstKM, agg.LatestKM),
FirstEventTime: agg.FirstEventTime,
LatestEventTime: agg.LatestEventTime,
QualityStatus: agg.QualityStatus,
QualityReason: agg.QualityReason,
}
stats.ApplyMileageQualityRules(&candidate)
agg.QualityStatus = candidate.QualityStatus
agg.QualityReason = candidate.QualityReason
return agg
}
func queryDurablePreviousSourceRow(ctx context.Context, db *sql.DB, date string, protocol envelope.Protocol, current dailySourceLast) (dailySourceLast, bool, error) {
if db == nil {
return dailySourceLast{}, false, nil
}
totalKM, eventTime, found, err := stats.LookupLatestSourceBaselineBefore(ctx, db, current.VIN, date, protocol, current.SourceKey)
if err != nil || !found {
return dailySourceLast{}, found, err
}
previous := current
previous.FirstTS = eventTime
previous.TS = eventTime
previous.FirstTotalKM = totalKM
previous.TotalKM = totalKM
return previous, true, nil
}
type trustedChoice struct {
@@ -409,8 +620,6 @@ type trustedChoice struct {
previous dailySourceLast
}
const maxTrustedDailyMileageKM = 1000
func chooseTrustedSource(current []dailySourceLast, previous []dailySourceLast) (trustedChoice, bool) {
previousBySource := map[string]dailySourceLast{}
for _, row := range previous {
@@ -423,8 +632,8 @@ func chooseTrustedSource(current []dailySourceLast, previous []dailySourceLast)
if !ok {
continue
}
delta := currentRow.TotalKM - previousRow.TotalKM
if delta < 0 || delta > maxTrustedDailyMileageKM {
delta, ok, _ := stats.NormalizeDailyMileageDeltaForWindow(stats.DailyMileageFromDayBoundary(previousRow.TotalKM, currentRow.TotalKM), previousRow.TS, currentRow.TS)
if !ok {
continue
}
if chosen.current.SourceKey == "" || delta < chosenDelta {
@@ -436,19 +645,18 @@ func chooseTrustedSource(current []dailySourceLast, previous []dailySourceLast)
}
func queryDailyLastSourceRows(ctx context.Context, db *sql.DB, cfg config, protocol envelope.Protocol, date string) (map[string][]dailySourceLast, error) {
where := []string{
fmt.Sprintf("ts >= '%s 00:00:00'", quote(date)),
fmt.Sprintf("ts < '%s 00:00:00'", quote(nextDate(date))),
where := backfillTimePredicates(cfg, date, nextDate(date))
where = append(where,
"parse_status = 'OK'",
"vin IS NOT NULL",
"vin <> ''",
fmt.Sprintf("protocol = '%s'", quote(string(protocol))),
realtimeMileageFramePredicate(),
}
)
if predicate := mileageBearingFramePredicate(protocol); predicate != "" {
where = append(where, predicate)
}
sqlText := fmt.Sprintf(`SELECT vin, phone, device_id, source_endpoint, FIRST(ts), FIRST(parsed_json), LAST(ts), LAST(parsed_json), COUNT(*)
sqlText := fmt.Sprintf(`SELECT vin, phone, device_id, source_endpoint, FIRST(event_time), FIRST(parsed_json), LAST(event_time), LAST(parsed_json), COUNT(*)
FROM %s.raw_frames
WHERE %s
GROUP BY vin, phone, device_id, source_endpoint`, ident(cfg.TDengineDatabase), strings.Join(where, " AND "))
@@ -456,24 +664,103 @@ GROUP BY vin, phone, device_id, source_endpoint`, ident(cfg.TDengineDatabase), s
}
func queryPreviousLastSourceRows(ctx context.Context, db *sql.DB, cfg config, protocol envelope.Protocol, date string) (map[string][]dailySourceLast, error) {
where := []string{
fmt.Sprintf("ts < '%s 00:00:00'", quote(date)),
// The baseline is the nearest earlier sample, not necessarily yesterday's.
// LAST aggregates the complete pre-window history once per source so empty
// calendar days do not make a backfill fall back to the current day's first
// sample.
where := backfillBeforePredicates(date)
where = append(where,
"parse_status = 'OK'",
"vin IS NOT NULL",
"vin <> ''",
fmt.Sprintf("protocol = '%s'", quote(string(protocol))),
realtimeMileageFramePredicate(),
}
)
if predicate := mileageBearingFramePredicate(protocol); predicate != "" {
where = append(where, predicate)
}
sqlText := fmt.Sprintf(`SELECT vin, phone, device_id, source_endpoint, LAST(ts), LAST(parsed_json), COUNT(*)
sqlText := fmt.Sprintf(`SELECT vin, phone, device_id, source_endpoint, LAST(event_time), LAST(parsed_json), COUNT(*)
FROM %s.raw_frames
WHERE %s
GROUP BY vin, phone, device_id, source_endpoint`, ident(cfg.TDengineDatabase), strings.Join(where, " AND "))
return querySourceRows(ctx, db, cfg, protocol, sqlText, false)
}
func backfillBeforePredicates(eventDateExclusive string) []string {
return []string{
fmt.Sprintf("event_time < '%s 00:00:00'", quote(eventDateExclusive)),
}
}
func queryRealtimeLocationLastRows(ctx context.Context, db *sql.DB, cfg config, protocol envelope.Protocol, date string) (map[string][]dailySourceLast, error) {
if db == nil {
return nil, nil
}
loc := cfg.Location
if loc == nil {
loc = time.FixedZone("Asia/Shanghai", 8*3600)
}
sqlText := `SELECT l.vin, COALESCE(NULLIF(s.peer, ''), ''), l.total_mileage_event_time, l.total_mileage_km
FROM vehicle_realtime_location l
LEFT JOIN vehicle_realtime_snapshot s ON s.protocol = l.protocol AND s.vin = l.vin
WHERE l.protocol = ?
AND l.vin IS NOT NULL AND l.vin <> ''
AND l.total_mileage_km IS NOT NULL AND l.total_mileage_km > 0
AND l.total_mileage_event_time >= ?
AND l.total_mileage_event_time < DATE_ADD(?, INTERVAL 1 DAY)`
rows, err := db.QueryContext(ctx, sqlText, string(protocol), date, date)
if err != nil {
return nil, err
}
defer rows.Close()
out := map[string][]dailySourceLast{}
for rows.Next() {
var vin string
var peer sql.NullString
var eventTime time.Time
var totalKM float64
if err := rows.Scan(&vin, &peer, &eventTime, &totalKM); err != nil {
return nil, err
}
vin = strings.TrimSpace(vin)
if vin == "" || totalKM <= 0 {
continue
}
sourceEndpoint := strings.TrimSpace(peer.String)
if sourceEndpoint == "" && protocol == envelope.ProtocolYutongMQTT {
sourceEndpoint = "mqtt://yutong/realtime-location"
}
deviceID := ""
if protocol == envelope.ProtocolYutongMQTT {
deviceID = vin
}
sourceCode, platformName, sourceKind := knownPlatformSourceMetadata(protocol, sourceEndpoint)
sourceKey := normalizedSourceKeyForSource(string(protocol), "", deviceID, sourceEndpoint, sourceKind, sourceCode)
if sourceKey == "" {
continue
}
row := dailySourceLast{
VIN: vin,
SourceKey: sourceKey,
DeviceID: deviceID,
SourceEndpoint: sourceEndpoint,
SourceCode: sourceCode,
PlatformName: platformName,
SourceKind: sourceKind,
FirstTS: eventTime.In(loc),
TS: eventTime.In(loc),
FirstTotalKM: totalKM,
TotalKM: totalKM,
RawSampleCount: 1,
}
out[vin] = append(out[vin], row)
}
if err := rows.Err(); err != nil {
return nil, err
}
return out, nil
}
func querySourceRows(ctx context.Context, db *sql.DB, cfg config, protocol envelope.Protocol, sqlText string, includeFirst bool) (map[string][]dailySourceLast, error) {
rows, err := db.QueryContext(ctx, sqlText)
if err != nil {
@@ -512,7 +799,8 @@ func querySourceRows(ctx context.Context, db *sql.DB, cfg config, protocol envel
firstTotalKM = parsedFirst
}
}
sourceKey := normalizedSourceKey(string(protocol), phone, deviceID, sourceEndpoint)
sourceCode, platformName, sourceKind := knownPlatformSourceMetadata(protocol, sourceEndpoint)
sourceKey := normalizedSourceKeyForSource(string(protocol), phone, deviceID, sourceEndpoint, sourceKind, sourceCode)
if sourceKey == "" {
continue
}
@@ -522,8 +810,11 @@ func querySourceRows(ctx context.Context, db *sql.DB, cfg config, protocol envel
Phone: strings.TrimSpace(phone),
DeviceID: strings.TrimSpace(deviceID),
SourceEndpoint: strings.TrimSpace(sourceEndpoint),
SourceCode: sourceCode,
PlatformName: platformName,
SourceKind: sourceKind,
FirstTS: firstTS.In(cfg.Location),
TS: ts,
TS: ts.In(cfg.Location),
FirstTotalKM: firstTotalKM,
TotalKM: latestTotalKM,
RawSampleCount: rawSampleCount,
@@ -562,8 +853,19 @@ func mileageFromParsed(protocol envelope.Protocol, vin string, parsedJSON string
}
func normalizedSourceKey(protocol string, phone string, deviceID string, endpoint string) string {
return normalizedSourceKeyForSource(protocol, phone, deviceID, endpoint, "", "")
}
func normalizedSourceKeyForSource(protocol string, phone string, deviceID string, endpoint string, sourceKind string, sourceCode string) string {
sourceIP := stats.NormalizeSourceIP(endpoint)
return stats.SourceKey(envelope.Protocol(protocol), phone, deviceID, sourceIP)
return stats.SourceKeyForSource(envelope.Protocol(protocol), phone, deviceID, sourceIP, sourceKind, sourceCode)
}
func knownPlatformSourceMetadata(protocol envelope.Protocol, endpoint string) (sourceCode string, platformName string, sourceKind string) {
if protocol == envelope.ProtocolYutongMQTT && stats.NormalizeSourceIP(endpoint) == "mqtt" {
return "yutong", "宇通", "PLATFORM"
}
return "", "", ""
}
func fieldsForStats(protocol envelope.Protocol, vin string, text string) map[string]any {
@@ -574,7 +876,7 @@ func fieldsForStats(protocol envelope.Protocol, vin string, text string) map[str
}
parsed := map[string]any{}
if err := json.Unmarshal([]byte(text), &parsed); err == nil && len(parsed) > 0 {
if flattened, _, ok := realtime.ParsedFieldsForEnvelope(envelope.FrameEnvelope{
if flattened, _, ok := realtime.ComputeParsedFieldsForEnvelope(envelope.FrameEnvelope{
Protocol: protocol,
VIN: vin,
Parsed: parsed,
@@ -604,7 +906,12 @@ func mileageKeys(protocol envelope.Protocol) []string {
case envelope.ProtocolJT808:
return []string{"jt808.location.total_mileage_km"}
case envelope.ProtocolYutongMQTT:
return []string{"yutong_mqtt.data.total_mileage", "yutong_mqtt.root.data.total_mileage"}
return []string{
"yutong_mqtt.data.total_mileage_km",
"yutong_mqtt.root.data.total_mileage_km",
"yutong_mqtt.data.total_mileage",
"yutong_mqtt.root.data.total_mileage",
}
default:
return nil
}
@@ -624,38 +931,67 @@ func loadConfig() (config, error) {
if err != nil {
loc = time.FixedZone("Asia/Shanghai", 8*3600)
}
dateTo := env("BACKFILL_DATE_TO", time.Now().In(loc).Format("2006-01-02"))
dateFrom := env("BACKFILL_DATE_FROM", dateTo)
dateFrom, dateTo := resolveBackfillDateRange(time.Now(), loc)
protocols, err := parseProtocols(env("BACKFILL_PROTOCOLS", "GB32960,JT808,YUTONG_MQTT"))
if err != nil {
return config{}, err
}
return config{
MySQLDSN: env("MYSQL_DSN", ""),
TDengineDriver: env("TDENGINE_DRIVER", "taosWS"),
TDengineDSN: env("TDENGINE_DSN", ""),
TDengineDatabase: env("TDENGINE_DATABASE", history.DefaultDatabase),
DateFrom: dateFrom,
DateTo: dateTo,
Protocols: protocols,
Method: env("BACKFILL_METHOD", "last_diff"),
Limit: envInt("BACKFILL_LIMIT", 0),
DryRun: envBool("BACKFILL_DRY_RUN", true),
Reset: envBool("BACKFILL_RESET", false),
Debug: envBool("BACKFILL_DEBUG", false),
ProgressEvery: int64(envInt("BACKFILL_PROGRESS_EVERY", 100000)),
Location: loc,
MySQLDSN: env("MYSQL_DSN", ""),
TDengineDriver: env("TDENGINE_DRIVER", "taosWS"),
TDengineDSN: env("TDENGINE_DSN", ""),
TDengineDatabase: env("TDENGINE_DATABASE", history.DefaultDatabase),
DateFrom: dateFrom,
DateTo: dateTo,
Protocols: protocols,
Method: env("BACKFILL_METHOD", "last_diff"),
Limit: envInt("BACKFILL_LIMIT", 0),
DryRun: envBool("BACKFILL_DRY_RUN", true),
Reset: envBool("BACKFILL_RESET", false),
Debug: envBool("BACKFILL_DEBUG", false),
EventTimeFullScan: envBool("BACKFILL_EVENT_TIME_FULL_SCAN", false),
ProgressEvery: int64(envInt("BACKFILL_PROGRESS_EVERY", 100000)),
Location: loc,
}, nil
}
func resolveBackfillDateRange(now time.Time, loc *time.Location) (string, string) {
if loc == nil {
loc = time.FixedZone("Asia/Shanghai", 8*3600)
}
explicitFrom := strings.TrimSpace(os.Getenv("BACKFILL_DATE_FROM"))
explicitTo := strings.TrimSpace(os.Getenv("BACKFILL_DATE_TO"))
if explicitTo != "" || explicitFrom != "" {
dateTo := explicitTo
if dateTo == "" {
dateTo = now.In(loc).Format("2006-01-02")
}
dateFrom := explicitFrom
if dateFrom == "" {
dateFrom = dateTo
}
return dateFrom, dateTo
}
daysBack := envInt("BACKFILL_DAYS_BACK", 0)
if daysBack < 0 {
daysBack = 0
}
windowDays := envInt("BACKFILL_WINDOW_DAYS", 1)
if windowDays < 1 {
windowDays = 1
}
dateToTime := now.In(loc).AddDate(0, 0, -daysBack)
dateFromTime := dateToTime.AddDate(0, 0, -(windowDays - 1))
return dateFromTime.Format("2006-01-02"), dateToTime.Format("2006-01-02")
}
func queryRawFrames(ctx context.Context, db *sql.DB, cfg config) (*sql.Rows, error) {
where := []string{
fmt.Sprintf("ts >= '%s 00:00:00'", quote(cfg.DateFrom)),
fmt.Sprintf("ts < '%s 00:00:00'", quote(nextDate(cfg.DateTo))),
where := backfillTimePredicates(cfg, cfg.DateFrom, nextDate(cfg.DateTo))
where = append(where,
"parse_status = 'OK'",
"vin IS NOT NULL",
"vin <> ''",
}
)
if len(cfg.Protocols) > 0 {
quoted := make([]string, 0, len(cfg.Protocols))
for _, protocol := range cfg.Protocols {
@@ -667,7 +1003,7 @@ func queryRawFrames(ctx context.Context, db *sql.DB, cfg config) (*sql.Rows, err
sqlText := fmt.Sprintf(`SELECT protocol, vin, phone, device_id, source_endpoint, event_id, message_id, event_time, received_at, parsed_json
FROM %s.raw_frames
WHERE %s
ORDER BY ts ASC`, ident(cfg.TDengineDatabase), strings.Join(where, " AND "))
ORDER BY event_time ASC, ts ASC`, ident(cfg.TDengineDatabase), strings.Join(where, " AND "))
if cfg.Limit > 0 {
sqlText += fmt.Sprintf(" LIMIT %d", cfg.Limit)
}
@@ -675,6 +1011,23 @@ ORDER BY ts ASC`, ident(cfg.TDengineDatabase), strings.Join(where, " AND "))
return db.QueryContext(ctx, sqlText)
}
func backfillTimePredicates(cfg config, eventDateFrom string, eventDateToExclusive string) []string {
where := make([]string, 0, 4)
if !cfg.EventTimeFullScan {
// ts is the TDengine primary timestamp and may be adjusted slightly to keep
// rows unique. Use it only as a broad index-friendly window; event_time is
// the protocol business boundary used for the final natural-day result.
where = append(where,
fmt.Sprintf("ts >= '%s 00:00:00'", quote(previousDate(eventDateFrom))),
fmt.Sprintf("ts < '%s 00:00:00'", quote(nextDate(eventDateToExclusive))),
)
}
return append(where,
fmt.Sprintf("event_time >= '%s 00:00:00'", quote(eventDateFrom)),
fmt.Sprintf("event_time < '%s 00:00:00'", quote(eventDateToExclusive)),
)
}
func realtimeMileageFramePredicate() string {
return `(
(protocol = 'GB32960' AND message_id IN (2,3))
@@ -684,17 +1037,39 @@ func realtimeMileageFramePredicate() string {
}
func mileageBearingFramePredicate(protocol envelope.Protocol) string {
keys := mileageKeys(protocol)
if len(keys) == 0 {
tokens := mileageSearchTokens(protocol)
if len(tokens) == 0 {
return ""
}
conditions := make([]string, 0, len(keys))
for _, key := range keys {
conditions = append(conditions, fmt.Sprintf("parsed_json LIKE '%%%s%%'", quote(key)))
conditions := make([]string, 0, len(tokens))
for _, token := range tokens {
conditions = append(conditions, fmt.Sprintf("parsed_json LIKE '%%%s%%'", quote(token)))
}
return "(" + strings.Join(conditions, " OR ") + ")"
}
func mileageSearchTokens(protocol envelope.Protocol) []string {
tokens := append([]string{}, mileageKeys(protocol)...)
switch protocol {
case envelope.ProtocolYutongMQTT:
tokens = append(tokens, "TOTAL_MILEAGE", "totalMileage", "total_mileage_km")
}
seen := make(map[string]struct{}, len(tokens))
out := make([]string, 0, len(tokens))
for _, token := range tokens {
token = strings.TrimSpace(token)
if token == "" {
continue
}
if _, ok := seen[token]; ok {
continue
}
seen[token] = struct{}{}
out = append(out, token)
}
return out
}
func scanRawFrame(rows *sql.Rows) (rawFrameRow, error) {
var protocol, vin, phone, deviceID, sourceEndpoint, eventID, parsed string
var messageID int64
@@ -804,6 +1179,26 @@ func loadEnvFiles(paths string) error {
return nil
}
func backfillEnvFiles(explicitFiles string, explicitFile string, defaults []string) string {
if value := strings.TrimSpace(explicitFiles); value != "" {
return value
}
if value := strings.TrimSpace(explicitFile); value != "" {
return value
}
var existing []string
for _, path := range defaults {
path = strings.TrimSpace(path)
if path == "" {
continue
}
if _, err := os.Stat(path); err == nil {
existing = append(existing, path)
}
}
return strings.Join(existing, ",")
}
func loadEnvFile(path string) error {
path = strings.TrimSpace(path)
if path == "" {

548
go/vehicle-gateway/cmd/stats-backfill/main_test.go
View File

@@ -2,6 +2,9 @@ package main
import (
"context"
"os"
"path/filepath"
"strings"
"testing"
"time"
@@ -47,6 +50,58 @@ func TestChooseTrustedSourceKeepsContinuingSourceAndRejectsNewJump(t *testing.T)
}
}
func TestChooseTrustedSourceAcceptsSmallNegativeMileageJitter(t *testing.T) {
previous := []dailySourceLast{{
VIN: "LB9A32A28R0LS1574",
SourceKey: normalizedSourceKey("JT808", "64115156034", "", "115.159.85.149:53330"),
Phone: "64115156034",
SourceEndpoint: "115.159.85.149:53330",
TotalKM: 15355.4,
}}
current := []dailySourceLast{{
VIN: "LB9A32A28R0LS1574",
SourceKey: normalizedSourceKey("JT808", "64115156034", "", "115.159.85.149:53338"),
Phone: "64115156034",
SourceEndpoint: "115.159.85.149:53338",
TotalKM: 15355.3,
}}
chosen, ok := chooseTrustedSource(current, previous)
if !ok {
t.Fatal("chooseTrustedSource() should accept tiny negative mileage jitter")
}
if chosen.current.SourceKey != current[0].SourceKey {
t.Fatalf("chosen source = %q", chosen.current.SourceKey)
}
}
func TestChooseTrustedSourceAcceptsPlausibleMultiDayFallbackDelta(t *testing.T) {
loc := time.FixedZone("Asia/Shanghai", 8*3600)
sourceKey := normalizedSourceKey("GB32960", "", "", "8.134.95.166:37720")
previous := []dailySourceLast{{
VIN: "LNXNEGRR1SR319498",
SourceKey: sourceKey,
SourceEndpoint: "8.134.95.166:37720",
TS: time.Date(2026, 7, 4, 11, 7, 58, 0, loc),
TotalKM: 8832.1,
}}
current := []dailySourceLast{{
VIN: "LNXNEGRR1SR319498",
SourceKey: sourceKey,
SourceEndpoint: "8.134.95.166:37720",
TS: time.Date(2026, 7, 12, 2, 52, 47, 0, loc),
TotalKM: 16665.6,
}}
chosen, ok := chooseTrustedSource(current, previous)
if !ok {
t.Fatal("chooseTrustedSource() should accept delta within the historical baseline window")
}
if delta := chosen.current.TotalKM - chosen.previous.TotalKM; delta < 7833.4 || delta > 7833.6 {
t.Fatalf("delta = %v, want historical gap delta", delta)
}
}
func TestDailySourceLastBuildsCandidateKeysBySourceIP(t *testing.T) {
sourceA := dailySourceLast{
VIN: "LA9GG64L7PBAF4001",
@@ -96,7 +151,7 @@ func TestAggregateFromDailySourceUsesOlderHistoricalBaseline(t *testing.T) {
if agg.FirstEventTime != previous.TS || agg.LatestEventTime != current.TS {
t.Fatalf("event range = %v -> %v", agg.FirstEventTime, agg.LatestEventTime)
}
if agg.QualityStatus != stats.QualityOK || agg.QualityReason != "historical_source_baseline" {
if agg.QualityStatus != stats.QualityOK || agg.QualityReason != stats.QualityReasonHistorical {
t.Fatalf("quality = %s/%s", agg.QualityStatus, agg.QualityReason)
}
if agg.Count != 15 {
@@ -104,7 +159,40 @@ func TestAggregateFromDailySourceUsesOlderHistoricalBaseline(t *testing.T) {
}
}
func TestAggregateFromDailySourceUsesCurrentFirstSampleWithoutHistory(t *testing.T) {
func TestAggregateFromDailySourceRejectsHistoricalBaselineJump(t *testing.T) {
loc := time.FixedZone("Asia/Shanghai", 8*3600)
current := dailySourceLast{
VIN: "LNXNEGRR6SR319464",
SourceKey: normalizedSourceKey("GB32960", "", "", "8.134.95.166:49206"),
SourceEndpoint: "8.134.95.166:49206",
FirstTS: time.Date(2026, 7, 12, 8, 5, 42, 0, loc),
TS: time.Date(2026, 7, 12, 10, 44, 56, 0, loc),
FirstTotalKM: 28004.2,
TotalKM: 40009.7,
RawSampleCount: 1938,
}
previous := dailySourceLast{
VIN: current.VIN,
SourceKey: current.SourceKey,
SourceEndpoint: current.SourceEndpoint,
TS: time.Date(2026, 7, 3, 19, 0, 39, 0, loc),
TotalKM: 10009.7,
}
agg := aggregateFromDailySource("2026-07-12", envelope.ProtocolGB32960, current, previous, true)
if agg.FirstKM != previous.TotalKM || agg.LatestKM != current.TotalKM {
t.Fatalf("km range = %v -> %v", agg.FirstKM, agg.LatestKM)
}
if !agg.FirstEventTime.Equal(previous.TS) || !agg.LatestEventTime.Equal(current.TS) {
t.Fatalf("event range = %v -> %v", agg.FirstEventTime, agg.LatestEventTime)
}
if agg.QualityStatus != stats.QualityInvalidDelta || agg.QualityReason != "outside_daily_range" {
t.Fatalf("quality = %s/%s", agg.QualityStatus, agg.QualityReason)
}
}
func TestAggregateFromDailySourceUsesCurrentDayFirstWhenHistoryIsEmpty(t *testing.T) {
current := dailySourceLast{
VIN: "LMRKH9AC2R1004087",
SourceKey: normalizedSourceKey("YUTONG_MQTT", "", "LMRKH9AC2R1004087", "mqtt://yutong/ytforward/shln/3"),
@@ -125,11 +213,264 @@ func TestAggregateFromDailySourceUsesCurrentFirstSampleWithoutHistory(t *testing
if agg.FirstEventTime != current.FirstTS || agg.LatestEventTime != current.TS {
t.Fatalf("event range = %v -> %v", agg.FirstEventTime, agg.LatestEventTime)
}
if agg.QualityStatus != stats.QualityOK || agg.QualityReason != "current_day_first_sample" {
if agg.QualityStatus != stats.QualityOK || agg.QualityReason != stats.QualityReasonCurrentDayFirst {
t.Fatalf("quality = %s/%s", agg.QualityStatus, agg.QualityReason)
}
}
func TestBuildLastDiffAggregatesCarriesNearestHistoryAcrossEmptyDays(t *testing.T) {
tdDB, mock, err := sqlmock.New()
if err != nil {
t.Fatalf("sqlmock.New() error = %v", err)
}
defer tdDB.Close()
mock.MatchExpectationsInOrder(true)
loc := time.FixedZone("Asia/Shanghai", 8*3600)
vin := "LMRKH9AC2R1004087"
endpoint := "mqtt://yutong/ytforward/shln/3"
dayOneTS := time.Date(2026, 7, 9, 23, 50, 0, 0, loc)
dayThreeTS := time.Date(2026, 7, 11, 17, 30, 0, 0, loc)
currentRows := func() *sqlmock.Rows {
return sqlmock.NewRows([]string{
"vin", "phone", "device_id", "source_endpoint", "FIRST(event_time)", "FIRST(parsed_json)", "LAST(event_time)", "LAST(parsed_json)", "COUNT(*)",
})
}
previousRows := func() *sqlmock.Rows {
return sqlmock.NewRows([]string{
"vin", "phone", "device_id", "source_endpoint", "LAST(event_time)", "LAST(parsed_json)", "COUNT(*)",
})
}
mock.ExpectQuery("(?s)SELECT vin.*LAST\\(event_time\\).*event_time < '2026-07-09 00:00:00'.*protocol = 'YUTONG_MQTT'").
WillReturnRows(previousRows())
mock.ExpectQuery("(?s)SELECT vin.*FIRST\\(event_time\\).*event_time >= '2026-07-09 00:00:00'.*protocol = 'YUTONG_MQTT'").
WillReturnRows(currentRows().AddRow(vin, "", vin, endpoint, dayOneTS, `{"data":{"TOTAL_MILEAGE":100000}}`, dayOneTS, `{"data":{"TOTAL_MILEAGE":100000}}`, int64(4)))
mock.ExpectQuery("(?s)SELECT vin.*FIRST\\(event_time\\).*event_time >= '2026-07-10 00:00:00'.*protocol = 'YUTONG_MQTT'").
WillReturnRows(currentRows())
mock.ExpectQuery("(?s)SELECT vin.*FIRST\\(event_time\\).*event_time >= '2026-07-11 00:00:00'.*protocol = 'YUTONG_MQTT'").
WillReturnRows(currentRows().AddRow(vin, "", vin, endpoint, dayThreeTS, `{"data":{"TOTAL_MILEAGE":120000}}`, dayThreeTS, `{"data":{"TOTAL_MILEAGE":120000}}`, int64(6)))
aggregates, err := buildLastDiffAggregates(context.Background(), nil, tdDB, config{
TDengineDatabase: "lingniu_vehicle_ts",
DateFrom: "2026-07-09",
DateTo: "2026-07-11",
Protocols: []envelope.Protocol{envelope.ProtocolYutongMQTT},
Location: loc,
})
if err != nil {
t.Fatalf("buildLastDiffAggregates() error = %v", err)
}
sourceKey := stats.SourceKeyForSource(envelope.ProtocolYutongMQTT, "", vin, "mqtt", "PLATFORM", "yutong")
agg := aggregates[vin+"|2026-07-11|YUTONG_MQTT|"+sourceKey]
if agg == nil {
t.Fatalf("missing day-three aggregate; keys=%v", aggregateKeys(aggregates))
}
if agg.FirstKM != 100 || agg.LatestKM != 120 {
t.Fatalf("km range = %v -> %v, want 100 -> 120", agg.FirstKM, agg.LatestKM)
}
if !agg.FirstEventTime.Equal(dayOneTS) || agg.QualityReason != stats.QualityReasonHistorical {
t.Fatalf("baseline = %v reason=%q, want day-one historical baseline", agg.FirstEventTime, agg.QualityReason)
}
if err := mock.ExpectationsWereMet(); err != nil {
t.Fatalf("sql expectations: %v", err)
}
}
func aggregateKeys(aggregates map[string]*metricAgg) []string {
keys := make([]string, 0, len(aggregates))
for key := range aggregates {
keys = append(keys, key)
}
return keys
}
func TestQueryRealtimeLocationLastRowsBuildsYutongSourceFromPeer(t *testing.T) {
db, mock, err := sqlmock.New()
if err != nil {
t.Fatalf("sqlmock.New() error = %v", err)
}
defer db.Close()
loc := time.FixedZone("Asia/Shanghai", 8*3600)
eventTime := time.Date(2026, 7, 12, 5, 54, 50, 0, loc)
mock.ExpectQuery("(?s)FROM vehicle_realtime_location l.*LEFT JOIN vehicle_realtime_snapshot s").
WithArgs("YUTONG_MQTT", "2026-07-12", "2026-07-12").
WillReturnRows(sqlmock.NewRows([]string{"vin", "peer", "total_mileage_event_time", "total_mileage_km"}).
AddRow("LMRKH9AC0R1004086", "mqtt://yutong/ytforward/shln/4", eventTime, 11578.0))
rows, err := queryRealtimeLocationLastRows(context.Background(), db, config{Location: loc}, envelope.ProtocolYutongMQTT, "2026-07-12")
if err != nil {
t.Fatalf("queryRealtimeLocationLastRows() error = %v", err)
}
sourceRows := rows["LMRKH9AC0R1004086"]
if len(sourceRows) != 1 {
t.Fatalf("rows = %d, want 1", len(sourceRows))
}
row := sourceRows[0]
wantSourceKey := stats.SourceKeyForSource(envelope.ProtocolYutongMQTT, "", "LMRKH9AC0R1004086", "mqtt", "PLATFORM", "yutong")
if row.SourceKey != wantSourceKey {
t.Fatalf("source key = %q", row.SourceKey)
}
if row.SourceCode != "yutong" || row.PlatformName != "宇通" || row.SourceKind != "PLATFORM" {
t.Fatalf("source metadata = code:%q platform:%q kind:%q", row.SourceCode, row.PlatformName, row.SourceKind)
}
if row.TotalKM != 11578 || row.DeviceID != "LMRKH9AC0R1004086" || row.RawSampleCount != 1 {
t.Fatalf("unexpected realtime fallback row: %#v", row)
}
if err := mock.ExpectationsWereMet(); err != nil {
t.Fatalf("sql expectations: %v", err)
}
}
func TestAddRealtimeLocationFallbackAggregatesUsesHistoricalBaseline(t *testing.T) {
mysqlDB, mysqlMock, err := sqlmock.New()
if err != nil {
t.Fatalf("mysql sqlmock.New() error = %v", err)
}
defer mysqlDB.Close()
tdDB, tdMock, err := sqlmock.New()
if err != nil {
t.Fatalf("td sqlmock.New() error = %v", err)
}
defer tdDB.Close()
loc := time.FixedZone("Asia/Shanghai", 8*3600)
currentTS := time.Date(2026, 7, 12, 5, 54, 50, 0, loc)
previousTS := time.Date(2026, 7, 11, 23, 58, 0, 0, loc)
mysqlMock.ExpectQuery("(?s)FROM vehicle_realtime_location l.*l.protocol = \\?").
WithArgs("YUTONG_MQTT", "2026-07-12", "2026-07-12").
WillReturnRows(sqlmock.NewRows([]string{"vin", "peer", "total_mileage_event_time", "total_mileage_km"}).
AddRow("LMRKH9AC0R1004086", "mqtt://yutong/ytforward/shln/4", currentTS, 11578.0))
tdMock.ExpectQuery("(?s)FROM lingniu_vehicle_ts\\.raw_frames.*event_time < '2026-07-12 00:00:00'.*protocol = 'YUTONG_MQTT'.*TOTAL_MILEAGE").
WillReturnRows(sqlmock.NewRows([]string{
"vin", "phone", "device_id", "source_endpoint", "LAST(event_time)", "LAST(parsed_json)", "COUNT(*)",
}).AddRow(
"LMRKH9AC0R1004086",
"",
"LMRKH9AC0R1004086",
"mqtt://yutong/ytforward/shln/4",
previousTS,
`{"data":{"TOTAL_MILEAGE":11500000}}`,
int64(12),
))
aggregates := map[string]*metricAgg{}
added, err := addRealtimeLocationFallbackAggregates(context.Background(), mysqlDB, tdDB, config{
TDengineDatabase: "lingniu_vehicle_ts",
DateFrom: "2026-07-12",
DateTo: "2026-07-12",
Protocols: []envelope.Protocol{envelope.ProtocolYutongMQTT},
Location: loc,
}, aggregates)
if err != nil {
t.Fatalf("addRealtimeLocationFallbackAggregates() error = %v", err)
}
if added != 1 || len(aggregates) != 1 {
t.Fatalf("added=%d aggregates=%d", added, len(aggregates))
}
for _, agg := range aggregates {
if agg.FirstKM != 11500 || agg.LatestKM != 11578 {
t.Fatalf("km range = %v -> %v", agg.FirstKM, agg.LatestKM)
}
if agg.SourceKey != stats.SourceKeyForSource(envelope.ProtocolYutongMQTT, "", "LMRKH9AC0R1004086", "mqtt", "PLATFORM", "yutong") {
t.Fatalf("source key = %q", agg.SourceKey)
}
if agg.SourceCode != "yutong" || agg.PlatformName != "宇通" || agg.SourceKind != "PLATFORM" {
t.Fatalf("source metadata = code:%q platform:%q kind:%q", agg.SourceCode, agg.PlatformName, agg.SourceKind)
}
if agg.QualityReason != "realtime_location_fallback_historical_baseline" {
t.Fatalf("quality reason = %q", agg.QualityReason)
}
}
if err := mysqlMock.ExpectationsWereMet(); err != nil {
t.Fatalf("mysql sql expectations: %v", err)
}
if err := tdMock.ExpectationsWereMet(); err != nil {
t.Fatalf("td sql expectations: %v", err)
}
}
func TestAddRealtimeLocationFallbackReusesAggregateHistoryAcrossEmptyDays(t *testing.T) {
mysqlDB, mysqlMock, err := sqlmock.New()
if err != nil {
t.Fatalf("mysql sqlmock.New() error = %v", err)
}
defer mysqlDB.Close()
tdDB, tdMock, err := sqlmock.New()
if err != nil {
t.Fatalf("td sqlmock.New() error = %v", err)
}
defer tdDB.Close()
loc := time.FixedZone("Asia/Shanghai", 8*3600)
vin := "LMRKH9AC2R1004087"
endpoint := "mqtt://yutong/ytforward/shln/3"
sourceKey := stats.SourceKeyForSource(envelope.ProtocolYutongMQTT, "", vin, "mqtt", "PLATFORM", "yutong")
dayOneTS := time.Date(2026, 7, 9, 23, 50, 0, 0, loc)
dayThreeTS := time.Date(2026, 7, 11, 17, 30, 0, 0, loc)
tdMock.ExpectQuery("(?s)SELECT vin.*LAST\\(event_time\\).*event_time < '2026-07-09 00:00:00'.*protocol = 'YUTONG_MQTT'").
WillReturnRows(sqlmock.NewRows([]string{
"vin", "phone", "device_id", "source_endpoint", "LAST(event_time)", "LAST(parsed_json)", "COUNT(*)",
}))
for _, date := range []string{"2026-07-09", "2026-07-10"} {
mysqlMock.ExpectQuery("(?s)FROM vehicle_realtime_location l.*l.protocol = \\?").
WithArgs("YUTONG_MQTT", date, date).
WillReturnRows(sqlmock.NewRows([]string{"vin", "peer", "total_mileage_event_time", "total_mileage_km"}))
}
mysqlMock.ExpectQuery("(?s)FROM vehicle_realtime_location l.*l.protocol = \\?").
WithArgs("YUTONG_MQTT", "2026-07-11", "2026-07-11").
WillReturnRows(sqlmock.NewRows([]string{"vin", "peer", "total_mileage_event_time", "total_mileage_km"}).
AddRow(vin, endpoint, dayThreeTS, 120.0))
aggregates := map[string]*metricAgg{
vin + "|2026-07-09|YUTONG_MQTT|" + sourceKey: {
VIN: vin,
Date: "2026-07-09",
Protocol: envelope.ProtocolYutongMQTT,
LatestKM: 100,
Count: 4,
SourceKey: sourceKey,
DeviceID: vin,
SourceEndpoint: endpoint,
SourceCode: "yutong",
PlatformName: "宇通",
SourceKind: "PLATFORM",
LatestEventTime: dayOneTS,
},
}
added, err := addRealtimeLocationFallbackAggregates(context.Background(), mysqlDB, tdDB, config{
TDengineDatabase: "lingniu_vehicle_ts",
DateFrom: "2026-07-09",
DateTo: "2026-07-11",
Protocols: []envelope.Protocol{envelope.ProtocolYutongMQTT},
Location: loc,
}, aggregates)
if err != nil {
t.Fatalf("addRealtimeLocationFallbackAggregates() error = %v", err)
}
if added != 1 || len(aggregates) != 2 {
t.Fatalf("added=%d aggregates=%d, want 1 and 2", added, len(aggregates))
}
agg := aggregates[vin+"|2026-07-11|YUTONG_MQTT|"+sourceKey]
if agg == nil {
t.Fatal("missing realtime-location fallback aggregate")
}
if agg.FirstKM != 100 || agg.LatestKM != 120 || !agg.FirstEventTime.Equal(dayOneTS) {
t.Fatalf("fallback range = %v@%v -> %v, want 100@day-one -> 120", agg.FirstKM, agg.FirstEventTime, agg.LatestKM)
}
if agg.QualityReason != "realtime_location_fallback_historical_baseline" {
t.Fatalf("quality reason = %q", agg.QualityReason)
}
if err := mysqlMock.ExpectationsWereMet(); err != nil {
t.Fatalf("mysql sql expectations: %v", err)
}
if err := tdMock.ExpectationsWereMet(); err != nil {
t.Fatalf("td sql expectations: %v", err)
}
}
func TestQueryDailyLastSourceRowsFiltersYutongToMileageFrames(t *testing.T) {
db, mock, err := sqlmock.New()
if err != nil {
@@ -140,9 +481,9 @@ func TestQueryDailyLastSourceRowsFiltersYutongToMileageFrames(t *testing.T) {
loc := time.FixedZone("Asia/Shanghai", 8*3600)
firstTS := time.Date(2026, 7, 8, 8, 0, 0, 0, loc)
lastTS := time.Date(2026, 7, 8, 18, 0, 0, 0, loc)
mock.ExpectQuery("(?s)FROM lingniu_vehicle_ts\\.raw_frames.*protocol = 'YUTONG_MQTT'.*parsed_json LIKE '%yutong_mqtt\\.data\\.total_mileage%'").
mock.ExpectQuery("(?s)FROM lingniu_vehicle_ts\\.raw_frames.*protocol = 'YUTONG_MQTT'.*parsed_json LIKE '%yutong_mqtt\\.data\\.total_mileage%'.*parsed_json LIKE '%TOTAL_MILEAGE%'").
WillReturnRows(sqlmock.NewRows([]string{
"vin", "phone", "device_id", "source_endpoint", "FIRST(ts)", "FIRST(parsed_json)", "LAST(ts)", "LAST(parsed_json)", "COUNT(*)",
"vin", "phone", "device_id", "source_endpoint", "FIRST(event_time)", "FIRST(parsed_json)", "LAST(event_time)", "LAST(parsed_json)", "COUNT(*)",
}).AddRow(
"LMRKH9AC6R1004108",
"",
@@ -183,9 +524,9 @@ func TestQueryPreviousLastSourceRowsFiltersYutongToMileageFrames(t *testing.T) {
loc := time.FixedZone("Asia/Shanghai", 8*3600)
lastTS := time.Date(2026, 7, 7, 23, 58, 0, 0, loc)
mock.ExpectQuery("(?s)FROM lingniu_vehicle_ts\\.raw_frames.*protocol = 'YUTONG_MQTT'.*parsed_json LIKE '%yutong_mqtt\\.data\\.total_mileage%'").
mock.ExpectQuery("(?s)FROM lingniu_vehicle_ts\\.raw_frames.*event_time < '2026-07-08 00:00:00'.*protocol = 'YUTONG_MQTT'.*parsed_json LIKE '%yutong_mqtt\\.data\\.total_mileage%'.*parsed_json LIKE '%TOTAL_MILEAGE%'").
WillReturnRows(sqlmock.NewRows([]string{
"vin", "phone", "device_id", "source_endpoint", "LAST(ts)", "LAST(parsed_json)", "COUNT(*)",
"vin", "phone", "device_id", "source_endpoint", "LAST(event_time)", "LAST(parsed_json)", "COUNT(*)",
}).AddRow(
"LMRKH9AC6R1004108",
"",
@@ -215,6 +556,89 @@ func TestQueryPreviousLastSourceRowsFiltersYutongToMileageFrames(t *testing.T) {
}
}
func TestBackfillBeforePredicatesSearchesAllEarlierHistory(t *testing.T) {
where := strings.Join(backfillBeforePredicates("2026-07-08"), " AND ")
if !strings.Contains(where, "event_time < '2026-07-08 00:00:00'") {
t.Fatalf("pre-window predicate missing exclusive upper bound: %s", where)
}
if strings.Contains(where, "event_time >=") || strings.Contains(where, "ts >=") {
t.Fatalf("pre-window predicate must not stop at the previous day: %s", where)
}
}
func TestQueryPreviousLastSourceRowsNormalizesScannedTimestampToBusinessTimezone(t *testing.T) {
db, mock, err := sqlmock.New()
if err != nil {
t.Fatalf("sqlmock.New() error = %v", err)
}
defer db.Close()
loc := time.FixedZone("Asia/Shanghai", 8*3600)
utcInstant := time.Date(2026, 7, 12, 15, 59, 59, 0, time.UTC)
mock.ExpectQuery("(?s)FROM lingniu_vehicle_ts\\.raw_frames.*protocol = 'YUTONG_MQTT'").
WillReturnRows(sqlmock.NewRows([]string{
"vin", "phone", "device_id", "source_endpoint", "LAST(event_time)", "LAST(parsed_json)", "COUNT(*)",
}).AddRow(
"LMRKH9AC7R1004098",
"",
"LMRKH9AC7R1004098",
"mqtt://yutong/ytforward/shln/4",
utcInstant,
`{"yutong_mqtt.data.total_mileage":"41249000"}`,
int64(1),
))
rows, err := queryPreviousLastSourceRows(context.Background(), db, config{
TDengineDatabase: "lingniu_vehicle_ts",
Location: loc,
}, envelope.ProtocolYutongMQTT, "2026-07-13")
if err != nil {
t.Fatalf("queryPreviousLastSourceRows() error = %v", err)
}
sourceRows := rows["LMRKH9AC7R1004098"]
if len(sourceRows) != 1 {
t.Fatalf("rows = %d, want 1", len(sourceRows))
}
want := time.Date(2026, 7, 12, 23, 59, 59, 0, loc)
if !sourceRows[0].TS.Equal(want) || sourceRows[0].TS.Location().String() != loc.String() {
t.Fatalf("previous event time = %s (%s), want %s (%s)", sourceRows[0].TS, sourceRows[0].TS.Location(), want, want.Location())
}
}
func TestFieldsForStatsExtractsRawYutongTotalMileage(t *testing.T) {
fields := fieldsForStats(envelope.ProtocolYutongMQTT, "LMRKH9AC6R1004108", `{
"data": {
"TOTAL_MILEAGE": 65423000,
"METER_SPEED": 12.3
},
"root": {
"device": "LMRKH9AC6R1004108"
}
}`)
if got := fields["yutong_mqtt.data.total_mileage"]; got == nil {
t.Fatalf("fields missing raw yutong total mileage: %#v", fields)
}
env := envelope.FrameEnvelope{
Protocol: envelope.ProtocolYutongMQTT,
VIN: "LMRKH9AC6R1004108",
EventTimeMS: time.Date(2026, 7, 8, 8, 0, 0, 0, time.FixedZone("Asia/Shanghai", 8*3600)).UnixMilli(),
ReceivedAtMS: time.Date(2026, 7, 8, 8, 0, 0, 0, time.FixedZone("Asia/Shanghai", 8*3600)).UnixMilli(),
Fields: fields,
}
samples, err := stats.SamplesFromEnvelope(env, time.FixedZone("Asia/Shanghai", 8*3600))
if err != nil {
t.Fatalf("SamplesFromEnvelope() error = %v", err)
}
if len(samples) != 1 {
t.Fatalf("samples = %d, want 1", len(samples))
}
if samples[0].TotalMileageKM != 65423 {
t.Fatalf("total mileage km = %v, want 65423", samples[0].TotalMileageKM)
}
}
func TestClearBackfillTargetMileageClearsExactKey(t *testing.T) {
db, mock, err := sqlmock.New()
if err != nil {
@@ -269,7 +693,7 @@ func TestWriteAggregatesClearsTargetRowsBeforeStaleCandidateUpsert(t *testing.T)
WithArgs("LA9GG64L7PBAF4001", "2026-07-08", "JT808").
WillReturnResult(sqlmock.NewResult(0, 1))
mock.ExpectExec(`INSERT INTO vehicle_data_source`).
WithArgs("JT808", "115.231.168.135", "115.231.168.135:20215", sqlmock.AnyArg(), sqlmock.AnyArg()).
WithArgs("JT808", "115.231.168.135", "115.231.168.135:20215", sqlmock.AnyArg(), sqlmock.AnyArg(), sqlmock.AnyArg(), sqlmock.AnyArg(), sqlmock.AnyArg()).
WillReturnResult(sqlmock.NewResult(0, 1))
mock.ExpectExec(`INSERT INTO vehicle_daily_mileage_source`).
WithArgs(
@@ -292,18 +716,16 @@ func TestWriteAggregatesClearsTargetRowsBeforeStaleCandidateUpsert(t *testing.T)
"outside_daily_range",
).
WillReturnResult(sqlmock.NewResult(0, 1))
mock.ExpectExec(`UPDATE vehicle_daily_mileage_source`).
WithArgs("LA9GG64L7PBAF4001", "2026-07-08", "JT808").
WillReturnResult(sqlmock.NewResult(0, 0))
mock.ExpectBegin()
mock.ExpectExec(`INSERT INTO vehicle_daily_mileage`).
WithArgs("LA9GG64L7PBAF4001", "2026-07-08", "JT808", int64(1000)).
WithArgs("LA9GG64L7PBAF4001", "2026-07-08", "JT808", int64(2500)).
WillReturnResult(sqlmock.NewResult(1, 0))
mock.ExpectExec(`UPDATE vehicle_daily_mileage_source s`).
WithArgs(
"LA9GG64L7PBAF4001",
"2026-07-08",
"JT808",
int64(1000),
int64(2500),
"LA9GG64L7PBAF4001",
"2026-07-08",
"JT808",
@@ -319,6 +741,7 @@ func TestWriteAggregatesClearsTargetRowsBeforeStaleCandidateUpsert(t *testing.T)
"JT808",
).
WillReturnResult(sqlmock.NewResult(0, 1))
mock.ExpectCommit()
written, err := writeAggregates(context.Background(), db, aggregates, 500)
if err != nil {
@@ -419,6 +842,8 @@ func TestAddSamplesUsesCurrentDayFirstSampleBaseline(t *testing.T) {
func TestLoadConfigDefaultsBackfillMethodToLastDiff(t *testing.T) {
t.Setenv("BACKFILL_METHOD", "")
t.Setenv("BACKFILL_DAYS_BACK", "")
t.Setenv("BACKFILL_WINDOW_DAYS", "")
t.Setenv("BACKFILL_DATE_FROM", "2026-07-08")
t.Setenv("BACKFILL_DATE_TO", "2026-07-08")
t.Setenv("BACKFILL_PROTOCOLS", "JT808")
@@ -431,4 +856,101 @@ func TestLoadConfigDefaultsBackfillMethodToLastDiff(t *testing.T) {
if cfg.Method != "last_diff" {
t.Fatalf("method = %q, want last_diff", cfg.Method)
}
if cfg.EventTimeFullScan {
t.Fatal("event-time full scan should be opt-in")
}
}
func TestBackfillTimePredicatesUsePrimaryTimeForCoarseScanAndEventTimeForBusinessDay(t *testing.T) {
where := strings.Join(backfillTimePredicates(config{}, "2026-07-13", "2026-07-14"), " AND ")
for _, want := range []string{
"ts >= '2026-07-12 00:00:00'",
"ts < '2026-07-15 00:00:00'",
"event_time >= '2026-07-13 00:00:00'",
"event_time < '2026-07-14 00:00:00'",
} {
if !strings.Contains(where, want) {
t.Fatalf("time predicates missing %q: %s", want, where)
}
}
}
func TestBackfillTimePredicatesAllowExplicitDeepEventTimeScan(t *testing.T) {
where := strings.Join(backfillTimePredicates(config{EventTimeFullScan: true}, "2026-07-13", "2026-07-14"), " AND ")
if strings.Contains(where, "ts >=") || strings.Contains(where, "ts <") {
t.Fatalf("deep event-time scan must not apply storage-time bounds: %s", where)
}
for _, want := range []string{
"event_time >= '2026-07-13 00:00:00'",
"event_time < '2026-07-14 00:00:00'",
} {
if !strings.Contains(where, want) {
t.Fatalf("deep event-time scan missing %q: %s", want, where)
}
}
}
func TestResolveBackfillDateRangeDefaultsToToday(t *testing.T) {
t.Setenv("BACKFILL_DATE_FROM", "")
t.Setenv("BACKFILL_DATE_TO", "")
t.Setenv("BACKFILL_DAYS_BACK", "")
t.Setenv("BACKFILL_WINDOW_DAYS", "")
loc := time.FixedZone("Asia/Shanghai", 8*3600)
dateFrom, dateTo := resolveBackfillDateRange(time.Date(2026, 7, 12, 10, 0, 0, 0, loc), loc)
if dateFrom != "2026-07-12" || dateTo != "2026-07-12" {
t.Fatalf("date range = %s -> %s", dateFrom, dateTo)
}
}
func TestResolveBackfillDateRangeUsesRelativeWindow(t *testing.T) {
t.Setenv("BACKFILL_DATE_FROM", "")
t.Setenv("BACKFILL_DATE_TO", "")
t.Setenv("BACKFILL_DAYS_BACK", "1")
t.Setenv("BACKFILL_WINDOW_DAYS", "3")
loc := time.FixedZone("Asia/Shanghai", 8*3600)
dateFrom, dateTo := resolveBackfillDateRange(time.Date(2026, 7, 12, 10, 0, 0, 0, loc), loc)
if dateFrom != "2026-07-09" || dateTo != "2026-07-11" {
t.Fatalf("date range = %s -> %s", dateFrom, dateTo)
}
}
func TestResolveBackfillDateRangePrefersExplicitDates(t *testing.T) {
t.Setenv("BACKFILL_DATE_FROM", "2026-07-01")
t.Setenv("BACKFILL_DATE_TO", "2026-07-03")
t.Setenv("BACKFILL_DAYS_BACK", "1")
t.Setenv("BACKFILL_WINDOW_DAYS", "3")
loc := time.FixedZone("Asia/Shanghai", 8*3600)
dateFrom, dateTo := resolveBackfillDateRange(time.Date(2026, 7, 12, 10, 0, 0, 0, loc), loc)
if dateFrom != "2026-07-01" || dateTo != "2026-07-03" {
t.Fatalf("date range = %s -> %s", dateFrom, dateTo)
}
}
func TestBackfillEnvFilesPrefersExplicitList(t *testing.T) {
got := backfillEnvFiles("/tmp/a.env,/tmp/b.env", "/tmp/legacy.env", []string{"/tmp/default.env"})
if got != "/tmp/a.env,/tmp/b.env" {
t.Fatalf("env files = %q", got)
}
}
func TestBackfillEnvFilesUsesExistingDefaults(t *testing.T) {
dir := t.TempDir()
missing := filepath.Join(dir, "missing.env")
base := filepath.Join(dir, "base.env")
stat := filepath.Join(dir, "stat-writer.env")
if err := os.WriteFile(base, []byte("A=1\n"), 0600); err != nil {
t.Fatalf("write base env: %v", err)
}
if err := os.WriteFile(stat, []byte("B=2\n"), 0600); err != nil {
t.Fatalf("write stat env: %v", err)
}
got := backfillEnvFiles("", "", []string{missing, base, stat})
want := base + "," + stat
if got != want {
t.Fatalf("env files = %q, want %q", got, want)
}
}

16
go/vehicle-gateway/go.mod
View File

@@ -11,7 +11,8 @@ require (
github.com/redis/go-redis/v9 v9.17.2
github.com/segmentio/kafka-go v0.4.49
github.com/taosdata/driver-go/v3 v3.8.1
golang.org/x/text v0.35.0
github.com/xuri/excelize/v2 v2.11.0
golang.org/x/text v0.38.0
)
require (
@@ -27,9 +28,14 @@ require (
github.com/nats-io/nkeys v0.4.15 // indirect
github.com/nats-io/nuid v1.0.1 // indirect
github.com/pierrec/lz4/v4 v4.1.15 // indirect
github.com/richardlehane/mscfb v1.0.7 // indirect
github.com/richardlehane/msoleps v1.0.6 // indirect
github.com/tiendc/go-deepcopy v1.7.2 // indirect
github.com/xuri/efp v0.0.1 // indirect
github.com/xuri/nfp v0.0.2-0.20250530014748-2ddeb826f9a9 // indirect
github.com/yuin/gopher-lua v1.1.1 // indirect
golang.org/x/crypto v0.49.0 // indirect
golang.org/x/net v0.51.0 // indirect
golang.org/x/sync v0.20.0 // indirect
golang.org/x/sys v0.42.0 // indirect
golang.org/x/crypto v0.53.0 // indirect
golang.org/x/net v0.56.0 // indirect
golang.org/x/sync v0.21.0 // indirect
golang.org/x/sys v0.46.0 // indirect
)

37
go/vehicle-gateway/go.sum
View File

@@ -46,6 +46,10 @@ github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZb
github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
github.com/redis/go-redis/v9 v9.17.2 h1:P2EGsA4qVIM3Pp+aPocCJ7DguDHhqrXNhVcEp4ViluI=
github.com/redis/go-redis/v9 v9.17.2/go.mod h1:u410H11HMLoB+TP67dz8rL9s6QW2j76l0//kSOd3370=
github.com/richardlehane/mscfb v1.0.7 h1:oeoiM0WE79vHwE8RpIYYvIAc8ajTH2mb6UZm55/+EB0=
github.com/richardlehane/mscfb v1.0.7/go.mod h1:pe0+IUIc0AHh0+teNzBlJCtSyZdFOGgV4ZK9bsoV+Jo=
github.com/richardlehane/msoleps v1.0.6 h1:9BvkpjvD+iUBalUY4esMwv6uBkfOip/Lzvd93jvR9gg=
github.com/richardlehane/msoleps v1.0.6/go.mod h1:BWev5JBpU9Ko2WAgmZEuiz4/u3ZYTKbjLycmwiWUfWg=
github.com/segmentio/kafka-go v0.4.49 h1:GJiNX1d/g+kG6ljyJEoi9++PUMdXGAxb7JGPiDCuNmk=
github.com/segmentio/kafka-go v0.4.49/go.mod h1:Y1gn60kzLEEaW28YshXyk2+VCUKbJ3Qr6DrnT3i4+9E=
github.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME=
@@ -55,28 +59,39 @@ github.com/stretchr/objx v0.5.0/go.mod h1:Yh+to48EsGEfYuaHDzXPcE3xhTkx73EhmCGUpE
github.com/stretchr/testify v1.3.0/go.mod h1:M5WIy9Dh21IEIfnGCwXGc5bZfKNJtfHm1UVUgZn+9EI=
github.com/stretchr/testify v1.7.1/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg=
github.com/stretchr/testify v1.8.0/go.mod h1:yNjHg4UonilssWZ8iaSj1OCr/vHnekPRkoO+kdMU+MU=
github.com/stretchr/testify v1.8.2 h1:+h33VjcLVPDHtOdpUCuF+7gSuG3yGIftsP1YvFihtJ8=
github.com/stretchr/testify v1.8.2/go.mod h1:w2LPCIKwWwSfY2zedu0+kehJoqGctiVI29o6fzry7u4=
github.com/stretchr/testify v1.11.1 h1:7s2iGBzp5EwR7/aIZr8ao5+dra3wiQyKjjFuvgVKu7U=
github.com/stretchr/testify v1.11.1/go.mod h1:wZwfW3scLgRK+23gO65QZefKpKQRnfz6sD981Nm4B6U=
github.com/taosdata/driver-go/v3 v3.8.1 h1:kkd4ABsGiU+oXDbsw/sic985LKAvnpF9Gb/TEunTnLE=
github.com/taosdata/driver-go/v3 v3.8.1/go.mod h1:S6OGOinfR0xxxaMGsvBi9cLkYxEIW1p6qqr8QJATTlg=
github.com/tiendc/go-deepcopy v1.7.2 h1:Ut2yYR7W9tWjTQitganoIue4UGxZwCcJy3orjrrIj44=
github.com/tiendc/go-deepcopy v1.7.2/go.mod h1:4bKjNC2r7boYOkD2IOuZpYjmlDdzjbpTRyCx+goBCJQ=
github.com/xdg-go/pbkdf2 v1.0.0 h1:Su7DPu48wXMwC3bs7MCNG+z4FhcyEuz5dlvchbq0B0c=
github.com/xdg-go/pbkdf2 v1.0.0/go.mod h1:jrpuAogTd400dnrH08LKmI/xc1MbPOebTwRqcT5RDeI=
github.com/xdg-go/scram v1.1.2 h1:FHX5I5B4i4hKRVRBCFRxq1iQRej7WO3hhBuJf+UUySY=
github.com/xdg-go/scram v1.1.2/go.mod h1:RT/sEzTbU5y00aCK8UOx6R7YryM0iF1N2MOmC3kKLN4=
github.com/xdg-go/stringprep v1.0.4 h1:XLI/Ng3O1Atzq0oBs3TWm+5ZVgkq2aqdlvP9JtoZ6c8=
github.com/xdg-go/stringprep v1.0.4/go.mod h1:mPGuuIYwz7CmR2bT9j4GbQqutWS1zV24gijq1dTyGkM=
github.com/xuri/efp v0.0.1 h1:fws5Rv3myXyYni8uwj2qKjVaRP30PdjeYe2Y6FDsCL8=
github.com/xuri/efp v0.0.1/go.mod h1:ybY/Jr0T0GTCnYjKqmdwxyxn2BQf2RcQIIvex5QldPI=
github.com/xuri/excelize/v2 v2.11.0 h1:HxaEFl6sRN2+8J5a8HaKq+0M4FsjBGMnWWtjOCPSG88=
github.com/xuri/excelize/v2 v2.11.0/go.mod h1:jxFLbzaIwGQ5ufFNvYfUOHqXhfPaNmP14KWfmNz2Uak=
github.com/xuri/nfp v0.0.2-0.20250530014748-2ddeb826f9a9 h1:+C0TIdyyYmzadGaL/HBLbf3WdLgC29pgyhTjAT/0nuE=
github.com/xuri/nfp v0.0.2-0.20250530014748-2ddeb826f9a9/go.mod h1:WwHg+CVyzlv/TX9xqBFXEZAuxOPxn2k1GNHwG41IIUQ=
github.com/yuin/gopher-lua v1.1.1 h1:kYKnWBjvbNP4XLT3+bPEwAXJx262OhaHDWDVOPjL46M=
github.com/yuin/gopher-lua v1.1.1/go.mod h1:GBR0iDaNXjAgGg9zfCvksxSRnQx76gclCIb7kdAd1Pw=
golang.org/x/crypto v0.49.0 h1:+Ng2ULVvLHnJ/ZFEq4KdcDd/cfjrrjjNSXNzxg0Y4U4=
golang.org/x/crypto v0.49.0/go.mod h1:ErX4dUh2UM+CFYiXZRTcMpEcN8b/1gxEuv3nODoYtCA=
golang.org/x/net v0.51.0 h1:94R/GTO7mt3/4wIKpcR5gkGmRLOuE/2hNGeWq/GBIFo=
golang.org/x/net v0.51.0/go.mod h1:aamm+2QF5ogm02fjy5Bb7CQ0WMt1/WVM7FtyaTLlA9Y=
golang.org/x/sync v0.20.0 h1:e0PTpb7pjO8GAtTs2dQ6jYa5BWYlMuX047Dco/pItO4=
golang.org/x/sync v0.20.0/go.mod h1:9xrNwdLfx4jkKbNva9FpL6vEN7evnE43NNNJQ2LF3+0=
golang.org/x/sys v0.42.0 h1:omrd2nAlyT5ESRdCLYdm3+fMfNFE/+Rf4bDIQImRJeo=
golang.org/x/sys v0.42.0/go.mod h1:4GL1E5IUh+htKOUEOaiffhrAeqysfVGipDYzABqnCmw=
golang.org/x/text v0.35.0 h1:JOVx6vVDFokkpaq1AEptVzLTpDe9KGpj5tR4/X+ybL8=
golang.org/x/text v0.35.0/go.mod h1:khi/HExzZJ2pGnjenulevKNX1W67CUy0AsXcNubPGCA=
golang.org/x/crypto v0.53.0 h1:QZ4Muo8THX6CizN2vPPd5fBGHyogrdK9fG4wLPFUsto=
golang.org/x/crypto v0.53.0/go.mod h1:DNLU434OwVakk9PzuwV8w62mAJpRJL3vsgcfp4Qnsio=
golang.org/x/image v0.38.0 h1:5l+q+Y9JDC7mBOMjo4/aPhMDcxEptsX+Tt3GgRQRPuE=
golang.org/x/image v0.38.0/go.mod h1:/3f6vaXC+6CEanU4KJxbcUZyEePbyKbaLoDOe4ehFYY=
golang.org/x/net v0.56.0 h1:Rw8j/hFzGvJUZwNBXnAtf5sVDVt+65SK2C7IxCxZt5o=
golang.org/x/net v0.56.0/go.mod h1:D3Ku6r+V6JROoZK144D2XfMHFcMq/0zSfLelVTCFKec=
golang.org/x/sync v0.21.0 h1:HLII4xRRTtCRkxYp4HNFF0Js/Og6q2i++KXbg0gHCwM=
golang.org/x/sync v0.21.0/go.mod h1:9xrNwdLfx4jkKbNva9FpL6vEN7evnE43NNNJQ2LF3+0=
golang.org/x/sys v0.46.0 h1:noSf2Fq6F8DBgS+LysIkx7rIExoNHJsxOAtPp4rthXw=
golang.org/x/sys v0.46.0/go.mod h1:4GL1E5IUh+htKOUEOaiffhrAeqysfVGipDYzABqnCmw=
golang.org/x/text v0.38.0 h1:sXmwo9DwP3OK9EZ7PqAdaooSGozfl/3a6/xJcbzPRhE=
golang.org/x/text v0.38.0/go.mod h1:YXZt3QhHUKYT53r2lLKFIVi6Ao1jdzrTR/KQ09qyxF4=
gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
gopkg.in/yaml.v3 v3.0.0-20200313102051-9f266ea9e77c/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA=

215
go/vehicle-gateway/internal/authentication/authentication.go Normal file
View File

@@ -0,0 +1,215 @@
package authentication
import (
"crypto/subtle"
"fmt"
"strings"
"lingniu-vehicle-ingest/go/vehicle-gateway/internal/envelope"
)
type Mode string
const (
ModeDisabled Mode = "disabled"
ModeObserve Mode = "observe"
ModeEnforce Mode = "enforce"
)
const (
StatusAccepted = "accepted"
StatusRejected = "rejected"
StatusUnknownAccount = "unknown_account"
StatusMissingCredential = "missing_credential"
StatusUnconfigured = "unconfigured"
)
type Result struct {
Applicable bool
Allowed bool
Mode Mode
Status string
Source string
}
type Authenticator interface {
Authenticate(envelope.FrameEnvelope) Result
}
func ParseMode(value string, fallback Mode) (Mode, error) {
value = strings.ToLower(strings.TrimSpace(value))
if value == "" {
value = string(fallback)
}
switch Mode(value) {
case ModeDisabled, ModeObserve, ModeEnforce:
return Mode(value), nil
default:
return "", fmt.Errorf("unsupported authentication mode %q", value)
}
}
type GB32960PlatformAuthenticator struct {
mode Mode
credentials map[string][]string
}
func NewGB32960PlatformAuthenticator(mode Mode, credentials map[string][]string) *GB32960PlatformAuthenticator {
normalized := make(map[string][]string, len(credentials))
for username, passwords := range credentials {
username = strings.TrimSpace(username)
if username == "" {
continue
}
for _, password := range passwords {
if password == "" {
continue
}
normalized[username] = append(normalized[username], password)
}
}
return &GB32960PlatformAuthenticator{mode: mode, credentials: normalized}
}
func (a *GB32960PlatformAuthenticator) Authenticate(env envelope.FrameEnvelope) Result {
if env.Protocol != envelope.ProtocolGB32960 || env.MessageID != "0x05" || a == nil || a.mode == ModeDisabled {
return Result{}
}
login := nestedMap(env.Parsed, "platform_login")
username := strings.TrimSpace(textValue(login, "username"))
password := textValue(login, "password")
status := StatusRejected
switch {
case len(a.credentials) == 0:
status = StatusUnconfigured
case username == "" || password == "":
status = StatusMissingCredential
default:
expected, ok := a.credentials[username]
if !ok || len(expected) == 0 {
status = StatusUnknownAccount
} else if anyConstantTimeEqual(expected, password) {
status = StatusAccepted
}
}
source := "none"
if status == StatusAccepted {
source = "configured"
}
return resultForMode(a.mode, status, source)
}
type JT808Authenticator struct {
mode Mode
authCode string
deviceTokens JT808DeviceTokenProvider
}
type JT808DeviceTokenProvider interface {
JT808AuthToken(phone string) (string, bool)
}
func NewJT808Authenticator(mode Mode, authCode string, deviceTokens JT808DeviceTokenProvider) *JT808Authenticator {
return &JT808Authenticator{mode: mode, authCode: authCode, deviceTokens: deviceTokens}
}
func (a *JT808Authenticator) Authenticate(env envelope.FrameEnvelope) Result {
if env.Protocol != envelope.ProtocolJT808 || env.MessageID != "0x0102" || a == nil || a.mode == ModeDisabled {
return Result{}
}
token := textValue(nestedMap(env.Parsed, "authentication"), "token")
status := StatusRejected
source := "none"
switch {
case token == "":
status = StatusMissingCredential
default:
if a.authCode != "" && constantTimeEqual(a.authCode, token) {
status = StatusAccepted
source = "configured"
break
}
deviceToken, knownDevice := "", false
if a.deviceTokens != nil {
deviceToken, knownDevice = a.deviceTokens.JT808AuthToken(env.Phone)
}
if knownDevice && deviceToken != "" && constantTimeEqual(deviceToken, token) {
status = StatusAccepted
source = "device"
} else if a.authCode == "" && !knownDevice {
status = StatusUnconfigured
}
}
return resultForMode(a.mode, status, source)
}
func Apply(env *envelope.FrameEnvelope, result Result) {
if env == nil || !result.Applicable {
return
}
env.AuthenticationMode = string(result.Mode)
env.AuthenticationStatus = result.Status
env.AuthenticationEnforced = result.Mode == ModeEnforce
}
// RedactParsedCredentials removes convenience copies of secrets before parsed
// fields are flattened and published. The original protocol frame remains in
// raw_hex for restricted forensic access.
func RedactParsedCredentials(env *envelope.FrameEnvelope) {
if env == nil || env.Protocol != envelope.ProtocolGB32960 {
return
}
login := nestedMap(env.Parsed, "platform_login")
if login == nil {
return
}
if password := textValue(login, "password"); password != "" {
login["password_present"] = true
}
delete(login, "password")
}
func resultForMode(mode Mode, status string, source ...string) Result {
allowed := status == StatusAccepted || mode != ModeEnforce
credentialSource := "none"
if len(source) > 0 && strings.TrimSpace(source[0]) != "" {
credentialSource = strings.TrimSpace(source[0])
}
return Result{Applicable: true, Allowed: allowed, Mode: mode, Status: status, Source: credentialSource}
}
func constantTimeEqual(expected string, actual string) bool {
if len(expected) != len(actual) {
return false
}
return subtle.ConstantTimeCompare([]byte(expected), []byte(actual)) == 1
}
func anyConstantTimeEqual(expected []string, actual string) bool {
matched := 0
for _, candidate := range expected {
if len(candidate) == len(actual) {
matched |= subtle.ConstantTimeCompare([]byte(candidate), []byte(actual))
}
}
return matched == 1
}
func nestedMap(parent map[string]any, key string) map[string]any {
if parent == nil {
return nil
}
value, _ := parent[key].(map[string]any)
return value
}
func textValue(values map[string]any, key string) string {
if values == nil {
return ""
}
value, ok := values[key]
if !ok || value == nil {
return ""
}
return strings.TrimSpace(fmt.Sprint(value))
}

111
go/vehicle-gateway/internal/authentication/authentication_test.go Normal file
View File

@@ -0,0 +1,111 @@
package authentication
import (
"strings"
"testing"
"lingniu-vehicle-ingest/go/vehicle-gateway/internal/envelope"
)
func TestGB32960PlatformAuthenticatorModes(t *testing.T) {
env := envelope.FrameEnvelope{
Protocol: envelope.ProtocolGB32960,
MessageID: "0x05",
Parsed: map[string]any{
"platform_login": map[string]any{"username": "platform-a", "password": "secret-a"},
},
}
accepted := NewGB32960PlatformAuthenticator(ModeEnforce, map[string][]string{"platform-a": {"secret-a"}}).Authenticate(env)
if !accepted.Applicable || !accepted.Allowed || accepted.Status != StatusAccepted {
t.Fatalf("accepted result = %#v", accepted)
}
rejected := NewGB32960PlatformAuthenticator(ModeEnforce, map[string][]string{"platform-a": {"other"}}).Authenticate(env)
if !rejected.Applicable || rejected.Allowed || rejected.Status != StatusRejected {
t.Fatalf("enforced rejected result = %#v", rejected)
}
observed := NewGB32960PlatformAuthenticator(ModeObserve, map[string][]string{"platform-a": {"other"}}).Authenticate(env)
if !observed.Allowed || observed.Status != StatusRejected {
t.Fatalf("observed rejected result = %#v", observed)
}
}
func TestGB32960PlatformAuthenticatorReportsUnknownAccount(t *testing.T) {
env := envelope.FrameEnvelope{
Protocol: envelope.ProtocolGB32960,
MessageID: "0x05",
Parsed: map[string]any{
"platform_login": map[string]any{"username": "unknown", "password": "secret"},
},
}
result := NewGB32960PlatformAuthenticator(ModeEnforce, map[string][]string{"known": {"secret"}}).Authenticate(env)
if result.Allowed || result.Status != StatusUnknownAccount {
t.Fatalf("result = %#v", result)
}
}
func TestJT808AuthenticatorValidatesAuthenticationFrame(t *testing.T) {
env := envelope.FrameEnvelope{
Protocol: envelope.ProtocolJT808,
MessageID: "0x0102",
Parsed: map[string]any{
"authentication": map[string]any{"token": "issued-code"},
},
}
accepted := NewJT808Authenticator(ModeEnforce, "issued-code", nil).Authenticate(env)
if !accepted.Allowed || accepted.Status != StatusAccepted {
t.Fatalf("accepted result = %#v", accepted)
}
rejected := NewJT808Authenticator(ModeEnforce, "different-code", nil).Authenticate(env)
if rejected.Allowed || rejected.Status != StatusRejected {
t.Fatalf("rejected result = %#v", rejected)
}
}
func TestJT808AuthenticatorAcceptsDeviceSnapshotToken(t *testing.T) {
env := envelope.FrameEnvelope{
Protocol: envelope.ProtocolJT808,
MessageID: "0x0102",
Phone: "013307795425",
Parsed: map[string]any{
"authentication": map[string]any{"token": "device-code"},
},
}
provider := staticJT808DeviceTokens{"13307795425": "device-code"}
result := NewJT808Authenticator(ModeEnforce, "configured-code", provider).Authenticate(env)
if !result.Allowed || result.Status != StatusAccepted || result.Source != "device" {
t.Fatalf("device token result = %#v", result)
}
}
type staticJT808DeviceTokens map[string]string
func (p staticJT808DeviceTokens) JT808AuthToken(phone string) (string, bool) {
phone = strings.TrimLeft(phone, "0")
token, ok := p[phone]
return token, ok
}
func TestRedactParsedCredentialsRemovesGB32960Password(t *testing.T) {
login := map[string]any{"username": "platform-a", "password": "secret-a"}
env := envelope.FrameEnvelope{
Protocol: envelope.ProtocolGB32960,
Parsed: map[string]any{"platform_login": login},
}
RedactParsedCredentials(&env)
if _, exists := login["password"]; exists {
t.Fatalf("password remained in parsed login: %#v", login)
}
if login["password_present"] != true {
t.Fatalf("password presence marker missing: %#v", login)
}
}
func TestParseModeRejectsUnknownValue(t *testing.T) {
if _, err := ParseMode("strict-ish", ModeObserve); err == nil {
t.Fatal("expected unsupported mode error")
}
}

4852
go/vehicle-gateway/internal/capacity/check.go
View File

File diff suppressed because it is too large Load Diff

4589
go/vehicle-gateway/internal/capacity/check_test.go
View File

File diff suppressed because it is too large Load Diff

62
go/vehicle-gateway/internal/envelope/envelope.go
View File

@@ -24,6 +24,14 @@ const (
ParseBadFrame ParseStatus = "BAD_FRAME"
)
type EventKind string
const (
EventKindRaw EventKind = "RAW"
EventKindFields EventKind = "FIELDS"
EventKindUnified EventKind = "UNIFIED"
)
const (
FieldSpeedKMH = "speed_kmh"
FieldTotalMileageKM = "total_mileage_km"
@@ -33,27 +41,36 @@ const (
)
type FrameEnvelope struct {
EventID string `json:"event_id"`
TraceID string `json:"trace_id"`
Protocol Protocol `json:"protocol"`
MessageID string `json:"message_id"`
Sequence uint16 `json:"sequence"`
VIN string `json:"vin,omitempty"`
VehicleKeyHint string `json:"vehicle_key,omitempty"`
Phone string `json:"phone,omitempty"`
DeviceID string `json:"device_id,omitempty"`
Plate string `json:"plate,omitempty"`
SourceEndpoint string `json:"source_endpoint,omitempty"`
EventTimeMS int64 `json:"event_time_ms"`
ReceivedAtMS int64 `json:"received_at_ms"`
RawHex string `json:"raw_hex,omitempty"`
RawText string `json:"raw_text,omitempty"`
Parsed map[string]any `json:"parsed,omitempty"`
ParsedFields map[string]any `json:"parsed_fields,omitempty"`
ParsedFieldTypes map[string]string `json:"parsed_field_types,omitempty"`
Fields map[string]any `json:"fields,omitempty"`
ParseStatus ParseStatus `json:"parse_status"`
ParseError string `json:"parse_error,omitempty"`
EventID string `json:"event_id"`
TraceID string `json:"trace_id"`
EventKind EventKind `json:"event_kind,omitempty"`
SourceEventID string `json:"source_event_id,omitempty"`
FieldMapping string `json:"field_mapping,omitempty"`
Protocol Protocol `json:"protocol"`
MessageID string `json:"message_id"`
Sequence uint16 `json:"sequence"`
VIN string `json:"vin,omitempty"`
VehicleKeyHint string `json:"vehicle_key,omitempty"`
Phone string `json:"phone,omitempty"`
DeviceID string `json:"device_id,omitempty"`
Plate string `json:"plate,omitempty"`
SourceEndpoint string `json:"source_endpoint,omitempty"`
SourceCode string `json:"source_code,omitempty"`
PlatformName string `json:"platform_name,omitempty"`
SourceKind string `json:"source_kind,omitempty"`
AuthenticationMode string `json:"authentication_mode,omitempty"`
AuthenticationStatus string `json:"authentication_status,omitempty"`
AuthenticationEnforced bool `json:"authentication_enforced,omitempty"`
EventTimeMS int64 `json:"event_time_ms"`
ReceivedAtMS int64 `json:"received_at_ms"`
RawHex string `json:"raw_hex,omitempty"`
RawText string `json:"raw_text,omitempty"`
Parsed map[string]any `json:"parsed,omitempty"`
ParsedFields map[string]any `json:"parsed_fields,omitempty"`
ParsedFieldTypes map[string]string `json:"parsed_field_types,omitempty"`
Fields map[string]any `json:"fields,omitempty"`
ParseStatus ParseStatus `json:"parse_status"`
ParseError string `json:"parse_error,omitempty"`
}
func (e FrameEnvelope) VehicleKey() string {
@@ -93,5 +110,8 @@ func (e FrameEnvelope) MarshalJSONBytes() ([]byte, error) {
if e.ParseStatus == "" {
e.ParseStatus = ParseOK
}
if e.EventKind == "" {
e.EventKind = EventKindRaw
}
return json.Marshal(e)
}

157
go/vehicle-gateway/internal/envelope/envelope_test.go
View File

@@ -3,6 +3,7 @@ package envelope
import (
"encoding/json"
"testing"
"time"
)
func TestFrameEnvelopeVehicleKeyPrefersVIN(t *testing.T) {
@@ -52,4 +53,160 @@ func TestFrameEnvelopeMarshalDefaults(t *testing.T) {
if decoded.ParseStatus != ParseOK {
t.Fatalf("parse status = %q", decoded.ParseStatus)
}
if decoded.EventKind != EventKindRaw {
t.Fatalf("event kind = %q, want %q", decoded.EventKind, EventKindRaw)
}
}
func TestNormalizedEventTimeMSFallsBackToReceivedWhenEventIsFarFuture(t *testing.T) {
received := time.Date(2026, 7, 12, 9, 30, 0, 0, time.UTC).UnixMilli()
event := time.Date(2026, 7, 13, 9, 30, 0, 0, time.UTC).UnixMilli()
got, ok := NormalizedEventTimeMS(FrameEnvelope{EventTimeMS: event, ReceivedAtMS: received})
if !ok {
t.Fatal("NormalizedEventTimeMS() ok = false")
}
if got != received {
t.Fatalf("normalized event time = %d, want received %d", got, received)
}
_, reason, ok := NormalizedEventTimeMSWithReason(FrameEnvelope{EventTimeMS: event, ReceivedAtMS: received})
if !ok || reason != EventTimeReasonReceivedFutureEvent {
t.Fatalf("reason = %q ok=%v, want future fallback", reason, ok)
}
}
func TestNormalizedEventTimeMSKeepsSmallFutureSkew(t *testing.T) {
received := time.Date(2026, 7, 12, 9, 30, 0, 0, time.UTC).UnixMilli()
event := time.Date(2026, 7, 12, 9, 35, 0, 0, time.UTC).UnixMilli()
got, ok := NormalizedEventTimeMS(FrameEnvelope{EventTimeMS: event, ReceivedAtMS: received})
if !ok {
t.Fatal("NormalizedEventTimeMS() ok = false")
}
if got != event {
t.Fatalf("normalized event time = %d, want event %d", got, event)
}
}
func TestIsRealtimeTelemetryFrameRejectsKnownNonRealtimeMessageIDs(t *testing.T) {
fields := map[string]any{
FieldLatitude: 30.590151,
FieldLongitude: 121.069881,
FieldTotalMileageKM: 10241.2,
}
tests := []FrameEnvelope{
{Protocol: ProtocolJT808, MessageID: "0x0100", Fields: fields, ParseStatus: ParseOK},
{Protocol: ProtocolGB32960, MessageID: "0x01", Fields: fields, Parsed: map[string]any{"data_units": []any{}}, ParseStatus: ParseOK},
}
for _, env := range tests {
if IsRealtimeTelemetryFrame(env) {
t.Fatalf("IsRealtimeTelemetryFrame(%s/%s) = true, want false", env.Protocol, env.MessageID)
}
}
}
func TestIsRealtimeTelemetryFrameAcceptsKnownRealtimeMessageIDs(t *testing.T) {
tests := []FrameEnvelope{
{Protocol: ProtocolJT808, MessageID: "0x0200", ParsedFields: map[string]any{"jt808.location.speed_kmh": 1}, ParseStatus: ParseOK},
{Protocol: ProtocolGB32960, MessageID: "0x02", ParsedFields: map[string]any{"gb32960.vehicle.speed_kmh": 1}, ParseStatus: ParseOK},
{Protocol: ProtocolGB32960, MessageID: "0x03", ParsedFields: map[string]any{"gb32960.vehicle.speed_kmh": 1}, ParseStatus: ParseOK},
}
for _, env := range tests {
if !IsRealtimeTelemetryFrame(env) {
t.Fatalf("IsRealtimeTelemetryFrame(%s/%s) = false, want true", env.Protocol, env.MessageID)
}
}
}
func TestIsRealtimeTelemetryFrameUsesCanonicalMQTTParsedFields(t *testing.T) {
realtime := FrameEnvelope{
Protocol: ProtocolYutongMQTT,
MessageID: "MQTT",
ParseStatus: ParseOK,
ParsedFields: map[string]any{
"yutong_mqtt.data.latitude": 30.590151,
},
}
if !IsRealtimeTelemetryFrame(realtime) {
t.Fatal("canonical MQTT data field should be classified as realtime")
}
metadataOnly := realtime
metadataOnly.ParsedFields = map[string]any{"yutong_mqtt.metadata.topic": "/ytforward/shln/3"}
if IsRealtimeTelemetryFrame(metadataOnly) {
t.Fatal("MQTT metadata-only field must not be classified as realtime")
}
}
func TestRequiresVehicleIdentity(t *testing.T) {
tests := []struct {
name string
env FrameEnvelope
want bool
}{
{
name: "gb32960 realtime upload",
env: FrameEnvelope{Protocol: ProtocolGB32960, MessageID: "0x02", Parsed: map[string]any{"data_units": []any{map[string]any{"name": "vehicle"}}}, ParseStatus: ParseOK},
want: true,
},
{
name: "gb32960 platform login",
env: FrameEnvelope{Protocol: ProtocolGB32960, MessageID: "0x05", ParseStatus: ParseOK},
want: false,
},
{
name: "gb32960 command response",
env: FrameEnvelope{Protocol: ProtocolGB32960, MessageID: "0x07", ParseStatus: ParseOK},
want: false,
},
{
name: "jt808 registration",
env: FrameEnvelope{Protocol: ProtocolJT808, MessageID: "0x0100", ParseStatus: ParseOK},
want: true,
},
{
name: "jt808 location",
env: FrameEnvelope{Protocol: ProtocolJT808, MessageID: "0x0200", Parsed: map[string]any{"location": map[string]any{"speed_kmh": 1}}, ParseStatus: ParseOK},
want: true,
},
{
name: "yutong empty data",
env: FrameEnvelope{Protocol: ProtocolYutongMQTT, MessageID: "MQTT", Parsed: map[string]any{"data": map[string]any{}}, ParseStatus: ParseOK},
want: false,
},
{
name: "yutong telemetry data",
env: FrameEnvelope{Protocol: ProtocolYutongMQTT, MessageID: "MQTT", Parsed: map[string]any{"data": map[string]any{"TOTAL_MILEAGE": 123}}, ParseStatus: ParseOK},
want: true,
},
{
name: "bad frame",
env: FrameEnvelope{Protocol: ProtocolJT808, MessageID: "0x0200", ParseStatus: ParseBadFrame},
want: false,
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
if got := RequiresVehicleIdentity(tt.env); got != tt.want {
t.Fatalf("RequiresVehicleIdentity() = %v, want %v", got, tt.want)
}
})
}
}
func TestNormalizeSourceEndpointKey(t *testing.T) {
tests := map[string]string{
"115.231.168.135:20215": "115.231.168.135",
"115.231.168.135": "115.231.168.135",
" 115.159.85.149:28316 ": "115.159.85.149",
"mqtt://yutong/ytforward/shln/3": "mqtt",
"MQTT://YUTONG/topic": "mqtt",
"": "",
" ": "",
}
for input, want := range tests {
if got := NormalizeSourceEndpointKey(input); got != want {
t.Fatalf("NormalizeSourceEndpointKey(%q) = %q, want %q", input, got, want)
}
}
}

78
go/vehicle-gateway/internal/envelope/realtime.go
View File

@@ -14,33 +14,93 @@ func IsRealtimeTelemetryFrame(env FrameEnvelope) bool {
command = strings.TrimSpace(stringAny(header["command"]))
}
}
if strings.EqualFold(command, "0x02") || strings.EqualFold(command, "0x03") {
return true
if command != "" && !strings.EqualFold(command, "0x02") && !strings.EqualFold(command, "0x03") {
return false
}
_, hasDataUnits := env.Parsed["data_units"]
return hasDataUnits || hasRealtimeField(env)
return hasDataUnits || hasGB32960ParsedTelemetryField(env) || hasRealtimeField(env)
case ProtocolJT808:
if strings.EqualFold(strings.TrimSpace(env.MessageID), "0x0200") {
return true
messageID := strings.TrimSpace(env.MessageID)
if messageID != "" && !strings.EqualFold(messageID, "0x0200") {
return false
}
if _, ok := env.Parsed["location"]; ok {
return true
}
return hasRealtimeField(env)
return hasParsedFieldPrefix(env, "jt808.location.") || hasRealtimeField(env)
case ProtocolYutongMQTT:
data, ok := env.Parsed["data"].(map[string]any)
return ok && len(data) > 0
if ok && len(data) > 0 {
return true
}
return hasParsedFieldPrefix(env, "yutong_mqtt.data.") ||
hasParsedFieldPrefix(env, "yutong_mqtt.root.data.")
default:
return false
}
}
func hasGB32960ParsedTelemetryField(env FrameEnvelope) bool {
for field := range env.ParsedFields {
if !strings.HasPrefix(field, "gb32960.") {
continue
}
if strings.HasPrefix(field, "gb32960.header.") ||
strings.HasPrefix(field, "gb32960.platform.") ||
strings.HasPrefix(field, "gb32960.identity.") ||
strings.HasPrefix(field, "gb32960.device_time.") {
continue
}
return true
}
return false
}
// RequiresVehicleIdentity keeps platform/control frames out of vehicle identity
// quality metrics while preserving JT808's phone-centric registration/auth flow.
func RequiresVehicleIdentity(env FrameEnvelope) bool {
if env.ParseStatus == ParseBadFrame {
return false
}
switch env.Protocol {
case ProtocolGB32960:
return IsRealtimeTelemetryFrame(env)
case ProtocolJT808:
return true
case ProtocolYutongMQTT:
return IsRealtimeTelemetryFrame(env)
default:
return false
}
}
func hasRealtimeField(env FrameEnvelope) bool {
return env.Fields[FieldLatitude] != nil ||
if env.Fields[FieldLatitude] != nil ||
env.Fields[FieldLongitude] != nil ||
env.Fields[FieldTotalMileageKM] != nil ||
env.Fields[FieldSpeedKMH] != nil ||
env.Fields[FieldSOCPercent] != nil
env.Fields[FieldSOCPercent] != nil {
return true
}
for field := range env.ParsedFields {
if strings.HasSuffix(field, ".latitude") ||
strings.HasSuffix(field, ".longitude") ||
strings.HasSuffix(field, ".total_mileage_km") ||
strings.HasSuffix(field, ".speed_kmh") ||
strings.HasSuffix(field, ".soc_percent") {
return true
}
}
return false
}
func hasParsedFieldPrefix(env FrameEnvelope, prefix string) bool {
for field := range env.ParsedFields {
if strings.HasPrefix(field, prefix) {
return true
}
}
return false
}
func stringAny(value any) string {

19
go/vehicle-gateway/internal/envelope/source.go Normal file
View File

@@ -0,0 +1,19 @@
package envelope
import "strings"
// NormalizeSourceEndpointKey returns the stable, low-cardinality source key used
// by identity lookup and statistics source tracking.
func NormalizeSourceEndpointKey(endpoint string) string {
endpoint = strings.TrimSpace(endpoint)
if endpoint == "" {
return ""
}
if strings.HasPrefix(strings.ToLower(endpoint), "mqtt://") {
return "mqtt"
}
if host, _, ok := strings.Cut(endpoint, ":"); ok {
return strings.TrimSpace(host)
}
return endpoint
}

31
go/vehicle-gateway/internal/envelope/time.go Normal file
View File

@@ -0,0 +1,31 @@
package envelope
import "time"
const (
MaxFutureEventSkew = 10 * time.Minute
MinPlausibleReceivedTimeMS = 1577836800000 // 2020-01-01T00:00:00Z
)
const (
EventTimeReasonEventTime = "event_time"
EventTimeReasonReceivedMissingEvent = "received_time_missing_event"
EventTimeReasonReceivedFutureEvent = "received_time_future_event"
)
func NormalizedEventTimeMS(env FrameEnvelope) (int64, bool) {
eventMS, _, ok := NormalizedEventTimeMSWithReason(env)
return eventMS, ok
}
func NormalizedEventTimeMSWithReason(env FrameEnvelope) (int64, string, bool) {
eventMS := env.EventTimeMS
receivedMS := env.ReceivedAtMS
if eventMS <= 0 {
return receivedMS, EventTimeReasonReceivedMissingEvent, receivedMS > 0
}
if receivedMS >= MinPlausibleReceivedTimeMS && eventMS > receivedMS+MaxFutureEventSkew.Milliseconds() {
return receivedMS, EventTimeReasonReceivedFutureEvent, true
}
return eventMS, EventTimeReasonEventTime, true
}

156
go/vehicle-gateway/internal/eventbus/async_sink.go
View File

@@ -13,6 +13,7 @@ import (
type AsyncConfig struct {
QueueSize int
Workers int
EnqueueTimeout time.Duration
OperationTimeout time.Duration
OnError func(error)
Metrics *metrics.Registry
@@ -20,12 +21,14 @@ type AsyncConfig struct {
}
type AsyncSink struct {
delegate Sink
jobs chan asyncJob
timeout time.Duration
onError func(error)
metrics *metrics.Registry
name string
delegate Sink
jobs chan asyncJob
enqueueTimeout time.Duration
timeout time.Duration
onError func(error)
metrics *metrics.Registry
name string
queueWait *metrics.RecentLatencyByKey
closeOnce sync.Once
closed chan struct{}
@@ -34,11 +37,13 @@ type AsyncSink struct {
}
type asyncJob struct {
kind string
env envelope.FrameEnvelope
kind string
env envelope.FrameEnvelope
enqueuedAt time.Time
}
var ErrAsyncSinkClosed = errors.New("async sink is closed")
var ErrAsyncSinkEnqueueTimeout = errors.New("async sink enqueue timeout")
var asyncSinkPublishDurationBucketsMS = []float64{1, 5, 10, 25, 50, 100, 250, 500, 1000, 5000}
@@ -52,6 +57,12 @@ func NewAsyncSink(delegate Sink, cfg AsyncConfig) *AsyncSink {
if cfg.Workers <= 0 {
cfg.Workers = 1
}
if cfg.EnqueueTimeout == 0 {
cfg.EnqueueTimeout = time.Second
}
if cfg.EnqueueTimeout < 0 {
cfg.EnqueueTimeout = 0
}
if cfg.OperationTimeout <= 0 {
cfg.OperationTimeout = 30 * time.Second
}
@@ -59,19 +70,23 @@ func NewAsyncSink(delegate Sink, cfg AsyncConfig) *AsyncSink {
cfg.Name = "async"
}
s := &AsyncSink{
delegate: delegate,
jobs: make(chan asyncJob, cfg.QueueSize),
timeout: cfg.OperationTimeout,
onError: cfg.OnError,
metrics: cfg.Metrics,
name: cfg.Name,
closed: make(chan struct{}),
done: make(chan struct{}),
delegate: delegate,
jobs: make(chan asyncJob, cfg.QueueSize),
enqueueTimeout: cfg.EnqueueTimeout,
timeout: cfg.OperationTimeout,
onError: cfg.OnError,
metrics: cfg.Metrics,
name: cfg.Name,
queueWait: metrics.NewRecentLatencyByKey(512),
closed: make(chan struct{}),
done: make(chan struct{}),
}
s.wg.Add(cfg.Workers)
for i := 0; i < cfg.Workers; i++ {
go s.worker()
}
s.recordQueueCapacity()
s.recordWorkers("default", cfg.Workers)
go func() {
s.wg.Wait()
close(s.done)
@@ -94,7 +109,6 @@ func (s *AsyncSink) PublishFields(ctx context.Context, env envelope.FrameEnvelop
func (s *AsyncSink) Close() error {
s.closeOnce.Do(func() {
close(s.closed)
close(s.jobs)
})
<-s.done
return s.delegate.Close()
@@ -107,6 +121,14 @@ func (s *AsyncSink) enqueue(ctx context.Context, job asyncJob) error {
return ErrAsyncSinkClosed
default:
}
var timeoutC <-chan time.Time
var timer *time.Timer
if s.enqueueTimeout > 0 {
timer = time.NewTimer(s.enqueueTimeout)
timeoutC = timer.C
defer timer.Stop()
}
job.enqueuedAt = time.Now()
select {
case s.jobs <- job:
s.recordEnqueue(job.kind, "queued")
@@ -119,37 +141,58 @@ func (s *AsyncSink) enqueue(ctx context.Context, job asyncJob) error {
s.recordEnqueue(job.kind, "timeout")
s.recordQueueDepth()
return ctx.Err()
case <-timeoutC:
s.recordEnqueue(job.kind, "timeout")
s.recordQueueDepth()
return ErrAsyncSinkEnqueueTimeout
}
}
func (s *AsyncSink) worker() {
defer s.wg.Done()
for job := range s.jobs {
s.recordQueueDepth()
ctx, cancel := context.WithTimeout(context.Background(), s.timeout)
started := time.Now()
var err error
switch job.kind {
case "raw":
err = s.delegate.PublishRaw(ctx, job.env)
case "unified":
err = s.delegate.PublishUnified(ctx, job.env)
case "fields":
err = s.delegate.PublishFields(ctx, job.env)
for {
select {
case job := <-s.jobs:
s.publishJob(job)
case <-s.closed:
for {
select {
case job := <-s.jobs:
s.publishJob(job)
default:
return
}
}
}
cancel()
status := "ok"
if err != nil {
status = "error"
}
s.recordPublish(job.kind, status, time.Since(started))
if err != nil && s.onError != nil {
s.onError(err)
}
s.recordQueueDepth()
}
}
func (s *AsyncSink) publishJob(job asyncJob) {
s.recordQueueDepth()
s.recordQueueWait("default", job)
ctx, cancel := context.WithTimeout(context.Background(), s.timeout)
started := time.Now()
var err error
switch job.kind {
case "raw":
err = s.delegate.PublishRaw(ctx, job.env)
case "unified":
err = s.delegate.PublishUnified(ctx, job.env)
case "fields":
err = s.delegate.PublishFields(ctx, job.env)
}
cancel()
status := "ok"
if err != nil {
status = "error"
}
s.recordPublish(job.kind, status, time.Since(started))
if err != nil && s.onError != nil {
s.onError(err)
}
s.recordQueueDepth()
}
func (s *AsyncSink) recordEnqueue(kind string, status string) {
if s.metrics == nil {
return
@@ -184,3 +227,38 @@ func (s *AsyncSink) recordQueueDepth() {
"sink": s.name,
}, float64(len(s.jobs)))
}
func (s *AsyncSink) recordQueueCapacity() {
if s.metrics == nil {
return
}
s.metrics.SetGauge("vehicle_async_sink_queue_capacity", metrics.Labels{
"sink": s.name,
}, float64(cap(s.jobs)))
}
func (s *AsyncSink) recordQueueWait(queueName string, job asyncJob) {
if s.metrics == nil || job.enqueuedAt.IsZero() {
return
}
elapsedMS := float64(time.Since(job.enqueuedAt)) / float64(time.Millisecond)
labels := metrics.Labels{
"sink": s.name,
"queue": queueName,
"kind": job.kind,
}
s.metrics.ObserveHistogram("vehicle_async_sink_queue_wait_duration_ms_histogram", labels, asyncSinkPublishDurationBucketsMS, elapsedMS)
p99, samples := s.queueWait.Observe(queueName+"\x00"+job.kind, elapsedMS)
s.metrics.SetGauge("vehicle_async_sink_queue_wait_recent_p99_ms", labels, p99)
s.metrics.SetGauge("vehicle_async_sink_queue_wait_recent_samples", labels, float64(samples))
}
func (s *AsyncSink) recordWorkers(queueName string, workers int) {
if s.metrics == nil {
return
}
s.metrics.SetGauge("vehicle_async_sink_workers", metrics.Labels{
"sink": s.name,
"queue": queueName,
}, float64(workers))
}

145
go/vehicle-gateway/internal/eventbus/async_sink_test.go
View File

@@ -2,6 +2,7 @@ package eventbus
import (
"context"
"errors"
"strings"
"testing"
"time"
@@ -80,12 +81,17 @@ func TestAsyncSinkRecordsQueueMetrics(t *testing.T) {
for _, want := range []string{
`vehicle_async_sink_enqueue_total{kind="raw",sink="nats",status="queued"} 1`,
`vehicle_async_sink_enqueue_total{kind="fields",sink="nats",status="queued"} 1`,
`vehicle_async_sink_queue_capacity{sink="nats"} 2`,
`vehicle_async_sink_queue_depth{sink="nats"}`,
`vehicle_async_sink_workers{queue="default",sink="nats"} 1`,
`vehicle_async_sink_publish_total{kind="raw",sink="nats",status="ok"} 1`,
`vehicle_async_sink_publish_total{kind="fields",sink="nats",status="ok"} 1`,
`vehicle_async_sink_publish_duration_ms_histogram_bucket{le="+Inf",kind="raw",sink="nats",status="ok"} 1`,
`vehicle_async_sink_publish_duration_ms_histogram_count{kind="raw",sink="nats",status="ok"} 1`,
`vehicle_async_sink_publish_duration_ms_histogram_sum{kind="raw",sink="nats",status="ok"}`,
`vehicle_async_sink_queue_wait_duration_ms_histogram_count{kind="raw",queue="default",sink="nats"} 1`,
`vehicle_async_sink_queue_wait_recent_p99_ms{kind="raw",queue="default",sink="nats"}`,
`vehicle_async_sink_queue_wait_recent_samples{kind="raw",queue="default",sink="nats"} 1`,
} {
if !strings.Contains(text, want) {
t.Fatalf("async sink metric missing %s:\n%s", want, text)
@@ -135,6 +141,145 @@ func TestAsyncSinkRecordsEnqueueTimeoutWhenQueueIsFull(t *testing.T) {
delegate.release()
}
func TestAsyncSinkEnqueueTimeoutDoesNotRequireCallerDeadline(t *testing.T) {
registry := metrics.NewRegistry()
delegate := newBlockingSink()
sink := NewAsyncSink(delegate, AsyncConfig{
QueueSize: 1,
Workers: 1,
EnqueueTimeout: 10 * time.Millisecond,
OperationTimeout: time.Second,
Metrics: registry,
Name: "nats",
})
defer sink.Close()
env := envelope.FrameEnvelope{Protocol: envelope.ProtocolJT808, Phone: "13307795425"}
if err := sink.PublishRaw(context.Background(), env); err != nil {
t.Fatalf("first PublishRaw() error = %v", err)
}
select {
case <-delegate.rawStarted:
case <-time.After(time.Second):
t.Fatal("delegate raw publish was not started")
}
if err := sink.PublishFields(context.Background(), env); err != nil {
t.Fatalf("second PublishFields() error = %v", err)
}
if err := sink.PublishUnified(context.Background(), env); !errors.Is(err, ErrAsyncSinkEnqueueTimeout) {
t.Fatalf("third PublishUnified() error = %v, want ErrAsyncSinkEnqueueTimeout", err)
}
text := registry.Render()
for _, want := range []string{
`vehicle_async_sink_enqueue_total{kind="unified",sink="nats",status="timeout"} 1`,
`vehicle_async_sink_queue_depth{sink="nats"} 1`,
} {
if !strings.Contains(text, want) {
t.Fatalf("async sink enqueue timeout metric missing %s:\n%s", want, text)
}
}
delegate.release()
}
func TestAsyncSinkCloseDoesNotPanicWithConcurrentBlockedEnqueue(t *testing.T) {
delegate := newBlockingSink()
sink := NewAsyncSink(delegate, AsyncConfig{
QueueSize: 1,
Workers: 1,
EnqueueTimeout: 20 * time.Millisecond,
OperationTimeout: time.Second,
})
env := envelope.FrameEnvelope{Protocol: envelope.ProtocolJT808, Phone: "13307795425"}
if err := sink.PublishRaw(context.Background(), env); err != nil {
t.Fatalf("first PublishRaw() error = %v", err)
}
select {
case <-delegate.rawStarted:
case <-time.After(time.Second):
t.Fatal("delegate raw publish was not started")
}
if err := sink.PublishFields(context.Background(), env); err != nil {
t.Fatalf("second PublishFields() error = %v", err)
}
publishErr := make(chan error, 1)
go func() {
defer func() {
if recovered := recover(); recovered != nil {
publishErr <- errors.New("publish panicked")
}
}()
publishErr <- sink.PublishUnified(context.Background(), env)
}()
closeErr := make(chan error, 1)
go func() {
closeErr <- sink.Close()
}()
if err := <-publishErr; !errors.Is(err, ErrAsyncSinkEnqueueTimeout) && !errors.Is(err, ErrAsyncSinkClosed) {
t.Fatalf("concurrent PublishUnified() error = %v, want timeout or closed", err)
}
delegate.release()
if err := <-closeErr; err != nil {
t.Fatalf("Close() error = %v", err)
}
if err := sink.PublishRaw(context.Background(), env); !errors.Is(err, ErrAsyncSinkClosed) {
t.Fatalf("PublishRaw() after Close error = %v, want ErrAsyncSinkClosed", err)
}
}
func TestAsyncSinkCloseUnblocksPublishWhenEnqueueTimeoutIsDisabled(t *testing.T) {
delegate := newBlockingSink()
sink := NewAsyncSink(delegate, AsyncConfig{
QueueSize: 1,
Workers: 1,
EnqueueTimeout: -1,
OperationTimeout: time.Second,
})
env := envelope.FrameEnvelope{Protocol: envelope.ProtocolJT808, Phone: "13307795425"}
if err := sink.PublishRaw(context.Background(), env); err != nil {
t.Fatalf("first PublishRaw() error = %v", err)
}
select {
case <-delegate.rawStarted:
case <-time.After(time.Second):
t.Fatal("delegate raw publish was not started")
}
if err := sink.PublishFields(context.Background(), env); err != nil {
t.Fatalf("second PublishFields() error = %v", err)
}
publishErr := make(chan error, 1)
go func() {
publishErr <- sink.PublishUnified(context.Background(), env)
}()
closeErr := make(chan error, 1)
go func() {
closeErr <- sink.Close()
}()
select {
case err := <-publishErr:
if !errors.Is(err, ErrAsyncSinkClosed) {
t.Fatalf("blocked PublishUnified() error = %v, want ErrAsyncSinkClosed", err)
}
case <-time.After(time.Second):
t.Fatal("blocked PublishUnified() was not released by Close")
}
delegate.release()
select {
case err := <-closeErr:
if err != nil {
t.Fatalf("Close() error = %v", err)
}
case <-time.After(time.Second):
t.Fatal("Close() did not finish after delegate release")
}
}
type blockingSink struct {
rawStarted chan struct{}
releaseRaw chan struct{}

348
go/vehicle-gateway/internal/eventbus/durable_outbox_sink.go Normal file
View File

@@ -0,0 +1,348 @@
package eventbus
import (
"context"
"errors"
"fmt"
"strings"
"sync"
"time"
"lingniu-vehicle-ingest/go/vehicle-gateway/internal/envelope"
"lingniu-vehicle-ingest/go/vehicle-gateway/internal/metrics"
)
type DurableOutboxConfig struct {
Directory string
ReplayBatchSize int
SyncWrites bool
CloseTimeout time.Duration
WALSegmentBytes int64
WALSegmentAge time.Duration
WALAppendQueue int
WALCommitBatch int
WALCommitWait time.Duration
Metrics *metrics.Registry
Name string
OnError func(error)
}
type asyncRecordPublishingSink interface {
Sink
ValidateRecord(durableRecord) error
PublishRecordAsync(durableRecord, func(error)) error
}
// DurableOutboxSink accepts a record only after its WAL commit is durable.
// Publishing is asynchronous; the WAL record remains replayable until the
// broker returns PubAck. Stable event IDs make crash-window replays idempotent.
type DurableOutboxSink struct {
delegate asyncRecordPublishingSink
store *durableOutboxWAL
replayBatchSize int
closeTimeout time.Duration
metrics *metrics.Registry
name string
onError func(error)
acceptMu sync.RWMutex
mu sync.Mutex
closed bool
inflight map[outboxWALRecordRef]struct{}
pending sync.WaitGroup
closeOne sync.Once
closeErr error
}
var ErrDurableOutboxClosed = errors.New("durable outbox is closed")
func NewDurableOutboxSink(delegate Sink, cfg DurableOutboxConfig) (*DurableOutboxSink, error) {
publisher, ok := delegate.(asyncRecordPublishingSink)
if !ok || publisher == nil {
return nil, errors.New("durable outbox delegate must support async record publishing")
}
dir := strings.TrimSpace(cfg.Directory)
if dir == "" {
return nil, errors.New("durable outbox directory is required")
}
if cfg.ReplayBatchSize <= 0 {
cfg.ReplayBatchSize = 1000
}
if cfg.CloseTimeout <= 0 {
cfg.CloseTimeout = 5 * time.Second
}
store, err := newDurableOutboxWAL(durableOutboxWALConfig{
Directory: dir,
SyncWrites: cfg.SyncWrites,
SegmentBytes: cfg.WALSegmentBytes,
SegmentAge: cfg.WALSegmentAge,
AppendQueue: cfg.WALAppendQueue,
CommitBatch: cfg.WALCommitBatch,
CommitInterval: cfg.WALCommitWait,
})
if err != nil {
return nil, err
}
s := &DurableOutboxSink{
delegate: publisher,
store: store,
replayBatchSize: cfg.ReplayBatchSize,
closeTimeout: cfg.CloseTimeout,
metrics: cfg.Metrics,
name: durableMetricName(cfg.Name),
onError: cfg.OnError,
inflight: map[outboxWALRecordRef]struct{}{},
}
s.recordBacklog()
return s, nil
}
func (s *DurableOutboxSink) PublishRaw(ctx context.Context, env envelope.FrameEnvelope) error {
return s.persistAndSubmit(ctx, "raw", env)
}
func (s *DurableOutboxSink) PublishUnified(ctx context.Context, env envelope.FrameEnvelope) error {
return s.persistAndSubmit(ctx, "unified", env)
}
func (s *DurableOutboxSink) PublishFields(ctx context.Context, env envelope.FrameEnvelope) error {
return s.persistAndSubmit(ctx, "fields", env)
}
func (s *DurableOutboxSink) Close() error {
s.closeOne.Do(func() {
s.acceptMu.Lock()
s.mu.Lock()
s.closed = true
s.mu.Unlock()
s.acceptMu.Unlock()
walErr := s.store.Close()
done := make(chan struct{})
go func() {
s.pending.Wait()
close(done)
}()
var waitErr error
select {
case <-done:
case <-time.After(s.closeTimeout):
waitErr = fmt.Errorf("durable outbox close timed out after %s", s.closeTimeout)
s.recordPublish("all", "close_timeout")
}
s.closeErr = errors.Join(walErr, waitErr, s.delegate.Close())
})
return s.closeErr
}
// ReplayOnce claims at most one configured batch. A bounded claim prevents a
// large recovered backlog from creating an unbounded number of PubAck futures.
func (s *DurableOutboxSink) ReplayOnce(ctx context.Context) error {
s.acceptMu.RLock()
defer s.acceptMu.RUnlock()
if s.isClosed() {
return ErrDurableOutboxClosed
}
return s.replay(ctx, s.replayBatchSize)
}
func (s *DurableOutboxSink) ReplayLoop(ctx context.Context, interval time.Duration) {
if interval <= 0 {
interval = time.Second
}
if err := s.ReplayOnce(ctx); err != nil && !errors.Is(err, ErrDurableOutboxClosed) {
s.reportError(err)
}
ticker := time.NewTicker(interval)
defer ticker.Stop()
for {
select {
case <-ctx.Done():
return
case <-ticker.C:
if err := s.ReplayOnce(ctx); err != nil && !errors.Is(err, ErrDurableOutboxClosed) {
s.reportError(err)
}
}
}
}
func (s *DurableOutboxSink) persistAndSubmit(ctx context.Context, kind string, env envelope.FrameEnvelope) error {
s.acceptMu.RLock()
defer s.acceptMu.RUnlock()
if s.isClosed() {
return ErrDurableOutboxClosed
}
record := durableRecord{Kind: kind, Envelope: normalizeDurableEnvelope(env)}
if err := s.delegate.ValidateRecord(record); err != nil {
s.recordPublish(kind, "validation_error")
return err
}
stored, err := s.store.Append(ctx, record)
if err != nil {
s.recordSpool(kind, "error")
return err
}
s.recordSpool(kind, "ok")
s.recordBacklog()
if err := s.submit(stored); err != nil {
// The durable commit is the device-facing acceptance boundary. Broker
// submission errors are surfaced operationally and recovered by replay.
s.reportError(err)
}
return nil
}
func normalizeDurableEnvelope(env envelope.FrameEnvelope) envelope.FrameEnvelope {
if env.EventID == "" {
env.EventID = env.StableEventID()
}
if env.ParseStatus == "" {
env.ParseStatus = envelope.ParseOK
}
return env
}
func (s *DurableOutboxSink) submit(stored storedOutboxRecord) error {
s.mu.Lock()
if s.closed {
s.mu.Unlock()
s.store.Release(stored.Ref)
return ErrDurableOutboxClosed
}
if _, exists := s.inflight[stored.Ref]; exists {
s.mu.Unlock()
return nil
}
s.inflight[stored.Ref] = struct{}{}
s.pending.Add(1)
inflight := len(s.inflight)
s.mu.Unlock()
s.recordInflight(inflight)
err := s.delegate.PublishRecordAsync(stored.Record, func(publishErr error) {
s.complete(stored, publishErr)
})
if err == nil {
s.recordPublish(stored.Record.Kind, "submitted")
return nil
}
s.mu.Lock()
delete(s.inflight, stored.Ref)
inflight = len(s.inflight)
s.mu.Unlock()
s.pending.Done()
s.store.Release(stored.Ref)
s.recordInflight(inflight)
s.recordPublish(stored.Record.Kind, "submit_error")
return fmt.Errorf("submit durable outbox record: %w", err)
}
func (s *DurableOutboxSink) complete(stored storedOutboxRecord, publishErr error) {
defer s.pending.Done()
status := "acked"
if publishErr != nil {
status = "ack_error"
s.store.Release(stored.Ref)
} else if err := s.store.Ack(stored.Ref); err != nil {
publishErr = err
status = "remove_error"
}
s.mu.Lock()
delete(s.inflight, stored.Ref)
inflight := len(s.inflight)
s.mu.Unlock()
s.recordInflight(inflight)
s.recordBacklog()
s.recordPublish(stored.Record.Kind, status)
if publishErr != nil {
s.reportError(publishErr)
}
}
func (s *DurableOutboxSink) replay(ctx context.Context, limit int) error {
if ctx != nil && ctx.Err() != nil {
return ctx.Err()
}
records, err := s.store.ClaimPending(limit)
if err != nil {
return fmt.Errorf("claim durable outbox records: %w", err)
}
for index, stored := range records {
if ctx != nil && ctx.Err() != nil {
for _, remaining := range records[index:] {
s.store.Release(remaining.Ref)
}
return ctx.Err()
}
if err := s.delegate.ValidateRecord(stored.Record); err != nil {
s.store.Release(stored.Ref)
for _, remaining := range records[index+1:] {
s.store.Release(remaining.Ref)
}
s.recordPublish(stored.Record.Kind, "validation_error")
return fmt.Errorf("validate durable outbox replay record: %w", err)
}
if err := s.submit(stored); err != nil && !errors.Is(err, ErrDurableOutboxClosed) {
s.reportError(err)
}
}
return nil
}
func (s *DurableOutboxSink) isClosed() bool {
s.mu.Lock()
defer s.mu.Unlock()
return s.closed
}
func (s *DurableOutboxSink) reportError(err error) {
if err != nil && s.onError != nil {
s.onError(err)
}
}
func (s *DurableOutboxSink) recordSpool(kind string, status string) {
if s.metrics == nil {
return
}
s.metrics.IncCounter("vehicle_durable_spool_records_total", metrics.Labels{
"name": s.name, "kind": kind, "status": status,
})
}
func (s *DurableOutboxSink) recordPublish(kind string, status string) {
if s.metrics == nil {
return
}
s.metrics.IncCounter("vehicle_durable_outbox_publish_total", metrics.Labels{
"name": s.name, "kind": kind, "status": status,
})
}
func (s *DurableOutboxSink) recordInflight(value int) {
if s.metrics == nil {
return
}
s.metrics.SetGauge("vehicle_durable_outbox_inflight", metrics.Labels{"name": s.name}, float64(value))
}
func (s *DurableOutboxSink) recordBacklog() {
if s.metrics == nil {
return
}
count, oldest := s.store.Stats()
labels := metrics.Labels{"name": s.name}
// Keep the legacy gauge during migration because capacity gates already
// consume it; its value now represents durable WAL records, not files.
s.metrics.SetGauge("vehicle_durable_spool_backlog_files", labels, float64(count))
s.metrics.SetGauge("vehicle_durable_outbox_backlog_records", labels, float64(count))
ageSeconds := 0.0
if count > 0 && !oldest.IsZero() {
ageSeconds = time.Since(oldest).Seconds()
if ageSeconds < 0 {
ageSeconds = 0
}
}
s.metrics.SetGauge("vehicle_durable_spool_oldest_age_seconds", labels, ageSeconds)
}

380
go/vehicle-gateway/internal/eventbus/durable_outbox_sink_test.go Normal file
View File

@@ -0,0 +1,380 @@
package eventbus
import (
"context"
"errors"
"strings"
"sync"
"sync/atomic"
"testing"
"time"
"lingniu-vehicle-ingest/go/vehicle-gateway/internal/envelope"
)
func TestDurableOutboxPersistsBeforeAsyncSubmitAndDeletesAfterAck(t *testing.T) {
dir := t.TempDir()
delegate := &outboxAsyncSink{}
var sink *DurableOutboxSink
delegate.onSubmit = func(record durableRecord) {
if got := outboxBacklog(sink); got != 1 {
t.Fatalf("durable backlog visible at submit = %d, want 1", got)
}
if record.Envelope.EventID == "" {
t.Fatal("submitted record must have a stable event id")
}
}
var err error
sink, err = NewDurableOutboxSink(delegate, DurableOutboxConfig{Directory: dir})
if err != nil {
t.Fatalf("NewDurableOutboxSink() error = %v", err)
}
if err := sink.PublishRaw(context.Background(), durableTestEnvelope()); err != nil {
t.Fatalf("PublishRaw() error = %v", err)
}
if got := outboxBacklog(sink); got != 1 {
t.Fatalf("backlog before ack = %d, want 1", got)
}
delegate.completeNext(t, nil)
if got := outboxBacklog(sink); got != 0 {
t.Fatalf("backlog after ack = %d, want 0", got)
}
}
func TestDurableOutboxImmediateSubmitErrorKeepsAcceptedRecord(t *testing.T) {
dir := t.TempDir()
submitErr := errors.New("nats pending limit")
delegate := &outboxAsyncSink{submitErrors: []error{submitErr}}
var reported error
sink, err := NewDurableOutboxSink(delegate, DurableOutboxConfig{
Directory: dir,
OnError: func(err error) {
reported = err
},
})
if err != nil {
t.Fatalf("NewDurableOutboxSink() error = %v", err)
}
if err := sink.PublishRaw(context.Background(), durableTestEnvelope()); err != nil {
t.Fatalf("accepted durable record should hide submit error, got %v", err)
}
if got := outboxBacklog(sink); got != 1 {
t.Fatalf("backlog after submit error = %d, want 1", got)
}
if reported == nil || !strings.Contains(reported.Error(), submitErr.Error()) {
t.Fatalf("reported error = %v", reported)
}
}
func TestDurableOutboxAckErrorIsRetriedAndStableRecordIsRemoved(t *testing.T) {
dir := t.TempDir()
delegate := &outboxAsyncSink{}
sink, err := NewDurableOutboxSink(delegate, DurableOutboxConfig{Directory: dir})
if err != nil {
t.Fatalf("NewDurableOutboxSink() error = %v", err)
}
if err := sink.PublishRaw(context.Background(), durableTestEnvelope()); err != nil {
t.Fatalf("PublishRaw() error = %v", err)
}
delegate.completeNext(t, errors.New("ack timeout"))
if got := outboxBacklog(sink); got != 1 {
t.Fatalf("backlog after ack error = %d, want 1", got)
}
if err := sink.ReplayOnce(context.Background()); err != nil {
t.Fatalf("ReplayOnce() error = %v", err)
}
if got := delegate.recordCount(); got != 2 {
t.Fatalf("async submissions = %d, want 2", got)
}
first, second := delegate.recordAt(0), delegate.recordAt(1)
if first.Envelope.EventID == "" || first.Envelope.EventID != second.Envelope.EventID {
t.Fatalf("replay event ids = %q and %q", first.Envelope.EventID, second.Envelope.EventID)
}
delegate.completeNext(t, nil)
if got := outboxBacklog(sink); got != 0 {
t.Fatalf("backlog after replay ack = %d, want 0", got)
}
}
func TestDurableOutboxReplaysPreexistingRecordAfterRestart(t *testing.T) {
dir := t.TempDir()
first := &outboxAsyncSink{submitErrors: []error{errors.New("nats unavailable")}}
writer, err := NewDurableOutboxSink(first, DurableOutboxConfig{Directory: dir})
if err != nil {
t.Fatalf("create first outbox: %v", err)
}
if err := writer.PublishRaw(context.Background(), durableTestEnvelope()); err != nil {
t.Fatalf("persist restart record: %v", err)
}
if err := writer.Close(); err != nil {
t.Fatalf("close first outbox: %v", err)
}
delegate := &outboxAsyncSink{}
sink, err := NewDurableOutboxSink(delegate, DurableOutboxConfig{Directory: dir, ReplayBatchSize: 10})
if err != nil {
t.Fatalf("NewDurableOutboxSink() error = %v", err)
}
if err := sink.ReplayOnce(context.Background()); err != nil {
t.Fatalf("ReplayOnce() error = %v", err)
}
if got := delegate.recordCount(); got != 1 {
t.Fatalf("replayed records = %d, want 1", got)
}
delegate.completeNext(t, nil)
if got := outboxBacklog(sink); got != 0 {
t.Fatalf("backlog after replay ack = %d, want 0", got)
}
}
func TestDurableOutboxDoesNotResubmitInflightRecord(t *testing.T) {
dir := t.TempDir()
delegate := &outboxAsyncSink{}
sink, err := NewDurableOutboxSink(delegate, DurableOutboxConfig{Directory: dir})
if err != nil {
t.Fatalf("NewDurableOutboxSink() error = %v", err)
}
if err := sink.PublishRaw(context.Background(), durableTestEnvelope()); err != nil {
t.Fatalf("PublishRaw() error = %v", err)
}
if err := sink.ReplayOnce(context.Background()); err != nil {
t.Fatalf("ReplayOnce() error = %v", err)
}
if got := delegate.recordCount(); got != 1 {
t.Fatalf("submissions while inflight = %d, want 1", got)
}
delegate.completeNext(t, nil)
}
func TestDurableOutboxRejectsInvalidRecordBeforePersistence(t *testing.T) {
dir := t.TempDir()
delegate := &outboxAsyncSink{validateErr: errors.New("subject not configured")}
sink, err := NewDurableOutboxSink(delegate, DurableOutboxConfig{Directory: dir})
if err != nil {
t.Fatalf("NewDurableOutboxSink() error = %v", err)
}
err = sink.PublishRaw(context.Background(), durableTestEnvelope())
if err == nil || !strings.Contains(err.Error(), "subject not configured") {
t.Fatalf("PublishRaw() error = %v", err)
}
if got := outboxBacklog(sink); got != 0 {
t.Fatalf("invalid record backlog = %d, want 0", got)
}
if got := delegate.recordCount(); got != 0 {
t.Fatalf("invalid record submissions = %d, want 0", got)
}
}
func TestDurableOutboxCloseWaitsForPendingAck(t *testing.T) {
dir := t.TempDir()
delegate := &outboxAsyncSink{}
sink, err := NewDurableOutboxSink(delegate, DurableOutboxConfig{
Directory: dir,
CloseTimeout: time.Second,
})
if err != nil {
t.Fatalf("NewDurableOutboxSink() error = %v", err)
}
if err := sink.PublishRaw(context.Background(), durableTestEnvelope()); err != nil {
t.Fatalf("PublishRaw() error = %v", err)
}
done := make(chan error, 1)
go func() { done <- sink.Close() }()
select {
case err := <-done:
t.Fatalf("Close() returned before ack: %v", err)
case <-time.After(20 * time.Millisecond):
}
delegate.completeNext(t, nil)
select {
case err := <-done:
if err != nil {
t.Fatalf("Close() error = %v", err)
}
case <-time.After(time.Second):
t.Fatal("Close() did not return after ack")
}
}
func TestDurableOutboxCloseTimeoutRetainsRecord(t *testing.T) {
dir := t.TempDir()
delegate := &outboxAsyncSink{}
sink, err := NewDurableOutboxSink(delegate, DurableOutboxConfig{
Directory: dir,
CloseTimeout: 10 * time.Millisecond,
})
if err != nil {
t.Fatalf("NewDurableOutboxSink() error = %v", err)
}
if err := sink.PublishRaw(context.Background(), durableTestEnvelope()); err != nil {
t.Fatalf("PublishRaw() error = %v", err)
}
err = sink.Close()
if err == nil || !strings.Contains(err.Error(), "close timed out") {
t.Fatalf("Close() error = %v", err)
}
if got := outboxBacklog(sink); got != 1 {
t.Fatalf("backlog after close timeout = %d, want 1", got)
}
delegate.completeNext(t, errors.New("connection closed"))
}
func BenchmarkDurableOutboxPublishRaw(b *testing.B) {
for _, syncWrites := range []bool{false, true} {
name := "fsync_off"
if syncWrites {
name = "fsync_on"
}
b.Run(name, func(b *testing.B) {
sink, err := NewDurableOutboxSink(autoAckOutboxSink{}, DurableOutboxConfig{
Directory: b.TempDir(),
SyncWrites: syncWrites,
})
if err != nil {
b.Fatalf("NewDurableOutboxSink() error = %v", err)
}
var sequence atomic.Uint32
b.ResetTimer()
b.RunParallel(func(pb *testing.PB) {
for pb.Next() {
env := durableTestEnvelope()
env.EventID = ""
env.Sequence = uint16(sequence.Add(1))
if err := sink.PublishRaw(context.Background(), env); err != nil {
b.Fatalf("PublishRaw() error = %v", err)
}
}
})
b.StopTimer()
if err := sink.Close(); err != nil {
b.Fatalf("Close() error = %v", err)
}
})
}
}
func outboxBacklog(sink *DurableOutboxSink) int {
count, _ := sink.store.Stats()
return count
}
type outboxAsyncSink struct {
mu sync.Mutex
validateErr error
submitErrors []error
records []durableRecord
callbacks []func(error)
onSubmit func(durableRecord)
closeCalls int
}
func (s *outboxAsyncSink) ValidateRecord(record durableRecord) error {
if s.validateErr != nil {
return s.validateErr
}
switch record.Kind {
case "raw", "unified", "fields":
return nil
default:
return errUnknownRecordKind(record.Kind)
}
}
func (s *outboxAsyncSink) PublishRecordAsync(record durableRecord, complete func(error)) error {
if s.onSubmit != nil {
s.onSubmit(record)
}
s.mu.Lock()
defer s.mu.Unlock()
if len(s.submitErrors) > 0 {
err := s.submitErrors[0]
s.submitErrors = s.submitErrors[1:]
if err != nil {
return err
}
}
s.records = append(s.records, record)
s.callbacks = append(s.callbacks, complete)
return nil
}
func (s *outboxAsyncSink) PublishRaw(context.Context, envelope.FrameEnvelope) error {
return nil
}
func (s *outboxAsyncSink) PublishUnified(context.Context, envelope.FrameEnvelope) error {
return nil
}
func (s *outboxAsyncSink) PublishFields(context.Context, envelope.FrameEnvelope) error {
return nil
}
func (s *outboxAsyncSink) Close() error {
s.mu.Lock()
s.closeCalls++
s.mu.Unlock()
return nil
}
func (s *outboxAsyncSink) completeNext(t *testing.T, err error) {
t.Helper()
s.mu.Lock()
if len(s.callbacks) == 0 {
s.mu.Unlock()
t.Fatal("no pending async callback")
}
callback := s.callbacks[0]
s.callbacks = s.callbacks[1:]
s.mu.Unlock()
callback(err)
}
func (s *outboxAsyncSink) recordCount() int {
s.mu.Lock()
defer s.mu.Unlock()
return len(s.records)
}
func (s *outboxAsyncSink) recordAt(index int) durableRecord {
s.mu.Lock()
defer s.mu.Unlock()
return s.records[index]
}
type autoAckOutboxSink struct{}
func (autoAckOutboxSink) ValidateRecord(record durableRecord) error {
switch record.Kind {
case "raw", "unified", "fields":
return nil
default:
return errUnknownRecordKind(record.Kind)
}
}
func (autoAckOutboxSink) PublishRecordAsync(_ durableRecord, complete func(error)) error {
complete(nil)
return nil
}
func (autoAckOutboxSink) PublishRaw(context.Context, envelope.FrameEnvelope) error {
return nil
}
func (autoAckOutboxSink) PublishUnified(context.Context, envelope.FrameEnvelope) error {
return nil
}
func (autoAckOutboxSink) PublishFields(context.Context, envelope.FrameEnvelope) error {
return nil
}
func (autoAckOutboxSink) Close() error { return nil }

804
go/vehicle-gateway/internal/eventbus/durable_outbox_wal.go Normal file
View File

@@ -0,0 +1,804 @@
package eventbus
import (
"context"
"encoding/binary"
"encoding/json"
"errors"
"fmt"
"hash/crc32"
"io"
"os"
"path/filepath"
"sort"
"strconv"
"strings"
"sync"
"time"
)
const (
outboxWALMagic = uint32(0x4c4e5731) // LNW1
outboxWALHeaderSize = 12
outboxWALMaxRecordBytes = 16 << 20
defaultWALSegmentBytes = 16 << 20
defaultWALAppendQueue = 100_000
defaultWALCommitBatch = 256
defaultWALCommitInterval = time.Millisecond
defaultWALSegmentAge = 5 * time.Second
)
var ErrDurableOutboxWALClosed = errors.New("durable outbox wal is closed")
type durableOutboxWALConfig struct {
Directory string
SyncWrites bool
SegmentBytes int64
SegmentAge time.Duration
AppendQueue int
CommitBatch int
CommitInterval time.Duration
}
type outboxWALRecordRef struct {
segmentID uint64
index int
}
type storedOutboxRecord struct {
Ref outboxWALRecordRef
Record durableRecord
}
type outboxWALRecordStatus uint8
const (
outboxWALPending outboxWALRecordStatus = iota
outboxWALClaimed
outboxWALAcknowledged
)
type outboxWALRecordState struct {
payloadOffset int64
payloadLength uint32
status outboxWALRecordStatus
}
type outboxWALSegment struct {
id uint64
path string
createdAt time.Time
size int64
closed bool
deleting bool
acked int
records []*outboxWALRecordState
}
type outboxWALAppendRequest struct {
record durableRecord
payload []byte
frame []byte
result chan outboxWALAppendResult
}
type outboxWALAppendResult struct {
stored storedOutboxRecord
err error
}
type durableOutboxWAL struct {
dir string
syncWrites bool
segmentBytes int64
segmentAge time.Duration
commitBatch int
commitInterval time.Duration
mu sync.Mutex
segments []*outboxWALSegment
segmentByID map[uint64]*outboxWALSegment
current *outboxWALSegment
currentFile *os.File
backlog int
fatalErr error
appendMu sync.RWMutex
closed bool
queue chan outboxWALAppendRequest
writerWG sync.WaitGroup
closeOne sync.Once
}
func newDurableOutboxWAL(cfg durableOutboxWALConfig) (*durableOutboxWAL, error) {
dir := strings.TrimSpace(cfg.Directory)
if dir == "" {
return nil, errors.New("durable outbox wal directory is required")
}
if cfg.SegmentBytes <= 0 {
cfg.SegmentBytes = defaultWALSegmentBytes
}
if cfg.SegmentAge <= 0 {
cfg.SegmentAge = defaultWALSegmentAge
}
if cfg.AppendQueue <= 0 {
cfg.AppendQueue = defaultWALAppendQueue
}
if cfg.CommitBatch <= 0 {
cfg.CommitBatch = defaultWALCommitBatch
}
if cfg.CommitInterval <= 0 {
cfg.CommitInterval = defaultWALCommitInterval
}
if err := os.MkdirAll(dir, 0o750); err != nil {
return nil, fmt.Errorf("create durable outbox wal directory: %w", err)
}
w := &durableOutboxWAL{
dir: dir,
syncWrites: cfg.SyncWrites,
segmentBytes: cfg.SegmentBytes,
segmentAge: cfg.SegmentAge,
commitBatch: cfg.CommitBatch,
commitInterval: cfg.CommitInterval,
segmentByID: map[uint64]*outboxWALSegment{},
queue: make(chan outboxWALAppendRequest, cfg.AppendQueue),
}
if err := w.loadSegments(); err != nil {
return nil, err
}
if err := w.createCurrentSegment(); err != nil {
return nil, err
}
w.writerWG.Add(1)
go w.appendLoop()
return w, nil
}
func (w *durableOutboxWAL) Append(ctx context.Context, record durableRecord) (storedOutboxRecord, error) {
payload, err := json.Marshal(record)
if err != nil {
return storedOutboxRecord{}, fmt.Errorf("marshal durable outbox wal record: %w", err)
}
if len(payload) > outboxWALMaxRecordBytes {
return storedOutboxRecord{}, fmt.Errorf("durable outbox wal record is %d bytes, max %d", len(payload), outboxWALMaxRecordBytes)
}
request := outboxWALAppendRequest{
record: record,
payload: payload,
frame: encodeOutboxWALFrame(payload),
result: make(chan outboxWALAppendResult, 1),
}
if ctx == nil {
ctx = context.Background()
}
w.appendMu.RLock()
if w.closed {
w.appendMu.RUnlock()
return storedOutboxRecord{}, ErrDurableOutboxWALClosed
}
select {
case w.queue <- request:
w.appendMu.RUnlock()
case <-ctx.Done():
w.appendMu.RUnlock()
return storedOutboxRecord{}, ctx.Err()
}
// Once admitted to the WAL queue, wait for the durability result even if
// the caller context is cancelled. Otherwise a committed record could be
// left claimed with no publisher responsible for it.
result := <-request.result
return result.stored, result.err
}
func (w *durableOutboxWAL) ClaimPending(limit int) ([]storedOutboxRecord, error) {
if limit <= 0 {
limit = defaultWALCommitBatch
}
w.mu.Lock()
refs := make([]outboxWALRecordRef, 0, limit)
for _, segment := range w.segments {
for index, state := range segment.records {
if state.status != outboxWALPending {
continue
}
state.status = outboxWALClaimed
refs = append(refs, outboxWALRecordRef{segmentID: segment.id, index: index})
if len(refs) >= limit {
break
}
}
if len(refs) >= limit {
break
}
}
w.mu.Unlock()
if len(refs) == 0 {
return nil, nil
}
records, err := w.readClaimedRecords(refs)
if err != nil {
for _, ref := range refs {
w.Release(ref)
}
return nil, err
}
return records, nil
}
func (w *durableOutboxWAL) Release(ref outboxWALRecordRef) {
w.mu.Lock()
defer w.mu.Unlock()
state := w.recordStateLocked(ref)
if state != nil && state.status == outboxWALClaimed {
state.status = outboxWALPending
}
}
func (w *durableOutboxWAL) Ack(ref outboxWALRecordRef) error {
w.mu.Lock()
segment := w.segmentByID[ref.segmentID]
if segment == nil || ref.index < 0 || ref.index >= len(segment.records) {
w.mu.Unlock()
return nil
}
state := segment.records[ref.index]
if state.status == outboxWALAcknowledged {
w.mu.Unlock()
return nil
}
state.status = outboxWALAcknowledged
segment.acked++
if w.backlog > 0 {
w.backlog--
}
shouldDelete := segment.closed && segment.acked == len(segment.records) && !segment.deleting
if shouldDelete {
segment.deleting = true
}
w.mu.Unlock()
if !shouldDelete {
return nil
}
return w.deleteAcknowledgedSegment(segment)
}
func (w *durableOutboxWAL) Stats() (backlog int, oldest time.Time) {
w.mu.Lock()
defer w.mu.Unlock()
backlog = w.backlog
if backlog == 0 {
return backlog, time.Time{}
}
for _, segment := range w.segments {
if segment.acked < len(segment.records) {
return backlog, segment.createdAt
}
}
return backlog, time.Time{}
}
func (w *durableOutboxWAL) Close() error {
w.closeOne.Do(func() {
w.appendMu.Lock()
w.closed = true
close(w.queue)
w.appendMu.Unlock()
w.writerWG.Wait()
})
w.mu.Lock()
err := w.fatalErr
w.mu.Unlock()
return err
}
func (w *durableOutboxWAL) appendLoop() {
defer w.writerWG.Done()
maintenanceInterval := minDuration(w.segmentAge/2, time.Second)
if maintenanceInterval < 10*time.Millisecond {
maintenanceInterval = 10 * time.Millisecond
}
maintenance := time.NewTicker(maintenanceInterval)
defer maintenance.Stop()
for {
select {
case request, ok := <-w.queue:
if !ok {
w.finishWriter()
return
}
batch := w.collectAppendBatch(request)
w.commitAppendBatch(batch)
case <-maintenance.C:
if err := w.rotateIfAged(); err != nil {
w.setFatal(err)
}
}
}
}
func (w *durableOutboxWAL) collectAppendBatch(first outboxWALAppendRequest) []outboxWALAppendRequest {
batch := make([]outboxWALAppendRequest, 0, w.commitBatch)
batch = append(batch, first)
timer := time.NewTimer(w.commitInterval)
defer timer.Stop()
for len(batch) < w.commitBatch {
select {
case request, ok := <-w.queue:
if !ok {
return batch
}
batch = append(batch, request)
case <-timer.C:
return batch
}
}
return batch
}
func (w *durableOutboxWAL) commitAppendBatch(batch []outboxWALAppendRequest) {
if len(batch) == 0 {
return
}
w.mu.Lock()
fatalErr := w.fatalErr
w.mu.Unlock()
if fatalErr != nil {
w.completeAppendErrors(batch, fatalErr)
return
}
for len(batch) > 0 {
if err := w.rotateBeforeAppend(len(batch[0].frame)); err != nil {
w.setFatal(err)
w.completeAppendErrors(batch, err)
return
}
count := w.batchCountForCurrentSegment(batch)
if count <= 0 {
count = 1
}
group := batch[:count]
if err := w.commitAppendGroup(group); err != nil {
w.setFatal(err)
w.completeAppendErrors(batch, err)
return
}
batch = batch[count:]
}
}
func (w *durableOutboxWAL) commitAppendGroup(group []outboxWALAppendRequest) error {
start := w.current.size
totalBytes := 0
for _, request := range group {
totalBytes += len(request.frame)
}
buffer := make([]byte, 0, totalBytes)
for _, request := range group {
buffer = append(buffer, request.frame...)
}
written, err := w.currentFile.Write(buffer)
if err != nil || written != len(buffer) {
if err == nil {
err = io.ErrShortWrite
}
w.rollbackAppend(start)
return fmt.Errorf("append durable outbox wal: %w", err)
}
if w.syncWrites {
if err := w.currentFile.Sync(); err != nil {
w.rollbackAppend(start)
return fmt.Errorf("sync durable outbox wal: %w", err)
}
}
w.mu.Lock()
segment := w.current
offset := start
results := make([]outboxWALAppendResult, 0, len(group))
for _, request := range group {
state := &outboxWALRecordState{
payloadOffset: offset + outboxWALHeaderSize,
payloadLength: uint32(len(request.payload)),
status: outboxWALClaimed,
}
index := len(segment.records)
segment.records = append(segment.records, state)
w.backlog++
results = append(results, outboxWALAppendResult{stored: storedOutboxRecord{
Ref: outboxWALRecordRef{segmentID: segment.id, index: index},
Record: request.record,
}})
offset += int64(len(request.frame))
}
segment.size += int64(len(buffer))
w.mu.Unlock()
for index, request := range group {
request.result <- results[index]
}
return nil
}
func (w *durableOutboxWAL) rollbackAppend(size int64) {
if w.currentFile == nil {
return
}
_ = w.currentFile.Truncate(size)
if w.syncWrites {
_ = w.currentFile.Sync()
}
}
func (w *durableOutboxWAL) rotateBeforeAppend(frameBytes int) error {
if w.current == nil {
return w.createCurrentSegment()
}
if len(w.current.records) == 0 {
return nil
}
tooLarge := w.current.size+int64(frameBytes) > w.segmentBytes
tooOld := time.Since(w.current.createdAt) >= w.segmentAge
if !tooLarge && !tooOld {
return nil
}
return w.rotateCurrentSegment()
}
func (w *durableOutboxWAL) rotateIfAged() error {
w.mu.Lock()
current := w.current
shouldRotate := current != nil && len(current.records) > 0 && time.Since(current.createdAt) >= w.segmentAge
w.mu.Unlock()
if !shouldRotate {
return nil
}
return w.rotateCurrentSegment()
}
func (w *durableOutboxWAL) rotateCurrentSegment() error {
if w.currentFile == nil || w.current == nil {
return w.createCurrentSegment()
}
if w.syncWrites {
if err := w.currentFile.Sync(); err != nil {
return fmt.Errorf("sync closing durable outbox wal segment: %w", err)
}
}
if err := w.currentFile.Close(); err != nil {
return fmt.Errorf("close durable outbox wal segment: %w", err)
}
w.mu.Lock()
old := w.current
old.closed = true
w.current = nil
w.currentFile = nil
deleteOld := old.acked == len(old.records) && !old.deleting
if deleteOld {
old.deleting = true
}
w.mu.Unlock()
if deleteOld {
if err := w.deleteAcknowledgedSegment(old); err != nil {
return err
}
}
return w.createCurrentSegment()
}
func (w *durableOutboxWAL) createCurrentSegment() error {
w.mu.Lock()
lastID := uint64(0)
if len(w.segments) > 0 {
lastID = w.segments[len(w.segments)-1].id
}
w.mu.Unlock()
id := uint64(time.Now().UnixNano())
if id <= lastID {
id = lastID + 1
}
path := filepath.Join(w.dir, outboxWALFileName(id))
file, err := os.OpenFile(path, os.O_CREATE|os.O_EXCL|os.O_RDWR|os.O_APPEND, 0o640)
if err != nil {
return fmt.Errorf("create durable outbox wal segment: %w", err)
}
if w.syncWrites {
if err := syncDirectory(w.dir); err != nil {
_ = file.Close()
return fmt.Errorf("sync durable outbox wal directory after create: %w", err)
}
}
segment := &outboxWALSegment{id: id, path: path, createdAt: time.Now()}
w.mu.Lock()
w.segments = append(w.segments, segment)
w.segmentByID[id] = segment
w.current = segment
w.currentFile = file
w.mu.Unlock()
return nil
}
func (w *durableOutboxWAL) finishWriter() {
if w.currentFile == nil || w.current == nil {
return
}
var finishErr error
if w.syncWrites {
finishErr = w.currentFile.Sync()
}
if closeErr := w.currentFile.Close(); finishErr == nil {
finishErr = closeErr
}
w.mu.Lock()
current := w.current
current.closed = true
w.current = nil
w.currentFile = nil
deleteCurrent := current.acked == len(current.records) && !current.deleting
if deleteCurrent {
current.deleting = true
}
w.mu.Unlock()
if deleteCurrent {
if err := w.deleteAcknowledgedSegment(current); finishErr == nil {
finishErr = err
}
}
if finishErr != nil {
w.setFatal(finishErr)
}
}
func (w *durableOutboxWAL) batchCountForCurrentSegment(batch []outboxWALAppendRequest) int {
remaining := w.segmentBytes - w.current.size
count := 0
for _, request := range batch {
if count > 0 && int64(len(request.frame)) > remaining {
break
}
remaining -= int64(len(request.frame))
count++
if remaining <= 0 {
break
}
}
return count
}
func (w *durableOutboxWAL) completeAppendErrors(batch []outboxWALAppendRequest, err error) {
for _, request := range batch {
request.result <- outboxWALAppendResult{err: err}
}
}
func (w *durableOutboxWAL) readClaimedRecords(refs []outboxWALRecordRef) ([]storedOutboxRecord, error) {
files := map[uint64]*os.File{}
defer func() {
for _, file := range files {
_ = file.Close()
}
}()
records := make([]storedOutboxRecord, 0, len(refs))
for _, ref := range refs {
w.mu.Lock()
segment := w.segmentByID[ref.segmentID]
state := w.recordStateLocked(ref)
w.mu.Unlock()
if segment == nil || state == nil {
return nil, fmt.Errorf("durable outbox wal record reference not found: segment=%d index=%d", ref.segmentID, ref.index)
}
file := files[segment.id]
if file == nil {
opened, err := os.Open(segment.path)
if err != nil {
return nil, fmt.Errorf("open durable outbox wal segment for replay: %w", err)
}
files[segment.id] = opened
file = opened
}
payload := make([]byte, state.payloadLength)
if _, err := file.ReadAt(payload, state.payloadOffset); err != nil {
return nil, fmt.Errorf("read durable outbox wal record: %w", err)
}
var record durableRecord
if err := json.Unmarshal(payload, &record); err != nil {
return nil, fmt.Errorf("decode durable outbox wal record: %w", err)
}
records = append(records, storedOutboxRecord{Ref: ref, Record: record})
}
return records, nil
}
func (w *durableOutboxWAL) loadSegments() error {
paths, err := filepath.Glob(filepath.Join(w.dir, "outbox-*.wal"))
if err != nil {
return fmt.Errorf("list durable outbox wal segments: %w", err)
}
sort.Strings(paths)
for _, path := range paths {
segment, err := loadOutboxWALSegment(path, w.syncWrites)
if err != nil {
return err
}
if len(segment.records) == 0 {
if err := os.Remove(path); err != nil && !os.IsNotExist(err) {
return fmt.Errorf("remove empty durable outbox wal segment: %w", err)
}
if w.syncWrites {
if err := syncDirectory(w.dir); err != nil {
return fmt.Errorf("sync durable outbox wal directory after removing empty segment: %w", err)
}
}
continue
}
w.segments = append(w.segments, segment)
w.segmentByID[segment.id] = segment
w.backlog += len(segment.records)
}
return nil
}
func loadOutboxWALSegment(path string, syncWrites bool) (*outboxWALSegment, error) {
id, err := parseOutboxWALFileName(filepath.Base(path))
if err != nil {
return nil, err
}
file, err := os.OpenFile(path, os.O_RDWR, 0)
if err != nil {
return nil, fmt.Errorf("open durable outbox wal segment: %w", err)
}
defer file.Close()
info, err := file.Stat()
if err != nil {
return nil, fmt.Errorf("stat durable outbox wal segment: %w", err)
}
segment := &outboxWALSegment{
id: id,
path: path,
createdAt: info.ModTime(),
closed: true,
}
offset := int64(0)
header := make([]byte, outboxWALHeaderSize)
for offset < info.Size() {
n, readErr := file.ReadAt(header, offset)
if readErr != nil {
if readErr == io.EOF && n < outboxWALHeaderSize {
if err := truncateOutboxWALTail(file, offset, syncWrites); err != nil {
return nil, err
}
break
}
return nil, fmt.Errorf("read durable outbox wal header at %d: %w", offset, readErr)
}
if binary.BigEndian.Uint32(header[0:4]) != outboxWALMagic {
return nil, fmt.Errorf("durable outbox wal corruption in %s at offset %d: invalid magic", path, offset)
}
length := binary.BigEndian.Uint32(header[4:8])
checksum := binary.BigEndian.Uint32(header[8:12])
if length == 0 || length > outboxWALMaxRecordBytes {
return nil, fmt.Errorf("durable outbox wal corruption in %s at offset %d: invalid length %d", path, offset, length)
}
frameEnd := offset + outboxWALHeaderSize + int64(length)
if frameEnd > info.Size() {
if err := truncateOutboxWALTail(file, offset, syncWrites); err != nil {
return nil, err
}
break
}
payload := make([]byte, length)
if _, err := file.ReadAt(payload, offset+outboxWALHeaderSize); err != nil {
return nil, fmt.Errorf("read durable outbox wal payload at %d: %w", offset, err)
}
if crc32.ChecksumIEEE(payload) != checksum {
return nil, fmt.Errorf("durable outbox wal corruption in %s at offset %d: checksum mismatch", path, offset)
}
segment.records = append(segment.records, &outboxWALRecordState{
payloadOffset: offset + outboxWALHeaderSize,
payloadLength: length,
status: outboxWALPending,
})
offset = frameEnd
}
segment.size = offset
return segment, nil
}
func truncateOutboxWALTail(file *os.File, size int64, syncWrites bool) error {
if err := file.Truncate(size); err != nil {
return fmt.Errorf("truncate incomplete durable outbox wal tail: %w", err)
}
if syncWrites {
if err := file.Sync(); err != nil {
return fmt.Errorf("sync truncated durable outbox wal tail: %w", err)
}
}
return nil
}
func (w *durableOutboxWAL) deleteAcknowledgedSegment(segment *outboxWALSegment) error {
err := os.Remove(segment.path)
if os.IsNotExist(err) {
err = nil
}
if err == nil && w.syncWrites {
err = syncDirectory(w.dir)
}
w.mu.Lock()
defer w.mu.Unlock()
if err != nil {
segment.deleting = false
return fmt.Errorf("delete acknowledged durable outbox wal segment: %w", err)
}
delete(w.segmentByID, segment.id)
for index, candidate := range w.segments {
if candidate != segment {
continue
}
w.segments = append(w.segments[:index], w.segments[index+1:]...)
break
}
return nil
}
func (w *durableOutboxWAL) recordStateLocked(ref outboxWALRecordRef) *outboxWALRecordState {
segment := w.segmentByID[ref.segmentID]
if segment == nil || ref.index < 0 || ref.index >= len(segment.records) {
return nil
}
return segment.records[ref.index]
}
func (w *durableOutboxWAL) setFatal(err error) {
if err == nil {
return
}
w.mu.Lock()
if w.fatalErr == nil {
w.fatalErr = err
}
w.mu.Unlock()
}
func encodeOutboxWALFrame(payload []byte) []byte {
frame := make([]byte, outboxWALHeaderSize+len(payload))
binary.BigEndian.PutUint32(frame[0:4], outboxWALMagic)
binary.BigEndian.PutUint32(frame[4:8], uint32(len(payload)))
binary.BigEndian.PutUint32(frame[8:12], crc32.ChecksumIEEE(payload))
copy(frame[outboxWALHeaderSize:], payload)
return frame
}
func outboxWALFileName(id uint64) string {
return fmt.Sprintf("outbox-%020d.wal", id)
}
func parseOutboxWALFileName(name string) (uint64, error) {
if !strings.HasPrefix(name, "outbox-") || !strings.HasSuffix(name, ".wal") {
return 0, fmt.Errorf("invalid durable outbox wal file name %q", name)
}
value := strings.TrimSuffix(strings.TrimPrefix(name, "outbox-"), ".wal")
id, err := strconv.ParseUint(value, 10, 64)
if err != nil {
return 0, fmt.Errorf("parse durable outbox wal file name %q: %w", name, err)
}
return id, nil
}
func syncDirectory(dir string) error {
handle, err := os.Open(dir)
if err != nil {
return err
}
syncErr := handle.Sync()
closeErr := handle.Close()
return errors.Join(syncErr, closeErr)
}
func minDuration(left, right time.Duration) time.Duration {
if left <= 0 {
return right
}
if right <= 0 || left < right {
return left
}
return right
}

229
go/vehicle-gateway/internal/eventbus/durable_outbox_wal_test.go Normal file
View File

@@ -0,0 +1,229 @@
package eventbus
import (
"context"
"encoding/json"
"errors"
"os"
"path/filepath"
"strings"
"sync"
"testing"
"time"
)
func TestDurableOutboxWALConcurrentGroupCommitRecoversEveryRecord(t *testing.T) {
dir := t.TempDir()
wal, err := newDurableOutboxWAL(durableOutboxWALConfig{
Directory: dir,
SyncWrites: true,
CommitBatch: 64,
CommitInterval: 5 * time.Millisecond,
})
if err != nil {
t.Fatalf("new WAL: %v", err)
}
const total = 256
refs := make(chan outboxWALRecordRef, total)
errs := make(chan error, total)
var workers sync.WaitGroup
for i := 0; i < total; i++ {
workers.Add(1)
go func(sequence int) {
defer workers.Done()
record := walTestRecord(sequence)
stored, appendErr := wal.Append(context.Background(), record)
if appendErr != nil {
errs <- appendErr
return
}
refs <- stored.Ref
}(i)
}
workers.Wait()
close(errs)
for appendErr := range errs {
t.Fatalf("concurrent append: %v", appendErr)
}
close(refs)
for ref := range refs {
wal.Release(ref)
}
if got, _ := wal.Stats(); got != total {
t.Fatalf("backlog before restart = %d, want %d", got, total)
}
if err := wal.Close(); err != nil {
t.Fatalf("close first WAL: %v", err)
}
recovered, err := newDurableOutboxWAL(durableOutboxWALConfig{Directory: dir, SyncWrites: true})
if err != nil {
t.Fatalf("reopen WAL: %v", err)
}
records, err := recovered.ClaimPending(total + 1)
if err != nil {
t.Fatalf("claim recovered records: %v", err)
}
if len(records) != total {
t.Fatalf("recovered records = %d, want %d", len(records), total)
}
seen := make(map[string]struct{}, total)
for _, record := range records {
seen[record.Record.Envelope.EventID] = struct{}{}
if err := recovered.Ack(record.Ref); err != nil {
t.Fatalf("ack recovered record: %v", err)
}
}
if len(seen) != total {
t.Fatalf("unique recovered event ids = %d, want %d", len(seen), total)
}
if got, _ := recovered.Stats(); got != 0 {
t.Fatalf("backlog after ack = %d, want 0", got)
}
if err := recovered.Close(); err != nil {
t.Fatalf("close recovered WAL: %v", err)
}
}
func TestDurableOutboxWALTruncatesIncompleteTrailingFrame(t *testing.T) {
dir := t.TempDir()
first := encodedWALTestFrame(t, 1)
second := encodedWALTestFrame(t, 2)
path := filepath.Join(dir, outboxWALFileName(1))
payload := append(append([]byte{}, first...), second[:len(second)/2]...)
if err := os.WriteFile(path, payload, 0o640); err != nil {
t.Fatalf("write incomplete WAL: %v", err)
}
wal, err := newDurableOutboxWAL(durableOutboxWALConfig{Directory: dir, SyncWrites: true})
if err != nil {
t.Fatalf("recover incomplete WAL: %v", err)
}
info, err := os.Stat(path)
if err != nil {
t.Fatalf("stat recovered segment: %v", err)
}
if got, want := info.Size(), int64(len(first)); got != want {
t.Fatalf("truncated size = %d, want %d", got, want)
}
if got, _ := wal.Stats(); got != 1 {
t.Fatalf("recovered backlog = %d, want 1", got)
}
if err := wal.Close(); err != nil {
t.Fatalf("close WAL: %v", err)
}
}
func TestDurableOutboxWALRejectsChecksumCorruption(t *testing.T) {
dir := t.TempDir()
frame := encodedWALTestFrame(t, 1)
frame[len(frame)-1] ^= 0xff
path := filepath.Join(dir, outboxWALFileName(1))
if err := os.WriteFile(path, frame, 0o640); err != nil {
t.Fatalf("write corrupt WAL: %v", err)
}
_, err := newDurableOutboxWAL(durableOutboxWALConfig{Directory: dir})
if err == nil || !strings.Contains(err.Error(), "checksum mismatch") {
t.Fatalf("corrupt WAL error = %v", err)
}
}
func TestDurableOutboxWALAckDeletesClosedSegment(t *testing.T) {
dir := t.TempDir()
frameSize := int64(len(encodedWALTestFrame(t, 1)))
wal, err := newDurableOutboxWAL(durableOutboxWALConfig{
Directory: dir,
SegmentBytes: frameSize,
CommitBatch: 1,
})
if err != nil {
t.Fatalf("new WAL: %v", err)
}
first, err := wal.Append(context.Background(), walTestRecord(1))
if err != nil {
t.Fatalf("append first: %v", err)
}
firstPath := filepath.Join(dir, outboxWALFileName(first.Ref.segmentID))
second, err := wal.Append(context.Background(), walTestRecord(2))
if err != nil {
t.Fatalf("append second: %v", err)
}
if first.Ref.segmentID == second.Ref.segmentID {
t.Fatal("second append should rotate to a new segment")
}
if err := wal.Ack(first.Ref); err != nil {
t.Fatalf("ack first: %v", err)
}
if _, err := os.Stat(firstPath); !errors.Is(err, os.ErrNotExist) {
t.Fatalf("closed acknowledged segment still exists: %v", err)
}
wal.Release(second.Ref)
if err := wal.Close(); err != nil {
t.Fatalf("close WAL: %v", err)
}
}
func TestDurableOutboxWALClaimIsBoundedAndNotDuplicated(t *testing.T) {
wal, err := newDurableOutboxWAL(durableOutboxWALConfig{Directory: t.TempDir(), CommitBatch: 1})
if err != nil {
t.Fatalf("new WAL: %v", err)
}
for i := 0; i < 10; i++ {
stored, err := wal.Append(context.Background(), walTestRecord(i))
if err != nil {
t.Fatalf("append %d: %v", i, err)
}
wal.Release(stored.Ref)
}
first, err := wal.ClaimPending(3)
if err != nil || len(first) != 3 {
t.Fatalf("first claim = %d, error = %v", len(first), err)
}
second, err := wal.ClaimPending(3)
if err != nil || len(second) != 3 {
t.Fatalf("second claim = %d, error = %v", len(second), err)
}
claimed := map[outboxWALRecordRef]struct{}{}
for _, record := range append(first, second...) {
if _, duplicate := claimed[record.Ref]; duplicate {
t.Fatalf("record claimed twice: %#v", record.Ref)
}
claimed[record.Ref] = struct{}{}
wal.Release(record.Ref)
}
if err := wal.Close(); err != nil {
t.Fatalf("close WAL: %v", err)
}
}
func TestDurableOutboxWALRejectsAppendAfterClose(t *testing.T) {
wal, err := newDurableOutboxWAL(durableOutboxWALConfig{Directory: t.TempDir()})
if err != nil {
t.Fatalf("new WAL: %v", err)
}
if err := wal.Close(); err != nil {
t.Fatalf("close WAL: %v", err)
}
_, err = wal.Append(context.Background(), walTestRecord(1))
if !errors.Is(err, ErrDurableOutboxWALClosed) {
t.Fatalf("append after close error = %v", err)
}
}
func walTestRecord(sequence int) durableRecord {
env := normalizeDurableEnvelope(durableTestEnvelope())
env.EventID = "wal-event-" + time.Unix(0, int64(sequence)+1).UTC().Format("150405.000000000")
env.Sequence = uint16(sequence)
return durableRecord{Kind: "raw", Envelope: env}
}
func encodedWALTestFrame(t *testing.T, sequence int) []byte {
t.Helper()
payload, err := json.Marshal(walTestRecord(sequence))
if err != nil {
t.Fatalf("marshal WAL test record: %v", err)
}
return encodeOutboxWALFrame(payload)
}

168
go/vehicle-gateway/internal/eventbus/durable_sink.go
View File

@@ -13,16 +13,21 @@ import (
"time"
"lingniu-vehicle-ingest/go/vehicle-gateway/internal/envelope"
"lingniu-vehicle-ingest/go/vehicle-gateway/internal/metrics"
)
type DurableConfig struct {
Directory string
ReplayBatchSize int
Metrics *metrics.Registry
Name string
}
type DurableSink struct {
delegate Sink
dir string
metrics *metrics.Registry
name string
mu sync.Mutex
seq uint64
@@ -50,12 +55,16 @@ func NewDurableSink(delegate Sink, cfg DurableConfig) *DurableSink {
if delegate == nil {
panic("durable delegate sink must not be nil")
}
return &DurableSink{
s := &DurableSink{
delegate: delegate,
dir: strings.TrimSpace(cfg.Directory),
metrics: cfg.Metrics,
name: durableMetricName(cfg.Name),
rawPending: map[string]struct{}{},
replayBatchSize: cfg.ReplayBatchSize,
}
s.recordBacklogAfterReplay()
return s
}
func (s *DurableSink) PublishRaw(ctx context.Context, env envelope.FrameEnvelope) error {
@@ -96,16 +105,29 @@ func (s *DurableSink) ReplayOnce(ctx context.Context) error {
func (s *DurableSink) replay(ctx context.Context, limit int) error {
files, err := durableFiles(s.dir, limit)
if err != nil {
s.recordReplay("list_error", 0)
return err
}
s.recordBacklogAfterReplay()
records := make([]durableRecordFile, 0, len(files))
for _, file := range files {
record, err := readDurableRecord(file)
if err != nil {
return err
s.recordReplay("read_error", 1)
if quarantineErr := quarantineDurableFile(file); quarantineErr != nil {
s.recordReplay("quarantine_error", 1)
return fmt.Errorf("quarantine unreadable durable record %s: read error: %w; quarantine error: %v", file, err, quarantineErr)
}
s.recordReplay("quarantined", 1)
continue
}
records = append(records, durableRecordFile{path: file, record: record})
}
if len(records) == 0 {
s.recordReplay("empty", 0)
s.recordBacklogAfterReplay()
return nil
}
sortDurableRecords(records)
if publisher, ok := s.delegate.(recordPublishingSink); ok {
durableRecords := make([]durableRecord, 0, len(records))
@@ -113,29 +135,41 @@ func (s *DurableSink) replay(ctx context.Context, limit int) error {
durableRecords = append(durableRecords, item.record)
}
if err := publisher.PublishRecords(ctx, durableRecords); err != nil {
s.recordReplay("publish_error", len(records))
s.recordReplayRecords(records, "publish_error")
return err
}
for _, item := range records {
if err := os.Remove(item.path); err != nil {
s.recordReplay("delete_error", len(records))
return err
}
if item.record.Kind == "raw" {
s.clearRawPending(item.record.Envelope)
}
}
s.recordReplay("ok", len(records))
s.recordReplayRecords(records, "ok")
s.recordBacklogAfterReplay()
return nil
}
for _, item := range records {
if err := s.publishRecord(ctx, item.record); err != nil {
s.recordReplay("publish_error", len(records))
s.recordReplayRecord(item.record, "publish_error")
return err
}
if err := os.Remove(item.path); err != nil {
s.recordReplay("delete_error", len(records))
return err
}
if item.record.Kind == "raw" {
s.clearRawPending(item.record.Envelope)
}
s.recordReplayRecord(item.record, "ok")
}
s.recordReplay("ok", len(records))
s.recordBacklogAfterReplay()
return nil
}
@@ -210,7 +244,7 @@ func errUnknownRecordKind(kind string) error {
func (s *DurableSink) spool(kind string, env envelope.FrameEnvelope) error {
if s.dir == "" {
return fmt.Errorf("durable spool directory is empty")
return s.spoolError(kind, fmt.Errorf("durable spool directory is empty"))
}
if env.EventID == "" {
env.EventID = env.StableEventID()
@@ -219,19 +253,24 @@ func (s *DurableSink) spool(kind string, env envelope.FrameEnvelope) error {
env.ParseStatus = envelope.ParseOK
}
if err := os.MkdirAll(s.dir, 0o750); err != nil {
return err
return s.spoolError(kind, err)
}
payload, err := json.Marshal(durableRecord{Kind: kind, Envelope: env})
if err != nil {
return err
return s.spoolError(kind, err)
}
name := s.nextFileName(env, kind)
path := filepath.Join(s.dir, name)
tmp := path + ".tmp"
if err := os.WriteFile(tmp, payload, 0o640); err != nil {
return err
return s.spoolError(kind, err)
}
return os.Rename(tmp, path)
if err := os.Rename(tmp, path); err != nil {
return s.spoolError(kind, err)
}
s.recordSpool(kind, "ok")
s.recordBacklogAfterReplay()
return nil
}
func (s *DurableSink) nextFileName(env envelope.FrameEnvelope, kind string) string {
@@ -273,6 +312,16 @@ func readDurableRecord(path string) (durableRecord, error) {
return record, json.Unmarshal(payload, &record)
}
func quarantineDurableFile(path string) error {
target := path + ".bad"
if _, err := os.Stat(target); err == nil {
target = fmt.Sprintf("%s.%d.bad", path, time.Now().UnixNano())
} else if !os.IsNotExist(err) {
return err
}
return os.Rename(path, target)
}
func durableFiles(dir string, limit int) ([]string, error) {
if limit > 0 {
handle, err := os.Open(dir)
@@ -330,3 +379,108 @@ func durableFilesFromReader(dir string, limit int, reader durableNameReader) ([]
func errorsIsEOF(err error) bool {
return err == io.EOF
}
func durableMetricName(name string) string {
name = strings.TrimSpace(name)
if name == "" {
return "default"
}
return name
}
func (s *DurableSink) spoolError(kind string, err error) error {
s.recordSpool(kind, "error")
return err
}
func (s *DurableSink) recordSpool(kind string, status string) {
if s == nil || s.metrics == nil {
return
}
s.metrics.IncCounter("vehicle_durable_spool_records_total", metrics.Labels{
"name": s.name,
"kind": kind,
"status": status,
})
}
func (s *DurableSink) recordReplay(status string, records int) {
if s == nil || s.metrics == nil {
return
}
labels := metrics.Labels{"name": s.name, "status": status}
s.metrics.IncCounter("vehicle_durable_spool_replay_total", labels)
if records > 0 {
s.metrics.AddCounter("vehicle_durable_spool_replay_records_total", metrics.Labels{
"name": s.name,
"kind": "all",
"status": status,
}, float64(records))
}
}
func (s *DurableSink) recordReplayRecords(records []durableRecordFile, status string) {
for _, item := range records {
s.recordReplayRecord(item.record, status)
}
}
func (s *DurableSink) recordReplayRecord(record durableRecord, status string) {
if s == nil || s.metrics == nil {
return
}
s.metrics.IncCounter("vehicle_durable_spool_replay_records_total", metrics.Labels{
"name": s.name,
"kind": record.Kind,
"status": status,
})
}
func (s *DurableSink) recordBacklogAfterReplay() {
if s == nil || s.metrics == nil {
return
}
if strings.TrimSpace(s.dir) == "" {
s.recordBacklog(0, 0)
return
}
files, oldestAge, err := durableBacklogStats(s.dir, time.Now())
if err != nil {
return
}
s.recordBacklog(files, oldestAge)
}
func (s *DurableSink) recordBacklog(files int, oldestAge time.Duration) {
if s == nil || s.metrics == nil {
return
}
s.metrics.SetGauge("vehicle_durable_spool_backlog_files", metrics.Labels{"name": s.name}, float64(files))
ageSeconds := 0.0
if files > 0 && oldestAge > 0 {
ageSeconds = oldestAge.Seconds()
}
s.metrics.SetGauge("vehicle_durable_spool_oldest_age_seconds", metrics.Labels{"name": s.name}, ageSeconds)
}
func durableBacklogStats(dir string, now time.Time) (count int, oldestAge time.Duration, err error) {
files, err := durableFiles(dir, 0)
if err != nil {
return 0, 0, err
}
for _, file := range files {
info, statErr := os.Stat(file)
if statErr != nil {
return 0, 0, statErr
}
age := now.Sub(info.ModTime())
if age < 0 {
age = 0
}
if count == 0 || age > oldestAge {
oldestAge = age
}
count++
}
return count, oldestAge, nil
}

152
go/vehicle-gateway/internal/eventbus/durable_sink_test.go
View File

@@ -8,8 +8,10 @@ import (
"os"
"path/filepath"
"testing"
"time"
"lingniu-vehicle-ingest/go/vehicle-gateway/internal/envelope"
"lingniu-vehicle-ingest/go/vehicle-gateway/internal/metrics"
)
func TestDurableSinkSpoolsUnifiedWhenRawWasSpooled(t *testing.T) {
@@ -167,6 +169,156 @@ func TestDurableSinkReplayUsesBatchPublisherWhenAvailable(t *testing.T) {
}
}
func TestDurableSinkRecordsSpoolAndReplayMetrics(t *testing.T) {
dir := t.TempDir()
registry := metrics.NewRegistry()
delegate := &scriptedSink{rawErrors: []error{errSpoolTest}}
sink := NewDurableSink(delegate, DurableConfig{Directory: dir, Metrics: registry, Name: "nats"})
env := durableTestEnvelope()
if err := sink.PublishRaw(context.Background(), env); err != nil {
t.Fatalf("PublishRaw() error = %v", err)
}
text := registry.Render()
for _, want := range []string{
`vehicle_durable_spool_records_total{kind="raw",name="nats",status="ok"} 1`,
`vehicle_durable_spool_backlog_files{name="nats"} 1`,
`vehicle_durable_spool_oldest_age_seconds{name="nats"}`,
} {
if !containsString(text, want) {
t.Fatalf("spool metric missing %s:\n%s", want, text)
}
}
delegate.rawErrors = nil
if err := sink.ReplayOnce(context.Background()); err != nil {
t.Fatalf("ReplayOnce() error = %v", err)
}
text = registry.Render()
for _, want := range []string{
`vehicle_durable_spool_replay_total{name="nats",status="ok"} 1`,
`vehicle_durable_spool_replay_records_total{kind="all",name="nats",status="ok"} 1`,
`vehicle_durable_spool_replay_records_total{kind="raw",name="nats",status="ok"} 1`,
`vehicle_durable_spool_backlog_files{name="nats"} 0`,
`vehicle_durable_spool_oldest_age_seconds{name="nats"} 0`,
} {
if !containsString(text, want) {
t.Fatalf("replay metric missing %s:\n%s", want, text)
}
}
}
func TestDurableSinkInitializesBacklogMetrics(t *testing.T) {
dir := t.TempDir()
registry := metrics.NewRegistry()
_ = NewDurableSink(&scriptedSink{}, DurableConfig{Directory: dir, Metrics: registry, Name: "nats"})
text := registry.Render()
for _, want := range []string{
`vehicle_durable_spool_backlog_files{name="nats"} 0`,
`vehicle_durable_spool_oldest_age_seconds{name="nats"} 0`,
} {
if !containsString(text, want) {
t.Fatalf("initial spool metric missing %s:\n%s", want, text)
}
}
}
func TestDurableSinkRecordsReplayPublishErrorMetrics(t *testing.T) {
dir := t.TempDir()
registry := metrics.NewRegistry()
env := durableTestEnvelope()
writeDurableRecord(t, filepath.Join(dir, "0001-raw.json"), durableRecord{Kind: "raw", Envelope: env})
delegate := &scriptedSink{rawErrors: []error{errSpoolTest}}
sink := NewDurableSink(delegate, DurableConfig{Directory: dir, Metrics: registry, Name: "kafka"})
if err := sink.ReplayOnce(context.Background()); err == nil {
t.Fatal("ReplayOnce() error = nil, want delegate publish error")
}
text := registry.Render()
for _, want := range []string{
`vehicle_durable_spool_replay_total{name="kafka",status="publish_error"} 1`,
`vehicle_durable_spool_replay_records_total{kind="all",name="kafka",status="publish_error"} 1`,
`vehicle_durable_spool_replay_records_total{kind="raw",name="kafka",status="publish_error"} 1`,
`vehicle_durable_spool_backlog_files{name="kafka"} 1`,
} {
if !containsString(text, want) {
t.Fatalf("replay error metric missing %s:\n%s", want, text)
}
}
if files := spoolFiles(t, dir); len(files) != 1 {
t.Fatalf("failed replay should keep spool file, files=%#v", files)
}
}
func TestDurableBacklogStatsCountsAllFilesAndOldestAge(t *testing.T) {
dir := t.TempDir()
env := durableTestEnvelope()
now := time.Date(2026, 7, 12, 15, 0, 0, 0, time.UTC)
oldFile := filepath.Join(dir, "0001-raw.json")
newFile := filepath.Join(dir, "0002-fields.json")
writeDurableRecord(t, oldFile, durableRecord{Kind: "raw", Envelope: env})
writeDurableRecord(t, newFile, durableRecord{Kind: "fields", Envelope: env})
if err := os.Chtimes(oldFile, now.Add(-10*time.Minute), now.Add(-10*time.Minute)); err != nil {
t.Fatalf("chtimes old file: %v", err)
}
if err := os.Chtimes(newFile, now.Add(-30*time.Second), now.Add(-30*time.Second)); err != nil {
t.Fatalf("chtimes new file: %v", err)
}
count, oldestAge, err := durableBacklogStats(dir, now)
if err != nil {
t.Fatalf("durableBacklogStats() error = %v", err)
}
if count != 2 {
t.Fatalf("count = %d, want 2", count)
}
if oldestAge != 10*time.Minute {
t.Fatalf("oldestAge = %s, want 10m", oldestAge)
}
}
func TestDurableSinkQuarantinesBadRecordAndReplaysRemainingFiles(t *testing.T) {
dir := t.TempDir()
registry := metrics.NewRegistry()
env := durableTestEnvelope()
if err := os.WriteFile(filepath.Join(dir, "0001-bad.json"), []byte("{bad json"), 0o640); err != nil {
t.Fatalf("write bad durable record: %v", err)
}
writeDurableRecord(t, filepath.Join(dir, "0002-raw.json"), durableRecord{Kind: "raw", Envelope: env})
delegate := &scriptedSink{}
sink := NewDurableSink(delegate, DurableConfig{Directory: dir, Metrics: registry, Name: "nats"})
if err := sink.ReplayOnce(context.Background()); err != nil {
t.Fatalf("ReplayOnce() error = %v", err)
}
if delegate.rawCalls != 1 {
t.Fatalf("raw calls = %d, want 1", delegate.rawCalls)
}
if files := spoolFiles(t, dir); len(files) != 0 {
t.Fatalf("normal spool files after replay = %#v, want none", files)
}
badFiles, err := filepath.Glob(filepath.Join(dir, "*.bad"))
if err != nil {
t.Fatalf("glob bad files: %v", err)
}
if len(badFiles) != 1 {
t.Fatalf("bad files = %#v, want one quarantined file", badFiles)
}
text := registry.Render()
for _, want := range []string{
`vehicle_durable_spool_replay_total{name="nats",status="read_error"} 1`,
`vehicle_durable_spool_replay_total{name="nats",status="quarantined"} 1`,
`vehicle_durable_spool_replay_total{name="nats",status="ok"} 1`,
`vehicle_durable_spool_replay_records_total{kind="raw",name="nats",status="ok"} 1`,
`vehicle_durable_spool_backlog_files{name="nats"} 0`,
} {
if !containsString(text, want) {
t.Fatalf("quarantine metric missing %s:\n%s", want, text)
}
}
}
func TestDurableFilesFromReaderStopsAfterLimitedJSONBatch(t *testing.T) {
reader := &fakeNameReader{
batches: [][]string{

30
go/vehicle-gateway/internal/eventbus/kafka_retry.go Normal file
View File

@@ -0,0 +1,30 @@
package eventbus
import "github.com/segmentio/kafka-go"
// MessagesAfterCommittedPrefixes keeps every fetched message that is not
// covered by the highest committed offset for its topic partition. This is
// used when a batch partially succeeds: later poison or valid messages must
// remain in memory until the failed offset ahead of them is durably handled.
func MessagesAfterCommittedPrefixes(messages []kafka.Message, committed []kafka.Message) []kafka.Message {
type partitionKey struct {
topic string
partition int
}
highest := make(map[partitionKey]int64, len(committed))
for _, message := range committed {
key := partitionKey{topic: message.Topic, partition: message.Partition}
if offset, ok := highest[key]; !ok || message.Offset > offset {
highest[key] = message.Offset
}
}
remaining := make([]kafka.Message, 0, len(messages))
for _, message := range messages {
key := partitionKey{topic: message.Topic, partition: message.Partition}
if offset, ok := highest[key]; ok && message.Offset <= offset {
continue
}
remaining = append(remaining, message)
}
return remaining
}

34
go/vehicle-gateway/internal/eventbus/kafka_retry_test.go Normal file
View File

@@ -0,0 +1,34 @@
package eventbus
import (
"testing"
"github.com/segmentio/kafka-go"
)
func TestMessagesAfterCommittedPrefixesKeepsPartitionGaps(t *testing.T) {
messages := []kafka.Message{
{Topic: "raw", Partition: 0, Offset: 10},
{Topic: "raw", Partition: 1, Offset: 20},
{Topic: "raw", Partition: 0, Offset: 11},
{Topic: "raw", Partition: 1, Offset: 21},
{Topic: "raw", Partition: 0, Offset: 12},
}
committed := []kafka.Message{
{Topic: "raw", Partition: 0, Offset: 10},
{Topic: "raw", Partition: 1, Offset: 21},
}
remaining := MessagesAfterCommittedPrefixes(messages, committed)
if len(remaining) != 2 || remaining[0].Partition != 0 || remaining[0].Offset != 11 || remaining[1].Offset != 12 {
t.Fatalf("remaining = %#v", remaining)
}
}
func TestMessagesAfterCommittedPrefixesWithoutCommitKeepsWholeBatch(t *testing.T) {
messages := []kafka.Message{{Topic: "raw", Partition: 0, Offset: 10}}
remaining := MessagesAfterCommittedPrefixes(messages, nil)
if len(remaining) != 1 || remaining[0].Offset != 10 {
t.Fatalf("remaining = %#v", remaining)
}
}

36
go/vehicle-gateway/internal/eventbus/kafka_sink.go
View File

@@ -41,16 +41,38 @@ func NewKafkaSink(cfg KafkaConfig) (*KafkaSink, error) {
if len(cfg.Brokers) == 0 {
return nil, errors.New("kafka brokers are required")
}
if err := ValidateKafkaConfig(cfg); err != nil {
return nil, err
}
rawTopics, fieldsTopics := kafkaTopicMaps(cfg)
return newKafkaSinkWithWriter(&kafka.Writer{
Addr: kafka.TCP(cfg.Brokers...),
Balancer: &kafka.Hash{},
AllowAutoTopicCreation: false,
RequiredAcks: kafka.RequireAll,
Async: false,
}, cfg), nil
}, KafkaConfig{
RawTopics: rawTopics,
FieldsTopics: fieldsTopics,
UnifiedTopic: cfg.UnifiedTopic,
}), nil
}
func ValidateKafkaConfig(cfg KafkaConfig) error {
rawTopics, fieldsTopics := kafkaTopicMaps(cfg)
return topics.ValidateKafkaRawFields(protocolTopicLabels(rawTopics), protocolTopicLabels(fieldsTopics))
}
func newKafkaSinkWithWriter(writer kafkaWriter, cfg KafkaConfig) *KafkaSink {
rawTopics, fieldsTopics := kafkaTopicMaps(cfg)
unifiedTopic := cfg.UnifiedTopic
if unifiedTopic == "" {
unifiedTopic = topics.Unified
}
return &KafkaSink{writer: writer, rawTopics: rawTopics, fieldsTopics: fieldsTopics, unifiedTopic: unifiedTopic}
}
func kafkaTopicMaps(cfg KafkaConfig) (map[envelope.Protocol]string, map[envelope.Protocol]string) {
rawTopics := map[envelope.Protocol]string{
envelope.ProtocolGB32960: topics.RawGB32960,
envelope.ProtocolJT808: topics.RawJT808,
@@ -71,11 +93,15 @@ func newKafkaSinkWithWriter(writer kafkaWriter, cfg KafkaConfig) *KafkaSink {
fieldsTopics[protocol] = topic
}
}
unifiedTopic := cfg.UnifiedTopic
if unifiedTopic == "" {
unifiedTopic = topics.Unified
return rawTopics, fieldsTopics
}
func protocolTopicLabels(values map[envelope.Protocol]string) map[string]string {
out := make(map[string]string, len(values))
for protocol, topic := range values {
out[string(protocol)] = topic
}
return &KafkaSink{writer: writer, rawTopics: rawTopics, fieldsTopics: fieldsTopics, unifiedTopic: unifiedTopic}
return out
}
func (s *KafkaSink) PublishRaw(ctx context.Context, env envelope.FrameEnvelope) error {

44
go/vehicle-gateway/internal/eventbus/kafka_sink_test.go
View File

@@ -4,6 +4,7 @@ import (
"context"
"encoding/json"
"reflect"
"strings"
"testing"
"github.com/segmentio/kafka-go"
@@ -112,6 +113,49 @@ func TestNewKafkaSinkUsesProductionDeliveryGuarantees(t *testing.T) {
}
}
func TestValidateKafkaConfigRejectsRawFieldsTopicOverlap(t *testing.T) {
err := ValidateKafkaConfig(KafkaConfig{
RawTopics: map[envelope.Protocol]string{
envelope.ProtocolJT808: "vehicle.raw.go.jt808.v1",
},
FieldsTopics: map[envelope.Protocol]string{
envelope.ProtocolJT808: "vehicle.raw.go.jt808.v1",
},
})
if err == nil {
t.Fatal("ValidateKafkaConfig() error = nil, want overlap rejection")
}
if !strings.Contains(err.Error(), "fields kafka topic") {
t.Fatalf("error = %q, want fields topic family hint", err)
}
}
func TestValidateKafkaConfigRejectsKnownProtocolTopicMismatch(t *testing.T) {
err := ValidateKafkaConfig(KafkaConfig{
RawTopics: map[envelope.Protocol]string{
envelope.ProtocolJT808: "vehicle.raw.go.gb32960.v1",
},
})
if err == nil {
t.Fatal("ValidateKafkaConfig() error = nil, want raw protocol mismatch")
}
if !strings.Contains(err.Error(), "must match protocol") {
t.Fatalf("error = %q, want protocol mismatch hint", err)
}
err = ValidateKafkaConfig(KafkaConfig{
FieldsTopics: map[envelope.Protocol]string{
envelope.ProtocolYutongMQTT: "vehicle.fields.go.jt808.v1",
},
})
if err == nil {
t.Fatal("ValidateKafkaConfig() error = nil, want fields protocol mismatch")
}
if !strings.Contains(err.Error(), "must match protocol") {
t.Fatalf("error = %q, want protocol mismatch hint", err)
}
}
type recordingWriter struct {
messages []kafka.Message
writeCalls int

183
go/vehicle-gateway/internal/eventbus/nats_sink.go
View File

@@ -4,6 +4,7 @@ import (
"context"
"errors"
"fmt"
"strings"
"time"
"github.com/nats-io/nats.go"
@@ -13,16 +14,19 @@ import (
)
type NATSConfig struct {
URL string
Name string
RawSubjects map[envelope.Protocol]string
FieldsSubjects map[envelope.Protocol]string
UnifiedSubject string
URL string
Name string
RawSubjects map[envelope.Protocol]string
FieldsSubjects map[envelope.Protocol]string
UnifiedSubject string
AsyncMaxPending int
AsyncAckTimeout time.Duration
}
type NATSSink struct {
conn *nats.Conn
publisher natsPublisher
asyncPublisher natsAsyncPublisher
rawSubjects map[envelope.Protocol]string
fieldsSubjects map[envelope.Protocol]string
unifiedSubject string
@@ -34,10 +38,18 @@ type natsPublisher interface {
Publish(context.Context, string, []byte, ...NATSPublishOption) error
}
type natsAsyncPublisher interface {
PublishAsync(string, []byte, ...NATSPublishOption) (nats.PubAckFuture, error)
}
func NewNATSSink(cfg NATSConfig) (*NATSSink, error) {
if cfg.URL == "" {
return nil, errors.New("nats url is required")
}
if err := ValidateNATSConfig(cfg); err != nil {
return nil, err
}
rawSubjects, fieldsSubjects := natsSubjectMaps(cfg)
name := cfg.Name
if name == "" {
name = "lingniu-vehicle-gateway"
@@ -46,17 +58,58 @@ func NewNATSSink(cfg NATSConfig) (*NATSSink, error) {
if err != nil {
return nil, err
}
js, err := conn.JetStream()
var jsOptions []nats.JSOpt
if cfg.AsyncMaxPending > 0 {
jsOptions = append(jsOptions, nats.PublishAsyncMaxPending(cfg.AsyncMaxPending))
}
if cfg.AsyncAckTimeout > 0 {
jsOptions = append(jsOptions, nats.PublishAsyncTimeout(cfg.AsyncAckTimeout))
}
js, err := conn.JetStream(jsOptions...)
if err != nil {
conn.Close()
return nil, err
}
sink := newNATSSinkWithPublisher(natsJetStreamPublisher{js: js}, cfg)
publisher := natsJetStreamPublisher{js: js}
sink := newNATSSinkWithPublishers(publisher, publisher, NATSConfig{
RawSubjects: rawSubjects,
FieldsSubjects: fieldsSubjects,
UnifiedSubject: cfg.UnifiedSubject,
})
sink.conn = conn
return sink, nil
}
func ValidateNATSConfig(cfg NATSConfig) error {
rawSubjects, fieldsSubjects := natsSubjectMaps(cfg)
raw := protocolTopicLabels(rawSubjects)
fields := protocolTopicLabels(fieldsSubjects)
if err := topics.ValidateKnownRawFieldsProtocols(raw, fields, "nats subject"); err != nil {
return err
}
return topics.ValidateRawFieldsDisjoint(raw, fields, "nats subject")
}
func newNATSSinkWithPublisher(publisher natsPublisher, cfg NATSConfig) *NATSSink {
return newNATSSinkWithPublishers(publisher, nil, cfg)
}
func newNATSSinkWithPublishers(publisher natsPublisher, asyncPublisher natsAsyncPublisher, cfg NATSConfig) *NATSSink {
rawSubjects, fieldsSubjects := natsSubjectMaps(cfg)
unifiedSubject := cfg.UnifiedSubject
if unifiedSubject == "" {
unifiedSubject = topics.Unified
}
return &NATSSink{
publisher: publisher,
asyncPublisher: asyncPublisher,
rawSubjects: rawSubjects,
fieldsSubjects: fieldsSubjects,
unifiedSubject: unifiedSubject,
}
}
func natsSubjectMaps(cfg NATSConfig) (map[envelope.Protocol]string, map[envelope.Protocol]string) {
rawSubjects := map[envelope.Protocol]string{
envelope.ProtocolGB32960: topics.RawGB32960,
envelope.ProtocolJT808: topics.RawJT808,
@@ -77,16 +130,7 @@ func newNATSSinkWithPublisher(publisher natsPublisher, cfg NATSConfig) *NATSSink
fieldsSubjects[protocol] = subject
}
}
unifiedSubject := cfg.UnifiedSubject
if unifiedSubject == "" {
unifiedSubject = topics.Unified
}
return &NATSSink{
publisher: publisher,
rawSubjects: rawSubjects,
fieldsSubjects: fieldsSubjects,
unifiedSubject: unifiedSubject,
}
return rawSubjects, fieldsSubjects
}
func (s *NATSSink) PublishRaw(ctx context.Context, env envelope.FrameEnvelope) error {
@@ -94,14 +138,14 @@ func (s *NATSSink) PublishRaw(ctx context.Context, env envelope.FrameEnvelope) e
if !ok || subject == "" {
return fmt.Errorf("raw subject not configured for protocol %s", env.Protocol)
}
return s.publish(ctx, subject, env)
return s.publish(ctx, subject, "raw", env)
}
func (s *NATSSink) PublishUnified(ctx context.Context, env envelope.FrameEnvelope) error {
if s.unifiedSubject == "" {
return errors.New("unified subject is empty")
}
return s.publish(ctx, s.unifiedSubject, env)
return s.publish(ctx, s.unifiedSubject, "unified", env)
}
func (s *NATSSink) PublishFields(ctx context.Context, env envelope.FrameEnvelope) error {
@@ -109,7 +153,62 @@ func (s *NATSSink) PublishFields(ctx context.Context, env envelope.FrameEnvelope
if !ok || subject == "" {
return fmt.Errorf("fields subject not configured for protocol %s", env.Protocol)
}
return s.publish(ctx, subject, env)
return s.publish(ctx, subject, "fields", env)
}
func (s *NATSSink) PublishRecords(ctx context.Context, records []durableRecord) error {
for _, record := range records {
subject, err := s.subjectForRecord(record)
if err != nil {
return err
}
if err := s.publish(ctx, subject, record.Kind, record.Envelope); err != nil {
return err
}
}
return nil
}
func (s *NATSSink) PublishRecordAsync(record durableRecord, complete func(error)) error {
if s == nil || s.asyncPublisher == nil {
return errors.New("nats async publisher is not configured")
}
if complete == nil {
return errors.New("nats async publish completion callback is required")
}
subject, err := s.subjectForRecord(record)
if err != nil {
return err
}
payload, err := record.Envelope.MarshalJSONBytes()
if err != nil {
return err
}
future, err := s.asyncPublisher.PublishAsync(
subject,
payload,
nats.MsgId(natsMessageID(record.Kind, subject, record.Envelope)),
)
if err != nil {
return err
}
go func() {
select {
case <-future.Ok():
complete(nil)
case asyncErr := <-future.Err():
if asyncErr == nil {
asyncErr = errors.New("nats async publish failed without error detail")
}
complete(asyncErr)
}
}()
return nil
}
func (s *NATSSink) ValidateRecord(record durableRecord) error {
_, err := s.subjectForRecord(record)
return err
}
func (s *NATSSink) Close() error {
@@ -121,12 +220,48 @@ func (s *NATSSink) Close() error {
return nil
}
func (s *NATSSink) publish(ctx context.Context, subject string, env envelope.FrameEnvelope) error {
func (s *NATSSink) publish(ctx context.Context, subject string, kind string, env envelope.FrameEnvelope) error {
payload, err := env.MarshalJSONBytes()
if err != nil {
return err
}
return s.publisher.Publish(ctx, subject, payload, nats.MsgId(env.StableEventID()))
return s.publisher.Publish(ctx, subject, payload, nats.MsgId(natsMessageID(kind, subject, env)))
}
func natsMessageID(kind string, subject string, env envelope.FrameEnvelope) string {
kind = strings.TrimSpace(kind)
if kind == "" {
kind = "unknown"
}
subject = strings.TrimSpace(subject)
if subject == "" {
subject = "unknown"
}
return kind + ":" + subject + ":" + env.StableEventID()
}
func (s *NATSSink) subjectForRecord(record durableRecord) (string, error) {
switch record.Kind {
case "raw":
subject, ok := s.rawSubjects[record.Envelope.Protocol]
if !ok || subject == "" {
return "", fmt.Errorf("raw subject not configured for protocol %s", record.Envelope.Protocol)
}
return subject, nil
case "unified":
if s.unifiedSubject == "" {
return "", errors.New("unified subject is empty")
}
return s.unifiedSubject, nil
case "fields":
subject, ok := s.fieldsSubjects[record.Envelope.Protocol]
if !ok || subject == "" {
return "", fmt.Errorf("fields subject not configured for protocol %s", record.Envelope.Protocol)
}
return subject, nil
default:
return "", errUnknownRecordKind(record.Kind)
}
}
type natsJetStreamPublisher struct {
@@ -137,3 +272,7 @@ func (p natsJetStreamPublisher) Publish(ctx context.Context, subject string, dat
_, err := p.js.Publish(subject, data, append(opts, nats.Context(ctx))...)
return err
}
func (p natsJetStreamPublisher) PublishAsync(subject string, data []byte, opts ...NATSPublishOption) (nats.PubAckFuture, error) {
return p.js.PublishAsync(subject, data, opts...)
}

222
go/vehicle-gateway/internal/eventbus/nats_sink_test.go
View File

@@ -3,7 +3,12 @@ package eventbus
import (
"context"
"encoding/json"
"errors"
"strings"
"testing"
"time"
"github.com/nats-io/nats.go"
"lingniu-vehicle-ingest/go/vehicle-gateway/internal/envelope"
)
@@ -57,6 +62,190 @@ func TestNATSSinkDefaultsToGoRawSubjects(t *testing.T) {
}
}
func TestNATSSinkPublishesDurableRecords(t *testing.T) {
publisher := &recordingNATSPublisher{}
sink := newNATSSinkWithPublisher(publisher, NATSConfig{
RawSubjects: map[envelope.Protocol]string{
envelope.ProtocolJT808: "vehicle.raw.go.jt808.v1",
},
FieldsSubjects: map[envelope.Protocol]string{
envelope.ProtocolJT808: "vehicle.fields.go.jt808.v1",
},
UnifiedSubject: "vehicle.event.go.unified.v1",
})
env := envelope.FrameEnvelope{Protocol: envelope.ProtocolJT808, Phone: "13307795425", MessageID: "0x0200"}
err := sink.PublishRecords(context.Background(), []durableRecord{
{Kind: "raw", Envelope: env},
{Kind: "fields", Envelope: env},
{Kind: "unified", Envelope: env},
})
if err != nil {
t.Fatalf("PublishRecords() error = %v", err)
}
if len(publisher.messages) != 3 {
t.Fatalf("published messages = %d, want 3", len(publisher.messages))
}
for i, want := range []string{
"vehicle.raw.go.jt808.v1",
"vehicle.fields.go.jt808.v1",
"vehicle.event.go.unified.v1",
} {
if got := publisher.messages[i].subject; got != want {
t.Fatalf("message %d subject = %q, want %q", i, got, want)
}
}
}
func TestNATSMessageIDSeparatesKindAndSubject(t *testing.T) {
env := envelope.FrameEnvelope{Protocol: envelope.ProtocolJT808, Phone: "13307795425", MessageID: "0x0200", Sequence: 7}
raw := natsMessageID("raw", "vehicle.raw.go.jt808.v1", env)
rawRetry := natsMessageID("raw", "vehicle.raw.go.jt808.v1", env)
fields := natsMessageID("fields", "vehicle.fields.go.jt808.v1", env)
unified := natsMessageID("unified", "vehicle.event.go.unified.v1", env)
rawOtherSubject := natsMessageID("raw", "vehicle.raw.go.gb32960.v1", env)
if raw != rawRetry {
t.Fatalf("same kind/subject/event id should be stable: %q vs %q", raw, rawRetry)
}
if raw == fields || raw == unified || raw == rawOtherSubject {
t.Fatalf("message ids should be unique per kind and subject: raw=%q fields=%q unified=%q rawOther=%q", raw, fields, unified, rawOtherSubject)
}
if !strings.Contains(raw, env.StableEventID()) {
t.Fatalf("message id %q should retain stable event id %q", raw, env.StableEventID())
}
}
func TestNATSSinkPublishRecordsRejectsUnknownKind(t *testing.T) {
sink := newNATSSinkWithPublisher(&recordingNATSPublisher{}, NATSConfig{})
err := sink.PublishRecords(context.Background(), []durableRecord{
{Kind: "unknown", Envelope: envelope.FrameEnvelope{Protocol: envelope.ProtocolJT808}},
})
if err == nil || !strings.Contains(err.Error(), "unknown durable record kind") {
t.Fatalf("PublishRecords() error = %v, want unknown kind", err)
}
}
func TestNATSSinkAsyncPublishCompletesOnlyAfterPubAck(t *testing.T) {
future := newTestPubAckFuture()
asyncPublisher := &recordingNATSAsyncPublisher{future: future}
sink := newNATSSinkWithPublishers(&recordingNATSPublisher{}, asyncPublisher, NATSConfig{})
record := durableRecord{Kind: "raw", Envelope: normalizeDurableEnvelope(durableTestEnvelope())}
completed := make(chan error, 1)
if err := sink.PublishRecordAsync(record, func(err error) { completed <- err }); err != nil {
t.Fatalf("PublishRecordAsync() error = %v", err)
}
select {
case err := <-completed:
t.Fatalf("completion fired before PubAck: %v", err)
case <-time.After(10 * time.Millisecond):
}
if got, want := asyncPublisher.subject, "vehicle.raw.go.jt808.v1"; got != want {
t.Fatalf("async subject = %q, want %q", got, want)
}
var decoded envelope.FrameEnvelope
if err := json.Unmarshal(asyncPublisher.data, &decoded); err != nil {
t.Fatalf("decode async payload: %v", err)
}
if decoded.EventID != record.Envelope.EventID {
t.Fatalf("async event id = %q, want %q", decoded.EventID, record.Envelope.EventID)
}
future.ok <- &nats.PubAck{Stream: "VEHICLE_RAW", Sequence: 1}
select {
case err := <-completed:
if err != nil {
t.Fatalf("completion error = %v", err)
}
case <-time.After(time.Second):
t.Fatal("completion did not fire after PubAck")
}
}
func TestNATSSinkAsyncPublishPropagatesFutureError(t *testing.T) {
future := newTestPubAckFuture()
sink := newNATSSinkWithPublishers(
&recordingNATSPublisher{},
&recordingNATSAsyncPublisher{future: future},
NATSConfig{},
)
completed := make(chan error, 1)
errWant := errors.New("jetstream ack timeout")
if err := sink.PublishRecordAsync(durableRecord{
Kind: "raw",
Envelope: normalizeDurableEnvelope(durableTestEnvelope()),
}, func(err error) { completed <- err }); err != nil {
t.Fatalf("PublishRecordAsync() error = %v", err)
}
future.err <- errWant
select {
case err := <-completed:
if !errors.Is(err, errWant) {
t.Fatalf("completion error = %v, want %v", err, errWant)
}
case <-time.After(time.Second):
t.Fatal("completion did not fire after future error")
}
}
func TestNATSSinkValidatesDurableRecordBeforeOutboxPersistence(t *testing.T) {
sink := newNATSSinkWithPublisher(&recordingNATSPublisher{}, NATSConfig{})
err := sink.ValidateRecord(durableRecord{
Kind: "raw",
Envelope: envelope.FrameEnvelope{Protocol: envelope.Protocol("UNKNOWN")},
})
if err == nil || !strings.Contains(err.Error(), "raw subject not configured") {
t.Fatalf("ValidateRecord() error = %v", err)
}
}
func TestValidateNATSConfigRejectsRawFieldsSubjectOverlap(t *testing.T) {
err := ValidateNATSConfig(NATSConfig{
RawSubjects: map[envelope.Protocol]string{
envelope.ProtocolJT808: "vehicle.same.jt808",
},
FieldsSubjects: map[envelope.Protocol]string{
envelope.ProtocolJT808: "vehicle.same.jt808",
},
})
if err == nil {
t.Fatal("ValidateNATSConfig() error = nil, want overlap rejection")
}
if !strings.Contains(err.Error(), "nats subject") {
t.Fatalf("error = %q, want nats subject hint", err)
}
}
func TestValidateNATSConfigRejectsKnownProtocolSubjectMismatch(t *testing.T) {
err := ValidateNATSConfig(NATSConfig{
RawSubjects: map[envelope.Protocol]string{
envelope.ProtocolJT808: "vehicle.raw.go.gb32960.v1",
},
})
if err == nil {
t.Fatal("ValidateNATSConfig() error = nil, want raw protocol mismatch")
}
if !strings.Contains(err.Error(), "must match protocol") {
t.Fatalf("error = %q, want protocol mismatch hint", err)
}
err = ValidateNATSConfig(NATSConfig{
FieldsSubjects: map[envelope.Protocol]string{
envelope.ProtocolGB32960: "vehicle.fields.go.yutong-mqtt.v1",
},
})
if err == nil {
t.Fatal("ValidateNATSConfig() error = nil, want fields protocol mismatch")
}
if !strings.Contains(err.Error(), "must match protocol") {
t.Fatalf("error = %q, want protocol mismatch hint", err)
}
}
type recordingNATSPublisher struct {
messages []recordedNATSMessage
}
@@ -70,3 +259,36 @@ func (p *recordingNATSPublisher) Publish(_ context.Context, subject string, data
p.messages = append(p.messages, recordedNATSMessage{subject: subject, data: append([]byte(nil), data...)})
return nil
}
type recordingNATSAsyncPublisher struct {
subject string
data []byte
future nats.PubAckFuture
err error
}
func (p *recordingNATSAsyncPublisher) PublishAsync(subject string, data []byte, _ ...NATSPublishOption) (nats.PubAckFuture, error) {
p.subject = subject
p.data = append([]byte(nil), data...)
return p.future, p.err
}
type testPubAckFuture struct {
ok chan *nats.PubAck
err chan error
msg *nats.Msg
}
func newTestPubAckFuture() *testPubAckFuture {
return &testPubAckFuture{
ok: make(chan *nats.PubAck, 1),
err: make(chan error, 1),
msg: &nats.Msg{},
}
}
func (f *testPubAckFuture) Ok() <-chan *nats.PubAck { return f.ok }
func (f *testPubAckFuture) Err() <-chan error { return f.err }
func (f *testPubAckFuture) Msg() *nats.Msg { return f.msg }

293
go/vehicle-gateway/internal/eventbus/partitioned_async_sink.go Normal file
View File

@@ -0,0 +1,293 @@
package eventbus
import (
"context"
"sync"
"time"
"lingniu-vehicle-ingest/go/vehicle-gateway/internal/envelope"
"lingniu-vehicle-ingest/go/vehicle-gateway/internal/metrics"
)
type PartitionedAsyncConfig struct {
RawQueueSize int
DerivedQueueSize int
RawWorkers int
DerivedWorkers int
EnqueueTimeout time.Duration
RawEnqueueTimeout time.Duration
DerivedEnqueueTimeout time.Duration
OperationTimeout time.Duration
OnError func(error)
Metrics *metrics.Registry
Name string
}
type PartitionedAsyncSink struct {
delegate Sink
rawJobs chan asyncJob
derivedJobs chan asyncJob
rawEnqueueTimeout time.Duration
derivedEnqueueTimeout time.Duration
timeout time.Duration
onError func(error)
metrics *metrics.Registry
name string
queueWait *metrics.RecentLatencyByKey
closeOnce sync.Once
closed chan struct{}
done chan struct{}
wg sync.WaitGroup
}
func NewPartitionedAsyncSink(delegate Sink, cfg PartitionedAsyncConfig) *PartitionedAsyncSink {
if delegate == nil {
panic("partitioned async delegate sink must not be nil")
}
if cfg.RawQueueSize <= 0 {
cfg.RawQueueSize = 100_000
}
if cfg.DerivedQueueSize <= 0 {
cfg.DerivedQueueSize = 50_000
}
if cfg.RawWorkers <= 0 {
cfg.RawWorkers = 4
}
if cfg.DerivedWorkers <= 0 {
cfg.DerivedWorkers = 2
}
enqueueTimeout := normalizePartitionedEnqueueTimeout(cfg.EnqueueTimeout, time.Second)
rawEnqueueTimeout := normalizePartitionedEnqueueTimeout(cfg.RawEnqueueTimeout, enqueueTimeout)
derivedEnqueueTimeout := normalizePartitionedEnqueueTimeout(cfg.DerivedEnqueueTimeout, enqueueTimeout)
if cfg.OperationTimeout <= 0 {
cfg.OperationTimeout = 30 * time.Second
}
if cfg.Name == "" {
cfg.Name = "partitioned-async"
}
s := &PartitionedAsyncSink{
delegate: delegate,
rawJobs: make(chan asyncJob, cfg.RawQueueSize),
derivedJobs: make(chan asyncJob, cfg.DerivedQueueSize),
rawEnqueueTimeout: rawEnqueueTimeout,
derivedEnqueueTimeout: derivedEnqueueTimeout,
timeout: cfg.OperationTimeout,
onError: cfg.OnError,
metrics: cfg.Metrics,
name: cfg.Name,
queueWait: metrics.NewRecentLatencyByKey(512),
closed: make(chan struct{}),
done: make(chan struct{}),
}
s.startWorkers("raw", s.rawJobs, cfg.RawWorkers)
s.startWorkers("derived", s.derivedJobs, cfg.DerivedWorkers)
s.recordWorkers("raw", cfg.RawWorkers)
s.recordWorkers("derived", cfg.DerivedWorkers)
s.recordQueueCapacity("raw", cap(s.rawJobs))
s.recordQueueCapacity("derived", cap(s.derivedJobs))
go func() {
s.wg.Wait()
close(s.done)
}()
return s
}
func (s *PartitionedAsyncSink) PublishRaw(ctx context.Context, env envelope.FrameEnvelope) error {
return s.enqueue(ctx, "raw", s.rawJobs, s.rawEnqueueTimeout, asyncJob{kind: "raw", env: env})
}
func (s *PartitionedAsyncSink) PublishUnified(ctx context.Context, env envelope.FrameEnvelope) error {
return s.enqueue(ctx, "derived", s.derivedJobs, s.derivedEnqueueTimeout, asyncJob{kind: "unified", env: env})
}
func (s *PartitionedAsyncSink) PublishFields(ctx context.Context, env envelope.FrameEnvelope) error {
return s.enqueue(ctx, "derived", s.derivedJobs, s.derivedEnqueueTimeout, asyncJob{kind: "fields", env: env})
}
func (s *PartitionedAsyncSink) Close() error {
s.closeOnce.Do(func() {
close(s.closed)
})
<-s.done
return s.delegate.Close()
}
func (s *PartitionedAsyncSink) startWorkers(queueName string, jobs <-chan asyncJob, workers int) {
s.wg.Add(workers)
for i := 0; i < workers; i++ {
go s.worker(queueName, jobs)
}
}
func (s *PartitionedAsyncSink) enqueue(ctx context.Context, queueName string, jobs chan<- asyncJob, enqueueTimeout time.Duration, job asyncJob) error {
select {
case <-s.closed:
s.recordEnqueue(job.kind, "closed")
return ErrAsyncSinkClosed
default:
}
var timeoutC <-chan time.Time
var timer *time.Timer
if enqueueTimeout > 0 {
timer = time.NewTimer(enqueueTimeout)
timeoutC = timer.C
defer timer.Stop()
}
job.enqueuedAt = time.Now()
select {
case jobs <- job:
s.recordEnqueue(job.kind, "queued")
s.recordQueueDepth(queueName)
return nil
case <-s.closed:
s.recordEnqueue(job.kind, "closed")
return ErrAsyncSinkClosed
case <-ctx.Done():
s.recordEnqueue(job.kind, "timeout")
s.recordQueueDepth(queueName)
return ctx.Err()
case <-timeoutC:
s.recordEnqueue(job.kind, "timeout")
s.recordQueueDepth(queueName)
return ErrAsyncSinkEnqueueTimeout
}
}
func normalizePartitionedEnqueueTimeout(value time.Duration, fallback time.Duration) time.Duration {
if value == 0 {
value = fallback
}
if value < 0 {
return 0
}
return value
}
func (s *PartitionedAsyncSink) worker(queueName string, jobs <-chan asyncJob) {
defer s.wg.Done()
for {
select {
case job := <-jobs:
s.publishJob(queueName, job)
case <-s.closed:
for {
select {
case job := <-jobs:
s.publishJob(queueName, job)
default:
return
}
}
}
}
}
func (s *PartitionedAsyncSink) publishJob(queueName string, job asyncJob) {
s.recordQueueDepth(queueName)
s.recordQueueWait(queueName, job)
ctx, cancel := context.WithTimeout(context.Background(), s.timeout)
started := time.Now()
var err error
switch job.kind {
case "raw":
err = s.delegate.PublishRaw(ctx, job.env)
case "unified":
err = s.delegate.PublishUnified(ctx, job.env)
case "fields":
err = s.delegate.PublishFields(ctx, job.env)
}
cancel()
status := "ok"
if err != nil {
status = "error"
}
s.recordPublish(job.kind, status, time.Since(started))
if err != nil && s.onError != nil {
s.onError(err)
}
s.recordQueueDepth(queueName)
}
func (s *PartitionedAsyncSink) recordEnqueue(kind string, status string) {
if s.metrics == nil {
return
}
s.metrics.IncCounter("vehicle_async_sink_enqueue_total", metrics.Labels{
"sink": s.name,
"kind": kind,
"status": status,
})
}
func (s *PartitionedAsyncSink) recordPublish(kind string, status string, elapsed time.Duration) {
if s.metrics == nil {
return
}
labels := metrics.Labels{
"sink": s.name,
"kind": kind,
"status": status,
}
s.metrics.IncCounter("vehicle_async_sink_publish_total", labels)
elapsedMS := float64(elapsed.Milliseconds())
s.metrics.SetGauge("vehicle_async_sink_publish_duration_ms", labels, elapsedMS)
s.metrics.ObserveHistogram("vehicle_async_sink_publish_duration_ms_histogram", labels, asyncSinkPublishDurationBucketsMS, elapsedMS)
}
func (s *PartitionedAsyncSink) recordQueueDepth(queueName string) {
if s.metrics == nil {
return
}
s.metrics.SetGauge("vehicle_async_sink_queue_depth", metrics.Labels{
"sink": s.name,
"queue": queueName,
}, float64(s.queueDepth(queueName)))
}
func (s *PartitionedAsyncSink) recordQueueCapacity(queueName string, value int) {
if s.metrics == nil {
return
}
s.metrics.SetGauge("vehicle_async_sink_queue_capacity", metrics.Labels{
"sink": s.name,
"queue": queueName,
}, float64(value))
}
func (s *PartitionedAsyncSink) recordQueueWait(queueName string, job asyncJob) {
if s.metrics == nil || job.enqueuedAt.IsZero() {
return
}
elapsedMS := float64(time.Since(job.enqueuedAt)) / float64(time.Millisecond)
labels := metrics.Labels{
"sink": s.name,
"queue": queueName,
"kind": job.kind,
}
s.metrics.ObserveHistogram("vehicle_async_sink_queue_wait_duration_ms_histogram", labels, asyncSinkPublishDurationBucketsMS, elapsedMS)
p99, samples := s.queueWait.Observe(queueName+"\x00"+job.kind, elapsedMS)
s.metrics.SetGauge("vehicle_async_sink_queue_wait_recent_p99_ms", labels, p99)
s.metrics.SetGauge("vehicle_async_sink_queue_wait_recent_samples", labels, float64(samples))
}
func (s *PartitionedAsyncSink) recordWorkers(queueName string, workers int) {
if s.metrics == nil {
return
}
s.metrics.SetGauge("vehicle_async_sink_workers", metrics.Labels{
"sink": s.name,
"queue": queueName,
}, float64(workers))
}
func (s *PartitionedAsyncSink) queueDepth(queueName string) int {
switch queueName {
case "raw":
return len(s.rawJobs)
case "derived":
return len(s.derivedJobs)
default:
return 0
}
}

230
go/vehicle-gateway/internal/eventbus/partitioned_async_sink_test.go Normal file
View File

@@ -0,0 +1,230 @@
package eventbus
import (
"context"
"errors"
"strings"
"testing"
"time"
"lingniu-vehicle-ingest/go/vehicle-gateway/internal/envelope"
"lingniu-vehicle-ingest/go/vehicle-gateway/internal/metrics"
)
func TestPartitionedAsyncSinkRawQueueIsIsolatedFromDerivedBacklog(t *testing.T) {
delegate := newBlockingDerivedSink()
sink := NewPartitionedAsyncSink(delegate, PartitionedAsyncConfig{
RawQueueSize: 1,
DerivedQueueSize: 1,
RawWorkers: 1,
DerivedWorkers: 1,
EnqueueTimeout: 20 * time.Millisecond,
OperationTimeout: time.Second,
})
defer sink.Close()
env := envelope.FrameEnvelope{Protocol: envelope.ProtocolJT808, Phone: "13307795425"}
if err := sink.PublishFields(context.Background(), env); err != nil {
t.Fatalf("PublishFields() error = %v", err)
}
select {
case <-delegate.fieldsStarted:
case <-time.After(time.Second):
t.Fatal("delegate fields publish was not started")
}
if err := sink.PublishUnified(context.Background(), env); err != nil {
t.Fatalf("PublishUnified() error = %v", err)
}
start := time.Now()
if err := sink.PublishRaw(context.Background(), env); err != nil {
t.Fatalf("PublishRaw() error = %v", err)
}
if elapsed := time.Since(start); elapsed > 50*time.Millisecond {
t.Fatalf("PublishRaw() blocked behind derived backlog for %s", elapsed)
}
delegate.release()
}
func TestPartitionedAsyncSinkRecordsPerQueueMetrics(t *testing.T) {
registry := metrics.NewRegistry()
delegate := newBlockingDerivedSink()
sink := NewPartitionedAsyncSink(delegate, PartitionedAsyncConfig{
RawQueueSize: 2,
DerivedQueueSize: 3,
RawWorkers: 1,
DerivedWorkers: 1,
OperationTimeout: time.Second,
Metrics: registry,
Name: "nats",
})
env := envelope.FrameEnvelope{Protocol: envelope.ProtocolGB32960, VIN: "LNBSCB3D4R1234567"}
if err := sink.PublishRaw(context.Background(), env); err != nil {
t.Fatalf("PublishRaw() error = %v", err)
}
if err := sink.PublishFields(context.Background(), env); err != nil {
t.Fatalf("PublishFields() error = %v", err)
}
delegate.release()
if err := sink.Close(); err != nil {
t.Fatalf("Close() error = %v", err)
}
text := registry.Render()
for _, want := range []string{
`vehicle_async_sink_queue_capacity{queue="raw",sink="nats"} 2`,
`vehicle_async_sink_queue_capacity{queue="derived",sink="nats"} 3`,
`vehicle_async_sink_workers{queue="raw",sink="nats"} 1`,
`vehicle_async_sink_workers{queue="derived",sink="nats"} 1`,
`vehicle_async_sink_enqueue_total{kind="raw",sink="nats",status="queued"} 1`,
`vehicle_async_sink_enqueue_total{kind="fields",sink="nats",status="queued"} 1`,
`vehicle_async_sink_publish_total{kind="raw",sink="nats",status="ok"} 1`,
`vehicle_async_sink_publish_total{kind="fields",sink="nats",status="ok"} 1`,
`vehicle_async_sink_queue_wait_duration_ms_histogram_count{kind="raw",queue="raw",sink="nats"} 1`,
`vehicle_async_sink_queue_wait_recent_p99_ms{kind="raw",queue="raw",sink="nats"}`,
`vehicle_async_sink_queue_wait_recent_samples{kind="raw",queue="raw",sink="nats"} 1`,
`vehicle_async_sink_queue_wait_duration_ms_histogram_count{kind="fields",queue="derived",sink="nats"} 1`,
} {
if !strings.Contains(text, want) {
t.Fatalf("partitioned async metric missing %s:\n%s", want, text)
}
}
}
func TestPartitionedAsyncSinkUsesIndependentDerivedEnqueueTimeout(t *testing.T) {
delegate := newBlockingDerivedSink()
sink := NewPartitionedAsyncSink(delegate, PartitionedAsyncConfig{
RawQueueSize: 1,
DerivedQueueSize: 1,
RawWorkers: 1,
DerivedWorkers: 1,
RawEnqueueTimeout: 200 * time.Millisecond,
DerivedEnqueueTimeout: 10 * time.Millisecond,
OperationTimeout: time.Second,
})
defer sink.Close()
env := envelope.FrameEnvelope{Protocol: envelope.ProtocolJT808, Phone: "13307795425"}
if err := sink.PublishFields(context.Background(), env); err != nil {
t.Fatalf("PublishFields() error = %v", err)
}
select {
case <-delegate.fieldsStarted:
case <-time.After(time.Second):
t.Fatal("delegate fields publish was not started")
}
if err := sink.PublishUnified(context.Background(), env); err != nil {
t.Fatalf("first PublishUnified() error = %v", err)
}
start := time.Now()
err := sink.PublishUnified(context.Background(), env)
elapsed := time.Since(start)
if !errors.Is(err, ErrAsyncSinkEnqueueTimeout) {
t.Fatalf("second PublishUnified() error = %v, want ErrAsyncSinkEnqueueTimeout", err)
}
if elapsed > 100*time.Millisecond {
t.Fatalf("derived enqueue timeout took %s, want quick failure", elapsed)
}
delegate.release()
}
func TestPartitionedAsyncSinkCloseUnblocksBlockedDerivedEnqueue(t *testing.T) {
delegate := newBlockingDerivedSink()
sink := NewPartitionedAsyncSink(delegate, PartitionedAsyncConfig{
RawQueueSize: 1,
DerivedQueueSize: 1,
RawWorkers: 1,
DerivedWorkers: 1,
EnqueueTimeout: -1,
OperationTimeout: time.Second,
})
env := envelope.FrameEnvelope{Protocol: envelope.ProtocolJT808, Phone: "13307795425"}
if err := sink.PublishFields(context.Background(), env); err != nil {
t.Fatalf("PublishFields() error = %v", err)
}
select {
case <-delegate.fieldsStarted:
case <-time.After(time.Second):
t.Fatal("delegate fields publish was not started")
}
if err := sink.PublishUnified(context.Background(), env); err != nil {
t.Fatalf("PublishUnified() error = %v", err)
}
publishErr := make(chan error, 1)
go func() {
publishErr <- sink.PublishUnified(context.Background(), env)
}()
closeErr := make(chan error, 1)
go func() {
closeErr <- sink.Close()
}()
select {
case err := <-publishErr:
if !errors.Is(err, ErrAsyncSinkClosed) {
t.Fatalf("blocked PublishUnified() error = %v, want ErrAsyncSinkClosed", err)
}
case <-time.After(time.Second):
t.Fatal("blocked PublishUnified() was not released by Close")
}
delegate.release()
select {
case err := <-closeErr:
if err != nil {
t.Fatalf("Close() error = %v", err)
}
case <-time.After(time.Second):
t.Fatal("Close() did not finish after delegate release")
}
}
type blockingDerivedSink struct {
fieldsStarted chan struct{}
releaseFields chan struct{}
}
func newBlockingDerivedSink() *blockingDerivedSink {
return &blockingDerivedSink{
fieldsStarted: make(chan struct{}),
releaseFields: make(chan struct{}),
}
}
func (s *blockingDerivedSink) PublishRaw(context.Context, envelope.FrameEnvelope) error {
return nil
}
func (s *blockingDerivedSink) PublishUnified(context.Context, envelope.FrameEnvelope) error {
return nil
}
func (s *blockingDerivedSink) PublishFields(context.Context, envelope.FrameEnvelope) error {
s.signalFieldsStarted()
<-s.releaseFields
return nil
}
func (s *blockingDerivedSink) Close() error {
s.release()
return nil
}
func (s *blockingDerivedSink) signalFieldsStarted() {
select {
case <-s.fieldsStarted:
default:
close(s.fieldsStarted)
}
}
func (s *blockingDerivedSink) release() {
select {
case <-s.releaseFields:
default:
close(s.releaseFields)
}
}

42
go/vehicle-gateway/internal/gateway/fields.go Normal file
View File

@@ -0,0 +1,42 @@
package gateway
import (
"lingniu-vehicle-ingest/go/vehicle-gateway/internal/envelope"
"lingniu-vehicle-ingest/go/vehicle-gateway/internal/realtime"
)
const (
gatewayFieldsPublished = "published"
gatewayFieldsDelegated = "delegated_to_bridge"
gatewayFieldsSkippedNonRealtime = "skipped_non_realtime"
gatewayFieldsSkippedMissing = "skipped_missing_fields"
gatewayFieldsPublishError = "publish_error"
)
func gatewayFieldsEnvelope(env envelope.FrameEnvelope) (envelope.FrameEnvelope, string, bool) {
if !envelope.IsRealtimeTelemetryFrame(env) {
return envelope.FrameEnvelope{}, gatewayFieldsSkippedNonRealtime, false
}
fieldsEnv, ok := realtime.BuildFieldsEnvelope(env)
if !ok {
return envelope.FrameEnvelope{}, gatewayFieldsSkippedMissing, false
}
return fieldsEnv, gatewayFieldsPublished, true
}
func gatewayDelegatedFields(env envelope.FrameEnvelope) (int, string, bool) {
if !envelope.IsRealtimeTelemetryFrame(env) {
return 0, gatewayFieldsSkippedNonRealtime, false
}
if len(env.ParsedFields) == 0 {
return 0, gatewayFieldsSkippedMissing, false
}
return len(env.ParsedFields), gatewayFieldsDelegated, true
}
func canonicalRawEnvelope(env envelope.FrameEnvelope) envelope.FrameEnvelope {
env.EventKind = envelope.EventKindRaw
env.Parsed = nil
env.Fields = nil
return env
}

76
go/vehicle-gateway/internal/gateway/fields_test.go Normal file
View File

@@ -0,0 +1,76 @@
package gateway
import (
"strings"
"testing"
"lingniu-vehicle-ingest/go/vehicle-gateway/internal/envelope"
)
func TestCanonicalRawEnvelopeDropsBareStandardizedFields(t *testing.T) {
original := envelope.FrameEnvelope{
Protocol: envelope.ProtocolJT808,
EventKind: envelope.EventKindFields,
Parsed: map[string]any{
"location": map[string]any{"latitude": 30.5, "speed_kmh": 20},
},
Fields: map[string]any{
envelope.FieldLatitude: 30.5,
envelope.FieldSpeedKMH: 20,
},
ParsedFields: map[string]any{
"jt808.location.latitude": 30.5,
"jt808.location.speed_kmh": 20,
},
}
canonical := canonicalRawEnvelope(original)
if canonical.EventKind != envelope.EventKindRaw {
t.Fatalf("event kind = %q", canonical.EventKind)
}
if len(canonical.Fields) != 0 {
t.Fatalf("canonical raw leaked bare fields: %#v", canonical.Fields)
}
if len(canonical.Parsed) != 0 {
t.Fatalf("canonical raw leaked duplicate parsed tree: %#v", canonical.Parsed)
}
if len(canonical.ParsedFields) != 2 {
t.Fatalf("canonical raw lost protocol fields: %#v", canonical.ParsedFields)
}
if len(original.Fields) != 2 {
t.Fatalf("canonical copy mutated in-process parser fields: %#v", original.Fields)
}
if len(original.Parsed) != 1 {
t.Fatalf("canonical copy mutated in-process parsed tree: %#v", original.Parsed)
}
}
func TestCanonicalRawEnvelopeReducesDuplicateParsedPayload(t *testing.T) {
original := envelope.FrameEnvelope{
Protocol: envelope.ProtocolGB32960,
MessageID: "0x02",
VIN: "LTESTVIN000000001",
ParseStatus: envelope.ParseOK,
Parsed: map[string]any{
"data_units": []any{map[string]any{
"name": "vendor",
"value": strings.Repeat("x", 4096),
}},
},
ParsedFields: map[string]any{
"gb32960.vendor.value": strings.Repeat("x", 4096),
},
}
before, err := original.MarshalJSONBytes()
if err != nil {
t.Fatal(err)
}
after, err := canonicalRawEnvelope(original).MarshalJSONBytes()
if err != nil {
t.Fatal(err)
}
if len(after)*100 >= len(before)*65 {
t.Fatalf("canonical raw should remove the duplicate parsed tree: before=%d after=%d", len(before), len(after))
}
t.Logf("canonical raw bytes before=%d after=%d reduction=%.1f%%", len(before), len(after), 100*(1-float64(len(after))/float64(len(before))))
}

104
go/vehicle-gateway/internal/gateway/identity_metrics.go Normal file
View File

@@ -0,0 +1,104 @@
package gateway
import (
"context"
"errors"
"strings"
"time"
"lingniu-vehicle-ingest/go/vehicle-gateway/internal/envelope"
"lingniu-vehicle-ingest/go/vehicle-gateway/internal/metrics"
)
var gatewayIdentityDurationBucketsMS = []float64{1, 5, 10, 25, 50, 100, 250, 500, 1000, 5000}
var gatewayFieldsCountBuckets = []float64{1, 5, 10, 25, 50, 100, 250, 500, 1000, 2500, 5000}
func identityErrorStatus(err error) string {
if errors.Is(err, context.DeadlineExceeded) {
return "timeout"
}
return "error"
}
func recordGatewayIdentityDuration(registry *metrics.Registry, protocol envelope.Protocol, status string, elapsed time.Duration) {
if registry == nil {
return
}
elapsedMS := float64(elapsed.Nanoseconds()) / float64(time.Millisecond)
labels := metrics.Labels{
"protocol": string(protocol),
"status": status,
}
registry.SetGauge("vehicle_gateway_identity_duration_ms", labels, elapsedMS)
registry.ObserveHistogram("vehicle_gateway_identity_duration_ms_histogram", labels, gatewayIdentityDurationBucketsMS, elapsedMS)
}
func recordGatewayIdentityCacheStatus(registry *metrics.Registry, protocol envelope.Protocol, env envelope.FrameEnvelope) {
if registry == nil {
return
}
status := identityCacheStatus(env)
if status == "" {
return
}
registry.IncCounter("vehicle_gateway_identity_cache_total", metrics.Labels{
"protocol": string(protocol),
"cache_status": status,
})
}
func identityCacheStatus(env envelope.FrameEnvelope) string {
identity, ok := env.Parsed["identity"].(map[string]any)
if !ok {
return ""
}
status, ok := identity["cache_status"].(string)
if !ok {
return ""
}
return strings.TrimSpace(status)
}
func annotateIdentityError(env *envelope.FrameEnvelope, err error) {
if env == nil || err == nil {
return
}
if env.Parsed == nil {
env.Parsed = map[string]any{}
}
identity, _ := env.Parsed["identity"].(map[string]any)
if identity == nil {
identity = map[string]any{}
}
if _, ok := identity["resolved"]; !ok {
identity["resolved"] = strings.TrimSpace(env.VIN) != ""
}
identity["error"] = err.Error()
env.Parsed["identity"] = identity
}
func recordGatewayFieldsMetric(registry *metrics.Registry, protocol envelope.Protocol, status string) {
if registry == nil {
return
}
registry.IncCounter("vehicle_gateway_fields_total", metrics.Labels{
"protocol": string(protocol),
"status": status,
})
}
func recordGatewayFieldsCount(registry *metrics.Registry, protocol envelope.Protocol, status string, fieldCount int) {
if registry == nil {
return
}
if fieldCount < 0 {
fieldCount = 0
}
labels := metrics.Labels{
"protocol": string(protocol),
"status": status,
}
value := float64(fieldCount)
registry.SetGauge("vehicle_gateway_fields_count", labels, value)
registry.ObserveHistogram("vehicle_gateway_fields_count_histogram", labels, gatewayFieldsCountBuckets, value)
}

27
go/vehicle-gateway/internal/gateway/identity_metrics_test.go Normal file
View File

@@ -0,0 +1,27 @@
package gateway
import (
"strings"
"testing"
"lingniu-vehicle-ingest/go/vehicle-gateway/internal/envelope"
"lingniu-vehicle-ingest/go/vehicle-gateway/internal/metrics"
)
func TestRecordGatewayIdentityCacheStatus(t *testing.T) {
registry := metrics.NewRegistry()
recordGatewayIdentityCacheStatus(registry, envelope.ProtocolJT808, envelope.FrameEnvelope{
Parsed: map[string]any{
"identity": map[string]any{
"resolved": true,
"cache_status": "stale",
},
},
})
got := registry.Render()
want := `vehicle_gateway_identity_cache_total{cache_status="stale",protocol="JT808"} 1`
if !strings.Contains(got, want) {
t.Fatalf("metrics missing %q:\n%s", want, got)
}
}

119
go/vehicle-gateway/internal/gateway/mqtt_client.go
View File

@@ -41,6 +41,7 @@ type MQTTClientConfig struct {
Logger *slog.Logger
Metrics *metrics.Registry
PublishUnified bool
DelegateFields bool
}
type MQTTClient struct {
@@ -204,28 +205,42 @@ func (c *MQTTClient) handleMessage(ctx context.Context, topic string, payload []
}
env.EventID = env.StableEventID()
c.recordParseErrorMetric(err)
} else {
} else if envelope.RequiresVehicleIdentity(env) {
resolveStarted := time.Now()
resolved, resolveErr := c.cfg.Resolver.Resolve(messageCtx, env)
if resolveErr != nil {
c.cfg.Logger.Warn("mqtt identity resolve failed", "topic", topic, "event_id", env.StableEventID(), "error", resolveErr)
if env.Parsed == nil {
env.Parsed = map[string]any{}
if resolved.Protocol != "" || strings.TrimSpace(resolved.VIN) != "" || strings.TrimSpace(resolved.Phone) != "" {
env = resolved
}
env.Parsed["identity"] = map[string]any{"resolved": false, "error": resolveErr.Error()}
identityStatus := identityErrorStatus(resolveErr)
c.recordIdentityDuration(identityStatus, time.Since(resolveStarted))
c.cfg.Logger.Warn("mqtt identity resolve failed", "topic", topic, "event_id", env.StableEventID(), "error", resolveErr)
annotateIdentityError(&env, resolveErr)
env.ParseStatus = envelope.ParsePartial
c.recordIdentityMetric("error")
c.recordIdentityMetric(identityStatus)
c.recordIdentityIssueMetric(identityStatus, env, identityIssueReason(identityStatus, resolveErr))
} else {
env = resolved
annotateIdentityUnresolved(&env)
c.recordIdentityMetric(identityStatus(env))
identityStatus := identityStatus(env)
c.recordIdentityDuration(identityStatus, time.Since(resolveStarted))
c.recordIdentityMetric(identityStatus)
if identityStatus != "resolved" {
c.recordIdentityIssueMetric(identityStatus, env, "no_binding")
}
recordGatewayIdentityCacheStatus(c.cfg.Metrics, envelope.ProtocolYutongMQTT, env)
}
} else {
c.recordIdentitySkipMetric("non_vehicle_frame")
}
frameStatus = env.ParseStatus
env.EventKind = envelope.EventKindRaw
c.recordFrameMetric(env.ParseStatus)
if env.ParseStatus != envelope.ParseBadFrame {
realtime.EnsureParsedFields(&env)
}
if err := c.cfg.Sink.PublishRaw(messageCtx, env); err != nil {
canonicalRaw := canonicalRawEnvelope(env)
if err := c.cfg.Sink.PublishRaw(messageCtx, canonicalRaw); err != nil {
c.recordPublishMetric("raw", "error")
c.cfg.Logger.Error("publish mqtt raw failed", "topic", topic, "event_id", env.StableEventID(), "error", err)
return
@@ -234,21 +249,37 @@ func (c *MQTTClient) handleMessage(ctx context.Context, topic string, payload []
if env.ParseStatus == envelope.ParseBadFrame {
return
}
if fieldsEnv, ok := realtime.BuildFieldsEnvelope(env); ok {
if err := c.cfg.Sink.PublishFields(messageCtx, fieldsEnv); err != nil {
c.recordPublishMetric("fields", "error")
c.cfg.Logger.Error("publish mqtt fields failed", "topic", topic, "event_id", env.StableEventID(), "error", err)
return
if c.cfg.DelegateFields {
fieldCount, fieldsStatus, ok := gatewayDelegatedFields(env)
c.recordFieldsMetric(fieldsStatus)
if ok {
c.recordFieldsCount(fieldsStatus, fieldCount)
c.recordPublishMetric("fields", "delegated")
}
} else {
fieldsEnv, fieldsStatus, ok := gatewayFieldsEnvelope(env)
if !ok {
c.recordFieldsMetric(fieldsStatus)
} else {
if err := c.cfg.Sink.PublishFields(messageCtx, fieldsEnv); err != nil {
c.recordFieldsMetric(gatewayFieldsPublishError)
c.recordFieldsCount(gatewayFieldsPublishError, len(fieldsEnv.Fields))
c.recordPublishMetric("fields", "error")
c.cfg.Logger.Error("publish mqtt fields failed", "topic", topic, "event_id", env.StableEventID(), "error", err)
} else {
c.recordFieldsMetric(fieldsStatus)
c.recordFieldsCount(fieldsStatus, len(fieldsEnv.Fields))
c.recordPublishMetric("fields", "ok")
}
}
c.recordPublishMetric("fields", "ok")
}
if c.cfg.PublishUnified {
if err := c.cfg.Sink.PublishUnified(messageCtx, env); err != nil {
if err := c.cfg.Sink.PublishUnified(messageCtx, canonicalRaw); err != nil {
c.recordPublishMetric("unified", "error")
c.cfg.Logger.Error("publish mqtt unified failed", "topic", topic, "event_id", env.StableEventID(), "error", err)
return
} else {
c.recordPublishMetric("unified", "ok")
}
c.recordPublishMetric("unified", "ok")
}
}
@@ -256,10 +287,12 @@ func (c *MQTTClient) recordFrameMetric(status envelope.ParseStatus) {
if c.cfg.Metrics == nil {
return
}
c.cfg.Metrics.IncCounter("vehicle_gateway_frames_total", metrics.Labels{
labels := metrics.Labels{
"protocol": string(envelope.ProtocolYutongMQTT),
"status": string(status),
})
}
c.cfg.Metrics.IncCounter("vehicle_gateway_frames_total", labels)
metrics.RecordLastActivity(c.cfg.Metrics, "vehicle_gateway_last_frame_unix_seconds", labels)
}
func (c *MQTTClient) recordFrameDuration(status envelope.ParseStatus, elapsed time.Duration) {
@@ -282,11 +315,21 @@ func (c *MQTTClient) recordPublishMetric(kind string, status string) {
if c.cfg.Metrics == nil {
return
}
c.cfg.Metrics.IncCounter("vehicle_gateway_publish_total", metrics.Labels{
labels := metrics.Labels{
"protocol": string(envelope.ProtocolYutongMQTT),
"kind": kind,
"status": status,
})
}
c.cfg.Metrics.IncCounter("vehicle_gateway_publish_total", labels)
metrics.RecordLastActivity(c.cfg.Metrics, "vehicle_gateway_last_publish_unix_seconds", labels)
}
func (c *MQTTClient) recordFieldsMetric(status string) {
recordGatewayFieldsMetric(c.cfg.Metrics, envelope.ProtocolYutongMQTT, status)
}
func (c *MQTTClient) recordFieldsCount(status string, fieldCount int) {
recordGatewayFieldsCount(c.cfg.Metrics, envelope.ProtocolYutongMQTT, status, fieldCount)
}
func (c *MQTTClient) recordParseErrorMetric(err error) {
@@ -308,3 +351,37 @@ func (c *MQTTClient) recordIdentityMetric(status string) {
"status": status,
})
}
func (c *MQTTClient) recordIdentitySkipMetric(reason string) {
if c.cfg.Metrics == nil {
return
}
c.cfg.Metrics.IncCounter("vehicle_gateway_identity_skips_total", metrics.Labels{
"protocol": string(envelope.ProtocolYutongMQTT),
"reason": reason,
})
}
func (c *MQTTClient) recordIdentityIssueMetric(status string, env envelope.FrameEnvelope, reason string) {
if c.cfg.Metrics == nil {
return
}
messageID := strings.TrimSpace(env.MessageID)
if messageID == "" {
messageID = "unknown"
}
reason = strings.TrimSpace(reason)
if reason == "" {
reason = "unknown"
}
c.cfg.Metrics.IncCounter("vehicle_gateway_identity_issues_total", metrics.Labels{
"protocol": string(envelope.ProtocolYutongMQTT),
"status": status,
"message_id": messageID,
"reason": reason,
})
}
func (c *MQTTClient) recordIdentityDuration(status string, elapsed time.Duration) {
recordGatewayIdentityDuration(c.cfg.Metrics, envelope.ProtocolYutongMQTT, status, elapsed)
}

207
go/vehicle-gateway/internal/gateway/mqtt_client_test.go
View File

@@ -7,6 +7,7 @@ import (
"crypto/x509"
"crypto/x509/pkix"
"encoding/pem"
"errors"
"log/slog"
"math/big"
"os"
@@ -19,7 +20,7 @@ import (
"lingniu-vehicle-ingest/go/vehicle-gateway/internal/metrics"
)
func TestMQTTClientHandleMessagePublishesOnlyRawByDefault(t *testing.T) {
func TestMQTTClientHandleMessagePublishesRawAndFieldsByDefault(t *testing.T) {
sink := &recordingSink{}
client, err := NewMQTTClient(MQTTClientConfig{
EndpointName: "endpoint-a",
@@ -39,18 +40,80 @@ func TestMQTTClientHandleMessagePublishesOnlyRawByDefault(t *testing.T) {
"data":{"METER_SPEED":52.3,"TOTAL_MILEAGE":123456.7}
}`))
if len(sink.raw) != 1 || len(sink.unified) != 0 {
t.Fatalf("raw=%d unified=%d", len(sink.raw), len(sink.unified))
if len(sink.raw) != 1 || len(sink.fields) != 1 || len(sink.unified) != 0 {
t.Fatalf("raw=%d fields=%d unified=%d", len(sink.raw), len(sink.fields), len(sink.unified))
}
if sink.raw[0].Protocol != envelope.ProtocolYutongMQTT || sink.raw[0].VIN != "LTEST000000000001" {
t.Fatalf("unexpected raw envelope: %#v", sink.raw[0])
}
if sink.raw[0].EventKind != envelope.EventKindRaw {
t.Fatalf("raw event kind = %q, want %q", sink.raw[0].EventKind, envelope.EventKindRaw)
}
if sink.raw[0].RawText == "" {
t.Fatal("mqtt raw envelope should keep text payload")
}
if sink.raw[0].RawHex != "" {
t.Fatalf("mqtt raw envelope should not duplicate text payload as hex: %q", sink.raw[0].RawHex)
}
if len(sink.raw[0].Fields) != 0 {
t.Fatalf("canonical raw must not carry bare standardized fields: %#v", sink.raw[0].Fields)
}
if got, want := sink.fields[0].Fields["yutong_mqtt.data.meter_speed"], sink.raw[0].ParsedFields["yutong_mqtt.data.meter_speed"]; got != want {
t.Fatalf("fields event should reuse raw parsed field, got %#v want %#v", got, want)
}
if sink.fields[0].EventKind != envelope.EventKindFields {
t.Fatalf("fields event kind = %q, want %q", sink.fields[0].EventKind, envelope.EventKindFields)
}
if got, want := sink.fields[0].SourceEventID, sink.raw[0].StableEventID(); got != want {
t.Fatalf("fields source event id = %#v, want %s", got, want)
}
if sink.fields[0].FieldMapping == "" {
t.Fatal("fields event should expose field mapping version")
}
if len(sink.fields[0].Parsed) != 0 || len(sink.fields[0].ParsedFields) != 0 {
t.Fatalf("fields envelope should not duplicate parsed payload: parsed=%#v parsed_fields=%#v", sink.fields[0].Parsed, sink.fields[0].ParsedFields)
}
}
func TestMQTTClientDelegatesFieldsProjectionWhenConfigured(t *testing.T) {
sink := &recordingSink{}
registry := metrics.NewRegistry()
client, err := NewMQTTClient(MQTTClientConfig{
EndpointName: "endpoint-a",
Broker: "tcp://127.0.0.1:1883",
ClientID: "test-client",
Topics: []string{"/ytforward/shln/+"},
Sink: sink,
Logger: slog.New(slog.NewTextHandler(testWriter{t: t}, nil)),
Metrics: registry,
DelegateFields: true,
})
if err != nil {
t.Fatalf("NewMQTTClient() error = %v", err)
}
client.handleMessage(context.Background(), "/ytforward/shln/dev1", []byte(`{
"device":"LTEST000000000001",
"time":"20260413100000",
"data":{"METER_SPEED":52.3,"TOTAL_MILEAGE":123456.7}
}`))
if len(sink.raw) != 1 || len(sink.fields) != 0 {
t.Fatalf("raw=%d fields=%d, want canonical raw only", len(sink.raw), len(sink.fields))
}
if len(sink.raw[0].Fields) != 0 {
t.Fatalf("delegated canonical raw must not carry bare standardized fields: %#v", sink.raw[0].Fields)
}
text := registry.Render()
for _, want := range []string{
`vehicle_gateway_fields_total{protocol="YUTONG_MQTT",status="delegated_to_bridge"} 1`,
`vehicle_gateway_fields_count{protocol="YUTONG_MQTT",status="delegated_to_bridge"} `,
`vehicle_gateway_publish_total{kind="fields",protocol="YUTONG_MQTT",status="delegated"} 1`,
} {
if !strings.Contains(text, want) {
t.Fatalf("delegated fields metric missing %s:\n%s", want, text)
}
}
}
func TestMQTTClientRecordsMessageMetrics(t *testing.T) {
@@ -77,8 +140,18 @@ func TestMQTTClientRecordsMessageMetrics(t *testing.T) {
text := registry.Render()
for _, want := range []string{
`vehicle_gateway_frames_total{protocol="YUTONG_MQTT",status="OK"} 1`,
`vehicle_gateway_last_frame_unix_seconds{protocol="YUTONG_MQTT",status="OK"} `,
`vehicle_gateway_identity_total{protocol="YUTONG_MQTT",status="resolved"} 1`,
`vehicle_gateway_identity_duration_ms{protocol="YUTONG_MQTT",status="resolved"}`,
`vehicle_gateway_identity_duration_ms_histogram_bucket{le="+Inf",protocol="YUTONG_MQTT",status="resolved"} 1`,
`vehicle_gateway_identity_duration_ms_histogram_count{protocol="YUTONG_MQTT",status="resolved"} 1`,
`vehicle_gateway_identity_duration_ms_histogram_sum{protocol="YUTONG_MQTT",status="resolved"}`,
`vehicle_gateway_publish_total{kind="raw",protocol="YUTONG_MQTT",status="ok"} 1`,
`vehicle_gateway_last_publish_unix_seconds{kind="raw",protocol="YUTONG_MQTT",status="ok"} `,
`vehicle_gateway_fields_total{protocol="YUTONG_MQTT",status="published"} 1`,
`vehicle_gateway_fields_count{protocol="YUTONG_MQTT",status="published"} `,
`vehicle_gateway_fields_count_histogram_count{protocol="YUTONG_MQTT",status="published"} 1`,
`vehicle_gateway_publish_total{kind="fields",protocol="YUTONG_MQTT",status="ok"} 1`,
`vehicle_gateway_frame_duration_ms{protocol="YUTONG_MQTT",status="OK"}`,
`vehicle_gateway_frame_duration_ms_histogram_bucket{le="+Inf",protocol="YUTONG_MQTT",status="OK"} 1`,
`vehicle_gateway_frame_duration_ms_histogram_count{protocol="YUTONG_MQTT",status="OK"} 1`,
@@ -93,6 +166,36 @@ func TestMQTTClientRecordsMessageMetrics(t *testing.T) {
}
}
func TestMQTTClientRecordsNonRealtimeFieldsSkipMetric(t *testing.T) {
registry := metrics.NewRegistry()
client, err := NewMQTTClient(MQTTClientConfig{
EndpointName: "endpoint-a",
Broker: "tcp://127.0.0.1:1883",
ClientID: "test-client",
Topics: []string{"/ytforward/shln/+"},
Sink: &recordingSink{},
Logger: slog.New(slog.NewTextHandler(testWriter{t: t}, nil)),
Metrics: registry,
})
if err != nil {
t.Fatalf("NewMQTTClient() error = %v", err)
}
client.handleMessage(context.Background(), "/ytforward/shln/dev1", []byte(`{
"device":"LTEST000000000001",
"time":"20260413100000",
"data":{}
}`))
text := registry.Render()
if !strings.Contains(text, `vehicle_gateway_identity_skips_total{protocol="YUTONG_MQTT",reason="non_vehicle_frame"} 1`) {
t.Fatalf("identity skip metric missing:\n%s", text)
}
if !strings.Contains(text, `vehicle_gateway_fields_total{protocol="YUTONG_MQTT",status="skipped_non_realtime"} 1`) {
t.Fatalf("non realtime fields skip metric missing:\n%s", text)
}
}
func TestMQTTClientHandleBadPayloadPublishesOnlyRaw(t *testing.T) {
sink := &recordingSink{}
registry := metrics.NewRegistry()
@@ -167,6 +270,66 @@ func TestMQTTClientUsesUncancelledMessageContextForReceivedMessage(t *testing.T)
}
}
func TestMQTTClientPreservesResolvedEnvelopeWhenIdentitySideEffectFails(t *testing.T) {
sink := &recordingSink{}
registry := metrics.NewRegistry()
wantErr := errors.New("registration upsert failed")
client, err := NewMQTTClient(MQTTClientConfig{
EndpointName: "endpoint-a",
Broker: "tcp://127.0.0.1:1883",
ClientID: "test-client",
Topics: []string{"/ytforward/shln/+"},
Sink: sink,
Resolver: resolvedErrorResolver{vin: "LRESOLVED00000001", err: wantErr},
Logger: slog.New(slog.NewTextHandler(testWriter{t: t}, nil)),
Metrics: registry,
})
if err != nil {
t.Fatalf("NewMQTTClient() error = %v", err)
}
client.handleMessage(context.Background(), "/ytforward/shln/dev1", []byte(`{
"device":"LTEST000000000001",
"time":"20260413100000",
"data":{"METER_SPEED":52.3,"TOTAL_MILEAGE":123456.7}
}`))
if len(sink.raw) != 1 {
t.Fatalf("raw=%d, want 1", len(sink.raw))
}
if sink.raw[0].VIN != "LRESOLVED00000001" {
t.Fatalf("raw vin = %q, want resolver vin", sink.raw[0].VIN)
}
if sink.raw[0].ParseStatus != envelope.ParsePartial {
t.Fatalf("parse status = %q, want PARTIAL", sink.raw[0].ParseStatus)
}
if len(sink.raw[0].Parsed) != 0 {
t.Fatalf("canonical raw should not duplicate parsed tree: %#v", sink.raw[0].Parsed)
}
if got := sink.raw[0].ParsedFields["yutong_mqtt.data.meter_speed"]; got != "52.3" {
t.Fatalf("protocol field lost after identity side-effect failure: %#v", got)
}
for field := range sink.raw[0].ParsedFields {
if strings.HasPrefix(field, "yutong_mqtt.identity.") {
t.Fatalf("derived identity annotation leaked into protocol fields: %s", field)
}
}
if len(sink.fields) != 1 || sink.fields[0].VIN != "LRESOLVED00000001" {
t.Fatalf("fields should preserve resolved vin, fields=%#v", sink.fields)
}
text := registry.Render()
for _, want := range []string{
`vehicle_gateway_identity_total{protocol="YUTONG_MQTT",status="error"} 1`,
`vehicle_gateway_identity_issues_total{message_id="MQTT",protocol="YUTONG_MQTT",reason="resolver_error",status="error"} 1`,
`vehicle_gateway_frames_total{protocol="YUTONG_MQTT",status="PARTIAL"} 1`,
`vehicle_gateway_fields_total{protocol="YUTONG_MQTT",status="published"} 1`,
} {
if !strings.Contains(text, want) {
t.Fatalf("metric missing %s:\n%s", want, text)
}
}
}
func TestMQTTClientPublishesUnifiedWhenExplicitlyEnabled(t *testing.T) {
sink := &recordingSink{}
client, err := NewMQTTClient(MQTTClientConfig{
@@ -193,6 +356,44 @@ func TestMQTTClientPublishesUnifiedWhenExplicitlyEnabled(t *testing.T) {
}
}
func TestMQTTClientPublishesUnifiedWhenFieldsPublishFails(t *testing.T) {
sink := &recordingSink{fieldsErr: errors.New("fields queue full")}
registry := metrics.NewRegistry()
client, err := NewMQTTClient(MQTTClientConfig{
EndpointName: "endpoint-a",
Broker: "tcp://127.0.0.1:1883",
ClientID: "test-client",
Topics: []string{"/ytforward/shln/+"},
PublishUnified: true,
Sink: sink,
Logger: slog.New(slog.NewTextHandler(testWriter{t: t}, nil)),
Metrics: registry,
})
if err != nil {
t.Fatalf("NewMQTTClient() error = %v", err)
}
client.handleMessage(context.Background(), "/ytforward/shln/dev1", []byte(`{
"device":"LTEST000000000001",
"time":"20260413100000",
"data":{"METER_SPEED":52.3,"TOTAL_MILEAGE":123456.7}
}`))
if len(sink.raw) != 1 || len(sink.fields) != 1 || len(sink.unified) != 1 {
t.Fatalf("raw=%d fields=%d unified=%d", len(sink.raw), len(sink.fields), len(sink.unified))
}
text := registry.Render()
for _, want := range []string{
`vehicle_gateway_fields_total{protocol="YUTONG_MQTT",status="publish_error"} 1`,
`vehicle_gateway_publish_total{kind="fields",protocol="YUTONG_MQTT",status="error"} 1`,
`vehicle_gateway_publish_total{kind="unified",protocol="YUTONG_MQTT",status="ok"} 1`,
} {
if !strings.Contains(text, want) {
t.Fatalf("metric missing %s:\n%s", want, text)
}
}
}
func TestMQTTClientBuildOptionsLoadsTLSCertificates(t *testing.T) {
dir := t.TempDir()
caPath, certPath, keyPath := writeTestTLSMaterial(t, dir)

368
go/vehicle-gateway/internal/gateway/tcp_server.go
View File

@@ -10,8 +10,10 @@ import (
"net"
"strings"
"sync"
"syscall"
"time"
"lingniu-vehicle-ingest/go/vehicle-gateway/internal/authentication"
"lingniu-vehicle-ingest/go/vehicle-gateway/internal/envelope"
"lingniu-vehicle-ingest/go/vehicle-gateway/internal/eventbus"
"lingniu-vehicle-ingest/go/vehicle-gateway/internal/identity"
@@ -26,11 +28,12 @@ type FrameParser func(raw []byte, receivedAtMS int64, sourceEndpoint string) (en
type FrameResponder func(raw []byte, env envelope.FrameEnvelope) (response []byte, ok bool, err error)
type TCPProtocol struct {
Protocol envelope.Protocol
Addr string
Extract FrameExtractor
Parse FrameParser
Respond FrameResponder
Protocol envelope.Protocol
Addr string
Extract FrameExtractor
Parse FrameParser
Authenticate authentication.Authenticator
Respond FrameResponder
}
type connectionState struct {
@@ -47,6 +50,9 @@ type TCPServer struct {
idleTimeout time.Duration
maxConnections int
publishUnified bool
delegateFields bool
activeMu sync.Mutex
activeConns map[net.Conn]struct{}
}
type TCPServerConfig struct {
@@ -59,11 +65,14 @@ type TCPServerConfig struct {
IdleTimeout time.Duration
MaxConnections int
PublishUnified bool
DelegateFields bool
}
const frameOperationTimeout = 30 * time.Second
var gatewayFrameDurationBucketsMS = []float64{1, 5, 10, 25, 50, 100, 250, 500, 1000, 5000}
var gatewayResponseDurationBucketsMS = []float64{1, 5, 10, 25, 50, 100, 250, 500, 1000, 5000}
var gatewayResponseE2ERecent = metrics.NewRecentLatencyByKey(512)
func NewTCPServer(cfg TCPServerConfig) (*TCPServer, error) {
if cfg.Protocol.Protocol == "" {
@@ -106,6 +115,8 @@ func NewTCPServer(cfg TCPServerConfig) (*TCPServer, error) {
idleTimeout: cfg.IdleTimeout,
maxConnections: cfg.MaxConnections,
publishUnified: cfg.PublishUnified,
delegateFields: cfg.DelegateFields,
activeConns: map[net.Conn]struct{}{},
}, nil
}
@@ -120,6 +131,7 @@ func (s *TCPServer) ListenAndServe(ctx context.Context) error {
go func() {
<-ctx.Done()
_ = listener.Close()
s.closeActiveConnections()
}()
s.logger.Info("tcp listener started", "protocol", s.protocol.Protocol, "addr", listener.Addr().String())
@@ -155,12 +167,15 @@ func (s *TCPServer) ListenAndServe(ctx context.Context) error {
func (s *TCPServer) handleConnection(ctx context.Context, conn net.Conn) {
defer conn.Close()
s.trackConnection(conn)
defer s.untrackConnection(conn)
source := conn.RemoteAddr().String()
log := s.logger.With("protocol", s.protocol.Protocol, "remote", source)
s.recordConnectionMetric(1)
defer s.recordConnectionMetric(-1)
log.Info("tcp connection opened")
defer log.Info("tcp connection closed")
log.Debug("tcp connection opened")
defer log.Debug("tcp connection closed")
readBuffer := make([]byte, s.readBufferSize)
var pending []byte
@@ -182,16 +197,21 @@ func (s *TCPServer) handleConnection(ctx context.Context, conn net.Conn) {
}
}
if err != nil {
if errors.Is(err, io.EOF) {
s.recordConnectionClose("eof")
if ctx.Err() != nil {
s.recordConnectionClose("context_cancelled")
return
}
var netErr net.Error
if errors.As(err, &netErr) && netErr.Timeout() {
log.Warn("tcp connection idle timeout")
log.Debug("tcp connection idle timeout")
s.recordConnectionClose("read_timeout")
return
}
if isRoutineTCPReadClose(err) {
log.Debug("tcp connection closed by peer", "error", err)
s.recordConnectionClose("remote_closed")
return
}
log.Warn("tcp read failed", "error", err)
s.recordConnectionClose("read_error")
return
@@ -203,6 +223,55 @@ func (s *TCPServer) handleConnection(ctx context.Context, conn net.Conn) {
}
}
func isRoutineTCPReadClose(err error) bool {
if err == nil {
return false
}
if errors.Is(err, io.EOF) ||
errors.Is(err, net.ErrClosed) ||
errors.Is(err, syscall.ECONNRESET) ||
errors.Is(err, syscall.EPIPE) {
return true
}
text := strings.ToLower(err.Error())
return strings.Contains(text, "connection reset by peer") ||
strings.Contains(text, "broken pipe") ||
strings.Contains(text, "use of closed network connection")
}
func (s *TCPServer) trackConnection(conn net.Conn) {
if conn == nil {
return
}
s.activeMu.Lock()
defer s.activeMu.Unlock()
if s.activeConns == nil {
s.activeConns = map[net.Conn]struct{}{}
}
s.activeConns[conn] = struct{}{}
}
func (s *TCPServer) untrackConnection(conn net.Conn) {
if conn == nil {
return
}
s.activeMu.Lock()
defer s.activeMu.Unlock()
delete(s.activeConns, conn)
}
func (s *TCPServer) closeActiveConnections() {
s.activeMu.Lock()
conns := make([]net.Conn, 0, len(s.activeConns))
for conn := range s.activeConns {
conns = append(conns, conn)
}
s.activeMu.Unlock()
for _, conn := range conns {
_ = conn.Close()
}
}
func (s *TCPServer) handleFrame(ctx context.Context, conn net.Conn, raw []byte, source string, state *connectionState) {
started := time.Now()
frameStatus := envelope.ParseBadFrame
@@ -228,29 +297,53 @@ func (s *TCPServer) handleFrame(ctx context.Context, conn net.Conn, raw []byte,
env.EventID = env.StableEventID()
s.recordParseErrorMetric(err)
} else {
resolved, resolveErr := s.resolver.Resolve(frameCtx, env)
if resolveErr != nil {
s.logger.Warn("identity resolve failed", "protocol", s.protocol.Protocol, "event_id", env.StableEventID(), "error", resolveErr)
if env.Parsed == nil {
env.Parsed = map[string]any{}
if s.protocol.Authenticate != nil {
result := s.protocol.Authenticate.Authenticate(env)
authentication.Apply(&env, result)
if result.Applicable {
s.recordAuthenticationMetric(result)
}
env.Parsed["identity"] = map[string]any{"resolved": false, "error": resolveErr.Error()}
env.ParseStatus = envelope.ParsePartial
s.recordIdentityMetric("error")
} else {
env = resolved
annotateIdentityUnresolved(&env)
s.recordIdentityMetric(identityStatus(env))
}
enrichConnectionPlatform(&env, state)
authentication.RedactParsedCredentials(&env)
if envelope.RequiresVehicleIdentity(env) {
resolveStarted := time.Now()
resolved, resolveErr := s.resolver.Resolve(frameCtx, env)
if resolveErr != nil {
if resolved.Protocol != "" || strings.TrimSpace(resolved.VIN) != "" || strings.TrimSpace(resolved.Phone) != "" {
env = resolved
}
identityStatus := identityErrorStatus(resolveErr)
s.recordIdentityDuration(identityStatus, time.Since(resolveStarted))
s.logger.Warn("identity resolve failed", "protocol", s.protocol.Protocol, "event_id", env.StableEventID(), "error", resolveErr)
annotateIdentityError(&env, resolveErr)
env.ParseStatus = envelope.ParsePartial
s.recordIdentityMetric(identityStatus)
s.recordIdentityIssueMetric(identityStatus, env, identityIssueReason(identityStatus, resolveErr))
} else {
env = resolved
annotateIdentityUnresolved(&env)
identityStatus := identityStatus(env)
s.recordIdentityDuration(identityStatus, time.Since(resolveStarted))
s.recordIdentityMetric(identityStatus)
if identityStatus != "resolved" {
s.recordIdentityIssueMetric(identityStatus, env, "no_binding")
}
recordGatewayIdentityCacheStatus(s.metrics, s.protocol.Protocol, env)
}
} else {
s.recordIdentitySkipMetric("non_vehicle_frame")
}
}
enrichConnectionPlatform(&env, state)
frameStatus = env.ParseStatus
env.EventKind = envelope.EventKindRaw
s.recordFrameMetric(env.ParseStatus)
if env.ParseStatus != envelope.ParseBadFrame {
realtime.EnsureParsedFields(&env)
}
if err := s.sink.PublishRaw(frameCtx, env); err != nil {
canonicalRaw := canonicalRawEnvelope(env)
if err := s.sink.PublishRaw(frameCtx, canonicalRaw); err != nil {
s.recordPublishMetric("raw", "error")
s.logger.Error("publish raw failed", "protocol", s.protocol.Protocol, "event_id", env.StableEventID(), "error", err)
return
@@ -259,37 +352,79 @@ func (s *TCPServer) handleFrame(ctx context.Context, conn net.Conn, raw []byte,
if env.ParseStatus == envelope.ParseBadFrame {
return
}
if fieldsEnv, ok := realtime.BuildFieldsEnvelope(env); ok {
if err := s.sink.PublishFields(frameCtx, fieldsEnv); err != nil {
s.recordPublishMetric("fields", "error")
s.logger.Error("publish fields failed", "protocol", s.protocol.Protocol, "event_id", env.StableEventID(), "error", err)
return
s.writeProtocolResponse(conn, raw, env, started)
if shouldCloseAfterAuthentication(env) && conn != nil {
_ = conn.Close()
return
}
if s.delegateFields {
fieldCount, fieldsStatus, ok := gatewayDelegatedFields(env)
s.recordFieldsMetric(fieldsStatus)
if ok {
s.recordFieldsCount(fieldsStatus, fieldCount)
s.recordPublishMetric("fields", "delegated")
}
} else {
fieldsEnv, fieldsStatus, ok := gatewayFieldsEnvelope(env)
if !ok {
s.recordFieldsMetric(fieldsStatus)
} else {
if err := s.sink.PublishFields(frameCtx, fieldsEnv); err != nil {
s.recordFieldsMetric(gatewayFieldsPublishError)
s.recordFieldsCount(gatewayFieldsPublishError, len(fieldsEnv.Fields))
s.recordPublishMetric("fields", "error")
s.logger.Error("publish fields failed", "protocol", s.protocol.Protocol, "event_id", env.StableEventID(), "error", err)
} else {
s.recordFieldsMetric(fieldsStatus)
s.recordFieldsCount(fieldsStatus, len(fieldsEnv.Fields))
s.recordPublishMetric("fields", "ok")
}
}
s.recordPublishMetric("fields", "ok")
}
if s.publishUnified {
if err := s.sink.PublishUnified(frameCtx, env); err != nil {
if err := s.sink.PublishUnified(frameCtx, canonicalRaw); err != nil {
s.recordPublishMetric("unified", "error")
s.logger.Error("publish unified failed", "protocol", s.protocol.Protocol, "event_id", env.StableEventID(), "error", err)
return
} else {
s.recordPublishMetric("unified", "ok")
}
s.recordPublishMetric("unified", "ok")
}
}
func (s *TCPServer) writeProtocolResponse(conn net.Conn, raw []byte, env envelope.FrameEnvelope, frameStarted time.Time) {
if s.protocol.Respond == nil {
return
}
status := "skipped"
defer func() {
s.recordResponseDuration(env.MessageID, status, time.Since(frameStarted))
}()
response, ok, err := s.protocol.Respond(raw, env)
if err != nil {
status = "build_error"
s.recordResponseMetric(env.MessageID, "build_error")
s.logger.Warn("build protocol response failed", "protocol", s.protocol.Protocol, "event_id", env.StableEventID(), "error", err)
return
}
if !ok || len(response) == 0 {
s.recordResponseMetric(env.MessageID, "skipped")
return
}
if conn == nil {
status = "write_error"
s.recordResponseMetric(env.MessageID, "write_error")
s.logger.Warn("write protocol response failed", "protocol", s.protocol.Protocol, "event_id", env.StableEventID(), "error", "nil connection")
return
}
_ = conn.SetWriteDeadline(time.Now().Add(5 * time.Second))
if _, err := conn.Write(response); err != nil {
status = "write_error"
s.recordResponseMetric(env.MessageID, "write_error")
s.logger.Warn("write protocol response failed", "protocol", s.protocol.Protocol, "event_id", env.StableEventID(), "error", err)
return
}
status = "ok"
s.recordResponseMetric(env.MessageID, "ok")
}
func enrichConnectionPlatform(env *envelope.FrameEnvelope, state *connectionState) {
@@ -299,6 +434,9 @@ func enrichConnectionPlatform(env *envelope.FrameEnvelope, state *connectionStat
if env.Fields == nil {
env.Fields = map[string]any{}
}
if shouldCloseAfterAuthentication(*env) {
return
}
if current := strings.TrimSpace(fmt.Sprint(env.Fields["platform_account"])); current != "" && current != "<nil>" {
state.platformName = current
}
@@ -308,6 +446,15 @@ func enrichConnectionPlatform(env *envelope.FrameEnvelope, state *connectionStat
if strings.TrimSpace(fmt.Sprint(env.Fields["platform_account"])) == "" || fmt.Sprint(env.Fields["platform_account"]) == "<nil>" {
env.Fields["platform_account"] = state.platformName
}
if strings.TrimSpace(env.PlatformName) == "" {
env.PlatformName = state.platformName
}
if strings.TrimSpace(env.SourceCode) == "" {
env.SourceCode = sourceCodeFromPlatformName(state.platformName)
}
if strings.TrimSpace(env.SourceKind) == "" || strings.EqualFold(strings.TrimSpace(env.SourceKind), "UNKNOWN") {
env.SourceKind = "PLATFORM"
}
if env.Parsed == nil {
env.Parsed = map[string]any{}
}
@@ -316,14 +463,37 @@ func enrichConnectionPlatform(env *envelope.FrameEnvelope, state *connectionStat
}
}
func sourceCodeFromPlatformName(value string) string {
value = strings.TrimSpace(value)
if value == "" {
return ""
}
var builder strings.Builder
for _, r := range value {
switch {
case r >= 'a' && r <= 'z':
builder.WriteRune(r)
case r >= 'A' && r <= 'Z':
builder.WriteRune(r)
case r >= '0' && r <= '9':
builder.WriteRune(r)
case r == '_' || r == '-' || r == '.':
builder.WriteRune(r)
}
}
return builder.String()
}
func (s *TCPServer) recordFrameMetric(status envelope.ParseStatus) {
if s.metrics == nil {
return
}
s.metrics.IncCounter("vehicle_gateway_frames_total", metrics.Labels{
labels := metrics.Labels{
"protocol": string(s.protocol.Protocol),
"status": string(status),
})
}
s.metrics.IncCounter("vehicle_gateway_frames_total", labels)
metrics.RecordLastActivity(s.metrics, "vehicle_gateway_last_frame_unix_seconds", labels)
}
func (s *TCPServer) recordFrameDuration(status envelope.ParseStatus, elapsed time.Duration) {
@@ -346,10 +516,74 @@ func (s *TCPServer) recordPublishMetric(kind string, status string) {
if s.metrics == nil {
return
}
s.metrics.IncCounter("vehicle_gateway_publish_total", metrics.Labels{
labels := metrics.Labels{
"protocol": string(s.protocol.Protocol),
"kind": kind,
"status": status,
}
s.metrics.IncCounter("vehicle_gateway_publish_total", labels)
metrics.RecordLastActivity(s.metrics, "vehicle_gateway_last_publish_unix_seconds", labels)
}
func (s *TCPServer) recordFieldsMetric(status string) {
recordGatewayFieldsMetric(s.metrics, s.protocol.Protocol, status)
}
func (s *TCPServer) recordFieldsCount(status string, fieldCount int) {
recordGatewayFieldsCount(s.metrics, s.protocol.Protocol, status, fieldCount)
}
func (s *TCPServer) recordResponseMetric(messageID string, status string) {
if s.metrics == nil {
return
}
messageID = strings.TrimSpace(messageID)
if messageID == "" {
messageID = "unknown"
}
labels := metrics.Labels{
"protocol": string(s.protocol.Protocol),
"message_id": messageID,
"status": status,
}
s.metrics.IncCounter("vehicle_gateway_response_total", labels)
metrics.RecordLastActivity(s.metrics, "vehicle_gateway_last_response_unix_seconds", labels)
}
func (s *TCPServer) recordResponseDuration(messageID string, status string, elapsed time.Duration) {
if s.metrics == nil {
return
}
messageID = strings.TrimSpace(messageID)
if messageID == "" {
messageID = "unknown"
}
elapsedMS := float64(elapsed.Microseconds()) / 1000
labels := metrics.Labels{
"protocol": string(s.protocol.Protocol),
"message_id": messageID,
"status": status,
}
s.metrics.ObserveHistogram("vehicle_gateway_response_duration_ms_histogram", labels, gatewayResponseDurationBucketsMS, elapsedMS)
if status != "ok" {
return
}
protocol := string(s.protocol.Protocol)
p99, samples := gatewayResponseE2ERecent.Observe(protocol, elapsedMS)
protocolLabels := metrics.Labels{"protocol": protocol}
s.metrics.SetGauge("vehicle_gateway_response_e2e_recent_p99_ms", protocolLabels, p99)
s.metrics.SetGauge("vehicle_gateway_response_e2e_recent_samples", protocolLabels, float64(samples))
}
func (s *TCPServer) recordAuthenticationMetric(result authentication.Result) {
if s.metrics == nil || !result.Applicable {
return
}
s.metrics.IncCounter("vehicle_gateway_authentication_total", metrics.Labels{
"protocol": string(s.protocol.Protocol),
"mode": string(result.Mode),
"source": result.Source,
"status": result.Status,
})
}
@@ -373,6 +607,40 @@ func (s *TCPServer) recordIdentityMetric(status string) {
})
}
func (s *TCPServer) recordIdentitySkipMetric(reason string) {
if s.metrics == nil {
return
}
s.metrics.IncCounter("vehicle_gateway_identity_skips_total", metrics.Labels{
"protocol": string(s.protocol.Protocol),
"reason": reason,
})
}
func (s *TCPServer) recordIdentityIssueMetric(status string, env envelope.FrameEnvelope, reason string) {
if s.metrics == nil {
return
}
messageID := strings.TrimSpace(env.MessageID)
if messageID == "" {
messageID = "unknown"
}
reason = strings.TrimSpace(reason)
if reason == "" {
reason = "unknown"
}
s.metrics.IncCounter("vehicle_gateway_identity_issues_total", metrics.Labels{
"protocol": string(s.protocol.Protocol),
"status": status,
"message_id": messageID,
"reason": reason,
})
}
func (s *TCPServer) recordIdentityDuration(status string, elapsed time.Duration) {
recordGatewayIdentityDuration(s.metrics, s.protocol.Protocol, status, elapsed)
}
func (s *TCPServer) recordConnectionMetric(delta float64) {
if s.metrics == nil {
return
@@ -402,6 +670,12 @@ func (s *TCPServer) recordConnectionClose(reason string) {
})
}
func shouldCloseAfterAuthentication(env envelope.FrameEnvelope) bool {
return env.AuthenticationEnforced &&
strings.TrimSpace(env.AuthenticationStatus) != "" &&
env.AuthenticationStatus != authentication.StatusAccepted
}
func identityStatus(env envelope.FrameEnvelope) string {
if strings.TrimSpace(env.VIN) != "" {
return "resolved"
@@ -412,6 +686,28 @@ func identityStatus(env envelope.FrameEnvelope) string {
return "unresolved"
}
func identityIssueReason(status string, err error) string {
switch status {
case "timeout":
return "timeout"
case "error":
if err == nil {
return "resolver_error"
}
text := strings.ToLower(strings.TrimSpace(err.Error()))
switch {
case strings.Contains(text, "connection"), strings.Contains(text, "tcp"), strings.Contains(text, "network"):
return "identity_store_connection"
case strings.Contains(text, "timeout"), strings.Contains(text, "deadline"):
return "timeout"
default:
return "resolver_error"
}
default:
return "no_binding"
}
}
func annotateIdentityUnresolved(env *envelope.FrameEnvelope) {
if env == nil || env.ParseStatus == envelope.ParseBadFrame || strings.TrimSpace(env.VIN) != "" {
return

587
go/vehicle-gateway/internal/gateway/tcp_server_test.go
View File

@@ -3,20 +3,48 @@ package gateway
import (
"context"
"encoding/hex"
"errors"
"fmt"
"io"
"log/slog"
"net"
"os"
"strings"
"syscall"
"testing"
"time"
"lingniu-vehicle-ingest/go/vehicle-gateway/internal/authentication"
"lingniu-vehicle-ingest/go/vehicle-gateway/internal/envelope"
"lingniu-vehicle-ingest/go/vehicle-gateway/internal/metrics"
"lingniu-vehicle-ingest/go/vehicle-gateway/internal/protocol/gb32960"
"lingniu-vehicle-ingest/go/vehicle-gateway/internal/protocol/jt808"
)
func TestTCPServerPublishesGoodFrameOnlyToRawByDefault(t *testing.T) {
func TestEnrichConnectionPlatformPromotesSourceMetadata(t *testing.T) {
env := envelope.FrameEnvelope{
Protocol: envelope.ProtocolGB32960,
MessageID: "0x02",
ParseStatus: envelope.ParseOK,
Fields: map[string]any{},
Parsed: map[string]any{},
}
state := &connectionState{platformName: "Hyundai"}
enrichConnectionPlatform(&env, state)
if env.PlatformName != "Hyundai" || env.SourceCode != "Hyundai" || env.SourceKind != "PLATFORM" {
t.Fatalf("source metadata = code:%q platform:%q kind:%q", env.SourceCode, env.PlatformName, env.SourceKind)
}
if got := fmt.Sprint(env.Fields["platform_account"]); got != "Hyundai" {
t.Fatalf("platform_account = %q", got)
}
if got := fmt.Sprint(env.Parsed["platform_name"]); got != "Hyundai" {
t.Fatalf("parsed platform_name = %q", got)
}
}
func TestTCPServerPublishesGoodFrameToRawAndFieldsByDefault(t *testing.T) {
frame, err := hex.DecodeString("7E020000320133077954250001000000000048000301D2C4C707376139000A00E6004F26063016235701040001900C2504000000000202000030011F31010F867E")
if err != nil {
t.Fatal(err)
@@ -28,6 +56,7 @@ func TestTCPServerPublishesGoodFrameOnlyToRawByDefault(t *testing.T) {
Extract: jt808.ExtractFrames,
Parse: jt808.ParseFrame,
}, sink)
server.resolver = vinResolver{vin: "LNBVIN00000000001"}
client, done := runPipe(t, server)
if _, err := client.Write(frame); err != nil {
@@ -39,11 +68,71 @@ func TestTCPServerPublishesGoodFrameOnlyToRawByDefault(t *testing.T) {
if len(sink.raw) != 1 || len(sink.unified) != 0 {
t.Fatalf("raw=%d unified=%d", len(sink.raw), len(sink.unified))
}
if len(sink.fields) != 1 {
t.Fatalf("fields=%d, want 1", len(sink.fields))
}
if sink.raw[0].Phone != "13307795425" {
t.Fatalf("phone = %q", sink.raw[0].Phone)
}
if sink.raw[0].Fields[envelope.FieldTotalMileageKM] != 10241.2 {
t.Fatalf("total mileage = %#v", sink.raw[0].Fields[envelope.FieldTotalMileageKM])
if sink.raw[0].EventKind != envelope.EventKindRaw {
t.Fatalf("raw event kind = %q, want %q", sink.raw[0].EventKind, envelope.EventKindRaw)
}
if len(sink.raw[0].Fields) != 0 {
t.Fatalf("canonical raw must not carry bare standardized fields: %#v", sink.raw[0].Fields)
}
if got := sink.raw[0].ParsedFields["jt808.location.total_mileage_km"]; got == nil {
t.Fatalf("raw parsed fields missing total mileage: %#v", sink.raw[0].ParsedFields)
}
if got, want := sink.fields[0].Fields["jt808.location.total_mileage_km"], sink.raw[0].ParsedFields["jt808.location.total_mileage_km"]; got != want {
t.Fatalf("fields event should reuse raw parsed field, got %#v want %#v", got, want)
}
if sink.fields[0].EventKind != envelope.EventKindFields {
t.Fatalf("fields event kind = %q, want %q", sink.fields[0].EventKind, envelope.EventKindFields)
}
if got, want := sink.fields[0].SourceEventID, sink.raw[0].StableEventID(); got != want {
t.Fatalf("fields source event id = %#v, want %s", got, want)
}
if sink.fields[0].FieldMapping == "" {
t.Fatal("fields event should expose field mapping version")
}
if len(sink.fields[0].Parsed) != 0 || len(sink.fields[0].ParsedFields) != 0 {
t.Fatalf("fields envelope should not duplicate parsed payload: parsed=%#v parsed_fields=%#v", sink.fields[0].Parsed, sink.fields[0].ParsedFields)
}
}
func TestTCPServerRecordsFieldsPublishedMetric(t *testing.T) {
frame, err := hex.DecodeString("7E020000320133077954250001000000000048000301D2C4C707376139000A00E6004F26063016235701040001900C2504000000000202000030011F31010F867E")
if err != nil {
t.Fatal(err)
}
sink := &recordingSink{}
registry := metrics.NewRegistry()
server := newTestServer(t, TCPProtocol{
Protocol: envelope.ProtocolJT808,
Addr: ":0",
Extract: jt808.ExtractFrames,
Parse: jt808.ParseFrame,
}, sink)
server.metrics = registry
server.resolver = vinResolver{vin: "LNBVIN00000000001"}
client, done := runPipe(t, server)
if _, err := client.Write(frame); err != nil {
t.Fatalf("client.Write() error = %v", err)
}
_ = client.Close()
<-done
text := registry.Render()
for _, want := range []string{
`vehicle_gateway_fields_total{protocol="JT808",status="published"} 1`,
`vehicle_gateway_fields_count{protocol="JT808",status="published"} `,
`vehicle_gateway_fields_count_histogram_count{protocol="JT808",status="published"} 1`,
`vehicle_gateway_publish_total{kind="fields",protocol="JT808",status="ok"} 1`,
} {
if !strings.Contains(text, want) {
t.Fatalf("metrics missing %s:\n%s", want, text)
}
}
}
@@ -79,8 +168,16 @@ func TestTCPServerRecordsFrameMetrics(t *testing.T) {
text := registry.Render()
for _, want := range []string{
`vehicle_gateway_frames_total{protocol="JT808",status="OK"} 1`,
`vehicle_gateway_last_frame_unix_seconds{protocol="JT808",status="OK"} `,
`vehicle_gateway_identity_total{protocol="JT808",status="unresolved"} 1`,
`vehicle_gateway_identity_issues_total{message_id="0x0200",protocol="JT808",reason="no_binding",status="unresolved"} 1`,
`vehicle_gateway_identity_duration_ms{protocol="JT808",status="unresolved"}`,
`vehicle_gateway_identity_duration_ms_histogram_bucket{le="+Inf",protocol="JT808",status="unresolved"} 1`,
`vehicle_gateway_identity_duration_ms_histogram_count{protocol="JT808",status="unresolved"} 1`,
`vehicle_gateway_identity_duration_ms_histogram_sum{protocol="JT808",status="unresolved"}`,
`vehicle_gateway_publish_total{kind="raw",protocol="JT808",status="ok"} 1`,
`vehicle_gateway_last_publish_unix_seconds{kind="raw",protocol="JT808",status="ok"} `,
`vehicle_gateway_fields_total{protocol="JT808",status="skipped_non_realtime"} 1`,
`vehicle_gateway_frame_duration_ms{protocol="JT808",status="OK"}`,
`vehicle_gateway_frame_duration_ms_histogram_bucket{le="+Inf",protocol="JT808",status="OK"} 1`,
`vehicle_gateway_frame_duration_ms_histogram_count{protocol="JT808",status="OK"} 1`,
@@ -95,6 +192,244 @@ func TestTCPServerRecordsFrameMetrics(t *testing.T) {
}
}
func TestTCPServerRecordsNonRealtimeFieldsSkipMetric(t *testing.T) {
registry := metrics.NewRegistry()
sink := &recordingSink{}
server := newTestServer(t, TCPProtocol{
Protocol: envelope.ProtocolGB32960,
Addr: ":0",
Extract: gb32960.ExtractFrames,
Parse: gb32960.ParseFrame,
}, sink)
server.metrics = registry
server.handleFrame(context.Background(), nil, buildGBFrame(0x07, 0xfe, "LNBSCB3D4R1234567", nil), "127.0.0.1:32960", &connectionState{})
if len(sink.raw) != 1 {
t.Fatalf("raw=%d, want 1", len(sink.raw))
}
if len(sink.fields) != 0 {
t.Fatalf("fields=%d, want 0", len(sink.fields))
}
if text := registry.Render(); !strings.Contains(text, `vehicle_gateway_fields_total{protocol="GB32960",status="skipped_non_realtime"} 1`) {
t.Fatalf("non realtime fields skip metric missing:\n%s", text)
}
}
func TestTCPServerSkipsIdentityForGB32960PlatformControlFrame(t *testing.T) {
registry := metrics.NewRegistry()
sink := &recordingSink{}
resolver := &countingResolver{vin: "LNBVIN00000000001"}
server := newTestServer(t, TCPProtocol{
Protocol: envelope.ProtocolGB32960,
Addr: ":0",
Extract: gb32960.ExtractFrames,
Parse: gb32960.ParseFrame,
}, sink)
server.metrics = registry
server.resolver = resolver
server.handleFrame(context.Background(), nil, buildGBFrame(0x05, 0xfe, "PLATFORMLOGIN0001", nil), "127.0.0.1:32960", &connectionState{})
if resolver.calls != 0 {
t.Fatalf("resolver calls = %d, want 0 for platform control frame", resolver.calls)
}
if len(sink.raw) != 1 {
t.Fatalf("raw=%d, want 1", len(sink.raw))
}
text := registry.Render()
if strings.Contains(text, "vehicle_gateway_identity_total") {
t.Fatalf("identity metric should not be recorded for platform control frame:\n%s", text)
}
if !strings.Contains(text, `vehicle_gateway_identity_skips_total{protocol="GB32960",reason="non_vehicle_frame"} 1`) {
t.Fatalf("identity skip metric missing:\n%s", text)
}
if !strings.Contains(text, `vehicle_gateway_fields_total{protocol="GB32960",status="skipped_non_realtime"} 1`) {
t.Fatalf("non realtime fields skip metric missing:\n%s", text)
}
}
func TestTCPServerAuthenticatesAndRedactsGB32960PlatformLogin(t *testing.T) {
registry := metrics.NewRegistry()
sink := &recordingSink{}
server := newTestServer(t, TCPProtocol{
Protocol: envelope.ProtocolGB32960,
Addr: ":0",
Extract: gb32960.ExtractFrames,
Parse: gb32960.ParseFrame,
Authenticate: authentication.NewGB32960PlatformAuthenticator(authentication.ModeObserve, map[string][]string{"platform-a": {"secret-a"}}),
}, sink)
server.metrics = registry
body := []byte{0x1a, 0x07, 0x0d, 0x14, 0x00, 0x00, 0x00, 0x01}
body = append(body, fixedASCIIBytes("platform-a", 12)...)
body = append(body, fixedASCIIBytes("secret-a", 20)...)
body = append(body, 0x01)
server.handleFrame(context.Background(), nil, buildGBFrame(0x05, 0xfe, "12345678901234567", body), "127.0.0.1:32960", &connectionState{})
if len(sink.raw) != 1 {
t.Fatalf("raw=%d, want 1", len(sink.raw))
}
raw := sink.raw[0]
if raw.AuthenticationMode != "observe" || raw.AuthenticationStatus != authentication.StatusAccepted || raw.AuthenticationEnforced {
t.Fatalf("authentication metadata = mode:%q status:%q enforced:%v", raw.AuthenticationMode, raw.AuthenticationStatus, raw.AuthenticationEnforced)
}
if _, exists := raw.ParsedFields["gb32960.platform_login.password"]; exists {
t.Fatalf("plaintext password leaked into parsed fields: %#v", raw.ParsedFields)
}
if got := raw.ParsedFields["gb32960.platform_login.password_present"]; got != "true" {
t.Fatalf("password presence marker = %#v", got)
}
if text := registry.Render(); !strings.Contains(text, `vehicle_gateway_authentication_total{mode="observe",protocol="GB32960",source="configured",status="accepted"} 1`) {
t.Fatalf("authentication metric missing:\n%s", text)
}
}
func TestTCPServerResolvesIdentityForGB32960RealtimeFrame(t *testing.T) {
registry := metrics.NewRegistry()
sink := &recordingSink{}
resolver := &countingResolver{vin: "LNBVIN00000000001"}
server := newTestServer(t, TCPProtocol{
Protocol: envelope.ProtocolGB32960,
Addr: ":0",
Extract: gb32960.ExtractFrames,
Parse: gb32960.ParseFrame,
}, sink)
server.metrics = registry
server.resolver = resolver
server.handleFrame(context.Background(), nil, buildGBFrame(0x02, 0xfe, "LNBSCB3D4R1234567", nil), "127.0.0.1:32960", &connectionState{})
if resolver.calls != 1 {
t.Fatalf("resolver calls = %d, want 1 for realtime frame", resolver.calls)
}
if len(sink.raw) != 1 {
t.Fatalf("raw=%d, want 1", len(sink.raw))
}
if sink.raw[0].VIN != "LNBVIN00000000001" {
t.Fatalf("raw vin = %q, want resolved vin", sink.raw[0].VIN)
}
if text := registry.Render(); !strings.Contains(text, `vehicle_gateway_identity_total{protocol="GB32960",status="resolved"} 1`) {
t.Fatalf("identity resolved metric missing:\n%s", text)
}
}
func TestTCPServerRecordsIdentityTimeoutMetrics(t *testing.T) {
registry := metrics.NewRegistry()
server, err := NewTCPServer(TCPServerConfig{
Protocol: TCPProtocol{
Protocol: envelope.ProtocolJT808,
Addr: ":0",
Extract: jt808.ExtractFrames,
Parse: func(_ []byte, receivedAtMS int64, sourceEndpoint string) (envelope.FrameEnvelope, error) {
return envelope.FrameEnvelope{
Protocol: envelope.ProtocolJT808,
MessageID: "0x0200",
Phone: "13307795425",
SourceEndpoint: sourceEndpoint,
ReceivedAtMS: receivedAtMS,
EventTimeMS: receivedAtMS,
ParseStatus: envelope.ParseOK,
}, nil
},
},
Sink: &recordingSink{},
Resolver: errorResolver{err: context.DeadlineExceeded},
Logger: slog.New(slog.NewTextHandler(testWriter{t: t}, nil)),
Metrics: registry,
})
if err != nil {
t.Fatalf("NewTCPServer() error = %v", err)
}
server.handleFrame(context.Background(), nil, []byte{0x01}, "127.0.0.1:808", &connectionState{})
text := registry.Render()
for _, want := range []string{
`vehicle_gateway_identity_total{protocol="JT808",status="timeout"} 1`,
`vehicle_gateway_identity_issues_total{message_id="0x0200",protocol="JT808",reason="timeout",status="timeout"} 1`,
`vehicle_gateway_identity_duration_ms{protocol="JT808",status="timeout"}`,
`vehicle_gateway_identity_duration_ms_histogram_bucket{le="+Inf",protocol="JT808",status="timeout"} 1`,
`vehicle_gateway_identity_duration_ms_histogram_count{protocol="JT808",status="timeout"} 1`,
`vehicle_gateway_frames_total{protocol="JT808",status="PARTIAL"} 1`,
} {
if !strings.Contains(text, want) {
t.Fatalf("metrics missing %s:\n%s", want, text)
}
}
}
func TestTCPServerPreservesResolvedEnvelopeWhenIdentitySideEffectFails(t *testing.T) {
registry := metrics.NewRegistry()
sink := &recordingSink{}
wantErr := errors.New("registration upsert failed")
server, err := NewTCPServer(TCPServerConfig{
Protocol: TCPProtocol{
Protocol: envelope.ProtocolJT808,
Addr: ":0",
Extract: jt808.ExtractFrames,
Parse: func(_ []byte, receivedAtMS int64, sourceEndpoint string) (envelope.FrameEnvelope, error) {
return envelope.FrameEnvelope{
Protocol: envelope.ProtocolJT808,
MessageID: "0x0200",
Phone: "13307795425",
SourceEndpoint: sourceEndpoint,
ReceivedAtMS: receivedAtMS,
EventTimeMS: receivedAtMS,
ParsedFields: map[string]any{
"jt808.location.total_mileage_km": "10241.2",
},
ParseStatus: envelope.ParseOK,
}, nil
},
},
Sink: sink,
Resolver: resolvedErrorResolver{vin: "LNBVIN00000000001", err: wantErr},
Logger: slog.New(slog.NewTextHandler(testWriter{t: t}, nil)),
Metrics: registry,
})
if err != nil {
t.Fatalf("NewTCPServer() error = %v", err)
}
server.handleFrame(context.Background(), nil, []byte{0x01}, "127.0.0.1:808", &connectionState{})
if len(sink.raw) != 1 {
t.Fatalf("raw count = %d, want 1", len(sink.raw))
}
if sink.raw[0].VIN != "LNBVIN00000000001" {
t.Fatalf("raw vin = %q, want resolved vin", sink.raw[0].VIN)
}
if sink.raw[0].ParseStatus != envelope.ParsePartial {
t.Fatalf("parse status = %q, want PARTIAL", sink.raw[0].ParseStatus)
}
if len(sink.raw[0].Parsed) != 0 {
t.Fatalf("canonical raw should not duplicate parsed tree: %#v", sink.raw[0].Parsed)
}
if got := sink.raw[0].ParsedFields["jt808.location.total_mileage_km"]; got != "10241.2" {
t.Fatalf("protocol field lost after identity side-effect failure: %#v", got)
}
for field := range sink.raw[0].ParsedFields {
if strings.HasPrefix(field, "jt808.identity.") {
t.Fatalf("derived identity annotation leaked into protocol fields: %s", field)
}
}
if len(sink.fields) != 1 || sink.fields[0].VIN != "LNBVIN00000000001" {
t.Fatalf("fields should preserve resolved vin, fields=%#v", sink.fields)
}
text := registry.Render()
for _, want := range []string{
`vehicle_gateway_identity_total{protocol="JT808",status="error"} 1`,
`vehicle_gateway_identity_issues_total{message_id="0x0200",protocol="JT808",reason="resolver_error",status="error"} 1`,
`vehicle_gateway_frames_total{protocol="JT808",status="PARTIAL"} 1`,
`vehicle_gateway_fields_total{protocol="JT808",status="published"} 1`,
} {
if !strings.Contains(text, want) {
t.Fatalf("metric missing %s:\n%s", want, text)
}
}
}
func TestTCPServerRecordsActiveConnectionGauge(t *testing.T) {
registry := metrics.NewRegistry()
server, err := NewTCPServer(TCPServerConfig{
@@ -140,6 +475,45 @@ func TestTCPServerRecordsActiveConnectionGauge(t *testing.T) {
}
}
func TestTCPServerClosesActiveConnectionWhenContextCancelled(t *testing.T) {
registry := metrics.NewRegistry()
server, err := NewTCPServer(TCPServerConfig{
Protocol: TCPProtocol{
Protocol: envelope.ProtocolJT808,
Addr: ":0",
Extract: jt808.ExtractFrames,
Parse: jt808.ParseFrame,
},
Sink: &recordingSink{},
Logger: slog.New(slog.NewTextHandler(testWriter{t: t}, nil)),
Metrics: registry,
IdleTimeout: time.Minute,
})
if err != nil {
t.Fatalf("NewTCPServer() error = %v", err)
}
ctx, cancel := context.WithCancel(context.Background())
client, srv := net.Pipe()
done := make(chan struct{})
go func() {
server.handleConnection(ctx, srv)
close(done)
}()
waitForMetric(t, registry, `vehicle_gateway_active_connections{protocol="JT808"} 1`)
cancel()
server.closeActiveConnections()
defer client.Close()
select {
case <-done:
case <-time.After(time.Second):
t.Fatal("connection handler did not exit after context cancellation")
}
if text := registry.Render(); !strings.Contains(text, `vehicle_gateway_connection_closes_total{protocol="JT808",reason="context_cancelled"} 1`) {
t.Fatalf("context cancellation close metric missing:\n%s", text)
}
}
func TestTCPServerRecordsConnectionRejectionMetric(t *testing.T) {
registry := metrics.NewRegistry()
server, err := NewTCPServer(TCPServerConfig{
@@ -188,6 +562,57 @@ func TestTCPServerRecordsConnectionCloseReasonMetric(t *testing.T) {
}
}
func TestTCPServerConnectionLifecycleLogsStayDebug(t *testing.T) {
source, err := os.ReadFile("tcp_server.go")
if err != nil {
t.Fatal(err)
}
text := string(source)
for _, forbidden := range []string{
`Info("tcp connection opened"`,
`Info("tcp connection closed"`,
`Warn("tcp connection idle timeout"`,
`Warn("tcp connection closed by peer"`,
} {
if strings.Contains(text, forbidden) {
t.Fatalf("high-volume connection lifecycle log should not be info/warn: %s", forbidden)
}
}
for _, want := range []string{
`Debug("tcp connection opened"`,
`Debug("tcp connection closed"`,
`Debug("tcp connection idle timeout"`,
`Debug("tcp connection closed by peer"`,
} {
if !strings.Contains(text, want) {
t.Fatalf("connection lifecycle log should remain debug: %s", want)
}
}
}
func TestIsRoutineTCPReadCloseClassifiesRemoteCloseNoise(t *testing.T) {
tests := []struct {
name string
err error
want bool
}{
{name: "eof", err: io.EOF, want: true},
{name: "net closed", err: net.ErrClosed, want: true},
{name: "reset wrapped", err: fmt.Errorf("read tcp: %w", syscall.ECONNRESET), want: true},
{name: "pipe wrapped", err: fmt.Errorf("write tcp: %w", syscall.EPIPE), want: true},
{name: "reset text", err: errors.New("read tcp 172.17.111.55:808->117.132.196.176:22187: read: connection reset by peer"), want: true},
{name: "unexpected", err: errors.New("checksum parser exploded"), want: false},
{name: "nil", err: nil, want: false},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
if got := isRoutineTCPReadClose(tt.err); got != tt.want {
t.Fatalf("isRoutineTCPReadClose(%v) = %v, want %v", tt.err, got, tt.want)
}
})
}
}
func TestTCPServerPublishesBadFrameOnlyToRaw(t *testing.T) {
good := buildGBFrame(0x02, 0xfe, "LNBSCB3D4R1234567", nil)
good[len(good)-1] ^= 0xff
@@ -248,14 +673,14 @@ func TestTCPServerAnnotatesUnresolvedIdentity(t *testing.T) {
if len(sink.raw) != 1 {
t.Fatalf("raw count = %d", len(sink.raw))
}
identity, ok := sink.raw[0].Parsed["identity"].(map[string]any)
if !ok || identity["resolved"] != false || identity["reason"] != "no_binding" {
t.Fatalf("identity metadata = %#v", sink.raw[0].Parsed["identity"])
if len(sink.raw[0].Parsed) != 0 || len(sink.raw[0].ParsedFields) != 0 {
t.Fatalf("unresolved derived metadata must not enter canonical protocol payload: parsed=%#v fields=%#v", sink.raw[0].Parsed, sink.raw[0].ParsedFields)
}
}
func TestTCPServerWritesProtocolResponseAfterPublish(t *testing.T) {
frame := buildGBFrame(0x07, 0xfe, "LNBSCB3D4R1234567", nil)
registry := metrics.NewRegistry()
sink := &recordingSink{}
server := newTestServer(t, TCPProtocol{
Protocol: envelope.ProtocolGB32960,
@@ -269,6 +694,7 @@ func TestTCPServerWritesProtocolResponseAfterPublish(t *testing.T) {
return []byte("ACK"), true, nil
},
}, sink)
server.metrics = registry
client, done := runPipe(t, server)
if _, err := client.Write(frame); err != nil {
@@ -283,6 +709,79 @@ func TestTCPServerWritesProtocolResponseAfterPublish(t *testing.T) {
}
_ = client.Close()
<-done
text := registry.Render()
for _, want := range []string{
`vehicle_gateway_response_total{message_id="0x07",protocol="GB32960",status="ok"} 1`,
`vehicle_gateway_last_response_unix_seconds{message_id="0x07",protocol="GB32960",status="ok"} `,
`vehicle_gateway_response_duration_ms_histogram_count{message_id="0x07",protocol="GB32960",status="ok"} 1`,
`vehicle_gateway_response_e2e_recent_p99_ms{protocol="GB32960"} `,
`vehicle_gateway_response_e2e_recent_samples{protocol="GB32960"} `,
} {
if !strings.Contains(text, want) {
t.Fatalf("response metric missing %s:\n%s", want, text)
}
}
}
func TestTCPServerWritesProtocolResponseWhenFieldsPublishFails(t *testing.T) {
registry := metrics.NewRegistry()
sink := &recordingSink{fieldsErr: errors.New("fields queue full")}
server := newTestServer(t, TCPProtocol{
Protocol: envelope.ProtocolJT808,
Addr: ":0",
Extract: func(raw []byte) ([][]byte, []byte, error) {
return [][]byte{raw}, nil, nil
},
Parse: func(_ []byte, receivedAtMS int64, sourceEndpoint string) (envelope.FrameEnvelope, error) {
return envelope.FrameEnvelope{
Protocol: envelope.ProtocolJT808,
MessageID: "0x0200",
VIN: "LNBVIN00000000001",
Phone: "13307795425",
SourceEndpoint: sourceEndpoint,
ReceivedAtMS: receivedAtMS,
EventTimeMS: receivedAtMS,
ParsedFields: map[string]any{
"jt808.location.total_mileage_km": "10241.2",
},
ParseStatus: envelope.ParseOK,
}, nil
},
Respond: func(_ []byte, env envelope.FrameEnvelope) ([]byte, bool, error) {
if len(sink.raw) != 1 || sink.raw[0].EventID != env.EventID {
t.Fatalf("response built before raw publish: raw=%d", len(sink.raw))
}
return []byte("ACK"), true, nil
},
}, sink)
server.metrics = registry
client, done := runPipe(t, server)
if _, err := client.Write([]byte{0x01}); err != nil {
t.Fatalf("client.Write() error = %v", err)
}
buf := make([]byte, 3)
if _, err := io.ReadFull(client, buf); err != nil {
t.Fatalf("read response error = %v", err)
}
if string(buf) != "ACK" {
t.Fatalf("response = %q", string(buf))
}
_ = client.Close()
<-done
if len(sink.fields) != 1 {
t.Fatalf("fields publish attempts = %d, want 1", len(sink.fields))
}
text := registry.Render()
for _, want := range []string{
`vehicle_gateway_response_total{message_id="0x0200",protocol="JT808",status="ok"} 1`,
`vehicle_gateway_fields_total{protocol="JT808",status="publish_error"} 1`,
`vehicle_gateway_publish_total{kind="fields",protocol="JT808",status="error"} 1`,
} {
if !strings.Contains(text, want) {
t.Fatalf("metric missing %s:\n%s", want, text)
}
}
}
func TestTCPServerUsesUncancelledFrameContextForAlreadyReadFrame(t *testing.T) {
@@ -368,6 +867,49 @@ func (r *contextCheckingResolver) Resolve(ctx context.Context, env envelope.Fram
return env, r.ctxErr
}
type vinResolver struct {
vin string
}
func (r vinResolver) Resolve(_ context.Context, env envelope.FrameEnvelope) (envelope.FrameEnvelope, error) {
env.VIN = r.vin
return env, nil
}
type countingResolver struct {
calls int
vin string
}
func (r *countingResolver) Resolve(_ context.Context, env envelope.FrameEnvelope) (envelope.FrameEnvelope, error) {
r.calls++
env.VIN = r.vin
return env, nil
}
type errorResolver struct {
err error
}
func (r errorResolver) Resolve(_ context.Context, env envelope.FrameEnvelope) (envelope.FrameEnvelope, error) {
return env, r.err
}
type resolvedErrorResolver struct {
vin string
err error
}
func (r resolvedErrorResolver) Resolve(_ context.Context, env envelope.FrameEnvelope) (envelope.FrameEnvelope, error) {
env.VIN = r.vin
if env.Parsed == nil {
env.Parsed = map[string]any{}
}
env.Parsed["identity"] = map[string]any{"resolved": true, "source": "test"}
env.EventID = env.StableEventID()
return env, r.err
}
type contextCheckingSink struct {
rawCtxErr error
unifiedCtxErr error
@@ -426,25 +968,40 @@ func runPipe(t *testing.T, server *TCPServer) (net.Conn, <-chan struct{}) {
return client, done
}
func waitForMetric(t *testing.T, registry *metrics.Registry, want string) {
t.Helper()
deadline := time.Now().Add(2 * time.Second)
for time.Now().Before(deadline) {
if strings.Contains(registry.Render(), want) {
return
}
time.Sleep(10 * time.Millisecond)
}
t.Fatalf("metric missing %s:\n%s", want, registry.Render())
}
type recordingSink struct {
raw []envelope.FrameEnvelope
unified []envelope.FrameEnvelope
fields []envelope.FrameEnvelope
raw []envelope.FrameEnvelope
unified []envelope.FrameEnvelope
fields []envelope.FrameEnvelope
rawErr error
unifiedErr error
fieldsErr error
}
func (s *recordingSink) PublishRaw(_ context.Context, env envelope.FrameEnvelope) error {
s.raw = append(s.raw, env)
return nil
return s.rawErr
}
func (s *recordingSink) PublishUnified(_ context.Context, env envelope.FrameEnvelope) error {
s.unified = append(s.unified, env)
return nil
return s.unifiedErr
}
func (s *recordingSink) PublishFields(_ context.Context, env envelope.FrameEnvelope) error {
s.fields = append(s.fields, env)
return nil
return s.fieldsErr
}
func (s *recordingSink) Close() error {
@@ -475,3 +1032,9 @@ func buildGBFrame(command byte, response byte, vin string, body []byte) []byte {
}
return append(frame, bcc)
}
func fixedASCIIBytes(value string, size int) []byte {
out := make([]byte, size)
copy(out, []byte(value))
return out
}

1
go/vehicle-gateway/internal/health/health.go
View File

@@ -40,6 +40,7 @@ func NewMux(service string, checks []Check, registry *metrics.Registry) *http.Se
mux.Handle("/healthz", handler)
mux.Handle("/readyz", handler)
if registry != nil {
metrics.RegisterServiceInfo(registry, handler.service)
mux.Handle("/metrics", metrics.NewHandler(registry))
}
return mux

7
go/vehicle-gateway/internal/health/health_test.go
View File

@@ -80,6 +80,13 @@ func TestNewMuxRegistersHealthAndReadinessRoutes(t *testing.T) {
t.Fatalf("%s status = %d body=%s", path, response.Code, response.Body.String())
}
}
request := httptest.NewRequest(http.MethodGet, "/metrics", nil)
response := httptest.NewRecorder()
mux.ServeHTTP(response, request)
if !strings.Contains(response.Body.String(), `vehicle_service_info{service="vehicle-stat-writer"} 1`) {
t.Fatalf("service info metric missing: %s", response.Body.String())
}
}
func TestNewServerReturnsNilWhenAddressIsEmpty(t *testing.T) {

10
go/vehicle-gateway/internal/history/query.go
View File

@@ -76,6 +76,7 @@ type LocationRow struct {
Latitude float64 `json:"latitude"`
AltitudeM *float64 `json:"altitude_m,omitempty"`
SpeedKMH *float64 `json:"speed_kmh,omitempty"`
SOCPercent *float64 `json:"soc_percent,omitempty"`
DirectionDeg *int64 `json:"direction_deg,omitempty"`
AlarmFlag *int64 `json:"alarm_flag,omitempty"`
StatusFlag *int64 `json:"status_flag,omitempty"`
@@ -218,6 +219,7 @@ func (r *LocationRepository) Query(ctx context.Context, query LocationQuery) ([]
var receivedAt scanDateTime
var altitude sql.NullFloat64
var speed sql.NullFloat64
var soc sql.NullFloat64
var direction sql.NullInt64
var alarm sql.NullInt64
var status sql.NullInt64
@@ -230,6 +232,7 @@ func (r *LocationRepository) Query(ctx context.Context, query LocationQuery) ([]
&row.Latitude,
&altitude,
&speed,
&soc,
&direction,
&alarm,
&status,
@@ -243,6 +246,7 @@ func (r *LocationRepository) Query(ctx context.Context, query LocationQuery) ([]
row.ReceivedAt = receivedAt.String
row.AltitudeM = nullableFloat(altitude)
row.SpeedKMH = nullableFloat(speed)
row.SOCPercent = nullableFloat(soc)
row.DirectionDeg = nullableInt(direction)
row.AlarmFlag = nullableInt(alarm)
row.StatusFlag = nullableInt(status)
@@ -563,7 +567,7 @@ func quotedList(values []string) string {
func buildLocationSQL(table string, query LocationQuery) (string, []any) {
where := locationWhere(query)
sqlText := `SELECT ts, event_id, received_at, longitude, latitude, altitude_m, speed_kmh, direction_deg, alarm_flag, status_flag, total_mileage_km, protocol, vin FROM ` + table
sqlText := `SELECT ts, event_id, received_at, longitude, latitude, altitude_m, speed_kmh, soc_percent, direction_deg, alarm_flag, status_flag, total_mileage_km, protocol, vin FROM ` + table
if len(where) > 0 {
sqlText += " WHERE " + strings.Join(where, " AND ")
}
@@ -954,11 +958,11 @@ func normalizeDateTimeLiteral(value string) string {
shanghai := time.FixedZone("Asia/Shanghai", 8*3600)
for _, layout := range []string{"2006-01-02T15:04:05", "2006-01-02 15:04:05"} {
if parsed, err := time.ParseInLocation(layout, value, shanghai); err == nil {
return parsed.UTC().Format("2006-01-02 15:04:05")
return parsed.In(shanghai).Format("2006-01-02 15:04:05")
}
}
if parsed, err := time.Parse(time.RFC3339, value); err == nil {
return parsed.UTC().Format("2006-01-02 15:04:05")
return parsed.In(shanghai).Format("2006-01-02 15:04:05")
}
return value
}

28
go/vehicle-gateway/internal/history/query_test.go
View File

@@ -391,10 +391,10 @@ func TestLocationHandlerReturnsLocationsByVIN(t *testing.T) {
mock.ExpectQuery("vin = 'LKLG7C4E3NA774736'").
WillReturnRows(sqlmock.NewRows([]string{
"ts", "event_id", "received_at", "longitude", "latitude", "altitude_m", "speed_kmh",
"direction_deg", "alarm_flag", "status_flag", "total_mileage_km", "protocol", "vin",
"soc_percent", "direction_deg", "alarm_flag", "status_flag", "total_mileage_km", "protocol", "vin",
}).AddRow(
"2026-07-02 00:18:22", "event-3", "2026-07-02 00:22:43",
121.07764, 30.585928, 11.0, 8.0, 171, 0, 4718595, 8792.8,
121.07764, 30.585928, 11.0, 8.0, 82.5, 171, 0, 4718595, 8792.8,
"JT808", "LKLG7C4E3NA774736",
))
@@ -408,7 +408,7 @@ func TestLocationHandlerReturnsLocationsByVIN(t *testing.T) {
t.Fatalf("status = %d body=%s", response.Code, response.Body.String())
}
body := response.Body.String()
for _, want := range []string{`"vin":"LKLG7C4E3NA774736"`, `"longitude":121.07764`, `"total_mileage_km":8792.8`, `"total":17`} {
for _, want := range []string{`"vin":"LKLG7C4E3NA774736"`, `"longitude":121.07764`, `"soc_percent":82.5`, `"total_mileage_km":8792.8`, `"total":17`} {
if !strings.Contains(body, want) {
t.Fatalf("response missing %s: %s", want, body)
}
@@ -432,10 +432,10 @@ func TestLocationHandlerSkipsTotalCountByDefault(t *testing.T) {
mock.ExpectQuery("vin = 'LKLG7C4E3NA774736'").
WillReturnRows(sqlmock.NewRows([]string{
"ts", "event_id", "received_at", "longitude", "latitude", "altitude_m", "speed_kmh",
"direction_deg", "alarm_flag", "status_flag", "total_mileage_km", "protocol", "vin",
"soc_percent", "direction_deg", "alarm_flag", "status_flag", "total_mileage_km", "protocol", "vin",
}).AddRow(
"2026-07-02 00:18:22", "event-3", "2026-07-02 00:22:43",
121.07764, 30.585928, 11.0, 8.0, 171, 0, 4718595, 8792.8,
121.07764, 30.585928, 11.0, 8.0, 82.5, 171, 0, 4718595, 8792.8,
"JT808", "LKLG7C4E3NA774736",
))
@@ -476,17 +476,17 @@ func TestParseRawFrameQueryAcceptsDatetimeLocalValues(t *testing.T) {
if err != nil {
t.Fatalf("parseRawFrameQuery() error = %v", err)
}
if query.DateFrom != "2026-06-30 16:00:00" || query.DateTo != "2026-07-01 16:00:00" {
if query.DateFrom != "2026-07-01 00:00:00" || query.DateTo != "2026-07-02 00:00:00" {
t.Fatalf("date range = %q -> %q", query.DateFrom, query.DateTo)
}
}
func TestNormalizeDateTimeLiteralConvertsInputToTDengineUTCTime(t *testing.T) {
func TestNormalizeDateTimeLiteralUsesAsiaShanghaiQueryTime(t *testing.T) {
for raw, want := range map[string]string{
"2026-07-01T00:00:00": "2026-06-30 16:00:00",
"2026-07-01 00:00:00": "2026-06-30 16:00:00",
"2026-07-01T00:00:00+08:00": "2026-06-30 16:00:00",
"2026-06-30T16:00:00Z": "2026-06-30 16:00:00",
"2026-07-01T00:00:00": "2026-07-01 00:00:00",
"2026-07-01 00:00:00": "2026-07-01 00:00:00",
"2026-07-01T00:00:00+08:00": "2026-07-01 00:00:00",
"2026-06-30T16:00:00Z": "2026-07-01 00:00:00",
} {
if got := normalizeDateTimeLiteral(raw); got != want {
t.Fatalf("normalizeDateTimeLiteral(%q) = %q, want %q", raw, got, want)
@@ -522,7 +522,7 @@ func TestBuildLocationSQLUsesLiteralsForTDengine(t *testing.T) {
for _, want := range []string{
"protocol = 'JT808'",
"vin = 'LKLG7C4E3NA774736'",
"ts >= '2026-07-01 16:00:00'",
"ts >= '2026-07-02 00:00:00'",
"LIMIT 20 OFFSET 5",
} {
if !strings.Contains(sqlText, want) {
@@ -556,8 +556,8 @@ func TestBuildRawFrameSQLUsesLiteralsForTDengine(t *testing.T) {
"vehicle_key = 'JT808:013307811350'",
"vin = 'VIN''1'",
"message_id = 512",
"ts >= '2026-06-30 16:00:00'",
"ts <= '2026-07-01 15:59:59'",
"ts >= '2026-07-01 00:00:00'",
"ts <= '2026-07-01 23:59:59'",
"LIMIT 20 OFFSET 5",
} {
if !strings.Contains(sqlText, want) {

11
go/vehicle-gateway/internal/history/schema.go
View File

@@ -54,6 +54,7 @@ func SchemaStatements(database string) []string {
latitude DOUBLE,
altitude_m DOUBLE,
speed_kmh DOUBLE,
soc_percent DOUBLE,
direction_deg INT,
alarm_flag BIGINT,
status_flag BIGINT,
@@ -64,3 +65,13 @@ func SchemaStatements(database string) []string {
)`,
}
}
// SchemaMigrationStatements contains additive TDengine changes that must also
// be applied to an already existing stable. The writer treats duplicate-column
// errors as success so startup remains idempotent across releases.
func SchemaMigrationStatements(database string) []string {
if database == "" {
database = DefaultDatabase
}
return []string{"ALTER STABLE " + database + ".vehicle_locations ADD COLUMN soc_percent DOUBLE"}
}

339
go/vehicle-gateway/internal/history/writer.go
View File

@@ -7,6 +7,7 @@ import (
"encoding/hex"
"encoding/json"
"fmt"
"sort"
"strconv"
"strings"
"sync"
@@ -15,6 +16,7 @@ import (
"lingniu-vehicle-ingest/go/vehicle-gateway/internal/envelope"
"lingniu-vehicle-ingest/go/vehicle-gateway/internal/realtime"
"lingniu-vehicle-ingest/go/vehicle-gateway/internal/telemetry"
)
type Execer interface {
@@ -27,9 +29,23 @@ type Writer struct {
cache tableCache
}
type AppendResult struct {
RawRows int
LocationRows int
LocationError error
}
const (
LocationStatusOK = "ok"
LocationStatusSkippedNonRealtime = "skipped_non_realtime"
LocationStatusSkippedMissingVIN = "skipped_missing_vin"
LocationStatusSkippedMissingCoordinates = "skipped_missing_coordinates"
)
const (
rawFramePayloadInlineLimit = 12_000
rawFramePayloadChunkSize = 16_000
tdengineInsertSoftLimit = 6 * 1024 * 1024
)
type payloadChunk struct {
@@ -100,9 +116,22 @@ func (w *Writer) EnsureSchema(ctx context.Context, database string) error {
return err
}
}
for _, statement := range SchemaMigrationStatements(database) {
if _, err := w.exec.ExecContext(ctx, statement); err != nil && !isDuplicateTDengineColumnError(err) {
return err
}
}
return nil
}
func isDuplicateTDengineColumnError(err error) bool {
if err == nil {
return false
}
message := strings.ToLower(err.Error())
return strings.Contains(message, "duplicate column") || strings.Contains(message, "duplicated column") || strings.Contains(message, "column already exists")
}
func (w *Writer) qualify(table string) string {
table = normalizeIdentifier(table)
if w.database == "" {
@@ -112,20 +141,52 @@ func (w *Writer) qualify(table string) string {
}
func (w *Writer) AppendAll(ctx context.Context, env envelope.FrameEnvelope) error {
if err := w.AppendRawFrame(ctx, env); err != nil {
result, err := w.AppendAllWithResult(ctx, env)
if err != nil {
return err
}
return w.AppendLocation(ctx, env)
return result.LocationError
}
func (w *Writer) AppendAllWithResult(ctx context.Context, env envelope.FrameEnvelope) (AppendResult, error) {
var result AppendResult
if err := w.AppendRawFrame(ctx, env); err != nil {
return result, err
}
result.RawRows = 1
rows, err := w.appendLocationWithCount(ctx, env)
if err != nil {
result.LocationError = err
return result, nil
}
result.LocationRows = rows
return result, nil
}
func (w *Writer) AppendAllBatch(ctx context.Context, envelopes []envelope.FrameEnvelope) error {
if len(envelopes) == 0 {
return nil
}
if err := w.AppendRawFrameBatch(ctx, envelopes); err != nil {
result, err := w.AppendAllBatchWithResult(ctx, envelopes)
if err != nil {
return err
}
return w.AppendLocationBatch(ctx, envelopes)
return result.LocationError
}
func (w *Writer) AppendAllBatchWithResult(ctx context.Context, envelopes []envelope.FrameEnvelope) (AppendResult, error) {
var result AppendResult
if len(envelopes) == 0 {
return result, nil
}
if err := w.AppendRawFrameBatch(ctx, envelopes); err != nil {
return result, err
}
result.RawRows = len(envelopes)
rows, err := w.appendLocationBatchWithCount(ctx, envelopes)
if err != nil {
result.LocationError = err
return result, nil
}
result.LocationRows = rows
return result, nil
}
func (w *Writer) AppendRawFrame(ctx context.Context, env envelope.FrameEnvelope) error {
@@ -165,7 +226,6 @@ VALUES (%s)`, w.qualify(chunkTable), joinLiterals(chunkValues(env, chunk)))); er
func (w *Writer) AppendRawFrameBatch(ctx context.Context, envelopes []envelope.FrameEnvelope) error {
rowsByTable := map[string][]string{}
chunkRowsByTable := map[string][]string{}
chunkEnvByTable := map[string]envelope.FrameEnvelope{}
for _, env := range envelopes {
table := tableName("raw", env)
if err := w.ensureRawChild(ctx, table, "raw_frames", env); err != nil {
@@ -184,87 +244,141 @@ func (w *Writer) AppendRawFrameBatch(ctx context.Context, envelopes []envelope.F
if err := w.ensureRawChild(ctx, chunkTable, "raw_frame_payload_chunks", env); err != nil {
return err
}
chunkEnvByTable[chunkTable] = env
for _, chunk := range chunks {
chunkRowsByTable[chunkTable] = append(chunkRowsByTable[chunkTable], "("+joinLiterals(chunkValues(env, chunk))+")")
}
}
for table, rows := range rowsByTable {
if len(rows) == 0 {
continue
}
if _, err := w.exec.ExecContext(ctx, fmt.Sprintf(`INSERT INTO %s
(ts, frame_id, event_id, message_id, event_time, received_at, raw_size_bytes,
raw_hex, raw_text, parsed_json, parse_status, parse_error, source_endpoint)
VALUES %s`, w.qualify(table), strings.Join(rows, ","))); err != nil {
return err
}
if err := w.execMultiTableInsert(ctx, rowsByTable, `ts, frame_id, event_id, message_id, event_time, received_at, raw_size_bytes,
raw_hex, raw_text, parsed_json, parse_status, parse_error, source_endpoint`); err != nil {
return err
}
for table, rows := range chunkRowsByTable {
if len(rows) == 0 {
continue
}
_ = chunkEnvByTable[table]
if _, err := w.exec.ExecContext(ctx, fmt.Sprintf(`INSERT INTO %s
(ts, event_id, frame_id, received_at, payload_kind, chunk_index, chunk_count, chunk_text)
VALUES %s`, w.qualify(table), strings.Join(rows, ","))); err != nil {
return err
}
if err := w.execMultiTableInsert(ctx, chunkRowsByTable, `ts, event_id, frame_id, received_at, payload_kind, chunk_index, chunk_count, chunk_text`); err != nil {
return err
}
return nil
}
func (w *Writer) AppendLocation(ctx context.Context, env envelope.FrameEnvelope) error {
if strings.TrimSpace(env.VIN) == "" {
return nil
}
longitude, okLon := floatField(env, envelope.FieldLongitude)
latitude, okLat := floatField(env, envelope.FieldLatitude)
if !okLon || !okLat {
return nil
}
table := locationTableName(env)
if err := w.ensureLocationChild(ctx, table, env); err != nil {
return err
}
_, err := w.exec.ExecContext(ctx, fmt.Sprintf(`INSERT INTO %s
(ts, event_id, received_at, longitude, latitude, altitude_m, speed_kmh,
direction_deg, alarm_flag, status_flag, total_mileage_km)
VALUES (%s)`, w.qualify(table), joinLiterals(locationValues(env, longitude, latitude))))
_, err := w.appendLocationWithCount(ctx, env)
return err
}
func (w *Writer) appendLocationWithCount(ctx context.Context, env envelope.FrameEnvelope) (int, error) {
longitude, latitude, status := locationCandidate(env)
if status != LocationStatusOK {
return 0, nil
}
table := locationTableName(env)
if err := w.ensureLocationChild(ctx, table, env); err != nil {
return 0, err
}
_, err := w.exec.ExecContext(ctx, fmt.Sprintf(`INSERT INTO %s
(ts, event_id, received_at, longitude, latitude, altitude_m, speed_kmh,
soc_percent, direction_deg, alarm_flag, status_flag, total_mileage_km)
VALUES (%s)`, w.qualify(table), joinLiterals(locationValues(env, longitude, latitude))))
if err != nil {
return 0, err
}
return 1, nil
}
func (w *Writer) AppendLocationBatch(ctx context.Context, envelopes []envelope.FrameEnvelope) error {
_, err := w.appendLocationBatchWithCount(ctx, envelopes)
return err
}
func (w *Writer) appendLocationBatchWithCount(ctx context.Context, envelopes []envelope.FrameEnvelope) (int, error) {
rowsByTable := map[string][]string{}
rowCount := 0
for _, env := range envelopes {
if strings.TrimSpace(env.VIN) == "" {
continue
}
longitude, okLon := floatField(env, envelope.FieldLongitude)
latitude, okLat := floatField(env, envelope.FieldLatitude)
if !okLon || !okLat {
longitude, latitude, status := locationCandidate(env)
if status != LocationStatusOK {
continue
}
table := locationTableName(env)
if err := w.ensureLocationChild(ctx, table, env); err != nil {
return err
return rowCount, err
}
rowsByTable[table] = append(rowsByTable[table], "("+joinLiterals(locationValues(env, longitude, latitude))+")")
rowCount++
}
for table, rows := range rowsByTable {
if len(rows) == 0 {
continue
}
if _, err := w.exec.ExecContext(ctx, fmt.Sprintf(`INSERT INTO %s
(ts, event_id, received_at, longitude, latitude, altitude_m, speed_kmh,
direction_deg, alarm_flag, status_flag, total_mileage_km)
VALUES %s`, w.qualify(table), strings.Join(rows, ","))); err != nil {
if err := w.execMultiTableInsert(ctx, rowsByTable, `ts, event_id, received_at, longitude, latitude, altitude_m, speed_kmh,
soc_percent, direction_deg, alarm_flag, status_flag, total_mileage_km`); err != nil {
return rowCount, err
}
return rowCount, nil
}
func (w *Writer) execMultiTableInsert(ctx context.Context, rowsByTable map[string][]string, columns string) error {
statements := buildMultiTableInsertStatements(rowsByTable, w.qualify, columns, tdengineInsertSoftLimit)
for _, statement := range statements {
if _, err := w.exec.ExecContext(ctx, statement); err != nil {
return err
}
}
return nil
}
func buildMultiTableInsertStatements(rowsByTable map[string][]string, qualify func(string) string, columns string, softLimit int) []string {
if len(rowsByTable) == 0 {
return nil
}
keys := make([]string, 0, len(rowsByTable))
for table, rows := range rowsByTable {
if len(rows) > 0 {
keys = append(keys, table)
}
}
sort.Strings(keys)
if len(keys) == 0 {
return nil
}
if qualify == nil {
qualify = func(table string) string { return table }
}
var statements []string
current := "INSERT INTO "
parts := 0
for _, table := range keys {
part := fmt.Sprintf(`%s
(%s)
VALUES %s`, qualify(table), columns, strings.Join(rowsByTable[table], ","))
if parts > 0 && softLimit > 0 && len(current)+1+len(part) > softLimit {
statements = append(statements, current)
current = "INSERT INTO "
parts = 0
}
if parts > 0 {
current += " "
}
current += part
parts++
}
if parts > 0 {
statements = append(statements, current)
}
return statements
}
func LocationStatus(env envelope.FrameEnvelope) string {
_, _, status := locationCandidate(env)
return status
}
func locationCandidate(env envelope.FrameEnvelope) (float64, float64, string) {
if !envelope.IsRealtimeTelemetryFrame(env) {
return 0, 0, LocationStatusSkippedNonRealtime
}
if strings.TrimSpace(env.VIN) == "" {
return 0, 0, LocationStatusSkippedMissingVIN
}
location, ok := telemetry.LocationProjectionForProtocol(env.Protocol, env.ParsedFields)
if !ok {
return 0, 0, LocationStatusSkippedMissingCoordinates
}
return location.Longitude, location.Latitude, LocationStatusOK
}
func (w *Writer) ensureRawChild(ctx context.Context, table string, stable string, env envelope.FrameEnvelope) error {
key := stable + "." + table
return w.cache.doOnce(key, func() error {
@@ -325,7 +439,7 @@ func rawValues(env envelope.FrameEnvelope, rawHex string, rawText string, parsed
func chunkValues(env envelope.FrameEnvelope, chunk payloadChunk) []any {
received := millis(env.ReceivedAtMS)
return []any{
received,
received.Add(time.Duration(chunk.Index) * time.Millisecond),
env.StableEventID(),
frameID(env),
received,
@@ -379,18 +493,21 @@ func safeChunkEnd(value string, start int, maxBytes int) int {
func locationValues(env envelope.FrameEnvelope, longitude float64, latitude float64) []any {
received := millis(env.ReceivedAtMS)
location, _ := telemetry.LocationProjectionForProtocol(env.Protocol, env.ParsedFields)
totalMileageKM, hasTotalMileage := telemetry.TotalMileageKM(env.Protocol, env.ParsedFields)
return []any{
eventTimeOrReceived(env),
env.StableEventID(),
received,
longitude,
latitude,
floatFieldOrNil(env, "altitude_m"),
floatFieldOrNil(env, envelope.FieldSpeedKMH),
intFieldOrNil(env, "direction_deg"),
intFieldOrNil(env, "alarm_flag"),
intFieldOrNil(env, "status_flag"),
floatFieldOrNil(env, envelope.FieldTotalMileageKM),
optionalFloat(location.AltitudeM),
optionalFloat(location.SpeedKMH),
optionalFloat(location.SOCPercent),
optionalInt(location.DirectionDeg),
optionalInt64(location.AlarmFlag),
optionalInt64(location.StatusFlag),
optionalPositiveFloat(totalMileageKM, hasTotalMileage),
}
}
@@ -463,76 +580,37 @@ func parsedFieldsJSONString(env envelope.FrameEnvelope) string {
return jsonString(fields)
}
func floatField(env envelope.FrameEnvelope, key string) (float64, bool) {
if env.Fields == nil {
return 0, false
}
value, ok := env.Fields[key]
if !ok || value == nil {
return 0, false
}
switch typed := value.(type) {
case float64:
return typed, true
case float32:
return float64(typed), true
case int:
return float64(typed), true
case int64:
return float64(typed), true
case uint16:
return float64(typed), true
case uint32:
return float64(typed), true
case string:
parsed, err := strconv.ParseFloat(strings.TrimSpace(typed), 64)
return parsed, err == nil
default:
return 0, false
func optionalFloat(value *float64) any {
if value == nil {
return nil
}
return *value
}
func floatFieldOrNil(env envelope.FrameEnvelope, key string) any {
value, ok := floatField(env, key)
if !ok {
func optionalInt(value *float64) any {
if value == nil {
return nil
}
return int64(*value)
}
func optionalInt64(value *int64) any {
if value == nil {
return nil
}
return *value
}
func optionalPositiveFloat(value float64, ok bool) any {
if !ok || value <= 0 {
return nil
}
return value
}
func intFieldOrNil(env envelope.FrameEnvelope, key string) any {
if env.Fields == nil {
return nil
}
value, ok := env.Fields[key]
if !ok || value == nil {
return nil
}
switch typed := value.(type) {
case int:
return typed
case int64:
return typed
case uint16:
return int64(typed)
case uint32:
return int64(typed)
case float64:
return int64(typed)
case string:
parsed, err := strconv.ParseInt(strings.TrimSpace(typed), 10, 64)
if err == nil {
return parsed
}
}
return nil
}
func eventTimeOrReceived(env envelope.FrameEnvelope) time.Time {
if env.EventTimeMS > 0 {
return millis(env.EventTimeMS)
}
return millis(env.ReceivedAtMS)
eventMS, _ := envelope.NormalizedEventTimeMS(env)
return millis(eventMS)
}
func millis(value int64) time.Time {
@@ -543,7 +621,10 @@ func millis(value int64) time.Time {
}
func quote(value string) string {
return strings.ReplaceAll(value, "'", "''")
return strings.NewReplacer(
`\`, `\\`,
`'`, `''`,
).Replace(value)
}
func joinLiterals(values []any) string {

235
go/vehicle-gateway/internal/history/writer_test.go
View File

@@ -3,6 +3,8 @@ package history
import (
"context"
"database/sql"
"errors"
"strconv"
"strings"
"sync"
"testing"
@@ -32,6 +34,18 @@ func TestSchemaStatementsCreateCoreStables(t *testing.T) {
}
}
func TestSchemaMigrationAddsTrackSOCIdempotently(t *testing.T) {
statements := strings.Join(SchemaMigrationStatements("test_ts"), "\n")
if !strings.Contains(statements, "ALTER STABLE test_ts.vehicle_locations ADD COLUMN soc_percent DOUBLE") {
t.Fatalf("SOC migration missing: %s", statements)
}
for _, message := range []string{"duplicate column name", "Duplicated column names", "column already exists"} {
if !isDuplicateTDengineColumnError(errors.New(message)) {
t.Fatalf("duplicate TDengine column error should be idempotent: %s", message)
}
}
}
func TestWriterAppendsRawAndLocationOnly(t *testing.T) {
exec := &recordingExec{}
writer := NewWriter(exec)
@@ -130,6 +144,16 @@ func TestWriterChunksOversizedParsedFields(t *testing.T) {
if got := countSQL(exec.calls, "INSERT INTO chunk_"); got < 2 {
t.Fatalf("chunk insert count = %d, calls=%v", got, exec.calls)
}
chunkInserts := matchingSQL(exec.calls, "INSERT INTO chunk_")
if len(chunkInserts) != 2 {
t.Fatalf("chunk inserts = %d, calls=%v", len(chunkInserts), exec.calls)
}
if !strings.Contains(chunkInserts[0], "VALUES (1782745114999, '") {
t.Fatalf("first chunk ts should use received_at: %s", chunkInserts[0])
}
if !strings.Contains(chunkInserts[1], "VALUES (1782745115000, '") {
t.Fatalf("second chunk ts should be offset by chunk_index ms: %s", chunkInserts[1])
}
}
func TestWriterUsesEnvelopeParsedFieldsWithoutReflattening(t *testing.T) {
@@ -185,11 +209,20 @@ func TestTimeLiteralsUseEpochMilliseconds(t *testing.T) {
}
}
func TestStringLiteralPreservesJSONEscapesForTDengine(t *testing.T) {
value := `{"bits":"{\"abs\":false}"}`
want := `'{"bits":"{\\"abs\\":false}"}'`
if got := literal(value); got != want {
t.Fatalf("literal() = %q, want %q", got, want)
}
}
func TestWriterSkipsSparseDerivedRows(t *testing.T) {
exec := &recordingExec{}
writer := NewWriter(exec)
env := sampleEnvelope()
env.Fields = map[string]any{}
env.ParsedFields = map[string]any{}
if err := writer.AppendAll(context.Background(), env); err != nil {
t.Fatalf("AppendAll() error = %v", err)
@@ -227,6 +260,86 @@ func TestWriterSkipsLocationWhenVINIsMissing(t *testing.T) {
}
}
func TestLocationStatusClassifiesDerivedLocationEligibility(t *testing.T) {
realtimeWithoutCoordinates := sampleEnvelope()
realtimeWithoutCoordinates.ParsedFields = map[string]any{
"jt808.location.speed_kmh": 30,
}
bareFieldsOnly := realtimeWithoutCoordinates
bareFieldsOnly.ParsedFields = nil
bareFieldsOnly.Fields = map[string]any{
envelope.FieldLatitude: 30.590151,
envelope.FieldLongitude: 121.069881,
}
withoutVIN := sampleEnvelope()
withoutVIN.VIN = ""
nonRealtimeWithCoordinates := sampleEnvelope()
nonRealtimeWithCoordinates.MessageID = "0x0100"
tests := []struct {
name string
env envelope.FrameEnvelope
want string
}{
{name: "ok", env: sampleEnvelope(), want: LocationStatusOK},
{name: "non realtime", env: nonRealtimeWithCoordinates, want: LocationStatusSkippedNonRealtime},
{name: "missing vin", env: withoutVIN, want: LocationStatusSkippedMissingVIN},
{name: "missing coordinates", env: realtimeWithoutCoordinates, want: LocationStatusSkippedMissingCoordinates},
{name: "bare fields are not canonical", env: bareFieldsOnly, want: LocationStatusSkippedMissingCoordinates},
}
for _, test := range tests {
if got := LocationStatus(test.env); got != test.want {
t.Fatalf("%s LocationStatus() = %q, want %q", test.name, got, test.want)
}
}
}
func TestWriterSkipsLocationForNonRealtimeFrame(t *testing.T) {
exec := &recordingExec{}
writer := NewWriter(exec)
env := sampleEnvelope()
env.MessageID = "0x0100"
env.Parsed = map[string]any{"registration": map[string]any{"plate": "沪A12345"}}
if err := writer.AppendAll(context.Background(), env); err != nil {
t.Fatalf("AppendAll() error = %v", err)
}
if got := countSQL(exec.calls, "INSERT INTO raw_"); got != 1 {
t.Fatalf("raw insert count = %d", got)
}
if got := countSQL(exec.calls, "USING vehicle_locations"); got != 0 {
t.Fatalf("location child create count = %d", got)
}
if got := countSQL(exec.calls, "INSERT INTO loc_"); got != 0 {
t.Fatalf("location insert count = %d", got)
}
}
func TestWriterAppendAllWithResultKeepsRawSuccessWhenLocationFails(t *testing.T) {
locationErr := errors.New("location insert failed")
exec := &recordingExec{errs: []error{nil, nil, nil, nil, locationErr}}
writer := NewWriter(exec)
env := sampleEnvelope()
result, err := writer.AppendAllWithResult(context.Background(), env)
if err != nil {
t.Fatalf("AppendAllWithResult() raw error = %v", err)
}
if !errors.Is(result.LocationError, locationErr) {
t.Fatalf("location error = %v, want %v", result.LocationError, locationErr)
}
if result.RawRows != 1 || result.LocationRows != 0 {
t.Fatalf("result = %+v, want raw row retained and no location rows", result)
}
if got := countSQL(exec.calls, "INSERT INTO raw_"); got != 1 {
t.Fatalf("raw insert count = %d", got)
}
if got := countSQL(exec.calls, "INSERT INTO loc_"); got != 1 {
t.Fatalf("location insert attempted count = %d", got)
}
}
func TestWriterAppendsBatchRowsByChildTable(t *testing.T) {
exec := &recordingExec{}
writer := NewWriter(exec)
@@ -262,6 +375,94 @@ func TestWriterAppendsBatchRowsByChildTable(t *testing.T) {
}
}
func TestWriterAppendsBatchAcrossChildTablesWithSingleMultiTableInsert(t *testing.T) {
exec := &recordingExec{}
writer := NewWriter(exec)
first := sampleEnvelope()
second := sampleEnvelope()
second.Sequence = 2
second.EventID = "second-event"
second.VIN = "LNBVIN00000000002"
second.Phone = "013307795426"
second.EventTimeMS += 1000
second.ReceivedAtMS += 1000
if err := writer.AppendAllBatch(context.Background(), []envelope.FrameEnvelope{first, second}); err != nil {
t.Fatalf("AppendAllBatch() error = %v", err)
}
if got := countSQL(exec.calls, "USING raw_frames"); got != 2 {
t.Fatalf("raw child create count = %d", got)
}
if got := countSQL(exec.calls, "USING vehicle_locations"); got != 2 {
t.Fatalf("location child create count = %d", got)
}
if got := countSQL(exec.calls, "INSERT INTO raw_"); got != 1 {
t.Fatalf("raw multi-table insert count = %d", got)
}
if got := countSQL(exec.calls, "INSERT INTO loc_"); got != 1 {
t.Fatalf("location multi-table insert count = %d", got)
}
rawInsert := findSQL(exec.calls, "INSERT INTO raw_")
if got := strings.Count(rawInsert, "\nVALUES "); got != 2 {
t.Fatalf("raw multi-table VALUES sections = %d, sql=%s", got, rawInsert)
}
locationInsert := findSQL(exec.calls, "INSERT INTO loc_")
if got := strings.Count(locationInsert, "\nVALUES "); got != 2 {
t.Fatalf("location multi-table VALUES sections = %d, sql=%s", got, locationInsert)
}
}
func TestWriterAppendAllBatchSkipsLocationForNonRealtimeFrames(t *testing.T) {
exec := &recordingExec{}
writer := NewWriter(exec)
first := sampleEnvelope()
first.MessageID = "0x0100"
first.Parsed = map[string]any{"registration": map[string]any{"plate": "沪A12345"}}
second := sampleEnvelope()
second.Sequence = 2
second.EventTimeMS += 1000
second.ReceivedAtMS += 1000
if err := writer.AppendAllBatch(context.Background(), []envelope.FrameEnvelope{first, second}); err != nil {
t.Fatalf("AppendAllBatch() error = %v", err)
}
if got := countSQL(exec.calls, "INSERT INTO raw_"); got != 1 {
t.Fatalf("raw batch insert count = %d", got)
}
locationInsert := findSQL(exec.calls, "INSERT INTO loc_")
if got := strings.Count(locationInsert, "),(") + 1; got != 1 {
t.Fatalf("location batch row count = %d, sql=%s", got, locationInsert)
}
}
func TestWriterNormalizesFarFutureEventTimeForLocationButKeepsRawEvidence(t *testing.T) {
exec := &recordingExec{}
writer := NewWriter(exec)
env := sampleEnvelope()
received := time.Date(2026, 7, 12, 9, 30, 0, 0, time.UTC)
futureEvent := time.Date(2026, 7, 13, 9, 30, 0, 0, time.UTC)
env.ReceivedAtMS = received.UnixMilli()
env.EventTimeMS = futureEvent.UnixMilli()
if err := writer.AppendAll(context.Background(), env); err != nil {
t.Fatalf("AppendAll() error = %v", err)
}
rawInsert := findSQL(exec.calls, "INSERT INTO raw_")
locationInsert := findSQL(exec.calls, "INSERT INTO loc_")
if !strings.Contains(rawInsert, strconv.FormatInt(futureEvent.UnixMilli(), 10)) {
t.Fatalf("raw insert should keep original event time %d: %s", futureEvent.UnixMilli(), rawInsert)
}
if strings.Contains(locationInsert, strconv.FormatInt(futureEvent.UnixMilli(), 10)) {
t.Fatalf("location insert should not use far future event time: %s", locationInsert)
}
if !strings.Contains(locationInsert, strconv.FormatInt(received.UnixMilli(), 10)) {
t.Fatalf("location insert should use received time %d: %s", received.UnixMilli(), locationInsert)
}
}
func TestWriterWithDatabaseQualifiesTDengineTables(t *testing.T) {
exec := &recordingExec{}
writer := NewWriterWithDatabase(exec, "vehicle_ts")
@@ -372,14 +573,14 @@ func sampleEnvelope() envelope.FrameEnvelope {
ReceivedAtMS: 1782745114999,
RawHex: "7E0200",
Parsed: map[string]any{"message": "location"},
Fields: map[string]any{
envelope.FieldLongitude: 121.069881,
envelope.FieldLatitude: 30.590151,
envelope.FieldSpeedKMH: 23.0,
envelope.FieldTotalMileageKM: 10241.2,
"direction_deg": uint16(79),
"alarm_flag": uint32(0),
"status_flag": uint32(72),
ParsedFields: map[string]any{
"jt808.location.longitude": 121.069881,
"jt808.location.latitude": 30.590151,
"jt808.location.speed_kmh": 23.0,
"jt808.location.total_mileage_km": 10241.2,
"jt808.location.direction_deg": uint16(79),
"jt808.location.alarm_flag": uint32(0),
"jt808.location.status_flag": uint32(72),
},
ParseStatus: envelope.ParseOK,
}
@@ -404,6 +605,16 @@ func findSQL(calls []execCall, pattern string) string {
return ""
}
func matchingSQL(calls []execCall, pattern string) []string {
out := []string{}
for _, call := range calls {
if strings.Contains(call.query, pattern) {
out = append(out, call.query)
}
}
return out
}
func containsSQL(calls []execCall, pattern string) bool {
return findSQL(calls, pattern) != ""
}
@@ -428,10 +639,16 @@ type execCall struct {
type recordingExec struct {
calls []execCall
errs []error
}
func (e *recordingExec) ExecContext(_ context.Context, query string, args ...any) (sql.Result, error) {
e.calls = append(e.calls, execCall{query: query, args: args})
if len(e.errs) > 0 {
err := e.errs[0]
e.errs = e.errs[1:]
return nil, err
}
return nil, nil
}

773
go/vehicle-gateway/internal/identity/mapping_import.go Normal file
View File

@@ -0,0 +1,773 @@
package identity
import (
"context"
"database/sql"
"errors"
"fmt"
"io/fs"
"path/filepath"
"regexp"
"sort"
"strconv"
"strings"
"github.com/xuri/excelize/v2"
)
const (
IdentifierTypeJT808Phone = "JT808_PHONE"
IdentifierTypePlate = "PLATE"
)
type MappingRecord struct {
File string `json:"file"`
Sheet string `json:"sheet"`
Row int `json:"row"`
SourceCode string `json:"source_code"`
SourceName string `json:"source_name"`
Protocol string `json:"protocol"`
IdentifierType string `json:"identifier_type"`
IdentifierValue string `json:"identifier_value"`
RawValue string `json:"raw_value,omitempty"`
Plate string `json:"plate,omitempty"`
OEM string `json:"oem,omitempty"`
}
type MappingScanReport struct {
Files int `json:"files"`
Sheets int `json:"sheets"`
Rows int `json:"rows"`
Records int `json:"records"`
Skipped int `json:"skipped"`
UnsupportedFiles int `json:"unsupported_files,omitempty"`
Sources []MappingSourceScanReport `json:"sources,omitempty"`
UnsupportedItems []MappingUnsupportedFileReport `json:"unsupported_items,omitempty"`
Errors []string `json:"errors,omitempty"`
}
type MappingSourceScanReport struct {
SourceCode string `json:"source_code"`
SourceName string `json:"source_name"`
Files int `json:"files"`
Sheets int `json:"sheets"`
Rows int `json:"rows"`
Records int `json:"records"`
PhoneRecords int `json:"phone_records"`
PlateRecords int `json:"plate_records"`
Skipped int `json:"skipped"`
}
type MappingUnsupportedFileReport struct {
File string `json:"file"`
Ext string `json:"ext"`
Reason string `json:"reason"`
}
type MappingImportOptions struct {
Apply bool
LegacyTable string
ReportItemLimit int
}
type MappingImportReport struct {
Scan MappingScanReport `json:"scan"`
Records int `json:"records"`
Deduplicated int `json:"deduplicated"`
Resolved int `json:"resolved"`
Unresolved int `json:"unresolved"`
Conflicts int `json:"conflicts"`
WouldInsert int `json:"would_insert,omitempty"`
WouldUpdate int `json:"would_update,omitempty"`
Inserted int `json:"inserted,omitempty"`
Updated int `json:"updated,omitempty"`
SourceResults []MappingSourceImportStat `json:"source_results,omitempty"`
UnresolvedItems []MappingRecord `json:"unresolved_items,omitempty"`
ConflictItems []MappingConflict `json:"conflict_items,omitempty"`
}
type MappingSourceImportStat struct {
SourceCode string `json:"source_code"`
SourceName string `json:"source_name"`
Records int `json:"records"`
Deduplicated int `json:"deduplicated"`
Resolved int `json:"resolved"`
Unresolved int `json:"unresolved"`
Conflicts int `json:"conflicts"`
WouldInsert int `json:"would_insert,omitempty"`
WouldUpdate int `json:"would_update,omitempty"`
Inserted int `json:"inserted,omitempty"`
Updated int `json:"updated,omitempty"`
}
type MappingConflict struct {
Record MappingRecord `json:"record"`
ExistingVIN string `json:"existing_vin,omitempty"`
NewVIN string `json:"new_vin,omitempty"`
Reason string `json:"reason"`
}
type resolvedMappingRecord struct {
MappingRecord
VIN string
}
type mappingStore interface {
ExecContext(context.Context, string, ...any) (sql.Result, error)
QueryRowContext(context.Context, string, ...any) *sql.Row
}
var digitPattern = regexp.MustCompile(`\D+`)
func EnsureVehicleIdentifierSchema(ctx context.Context, db *sql.DB) error {
if db == nil {
return errors.New("identity db must not be nil")
}
if _, err := db.ExecContext(ctx, vehicleTableSQL); err != nil {
return err
}
_, err := db.ExecContext(ctx, vehicleIdentifierTableSQL)
return err
}
func ReadMappingDirectory(root string) ([]MappingRecord, MappingScanReport, error) {
root = strings.TrimSpace(root)
if root == "" {
return nil, MappingScanReport{}, errors.New("mapping input directory is empty")
}
var records []MappingRecord
report := MappingScanReport{}
err := filepath.WalkDir(root, func(path string, entry fs.DirEntry, err error) error {
if err != nil {
report.Errors = append(report.Errors, err.Error())
return nil
}
if entry == nil || entry.IsDir() {
return nil
}
name := entry.Name()
if strings.HasPrefix(name, "~$") || strings.HasPrefix(name, "._") {
return nil
}
ext := strings.ToLower(filepath.Ext(name))
if !isSupportedMappingWorkbookExt(ext) {
if isUnsupportedMappingWorkbookExt(ext) {
report.UnsupportedFiles++
report.UnsupportedItems = append(report.UnsupportedItems, MappingUnsupportedFileReport{
File: path,
Ext: ext,
Reason: "convert legacy workbook to .xlsx before import",
})
}
return nil
}
fileRecords, fileReport, err := readMappingWorkbook(root, path)
report.Files++
report.Sheets += fileReport.Sheets
report.Rows += fileReport.Rows
report.Records += fileReport.Records
report.Skipped += fileReport.Skipped
report.Errors = append(report.Errors, fileReport.Errors...)
for _, source := range fileReport.Sources {
mergeMappingSourceScan(&report, source)
}
if err != nil {
report.Errors = append(report.Errors, fmt.Sprintf("%s: %v", path, err))
return nil
}
records = append(records, fileRecords...)
return nil
})
if err != nil {
return records, report, err
}
sortMappingSourceScans(report.Sources)
return records, report, nil
}
func readMappingWorkbook(root string, path string) ([]MappingRecord, MappingScanReport, error) {
workbook, err := excelize.OpenFile(path)
if err != nil {
return nil, MappingScanReport{}, err
}
defer func() { _ = workbook.Close() }()
sourceCode, sourceName := mappingSource(root, path)
sourceReport := MappingSourceScanReport{
SourceCode: sourceCode,
SourceName: sourceName,
Files: 1,
}
var records []MappingRecord
report := MappingScanReport{}
for _, sheet := range workbook.GetSheetList() {
rows, err := workbook.GetRows(sheet)
if err != nil {
report.Errors = append(report.Errors, fmt.Sprintf("%s/%s: %v", path, sheet, err))
continue
}
report.Sheets++
sourceReport.Sheets++
report.Rows += len(rows)
sourceReport.Rows += len(rows)
if len(rows) == 0 {
continue
}
header, dataStart := mappingHeader(rows)
for rowIndex := dataStart; rowIndex < len(rows); rowIndex++ {
row := rows[rowIndex]
plate := normalizePlate(cellByHeader(row, header, "plate"))
rawPhone := cellByHeader(row, header, "phone")
phone := normalizeMappingPhone(rawPhone)
if len(header) == 0 {
plate = normalizePlate(cell(row, 0))
rawPhone = cell(row, 1)
phone = normalizeMappingPhone(rawPhone)
}
if plate == "" && phone == "" {
report.Skipped++
sourceReport.Skipped++
continue
}
if phone != "" {
records = append(records, MappingRecord{
File: path,
Sheet: sheet,
Row: rowIndex + 1,
SourceCode: sourceCode,
SourceName: sourceName,
Protocol: "JT808",
IdentifierType: IdentifierTypeJT808Phone,
IdentifierValue: phone,
RawValue: strings.TrimSpace(rawPhone),
Plate: plate,
OEM: sourceName,
})
sourceReport.PhoneRecords++
sourceReport.Records++
}
if plate != "" {
records = append(records, MappingRecord{
File: path,
Sheet: sheet,
Row: rowIndex + 1,
SourceCode: sourceCode,
SourceName: sourceName,
Protocol: "JT808",
IdentifierType: IdentifierTypePlate,
IdentifierValue: plate,
RawValue: plate,
Plate: plate,
OEM: sourceName,
})
sourceReport.PlateRecords++
sourceReport.Records++
}
}
}
report.Records = len(records)
report.Sources = []MappingSourceScanReport{sourceReport}
return records, report, nil
}
func mergeMappingSourceScan(report *MappingScanReport, source MappingSourceScanReport) {
if report == nil || strings.TrimSpace(source.SourceCode) == "" {
return
}
for index := range report.Sources {
if report.Sources[index].SourceCode != source.SourceCode {
continue
}
report.Sources[index].Files += source.Files
report.Sources[index].Sheets += source.Sheets
report.Sources[index].Rows += source.Rows
report.Sources[index].Records += source.Records
report.Sources[index].PhoneRecords += source.PhoneRecords
report.Sources[index].PlateRecords += source.PlateRecords
report.Sources[index].Skipped += source.Skipped
if report.Sources[index].SourceName == "" {
report.Sources[index].SourceName = source.SourceName
}
return
}
report.Sources = append(report.Sources, source)
}
func sortMappingSourceScans(sources []MappingSourceScanReport) {
sort.SliceStable(sources, func(i, j int) bool {
return sources[i].SourceCode < sources[j].SourceCode
})
}
func isSupportedMappingWorkbookExt(ext string) bool {
switch strings.ToLower(strings.TrimSpace(ext)) {
case ".xlsx", ".xlsm":
return true
default:
return false
}
}
func isUnsupportedMappingWorkbookExt(ext string) bool {
switch strings.ToLower(strings.TrimSpace(ext)) {
case ".xls", ".xlsb":
return true
default:
return false
}
}
func ImportMappingRecords(ctx context.Context, db *sql.DB, records []MappingRecord, scan MappingScanReport, opts MappingImportOptions) (MappingImportReport, error) {
if db == nil {
return MappingImportReport{}, errors.New("identity db must not be nil")
}
legacyTable := strings.TrimSpace(opts.LegacyTable)
if legacyTable == "" || !safeIdentifier(legacyTable) {
legacyTable = "vehicle_identity_binding"
}
report := MappingImportReport{
Scan: scan,
Records: len(records),
}
sourceStats := map[string]*MappingSourceImportStat{}
for _, record := range records {
sourceImportStat(sourceStats, record).Records++
}
reportLimit := opts.ReportItemLimit
if reportLimit == 0 {
reportLimit = 50
}
deduped, conflicts := dedupeMappingRecords(records)
report.Deduplicated = len(deduped)
report.Conflicts += len(conflicts)
for _, conflict := range conflicts {
sourceImportStat(sourceStats, conflict.Record).Conflicts++
appendConflictItem(&report, conflict, reportLimit)
}
store := mappingStore(db)
var tx *sql.Tx
if opts.Apply {
var err error
tx, err = db.BeginTx(ctx, nil)
if err != nil {
return report, err
}
store = tx
defer func() {
if tx != nil {
_ = tx.Rollback()
}
}()
}
for _, record := range deduped {
sourceStat := sourceImportStat(sourceStats, record)
sourceStat.Deduplicated++
vin, err := resolveMappingVIN(ctx, store, legacyTable, record)
if err != nil {
return report, err
}
if vin == "" {
report.Unresolved++
sourceStat.Unresolved++
appendUnresolvedItem(&report, record, reportLimit)
continue
}
resolved := resolvedMappingRecord{MappingRecord: record, VIN: vin}
existingVIN, exists, err := existingIdentifierVIN(ctx, store, resolved)
if err != nil {
return report, err
}
if exists && !strings.EqualFold(existingVIN, vin) {
report.Conflicts++
sourceStat.Conflicts++
appendConflictItem(&report, MappingConflict{
Record: record,
ExistingVIN: existingVIN,
NewVIN: vin,
Reason: "identifier already points to another vin",
}, reportLimit)
continue
}
report.Resolved++
sourceStat.Resolved++
if exists {
if opts.Apply {
if err := updateVehicleIdentifier(ctx, store, resolved); err != nil {
return report, err
}
report.Updated++
sourceStat.Updated++
} else {
report.WouldUpdate++
sourceStat.WouldUpdate++
}
continue
}
if opts.Apply {
if err := upsertVehicle(ctx, store, resolved); err != nil {
return report, err
}
if err := insertVehicleIdentifier(ctx, store, resolved); err != nil {
return report, err
}
report.Inserted++
sourceStat.Inserted++
} else {
report.WouldInsert++
sourceStat.WouldInsert++
}
}
if tx != nil {
if err := tx.Commit(); err != nil {
return report, err
}
tx = nil
}
report.SourceResults = sortedMappingSourceImportStats(sourceStats)
return report, nil
}
func sourceImportStat(stats map[string]*MappingSourceImportStat, record MappingRecord) *MappingSourceImportStat {
sourceCode := strings.TrimSpace(record.SourceCode)
if sourceCode == "" {
sourceCode = "unknown"
}
stat := stats[sourceCode]
if stat != nil {
if stat.SourceName == "" {
stat.SourceName = strings.TrimSpace(record.SourceName)
}
return stat
}
stat = &MappingSourceImportStat{
SourceCode: sourceCode,
SourceName: strings.TrimSpace(record.SourceName),
}
stats[sourceCode] = stat
return stat
}
func sortedMappingSourceImportStats(stats map[string]*MappingSourceImportStat) []MappingSourceImportStat {
if len(stats) == 0 {
return nil
}
keys := make([]string, 0, len(stats))
for key := range stats {
keys = append(keys, key)
}
sort.Strings(keys)
out := make([]MappingSourceImportStat, 0, len(keys))
for _, key := range keys {
out = append(out, *stats[key])
}
return out
}
func appendUnresolvedItem(report *MappingImportReport, record MappingRecord, limit int) {
if report == nil {
return
}
if limit >= 0 && len(report.UnresolvedItems) >= limit {
return
}
report.UnresolvedItems = append(report.UnresolvedItems, record)
}
func appendConflictItem(report *MappingImportReport, conflict MappingConflict, limit int) {
if report == nil {
return
}
if limit >= 0 && len(report.ConflictItems) >= limit {
return
}
report.ConflictItems = append(report.ConflictItems, conflict)
}
func dedupeMappingRecords(records []MappingRecord) ([]MappingRecord, []MappingConflict) {
seen := map[string]MappingRecord{}
indexByKey := map[string]int{}
var out []MappingRecord
var conflicts []MappingConflict
for _, record := range records {
record.IdentifierValue = normalizeIdentifierValue(record.IdentifierType, record.IdentifierValue)
record.Plate = normalizePlate(record.Plate)
if record.Protocol == "" {
record.Protocol = "JT808"
}
if record.IdentifierValue == "" || record.IdentifierType == "" {
continue
}
key := strings.Join([]string{record.Protocol, record.SourceCode, record.IdentifierType, record.IdentifierValue}, "\x00")
existing, ok := seen[key]
if !ok {
seen[key] = record
indexByKey[key] = len(out)
out = append(out, record)
continue
}
if existing.Plate != "" && record.Plate != "" && existing.Plate != record.Plate {
conflicts = append(conflicts, MappingConflict{
Record: record,
Reason: fmt.Sprintf("same source identifier maps to multiple plates: %s/%s", existing.Plate, record.Plate),
})
continue
}
merged := mergeMappingRecord(existing, record)
seen[key] = merged
if index, ok := indexByKey[key]; ok && index >= 0 && index < len(out) {
out[index] = merged
}
}
return out, conflicts
}
func mergeMappingRecord(existing MappingRecord, incoming MappingRecord) MappingRecord {
merged := existing
if merged.File == "" {
merged.File = incoming.File
}
if merged.Sheet == "" {
merged.Sheet = incoming.Sheet
}
if merged.Row == 0 {
merged.Row = incoming.Row
}
if merged.SourceName == "" {
merged.SourceName = incoming.SourceName
}
if merged.Protocol == "" {
merged.Protocol = incoming.Protocol
}
if merged.IdentifierType == "" {
merged.IdentifierType = incoming.IdentifierType
}
if merged.IdentifierValue == "" {
merged.IdentifierValue = incoming.IdentifierValue
}
if merged.RawValue == "" {
merged.RawValue = incoming.RawValue
}
if merged.Plate == "" {
merged.Plate = incoming.Plate
}
if merged.OEM == "" {
merged.OEM = incoming.OEM
}
return merged
}
func resolveMappingVIN(ctx context.Context, db mappingStore, legacyTable string, record MappingRecord) (string, error) {
if record.Plate != "" {
vin, err := lookupLegacyVIN(ctx, db, legacyTable, "plate", record.Plate)
if err != nil && !errors.Is(err, sql.ErrNoRows) {
return "", err
}
if vin != "" {
return vin, nil
}
}
if record.IdentifierType == IdentifierTypeJT808Phone && record.IdentifierValue != "" {
vin, err := lookupLegacyVIN(ctx, db, legacyTable, "phone", record.IdentifierValue)
if err != nil && !errors.Is(err, sql.ErrNoRows) {
return "", err
}
return vin, nil
}
return "", nil
}
func lookupLegacyVIN(ctx context.Context, db mappingStore, table string, column string, value string) (string, error) {
if !safeIdentifier(table) || !safeIdentifier(column) {
return "", sql.ErrNoRows
}
query := "SELECT vin FROM " + table + " WHERE " + column + " = ? AND vin IS NOT NULL AND vin <> '' LIMIT 1"
var vin string
err := db.QueryRowContext(ctx, query, value).Scan(&vin)
if err != nil {
return "", err
}
return strings.TrimSpace(vin), nil
}
func existingIdentifierVIN(ctx context.Context, db mappingStore, record resolvedMappingRecord) (string, bool, error) {
var vin string
err := db.QueryRowContext(ctx, `SELECT vin FROM vehicle_identifier
WHERE protocol = ? AND source_code = ? AND identifier_type = ? AND identifier_value = ?`,
record.Protocol, record.SourceCode, record.IdentifierType, record.IdentifierValue).Scan(&vin)
if errors.Is(err, sql.ErrNoRows) {
return "", false, nil
}
if err != nil {
return "", false, err
}
return strings.TrimSpace(vin), true, nil
}
func upsertVehicle(ctx context.Context, db mappingStore, record resolvedMappingRecord) error {
_, err := db.ExecContext(ctx, `INSERT INTO vehicle (vin, plate, oem, enabled)
VALUES (?, ?, ?, 1)
ON DUPLICATE KEY UPDATE
plate = IF(VALUES(plate) <> '', VALUES(plate), plate),
oem = IF(VALUES(oem) <> '', VALUES(oem), oem),
enabled = 1`,
record.VIN, record.Plate, record.OEM)
return err
}
func insertVehicleIdentifier(ctx context.Context, db mappingStore, record resolvedMappingRecord) error {
_, err := db.ExecContext(ctx, `INSERT INTO vehicle_identifier
(protocol, source_code, identifier_type, identifier_value, vin, plate, oem, raw_value, enabled, latest_import_file)
VALUES (?, ?, ?, ?, ?, ?, ?, ?, 1, ?)`,
record.Protocol,
record.SourceCode,
record.IdentifierType,
record.IdentifierValue,
record.VIN,
record.Plate,
record.OEM,
record.RawValue,
record.File,
)
return err
}
func updateVehicleIdentifier(ctx context.Context, db mappingStore, record resolvedMappingRecord) error {
_, err := db.ExecContext(ctx, `UPDATE vehicle_identifier
SET plate = IF(? <> '', ?, plate),
oem = IF(? <> '', ?, oem),
raw_value = IF(? <> '', ?, raw_value),
latest_import_file = ?,
enabled = 1
WHERE protocol = ? AND source_code = ? AND identifier_type = ? AND identifier_value = ?`,
record.Plate, record.Plate,
record.OEM, record.OEM,
record.RawValue, record.RawValue,
record.File,
record.Protocol,
record.SourceCode,
record.IdentifierType,
record.IdentifierValue,
)
return err
}
func mappingSource(root string, path string) (string, string) {
rel, err := filepath.Rel(root, path)
if err != nil {
rel = filepath.Base(path)
}
parts := strings.Split(filepath.ToSlash(rel), "/")
name := strings.TrimSpace(parts[0])
if name == "" || strings.EqualFold(name, ".") {
name = strings.TrimSuffix(filepath.Base(path), filepath.Ext(path))
}
code := sourceCode(name)
return code, name
}
func sourceCode(name string) string {
switch strings.TrimSpace(strings.ToLower(name)) {
case "g7s":
return "g7s"
case "信达":
return "xinda"
case "广安北斗", "广安车联":
return "guangan_beidou"
case "东方北斗":
return "dongfang_beidou"
case "赛格":
return "saige"
default:
return normalizeASCIIKey(name)
}
}
func mappingHeader(rows [][]string) (map[string]int, int) {
for index, row := range rows {
header := map[string]int{}
for columnIndex, value := range row {
key := normalizeHeader(value)
switch key {
case "车牌", "车牌号", "车牌号码":
header["plate"] = columnIndex
case "sim", "sim卡号", "手机号", "终端手机号", "终端id", "终端标识":
header["phone"] = columnIndex
}
}
if len(header) > 0 {
return header, index + 1
}
if index >= 5 {
break
}
}
return nil, 0
}
func cellByHeader(row []string, header map[string]int, key string) string {
if len(header) == 0 {
return ""
}
index, ok := header[key]
if !ok {
return ""
}
return cell(row, index)
}
func cell(row []string, index int) string {
if index < 0 || index >= len(row) {
return ""
}
return strings.TrimSpace(row[index])
}
func normalizeHeader(value string) string {
value = strings.TrimSpace(strings.ToLower(value))
value = strings.ReplaceAll(value, " ", "")
value = strings.ReplaceAll(value, "\t", "")
value = strings.ReplaceAll(value, "（", "(")
value = strings.ReplaceAll(value, "）", ")")
return value
}
func normalizePlate(value string) string {
value = strings.TrimSpace(value)
value = strings.ReplaceAll(value, " ", "")
value = strings.ReplaceAll(value, "\t", "")
return value
}
func normalizeMappingPhone(value string) string {
value = strings.TrimSpace(value)
if value == "" {
return ""
}
if strings.ContainsAny(value, ".eE") {
if parsed, err := strconv.ParseFloat(value, 64); err == nil && parsed > 0 {
return normalizePhone(strconv.FormatFloat(parsed, 'f', 0, 64))
}
}
digits := digitPattern.ReplaceAllString(value, "")
return normalizePhone(digits)
}
func normalizeASCIIKey(value string) string {
value = strings.TrimSpace(strings.ToLower(value))
var b strings.Builder
lastUnderscore := false
for _, r := range value {
if (r >= 'a' && r <= 'z') || (r >= '0' && r <= '9') {
b.WriteRune(r)
lastUnderscore = false
continue
}
if !lastUnderscore {
b.WriteByte('_')
lastUnderscore = true
}
}
return strings.Trim(b.String(), "_")
}

345
go/vehicle-gateway/internal/identity/mapping_import_test.go Normal file
View File

@@ -0,0 +1,345 @@
package identity
import (
"context"
"errors"
"os"
"path/filepath"
"testing"
"github.com/DATA-DOG/go-sqlmock"
"github.com/xuri/excelize/v2"
)
func TestReadMappingDirectoryExtractsPhoneAndPlate(t *testing.T) {
dir := t.TempDir()
writeWorkbook(t, filepath.Join(dir, "G7s", "宇速全量.xlsx"), [][]string{
{"车牌号", "sim卡号", "设备号"},
{"粤AG18312", "013307795425", "DEV001"},
})
writeWorkbook(t, filepath.Join(dir, "东方北斗", "无标题0703.xlsx"), [][]string{
{"沪A01559F", "64341233712"},
})
if err := os.MkdirAll(filepath.Join(dir, "信达"), 0o755); err != nil {
t.Fatalf("MkdirAll() error = %v", err)
}
if err := os.WriteFile(filepath.Join(dir, "信达", "旧格式.xls"), []byte("legacy xls"), 0o644); err != nil {
t.Fatalf("WriteFile() error = %v", err)
}
if err := os.WriteFile(filepath.Join(dir, "信达", "说明.txt"), []byte("ignored"), 0o644); err != nil {
t.Fatalf("WriteFile() error = %v", err)
}
records, report, err := ReadMappingDirectory(dir)
if err != nil {
t.Fatalf("ReadMappingDirectory() error = %v", err)
}
if report.Files != 2 {
t.Fatalf("files = %d, report=%#v", report.Files, report)
}
if report.UnsupportedFiles != 1 || len(report.UnsupportedItems) != 1 {
t.Fatalf("unsupported files = %d, items=%#v", report.UnsupportedFiles, report.UnsupportedItems)
}
if got := report.UnsupportedItems[0]; got.Ext != ".xls" || got.Reason == "" {
t.Fatalf("unsupported item = %#v", got)
}
sources := map[string]MappingSourceScanReport{}
for _, source := range report.Sources {
sources[source.SourceCode] = source
}
if got := sources["g7s"]; got.Files != 1 || got.Sheets != 1 || got.PhoneRecords != 1 || got.PlateRecords != 1 || got.Records != 2 {
t.Fatalf("g7s source report = %#v", got)
}
if got := sources["dongfang_beidou"]; got.Files != 1 || got.Sheets != 1 || got.PhoneRecords != 1 || got.PlateRecords != 1 || got.Records != 2 {
t.Fatalf("dongfang source report = %#v", got)
}
var phoneSeen bool
var plateSeen bool
var headerlessSeen bool
for _, record := range records {
if record.SourceCode == "g7s" && record.IdentifierType == IdentifierTypeJT808Phone && record.IdentifierValue == "13307795425" && record.Plate == "粤AG18312" {
phoneSeen = true
}
if record.SourceCode == "g7s" && record.IdentifierType == IdentifierTypePlate && record.IdentifierValue == "粤AG18312" {
plateSeen = true
}
if record.SourceCode == "dongfang_beidou" && record.IdentifierType == IdentifierTypeJT808Phone && record.IdentifierValue == "64341233712" && record.Plate == "沪A01559F" {
headerlessSeen = true
}
}
if !phoneSeen || !plateSeen || !headerlessSeen {
t.Fatalf("records missing expected mappings: %#v", records)
}
}
func TestImportMappingRecordsDryRunResolvesVINFromLegacyPlate(t *testing.T) {
db, mock := newMockDB(t)
defer db.Close()
mock.ExpectQuery("SELECT vin FROM vehicle_identity_binding WHERE plate = \\?").
WithArgs("粤AG18312").
WillReturnRows(sqlmock.NewRows([]string{"vin"}).AddRow("LB9A32A22P0LS1230"))
mock.ExpectQuery("SELECT vin FROM vehicle_identifier").
WithArgs("JT808", "g7s", IdentifierTypeJT808Phone, "13307795425").
WillReturnRows(sqlmock.NewRows([]string{"vin"}))
report, err := ImportMappingRecords(context.Background(), db, []MappingRecord{
{
SourceCode: "g7s",
SourceName: "G7s",
Protocol: "JT808",
IdentifierType: IdentifierTypeJT808Phone,
IdentifierValue: "013307795425",
Plate: "粤AG18312",
OEM: "G7s",
},
}, MappingScanReport{}, MappingImportOptions{LegacyTable: "vehicle_identity_binding"})
if err != nil {
t.Fatalf("ImportMappingRecords() error = %v", err)
}
if report.Resolved != 1 || report.WouldInsert != 1 || report.Unresolved != 0 {
t.Fatalf("report = %#v", report)
}
if len(report.SourceResults) != 1 {
t.Fatalf("source results = %#v, want one source", report.SourceResults)
}
if got := report.SourceResults[0]; got.SourceCode != "g7s" || got.Records != 1 || got.Deduplicated != 1 || got.Resolved != 1 || got.WouldInsert != 1 {
t.Fatalf("source result = %#v", got)
}
if err := mock.ExpectationsWereMet(); err != nil {
t.Fatalf("sql expectations: %v", err)
}
}
func TestImportMappingRecordsReportsSourceResults(t *testing.T) {
db, mock := newMockDB(t)
defer db.Close()
mock.ExpectQuery("SELECT vin FROM vehicle_identity_binding WHERE plate = \\?").
WithArgs("粤AG18312").
WillReturnRows(sqlmock.NewRows([]string{"vin"}).AddRow("LB9A32A22P0LS1230"))
mock.ExpectQuery("SELECT vin FROM vehicle_identifier").
WithArgs("JT808", "g7s", IdentifierTypeJT808Phone, "13307795425").
WillReturnRows(sqlmock.NewRows([]string{"vin"}))
mock.ExpectQuery("SELECT vin FROM vehicle_identity_binding WHERE plate = \\?").
WithArgs("粤B00000").
WillReturnRows(sqlmock.NewRows([]string{"vin"}))
mock.ExpectQuery("SELECT vin FROM vehicle_identity_binding WHERE phone = \\?").
WithArgs("14400000000").
WillReturnRows(sqlmock.NewRows([]string{"vin"}))
report, err := ImportMappingRecords(context.Background(), db, []MappingRecord{
{
SourceCode: "g7s",
SourceName: "G7s",
Protocol: "JT808",
IdentifierType: IdentifierTypeJT808Phone,
IdentifierValue: "13307795425",
Plate: "粤AG18312",
OEM: "G7s",
},
{
SourceCode: "g7s",
SourceName: "G7s",
Protocol: "JT808",
IdentifierType: IdentifierTypeJT808Phone,
IdentifierValue: "13307795425",
Plate: "粤B99999",
OEM: "G7s",
},
{
SourceCode: "xinda",
SourceName: "信达",
Protocol: "JT808",
IdentifierType: IdentifierTypeJT808Phone,
IdentifierValue: "14400000000",
Plate: "粤B00000",
OEM: "信达",
},
}, MappingScanReport{}, MappingImportOptions{LegacyTable: "vehicle_identity_binding"})
if err != nil {
t.Fatalf("ImportMappingRecords() error = %v", err)
}
if report.Records != 3 || report.Deduplicated != 2 || report.Resolved != 1 || report.Unresolved != 1 || report.Conflicts != 1 || report.WouldInsert != 1 {
t.Fatalf("report = %#v", report)
}
got := map[string]MappingSourceImportStat{}
for _, source := range report.SourceResults {
got[source.SourceCode] = source
}
if source := got["g7s"]; source.Records != 2 || source.Deduplicated != 1 || source.Resolved != 1 || source.Conflicts != 1 || source.WouldInsert != 1 {
t.Fatalf("g7s source result = %#v", source)
}
if source := got["xinda"]; source.Records != 1 || source.Deduplicated != 1 || source.Unresolved != 1 {
t.Fatalf("xinda source result = %#v", source)
}
if err := mock.ExpectationsWereMet(); err != nil {
t.Fatalf("sql expectations: %v", err)
}
}
func TestImportMappingRecordsMergesDuplicateIdentifierDetails(t *testing.T) {
db, mock := newMockDB(t)
defer db.Close()
mock.ExpectQuery("SELECT vin FROM vehicle_identity_binding WHERE plate = \\?").
WithArgs("粤AG18312").
WillReturnRows(sqlmock.NewRows([]string{"vin"}).AddRow("LB9A32A22P0LS1230"))
mock.ExpectQuery("SELECT vin FROM vehicle_identifier").
WithArgs("JT808", "g7s", IdentifierTypeJT808Phone, "13307795425").
WillReturnRows(sqlmock.NewRows([]string{"vin"}))
report, err := ImportMappingRecords(context.Background(), db, []MappingRecord{
{
File: "G7s/no-plate.xlsx",
SourceCode: "g7s",
SourceName: "G7s",
Protocol: "JT808",
IdentifierType: IdentifierTypeJT808Phone,
IdentifierValue: "13307795425",
OEM: "G7s",
},
{
File: "G7s/with-plate.xlsx",
SourceCode: "g7s",
SourceName: "G7s",
Protocol: "JT808",
IdentifierType: IdentifierTypeJT808Phone,
IdentifierValue: "13307795425",
Plate: "粤AG18312",
OEM: "G7s",
},
}, MappingScanReport{}, MappingImportOptions{LegacyTable: "vehicle_identity_binding"})
if err != nil {
t.Fatalf("ImportMappingRecords() error = %v", err)
}
if report.Records != 2 || report.Deduplicated != 1 || report.Resolved != 1 || report.WouldInsert != 1 || report.Unresolved != 0 {
t.Fatalf("report = %#v", report)
}
if err := mock.ExpectationsWereMet(); err != nil {
t.Fatalf("sql expectations: %v", err)
}
}
func TestImportMappingRecordsApplyCommitsSingleTransaction(t *testing.T) {
db, mock := newMockDB(t)
defer db.Close()
mock.ExpectBegin()
mock.ExpectQuery("SELECT vin FROM vehicle_identity_binding WHERE plate = \\?").
WithArgs("粤AG18312").
WillReturnRows(sqlmock.NewRows([]string{"vin"}).AddRow("LB9A32A22P0LS1230"))
mock.ExpectQuery("SELECT vin FROM vehicle_identifier").
WithArgs("JT808", "g7s", IdentifierTypeJT808Phone, "13307795425").
WillReturnRows(sqlmock.NewRows([]string{"vin"}))
mock.ExpectExec("INSERT INTO vehicle").
WithArgs("LB9A32A22P0LS1230", "粤AG18312", "G7s").
WillReturnResult(sqlmock.NewResult(0, 1))
mock.ExpectExec("INSERT INTO vehicle_identifier").
WithArgs("JT808", "g7s", IdentifierTypeJT808Phone, "13307795425", "LB9A32A22P0LS1230", "粤AG18312", "G7s", "13307795425", "G7s/example.xlsx").
WillReturnResult(sqlmock.NewResult(0, 1))
mock.ExpectCommit()
report, err := ImportMappingRecords(context.Background(), db, []MappingRecord{
{
File: "G7s/example.xlsx",
SourceCode: "g7s",
SourceName: "G7s",
Protocol: "JT808",
IdentifierType: IdentifierTypeJT808Phone,
IdentifierValue: "13307795425",
RawValue: "13307795425",
Plate: "粤AG18312",
OEM: "G7s",
},
}, MappingScanReport{}, MappingImportOptions{
Apply: true,
LegacyTable: "vehicle_identity_binding",
})
if err != nil {
t.Fatalf("ImportMappingRecords() error = %v", err)
}
if report.Inserted != 1 || report.Resolved != 1 || report.WouldInsert != 0 {
t.Fatalf("report = %#v", report)
}
if err := mock.ExpectationsWereMet(); err != nil {
t.Fatalf("sql expectations: %v", err)
}
}
func TestImportMappingRecordsApplyRollsBackOnWriteError(t *testing.T) {
db, mock := newMockDB(t)
defer db.Close()
errWrite := errors.New("insert vehicle failed")
mock.ExpectBegin()
mock.ExpectQuery("SELECT vin FROM vehicle_identity_binding WHERE plate = \\?").
WithArgs("粤AG18312").
WillReturnRows(sqlmock.NewRows([]string{"vin"}).AddRow("LB9A32A22P0LS1230"))
mock.ExpectQuery("SELECT vin FROM vehicle_identifier").
WithArgs("JT808", "g7s", IdentifierTypeJT808Phone, "13307795425").
WillReturnRows(sqlmock.NewRows([]string{"vin"}))
mock.ExpectExec("INSERT INTO vehicle").
WithArgs("LB9A32A22P0LS1230", "粤AG18312", "G7s").
WillReturnError(errWrite)
mock.ExpectRollback()
report, err := ImportMappingRecords(context.Background(), db, []MappingRecord{
{
SourceCode: "g7s",
SourceName: "G7s",
Protocol: "JT808",
IdentifierType: IdentifierTypeJT808Phone,
IdentifierValue: "13307795425",
RawValue: "13307795425",
Plate: "粤AG18312",
OEM: "G7s",
},
}, MappingScanReport{}, MappingImportOptions{
Apply: true,
LegacyTable: "vehicle_identity_binding",
})
if !errors.Is(err, errWrite) {
t.Fatalf("ImportMappingRecords() error = %v, want %v", err, errWrite)
}
if report.Inserted != 0 || report.Resolved != 1 {
t.Fatalf("report = %#v", report)
}
if err := mock.ExpectationsWereMet(); err != nil {
t.Fatalf("sql expectations: %v", err)
}
}
func TestNormalizeMappingPhoneHandlesExcelNumericFormats(t *testing.T) {
tests := map[string]string{
"013307795425": "13307795425",
"13307795425.0": "13307795425",
"1.3307795425E10": "13307795425",
}
for input, want := range tests {
if got := normalizeMappingPhone(input); got != want {
t.Fatalf("normalizeMappingPhone(%q) = %q, want %q", input, got, want)
}
}
}
func writeWorkbook(t *testing.T, path string, rows [][]string) {
t.Helper()
if err := os.MkdirAll(filepath.Dir(path), 0o755); err != nil {
t.Fatalf("MkdirAll() error = %v", err)
}
workbook := excelize.NewFile()
sheet := "Sheet1"
for rowIndex, row := range rows {
for columnIndex, value := range row {
cellName, err := excelize.CoordinatesToCellName(columnIndex+1, rowIndex+1)
if err != nil {
t.Fatalf("CoordinatesToCellName() error = %v", err)
}
if err := workbook.SetCellValue(sheet, cellName, value); err != nil {
t.Fatalf("SetCellValue() error = %v", err)
}
}
}
if err := workbook.SaveAs(path); err != nil {
t.Fatalf("SaveAs() error = %v", err)
}
if err := workbook.Close(); err != nil {
t.Fatalf("Close() error = %v", err)
}
}

357
go/vehicle-gateway/internal/identity/registration_writer.go Normal file
View File

@@ -0,0 +1,357 @@
package identity
import (
"context"
"database/sql"
"fmt"
"strings"
"sync"
"time"
"lingniu-vehicle-ingest/go/vehicle-gateway/internal/envelope"
)
const (
JT808RegisterMessageID = "0x0100"
JT808AuthMessageID = "0x0102"
JT808LocationMessageID = "0x0200"
)
// JT808RegistrationFact is the durable identity projection carried by a raw
// JT808 envelope. SeenAt uses gateway receive time so replay cannot move a
// terminal to a future date because its device clock was wrong.
type JT808RegistrationFact struct {
Phone string
DeviceID string
Plate string
VIN string
Province string
City string
Manufacturer string
DeviceType string
PlateColor string
AuthToken string
AuthIMEI string
AuthSoftwareVersion string
SourceEndpoint string
SourceIP string
FirstRegisteredAt *time.Time
LatestRegisteredAt *time.Time
LatestAuthenticated *time.Time
SeenAt time.Time
}
// JT808RegistrationProjector throttles ordinary location touches in memory.
// Registration and authentication frames are never throttled.
type JT808RegistrationProjector struct {
location *time.Location
touchInterval time.Duration
retention time.Duration
mu sync.Mutex
lastTouches map[string]time.Time
nextCleanup time.Time
}
func NewJT808RegistrationProjector(location *time.Location, touchInterval time.Duration) *JT808RegistrationProjector {
if location == nil {
location = time.Local
}
if touchInterval <= 0 {
touchInterval = 10 * time.Minute
}
return &JT808RegistrationProjector{
location: location,
touchInterval: touchInterval,
retention: 24 * time.Hour,
lastTouches: map[string]time.Time{},
}
}
// ProjectBatch returns at most one merged fact per phone. Call MarkPersisted
// only after the database transaction succeeds; otherwise replay must remain
// eligible immediately.
func (p *JT808RegistrationProjector) ProjectBatch(envelopes []envelope.FrameEnvelope) []JT808RegistrationFact {
if p == nil || len(envelopes) == 0 {
return nil
}
p.mu.Lock()
defer p.mu.Unlock()
byPhone := make(map[string]JT808RegistrationFact)
order := make([]string, 0, len(envelopes))
for _, env := range envelopes {
fact, ok := p.projectLocked(env)
if !ok {
continue
}
if current, exists := byPhone[fact.Phone]; exists {
byPhone[fact.Phone] = mergeJT808RegistrationFact(current, fact)
continue
}
byPhone[fact.Phone] = fact
order = append(order, fact.Phone)
}
result := make([]JT808RegistrationFact, 0, len(order))
for _, phone := range order {
result = append(result, byPhone[phone])
}
return result
}
func (p *JT808RegistrationProjector) MarkPersisted(facts []JT808RegistrationFact) {
if p == nil || len(facts) == 0 {
return
}
p.mu.Lock()
defer p.mu.Unlock()
for _, fact := range facts {
phone := normalizePhone(fact.Phone)
if phone == "" || fact.SeenAt.IsZero() {
continue
}
if current := p.lastTouches[phone]; fact.SeenAt.After(current) {
p.lastTouches[phone] = fact.SeenAt
}
}
p.cleanupLocked(time.Now())
}
func (p *JT808RegistrationProjector) projectLocked(env envelope.FrameEnvelope) (JT808RegistrationFact, bool) {
if env.Protocol != envelope.ProtocolJT808 || env.ParseStatus == envelope.ParseBadFrame {
return JT808RegistrationFact{}, false
}
messageID := strings.TrimSpace(env.MessageID)
if messageID != JT808RegisterMessageID && messageID != JT808AuthMessageID && messageID != JT808LocationMessageID {
return JT808RegistrationFact{}, false
}
phone := normalizePhone(env.Phone)
if phone == "" {
return JT808RegistrationFact{}, false
}
seenAt := p.receivedAt(env)
if seenAt.IsZero() {
return JT808RegistrationFact{}, false
}
if messageID == JT808LocationMessageID {
if last := p.lastTouches[phone]; !last.IsZero() && seenAt.Before(last.Add(p.touchInterval)) {
return JT808RegistrationFact{}, false
}
}
registration := mapValue(env.Parsed, "registration")
authentication := mapValue(env.Parsed, "authentication")
authenticationAccepted := messageID != JT808AuthMessageID ||
!env.AuthenticationEnforced || env.AuthenticationStatus == "accepted"
vin := strings.TrimSpace(env.VIN)
if vin == "" {
vin = "unknown"
}
fact := JT808RegistrationFact{
Phone: phone,
DeviceID: firstNonEmpty(env.DeviceID, textValue(registration, "device_id"), parsedFieldText(env.ParsedFields, "jt808.registration.device_id")),
Plate: firstNonEmpty(env.Plate, textValue(registration, "plate"), parsedFieldText(env.ParsedFields, "jt808.registration.plate")),
VIN: vin,
Province: firstNonEmpty(textValue(registration, "province"), parsedFieldText(env.ParsedFields, "jt808.registration.province")),
City: firstNonEmpty(textValue(registration, "city"), parsedFieldText(env.ParsedFields, "jt808.registration.city")),
Manufacturer: firstNonEmpty(textValue(registration, "manufacturer"), parsedFieldText(env.ParsedFields, "jt808.registration.manufacturer")),
DeviceType: firstNonEmpty(textValue(registration, "device_type"), parsedFieldText(env.ParsedFields, "jt808.registration.device_type")),
PlateColor: firstNonEmpty(textValue(registration, "plate_color"), parsedFieldText(env.ParsedFields, "jt808.registration.plate_color")),
AuthToken: firstNonEmpty(textValue(authentication, "token"), parsedFieldText(env.ParsedFields, "jt808.authentication.token")),
AuthIMEI: firstNonEmpty(textValue(authentication, "imei"), parsedFieldText(env.ParsedFields, "jt808.authentication.imei")),
AuthSoftwareVersion: firstNonEmpty(textValue(authentication, "software_version"), parsedFieldText(env.ParsedFields, "jt808.authentication.software_version")),
SourceEndpoint: strings.TrimSpace(env.SourceEndpoint),
SourceIP: normalizeEndpointIP(env.SourceEndpoint),
SeenAt: seenAt,
}
if !authenticationAccepted {
fact.AuthToken = ""
fact.AuthIMEI = ""
fact.AuthSoftwareVersion = ""
}
if messageID == JT808RegisterMessageID {
fact.FirstRegisteredAt = timePointer(seenAt)
fact.LatestRegisteredAt = timePointer(seenAt)
}
if messageID == JT808AuthMessageID && authenticationAccepted {
fact.LatestAuthenticated = timePointer(seenAt)
}
return fact, true
}
func parsedFieldText(fields map[string]any, key string) string {
value, ok := fields[key]
if !ok || value == nil {
return ""
}
text := strings.TrimSpace(fmt.Sprint(value))
if text == "<nil>" {
return ""
}
return text
}
func (p *JT808RegistrationProjector) receivedAt(env envelope.FrameEnvelope) time.Time {
milliseconds := env.ReceivedAtMS
if milliseconds <= 0 {
milliseconds = env.EventTimeMS
}
if milliseconds <= 0 {
return time.Time{}
}
return time.UnixMilli(milliseconds).In(p.location).Truncate(time.Second)
}
func (p *JT808RegistrationProjector) cleanupLocked(now time.Time) {
if !p.nextCleanup.IsZero() && now.Before(p.nextCleanup) {
return
}
p.nextCleanup = now.Add(time.Hour)
cutoff := now.Add(-p.retention)
for phone, touchedAt := range p.lastTouches {
if touchedAt.Before(cutoff) {
delete(p.lastTouches, phone)
}
}
}
func mergeJT808RegistrationFact(current JT808RegistrationFact, candidate JT808RegistrationFact) JT808RegistrationFact {
if candidate.SeenAt.After(current.SeenAt) || candidate.SeenAt.Equal(current.SeenAt) {
current.DeviceID = firstNonEmpty(candidate.DeviceID, current.DeviceID)
current.Plate = firstNonEmpty(candidate.Plate, current.Plate)
current.VIN = preferKnownVIN(candidate.VIN, current.VIN)
current.Province = firstNonEmpty(candidate.Province, current.Province)
current.City = firstNonEmpty(candidate.City, current.City)
current.Manufacturer = firstNonEmpty(candidate.Manufacturer, current.Manufacturer)
current.DeviceType = firstNonEmpty(candidate.DeviceType, current.DeviceType)
current.PlateColor = firstNonEmpty(candidate.PlateColor, current.PlateColor)
current.AuthToken = firstNonEmpty(candidate.AuthToken, current.AuthToken)
current.AuthIMEI = firstNonEmpty(candidate.AuthIMEI, current.AuthIMEI)
current.AuthSoftwareVersion = firstNonEmpty(candidate.AuthSoftwareVersion, current.AuthSoftwareVersion)
current.SourceEndpoint = firstNonEmpty(candidate.SourceEndpoint, current.SourceEndpoint)
current.SourceIP = firstNonEmpty(candidate.SourceIP, current.SourceIP)
current.SeenAt = candidate.SeenAt
} else {
current.DeviceID = firstNonEmpty(current.DeviceID, candidate.DeviceID)
current.Plate = firstNonEmpty(current.Plate, candidate.Plate)
current.VIN = preferKnownVIN(current.VIN, candidate.VIN)
current.Province = firstNonEmpty(current.Province, candidate.Province)
current.City = firstNonEmpty(current.City, candidate.City)
current.Manufacturer = firstNonEmpty(current.Manufacturer, candidate.Manufacturer)
current.DeviceType = firstNonEmpty(current.DeviceType, candidate.DeviceType)
current.PlateColor = firstNonEmpty(current.PlateColor, candidate.PlateColor)
current.AuthToken = firstNonEmpty(current.AuthToken, candidate.AuthToken)
current.AuthIMEI = firstNonEmpty(current.AuthIMEI, candidate.AuthIMEI)
current.AuthSoftwareVersion = firstNonEmpty(current.AuthSoftwareVersion, candidate.AuthSoftwareVersion)
}
current.FirstRegisteredAt = earlierTimePointer(current.FirstRegisteredAt, candidate.FirstRegisteredAt)
current.LatestRegisteredAt = laterTimePointer(current.LatestRegisteredAt, candidate.LatestRegisteredAt)
current.LatestAuthenticated = laterTimePointer(current.LatestAuthenticated, candidate.LatestAuthenticated)
return current
}
func timePointer(value time.Time) *time.Time {
copy := value
return &copy
}
func earlierTimePointer(left *time.Time, right *time.Time) *time.Time {
if left == nil {
return right
}
if right == nil || left.Before(*right) || left.Equal(*right) {
return left
}
return right
}
func laterTimePointer(left *time.Time, right *time.Time) *time.Time {
if left == nil {
return right
}
if right == nil || left.After(*right) || left.Equal(*right) {
return left
}
return right
}
type JT808RegistrationStore struct {
db *sql.DB
}
func NewJT808RegistrationStore(db *sql.DB) *JT808RegistrationStore {
if db == nil {
panic("jt808 registration db must not be nil")
}
return &JT808RegistrationStore{db: db}
}
func EnsureJT808RegistrationSchema(ctx context.Context, db *sql.DB) error {
if db == nil {
return fmt.Errorf("jt808 registration db is nil")
}
if _, err := db.ExecContext(ctx, jt808RegistrationTableSQL); err != nil {
return err
}
for _, statement := range jt808RegistrationAlterSQL {
if _, err := db.ExecContext(ctx, statement); err != nil && !isIgnoredJT808RegistrationAlterError(err) {
return err
}
}
_, err := db.ExecContext(ctx, jt808RegistrationSourceIPBackfillSQL)
return err
}
func (s *JT808RegistrationStore) UpsertBatch(ctx context.Context, facts []JT808RegistrationFact) error {
if s == nil || s.db == nil || len(facts) == 0 {
return nil
}
const columns = `phone, device_id, plate, vin, province, city, manufacturer, device_type, plate_color,
auth_token, auth_imei, auth_software_version, source_endpoint, source_ip,
first_registered_at, latest_registered_at, latest_authenticated_at, latest_seen_at`
values := make([]string, 0, len(facts))
args := make([]any, 0, len(facts)*18)
for _, fact := range facts {
phone := normalizePhone(fact.Phone)
if phone == "" || fact.SeenAt.IsZero() {
continue
}
vin := strings.TrimSpace(fact.VIN)
if vin == "" {
vin = "unknown"
}
values = append(values, "(?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)")
args = append(args,
phone, strings.TrimSpace(fact.DeviceID), strings.TrimSpace(fact.Plate), vin,
strings.TrimSpace(fact.Province), strings.TrimSpace(fact.City), strings.TrimSpace(fact.Manufacturer),
strings.TrimSpace(fact.DeviceType), strings.TrimSpace(fact.PlateColor), strings.TrimSpace(fact.AuthToken),
strings.TrimSpace(fact.AuthIMEI), strings.TrimSpace(fact.AuthSoftwareVersion),
strings.TrimSpace(fact.SourceEndpoint), strings.TrimSpace(fact.SourceIP),
fact.FirstRegisteredAt, fact.LatestRegisteredAt, fact.LatestAuthenticated, fact.SeenAt,
)
}
if len(values) == 0 {
return nil
}
query := `INSERT INTO jt808_registration (` + columns + `) VALUES ` + strings.Join(values, ",") + `
ON DUPLICATE KEY UPDATE
device_id = IF(VALUES(device_id) <> '' AND (device_id IS NULL OR device_id = '' OR VALUES(latest_seen_at) >= latest_seen_at), VALUES(device_id), device_id),
plate = IF(VALUES(plate) <> '' AND (plate IS NULL OR plate = '' OR VALUES(latest_seen_at) >= latest_seen_at), VALUES(plate), plate),
vin = IF(VALUES(vin) <> '' AND VALUES(vin) <> 'unknown' AND (vin IS NULL OR vin = '' OR vin = 'unknown' OR VALUES(latest_seen_at) >= latest_seen_at), VALUES(vin), vin),
province = IF(VALUES(province) <> '' AND (province IS NULL OR province = '' OR VALUES(latest_seen_at) >= latest_seen_at), VALUES(province), province),
city = IF(VALUES(city) <> '' AND (city IS NULL OR city = '' OR VALUES(latest_seen_at) >= latest_seen_at), VALUES(city), city),
manufacturer = IF(VALUES(manufacturer) <> '' AND (manufacturer IS NULL OR manufacturer = '' OR VALUES(latest_seen_at) >= latest_seen_at), VALUES(manufacturer), manufacturer),
device_type = IF(VALUES(device_type) <> '' AND (device_type IS NULL OR device_type = '' OR VALUES(latest_seen_at) >= latest_seen_at), VALUES(device_type), device_type),
plate_color = IF(VALUES(plate_color) <> '' AND (plate_color IS NULL OR plate_color = '' OR VALUES(latest_seen_at) >= latest_seen_at), VALUES(plate_color), plate_color),
auth_token = IF(VALUES(auth_token) <> '' AND (auth_token IS NULL OR auth_token = '' OR latest_authenticated_at IS NULL OR VALUES(latest_authenticated_at) >= latest_authenticated_at), VALUES(auth_token), auth_token),
auth_imei = IF(VALUES(auth_imei) <> '' AND (auth_imei IS NULL OR auth_imei = '' OR latest_authenticated_at IS NULL OR VALUES(latest_authenticated_at) >= latest_authenticated_at), VALUES(auth_imei), auth_imei),
auth_software_version = IF(VALUES(auth_software_version) <> '' AND (auth_software_version IS NULL OR auth_software_version = '' OR latest_authenticated_at IS NULL OR VALUES(latest_authenticated_at) >= latest_authenticated_at), VALUES(auth_software_version), auth_software_version),
source_endpoint = IF(VALUES(source_endpoint) <> '' AND (source_endpoint IS NULL OR source_endpoint = '' OR VALUES(latest_seen_at) >= latest_seen_at), VALUES(source_endpoint), source_endpoint),
source_ip = IF(VALUES(source_ip) <> '' AND (source_ip IS NULL OR source_ip = '' OR VALUES(latest_seen_at) >= latest_seen_at), VALUES(source_ip), source_ip),
first_registered_at = CASE WHEN VALUES(first_registered_at) IS NULL THEN first_registered_at WHEN first_registered_at IS NULL THEN VALUES(first_registered_at) ELSE LEAST(first_registered_at, VALUES(first_registered_at)) END,
latest_registered_at = CASE WHEN VALUES(latest_registered_at) IS NULL THEN latest_registered_at WHEN latest_registered_at IS NULL THEN VALUES(latest_registered_at) ELSE GREATEST(latest_registered_at, VALUES(latest_registered_at)) END,
latest_authenticated_at = CASE WHEN VALUES(latest_authenticated_at) IS NULL THEN latest_authenticated_at WHEN latest_authenticated_at IS NULL THEN VALUES(latest_authenticated_at) ELSE GREATEST(latest_authenticated_at, VALUES(latest_authenticated_at)) END,
latest_seen_at = GREATEST(latest_seen_at, VALUES(latest_seen_at))`
_, err := s.db.ExecContext(ctx, query, args...)
return err
}

221
go/vehicle-gateway/internal/identity/registration_writer_test.go Normal file
View File

@@ -0,0 +1,221 @@
package identity
import (
"context"
"regexp"
"testing"
"time"
"github.com/DATA-DOG/go-sqlmock"
"lingniu-vehicle-ingest/go/vehicle-gateway/internal/envelope"
)
func TestJT808RegistrationProjectorProjectsRegistrationUsingReceiveTime(t *testing.T) {
loc := time.FixedZone("Asia/Shanghai", 8*60*60)
projector := NewJT808RegistrationProjector(loc, 10*time.Minute)
receivedAt := time.Date(2026, 7, 13, 17, 20, 30, 987000000, loc)
env := envelope.FrameEnvelope{
Protocol: envelope.ProtocolJT808,
MessageID: JT808RegisterMessageID,
Phone: "0013307795425",
VIN: "LTESTVIN000000001",
DeviceID: "DEV-1",
Plate: "粤A00001",
SourceEndpoint: "115.231.168.135:43625",
ReceivedAtMS: receivedAt.UnixMilli(),
EventTimeMS: receivedAt.Add(24 * time.Hour).UnixMilli(),
ParseStatus: envelope.ParseOK,
ParsedFields: map[string]any{
"jt808.registration.province": "44",
"jt808.registration.city": "1",
"jt808.registration.manufacturer": "YUTNG",
"jt808.registration.device_type": "TBOX-1",
"jt808.registration.plate_color": "2",
},
}
facts := projector.ProjectBatch([]envelope.FrameEnvelope{env})
if len(facts) != 1 {
t.Fatalf("facts = %d, want 1", len(facts))
}
fact := facts[0]
if fact.Phone != "13307795425" || fact.VIN != env.VIN || fact.Plate != env.Plate {
t.Fatalf("identity fact = %+v", fact)
}
wantTime := receivedAt.Truncate(time.Second)
if !fact.SeenAt.Equal(wantTime) || fact.FirstRegisteredAt == nil || !fact.FirstRegisteredAt.Equal(wantTime) {
t.Fatalf("fact times = seen %s first %#v, want %s", fact.SeenAt, fact.FirstRegisteredAt, wantTime)
}
if fact.SourceIP != "115.231.168.135" || fact.Manufacturer != "YUTNG" || fact.PlateColor != "2" {
t.Fatalf("registration details = %+v", fact)
}
}
func TestJT808RegistrationProjectorThrottlesLocationOnlyAfterPersist(t *testing.T) {
loc := time.UTC
projector := NewJT808RegistrationProjector(loc, 10*time.Minute)
base := time.Date(2026, 7, 13, 8, 0, 0, 0, loc)
location := func(at time.Time) envelope.FrameEnvelope {
return envelope.FrameEnvelope{
Protocol: envelope.ProtocolJT808,
MessageID: JT808LocationMessageID,
Phone: "013307795425",
VIN: "LTESTVIN000000001",
ReceivedAtMS: at.UnixMilli(),
ParseStatus: envelope.ParseOK,
}
}
first := projector.ProjectBatch([]envelope.FrameEnvelope{location(base)})
if len(first) != 1 {
t.Fatalf("first facts = %d, want 1", len(first))
}
// A failed database attempt must remain immediately replayable.
if replay := projector.ProjectBatch([]envelope.FrameEnvelope{location(base)}); len(replay) != 1 {
t.Fatalf("uncommitted replay facts = %d, want 1", len(replay))
}
projector.MarkPersisted(first)
if throttled := projector.ProjectBatch([]envelope.FrameEnvelope{location(base.Add(9 * time.Minute))}); len(throttled) != 0 {
t.Fatalf("throttled facts = %d, want 0", len(throttled))
}
if due := projector.ProjectBatch([]envelope.FrameEnvelope{location(base.Add(10 * time.Minute))}); len(due) != 1 {
t.Fatalf("due facts = %d, want 1", len(due))
}
}
func TestJT808RegistrationProjectorMergesRegisterAndAuthForPhone(t *testing.T) {
projector := NewJT808RegistrationProjector(time.UTC, 10*time.Minute)
base := time.Date(2026, 7, 13, 8, 0, 0, 0, time.UTC)
register := envelope.FrameEnvelope{
Protocol: envelope.ProtocolJT808,
MessageID: JT808RegisterMessageID,
Phone: "013307795425",
VIN: "LTESTVIN000000001",
Plate: "粤A00001",
ReceivedAtMS: base.UnixMilli(),
ParseStatus: envelope.ParseOK,
}
auth := envelope.FrameEnvelope{
Protocol: envelope.ProtocolJT808,
MessageID: JT808AuthMessageID,
Phone: "13307795425",
ReceivedAtMS: base.Add(time.Second).UnixMilli(),
ParseStatus: envelope.ParseOK,
ParsedFields: map[string]any{
"jt808.authentication.token": "g7gps",
"jt808.authentication.imei": "123456789012345",
},
}
facts := projector.ProjectBatch([]envelope.FrameEnvelope{register, auth})
if len(facts) != 1 {
t.Fatalf("facts = %d, want 1", len(facts))
}
fact := facts[0]
if fact.VIN != register.VIN || fact.Plate != register.Plate || fact.AuthToken != "g7gps" {
t.Fatalf("merged fact = %+v", fact)
}
if fact.FirstRegisteredAt == nil || fact.LatestRegisteredAt == nil || fact.LatestAuthenticated == nil {
t.Fatalf("merged timestamps missing: %+v", fact)
}
}
func TestJT808RegistrationProjectorDoesNotTrustEnforcedRejectedAuth(t *testing.T) {
projector := NewJT808RegistrationProjector(time.UTC, 10*time.Minute)
seenAt := time.Date(2026, 7, 13, 8, 0, 0, 0, time.UTC)
facts := projector.ProjectBatch([]envelope.FrameEnvelope{{
Protocol: envelope.ProtocolJT808,
MessageID: JT808AuthMessageID,
Phone: "13307795425",
ReceivedAtMS: seenAt.UnixMilli(),
ParseStatus: envelope.ParseOK,
AuthenticationEnforced: true,
AuthenticationStatus: "rejected",
Parsed: map[string]any{
"authentication": map[string]any{
"token": "untrusted-token",
"imei": "untrusted-imei",
"software_version": "untrusted-version",
},
},
}})
if len(facts) != 1 {
t.Fatalf("facts = %d, want 1 audit touch", len(facts))
}
fact := facts[0]
if fact.AuthToken != "" || fact.AuthIMEI != "" || fact.AuthSoftwareVersion != "" || fact.LatestAuthenticated != nil {
t.Fatalf("rejected credential leaked into registration fact: %+v", fact)
}
}
func TestJT808RegistrationStoreUsesIdempotentEventTimeUpsert(t *testing.T) {
db, mock, err := sqlmock.New()
if err != nil {
t.Fatal(err)
}
defer db.Close()
store := NewJT808RegistrationStore(db)
seenAt := time.Date(2026, 7, 13, 8, 0, 0, 0, time.UTC)
mock.ExpectExec(regexp.QuoteMeta("INSERT INTO jt808_registration")).
WithArgs(
"13307795425", "DEV-1", "粤A00001", "LTESTVIN000000001", "", "", "YUTNG", "", "",
"g7gps", "", "", "115.231.168.135:43625", "115.231.168.135",
sqlmock.AnyArg(), sqlmock.AnyArg(), sqlmock.AnyArg(), seenAt,
).
WillReturnResult(sqlmock.NewResult(1, 1))
err = store.UpsertBatch(context.Background(), []JT808RegistrationFact{{
Phone: "013307795425",
DeviceID: "DEV-1",
Plate: "粤A00001",
VIN: "LTESTVIN000000001",
Manufacturer: "YUTNG",
AuthToken: "g7gps",
SourceEndpoint: "115.231.168.135:43625",
SourceIP: "115.231.168.135",
FirstRegisteredAt: timePointer(seenAt),
LatestRegisteredAt: timePointer(seenAt),
LatestAuthenticated: timePointer(seenAt),
SeenAt: seenAt,
}})
if err != nil {
t.Fatalf("UpsertBatch() error = %v", err)
}
if err := mock.ExpectationsWereMet(); err != nil {
t.Fatal(err)
}
}
func TestRegistrationUpsertProtectsLatestValuesFromOldReplay(t *testing.T) {
var query string
db, mock, err := sqlmock.New(sqlmock.QueryMatcherOption(sqlmock.QueryMatcherFunc(func(_ string, actual string) error {
query = actual
return nil
})))
if err != nil {
t.Fatal(err)
}
defer db.Close()
store := NewJT808RegistrationStore(db)
mock.ExpectExec("INSERT INTO jt808_registration").WillReturnResult(sqlmock.NewResult(1, 1))
if err := store.UpsertBatch(context.Background(), []JT808RegistrationFact{{
Phone: "13307795425",
VIN: "unknown",
SeenAt: time.Date(2026, 7, 13, 8, 0, 0, 0, time.UTC),
}}); err != nil {
t.Fatal(err)
}
if err := mock.ExpectationsWereMet(); err != nil {
t.Fatal(err)
}
for _, required := range []string{
"VALUES(latest_seen_at) >= latest_seen_at",
"LEAST(first_registered_at, VALUES(first_registered_at))",
"GREATEST(latest_authenticated_at, VALUES(latest_authenticated_at))",
"latest_seen_at = GREATEST(latest_seen_at, VALUES(latest_seen_at))",
} {
if !regexp.MustCompile(regexp.QuoteMeta(required)).MatchString(query) {
t.Fatalf("upsert SQL missing %q", required)
}
}
}

1344
go/vehicle-gateway/internal/identity/resolver.go
View File

File diff suppressed because it is too large Load Diff

759
go/vehicle-gateway/internal/identity/resolver_test.go
View File

@@ -5,6 +5,7 @@ import (
"database/sql"
"errors"
"strings"
"sync"
"testing"
"time"
@@ -57,9 +58,7 @@ func TestCandidateKeysNormalizesPhone(t *testing.T) {
func TestMySQLResolverFillsVINFromPhone(t *testing.T) {
db, mock := newMockDB(t)
defer db.Close()
mock.ExpectQuery("SELECT vin FROM vehicle_identity_binding WHERE phone = \\?").
WithArgs("13307795425").
WillReturnRows(sqlmock.NewRows([]string{"vin"}).AddRow("LNBVIN00000000001"))
expectVehicleIdentifierHit(mock, "JT808_PHONE", "13307795425", "LNBVIN00000000001")
resolver := NewMySQLResolver(db, "vehicle_identity_binding")
env, err := resolver.Resolve(context.Background(), envelope.FrameEnvelope{
@@ -74,7 +73,7 @@ func TestMySQLResolverFillsVINFromPhone(t *testing.T) {
t.Fatalf("vin = %q", env.VIN)
}
identity, ok := env.Parsed["identity"].(map[string]any)
if !ok || identity["source"] != "phone" {
if !ok || identity["source"] != "vehicle_identifier.JT808_PHONE" {
t.Fatalf("identity metadata = %#v", env.Parsed["identity"])
}
if err := mock.ExpectationsWereMet(); err != nil {
@@ -82,9 +81,254 @@ func TestMySQLResolverFillsVINFromPhone(t *testing.T) {
}
}
func TestMySQLResolverUsesDataSourceCodeForJT808IdentifierLookup(t *testing.T) {
db, mock := newMockDB(t)
defer db.Close()
expectSourceCodeHit(mock, "115.231.168.135", "xinda")
expectVehicleIdentifierHitForSource(mock, "xinda", "JT808_PHONE", "13307795425", "LNBVIN00000000001")
resolver := NewMySQLResolverWithOptions(db, "vehicle_identity_binding", MySQLResolverOptions{
SourceCodeLookup: true,
LookupCacheTTL: time.Hour,
})
env, err := resolver.Resolve(context.Background(), envelope.FrameEnvelope{
Protocol: envelope.ProtocolJT808,
Phone: "013307795425",
SourceEndpoint: "115.231.168.135:43625",
Parsed: map[string]any{},
})
if err != nil {
t.Fatalf("Resolve() error = %v", err)
}
if env.VIN != "LNBVIN00000000001" {
t.Fatalf("vin = %q", env.VIN)
}
if env.SourceCode != "xinda" || env.PlatformName != "G7s" || env.SourceKind != "PLATFORM" {
t.Fatalf("source metadata = code:%q platform:%q kind:%q", env.SourceCode, env.PlatformName, env.SourceKind)
}
identity, ok := env.Parsed["identity"].(map[string]any)
if !ok || identity["source"] != "vehicle_identifier.JT808_PHONE.xinda" {
t.Fatalf("identity metadata = %#v", env.Parsed["identity"])
}
if identity["source_code"] != "xinda" || identity["platform_name"] != "G7s" || identity["source_kind"] != "PLATFORM" {
t.Fatalf("identity source metadata = %#v", identity)
}
if err := mock.ExpectationsWereMet(); err != nil {
t.Fatalf("sql expectations: %v", err)
}
}
func TestMySQLResolverVehicleIdentifierSourceOverridesDataSourceCode(t *testing.T) {
db, mock := newMockDB(t)
defer db.Close()
mock.ExpectQuery("SELECT source_code").
WithArgs("JT808", "115.159.85.149").
WillReturnRows(sqlmock.NewRows([]string{"source_code", "platform_name", "source_kind"}).AddRow("dongfang_beidou", "G7易流", "PLATFORM"))
expectVehicleIdentifierMissForSource(mock, "dongfang_beidou", "JT808_PHONE", "64341232682")
expectVehicleIdentifierHitWithSource(mock, "JT808_PHONE", "64341232682", "LB9A32A23R0LS1045", "g7s", "G7s")
resolver := NewMySQLResolverWithOptions(db, "vehicle_identity_binding", MySQLResolverOptions{
SourceCodeLookup: true,
LookupCacheTTL: time.Hour,
})
env, err := resolver.Resolve(context.Background(), envelope.FrameEnvelope{
Protocol: envelope.ProtocolJT808,
Phone: "064341232682",
SourceEndpoint: "115.159.85.149:42823",
Parsed: map[string]any{},
})
if err != nil {
t.Fatalf("Resolve() error = %v", err)
}
if env.VIN != "LB9A32A23R0LS1045" {
t.Fatalf("vin = %q", env.VIN)
}
if env.SourceCode != "g7s" || env.PlatformName != "G7s" || env.SourceKind != "PLATFORM" {
t.Fatalf("source metadata = code:%q platform:%q kind:%q", env.SourceCode, env.PlatformName, env.SourceKind)
}
identity, ok := env.Parsed["identity"].(map[string]any)
if !ok || identity["source"] != "vehicle_identifier.JT808_PHONE.g7s" {
t.Fatalf("identity metadata = %#v", env.Parsed["identity"])
}
if identity["source_code"] != "g7s" || identity["platform_name"] != "G7s" || identity["source_kind"] != "PLATFORM" {
t.Fatalf("identity source metadata = %#v", identity)
}
if err := mock.ExpectationsWereMet(); err != nil {
t.Fatalf("sql expectations: %v", err)
}
}
func TestMySQLResolverUsesStaleIdentifierCacheWhenLookupFails(t *testing.T) {
db, mock := newMockDB(t)
defer db.Close()
expectVehicleIdentifierHit(mock, "JT808_PHONE", "13307795425", "LNBVIN00000000001")
resolver := NewMySQLResolverWithOptions(db, "vehicle_identity_binding", MySQLResolverOptions{
LookupCacheTTL: time.Nanosecond,
StaleLookupTTL: time.Hour,
})
first, err := resolver.Resolve(context.Background(), envelope.FrameEnvelope{
Protocol: envelope.ProtocolJT808,
Phone: "013307795425",
Parsed: map[string]any{},
})
if err != nil {
t.Fatalf("first Resolve() error = %v", err)
}
if first.VIN != "LNBVIN00000000001" {
t.Fatalf("first vin = %q", first.VIN)
}
time.Sleep(time.Millisecond)
mock.ExpectQuery("SELECT DISTINCT vin").
WithArgs("JT808", "JT808_PHONE", "13307795425").
WillReturnError(errors.New("mysql temporarily unavailable"))
second, err := resolver.Resolve(context.Background(), envelope.FrameEnvelope{
Protocol: envelope.ProtocolJT808,
Phone: "013307795425",
Parsed: map[string]any{},
})
if err != nil {
t.Fatalf("second Resolve() error = %v", err)
}
if second.VIN != "LNBVIN00000000001" {
t.Fatalf("second vin = %q, want stale cached vin", second.VIN)
}
identity, ok := second.Parsed["identity"].(map[string]any)
if !ok || identity["source"] != "vehicle_identifier.JT808_PHONE" {
t.Fatalf("identity metadata = %#v", second.Parsed["identity"])
}
if identity["cache_status"] != "stale" {
t.Fatalf("identity cache_status = %#v, want stale", identity["cache_status"])
}
if err := mock.ExpectationsWereMet(); err != nil {
t.Fatalf("sql expectations: %v", err)
}
}
func TestNormalizeEndpointIPUsesSharedSourceEndpointKey(t *testing.T) {
tests := map[string]string{
"115.231.168.135:43625": "115.231.168.135",
" 115.159.85.149:28316 ": "115.159.85.149",
"mqtt://yutong/ytforward/shln/3": "mqtt",
"MQTT://YUTONG/topic": "mqtt",
}
for input, want := range tests {
if got := normalizeEndpointIP(input); got != want {
t.Fatalf("normalizeEndpointIP(%q) = %q, want %q", input, got, want)
}
}
}
func TestMySQLResolverKeepsDirectSourceKindWithoutSourceCode(t *testing.T) {
db, mock := newMockDB(t)
defer db.Close()
mock.ExpectQuery("SELECT source_code").
WithArgs("JT808", "39.144.3.22").
WillReturnRows(sqlmock.NewRows([]string{"source_code", "platform_name", "source_kind"}).AddRow("", "", "DIRECT"))
expectVehicleIdentifierHit(mock, "JT808_PHONE", "13307765812", "LA9GG64L7PBAF4001")
resolver := NewMySQLResolverWithOptions(db, "vehicle_identity_binding", MySQLResolverOptions{
SourceCodeLookup: true,
LookupCacheTTL: time.Hour,
})
env, err := resolver.Resolve(context.Background(), envelope.FrameEnvelope{
Protocol: envelope.ProtocolJT808,
Phone: "013307765812",
SourceEndpoint: "39.144.3.22:60177",
Parsed: map[string]any{},
})
if err != nil {
t.Fatalf("Resolve() error = %v", err)
}
if env.VIN != "LA9GG64L7PBAF4001" {
t.Fatalf("vin = %q", env.VIN)
}
if env.SourceCode != "" || env.SourceKind != "DIRECT" {
t.Fatalf("source metadata = code:%q kind:%q", env.SourceCode, env.SourceKind)
}
identity, ok := env.Parsed["identity"].(map[string]any)
if !ok || identity["source_kind"] != "DIRECT" {
t.Fatalf("identity source metadata = %#v", env.Parsed["identity"])
}
if err := mock.ExpectationsWereMet(); err != nil {
t.Fatalf("sql expectations: %v", err)
}
}
func TestMySQLResolverFallsBackToGlobalIdentifierWhenSourceCodeMisses(t *testing.T) {
db, mock := newMockDB(t)
defer db.Close()
expectSourceCodeHit(mock, "115.231.168.135", "xinda")
expectVehicleIdentifierMissForSource(mock, "xinda", "JT808_PHONE", "13307795425")
expectVehicleIdentifierHit(mock, "JT808_PHONE", "13307795425", "LNBVIN00000000002")
resolver := NewMySQLResolverWithOptions(db, "vehicle_identity_binding", MySQLResolverOptions{
SourceCodeLookup: true,
LookupCacheTTL: time.Hour,
})
env, err := resolver.Resolve(context.Background(), envelope.FrameEnvelope{
Protocol: envelope.ProtocolJT808,
Phone: "013307795425",
SourceEndpoint: "115.231.168.135:43625",
Parsed: map[string]any{},
})
if err != nil {
t.Fatalf("Resolve() error = %v", err)
}
if env.VIN != "LNBVIN00000000002" {
t.Fatalf("vin = %q", env.VIN)
}
identity, ok := env.Parsed["identity"].(map[string]any)
if !ok || identity["source"] != "vehicle_identifier.JT808_PHONE" {
t.Fatalf("identity metadata = %#v", env.Parsed["identity"])
}
if err := mock.ExpectationsWereMet(); err != nil {
t.Fatalf("sql expectations: %v", err)
}
}
func TestMySQLResolverUsesSingleGlobalIdentifierSourceCodeForJT808SourceMetadata(t *testing.T) {
db, mock := newMockDB(t)
defer db.Close()
expectSourceCodeMiss(mock, "117.132.196.119")
expectVehicleIdentifierHitWithSource(mock, "JT808_PHONE", "41456413943", "LNXNEGRR0SR321372", "xinda", "信达")
resolver := NewMySQLResolverWithOptions(db, "vehicle_identity_binding", MySQLResolverOptions{
SourceCodeLookup: true,
LookupCacheTTL: time.Hour,
})
env, err := resolver.Resolve(context.Background(), envelope.FrameEnvelope{
Protocol: envelope.ProtocolJT808,
Phone: "41456413943",
SourceEndpoint: "117.132.196.119:3275",
Parsed: map[string]any{},
})
if err != nil {
t.Fatalf("Resolve() error = %v", err)
}
if env.VIN != "LNXNEGRR0SR321372" {
t.Fatalf("vin = %q", env.VIN)
}
if env.SourceCode != "xinda" || env.PlatformName != "信达" || env.SourceKind != "PLATFORM" {
t.Fatalf("source metadata = code:%q platform:%q kind:%q", env.SourceCode, env.PlatformName, env.SourceKind)
}
identity, ok := env.Parsed["identity"].(map[string]any)
if !ok || identity["source"] != "vehicle_identifier.JT808_PHONE.xinda" {
t.Fatalf("identity metadata = %#v", env.Parsed["identity"])
}
if identity["source_code"] != "xinda" || identity["platform_name"] != "信达" || identity["source_kind"] != "PLATFORM" {
t.Fatalf("identity source metadata = %#v", identity)
}
if err := mock.ExpectationsWereMet(); err != nil {
t.Fatalf("sql expectations: %v", err)
}
}
func TestMySQLResolverLooksUpVINByUniqueKeyWithoutSort(t *testing.T) {
db, mock := newMockDB(t)
defer db.Close()
expectVehicleIdentifierMiss(mock, "JT808_PHONE", "13307795425")
mock.ExpectQuery("SELECT vin FROM vehicle_identity_binding WHERE phone = \\? AND vin IS NOT NULL AND vin <> ''$").
WithArgs("13307795425").
WillReturnRows(sqlmock.NewRows([]string{"vin"}).AddRow("LNBVIN00000000001"))
@@ -109,6 +353,7 @@ func TestMySQLResolverLooksUpVINByUniqueKeyWithoutSort(t *testing.T) {
func TestMySQLResolverDoesNotLookupBindingByDeviceID(t *testing.T) {
db, mock := newMockDB(t)
defer db.Close()
expectVehicleIdentifierMiss(mock, "JT808_PHONE", "13307795425")
mock.ExpectQuery("SELECT vin FROM vehicle_identity_binding WHERE phone = \\?").
WithArgs("13307795425").
WillReturnError(sql.ErrNoRows)
@@ -133,9 +378,11 @@ func TestMySQLResolverDoesNotLookupBindingByDeviceID(t *testing.T) {
func TestMySQLResolverTracksJT808RegistrationWithoutWritingBinding(t *testing.T) {
db, mock := newMockDB(t)
defer db.Close()
expectVehicleIdentifierMiss(mock, "JT808_PHONE", "13079963379")
mock.ExpectQuery("SELECT vin FROM vehicle_identity_binding WHERE phone = \\?").
WithArgs("13079963379").
WillReturnError(sql.ErrNoRows)
expectVehicleIdentifierMiss(mock, "PLATE", "TEST123")
mock.ExpectQuery("SELECT vin FROM vehicle_identity_binding WHERE plate = \\?").
WithArgs("TEST123").
WillReturnRows(sqlmock.NewRows([]string{"vin"}).AddRow("LKLG7C4E3NA774736"))
@@ -176,6 +423,7 @@ func TestMySQLResolverTracksJT808RegistrationWithoutWritingBinding(t *testing.T)
func TestMySQLResolverTracksFirstJT808LocationThenThrottles(t *testing.T) {
db, mock := newMockDB(t)
defer db.Close()
expectVehicleIdentifierMiss(mock, "JT808_PHONE", "13307795425")
mock.ExpectQuery("SELECT vin FROM vehicle_identity_binding WHERE phone = \\?").
WithArgs("13307795425").
WillReturnError(sql.ErrNoRows)
@@ -206,12 +454,14 @@ func TestMySQLResolverTracksFirstJT808LocationThenThrottles(t *testing.T) {
func TestMySQLResolverUsesJT808RegistrationPlateForLocationVIN(t *testing.T) {
db, mock := newMockDB(t)
defer db.Close()
expectVehicleIdentifierMiss(mock, "JT808_PHONE", "40692934322")
mock.ExpectQuery("SELECT vin FROM vehicle_identity_binding WHERE phone = \\?").
WithArgs("40692934322").
WillReturnError(sql.ErrNoRows)
mock.ExpectQuery("SELECT vin, device_id, plate FROM jt808_registration WHERE phone = \\?").
WithArgs("40692934322").
WillReturnRows(sqlmock.NewRows([]string{"vin", "device_id", "plate"}).AddRow("unknown", "18285", "粤AG18285"))
expectVehicleIdentifierMiss(mock, "PLATE", "粤AG18285")
mock.ExpectQuery("SELECT vin FROM vehicle_identity_binding WHERE plate = \\?").
WithArgs("粤AG18285").
WillReturnRows(sqlmock.NewRows([]string{"vin"}).AddRow("LNXNEGRR7SR318212"))
@@ -248,9 +498,352 @@ func TestMySQLResolverUsesJT808RegistrationPlateForLocationVIN(t *testing.T) {
}
}
func TestMySQLResolverRefreshesRegistrationCacheAfterRegisterFrame(t *testing.T) {
db, mock := newMockDB(t)
defer db.Close()
expectVehicleIdentifierMiss(mock, "JT808_PHONE", "40692934322")
mock.ExpectQuery("SELECT vin FROM vehicle_identity_binding WHERE phone = \\?").
WithArgs("40692934322").
WillReturnError(sql.ErrNoRows)
mock.ExpectQuery("SELECT vin, device_id, plate FROM jt808_registration WHERE phone = \\?").
WithArgs("40692934322").
WillReturnError(sql.ErrNoRows)
mock.ExpectExec("INSERT INTO jt808_registration").
WillReturnResult(sqlmock.NewResult(1, 1))
expectVehicleIdentifierMiss(mock, "PLATE", "粤AG18285")
mock.ExpectQuery("SELECT vin FROM vehicle_identity_binding WHERE plate = \\?").
WithArgs("粤AG18285").
WillReturnRows(sqlmock.NewRows([]string{"vin"}).AddRow("LNXNEGRR7SR318212"))
mock.ExpectExec("INSERT INTO jt808_registration").
WillReturnResult(sqlmock.NewResult(1, 1))
resolver := NewMySQLResolverWithOptions(db, "vehicle_identity_binding", MySQLResolverOptions{
LocationTouchInterval: time.Hour,
LookupCacheTTL: time.Hour,
})
firstLocation, err := resolver.Resolve(context.Background(), envelope.FrameEnvelope{
Protocol: envelope.ProtocolJT808,
MessageID: "0x0200",
Phone: "040692934322",
SourceEndpoint: "115.231.168.135:47822",
Parsed: map[string]any{},
})
if err != nil {
t.Fatalf("first location Resolve() error = %v", err)
}
if firstLocation.VIN != "" {
t.Fatalf("first location vin = %q, want unresolved", firstLocation.VIN)
}
registered, err := resolver.Resolve(context.Background(), envelope.FrameEnvelope{
Protocol: envelope.ProtocolJT808,
MessageID: "0x0100",
Phone: "040692934322",
DeviceID: "18285",
Plate: "粤AG18285",
SourceEndpoint: "115.231.168.135:47822",
Parsed: map[string]any{
"registration": map[string]any{
"device_id": "18285",
"plate": "粤AG18285",
},
},
})
if err != nil {
t.Fatalf("registration Resolve() error = %v", err)
}
if registered.VIN != "LNXNEGRR7SR318212" {
t.Fatalf("registered vin = %q", registered.VIN)
}
secondLocation, err := resolver.Resolve(context.Background(), envelope.FrameEnvelope{
Protocol: envelope.ProtocolJT808,
MessageID: "0x0200",
Phone: "040692934322",
SourceEndpoint: "115.231.168.135:47822",
Parsed: map[string]any{},
})
if err != nil {
t.Fatalf("second location Resolve() error = %v", err)
}
if secondLocation.VIN != "LNXNEGRR7SR318212" {
t.Fatalf("second location vin = %q, want cache-refreshed registration vin", secondLocation.VIN)
}
if secondLocation.DeviceID != "18285" || secondLocation.Plate != "粤AG18285" {
t.Fatalf("second location identity not copied: device=%q plate=%q", secondLocation.DeviceID, secondLocation.Plate)
}
if err := mock.ExpectationsWereMet(); err != nil {
t.Fatalf("sql expectations: %v", err)
}
}
func TestMySQLResolverDelegatesRegistrationPersistenceButKeepsLocalSession(t *testing.T) {
db, mock, err := sqlmock.New()
if err != nil {
t.Fatal(err)
}
defer db.Close()
var results []RegistrationWriteResult
resolver := NewMySQLResolverWithOptions(db, "vehicle_identity_binding", MySQLResolverOptions{
DisableRegistrationWrites: true,
OnRegistrationWriteResult: func(result RegistrationWriteResult) {
results = append(results, result)
},
})
env := envelope.FrameEnvelope{
Protocol: envelope.ProtocolJT808,
MessageID: JT808RegisterMessageID,
Phone: "013307795425",
VIN: "LTESTVIN000000001",
DeviceID: "DEV-1",
Plate: "粤A00001",
SourceEndpoint: "115.231.168.135:43625",
ParseStatus: envelope.ParseOK,
}
resolved, err := resolver.Resolve(context.Background(), env)
if err != nil {
t.Fatalf("Resolve() error = %v", err)
}
if resolved.VIN != env.VIN {
t.Fatalf("resolved vin = %q, want %q", resolved.VIN, env.VIN)
}
entry, ok := resolver.registrationCache["13307795425"]
if !ok || entry.vin != env.VIN || entry.plate != env.Plate || entry.deviceID != env.DeviceID {
t.Fatalf("local session = %+v exists=%v", entry, ok)
}
if len(results) != 1 || results[0].Mode != "delegated" || results[0].Status != "ok" {
t.Fatalf("registration write results = %#v", results)
}
if err := mock.ExpectationsWereMet(); err != nil {
t.Fatalf("unexpected mysql access: %v", err)
}
}
func TestMySQLResolverRetriesRegistrationUpsertTransientFailure(t *testing.T) {
db, mock := newMockDB(t)
defer db.Close()
mock.ExpectExec("INSERT INTO jt808_registration").
WillReturnError(errors.New("driver: bad connection: read tcp: connection reset by peer"))
mock.ExpectExec("INSERT INTO jt808_registration").
WillReturnResult(sqlmock.NewResult(1, 1))
resolver := NewMySQLResolverWithOptions(db, "vehicle_identity_binding", MySQLResolverOptions{
LocationTouchInterval: time.Hour,
RegistrationWriteAttempts: 2,
RegistrationWriteRetryDelay: -1,
LookupCacheTTL: time.Hour,
})
env := envelope.FrameEnvelope{
Protocol: envelope.ProtocolJT808,
MessageID: "0x0200",
Phone: "040692934322",
VIN: "LNXNEGRR7SR318212",
SourceEndpoint: "115.231.168.135:47822",
Parsed: map[string]any{},
}
if _, err := resolver.Resolve(context.Background(), env); err != nil {
t.Fatalf("Resolve() error = %v, want internal retry to recover", err)
}
if err := mock.ExpectationsWereMet(); err != nil {
t.Fatalf("sql expectations: %v", err)
}
}
func TestMySQLResolverAsyncRegistrationWritesDrainOnClose(t *testing.T) {
db, mock := newMockDB(t)
defer db.Close()
mock.ExpectExec("INSERT INTO jt808_registration").
WillReturnResult(sqlmock.NewResult(1, 1))
results := make(chan RegistrationWriteResult, 2)
resolver := NewMySQLResolverWithOptions(db, "vehicle_identity_binding", MySQLResolverOptions{
AsyncRegistrationWrites: true,
RegistrationWriteQueueSize: 8,
RegistrationWriteWorkers: 1,
RegistrationWriteTimeout: time.Second,
OnRegistrationWriteResult: func(result RegistrationWriteResult) {
results <- result
},
LocationTouchInterval: time.Hour,
LookupCacheTTL: time.Hour,
})
env := envelope.FrameEnvelope{
Protocol: envelope.ProtocolJT808,
MessageID: "0x0200",
Phone: "040692934322",
VIN: "LNXNEGRR7SR318212",
SourceEndpoint: "115.231.168.135:47822",
Parsed: map[string]any{},
}
if _, err := resolver.Resolve(context.Background(), env); err != nil {
t.Fatalf("Resolve() error = %v", err)
}
if err := resolver.Close(); err != nil {
t.Fatalf("Close() error = %v", err)
}
close(results)
got := map[string]int{}
for result := range results {
got[result.Mode+":"+result.Status]++
}
if got["async_enqueue:ok"] != 1 || got["async_background:ok"] != 1 {
t.Fatalf("registration write results = %#v, want enqueue/background ok", got)
}
if err := mock.ExpectationsWereMet(); err != nil {
t.Fatalf("sql expectations: %v", err)
}
}
func TestMySQLResolverAsyncRegistrationWriteFailureMarksLocationRetry(t *testing.T) {
db, mock := newMockDB(t)
defer db.Close()
mock.ExpectExec("INSERT INTO jt808_registration").
WillReturnError(errors.New("driver: bad connection"))
errs := make(chan error, 1)
results := make(chan RegistrationWriteResult, 2)
resolver := NewMySQLResolverWithOptions(db, "vehicle_identity_binding", MySQLResolverOptions{
AsyncRegistrationWrites: true,
RegistrationWriteQueueSize: 8,
RegistrationWriteWorkers: 1,
RegistrationWriteTimeout: time.Second,
OnRegistrationWriteResult: func(result RegistrationWriteResult) {
results <- result
},
OnRegistrationWriteError: func(err error) {
errs <- err
},
LocationTouchInterval: time.Hour,
LocationTouchRetryInterval: time.Hour,
RegistrationWriteAttempts: 1,
LookupCacheTTL: time.Hour,
})
env := envelope.FrameEnvelope{
Protocol: envelope.ProtocolJT808,
MessageID: "0x0200",
Phone: "040692934322",
VIN: "LNXNEGRR7SR318212",
SourceEndpoint: "115.231.168.135:47822",
Parsed: map[string]any{},
}
if _, err := resolver.Resolve(context.Background(), env); err != nil {
t.Fatalf("Resolve() error = %v, async write failure should be reported out-of-band", err)
}
if err := resolver.Close(); err != nil {
t.Fatalf("Close() error = %v", err)
}
select {
case err := <-errs:
if err == nil {
t.Fatal("async error callback received nil")
}
default:
t.Fatal("async write failure was not reported")
}
stats := resolver.CacheStats()
if stats.LocationTouchFailureEntries != 1 {
t.Fatalf("location touch failure cache entries = %d, want 1", stats.LocationTouchFailureEntries)
}
close(results)
got := map[string]int{}
for result := range results {
got[result.Mode+":"+result.Status]++
}
if got["async_enqueue:ok"] != 1 || got["async_background:error"] != 1 {
t.Fatalf("registration write results = %#v, want enqueue ok/background error", got)
}
if err := mock.ExpectationsWereMet(); err != nil {
t.Fatalf("sql expectations: %v", err)
}
}
func TestMySQLResolverBacksOffLocationTouchAfterExhaustedUpsert(t *testing.T) {
db, mock := newMockDB(t)
defer db.Close()
mock.ExpectExec("INSERT INTO jt808_registration").
WillReturnError(errors.New("driver: bad connection"))
resolver := NewMySQLResolverWithOptions(db, "vehicle_identity_binding", MySQLResolverOptions{
LocationTouchInterval: time.Hour,
LocationTouchRetryInterval: time.Hour,
RegistrationWriteAttempts: 1,
LookupCacheTTL: time.Hour,
})
env := envelope.FrameEnvelope{
Protocol: envelope.ProtocolJT808,
MessageID: "0x0200",
Phone: "040692934322",
VIN: "LNXNEGRR7SR318212",
SourceEndpoint: "115.231.168.135:47822",
Parsed: map[string]any{},
}
if _, err := resolver.Resolve(context.Background(), env); err == nil {
t.Fatal("first Resolve() error = nil, want exhausted transient registration upsert failure")
}
stats := resolver.CacheStats()
if stats.LocationTouchFailureEntries != 1 {
t.Fatalf("location touch failure cache entries = %d, want 1", stats.LocationTouchFailureEntries)
}
if _, err := resolver.Resolve(context.Background(), env); err != nil {
t.Fatalf("second Resolve() error = %v, want short backoff to skip immediate retry", err)
}
stats = resolver.CacheStats()
if stats.LocationTouchFailureEntries != 1 {
t.Fatalf("location touch failure cache entries after backoff = %d, want 1", stats.LocationTouchFailureEntries)
}
if err := mock.ExpectationsWereMet(); err != nil {
t.Fatalf("sql expectations: %v", err)
}
}
func TestIsTransientMySQLIdentityWriteError(t *testing.T) {
for _, err := range []error{
errors.New("dial tcp 127.0.0.1:3306: connection refused"),
errors.New("read tcp: connection reset by peer"),
errors.New("write tcp: broken pipe"),
errors.New("driver: bad connection"),
errors.New("invalid connection"),
errors.New("i/o timeout"),
errors.New("EOF"),
errors.New("server is down"),
errors.New("network is unreachable"),
} {
if !isTransientMySQLIdentityWriteError(err) {
t.Fatalf("isTransientMySQLIdentityWriteError(%q) = false, want true", err.Error())
}
}
for _, err := range []error{
context.Canceled,
context.DeadlineExceeded,
errors.New("duplicate key conflict"),
nil,
} {
if isTransientMySQLIdentityWriteError(err) {
t.Fatalf("isTransientMySQLIdentityWriteError(%v) = true, want false", err)
}
}
}
func TestTimeoutResolverAppliesDeadlineToDelegate(t *testing.T) {
delegate := &deadlineCheckingResolver{}
resolver := TimeoutResolver{Delegate: delegate, Timeout: 50 * time.Millisecond}
if _, err := resolver.Resolve(context.Background(), envelope.FrameEnvelope{Protocol: envelope.ProtocolJT808}); err != nil {
t.Fatalf("Resolve() error = %v", err)
}
if delegate.deadline.IsZero() {
t.Fatal("delegate did not receive a deadline")
}
if remaining := time.Until(delegate.deadline); remaining <= 0 || remaining > time.Second {
t.Fatalf("deadline remaining = %s", remaining)
}
}
func TestMySQLResolverCachesIdentityMissesForHighFrequencyFrames(t *testing.T) {
db, mock := newMockDB(t)
defer db.Close()
expectVehicleIdentifierMiss(mock, "JT808_PHONE", "13307795425")
mock.ExpectQuery("SELECT vin FROM vehicle_identity_binding WHERE phone = \\?").
WithArgs("13307795425").
WillReturnError(sql.ErrNoRows)
@@ -274,6 +867,95 @@ func TestMySQLResolverCachesIdentityMissesForHighFrequencyFrames(t *testing.T) {
}
}
func TestMySQLResolverBoundsIdentityLookupCaches(t *testing.T) {
db, _ := newMockDB(t)
defer db.Close()
resolver := NewMySQLResolverWithOptions(db, "vehicle_identity_binding", MySQLResolverOptions{
LookupCacheTTL: time.Hour,
CacheCleanupInterval: time.Hour,
MaxCacheEntries: 2,
LocationTouchInterval: time.Hour,
})
now := time.Date(2026, 7, 12, 10, 0, 0, 0, time.UTC)
for i, key := range []string{"lookup-1", "lookup-2", "lookup-3"} {
entryNow := now.Add(time.Duration(i) * time.Second)
resolver.cacheLookup(key, lookupCacheEntry{
vin: key,
expiresAt: entryNow.Add(time.Hour),
}, entryNow)
resolver.cacheRegistration("phone-"+key, registrationCacheEntry{
vin: key,
expiresAt: entryNow.Add(time.Hour),
}, entryNow)
resolver.cacheSourceMetadata("source-"+key, sourceMetadata{SourceCode: key}, false, entryNow)
}
stats := resolver.CacheStats()
if stats.LookupEntries != 2 || stats.RegistrationEntries != 2 || stats.SourceCodeEntries != 2 || stats.MaxEntries != 2 {
t.Fatalf("cache stats = %+v, want all identity caches capped at 2", stats)
}
resolver.lookupMu.Lock()
_, hasOldLookup := resolver.lookupCache["lookup-1"]
_, hasOldRegistration := resolver.registrationCache["phone-lookup-1"]
_, hasOldSource := resolver.sourceCodeCache["source-lookup-1"]
resolver.lookupMu.Unlock()
if hasOldLookup || hasOldRegistration || hasOldSource {
t.Fatalf("oldest cache entries should be evicted")
}
}
func TestMySQLResolverBoundsLocationTouchCache(t *testing.T) {
db, _ := newMockDB(t)
defer db.Close()
resolver := NewMySQLResolverWithOptions(db, "vehicle_identity_binding", MySQLResolverOptions{
LookupCacheTTL: time.Hour,
CacheCleanupInterval: time.Hour,
MaxCacheEntries: 2,
LocationTouchInterval: time.Hour,
})
now := time.Date(2026, 7, 12, 10, 0, 0, 0, time.UTC)
resolver.touchMu.Lock()
resolver.locationTouches["old-expired"] = now.Add(-2 * time.Hour)
resolver.locationTouches["phone-1"] = now.Add(-2 * time.Minute)
resolver.locationTouches["phone-2"] = now.Add(-time.Minute)
resolver.locationTouches["phone-3"] = now
resolver.locationTouchFailures["old-failure"] = now.Add(-time.Minute)
resolver.locationTouchFailures["phone-2"] = now.Add(time.Minute)
resolver.locationTouchFailures["phone-3"] = now.Add(2 * time.Minute)
resolver.locationTouchFailures["phone-4"] = now.Add(3 * time.Minute)
resolver.cleanupLocationTouchesLocked(now, false)
resolver.touchMu.Unlock()
stats := resolver.CacheStats()
if stats.LocationTouchEntries != 2 || stats.LocationTouchFailureEntries != 2 || stats.MaxEntries != 2 {
t.Fatalf("cache stats = %+v, want location touch caches capped at 2", stats)
}
resolver.touchMu.Lock()
_, hasExpired := resolver.locationTouches["old-expired"]
_, hasOldest := resolver.locationTouches["phone-1"]
_, hasExpiredFailure := resolver.locationTouchFailures["old-failure"]
_, hasOldestFailure := resolver.locationTouchFailures["phone-2"]
resolver.touchMu.Unlock()
if hasExpired || hasOldest || hasExpiredFailure || hasOldestFailure {
t.Fatalf("expired and oldest location touch entries should be evicted")
}
}
type deadlineCheckingResolver struct {
mu sync.Mutex
deadline time.Time
}
func (r *deadlineCheckingResolver) Resolve(ctx context.Context, env envelope.FrameEnvelope) (envelope.FrameEnvelope, error) {
deadline, _ := ctx.Deadline()
r.mu.Lock()
r.deadline = deadline
r.mu.Unlock()
return env, nil
}
func TestMySQLResolverEnsuresMinimalIdentitySchema(t *testing.T) {
db, mock := newMockDB(t)
defer db.Close()
@@ -285,8 +967,18 @@ func TestMySQLResolverEnsuresMinimalIdentitySchema(t *testing.T) {
WillReturnResult(sqlmock.NewResult(0, 0))
mock.ExpectExec("ALTER TABLE vehicle_identity_binding DROP COLUMN device_id").
WillReturnResult(sqlmock.NewResult(0, 0))
mock.ExpectExec("CREATE TABLE IF NOT EXISTS vehicle \\(").
WillReturnResult(sqlmock.NewResult(0, 0))
mock.ExpectExec("CREATE TABLE IF NOT EXISTS vehicle_identifier").
WillReturnResult(sqlmock.NewResult(0, 0))
mock.ExpectExec("CREATE TABLE IF NOT EXISTS jt808_registration").
WillReturnResult(sqlmock.NewResult(0, 0))
mock.ExpectExec("ALTER TABLE jt808_registration ADD COLUMN source_ip").
WillReturnResult(sqlmock.NewResult(0, 0))
mock.ExpectExec("ALTER TABLE jt808_registration ADD KEY idx_jt808_registration_source_ip").
WillReturnResult(sqlmock.NewResult(0, 0))
mock.ExpectExec("UPDATE jt808_registration").
WillReturnResult(sqlmock.NewResult(0, 0))
resolver := NewMySQLResolver(db, "vehicle_identity_binding")
if err := resolver.EnsureSchema(context.Background()); err != nil {
@@ -308,8 +1000,18 @@ func TestMySQLResolverIgnoresExistingOEMColumn(t *testing.T) {
WillReturnError(errors.New("Error 1091 (42000): Can't DROP 'uk_identity_device'; check that column/key exists"))
mock.ExpectExec("ALTER TABLE vehicle_identity_binding DROP COLUMN device_id").
WillReturnError(errors.New("Error 1091 (42000): Can't DROP 'device_id'; check that column/key exists"))
mock.ExpectExec("CREATE TABLE IF NOT EXISTS vehicle \\(").
WillReturnResult(sqlmock.NewResult(0, 0))
mock.ExpectExec("CREATE TABLE IF NOT EXISTS vehicle_identifier").
WillReturnResult(sqlmock.NewResult(0, 0))
mock.ExpectExec("CREATE TABLE IF NOT EXISTS jt808_registration").
WillReturnResult(sqlmock.NewResult(0, 0))
mock.ExpectExec("ALTER TABLE jt808_registration ADD COLUMN source_ip").
WillReturnError(errors.New("Error 1060 (42S21): Duplicate column name 'source_ip'"))
mock.ExpectExec("ALTER TABLE jt808_registration ADD KEY idx_jt808_registration_source_ip").
WillReturnError(errors.New("Error 1061 (42000): Duplicate key name 'idx_jt808_registration_source_ip'"))
mock.ExpectExec("UPDATE jt808_registration").
WillReturnResult(sqlmock.NewResult(0, 0))
resolver := NewMySQLResolver(db, "vehicle_identity_binding")
if err := resolver.EnsureSchema(context.Background()); err != nil {
@@ -339,6 +1041,15 @@ func TestIdentitySchemaUsesBusinessKeysOnly(t *testing.T) {
if !strings.Contains(registration, "phone VARCHAR(32) PRIMARY KEY") {
t.Fatalf("registration table should key by phone:\n%s", registration)
}
if !strings.Contains(registration, "source_ip VARCHAR(64)") || !strings.Contains(registration, "idx_jt808_registration_source_ip") {
t.Fatalf("registration table should keep indexed source_ip:\n%s", registration)
}
if !strings.Contains(vehicleIdentifierTableSQL, "PRIMARY KEY (protocol, source_code, identifier_type, identifier_value)") {
t.Fatalf("vehicle identifier should use protocol/source/type/value as key:\n%s", vehicleIdentifierTableSQL)
}
if strings.Contains(vehicleIdentifierTableSQL, "AUTO_INCREMENT") {
t.Fatalf("vehicle identifier should not use surrogate auto increment id:\n%s", vehicleIdentifierTableSQL)
}
}
func newMockDB(t *testing.T) (*sql.DB, sqlmock.Sqlmock) {
@@ -349,3 +1060,43 @@ func newMockDB(t *testing.T) (*sql.DB, sqlmock.Sqlmock) {
}
return db, mock
}
func expectVehicleIdentifierMiss(mock sqlmock.Sqlmock, identifierType string, value string) {
mock.ExpectQuery("SELECT DISTINCT vin").
WithArgs("JT808", identifierType, value).
WillReturnRows(sqlmock.NewRows([]string{"vin", "source_code", "platform_name"}))
}
func expectVehicleIdentifierHit(mock sqlmock.Sqlmock, identifierType string, value string, vin string) {
expectVehicleIdentifierHitWithSource(mock, identifierType, value, vin, "", "")
}
func expectVehicleIdentifierHitWithSource(mock sqlmock.Sqlmock, identifierType string, value string, vin string, sourceCode string, platformName string) {
mock.ExpectQuery("SELECT DISTINCT vin").
WithArgs("JT808", identifierType, value).
WillReturnRows(sqlmock.NewRows([]string{"vin", "source_code", "platform_name"}).AddRow(vin, sourceCode, platformName))
}
func expectSourceCodeHit(mock sqlmock.Sqlmock, sourceIP string, sourceCode string) {
mock.ExpectQuery("SELECT source_code").
WithArgs("JT808", sourceIP).
WillReturnRows(sqlmock.NewRows([]string{"source_code", "platform_name", "source_kind"}).AddRow(sourceCode, "G7s", "PLATFORM"))
}
func expectSourceCodeMiss(mock sqlmock.Sqlmock, sourceIP string) {
mock.ExpectQuery("SELECT source_code").
WithArgs("JT808", sourceIP).
WillReturnRows(sqlmock.NewRows([]string{"source_code", "platform_name", "source_kind"}))
}
func expectVehicleIdentifierMissForSource(mock sqlmock.Sqlmock, sourceCode string, identifierType string, value string) {
mock.ExpectQuery("SELECT DISTINCT vin").
WithArgs("JT808", identifierType, value, sourceCode).
WillReturnRows(sqlmock.NewRows([]string{"vin", "source_code", "platform_name"}))
}
func expectVehicleIdentifierHitForSource(mock sqlmock.Sqlmock, sourceCode string, identifierType string, value string, vin string) {
mock.ExpectQuery("SELECT DISTINCT vin").
WithArgs("JT808", identifierType, value, sourceCode).
WillReturnRows(sqlmock.NewRows([]string{"vin", "source_code", "platform_name"}).AddRow(vin, sourceCode, "G7s"))
}

325
go/vehicle-gateway/internal/identity/snapshot.go Normal file
View File

@@ -0,0 +1,325 @@
package identity
import (
"context"
"database/sql"
"fmt"
"strings"
"time"
)
type SnapshotRefreshResult struct {
BindingEntries int
IdentifierEntries int
RegistrationEntries int
SourceEntries int
RefreshedAt time.Time
}
// identitySnapshot is immutable after atomic publication, so frame handling
// performs only local map lookups and never waits for MySQL or a refresh lock.
type identitySnapshot struct {
bindings map[string]string
identifiers map[string]vehicleIdentifierMatch
registrations map[string]registrationCacheEntry
sources map[string]sourceMetadata
refreshedAt time.Time
}
func (r *MySQLResolver) RefreshSnapshot(ctx context.Context) (SnapshotRefreshResult, error) {
if r == nil || r.db == nil {
return SnapshotRefreshResult{}, fmt.Errorf("identity snapshot database is not configured")
}
r.snapshotRefreshMu.Lock()
defer r.snapshotRefreshMu.Unlock()
next := &identitySnapshot{
bindings: map[string]string{},
identifiers: map[string]vehicleIdentifierMatch{},
registrations: map[string]registrationCacheEntry{},
sources: map[string]sourceMetadata{},
}
if err := r.loadBindingSnapshot(ctx, next); err != nil {
return SnapshotRefreshResult{}, err
}
if err := r.loadIdentifierSnapshot(ctx, next); err != nil {
return SnapshotRefreshResult{}, err
}
if err := r.loadRegistrationSnapshot(ctx, next); err != nil {
return SnapshotRefreshResult{}, err
}
if err := r.loadSourceSnapshot(ctx, next); err != nil {
return SnapshotRefreshResult{}, err
}
next.refreshedAt = time.Now()
r.snapshot.Store(next)
return SnapshotRefreshResult{
BindingEntries: len(next.bindings),
IdentifierEntries: len(next.identifiers),
RegistrationEntries: len(next.registrations),
SourceEntries: len(next.sources),
RefreshedAt: next.refreshedAt,
}, nil
}
func (r *MySQLResolver) loadBindingSnapshot(ctx context.Context, target *identitySnapshot) error {
rows, err := r.db.QueryContext(ctx, "SELECT vin, plate, phone FROM "+r.table+" WHERE vin IS NOT NULL AND TRIM(vin) <> ''")
if err != nil {
return fmt.Errorf("load identity binding snapshot: %w", err)
}
defer rows.Close()
ambiguous := map[string]struct{}{}
for rows.Next() {
var vin, plate, phone sql.NullString
if err := rows.Scan(&vin, &plate, &phone); err != nil {
return fmt.Errorf("scan identity binding snapshot: %w", err)
}
vinValue := strings.TrimSpace(vin.String)
if vinValue == "" {
continue
}
addSnapshotBinding(target.bindings, ambiguous, bindingSnapshotKey("vin", vinValue), vinValue)
if value := strings.TrimSpace(plate.String); value != "" {
addSnapshotBinding(target.bindings, ambiguous, bindingSnapshotKey("plate", value), vinValue)
}
if value := normalizePhone(phone.String); value != "" {
addSnapshotBinding(target.bindings, ambiguous, bindingSnapshotKey("phone", value), vinValue)
}
}
if err := rows.Err(); err != nil {
return fmt.Errorf("iterate identity binding snapshot: %w", err)
}
return nil
}
func (r *MySQLResolver) loadIdentifierSnapshot(ctx context.Context, target *identitySnapshot) error {
rows, err := r.db.QueryContext(ctx, `SELECT protocol, source_code, identifier_type, identifier_value,
vin, COALESCE(NULLIF(TRIM(oem), ''), source_code) AS platform_name
FROM vehicle_identifier
WHERE enabled = 1 AND vin IS NOT NULL AND TRIM(vin) <> ''
AND identifier_value IS NOT NULL AND TRIM(identifier_value) <> ''`)
if err != nil {
return fmt.Errorf("load vehicle identifier snapshot: %w", err)
}
defer rows.Close()
ambiguous := map[string]struct{}{}
for rows.Next() {
var protocol, sourceCode, identifierType, identifierValue, vin, platformName sql.NullString
if err := rows.Scan(&protocol, &sourceCode, &identifierType, &identifierValue, &vin, &platformName); err != nil {
return fmt.Errorf("scan vehicle identifier snapshot: %w", err)
}
protocolValue := strings.TrimSpace(protocol.String)
typeValue := strings.ToUpper(strings.TrimSpace(identifierType.String))
value := normalizeIdentifierValue(typeValue, identifierValue.String)
vinValue := strings.TrimSpace(vin.String)
if protocolValue == "" || typeValue == "" || value == "" || vinValue == "" {
continue
}
match := vehicleIdentifierMatch{
VIN: vinValue,
SourceCode: strings.TrimSpace(sourceCode.String),
PlatformName: strings.TrimSpace(platformName.String),
}
scopedKey := vehicleIdentifierSnapshotKey(protocolValue, match.SourceCode, typeValue, value)
addSnapshotIdentifier(target.identifiers, ambiguous, scopedKey, match)
globalKey := vehicleIdentifierSnapshotKey(protocolValue, "", typeValue, value)
addSnapshotIdentifier(target.identifiers, ambiguous, globalKey, match)
}
if err := rows.Err(); err != nil {
return fmt.Errorf("iterate vehicle identifier snapshot: %w", err)
}
return nil
}
func (r *MySQLResolver) loadRegistrationSnapshot(ctx context.Context, target *identitySnapshot) error {
rows, err := r.db.QueryContext(ctx, `SELECT phone, vin, device_id, plate, auth_token
FROM jt808_registration
WHERE phone IS NOT NULL AND TRIM(phone) <> ''`)
if err != nil {
return fmt.Errorf("load jt808 registration snapshot: %w", err)
}
defer rows.Close()
for rows.Next() {
var phone, vin, deviceID, plate, authToken sql.NullString
if err := rows.Scan(&phone, &vin, &deviceID, &plate, &authToken); err != nil {
return fmt.Errorf("scan jt808 registration snapshot: %w", err)
}
phoneValue := normalizePhone(phone.String)
if phoneValue == "" {
continue
}
target.registrations[phoneValue] = registrationCacheEntry{
vin: strings.TrimSpace(vin.String),
deviceID: strings.TrimSpace(deviceID.String),
plate: strings.TrimSpace(plate.String),
authToken: strings.TrimSpace(authToken.String),
}
}
if err := rows.Err(); err != nil {
return fmt.Errorf("iterate jt808 registration snapshot: %w", err)
}
return nil
}
func (r *MySQLResolver) loadSourceSnapshot(ctx context.Context, target *identitySnapshot) error {
rows, err := r.db.QueryContext(ctx, `SELECT protocol, source_ip, source_code, platform_name, source_kind
FROM vehicle_data_source
WHERE enabled = 1 AND source_ip IS NOT NULL AND TRIM(source_ip) <> ''
AND (
(source_code IS NOT NULL AND TRIM(source_code) <> '')
OR source_kind IN ('PLATFORM', 'DIRECT')
OR (platform_name IS NOT NULL AND TRIM(platform_name) <> '')
)`)
if err != nil {
if isOptionalSourceCodeLookupError(err) {
return nil
}
return fmt.Errorf("load vehicle data source snapshot: %w", err)
}
defer rows.Close()
for rows.Next() {
var protocol, sourceIP, sourceCode, platformName, sourceKind sql.NullString
if err := rows.Scan(&protocol, &sourceIP, &sourceCode, &platformName, &sourceKind); err != nil {
return fmt.Errorf("scan vehicle data source snapshot: %w", err)
}
key := sourceSnapshotKey(protocol.String, sourceIP.String)
if key == "" {
continue
}
target.sources[key] = sourceMetadata{
SourceCode: strings.TrimSpace(sourceCode.String),
PlatformName: strings.TrimSpace(platformName.String),
SourceKind: strings.TrimSpace(sourceKind.String),
}
}
if err := rows.Err(); err != nil {
return fmt.Errorf("iterate vehicle data source snapshot: %w", err)
}
return nil
}
func addSnapshotBinding(values map[string]string, ambiguous map[string]struct{}, key string, vin string) {
if key == "" {
return
}
if _, exists := ambiguous[key]; exists {
return
}
if current, exists := values[key]; exists && !strings.EqualFold(current, vin) {
delete(values, key)
ambiguous[key] = struct{}{}
return
}
values[key] = vin
}
func addSnapshotIdentifier(values map[string]vehicleIdentifierMatch, ambiguous map[string]struct{}, key string, match vehicleIdentifierMatch) {
if key == "" {
return
}
if _, exists := ambiguous[key]; exists {
return
}
current, exists := values[key]
if !exists {
values[key] = match
return
}
if !strings.EqualFold(current.VIN, match.VIN) {
delete(values, key)
ambiguous[key] = struct{}{}
return
}
if !strings.EqualFold(current.SourceCode, match.SourceCode) {
current.SourceCode = ""
current.PlatformName = ""
} else if current.PlatformName != match.PlatformName {
current.PlatformName = ""
}
values[key] = current
}
func (r *MySQLResolver) snapshotBinding(column string, value string) (string, bool) {
key := bindingSnapshotKey(column, value)
if key == "" {
return "", false
}
snapshot := r.snapshot.Load()
if snapshot == nil {
return "", false
}
vin, ok := snapshot.bindings[key]
return vin, ok
}
func (r *MySQLResolver) snapshotIdentifier(protocolValue string, sourceCode string, identifierType string, value string) (vehicleIdentifierMatch, bool) {
key := vehicleIdentifierSnapshotKey(protocolValue, sourceCode, identifierType, value)
snapshot := r.snapshot.Load()
if snapshot == nil {
return vehicleIdentifierMatch{}, false
}
match, ok := snapshot.identifiers[key]
return match, ok
}
func (r *MySQLResolver) snapshotRegistration(phone string) (registrationCacheEntry, bool) {
phone = normalizePhone(phone)
snapshot := r.snapshot.Load()
if snapshot == nil {
return registrationCacheEntry{}, false
}
entry, ok := snapshot.registrations[phone]
return entry, ok
}
func (r *MySQLResolver) snapshotSource(protocolValue string, sourceIP string) (sourceMetadata, bool) {
key := sourceSnapshotKey(protocolValue, sourceIP)
snapshot := r.snapshot.Load()
if snapshot == nil {
return sourceMetadata{}, false
}
metadata, ok := snapshot.sources[key]
return metadata, ok
}
// JT808AuthToken serves authentication from the same immutable snapshot used
// by identity resolution. It deliberately never falls back to a per-frame SQL
// query because authentication is on the protocol response hot path.
func (r *MySQLResolver) JT808AuthToken(phone string) (string, bool) {
entry, ok := r.snapshotRegistration(phone)
token := strings.TrimSpace(entry.authToken)
return token, ok && token != ""
}
func bindingSnapshotKey(column string, value string) string {
column = strings.ToLower(strings.TrimSpace(column))
value = strings.TrimSpace(value)
if column == "phone" {
value = normalizePhone(value)
}
if value == "" || (column != "vin" && column != "plate" && column != "phone") {
return ""
}
return column + "\x00" + value
}
func vehicleIdentifierSnapshotKey(protocolValue string, sourceCode string, identifierType string, value string) string {
protocolValue = strings.TrimSpace(protocolValue)
sourceCode = strings.TrimSpace(sourceCode)
identifierType = strings.ToUpper(strings.TrimSpace(identifierType))
value = normalizeIdentifierValue(identifierType, value)
if protocolValue == "" || identifierType == "" || value == "" {
return ""
}
return "vehicle_identifier\x00" + protocolValue + "\x00" + sourceCode + "\x00" + identifierType + "\x00" + value
}
func sourceSnapshotKey(protocolValue string, sourceIP string) string {
protocolValue = strings.TrimSpace(protocolValue)
sourceIP = normalizeEndpointIP(sourceIP)
if protocolValue == "" || sourceIP == "" {
return ""
}
return protocolValue + "\x00" + sourceIP
}

166
go/vehicle-gateway/internal/identity/snapshot_test.go Normal file
View File

@@ -0,0 +1,166 @@
package identity
import (
"context"
"errors"
"testing"
"time"
"github.com/DATA-DOG/go-sqlmock"
"lingniu-vehicle-ingest/go/vehicle-gateway/internal/envelope"
)
func TestRefreshSnapshotResolvesKnownJT808WithoutPerFrameQueries(t *testing.T) {
db, mock := newMockDB(t)
defer db.Close()
expectIdentitySnapshot(mock)
resolver := NewMySQLResolverWithOptions(db, "vehicle_identity_binding", MySQLResolverOptions{
SnapshotOnlyLookups: true,
})
result, err := resolver.RefreshSnapshot(context.Background())
if err != nil {
t.Fatalf("RefreshSnapshot() error = %v", err)
}
if result.BindingEntries != 3 || result.IdentifierEntries != 2 || result.RegistrationEntries != 1 || result.SourceEntries != 1 {
t.Fatalf("snapshot result = %+v", result)
}
resolved, err := resolver.Resolve(context.Background(), envelope.FrameEnvelope{
Protocol: envelope.ProtocolJT808,
Phone: "013307795425",
SourceEndpoint: "115.231.168.135:43625",
Parsed: map[string]any{},
})
if err != nil {
t.Fatalf("Resolve() error = %v", err)
}
if resolved.VIN != "LNBVIN00000000001" {
t.Fatalf("vin = %q", resolved.VIN)
}
if resolved.SourceCode != "g7s" || resolved.PlatformName != "G7s" || resolved.SourceKind != "PLATFORM" {
t.Fatalf("source metadata = code:%q platform:%q kind:%q", resolved.SourceCode, resolved.PlatformName, resolved.SourceKind)
}
identityMetadata, _ := resolved.Parsed["identity"].(map[string]any)
if identityMetadata["cache_status"] != "snapshot" {
t.Fatalf("identity metadata = %#v, want snapshot cache status", identityMetadata)
}
if err := mock.ExpectationsWereMet(); err != nil {
t.Fatalf("sql expectations: %v", err)
}
}
func TestSnapshotOnlyResolverMissDoesNotQueryMySQL(t *testing.T) {
db, mock := newMockDB(t)
defer db.Close()
resolver := NewMySQLResolverWithOptions(db, "vehicle_identity_binding", MySQLResolverOptions{
SnapshotOnlyLookups: true,
})
resolved, err := resolver.Resolve(context.Background(), envelope.FrameEnvelope{
Protocol: envelope.ProtocolJT808,
Phone: "013307700000",
Parsed: map[string]any{},
})
if err != nil {
t.Fatalf("Resolve() error = %v", err)
}
if resolved.VIN != "" {
t.Fatalf("vin = %q, want unresolved", resolved.VIN)
}
if err := mock.ExpectationsWereMet(); err != nil {
t.Fatalf("snapshot-only miss should not query mysql: %v", err)
}
}
func TestRefreshSnapshotFailureKeepsLastKnownGoodSnapshot(t *testing.T) {
db, mock := newMockDB(t)
defer db.Close()
expectIdentitySnapshot(mock)
resolver := NewMySQLResolverWithOptions(db, "vehicle_identity_binding", MySQLResolverOptions{
SnapshotOnlyLookups: true,
})
first, err := resolver.RefreshSnapshot(context.Background())
if err != nil {
t.Fatalf("first RefreshSnapshot() error = %v", err)
}
mock.ExpectQuery("SELECT vin, plate, phone").
WillReturnError(errors.New("mysql unavailable"))
if _, err := resolver.RefreshSnapshot(context.Background()); err == nil {
t.Fatal("second RefreshSnapshot() error = nil, want failure")
}
resolved, err := resolver.Resolve(context.Background(), envelope.FrameEnvelope{
Protocol: envelope.ProtocolJT808,
Phone: "013307795425",
Parsed: map[string]any{},
})
if err != nil {
t.Fatalf("Resolve() after failed refresh error = %v", err)
}
if resolved.VIN != "LNBVIN00000000001" {
t.Fatalf("vin after failed refresh = %q", resolved.VIN)
}
stats := resolver.CacheStats()
if !stats.SnapshotReady || stats.SnapshotRefreshedAt.IsZero() || !stats.SnapshotRefreshedAt.Equal(first.RefreshedAt) {
t.Fatalf("snapshot stats after failed refresh = %+v, first = %+v", stats, first)
}
if err := mock.ExpectationsWereMet(); err != nil {
t.Fatalf("sql expectations: %v", err)
}
}
func expectIdentitySnapshot(mock sqlmock.Sqlmock) {
mock.ExpectQuery("SELECT vin, plate, phone").
WillReturnRows(sqlmock.NewRows([]string{"vin", "plate", "phone"}).
AddRow("LNBVIN00000000001", "粤A00001", "13307795425"))
mock.ExpectQuery("SELECT protocol, source_code, identifier_type, identifier_value").
WillReturnRows(sqlmock.NewRows([]string{"protocol", "source_code", "identifier_type", "identifier_value", "vin", "platform_name"}).
AddRow("JT808", "g7s", "JT808_PHONE", "13307795425", "LNBVIN00000000001", "G7s"))
mock.ExpectQuery("SELECT phone, vin, device_id, plate, auth_token").
WillReturnRows(sqlmock.NewRows([]string{"phone", "vin", "device_id", "plate", "auth_token"}).
AddRow("13307795425", "LNBVIN00000000001", "DEVICE-1", "粤A00001", "device-code"))
mock.ExpectQuery("SELECT protocol, source_ip, source_code, platform_name, source_kind").
WillReturnRows(sqlmock.NewRows([]string{"protocol", "source_ip", "source_code", "platform_name", "source_kind"}).
AddRow("JT808", "115.231.168.135", "g7s", "G7s", "PLATFORM"))
}
func TestSnapshotServesJT808AuthenticationTokenByNormalizedPhone(t *testing.T) {
db, mock := newMockDB(t)
defer db.Close()
expectIdentitySnapshot(mock)
resolver := NewMySQLResolverWithOptions(db, "vehicle_identity_binding", MySQLResolverOptions{
SnapshotOnlyLookups: true,
})
if _, err := resolver.RefreshSnapshot(context.Background()); err != nil {
t.Fatalf("RefreshSnapshot() error = %v", err)
}
token, ok := resolver.JT808AuthToken("0013307795425")
if !ok || token != "device-code" {
t.Fatalf("JT808AuthToken() = %q, %v", token, ok)
}
if err := mock.ExpectationsWereMet(); err != nil {
t.Fatalf("sql expectations: %v", err)
}
}
func TestSnapshotResultRefreshedAtUsesCurrentTime(t *testing.T) {
db, mock := newMockDB(t)
defer db.Close()
expectIdentitySnapshot(mock)
resolver := NewMySQLResolverWithOptions(db, "vehicle_identity_binding", MySQLResolverOptions{
SnapshotOnlyLookups: true,
})
before := time.Now()
result, err := resolver.RefreshSnapshot(context.Background())
if err != nil {
t.Fatalf("RefreshSnapshot() error = %v", err)
}
if result.RefreshedAt.Before(before) || result.RefreshedAt.After(time.Now()) {
t.Fatalf("refreshed_at = %v, want current time", result.RefreshedAt)
}
}

35
go/vehicle-gateway/internal/loadsim/config.go
View File

@@ -14,6 +14,9 @@ const (
ProtocolGB32960 Protocol = "gb32960"
ProtocolJT808 Protocol = "jt808"
ProtocolYutongMQTT Protocol = "yutong-mqtt"
DefaultJT808PhoneBase int64 = 139000000000
maxJT808Phone int64 = 999999999999
)
type Config struct {
@@ -25,17 +28,21 @@ type Config struct {
Duration time.Duration
Template string
SendFrames bool
JT808PhoneBase int64
DrainResponses bool
}
type FlagConfig struct {
protocol string
addr string
connections int
connectRate int
sendInterval time.Duration
duration time.Duration
template string
sendFrames bool
protocol string
addr string
connections int
connectRate int
sendInterval time.Duration
duration time.Duration
template string
sendFrames bool
jt808PhoneBase int64
drainResponses bool
}
func RegisterFlags(fs *flag.FlagSet) *FlagConfig {
@@ -48,6 +55,8 @@ func RegisterFlags(fs *flag.FlagSet) *FlagConfig {
fs.DurationVar(&cfg.duration, "duration", 10*time.Minute, "load test duration")
fs.StringVar(&cfg.template, "template", "", "frame template name")
fs.BoolVar(&cfg.sendFrames, "send", true, "send protocol frames while connections are open")
fs.Int64Var(&cfg.jt808PhoneBase, "jt808-phone-base", DefaultJT808PhoneBase, "first synthetic JT808 phone; one consecutive phone is assigned per connection")
fs.BoolVar(&cfg.drainResponses, "drain-responses", true, "continuously read protocol responses while frames are being sent")
return cfg
}
@@ -73,6 +82,14 @@ func (c *FlagConfig) Build() (Config, error) {
if c.duration <= 0 {
return Config{}, errors.New("duration must be positive")
}
if protocol == ProtocolJT808 {
if c.jt808PhoneBase <= 0 || c.jt808PhoneBase > maxJT808Phone {
return Config{}, errors.New("jt808-phone-base must be a positive 12-digit-or-shorter number")
}
if int64(c.connections-1) > maxJT808Phone-c.jt808PhoneBase {
return Config{}, errors.New("jt808 synthetic phone range exceeds 12 digits")
}
}
return Config{
Protocol: protocol,
Addr: strings.TrimSpace(c.addr),
@@ -82,5 +99,7 @@ func (c *FlagConfig) Build() (Config, error) {
Duration: c.duration,
Template: strings.TrimSpace(c.template),
SendFrames: c.sendFrames,
JT808PhoneBase: c.jt808PhoneBase,
DrainResponses: c.drainResponses,
}, nil
}

8
go/vehicle-gateway/internal/loadsim/config_test.go
View File

@@ -18,6 +18,7 @@ func TestConfigFromFlagsParsesCapacityKnobs(t *testing.T) {
"-send-interval", "5s",
"-duration", "30m",
"-template", "0200",
"-jt808-phone-base", "138900000000",
"-send=false",
})
if err != nil {
@@ -53,6 +54,12 @@ func TestConfigFromFlagsParsesCapacityKnobs(t *testing.T) {
if got.SendFrames {
t.Fatal("SendFrames = true, want false")
}
if got.JT808PhoneBase != 138900000000 {
t.Fatalf("JT808PhoneBase = %d, want 138900000000", got.JT808PhoneBase)
}
if !got.DrainResponses {
t.Fatal("DrainResponses = false, want true by default")
}
}
func TestConfigFromFlagsRejectsUnsafeValues(t *testing.T) {
@@ -62,6 +69,7 @@ func TestConfigFromFlagsRejectsUnsafeValues(t *testing.T) {
"zero connections": {"-protocol", "jt808", "-addr", "127.0.0.1:808", "-connections", "0"},
"zero connect rate": {"-protocol", "jt808", "-addr", "127.0.0.1:808", "-connections", "1", "-connect-rate", "0"},
"zero interval": {"-protocol", "jt808", "-addr", "127.0.0.1:808", "-connections", "1", "-send-interval", "0s"},
"phone overflow": {"-protocol", "jt808", "-addr", "127.0.0.1:808", "-connections", "2", "-jt808-phone-base", "999999999999"},
} {
t.Run(name, func(t *testing.T) {
fs := flag.NewFlagSet("load-sim", flag.ContinueOnError)

20
go/vehicle-gateway/internal/loadsim/mutator.go
View File

@@ -7,22 +7,30 @@ import (
)
type FrameFactory struct {
protocol Protocol
base []byte
protocol Protocol
base []byte
jt808PhoneBase int64
}
func NewFrameFactory(protocol Protocol, template string) (*FrameFactory, error) {
return NewFrameFactoryWithJT808PhoneBase(protocol, template, DefaultJT808PhoneBase)
}
func NewFrameFactoryWithJT808PhoneBase(protocol Protocol, template string, phoneBase int64) (*FrameFactory, error) {
base, err := FrameTemplate(protocol, template)
if err != nil {
return nil, err
}
return &FrameFactory{protocol: protocol, base: base}, nil
if phoneBase <= 0 {
phoneBase = DefaultJT808PhoneBase
}
return &FrameFactory{protocol: protocol, base: base, jt808PhoneBase: phoneBase}, nil
}
func (f *FrameFactory) Frame(connectionIndex int, frameIndex int64) ([]byte, error) {
switch f.protocol {
case ProtocolJT808:
return mutateJT808Frame(f.base, connectionIndex, frameIndex)
return mutateJT808Frame(f.base, f.jt808PhoneBase, connectionIndex, frameIndex)
case ProtocolGB32960:
return mutateGB32960Frame(f.base, connectionIndex, frameIndex)
default:
@@ -31,7 +39,7 @@ func (f *FrameFactory) Frame(connectionIndex int, frameIndex int64) ([]byte, err
}
}
func mutateJT808Frame(base []byte, connectionIndex int, frameIndex int64) ([]byte, error) {
func mutateJT808Frame(base []byte, phoneBase int64, connectionIndex int, frameIndex int64) ([]byte, error) {
if len(base) < 2 || base[0] != 0x7e || base[len(base)-1] != 0x7e {
return nil, fmt.Errorf("jt808 template must include 0x7e delimiters")
}
@@ -42,7 +50,7 @@ func mutateJT808Frame(base []byte, connectionIndex int, frameIndex int64) ([]byt
if len(payload) < 13 {
return nil, fmt.Errorf("jt808 template too short: %d", len(payload))
}
phone := fmt.Sprintf("%012d", 139000000000+connectionIndex%100000000)
phone := fmt.Sprintf("%012d", phoneBase+int64(connectionIndex))
copy(payload[4:10], encodeBCD(phone, 6))
binary.BigEndian.PutUint16(payload[10:12], uint16((int(frameIndex)+connectionIndex)%65536))
bodySize := int(binary.BigEndian.Uint16(payload[2:4]) & 0x03ff)

14
go/vehicle-gateway/internal/loadsim/mutator_test.go
View File

@@ -38,6 +38,20 @@ func TestFrameFactoryGeneratesUniqueParsableJT808Frames(t *testing.T) {
}
}
func TestFrameFactoryUsesConfiguredJT808PhoneBase(t *testing.T) {
factory, err := NewFrameFactoryWithJT808PhoneBase(ProtocolJT808, "0200", 138900000000)
if err != nil {
t.Fatalf("NewFrameFactoryWithJT808PhoneBase() error = %v", err)
}
frame, err := factory.Frame(42, 0)
if err != nil {
t.Fatalf("Frame() error = %v", err)
}
if got := parseJT808Frame(t, frame).Phone; got != "138900000042" {
t.Fatalf("phone = %q, want 138900000042", got)
}
}
func TestFrameFactoryGeneratesUniqueParsableGB32960Frames(t *testing.T) {
factory, err := NewFrameFactory(ProtocolGB32960, "realtime")
if err != nil {

62
go/vehicle-gateway/internal/loadsim/runner.go
View File

@@ -2,6 +2,8 @@ package loadsim
import (
"context"
"errors"
"io"
"net"
"sync"
"sync/atomic"
@@ -19,24 +21,32 @@ type Stats struct {
ConnectionsFailed int64
FramesWritten int64
WriteErrors int64
ResponseBytes int64
ReadErrors int64
}
func (r Runner) Run(ctx context.Context, cfg Config) (Stats, error) {
phoneBase := cfg.JT808PhoneBase
if phoneBase <= 0 {
phoneBase = DefaultJT808PhoneBase
}
if _, err := (&FlagConfig{
protocol: string(cfg.Protocol),
addr: cfg.Addr,
connections: cfg.Connections,
connectRate: cfg.ConnectRatePerSecond,
sendInterval: cfg.SendInterval,
duration: cfg.Duration,
template: cfg.Template,
protocol: string(cfg.Protocol),
addr: cfg.Addr,
connections: cfg.Connections,
connectRate: cfg.ConnectRatePerSecond,
sendInterval: cfg.SendInterval,
duration: cfg.Duration,
template: cfg.Template,
jt808PhoneBase: phoneBase,
drainResponses: cfg.DrainResponses,
}).Build(); err != nil {
return Stats{}, err
}
var factory *FrameFactory
if cfg.SendFrames {
var err error
factory, err = NewFrameFactory(cfg.Protocol, cfg.Template)
factory, err = NewFrameFactoryWithJT808PhoneBase(cfg.Protocol, cfg.Template, phoneBase)
if err != nil {
return Stats{}, err
}
@@ -74,12 +84,24 @@ func (r Runner) Run(ctx context.Context, cfg Config) (Stats, error) {
connectionIndex++
go func() {
defer wg.Done()
defer conn.Close()
if cfg.SendFrames {
var readDone chan struct{}
if cfg.DrainResponses {
readDone = make(chan struct{})
go func() {
drainResponses(runCtx, conn, &stats)
close(readDone)
}()
}
writeLoop(runCtx, conn, factory, connIndex, cfg.SendInterval, &stats)
_ = conn.Close()
if readDone != nil {
<-readDone
}
return
}
<-runCtx.Done()
_ = conn.Close()
}()
}
}
@@ -88,6 +110,28 @@ func (r Runner) Run(ctx context.Context, cfg Config) (Stats, error) {
return stats, nil
}
func drainResponses(ctx context.Context, conn net.Conn, stats *Stats) {
buffer := make([]byte, 4096)
for {
_ = conn.SetReadDeadline(time.Now().Add(time.Second))
read, err := conn.Read(buffer)
if read > 0 {
atomic.AddInt64(&stats.ResponseBytes, int64(read))
}
if err == nil {
continue
}
if ctx.Err() != nil || errors.Is(err, net.ErrClosed) || errors.Is(err, io.EOF) {
return
}
if timeout, ok := err.(net.Error); ok && timeout.Timeout() {
continue
}
atomic.AddInt64(&stats.ReadErrors, 1)
return
}
}
func writeLoop(ctx context.Context, conn net.Conn, factory *FrameFactory, connectionIndex int, interval time.Duration, stats *Stats) {
for frameIndex := int64(0); ; frameIndex++ {
payload, err := factory.Frame(connectionIndex, frameIndex)

42
go/vehicle-gateway/internal/loadsim/runner_test.go
View File

@@ -131,6 +131,48 @@ func TestRunnerCanHoldConnectionsWithoutWritingFrames(t *testing.T) {
}
}
func TestRunnerDrainsProtocolResponses(t *testing.T) {
runner := Runner{
Dial: func(ctx context.Context, network, addr string) (net.Conn, error) {
client, server := net.Pipe()
go func() {
defer server.Close()
buffer := make([]byte, 4096)
for {
if _, err := server.Read(buffer); err != nil {
return
}
if _, err := server.Write([]byte{0x7e, 0x80, 0x01, 0x7e}); err != nil {
return
}
}
}()
return client, nil
},
}
stats, err := runner.Run(context.Background(), Config{
Protocol: ProtocolJT808,
Addr: "127.0.0.1:808",
Connections: 1,
ConnectRatePerSecond: 1000,
SendInterval: time.Millisecond,
Duration: 10 * time.Millisecond,
Template: "0200",
SendFrames: true,
DrainResponses: true,
})
if err != nil {
t.Fatalf("Run() error = %v", err)
}
if stats.ResponseBytes == 0 {
t.Fatalf("ResponseBytes = 0, want drained protocol responses")
}
if stats.ReadErrors != 0 {
t.Fatalf("ReadErrors = %d, want 0", stats.ReadErrors)
}
}
type recordingConn struct {
write func([]byte) (int, error)
close func() error

62
go/vehicle-gateway/internal/metrics/metrics.go
View File

@@ -4,12 +4,16 @@ import (
"fmt"
"math"
"net/http"
"runtime"
"sort"
"strconv"
"strings"
"sync"
"time"
)
var processStartedAt = time.Now()
type Labels map[string]string
type Registry struct {
@@ -33,6 +37,56 @@ func NewRegistry() *Registry {
}
}
func RegisterServiceInfo(registry *Registry, service string) {
if registry == nil {
return
}
service = strings.TrimSpace(service)
if service == "" {
service = "vehicle-service"
}
registry.SetGauge("vehicle_service_info", Labels{"service": service}, 1)
RecordProcessRuntime(registry)
}
func RecordProcessRuntime(registry *Registry) {
if registry == nil {
return
}
var mem runtime.MemStats
runtime.ReadMemStats(&mem)
registry.SetGauge("vehicle_process_start_time_unix_seconds", nil, float64(processStartedAt.Unix()))
registry.SetGauge("vehicle_process_uptime_seconds", nil, time.Since(processStartedAt).Seconds())
registry.SetGauge("vehicle_process_heap_alloc_bytes", nil, float64(mem.HeapAlloc))
registry.SetGauge("vehicle_process_heap_sys_bytes", nil, float64(mem.HeapSys))
registry.SetGauge("vehicle_process_goroutines", nil, float64(runtime.NumGoroutine()))
}
func RegisterKafkaConsumerInfo(registry *Registry, service string, group string, topics []string) {
if registry == nil {
return
}
service = strings.TrimSpace(service)
if service == "" {
service = "vehicle-service"
}
group = strings.TrimSpace(group)
if group == "" {
return
}
for _, topic := range topics {
topic = strings.TrimSpace(topic)
if topic == "" {
continue
}
registry.SetGauge("vehicle_kafka_consumer_info", Labels{
"service": service,
"group": group,
"topic": topic,
}, 1)
}
}
func (r *Registry) IncCounter(name string, labels Labels) {
r.AddCounter(name, labels, 1)
}
@@ -90,6 +144,13 @@ func (r *Registry) SetKafkaLag(name string, topic string, partition int, offset
}, float64(lag))
}
func RecordLastActivity(registry *Registry, name string, labels Labels) {
if registry == nil {
return
}
registry.SetGauge(name, labels, float64(time.Now().Unix()))
}
func (r *Registry) ObserveHistogram(name string, labels Labels, buckets []float64, value float64) {
name = strings.TrimSpace(name)
if r == nil || name == "" {
@@ -265,6 +326,7 @@ func NewHandler(registry *Registry) http.Handler {
http.NotFound(w, r)
return
}
RecordProcessRuntime(registry)
w.Header().Set("Content-Type", "text/plain; version=0.0.4; charset=utf-8")
_, _ = w.Write([]byte(registry.Render()))
})

59
go/vehicle-gateway/internal/metrics/metrics_test.go
View File

@@ -85,6 +85,65 @@ func TestRegistryRecordsKafkaLagGauge(t *testing.T) {
}
}
func TestRecordLastActivityRendersUnixGauge(t *testing.T) {
registry := NewRegistry()
RecordLastActivity(registry, "vehicle_stat_last_message_unix_seconds", Labels{"topic": "vehicle.fields.go.jt808.v1", "status": "received"})
text := registry.Render()
if !strings.Contains(text, `# TYPE vehicle_stat_last_message_unix_seconds gauge`) ||
!strings.Contains(text, `vehicle_stat_last_message_unix_seconds{status="received",topic="vehicle.fields.go.jt808.v1"} `) {
t.Fatalf("last activity gauge missing:\n%s", text)
}
}
func TestRegisterServiceInfoRendersStableGauge(t *testing.T) {
registry := NewRegistry()
RegisterServiceInfo(registry, "vehicle-realtime-api")
text := registry.Render()
if !strings.Contains(text, `vehicle_service_info{service="vehicle-realtime-api"} 1`) {
t.Fatalf("service info metric missing:\n%s", text)
}
for _, want := range []string{
`# TYPE vehicle_process_start_time_unix_seconds gauge`,
`# TYPE vehicle_process_uptime_seconds gauge`,
`# TYPE vehicle_process_heap_alloc_bytes gauge`,
`# TYPE vehicle_process_heap_sys_bytes gauge`,
`# TYPE vehicle_process_goroutines gauge`,
`vehicle_process_start_time_unix_seconds `,
`vehicle_process_uptime_seconds `,
`vehicle_process_heap_alloc_bytes `,
`vehicle_process_heap_sys_bytes `,
`vehicle_process_goroutines `,
} {
if !strings.Contains(text, want) {
t.Fatalf("process runtime metric missing %s:\n%s", want, text)
}
}
}
func TestRegisterKafkaConsumerInfoRendersTopics(t *testing.T) {
registry := NewRegistry()
RegisterKafkaConsumerInfo(registry, "vehicle-stat-writer", "go-stat-writer", []string{
"vehicle.fields.go.gb32960.v1",
"vehicle.fields.go.jt808.v1",
"",
})
text := registry.Render()
for _, want := range []string{
`vehicle_kafka_consumer_info{group="go-stat-writer",service="vehicle-stat-writer",topic="vehicle.fields.go.gb32960.v1"} 1`,
`vehicle_kafka_consumer_info{group="go-stat-writer",service="vehicle-stat-writer",topic="vehicle.fields.go.jt808.v1"} 1`,
} {
if !strings.Contains(text, want) {
t.Fatalf("consumer info metric missing %s:\n%s", want, text)
}
}
}
func TestHandlerServesPrometheusText(t *testing.T) {
registry := NewRegistry()
registry.IncCounter("vehicle_kafka_commits_total", Labels{"service": "history"})

35
go/vehicle-gateway/internal/metrics/pending.go Normal file
View File

@@ -0,0 +1,35 @@
package metrics
import "sync"
// PendingPairGauge keeps process-wide in-flight totals correct when a service
// has multiple workers updating the same pair of pending gauges.
type PendingPairGauge struct {
mu sync.Mutex
first int
second int
}
func (g *PendingPairGauge) Add(registry *Registry, firstMetric string, secondMetric string, firstDelta int, secondDelta int) {
if g == nil {
return
}
g.mu.Lock()
g.first += firstDelta
g.second += secondDelta
if g.first < 0 {
g.first = 0
}
if g.second < 0 {
g.second = 0
}
first := g.first
second := g.second
g.mu.Unlock()
if registry == nil {
return
}
registry.SetGauge(firstMetric, nil, float64(first))
registry.SetGauge(secondMetric, nil, float64(second))
}

79
go/vehicle-gateway/internal/metrics/recent_latency.go Normal file
View File

@@ -0,0 +1,79 @@
package metrics
import (
"math"
"sort"
"strings"
"sync"
)
// RecentLatencyByKey tracks a bounded in-memory latency window per logical key.
// It is intentionally small and process-local; Prometheus histograms keep the
// lifetime view, while this gives capacity-check a recent-window signal.
type RecentLatencyByKey struct {
mu sync.Mutex
size int
windows map[string]*recentLatencyWindow
}
type recentLatencyWindow struct {
values []float64
next int
count int
}
func NewRecentLatencyByKey(size int) *RecentLatencyByKey {
if size <= 0 {
size = 1
}
return &RecentLatencyByKey{
size: size,
windows: map[string]*recentLatencyWindow{},
}
}
func (r *RecentLatencyByKey) Observe(key string, value float64) (p99 float64, samples int) {
if r == nil {
return 0, 0
}
key = strings.TrimSpace(key)
if key == "" {
key = "unknown"
}
if value < 0 || math.IsNaN(value) {
value = 0
}
if math.IsInf(value, 1) {
value = math.MaxFloat64
}
r.mu.Lock()
defer r.mu.Unlock()
window := r.windows[key]
if window == nil {
window = &recentLatencyWindow{values: make([]float64, r.size)}
r.windows[key] = window
}
window.values[window.next] = value
window.next = (window.next + 1) % len(window.values)
if window.count < len(window.values) {
window.count++
}
return window.quantileLocked(0.99), window.count
}
func (w *recentLatencyWindow) quantileLocked(q float64) float64 {
if w == nil || w.count == 0 {
return 0
}
snapshot := append([]float64(nil), w.values[:w.count]...)
sort.Float64s(snapshot)
index := int(math.Ceil(q*float64(len(snapshot)))) - 1
if index < 0 {
index = 0
}
if index >= len(snapshot) {
index = len(snapshot) - 1
}
return snapshot[index]
}

34
go/vehicle-gateway/internal/metrics/recent_latency_test.go Normal file
View File

@@ -0,0 +1,34 @@
package metrics
import "testing"
func TestRecentLatencyByKeyTracksBoundedP99PerKey(t *testing.T) {
tracker := NewRecentLatencyByKey(3)
p99, samples := tracker.Observe("a", 10)
if p99 != 10 || samples != 1 {
t.Fatalf("first p99=%v samples=%d", p99, samples)
}
tracker.Observe("a", 20)
p99, samples = tracker.Observe("a", 30)
if p99 != 30 || samples != 3 {
t.Fatalf("filled p99=%v samples=%d", p99, samples)
}
p99, samples = tracker.Observe("a", 5)
if p99 != 30 || samples != 3 {
t.Fatalf("bounded p99=%v samples=%d", p99, samples)
}
p99, samples = tracker.Observe("b", 7)
if p99 != 7 || samples != 1 {
t.Fatalf("second key p99=%v samples=%d", p99, samples)
}
}
func TestRecentLatencyByKeySanitizesBadValues(t *testing.T) {
tracker := NewRecentLatencyByKey(2)
p99, samples := tracker.Observe("", -10)
if p99 != 0 || samples != 1 {
t.Fatalf("p99=%v samples=%d", p99, samples)
}
}

19
go/vehicle-gateway/internal/observability/logger.go
View File

@@ -3,11 +3,28 @@ package observability
import (
"log/slog"
"os"
"strings"
)
// NewLogger returns a JSON logger with a stable service field so ECS logs from
// different Go processes can be filtered without relying on container names.
func NewLogger(service string) *slog.Logger {
handler := slog.NewJSONHandler(os.Stdout, &slog.HandlerOptions{AddSource: true})
handler := slog.NewJSONHandler(os.Stdout, &slog.HandlerOptions{
AddSource: true,
Level: parseLogLevel(os.Getenv("LOG_LEVEL")),
})
return slog.New(handler).With("service", service)
}
func parseLogLevel(value string) slog.Level {
switch strings.ToLower(strings.TrimSpace(value)) {
case "debug":
return slog.LevelDebug
case "warn", "warning":
return slog.LevelWarn
case "error":
return slog.LevelError
default:
return slog.LevelInfo
}
}

27
go/vehicle-gateway/internal/observability/logger_test.go Normal file
View File

@@ -0,0 +1,27 @@
package observability
import (
"log/slog"
"testing"
)
func TestParseLogLevel(t *testing.T) {
tests := []struct {
name string
value string
want slog.Level
}{
{name: "empty defaults to info", value: "", want: slog.LevelInfo},
{name: "unknown defaults to info", value: "trace", want: slog.LevelInfo},
{name: "debug", value: "debug", want: slog.LevelDebug},
{name: "warn alias", value: "warning", want: slog.LevelWarn},
{name: "error trims and lowercases", value: " ERROR ", want: slog.LevelError},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
if got := parseLogLevel(tt.value); got != tt.want {
t.Fatalf("parseLogLevel(%q) = %v, want %v", tt.value, got, tt.want)
}
})
}
}

Some files were not shown because too many files have changed in this diff Show More