feat(go): reuse parsed fields across projections
This commit is contained in:
lingniu
341
docs/ops/go-vehicle-ingest-memory.md
Normal file
341
docs/ops/go-vehicle-ingest-memory.md
Normal file
@@ -0,0 +1,341 @@
|
||||
# Go 车辆接入上下文记忆
|
||||
|
||||
更新时间：2026-07-03
|
||||
|
||||
这份文档用于在 Codex 上下文重载后快速恢复项目状态。更完整的运行排查步骤见：
|
||||
|
||||
- `docs/ops/vehicle-ingest-runbook.md`
|
||||
- `docs/architecture/production-data-plane-inventory.md`
|
||||
|
||||
## 当前目标
|
||||
|
||||
Go 版本车辆数据接入链路已经作为生产主链路运行在 ECS `115.29.187.205`。
|
||||
|
||||
核心目标：
|
||||
|
||||
1. 高性能可靠接收 GB32960、JT808、宇通 MQTT。
|
||||
2. 协议解析后先写 NATS，再桥接 Kafka。
|
||||
3. Kafka raw 供 TDengine 历史、Redis 实时、MySQL 当前态/统计消费。
|
||||
4. 尽量避免 Java 老链路、Xinda 相关链路和重复存储。
|
||||
5. 解析字段只投影一次，避免 TDengine/Redis/MySQL 各自重复 flatten。
|
||||
|
||||
## 服务和端口
|
||||
|
||||
ECS：`115.29.187.205`
|
||||
|
||||
| 服务 | systemd | 端口 | 作用 |
|
||||
| --- | --- | --- | --- |
|
||||
| Gateway | `lingniu-go-gateway.service` | `808`、`32960`、`20211` | JT808、GB32960、宇通 MQTT 接入和解析 |
|
||||
| NATS fast writer | `lingniu-go-nats-fast-writer.service` | 健康端口随 env | NATS 快速写 TDengine/Redis |
|
||||
| NATS Kafka bridge | `lingniu-go-nats-kafka-bridge.service` | `20214` | NATS 到 Kafka |
|
||||
| History writer | `lingniu-go-history-writer.service` | `20212` | Kafka raw 写 TDengine |
|
||||
| Stat writer | `lingniu-go-stat-writer.service` | `20213` | Kafka raw 写每日里程 |
|
||||
| Realtime API | `lingniu-go-realtime-api.service` | `20200` | Redis/MySQL 投影和 HTTP API |
|
||||
|
||||
业务端口：
|
||||
|
||||
- JT808：`0.0.0.0:808`
|
||||
- GB32960：`0.0.0.0:32960`
|
||||
- Swagger/API：`http://115.29.187.205:20200/swagger-ui/index.html`
|
||||
|
||||
## 部署路径
|
||||
|
||||
Go 服务使用 systemd 裸机部署，不使用 Docker。
|
||||
|
||||
```bash
|
||||
/opt/lingniu-go-native/current
|
||||
/opt/lingniu-go-native/releases/*
|
||||
/opt/lingniu-go-native/env/*.env
|
||||
```
|
||||
|
||||
最近 release：
|
||||
|
||||
```text
|
||||
/opt/lingniu-go-native/releases/parsed-fields-once-20260703163821
|
||||
```
|
||||
|
||||
服务重启：
|
||||
|
||||
```bash
|
||||
systemctl restart lingniu-go-gateway.service
|
||||
systemctl restart lingniu-go-nats-fast-writer.service
|
||||
systemctl restart lingniu-go-nats-kafka-bridge.service
|
||||
systemctl restart lingniu-go-history-writer.service
|
||||
systemctl restart lingniu-go-stat-writer.service
|
||||
systemctl restart lingniu-go-realtime-api.service
|
||||
```
|
||||
|
||||
健康检查：
|
||||
|
||||
```bash
|
||||
for port in 20211 20212 20213 20214 20200; do
|
||||
curl -fsS "http://127.0.0.1:${port}/readyz"
|
||||
echo
|
||||
done
|
||||
```
|
||||
|
||||
## 中间件
|
||||
|
||||
Kafka / NATS ECS：
|
||||
|
||||
- 公网：`114.55.58.251`
|
||||
- 内网：`172.17.111.56`
|
||||
- NATS：`172.17.111.56:4222`
|
||||
|
||||
TDengine：
|
||||
|
||||
- 内网：`172.17.111.57`
|
||||
- WS：`172.17.111.57:6041`
|
||||
- 数据库：`lingniu_vehicle_ts`
|
||||
|
||||
MySQL RDS：
|
||||
|
||||
- 内网：`rm-bp179zbv481rnw3e2.mysql.rds.aliyuncs.com:3306`
|
||||
- 数据库：`lingniu_vehicle_data`
|
||||
|
||||
Redis：
|
||||
|
||||
- 内网：`r-bp1u741kij7e51i481.redis.rds.aliyuncs.com:6379`
|
||||
- DB：`50`
|
||||
|
||||
不要在最终回答中明文输出生产密码。
|
||||
|
||||
## Topic
|
||||
|
||||
当前 Go raw topic：
|
||||
|
||||
```text
|
||||
vehicle.raw.go.gb32960.v1
|
||||
vehicle.raw.go.jt808.v1
|
||||
vehicle.raw.go.yutong-mqtt.v1
|
||||
```
|
||||
|
||||
fields topic 已存在：
|
||||
|
||||
```text
|
||||
vehicle.fields.go.gb32960.v1
|
||||
vehicle.fields.go.jt808.v1
|
||||
vehicle.fields.go.yutong-mqtt.v1
|
||||
```
|
||||
|
||||
当前核心链路仍以 raw topic 作为事实输入；raw envelope 里已经包含一次性预计算的 `parsed_fields`。
|
||||
|
||||
## 代码结构
|
||||
|
||||
主要目录：
|
||||
|
||||
```text
|
||||
go/vehicle-gateway/cmd/gateway
|
||||
go/vehicle-gateway/cmd/history-writer
|
||||
go/vehicle-gateway/cmd/realtime-api
|
||||
go/vehicle-gateway/cmd/stat-writer
|
||||
go/vehicle-gateway/cmd/nats-fast-writer
|
||||
go/vehicle-gateway/cmd/nats-kafka-bridge
|
||||
go/vehicle-gateway/internal/protocol/gb32960
|
||||
go/vehicle-gateway/internal/protocol/jt808
|
||||
go/vehicle-gateway/internal/protocol/yutongmqtt
|
||||
go/vehicle-gateway/internal/realtime
|
||||
go/vehicle-gateway/internal/history
|
||||
go/vehicle-gateway/internal/identity
|
||||
```
|
||||
|
||||
每次改代码后在本地跑：
|
||||
|
||||
```bash
|
||||
cd go/vehicle-gateway
|
||||
go test ./...
|
||||
```
|
||||
|
||||
## 解析和存储原则
|
||||
|
||||
### 一次解析字段
|
||||
|
||||
2026-07-03 已完成优化：
|
||||
|
||||
- `FrameEnvelope` 增加：
|
||||
- `parsed_fields`
|
||||
- `parsed_field_types`
|
||||
- Gateway 在发布 raw 前调用 `realtime.EnsureParsedFields(&env)`。
|
||||
- TDengine writer、Redis realtime KV、MySQL snapshot 优先使用 `env.ParsedFields`。
|
||||
- 旧消息没有 `parsed_fields` 时，保留 fallback，从 `Parsed` 展开一次。
|
||||
|
||||
相关文件：
|
||||
|
||||
```text
|
||||
go/vehicle-gateway/internal/envelope/envelope.go
|
||||
go/vehicle-gateway/internal/realtime/kv.go
|
||||
go/vehicle-gateway/internal/gateway/tcp_server.go
|
||||
go/vehicle-gateway/internal/gateway/mqtt_client.go
|
||||
go/vehicle-gateway/internal/history/writer.go
|
||||
go/vehicle-gateway/internal/realtime/repository.go
|
||||
go/vehicle-gateway/internal/realtime/snapshot_writer.go
|
||||
```
|
||||
|
||||
### GB32960 多帧
|
||||
|
||||
GB32960 有两类实际模式：
|
||||
|
||||
1. 单条 `0x02` 同时包含标准 data unit 和 vendor data unit。
|
||||
2. 标准帧加 vendor 栈分片帧。
|
||||
|
||||
例子：
|
||||
|
||||
- `LB9A32A24R0LS1426`：单帧完整混合上报，单帧约 `774 bytes`，`gd_fc_stack.cell_count=108`。
|
||||
- `LNXNEGRR7SR318212`：多帧上报，`gd_fc_stack.cell_count=432`，按 `200 + 200 + 32` 分片。
|
||||
|
||||
TDengine raw_frames 必须保持单帧事实，不跨帧合并。
|
||||
|
||||
Redis/MySQL 实时视图可以按 `protocol + vin` 增量合并，但不能让后来的 stack-only 帧覆盖标准车辆字段。
|
||||
|
||||
### JT808
|
||||
|
||||
JT808 主入口端口是 `808`，不再使用 `8089`。
|
||||
|
||||
解析重点：
|
||||
|
||||
- 包头手机号按 BCD 去前导 0。
|
||||
- `0x0100` 注册帧写 `jt808_registration`。
|
||||
- `0x0102` 鉴权帧尝试按 phone 查 registration/binding 关联 VIN。
|
||||
- `0x0200` 位置帧即使没有注册/鉴权，也会按 phone 降级查 `vehicle_identity_binding`，建立 session。
|
||||
- 位置附加信息 `0x01` 是 GPS 总里程，按差值用于每日里程。
|
||||
|
||||
## MySQL 身份表
|
||||
|
||||
`vehicle_identity_binding` 是外部维护事实表，服务只读，不允许运行时写入。
|
||||
|
||||
核心字段：
|
||||
|
||||
```text
|
||||
vin
|
||||
plate
|
||||
phone
|
||||
oem
|
||||
updated_at
|
||||
```
|
||||
|
||||
用途：
|
||||
|
||||
- JT808 用 phone 或 plate 找 VIN。
|
||||
- Realtime snapshot/location 用 VIN 反查 plate。
|
||||
- 后续导入 G7s、广安车联等来源时更新 phone/oem。
|
||||
|
||||
`jt808_registration` 是 808 注册鉴权运行状态表，以 phone 为主键，记录注册、鉴权、source_endpoint、关联 VIN 状态。
|
||||
|
||||
## 2026-07-03 数据维护记录
|
||||
|
||||
### G7s binding
|
||||
|
||||
源文件：
|
||||
|
||||
```text
|
||||
/Users/lingniu/Library/Mobile Documents/com~apple~CloudDocs/rsync/2026_07_02_17_11_37转发配置列表-1(2).xlsx
|
||||
```
|
||||
|
||||
处理规则：
|
||||
|
||||
1. 删除原有 G7/G7s 的 `phone/oem`。
|
||||
2. 使用文件中的 `手机号` 列更新。
|
||||
3. 冲突的不更新，其他更新。
|
||||
|
||||
执行结果：
|
||||
|
||||
- 清空旧 G7/G7s：`666` 行。
|
||||
- 更新安全 G7s：`561` 行。
|
||||
- 跳过：`72` 行。
|
||||
- 缺 plate 或 VIN：`14`
|
||||
- 与非 G7/G7s 绑定冲突：`58`
|
||||
|
||||
跳过清单：
|
||||
|
||||
```text
|
||||
outputs/g7s_binding_skipped_apply.csv
|
||||
```
|
||||
|
||||
### JT808 unknown VIN 导出
|
||||
|
||||
导出条件：
|
||||
|
||||
- `jt808_registration.vin` 为空、`unknown` 或 `unknow`
|
||||
- 且 `phone` 不存在于 `vehicle_identity_binding.phone`
|
||||
|
||||
结果：
|
||||
|
||||
- `30` 个 phone
|
||||
|
||||
导出文件：
|
||||
|
||||
```text
|
||||
outputs/jt808_unknown_vin_phone_not_in_binding.csv
|
||||
```
|
||||
|
||||
## 常用查询
|
||||
|
||||
历史 raw frame：
|
||||
|
||||
```bash
|
||||
curl -s 'http://115.29.187.205:20200/api/history/raw-frames?protocol=GB32960&vin=LB9A32A24R0LS1426&limit=1&includeFields=true'
|
||||
```
|
||||
|
||||
只取指定 parsed fields：
|
||||
|
||||
```bash
|
||||
curl -s 'http://115.29.187.205:20200/api/history/raw-frames?protocol=GB32960&vin=LB9A32A24R0LS1426&limit=1&includeFields=true&fields=gb32960.vehicle.speed_kmh&fields=gb32960.gd_fc_stack.stack_water_outlet_temp_c'
|
||||
```
|
||||
|
||||
808 源 IP：
|
||||
|
||||
```bash
|
||||
journalctl -u lingniu-go-gateway.service --since '10 minutes ago' --no-pager
|
||||
ss -tn sport = :808
|
||||
```
|
||||
|
||||
最近已观察到 808 源 IP：
|
||||
|
||||
```text
|
||||
115.159.85.149
|
||||
222.66.200.68
|
||||
122.152.221.156
|
||||
115.231.168.135
|
||||
```
|
||||
|
||||
## 部署步骤
|
||||
|
||||
本地构建：
|
||||
|
||||
```bash
|
||||
cd go/vehicle-gateway
|
||||
GOOS=linux GOARCH=amd64 CGO_ENABLED=0 go build -trimpath -ldflags='-s -w' -o /tmp/lingniu-go-deploy/gateway ./cmd/gateway
|
||||
GOOS=linux GOARCH=amd64 CGO_ENABLED=0 go build -trimpath -ldflags='-s -w' -o /tmp/lingniu-go-deploy/history-writer ./cmd/history-writer
|
||||
GOOS=linux GOARCH=amd64 CGO_ENABLED=0 go build -trimpath -ldflags='-s -w' -o /tmp/lingniu-go-deploy/realtime-api ./cmd/realtime-api
|
||||
GOOS=linux GOARCH=amd64 CGO_ENABLED=0 go build -trimpath -ldflags='-s -w' -o /tmp/lingniu-go-deploy/stat-writer ./cmd/stat-writer
|
||||
GOOS=linux GOARCH=amd64 CGO_ENABLED=0 go build -trimpath -ldflags='-s -w' -o /tmp/lingniu-go-deploy/nats-fast-writer ./cmd/nats-fast-writer
|
||||
GOOS=linux GOARCH=amd64 CGO_ENABLED=0 go build -trimpath -ldflags='-s -w' -o /tmp/lingniu-go-deploy/nats-kafka-bridge ./cmd/nats-kafka-bridge
|
||||
```
|
||||
|
||||
上传部署：
|
||||
|
||||
```bash
|
||||
release="name-$(date +%Y%m%d%H%M%S)"
|
||||
ssh root@115.29.187.205 "mkdir -p /opt/lingniu-go-native/releases/$release"
|
||||
scp /tmp/lingniu-go-deploy/* root@115.29.187.205:/opt/lingniu-go-native/releases/$release/
|
||||
ssh root@115.29.187.205 "chmod 755 /opt/lingniu-go-native/releases/$release/* && ln -sfn /opt/lingniu-go-native/releases/$release /opt/lingniu-go-native/current"
|
||||
```
|
||||
|
||||
部署后验证：
|
||||
|
||||
```bash
|
||||
ssh root@115.29.187.205 '
|
||||
systemctl is-active lingniu-go-gateway.service lingniu-go-nats-fast-writer.service lingniu-go-nats-kafka-bridge.service lingniu-go-history-writer.service lingniu-go-stat-writer.service lingniu-go-realtime-api.service
|
||||
for port in 20211 20212 20213 20214 20200; do curl -fsS "http://127.0.0.1:${port}/readyz"; echo; done
|
||||
journalctl -u lingniu-go-gateway.service --since "2 minutes ago" --no-pager | grep -Ei "error|failed|panic|fatal" || true
|
||||
'
|
||||
```
|
||||
|
||||
## 下次恢复上下文时先做
|
||||
|
||||
1. 读本文件。
|
||||
2. 读 `docs/ops/vehicle-ingest-runbook.md`。
|
||||
3. 读 `docs/architecture/production-data-plane-inventory.md`。
|
||||
4. 执行 `git status --short`。
|
||||
5. 如果涉及生产，先查 ECS 服务健康和最近日志。
|
||||
|
||||
Reference in New Issue
Block a user