202 lines
9.4 KiB
Markdown
202 lines
9.4 KiB
Markdown
# 当前 ECS Go 原生部署说明
|
||
|
||
更新时间：2026-07-02 01:30 CST
|
||
|
||
本文档记录当前 `lingniu-vehicle-ingest` 的生产运行面。当前 goal 的接入链路已经切到 Go 原生 systemd 部署，不再以 Docker/Portainer 作为生产运行方式。密钥只维护在 ECS 环境文件或受控凭据中，不写入 Git。
|
||
|
||
## 部署范围
|
||
|
||
| systemd 服务 | 二进制 | 说明 | 对外端口 |
|
||
| --- | --- | --- | --- |
|
||
| `lingniu-go-gateway.service` | `gateway` | GB32960 TCP、JT808 TCP、宇通 MQTT 接入，解析后写 Kafka RAW 和统一事件 | TCP `32960`、TCP `808` |
|
||
| `lingniu-go-history-writer.service` | `history-writer` | 消费 Kafka RAW，写 TDengine RAW、位置点、里程点核心表 | 无 HTTP |
|
||
| `lingniu-go-stat-writer.service` | `stat-writer` | 消费 Kafka RAW，按总里程差值法写 MySQL 每日指标 | 无 HTTP |
|
||
| `lingniu-go-realtime-api.service` | `realtime-api` | 消费统一事件写 Redis，并提供实时、历史、统计查询 API | HTTP `20210` |
|
||
|
||
信达 Push 已废弃，不参与当前 Go 生产链路。旧 Java/Docker 服务不应占用 `808`、`32960`、`20210`。
|
||
|
||
## 应用 ECS
|
||
|
||
| 项 | 值 |
|
||
| --- | --- |
|
||
| 公网 IP | `115.29.187.205` |
|
||
| 登录用户 | `root` |
|
||
| 部署目录 | `/opt/lingniu-go-native` |
|
||
| 当前 release | `/opt/lingniu-go-native/current` |
|
||
| 环境文件目录 | `/opt/lingniu-go-native/env` |
|
||
| systemd unit 目录 | `/etc/systemd/system` |
|
||
| 对外服务 | `32960`、`808`、`20210` |
|
||
|
||
运行目录结构：
|
||
|
||
```text
|
||
/opt/lingniu-go-native/
|
||
current -> /opt/lingniu-go-native/releases/<git-short-sha>
|
||
releases/<git-short-sha>/
|
||
gateway
|
||
history-writer
|
||
stat-writer
|
||
realtime-api
|
||
env/
|
||
gateway.env
|
||
history-writer.env
|
||
stat-writer.env
|
||
realtime-api.env
|
||
spool/gateway/
|
||
```
|
||
|
||
## 中间件
|
||
|
||
| 组件 | 内网地址 | 公网地址 | 用途 |
|
||
| --- | --- | --- | --- |
|
||
| Kafka | `172.17.111.56:9092` | `114.55.58.251:9092` | RAW 和统一事件消息总线 |
|
||
| TDengine | `172.17.111.57:6041` | `115.29.185.82:6041` | RAW、位置、里程点时序热存储 |
|
||
| MySQL RDS | `rm-bp179zbv481rnw3e2.mysql.rds.aliyuncs.com:3306` | `rm-bp179zbv481rnw3e2no.mysql.rds.aliyuncs.com:3306` | 身份映射、JT808 注册、每日指标 |
|
||
| Redis RDS | `r-bp1u741kij7e51i481.redis.rds.aliyuncs.com:6379` | 无 | 准实时车辆状态缓存 |
|
||
|
||
生产应用之间访问中间件优先使用内网地址。RDS 白名单需要允许应用 ECS 私网访问。
|
||
|
||
## Kafka Topic
|
||
|
||
| Topic | 生产者 | 消费者 | 内容 |
|
||
| --- | --- | --- | --- |
|
||
| `vehicle.raw.go.gb32960.v1` | `gateway` | `history-writer`、`stat-writer` | GB32960 完整 RAW 记录 |
|
||
| `vehicle.raw.go.jt808.v1` | `gateway` | `history-writer`、`stat-writer` | JT808 完整 RAW 记录 |
|
||
| `vehicle.raw.go.yutong-mqtt.v1` | `gateway` | `history-writer` | 宇通 MQTT 完整 RAW 记录 |
|
||
| `vehicle.event.go.unified.v1` | `gateway` | `realtime-api` | 三类协议统一事件，用于 Redis 实时状态 |
|
||
|
||
`gateway` 配置了 Kafka retry；生产 env 中启用 `KAFKA_SPOOL_DIR` 时，本地 spool 可在 Kafka 短暂不可用后回放。
|
||
|
||
Kafka 消费组：
|
||
|
||
| Consumer Group | Topic | 当前验证 |
|
||
| --- | --- | --- |
|
||
| `go-history-writer` | 三个 `vehicle.raw.go.*` topic | 2026-07-02 验证 lag 为 `0` |
|
||
| `go-stat-writer` | `vehicle.raw.go.gb32960.v1`、`vehicle.raw.go.jt808.v1` | 2026-07-02 验证 lag 为 `0` |
|
||
| `go-realtime-api` | `vehicle.event.go.unified.v1` | 2026-07-02 验证 lag 为 `0` |
|
||
|
||
## TDengine 数据层
|
||
|
||
默认数据库：`lingniu_vehicle_ts`
|
||
|
||
| 表 | 类型 | 说明 |
|
||
| --- | --- | --- |
|
||
| `raw_frames` | stable | 完整 RAW 帧，包含 `raw_hex`、`parsed_json`、`fields_json`、`source_endpoint` 和协议/车辆 tags |
|
||
| `vehicle_locations` | stable | 最小化位置历史，保留 `frame_id` 回链 RAW |
|
||
| `vehicle_mileage_points` | stable | 最小化总里程点，供历史查询和统计复核 |
|
||
|
||
设计原则：
|
||
|
||
- 完整结构化 payload 只落在 `raw_frames.parsed_json` / `fields_json`。
|
||
- `vehicle_locations` 和 `vehicle_mileage_points` 只保留核心查询字段，不重复保存完整 JSON。
|
||
- `telemetry_fields` 不由当前服务维护，字段配置解析由独立子服务处理。
|
||
- API 传入东八区时间时，服务会转换为 TDengine 当前存储使用的 UTC 字面量再查询。
|
||
|
||
## MySQL 数据层
|
||
|
||
默认业务库：`lingniu_vehicle_data`
|
||
|
||
| 表 | 维护方 | 说明 |
|
||
| --- | --- | --- |
|
||
| `vehicle_identity_binding` | 人工/外部主数据 | VIN 映射主表，用 `phone`、`device_id`、`plate` 降级定位 VIN |
|
||
| `jt808_registration` | `gateway` 自动写入 | 仅 JT808 注册和鉴权记录，以 `phone` 作为主键，记录设备、车牌、厂家、鉴权、来源端点 |
|
||
| `vehicle_daily_metric` | `stat-writer` 自动写入 | 每日指标表，保存 `daily_mileage_km` 和 `daily_total_mileage_km` |
|
||
|
||
每日里程算法：
|
||
|
||
```text
|
||
daily_mileage_km = 当日最大 total_mileage_km - 当日最小 total_mileage_km
|
||
daily_total_mileage_km = 当日最大 total_mileage_km
|
||
```
|
||
|
||
统计按 `Asia/Shanghai` 自然日归档，当前支持 GB32960、JT808、宇通 MQTT 中能解析出 `total_mileage_km` 的数据。JT808 的总里程来自位置附加信息 `0x01`，单位按协议转换为 km；宇通 MQTT 的 `TOTAL_MILEAGE` 原始单位为米，入库前转换为 km。
|
||
|
||
## Redis 实时层
|
||
|
||
`realtime-api` 消费 `vehicle.event.go.unified.v1`，按 VIN/vehicle key 维护准实时快照。主要能力：
|
||
|
||
| API | 说明 |
|
||
| --- | --- |
|
||
| `/api/realtime/vehicles/{vin}` | 查询车辆合并实时快照 |
|
||
| `/api/realtime/vehicles/{vin}/online` | 查询车辆是否在线 |
|
||
| `/api/realtime/vehicles/{vin}/protocols/{protocol}` | 查询指定协议的实时快照 |
|
||
|
||
Redis 只作为实时缓存；历史和统计以 TDengine/MySQL 为准。
|
||
|
||
## API 入口
|
||
|
||
公网基础地址：
|
||
|
||
```text
|
||
http://115.29.187.205:20210
|
||
```
|
||
|
||
常用查询：
|
||
|
||
```bash
|
||
curl -sS 'http://115.29.187.205:20210/api/history/raw-frames?protocol=GB32960&dateFrom=2026-07-02T00:00:00%2B08:00&dateTo=2026-07-03T00:00:00%2B08:00&limit=1'
|
||
curl -sS 'http://115.29.187.205:20210/api/history/raw-frames?protocol=JT808&dateFrom=2026-07-02T00:00:00%2B08:00&dateTo=2026-07-03T00:00:00%2B08:00&limit=1'
|
||
curl -sS 'http://115.29.187.205:20210/api/history/raw-frames?protocol=YUTONG_MQTT&dateFrom=2026-07-02T00:00:00%2B08:00&dateTo=2026-07-03T00:00:00%2B08:00&limit=1'
|
||
curl -sS 'http://115.29.187.205:20210/api/history/locations?protocol=JT808&dateFrom=2026-07-02T00:00:00%2B08:00&dateTo=2026-07-03T00:00:00%2B08:00&limit=1'
|
||
curl -sS 'http://115.29.187.205:20210/api/history/mileage-points?protocol=JT808&dateFrom=2026-07-02T00:00:00%2B08:00&dateTo=2026-07-03T00:00:00%2B08:00&limit=1'
|
||
curl -sS 'http://115.29.187.205:20210/api/stats/daily-metrics?dateFrom=2026-07-02&dateTo=2026-07-02&limit=20'
|
||
```
|
||
|
||
分页参数统一使用 `limit` 和 `offset`。
|
||
|
||
部署后推荐直接运行 Go 原生生产 smoke：
|
||
|
||
```bash
|
||
python3 tools/go_native_prod_smoke.py --date 2026-07-02 --timeout 8
|
||
```
|
||
|
||
该脚本通过 `20210` HTTP API 验证 GB32960、JT808、宇通 MQTT 的 RAW 查询，三类协议的位置和里程点查询，GB32960/JT808 的 `daily_mileage_km`、`daily_total_mileage_km` 统计，以及三类协议的 Redis realtime snapshot、online、protocol 查询。默认检查今天东八区数据时，还会要求三类 RAW 最新样本不超过 15 分钟，避免旧数据误判为接收正常；任一检查不达标时退出码为非 0。
|
||
|
||
## 部署命令
|
||
|
||
在开发机或 CI 构建 Linux amd64：
|
||
|
||
```bash
|
||
cd go/vehicle-gateway
|
||
CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go build -trimpath -ldflags='-s -w' -o /tmp/lingniu-go-native/gateway ./cmd/gateway
|
||
CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go build -trimpath -ldflags='-s -w' -o /tmp/lingniu-go-native/history-writer ./cmd/history-writer
|
||
CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go build -trimpath -ldflags='-s -w' -o /tmp/lingniu-go-native/stat-writer ./cmd/stat-writer
|
||
CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go build -trimpath -ldflags='-s -w' -o /tmp/lingniu-go-native/realtime-api ./cmd/realtime-api
|
||
```
|
||
|
||
上传到 ECS 新 release 目录后切换：
|
||
|
||
```bash
|
||
systemctl daemon-reload
|
||
systemctl enable lingniu-go-gateway lingniu-go-history-writer lingniu-go-stat-writer lingniu-go-realtime-api
|
||
systemctl restart lingniu-go-gateway lingniu-go-history-writer lingniu-go-stat-writer lingniu-go-realtime-api
|
||
```
|
||
|
||
只更新查询 API 时，可以仅替换 `/opt/lingniu-go-native/current/realtime-api` 并重启：
|
||
|
||
```bash
|
||
systemctl restart lingniu-go-realtime-api.service
|
||
```
|
||
|
||
## 运行检查
|
||
|
||
```bash
|
||
systemctl is-active lingniu-go-gateway lingniu-go-history-writer lingniu-go-stat-writer lingniu-go-realtime-api
|
||
ss -lntp | egrep ':(808|32960|20210)\b'
|
||
journalctl -u lingniu-go-gateway --since '5 minutes ago' --no-pager
|
||
journalctl -u lingniu-go-history-writer --since '5 minutes ago' --no-pager
|
||
journalctl -u lingniu-go-stat-writer --since '5 minutes ago' --no-pager
|
||
journalctl -u lingniu-go-realtime-api --since '5 minutes ago' --no-pager
|
||
```
|
||
|
||
2026-07-02 01:30 CST 已验证：
|
||
|
||
- 四个 Go systemd 服务均为 `active`。
|
||
- `gateway` 正在监听 `32960` 和 `808`。
|
||
- `realtime-api` 正在监听 `20210`。
|
||
- GB32960、JT808、YUTONG_MQTT 的东八区 RAW 查询均可命中生产数据。
|
||
- GB32960、JT808、YUTONG_MQTT 的最新 RAW 样本均在 15 分钟内。
|
||
- GB32960、JT808、YUTONG_MQTT 的 `vehicle_locations`、`vehicle_mileage_points` 查询均可命中生产数据。
|
||
- `vehicle_daily_metric` 可查到 `daily_mileage_km` 和 `daily_total_mileage_km`。
|
||
- Redis realtime 可查到 GB32960、JT808、YUTONG_MQTT 的在线状态和协议快照。
|