752 lines
27 KiB
HTML
752 lines
27 KiB
HTML
<!doctype html>
|
||
<html lang="zh-CN">
|
||
<head>
|
||
<meta charset="utf-8" />
|
||
<meta name="viewport" content="width=device-width, initial-scale=1" />
|
||
<title>Lingniu Vehicle Ingest Go Version Data Flow</title>
|
||
<style>
|
||
:root {
|
||
--bg: #f6f7f9;
|
||
--ink: #17202a;
|
||
--muted: #5f6b7a;
|
||
--line: #ccd3dd;
|
||
--panel: #ffffff;
|
||
--blue: #1f6feb;
|
||
--green: #16803c;
|
||
--amber: #a15c00;
|
||
--red: #c23131;
|
||
--purple: #7c3aed;
|
||
--teal: #087f8c;
|
||
--shadow: 0 10px 30px rgba(25, 33, 45, .08);
|
||
}
|
||
|
||
* { box-sizing: border-box; }
|
||
|
||
body {
|
||
margin: 0;
|
||
color: var(--ink);
|
||
background: var(--bg);
|
||
font: 14px/1.55 -apple-system, BlinkMacSystemFont, "Segoe UI", Arial, "PingFang SC", "Microsoft YaHei", sans-serif;
|
||
}
|
||
|
||
header {
|
||
padding: 28px 34px 22px;
|
||
background: #111827;
|
||
color: white;
|
||
}
|
||
|
||
h1 {
|
||
margin: 0 0 8px;
|
||
font-size: 28px;
|
||
line-height: 1.2;
|
||
letter-spacing: 0;
|
||
}
|
||
|
||
.subtitle {
|
||
max-width: 1180px;
|
||
color: #d1d5db;
|
||
font-size: 15px;
|
||
}
|
||
|
||
main {
|
||
width: min(1480px, calc(100vw - 36px));
|
||
margin: 22px auto 48px;
|
||
}
|
||
|
||
h2 {
|
||
margin: 0 0 14px;
|
||
font-size: 20px;
|
||
line-height: 1.25;
|
||
}
|
||
|
||
h3 {
|
||
margin: 0 0 10px;
|
||
font-size: 15px;
|
||
}
|
||
|
||
.section {
|
||
margin: 18px 0;
|
||
padding: 20px;
|
||
background: var(--panel);
|
||
border: 1px solid #e4e8ef;
|
||
border-radius: 8px;
|
||
box-shadow: var(--shadow);
|
||
}
|
||
|
||
.legend {
|
||
display: flex;
|
||
flex-wrap: wrap;
|
||
gap: 10px;
|
||
margin-top: 16px;
|
||
}
|
||
|
||
.tag {
|
||
display: inline-flex;
|
||
align-items: center;
|
||
gap: 7px;
|
||
padding: 4px 8px;
|
||
border: 1px solid #d8dee8;
|
||
border-radius: 6px;
|
||
background: #fff;
|
||
color: var(--muted);
|
||
font-size: 12px;
|
||
}
|
||
|
||
.dot {
|
||
width: 10px;
|
||
height: 10px;
|
||
border-radius: 50%;
|
||
background: var(--blue);
|
||
}
|
||
|
||
.dot.source { background: var(--amber); }
|
||
.dot.gateway { background: var(--blue); }
|
||
.dot.bus { background: var(--purple); }
|
||
.dot.store { background: var(--green); }
|
||
.dot.api { background: var(--teal); }
|
||
.dot.warning { background: var(--red); }
|
||
|
||
.flow {
|
||
display: grid;
|
||
grid-template-columns: 1.15fr 1.35fr 1.15fr 1.2fr 1.4fr 1.1fr;
|
||
gap: 14px;
|
||
align-items: stretch;
|
||
overflow-x: auto;
|
||
padding-bottom: 8px;
|
||
}
|
||
|
||
.lane {
|
||
min-width: 190px;
|
||
display: flex;
|
||
flex-direction: column;
|
||
gap: 12px;
|
||
}
|
||
|
||
.lane-title {
|
||
padding: 8px 10px;
|
||
border-radius: 6px;
|
||
color: white;
|
||
font-weight: 700;
|
||
text-align: center;
|
||
background: #374151;
|
||
}
|
||
|
||
.node {
|
||
position: relative;
|
||
min-height: 88px;
|
||
padding: 13px 13px 12px;
|
||
border: 1px solid #d9e0ea;
|
||
border-left: 5px solid var(--blue);
|
||
border-radius: 8px;
|
||
background: #fff;
|
||
}
|
||
|
||
.node.source { border-left-color: var(--amber); }
|
||
.node.gateway { border-left-color: var(--blue); }
|
||
.node.bus { border-left-color: var(--purple); }
|
||
.node.store { border-left-color: var(--green); }
|
||
.node.api { border-left-color: var(--teal); }
|
||
.node.warning { border-left-color: var(--red); }
|
||
|
||
.node strong {
|
||
display: block;
|
||
margin-bottom: 5px;
|
||
font-size: 14px;
|
||
}
|
||
|
||
.node p {
|
||
margin: 0;
|
||
color: var(--muted);
|
||
font-size: 12px;
|
||
}
|
||
|
||
.arrow {
|
||
display: flex;
|
||
align-items: center;
|
||
justify-content: center;
|
||
height: 28px;
|
||
color: var(--muted);
|
||
font-size: 12px;
|
||
white-space: nowrap;
|
||
}
|
||
|
||
.arrow:before, .arrow:after {
|
||
content: "";
|
||
height: 1px;
|
||
background: var(--line);
|
||
flex: 1;
|
||
margin: 0 8px;
|
||
}
|
||
|
||
.arrow:after {
|
||
max-width: 16px;
|
||
height: 0;
|
||
border-top: 5px solid transparent;
|
||
border-bottom: 5px solid transparent;
|
||
border-left: 8px solid var(--line);
|
||
background: transparent;
|
||
margin-left: 0;
|
||
}
|
||
|
||
.grid-2 {
|
||
display: grid;
|
||
grid-template-columns: repeat(2, minmax(0, 1fr));
|
||
gap: 14px;
|
||
}
|
||
|
||
.grid-3 {
|
||
display: grid;
|
||
grid-template-columns: repeat(3, minmax(0, 1fr));
|
||
gap: 14px;
|
||
}
|
||
|
||
table {
|
||
width: 100%;
|
||
border-collapse: collapse;
|
||
overflow: hidden;
|
||
border: 1px solid #e1e6ef;
|
||
border-radius: 8px;
|
||
background: white;
|
||
}
|
||
|
||
th, td {
|
||
padding: 10px 11px;
|
||
border-bottom: 1px solid #edf1f6;
|
||
text-align: left;
|
||
vertical-align: top;
|
||
font-size: 13px;
|
||
}
|
||
|
||
th {
|
||
background: #f0f4f8;
|
||
color: #263241;
|
||
font-weight: 700;
|
||
}
|
||
|
||
tr:last-child td { border-bottom: 0; }
|
||
|
||
code {
|
||
padding: 1px 5px;
|
||
border-radius: 4px;
|
||
background: #eef2f7;
|
||
color: #1f2937;
|
||
font-family: ui-monospace, SFMono-Regular, Menlo, Consolas, monospace;
|
||
font-size: 12px;
|
||
}
|
||
|
||
.pill {
|
||
display: inline-block;
|
||
margin: 2px 4px 2px 0;
|
||
padding: 2px 7px;
|
||
border-radius: 999px;
|
||
background: #eef2ff;
|
||
color: #3730a3;
|
||
font-size: 12px;
|
||
white-space: nowrap;
|
||
}
|
||
|
||
.note {
|
||
padding: 11px 12px;
|
||
border-left: 4px solid var(--amber);
|
||
background: #fff7ed;
|
||
color: #61410b;
|
||
border-radius: 6px;
|
||
}
|
||
|
||
.ok {
|
||
padding: 11px 12px;
|
||
border-left: 4px solid var(--green);
|
||
background: #effaf1;
|
||
color: #164b25;
|
||
border-radius: 6px;
|
||
}
|
||
|
||
.mini-flow {
|
||
display: grid;
|
||
grid-template-columns: repeat(7, minmax(128px, 1fr));
|
||
gap: 8px;
|
||
overflow-x: auto;
|
||
padding-bottom: 8px;
|
||
}
|
||
|
||
.mini-step {
|
||
min-height: 82px;
|
||
padding: 10px;
|
||
border: 1px solid #dfe6f1;
|
||
border-radius: 8px;
|
||
background: #fff;
|
||
}
|
||
|
||
.mini-step b {
|
||
display: block;
|
||
margin-bottom: 4px;
|
||
color: #111827;
|
||
font-size: 13px;
|
||
}
|
||
|
||
.mini-step span {
|
||
color: var(--muted);
|
||
font-size: 12px;
|
||
}
|
||
|
||
.footer {
|
||
color: var(--muted);
|
||
font-size: 12px;
|
||
text-align: center;
|
||
padding: 20px;
|
||
}
|
||
|
||
@media (max-width: 980px) {
|
||
.grid-2, .grid-3 { grid-template-columns: 1fr; }
|
||
header { padding: 22px 18px; }
|
||
main { width: calc(100vw - 20px); }
|
||
.section { padding: 14px; }
|
||
}
|
||
</style>
|
||
</head>
|
||
<body>
|
||
<header>
|
||
<h1>Go 版本车辆数据处理与流转全图</h1>
|
||
<div class="subtitle">
|
||
当前 Go 架构以 gateway 做高性能协议接入，NATS JetStream 做入口缓冲和解耦，nats-kafka-bridge 保持现有 Kafka 下游兼容，最终进入 TDengine、Redis、MySQL，并由 realtime-api 对外提供查询能力。
|
||
</div>
|
||
<div class="legend">
|
||
<span class="tag"><span class="dot source"></span>外部数据源</span>
|
||
<span class="tag"><span class="dot gateway"></span>Go 接入与解析</span>
|
||
<span class="tag"><span class="dot bus"></span>NATS / Kafka 解耦</span>
|
||
<span class="tag"><span class="dot store"></span>存储</span>
|
||
<span class="tag"><span class="dot api"></span>查询 API</span>
|
||
<span class="tag"><span class="dot warning"></span>可靠性边界</span>
|
||
</div>
|
||
</header>
|
||
|
||
<main>
|
||
<section class="section">
|
||
<h2>1. 总览图：从车辆平台到 API</h2>
|
||
<div class="flow">
|
||
<div class="lane">
|
||
<div class="lane-title">外部输入</div>
|
||
<div class="node source">
|
||
<strong>GB/T 32960 平台</strong>
|
||
<p>TCP 连接到 ECS <code>:32960</code>，包含登录、登出、实时信息、补发等帧。</p>
|
||
</div>
|
||
<div class="node source">
|
||
<strong>JT/T 808 平台</strong>
|
||
<p>TCP 连接到 ECS <code>:808</code>，包含注册、鉴权、位置上报 <code>0x0200</code> 等。</p>
|
||
</div>
|
||
<div class="node source">
|
||
<strong>宇通 MQTT</strong>
|
||
<p>gateway 作为 MQTT client 订阅 <code>/ytforward/shln/+</code>。</p>
|
||
</div>
|
||
</div>
|
||
|
||
<div class="lane">
|
||
<div class="lane-title">gateway 接入层</div>
|
||
<div class="node gateway">
|
||
<strong>TCPServer / MQTTClient</strong>
|
||
<p>读取连接、拆包、处理半包/粘包、按协议路由到解析器。</p>
|
||
</div>
|
||
<div class="node gateway">
|
||
<strong>协议解析器</strong>
|
||
<p><code>gb32960</code>、<code>jt808</code>、<code>yutongmqtt</code> 解析 RAW，抽取核心 fields。</p>
|
||
</div>
|
||
<div class="node gateway">
|
||
<strong>身份解析</strong>
|
||
<p>通过 MySQL <code>vehicle_identity_binding</code> 将 phone/device/plate 尽量映射到 VIN。</p>
|
||
</div>
|
||
<div class="node gateway">
|
||
<strong>协议响应</strong>
|
||
<p>32960/808 需要应答时，在成功发布后写 ACK。</p>
|
||
</div>
|
||
</div>
|
||
|
||
<div class="lane">
|
||
<div class="lane-title">统一消息</div>
|
||
<div class="node gateway">
|
||
<strong>FrameEnvelope</strong>
|
||
<p>统一封装：protocol、message_id、vin、phone、device_id、source_endpoint、raw、parsed、fields。</p>
|
||
</div>
|
||
<div class="node warning">
|
||
<strong>StableEventID</strong>
|
||
<p>基于协议、消息、车辆键、序号、时间、raw 生成稳定 ID，用于去重和追踪。</p>
|
||
</div>
|
||
</div>
|
||
|
||
<div class="lane">
|
||
<div class="lane-title">入口解耦</div>
|
||
<div class="node bus">
|
||
<strong>NATS JetStream</strong>
|
||
<p>gateway 发布到 <code>VEHICLE_INGEST</code> stream，文件存储，24 小时保留。</p>
|
||
</div>
|
||
<div class="node bus">
|
||
<strong>Subjects</strong>
|
||
<p><code>vehicle.raw.go.gb32960.v1</code><br><code>vehicle.raw.go.jt808.v1</code><br><code>vehicle.raw.go.yutong-mqtt.v1</code></p>
|
||
</div>
|
||
<div class="node warning">
|
||
<strong>Async + Retry</strong>
|
||
<p>gateway 内部异步队列发布 NATS，降低协议连接被 Kafka 慢写拖住的风险。</p>
|
||
</div>
|
||
</div>
|
||
|
||
<div class="lane">
|
||
<div class="lane-title">Kafka 兼容层</div>
|
||
<div class="node bus">
|
||
<strong>nats-kafka-bridge</strong>
|
||
<p>durable pull consumer 批量拉 NATS，写 Kafka 成功后才 ACK NATS。</p>
|
||
</div>
|
||
<div class="node bus">
|
||
<strong>Kafka Topics</strong>
|
||
<p>与 NATS subject 同名：三个协议 RAW topic。<code>vehicle.event.go.unified.v1</code> 仅作为显式兼容开关保留。</p>
|
||
</div>
|
||
<div class="node warning">
|
||
<strong>失败语义</strong>
|
||
<p>Kafka 写失败不 ACK，NATS 保留消息等待重试；未知 subject 不 ACK，防止静默丢数据。</p>
|
||
</div>
|
||
</div>
|
||
|
||
<div class="lane">
|
||
<div class="lane-title">落库与查询</div>
|
||
<div class="node store">
|
||
<strong>history-writer</strong>
|
||
<p>消费 RAW topic，写 TDengine：raw_frames、payload_chunks、locations。</p>
|
||
</div>
|
||
<div class="node store">
|
||
<strong>stat-writer</strong>
|
||
<p>消费 RAW topic，根据总里程差值写 MySQL 每日指标。</p>
|
||
</div>
|
||
<div class="node store">
|
||
<strong>realtime-api 消费器</strong>
|
||
<p>消费三个 RAW topic，写 Redis 在线状态、实时合并快照、各协议 realtime-raw。</p>
|
||
</div>
|
||
<div class="node api">
|
||
<strong>HTTP API</strong>
|
||
<p>realtime-api 暴露实时、历史、统计查询接口，端口 <code>:20200</code>。</p>
|
||
</div>
|
||
</div>
|
||
</div>
|
||
</section>
|
||
|
||
<section class="section">
|
||
<h2>2. 单条报文的完整生命周期</h2>
|
||
<div class="mini-flow">
|
||
<div class="mini-step"><b>1. 到达</b><span>32960/808 TCP 报文或 MQTT 消息到 gateway。</span></div>
|
||
<div class="mini-step"><b>2. 拆包</b><span>TCP 按协议提取完整帧；MQTT 直接按消息处理。</span></div>
|
||
<div class="mini-step"><b>3. 解析</b><span>生成 parsed 全量结构化数据和 fields 核心字段。</span></div>
|
||
<div class="mini-step"><b>4. 绑定 VIN</b><span>根据 VIN/phone/device_id/plate 生成 vehicle_key。</span></div>
|
||
<div class="mini-step"><b>5. 发 NATS</b><span>写协议 RAW subject；unified 只在兼容开关打开时写。</span></div>
|
||
<div class="mini-step"><b>6. 桥接 Kafka</b><span>bridge 写 Kafka 成功后 ACK NATS。</span></div>
|
||
<div class="mini-step"><b>7. 多路消费</b><span>历史、实时、统计各自消费 Kafka，互不阻塞。</span></div>
|
||
</div>
|
||
</section>
|
||
|
||
<section class="section">
|
||
<h2>3. FrameEnvelope：Go 版本内部统一数据格式</h2>
|
||
<div class="grid-2">
|
||
<div>
|
||
<table>
|
||
<thead>
|
||
<tr><th>字段</th><th>含义</th><th>来源</th></tr>
|
||
</thead>
|
||
<tbody>
|
||
<tr><td><code>event_id</code></td><td>稳定事件 ID</td><td>已有则保留，否则按协议/消息/车辆/时间/raw 计算</td></tr>
|
||
<tr><td><code>protocol</code></td><td>协议</td><td><code>GB32960</code>、<code>JT808</code>、<code>YUTONG_MQTT</code></td></tr>
|
||
<tr><td><code>message_id</code></td><td>协议消息类型</td><td>如 808 <code>0x0200</code>，32960 命令标识等</td></tr>
|
||
<tr><td><code>vin</code></td><td>VIN</td><td>协议原始字段或 MySQL binding 反查</td></tr>
|
||
<tr><td><code>phone</code></td><td>808 终端手机号</td><td>808 消息头 BCD 解析后去前导 0</td></tr>
|
||
<tr><td><code>device_id</code></td><td>设备标识</td><td>协议内设备号、MQTT 设备字段等</td></tr>
|
||
<tr><td><code>plate</code></td><td>车牌</td><td>注册帧或 binding 表</td></tr>
|
||
<tr><td><code>source_endpoint</code></td><td>来源地址</td><td>TCP remote ip:port 或 MQTT endpoint/topic</td></tr>
|
||
</tbody>
|
||
</table>
|
||
</div>
|
||
<div>
|
||
<table>
|
||
<thead>
|
||
<tr><th>字段</th><th>含义</th><th>去向</th></tr>
|
||
</thead>
|
||
<tbody>
|
||
<tr><td><code>raw_hex</code> / <code>raw_text</code></td><td>原始报文</td><td>TDengine <code>raw_frames</code>，超长进入 chunks</td></tr>
|
||
<tr><td><code>parsed</code></td><td>全量解析 JSON</td><td>TDengine <code>parsed_json</code>；Redis <code>realtime-raw</code></td></tr>
|
||
<tr><td><code>fields</code></td><td>最小核心字段</td><td>TDengine 位置/里程点；Redis 合并快照；统计计算</td></tr>
|
||
<tr><td><code>event_time_ms</code></td><td>车端事件时间</td><td>TDengine 主时间；Redis field time</td></tr>
|
||
<tr><td><code>received_at_ms</code></td><td>平台接收时间</td><td>延迟分析、在线状态 last_seen</td></tr>
|
||
<tr><td><code>parse_status</code></td><td>解析状态</td><td><code>OK</code>、<code>PARTIAL</code>、<code>BAD_FRAME</code></td></tr>
|
||
<tr><td><code>parse_error</code></td><td>解析错误</td><td>RAW 查询排查使用</td></tr>
|
||
<tr><td><code>vehicle_key</code></td><td>车辆主键</td><td>优先 VIN；否则 <code>PROTOCOL:phone/device</code></td></tr>
|
||
</tbody>
|
||
</table>
|
||
</div>
|
||
</div>
|
||
</section>
|
||
|
||
<section class="section">
|
||
<h2>4. 三个协议的处理差异</h2>
|
||
<table>
|
||
<thead>
|
||
<tr>
|
||
<th>协议</th>
|
||
<th>接入方式</th>
|
||
<th>身份主线</th>
|
||
<th>RAW 数据</th>
|
||
<th>核心 fields</th>
|
||
<th>特殊动作</th>
|
||
</tr>
|
||
</thead>
|
||
<tbody>
|
||
<tr>
|
||
<td><strong>GB32960</strong></td>
|
||
<td>gateway TCP <code>:32960</code></td>
|
||
<td>报文本身通常有 VIN；平台账号用于登录鉴权</td>
|
||
<td>完整 parsed JSON 写 <code>raw_frames.parsed_json</code></td>
|
||
<td>速度、经纬度、SOC、累计里程等按帧类型抽取</td>
|
||
<td>登录/数据帧需要平台应答，成功发布后 ACK</td>
|
||
</tr>
|
||
<tr>
|
||
<td><strong>JT808</strong></td>
|
||
<td>gateway TCP <code>:808</code></td>
|
||
<td>消息头 phone 为主；注册/鉴权/位置帧结合 binding 找 VIN</td>
|
||
<td>完整 parsed JSON 写 <code>raw_frames.parsed_json</code></td>
|
||
<td>位置、速度、方向、状态、报警、GPS 总里程</td>
|
||
<td>注册/鉴权/通用应答；每日里程按总里程差值统计</td>
|
||
</tr>
|
||
<tr>
|
||
<td><strong>YUTONG_MQTT</strong></td>
|
||
<td>gateway MQTT client 订阅</td>
|
||
<td>MQTT payload 中 VIN/设备字段；source endpoint 为 MQTT endpoint/topic</td>
|
||
<td>完整 parsed JSON 写 <code>raw_frames.parsed_json</code></td>
|
||
<td>速度、经纬度、总里程、SOC 等</td>
|
||
<td>不需要 TCP ACK；实时多帧在 Redis 按协议合并</td>
|
||
</tr>
|
||
</tbody>
|
||
</table>
|
||
</section>
|
||
|
||
<section class="section">
|
||
<h2>5. NATS + Kafka 的职责分工</h2>
|
||
<div class="grid-3">
|
||
<div class="node bus">
|
||
<strong>NATS JetStream：入口缓冲层</strong>
|
||
<p>部署在 Kafka ECS，Docker Compose 管理；监听内网 <code>172.17.111.56:4222</code>，监控 <code>172.17.111.56:8222</code>。gateway 只需要把事件快速、可靠地写入 NATS。</p>
|
||
</div>
|
||
<div class="node bus">
|
||
<strong>nats-kafka-bridge：可靠桥接层</strong>
|
||
<p>使用 durable pull consumer；批量写 Kafka；只有 Kafka 写入成功才 ACK NATS。Kafka 短暂异常时，NATS 消息保留并重试。</p>
|
||
</div>
|
||
<div class="node bus">
|
||
<strong>Kafka：业务消费层</strong>
|
||
<p>保持现有 topic 和消费者模型，history/realtime/stat 不直接依赖 gateway，也不直接依赖 NATS API。</p>
|
||
</div>
|
||
</div>
|
||
<div class="note" style="margin-top:14px;">
|
||
当前阶段没有让下游直接消费 NATS，这是为了最小化切换风险：入口换成 NATS 后，下游仍沿用 Kafka。后续如果要进一步实时化，可以让实时缓存或控制命令直接使用 NATS subject。
|
||
</div>
|
||
</section>
|
||
|
||
<section class="section">
|
||
<h2>6. 存储模型：哪些数据进入哪里</h2>
|
||
<table>
|
||
<thead>
|
||
<tr>
|
||
<th>存储</th>
|
||
<th>表/Key</th>
|
||
<th>写入方</th>
|
||
<th>内容</th>
|
||
<th>用途</th>
|
||
</tr>
|
||
</thead>
|
||
<tbody>
|
||
<tr>
|
||
<td><strong>TDengine</strong></td>
|
||
<td><code>raw_frames</code></td>
|
||
<td>history-writer</td>
|
||
<td>原始报文、完整 parsed JSON、parse status、source endpoint</td>
|
||
<td>RAW 查询、追溯、排查解析问题</td>
|
||
</tr>
|
||
<tr>
|
||
<td><strong>TDengine</strong></td>
|
||
<td><code>raw_frame_payload_chunks</code></td>
|
||
<td>history-writer</td>
|
||
<td>超长 raw/parsed payload 分片</td>
|
||
<td>避免 BINARY 长度限制导致完整 JSON 丢失</td>
|
||
</tr>
|
||
<tr>
|
||
<td><strong>TDengine</strong></td>
|
||
<td><code>vehicle_locations</code></td>
|
||
<td>history-writer</td>
|
||
<td>经纬度、速度、方向、状态、报警、总里程等核心字段</td>
|
||
<td>高频历史位置分页查询</td>
|
||
</tr>
|
||
<tr>
|
||
<td><strong>Redis</strong></td>
|
||
<td><code>vehicle:latest:{vin}</code></td>
|
||
<td>realtime-api Kafka consumer</td>
|
||
<td>跨协议合并后的实时字段快照</td>
|
||
<td>查 VIN 是否在线、查 VIN 实时数据</td>
|
||
</tr>
|
||
<tr>
|
||
<td><strong>Redis</strong></td>
|
||
<td><code>vehicle:latest:{vin}:{protocol}</code></td>
|
||
<td>realtime-api Kafka consumer</td>
|
||
<td>单协议实时轻量快照</td>
|
||
<td>区分 32960/808/MQTT 的实时状态</td>
|
||
</tr>
|
||
<tr>
|
||
<td><strong>Redis</strong></td>
|
||
<td><code>vehicle:realtime-raw:{protocol}:{vin}</code></td>
|
||
<td>realtime-api Kafka consumer</td>
|
||
<td>单协议最新 parsed 全量字段</td>
|
||
<td>实时 RAW 字段查看，不走 TDengine 历史扫描</td>
|
||
</tr>
|
||
<tr>
|
||
<td><strong>Redis</strong></td>
|
||
<td><code>vehicle:online:{vin}</code>、<code>vehicle:last_seen</code></td>
|
||
<td>realtime-api Kafka consumer</td>
|
||
<td>在线状态、最后接收时间、协议列表</td>
|
||
<td>在线判断和近期车辆列表</td>
|
||
</tr>
|
||
<tr>
|
||
<td><strong>MySQL</strong></td>
|
||
<td><code>vehicle_identity_binding</code></td>
|
||
<td>人工/导入维护，gateway 读取</td>
|
||
<td>phone/device/plate 到 VIN 的映射</td>
|
||
<td>808 等缺 VIN 协议反查正确 VIN</td>
|
||
</tr>
|
||
<tr>
|
||
<td><strong>MySQL</strong></td>
|
||
<td><code>jt808_registration</code></td>
|
||
<td>gateway</td>
|
||
<td>808 注册、鉴权、首次/最新上报、phone、device、plate、vin 解析结果</td>
|
||
<td>定位哪些手机号没有映射 VIN，排查注册鉴权</td>
|
||
</tr>
|
||
<tr>
|
||
<td><strong>MySQL</strong></td>
|
||
<td><code>vehicle_daily_mileage</code></td>
|
||
<td>stat-writer</td>
|
||
<td>每日里程，按首末总里程差值计算，保留样本数</td>
|
||
<td>统计查询</td>
|
||
</tr>
|
||
</tbody>
|
||
</table>
|
||
</section>
|
||
|
||
<section class="section">
|
||
<h2>7. 查询入口</h2>
|
||
<div class="grid-2">
|
||
<div>
|
||
<h3>实时查询</h3>
|
||
<table>
|
||
<thead><tr><th>接口类型</th><th>数据源</th><th>说明</th></tr></thead>
|
||
<tbody>
|
||
<tr><td>VIN 是否在线</td><td>Redis online key</td><td>TTL 内有数据即在线，默认 TTL 600 秒。</td></tr>
|
||
<tr><td>VIN 实时数据</td><td>Redis latest snapshot</td><td>跨协议合并核心 fields；完整协议字段通过 realtime-raw 查询。</td></tr>
|
||
<tr><td>单协议实时 RAW</td><td>Redis realtime-raw</td><td>查看某 VIN 在某协议下最新 parsed 全量字段。</td></tr>
|
||
</tbody>
|
||
</table>
|
||
</div>
|
||
<div>
|
||
<h3>历史查询</h3>
|
||
<table>
|
||
<thead><tr><th>接口类型</th><th>数据源</th><th>说明</th></tr></thead>
|
||
<tbody>
|
||
<tr><td>RAW 帧查询</td><td>TDengine raw_frames + chunks</td><td>按协议、VIN/phone、时间、消息类型分页。</td></tr>
|
||
<tr><td>位置/总里程历史</td><td>TDengine vehicle_locations</td><td>高频位置分页查询，包含 total_mileage_km，避免重复里程点接口。</td></tr>
|
||
<tr><td>每日里程</td><td>MySQL vehicle_daily_mileage</td><td>按日期、协议查询首末总里程差值结果。</td></tr>
|
||
</tbody>
|
||
</table>
|
||
</div>
|
||
</div>
|
||
</section>
|
||
|
||
<section class="section">
|
||
<h2>8. 生产部署视图</h2>
|
||
<table>
|
||
<thead>
|
||
<tr><th>节点</th><th>组件</th><th>地址/端口</th><th>说明</th></tr>
|
||
</thead>
|
||
<tbody>
|
||
<tr>
|
||
<td>Gateway ECS</td>
|
||
<td><code>lingniu-go-gateway</code></td>
|
||
<td>公网 <code>115.29.187.205</code>；生产端口 <code>32960</code>、<code>808</code></td>
|
||
<td>承接外部平台真实连接，发布到 NATS。</td>
|
||
</tr>
|
||
<tr>
|
||
<td>Gateway ECS</td>
|
||
<td><code>lingniu-go-nats-kafka-bridge</code></td>
|
||
<td>连接 NATS <code>172.17.111.56:4222</code>；Kafka <code>172.17.111.56:9092</code></td>
|
||
<td>当前部署在 gateway ECS，负责 NATS 到 Kafka 桥接。</td>
|
||
</tr>
|
||
<tr>
|
||
<td>Gateway ECS</td>
|
||
<td><code>history-writer</code>、<code>stat-writer</code>、<code>realtime-api</code></td>
|
||
<td>API <code>115.29.187.205:20200</code></td>
|
||
<td>消费 Kafka，写 TDengine/Redis/MySQL，并提供查询。</td>
|
||
</tr>
|
||
<tr>
|
||
<td>Kafka ECS</td>
|
||
<td>Kafka + NATS</td>
|
||
<td>公网 <code>114.55.58.251</code>；内网 <code>172.17.111.56</code></td>
|
||
<td>NATS 使用 Docker Compose，端口只绑定内网 <code>4222</code>/<code>8222</code>。</td>
|
||
</tr>
|
||
<tr>
|
||
<td>TDengine ECS</td>
|
||
<td>TDengine</td>
|
||
<td>内网 <code>172.17.111.57:6041</code></td>
|
||
<td>历史 RAW、位置时序存储。</td>
|
||
</tr>
|
||
<tr>
|
||
<td>云服务</td>
|
||
<td>MySQL / Redis</td>
|
||
<td>RDS 内网、Redis 内网</td>
|
||
<td>身份绑定、808 注册、每日指标、实时缓存。</td>
|
||
</tr>
|
||
</tbody>
|
||
</table>
|
||
</section>
|
||
|
||
<section class="section">
|
||
<h2>9. 关键可靠性设计</h2>
|
||
<div class="grid-3">
|
||
<div class="ok">
|
||
<strong>接入不直接等待 Kafka</strong><br>
|
||
gateway 先写 NATS，Kafka 慢写由 bridge 消化，降低 32960/808 连接积压风险。
|
||
</div>
|
||
<div class="ok">
|
||
<strong>NATS ACK 在 Kafka 成功之后</strong><br>
|
||
bridge 写 Kafka 失败时不 ACK，消息仍在 JetStream，恢复后继续拉取。
|
||
</div>
|
||
<div class="ok">
|
||
<strong>RAW 完整 JSON 保留</strong><br>
|
||
<code>parsed</code> 全量写 RAW，核心表只存查询常用字段，避免位置查询每次扫完整 JSON。
|
||
</div>
|
||
<div class="ok">
|
||
<strong>实时多协议合并</strong><br>
|
||
Redis latest snapshot 只保留统一实时核心字段；完整协议 parsed 只放 realtime-raw，避免重复缓存大 JSON。
|
||
</div>
|
||
<div class="ok">
|
||
<strong>身份解析边界</strong><br>
|
||
实时缓存只写 VIN 非空车辆；无 VIN 帧保留在 RAW 和注册/绑定排查链路，补齐绑定后通过新帧或回放进入实时态。
|
||
</div>
|
||
<div class="ok">
|
||
<strong>数据分层查询</strong><br>
|
||
高频查询走 TDengine 核心字段表和 Redis；完整追溯走 RAW JSON，性能和完整性分开处理。
|
||
</div>
|
||
</div>
|
||
</section>
|
||
|
||
<section class="section">
|
||
<h2>10. 当前运行链路一句话</h2>
|
||
<p>
|
||
外部 32960/808/MQTT 数据进入 Go gateway 后，被解析成 <code>FrameEnvelope</code>，先写入 NATS JetStream；
|
||
<code>nats-kafka-bridge</code> 把 NATS 消息可靠转写到 Kafka；Kafka 再分发给 <code>history-writer</code> 写 TDengine、
|
||
<code>stat-writer</code> 写 MySQL 每日指标、<code>realtime-api</code> 写 Redis 实时缓存并提供 HTTP 查询。
|
||
</p>
|
||
</section>
|
||
</main>
|
||
|
||
<div class="footer">
|
||
Generated for lingniu-vehicle-ingest Go architecture. Update this file when gateway subjects, storage tables, or deployment topology changes.
|
||
</div>
|
||
</body>
|
||
</html>
|