feat(go): expose gateway frame duration histogram
This commit is contained in:
lingniu
@@ -38,7 +38,7 @@
|
||||
- `net.core.somaxconn=128`，无法作为 100K 连接生产基线。
|
||||
- Gateway 默认 `TCP_MAX_CONNECTIONS` 已调整为 `120000`，生产仍可通过环境变量覆盖。
|
||||
- 已新增可重复的 TCP 连接压测工具，后续需要用它跑 10K/50K/100K 阶段测试并记录结果。
|
||||
- 仍缺按协议维度的连接生命周期、读超时、parse/publish p95/p99 直方图。
|
||||
- 仍缺读超时按协议维度计数；Gateway frame duration 已提供 histogram，可用于 parse + enqueue + response 的 p95/p99 估算。
|
||||
- TDengine history writer 当前逐消息插入，后续需要批量写入。
|
||||
- Kafka topic 当前 12 分区，100K 目标下需要结合实际 FPS 再评估分区数。
|
||||
|
||||
@@ -65,7 +65,7 @@ systemd gateway 已配置 `LimitNOFILE=1048576`，需要持续保持。
|
||||
| Gateway async sink | `vehicle_async_sink_enqueue_total{sink,kind,status}` | `timeout/closed` 不增长 |
|
||||
| Gateway async sink | `vehicle_async_sink_publish_total{sink,kind,status}` | `error` 不增长 |
|
||||
| Gateway | `vehicle_gateway_publish_total{status="ok"}` | 持续增长 |
|
||||
| Gateway | parse/publish duration | p99 小于容量目标 |
|
||||
| Gateway | `vehicle_gateway_frame_duration_ms_histogram_bucket` | p99 小于容量目标 |
|
||||
| History writer | `vehicle_history_batch_flush_total{status}` | `error` 不增长 |
|
||||
| History writer | `vehicle_history_batch_rows_total{status}` | batch 行数持续增长 |
|
||||
| History writer | `vehicle_history_batch_pending_messages` | 稳态接近 0，burst 后下降 |
|
||||
|
||||
@@ -35,6 +35,7 @@ curl -fsS http://127.0.0.1:20200/readyz
|
||||
| --- | --- |
|
||||
| `vehicle_gateway_active_connections` | Current TCP connections by protocol. Labels: `protocol`. |
|
||||
| `vehicle_gateway_frames_total` | Protocol frames received and parsed by the gateway. Labels: `protocol`, `status`. |
|
||||
| `vehicle_gateway_frame_duration_ms` | Last observed Gateway frame handling duration and histogram buckets for p95/p99. Labels: `protocol`, `status`. |
|
||||
| `vehicle_gateway_publish_total` | Gateway publish result for raw events. Unified appears only when the compatibility stream is explicitly enabled. Labels: `protocol`, `kind`, `status`. |
|
||||
| `vehicle_bridge_messages_total` | NATS messages fetched by the bridge. Labels: `subject`, `status`. |
|
||||
| `vehicle_bridge_kafka_writes_total` | Bridge writes to Kafka. Labels: `topic`, `status`. |
|
||||
|
||||
@@ -406,6 +406,18 @@ vehicle_async_sink_queue_depth{sink}
|
||||
|
||||
这些指标是进入带帧率压测前的关键保护栏：如果 `queue_depth` 持续增长或 `enqueue_total{status="timeout"}` 增长，说明入口 publish 队列已成为瓶颈。
|
||||
|
||||
### Gateway frame duration histogram
|
||||
|
||||
2026-07-03 已为 Gateway 帧处理耗时增加 histogram：
|
||||
|
||||
```text
|
||||
vehicle_gateway_frame_duration_ms_histogram_bucket{protocol,status,le}
|
||||
vehicle_gateway_frame_duration_ms_histogram_count{protocol,status}
|
||||
vehicle_gateway_frame_duration_ms_histogram_sum{protocol,status}
|
||||
```
|
||||
|
||||
该耗时覆盖单帧从解析、identity resolve、parsed fields 生成、publish enqueue 到协议响应写入的整体入口处理路径。带帧率压测时用它按协议观察 p95/p99，不能只看 `vehicle_gateway_frame_duration_ms` 最后一条 gauge。
|
||||
|
||||
### History writer TDengine 批写
|
||||
|
||||
2026-07-03 已实现 history-writer 第一阶段批写：
|
||||
|
||||
@@ -116,6 +116,7 @@ go run ./cmd/load-sim \
|
||||
| `/readyz` 非 ok | 立即处理 | 服务或依赖不可用。 |
|
||||
| Gateway 活跃连接 | 预期有流量时某协议降为 `0` | 上游网络、监听端口或进程可能异常。 |
|
||||
| `vehicle_gateway_frames_total{status!="OK"}` | 连续 2 分钟增长 | 解析器或上游报文质量异常。 |
|
||||
| `vehicle_gateway_frame_duration_ms_histogram_bucket` | p99 连续 5 分钟超过容量目标 | Gateway parse、identity resolve、publish enqueue 或响应链路变慢。 |
|
||||
| `vehicle_async_sink_queue_depth{sink="nats"}` | 持续增长且不回落 | Gateway 到 NATS/Kafka 的异步 publish 队列开始积压。 |
|
||||
| `vehicle_async_sink_enqueue_total{status="timeout"}` | 任意增长 | Gateway publish 队列已满或 worker 长时间阻塞，入口可能开始丢实时性。 |
|
||||
| `vehicle_async_sink_publish_total{status="error"}` | 连续增长 | NATS/Kafka publish 失败，需要先查中间件连接和日志。 |
|
||||
|
||||
Reference in New Issue
Block a user