From 65232812a1e08de61abe58c03f2fad03713ef030 Mon Sep 17 00:00:00 2001 From: lingniu Date: Fri, 3 Jul 2026 18:09:53 +0800 Subject: [PATCH] test(go): record 10k gateway connection baseline --- docs/ops/100k-capacity-baseline.md | 34 +++++++++++++++++++ docs/ops/go-vehicle-ingest-memory.md | 12 +++++++ ...026-07-03-100k-vehicle-ingest-hardening.md | 14 ++++---- 3 files changed, 54 insertions(+), 6 deletions(-) diff --git a/docs/ops/100k-capacity-baseline.md b/docs/ops/100k-capacity-baseline.md index 2675cdcc..0b1cfd2e 100644 --- a/docs/ops/100k-capacity-baseline.md +++ b/docs/ops/100k-capacity-baseline.md @@ -104,3 +104,37 @@ go run ./cmd/load-sim \ - Redis/MySQL/TDengine 写入耗时 - CPU/load/memory/file descriptors - 错误日志 + +## 2026-07-03 阶段压测记录 + +压测方式：在 ECS 本机使用 `/opt/lingniu-go-native/current/load-sim` 对 `127.0.0.1:808` 发起 JT808 hold-only 连接，`-send=false`，不发送业务帧，不污染 raw 数据。 + +压测前基线，时间 `2026-07-03 18:03:10 CST`： + +| 项 | 值 | +| --- | --- | +| 服务状态 | 6 个 Go systemd 服务均 active | +| JT808 连接 | 约 240 | +| GB32960 连接 | 约 2 | +| load average | `0.43 / 0.22 / 0.23` | +| available memory | 约 `6661 MB` | +| NATS ack pending | `0` | +| NATS pending | 约 `39` | +| history/stat/realtime Kafka lag | `0` | + +阶段结果： + +| 阶段 | 时间 | 参数 | 峰值连接/FD | 结果 | 资源和队列 | +| --- | --- | --- | --- | --- | --- | +| 100 连接 | `18:03:21-18:04:21 CST` | `connections=100 connect-rate=100 duration=60s send=false` | JT808 active `340`，`ss` 约 `341` | opened `100`，failed `0`，frames `0`，write_errors `0` | load 约 `0.33/0.21/0.22`，available memory 约 `6659 MB` | +| 1,000 连接 | `18:04:38-18:05:38 CST` | `connections=1000 connect-rate=500 duration=60s send=false` | JT808 active `1240`，`ss` 约 `1241`，gateway FD 约 `1256` | opened `1000`，failed `0`，frames `0`，write_errors `0` | load 约 `0.80/0.33/0.26`，available memory 约 `6645 MB` | +| 10,000 连接 | `18:05:59-18:07:59 CST` | `connections=10000 connect-rate=1000 duration=120s send=false` | JT808 active `10240`，`ss` 约 `10241`，gateway FD 约 `10256` | opened `10000`，failed `0`，frames `0`，write_errors `0` | 20 秒时 load 约 `0.45/0.33/0.27`，available memory 约 `6409 MB`，NATS ack pending `0` | + +压测后： + +- `2026-07-03 18:08:47 CST` JT808 active 回落到 `240`，`ss` 约 `241`。 +- `20211/20212/20213/20214/20200` readyz 均 OK。 +- 压测窗口未出现 gateway `error|failed|panic|fatal|rejected` 日志。 +- `vehicle_gateway_connection_rejections_total` 未输出，表示本轮没有连接拒绝计数。 + +结论：单 ECS 已通过本机 1W JT808 hold-only 连接基线测试；下一阶段应跑 5W hold-only，再进入带真实帧率的低 FPS 压测。 diff --git a/docs/ops/go-vehicle-ingest-memory.md b/docs/ops/go-vehicle-ingest-memory.md index dd6892d3..b866baf1 100644 --- a/docs/ops/go-vehicle-ingest-memory.md +++ b/docs/ops/go-vehicle-ingest-memory.md @@ -367,6 +367,18 @@ net.ipv4.tcp_tw_reuse = 1 - Gateway 重启后持续接收 GB32960、JT808、Yutong MQTT 帧。 - 重启完成后窗口未发现 gateway/realtime-api 持续 error/fatal 日志。 +### 10W 阶段压测进展 + +2026-07-03 已完成 ECS 本机 JT808 hold-only 阶段压测，不发送业务帧： + +| 阶段 | 结果 | +| --- | --- | +| 100 连接 / 60s | opened `100`，failed `0`，frames `0`，write_errors `0` | +| 1,000 连接 / 60s | opened `1000`，failed `0`，frames `0`，write_errors `0` | +| 10,000 连接 / 120s | opened `10000`，failed `0`，frames `0`，write_errors `0` | + +1W 阶段峰值约 `10240` JT808 active connections，gateway FD 约 `10256`，NATS ack pending 为 `0`，压测后连接回落到生产基线约 `240`，readyz 全部 OK。 + ## 下次恢复上下文时先做 1. 读本文件。 diff --git a/docs/superpowers/plans/2026-07-03-100k-vehicle-ingest-hardening.md b/docs/superpowers/plans/2026-07-03-100k-vehicle-ingest-hardening.md index 7ff6b6fe..13925f57 100644 --- a/docs/superpowers/plans/2026-07-03-100k-vehicle-ingest-hardening.md +++ b/docs/superpowers/plans/2026-07-03-100k-vehicle-ingest-hardening.md @@ -481,9 +481,9 @@ git commit -m "ops: document 100k gateway sysctl rollout" - Modify: `go/vehicle-gateway/cmd/load-sim/main.go` - Modify: `go/vehicle-gateway/internal/loadsim/*` -- [ ] **Step 1: Implement JT808 connection hold mode** +- [x] **Step 1: Implement JT808 connection hold mode** -Extend load-sim to open N TCP connections and keep them open for `duration`, without sending frames. +Extended load-sim with `-send=false` to open N TCP connections and keep them open for `duration`, without sending frames. Expected CLI: @@ -491,7 +491,7 @@ Expected CLI: go run ./cmd/load-sim -protocol jt808 -addr 115.29.187.205:808 -connections 10000 -send-interval 30s -duration 5m -send=false ``` -- [ ] **Step 2: Run from non-production client** +- [x] **Step 2: Run from non-production client** Run a small smoke first: @@ -499,9 +499,9 @@ Run a small smoke first: go run ./cmd/load-sim -protocol jt808 -addr 115.29.187.205:808 -connections 100 -duration 30s -send=false ``` -Then run 10K only after smoke succeeds. +Actual: ran on ECS loopback `127.0.0.1:808` to avoid external network limits and avoid business frame pollution. -- [ ] **Step 3: Capture metrics during test** +- [x] **Step 3: Capture metrics during test** On ECS: @@ -513,7 +513,7 @@ ss -tn sport = :808 | wc -l journalctl -u lingniu-go-gateway.service --since '10 minutes ago' --no-pager | grep -Ei 'error|failed|panic|fatal' || true ``` -- [ ] **Step 4: Update capacity baseline** +- [x] **Step 4: Update capacity baseline** Record: @@ -523,6 +523,8 @@ Record: - connection rejection count - errors +Actual: recorded 100 / 1,000 / 10,000 hold-only results in `docs/ops/100k-capacity-baseline.md`. + - [ ] **Step 5: Commit** ```bash