test(go): record 10k gateway connection baseline
This commit is contained in:
@@ -104,3 +104,37 @@ go run ./cmd/load-sim \
|
|||||||
- Redis/MySQL/TDengine 写入耗时
|
- Redis/MySQL/TDengine 写入耗时
|
||||||
- CPU/load/memory/file descriptors
|
- CPU/load/memory/file descriptors
|
||||||
- 错误日志
|
- 错误日志
|
||||||
|
|
||||||
|
## 2026-07-03 阶段压测记录
|
||||||
|
|
||||||
|
压测方式:在 ECS 本机使用 `/opt/lingniu-go-native/current/load-sim` 对 `127.0.0.1:808` 发起 JT808 hold-only 连接,`-send=false`,不发送业务帧,不污染 raw 数据。
|
||||||
|
|
||||||
|
压测前基线,时间 `2026-07-03 18:03:10 CST`:
|
||||||
|
|
||||||
|
| 项 | 值 |
|
||||||
|
| --- | --- |
|
||||||
|
| 服务状态 | 6 个 Go systemd 服务均 active |
|
||||||
|
| JT808 连接 | 约 240 |
|
||||||
|
| GB32960 连接 | 约 2 |
|
||||||
|
| load average | `0.43 / 0.22 / 0.23` |
|
||||||
|
| available memory | 约 `6661 MB` |
|
||||||
|
| NATS ack pending | `0` |
|
||||||
|
| NATS pending | 约 `39` |
|
||||||
|
| history/stat/realtime Kafka lag | `0` |
|
||||||
|
|
||||||
|
阶段结果:
|
||||||
|
|
||||||
|
| 阶段 | 时间 | 参数 | 峰值连接/FD | 结果 | 资源和队列 |
|
||||||
|
| --- | --- | --- | --- | --- | --- |
|
||||||
|
| 100 连接 | `18:03:21-18:04:21 CST` | `connections=100 connect-rate=100 duration=60s send=false` | JT808 active `340`,`ss` 约 `341` | opened `100`,failed `0`,frames `0`,write_errors `0` | load 约 `0.33/0.21/0.22`,available memory 约 `6659 MB` |
|
||||||
|
| 1,000 连接 | `18:04:38-18:05:38 CST` | `connections=1000 connect-rate=500 duration=60s send=false` | JT808 active `1240`,`ss` 约 `1241`,gateway FD 约 `1256` | opened `1000`,failed `0`,frames `0`,write_errors `0` | load 约 `0.80/0.33/0.26`,available memory 约 `6645 MB` |
|
||||||
|
| 10,000 连接 | `18:05:59-18:07:59 CST` | `connections=10000 connect-rate=1000 duration=120s send=false` | JT808 active `10240`,`ss` 约 `10241`,gateway FD 约 `10256` | opened `10000`,failed `0`,frames `0`,write_errors `0` | 20 秒时 load 约 `0.45/0.33/0.27`,available memory 约 `6409 MB`,NATS ack pending `0` |
|
||||||
|
|
||||||
|
压测后:
|
||||||
|
|
||||||
|
- `2026-07-03 18:08:47 CST` JT808 active 回落到 `240`,`ss` 约 `241`。
|
||||||
|
- `20211/20212/20213/20214/20200` readyz 均 OK。
|
||||||
|
- 压测窗口未出现 gateway `error|failed|panic|fatal|rejected` 日志。
|
||||||
|
- `vehicle_gateway_connection_rejections_total` 未输出,表示本轮没有连接拒绝计数。
|
||||||
|
|
||||||
|
结论:单 ECS 已通过本机 1W JT808 hold-only 连接基线测试;下一阶段应跑 5W hold-only,再进入带真实帧率的低 FPS 压测。
|
||||||
|
|||||||
@@ -367,6 +367,18 @@ net.ipv4.tcp_tw_reuse = 1
|
|||||||
- Gateway 重启后持续接收 GB32960、JT808、Yutong MQTT 帧。
|
- Gateway 重启后持续接收 GB32960、JT808、Yutong MQTT 帧。
|
||||||
- 重启完成后窗口未发现 gateway/realtime-api 持续 error/fatal 日志。
|
- 重启完成后窗口未发现 gateway/realtime-api 持续 error/fatal 日志。
|
||||||
|
|
||||||
|
### 10W 阶段压测进展
|
||||||
|
|
||||||
|
2026-07-03 已完成 ECS 本机 JT808 hold-only 阶段压测,不发送业务帧:
|
||||||
|
|
||||||
|
| 阶段 | 结果 |
|
||||||
|
| --- | --- |
|
||||||
|
| 100 连接 / 60s | opened `100`,failed `0`,frames `0`,write_errors `0` |
|
||||||
|
| 1,000 连接 / 60s | opened `1000`,failed `0`,frames `0`,write_errors `0` |
|
||||||
|
| 10,000 连接 / 120s | opened `10000`,failed `0`,frames `0`,write_errors `0` |
|
||||||
|
|
||||||
|
1W 阶段峰值约 `10240` JT808 active connections,gateway FD 约 `10256`,NATS ack pending 为 `0`,压测后连接回落到生产基线约 `240`,readyz 全部 OK。
|
||||||
|
|
||||||
## 下次恢复上下文时先做
|
## 下次恢复上下文时先做
|
||||||
|
|
||||||
1. 读本文件。
|
1. 读本文件。
|
||||||
|
|||||||
@@ -481,9 +481,9 @@ git commit -m "ops: document 100k gateway sysctl rollout"
|
|||||||
- Modify: `go/vehicle-gateway/cmd/load-sim/main.go`
|
- Modify: `go/vehicle-gateway/cmd/load-sim/main.go`
|
||||||
- Modify: `go/vehicle-gateway/internal/loadsim/*`
|
- Modify: `go/vehicle-gateway/internal/loadsim/*`
|
||||||
|
|
||||||
- [ ] **Step 1: Implement JT808 connection hold mode**
|
- [x] **Step 1: Implement JT808 connection hold mode**
|
||||||
|
|
||||||
Extend load-sim to open N TCP connections and keep them open for `duration`, without sending frames.
|
Extended load-sim with `-send=false` to open N TCP connections and keep them open for `duration`, without sending frames.
|
||||||
|
|
||||||
Expected CLI:
|
Expected CLI:
|
||||||
|
|
||||||
@@ -491,7 +491,7 @@ Expected CLI:
|
|||||||
go run ./cmd/load-sim -protocol jt808 -addr 115.29.187.205:808 -connections 10000 -send-interval 30s -duration 5m -send=false
|
go run ./cmd/load-sim -protocol jt808 -addr 115.29.187.205:808 -connections 10000 -send-interval 30s -duration 5m -send=false
|
||||||
```
|
```
|
||||||
|
|
||||||
- [ ] **Step 2: Run from non-production client**
|
- [x] **Step 2: Run from non-production client**
|
||||||
|
|
||||||
Run a small smoke first:
|
Run a small smoke first:
|
||||||
|
|
||||||
@@ -499,9 +499,9 @@ Run a small smoke first:
|
|||||||
go run ./cmd/load-sim -protocol jt808 -addr 115.29.187.205:808 -connections 100 -duration 30s -send=false
|
go run ./cmd/load-sim -protocol jt808 -addr 115.29.187.205:808 -connections 100 -duration 30s -send=false
|
||||||
```
|
```
|
||||||
|
|
||||||
Then run 10K only after smoke succeeds.
|
Actual: ran on ECS loopback `127.0.0.1:808` to avoid external network limits and avoid business frame pollution.
|
||||||
|
|
||||||
- [ ] **Step 3: Capture metrics during test**
|
- [x] **Step 3: Capture metrics during test**
|
||||||
|
|
||||||
On ECS:
|
On ECS:
|
||||||
|
|
||||||
@@ -513,7 +513,7 @@ ss -tn sport = :808 | wc -l
|
|||||||
journalctl -u lingniu-go-gateway.service --since '10 minutes ago' --no-pager | grep -Ei 'error|failed|panic|fatal' || true
|
journalctl -u lingniu-go-gateway.service --since '10 minutes ago' --no-pager | grep -Ei 'error|failed|panic|fatal' || true
|
||||||
```
|
```
|
||||||
|
|
||||||
- [ ] **Step 4: Update capacity baseline**
|
- [x] **Step 4: Update capacity baseline**
|
||||||
|
|
||||||
Record:
|
Record:
|
||||||
|
|
||||||
@@ -523,6 +523,8 @@ Record:
|
|||||||
- connection rejection count
|
- connection rejection count
|
||||||
- errors
|
- errors
|
||||||
|
|
||||||
|
Actual: recorded 100 / 1,000 / 10,000 hold-only results in `docs/ops/100k-capacity-baseline.md`.
|
||||||
|
|
||||||
- [ ] **Step 5: Commit**
|
- [ ] **Step 5: Commit**
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
|
|||||||
Reference in New Issue
Block a user