fix(stats): recover durable daily mileage projections
This commit is contained in:
@@ -151,6 +151,8 @@ systemctl start lingniu-go-capacity-check.service
|
||||
| `vehicle_stat_sources_total` | Stat writer source tracking results for `vehicle_data_source`. Labels: `topic`, `protocol`, `status`; status includes `attempted`, `written`, `skipped_throttled`, `skipped_missing_endpoint`. This intentionally avoids source IP labels to keep metrics low-cardinality. |
|
||||
| `vehicle_stat_projections_total` | Final daily mileage projection results from `vehicle_daily_mileage_source` to `vehicle_daily_mileage`. Labels: `topic`, `protocol`, `status`; status includes `attempted`, `written`, `skipped_throttled`. |
|
||||
| `vehicle_stat_pending_projections_total` / `vehicle_stat_pending_projection_entries` | Eventual projection tail flush. `attempted` and `written` should grow together; `error` indicates a pending final projection remains queued for retry. The entries gauge should stay bounded by active vehicle/protocol/day keys and drain after the projection interval. |
|
||||
| `vehicle_stat_startup_projection_reconcile_targets` / `vehicle_stat_startup_projection_reconcile_written` / `vehicle_stat_startup_projection_reconcile_failed` | Startup recovery for durable daily-mileage candidates that are newer than, or missing from, the final projection. Healthy startup has `written = targets` and `failed = 0`. |
|
||||
| `vehicle_stat_last_startup_projection_reconcile_unix_seconds` | Last startup recovery result, labeled `status="ok"` or `status="error"`. An error preserves the gap as observable evidence and must be followed by a targeted backfill or another successful restart reconciliation. |
|
||||
| `vehicle_realtime_updates_total` | Redis/MySQL realtime projector updates. Labels: `topic`, `status`. |
|
||||
| `vehicle_realtime_kafka_messages_total` | Realtime projector Kafka message results by raw topic. Labels: `topic`, `status`; `invalid_json` messages are committed and isolated before Redis/MySQL projection. |
|
||||
| `vehicle_realtime_last_message_unix_seconds` | Last realtime Kafka message by topic/status. Labels: `topic`, `status`. |
|
||||
|
||||
@@ -47,6 +47,8 @@ Gateway 在 NATS 模式下设置 `FIELDS_DERIVE_FROM_RAW_ENABLED=true`,每帧
|
||||
|
||||
如果 stat-writer 少消费某个 fields topic,对应协议的每日里程不会进入 `vehicle_daily_mileage`。stat-writer 启动时会拒绝 `vehicle.raw.*` 这类 RAW topic 配置,避免统计链路重新从原始报文解析并破坏 RAW/fields 解耦。修正后可能出现短时间 Kafka lag,这是在追补历史 backlog;`vehicle_stat_writes_total{status="ok"}` 只表示 append 调用成功,真正证明里程样本入库的是 `vehicle_stat_samples_total{status="written"}` 持续增长且 lag 下降。
|
||||
|
||||
stat-writer 的节流尾部投影队列保存在内存中,但来源候选事实已经落在 MySQL。服务启动时默认执行 `STATS_RECONCILE_PROJECTIONS_ON_START=true` 的当天持久化候选扫描,把“候选比最终日表新”或“最终日表缺失”的车辆/协议重新投影;这用于关闭候选提交后进程重启造成的漏算窗口,尤其避免静止车辆因没有后续里程变化而一直漏到次日回补。默认超时由 `STATS_RECONCILE_PROJECTIONS_TIMEOUT_SECONDS=30` 控制。核验 `vehicle_stat_startup_projection_reconcile_targets`、`vehicle_stat_startup_projection_reconcile_written`、`vehicle_stat_startup_projection_reconcile_failed` 和 `vehicle_stat_last_startup_projection_reconcile_unix_seconds{status}`;失败不会伪装成功,仍由日志、指标和次日 backfill 继续暴露。
|
||||
|
||||
stat-writer 默认 `STATS_WORKERS=3`,每个 worker 是同一 consumer group 的独立 Kafka reader。Kafka 按车辆 key 固定分区,因此单车累计里程仍按分区顺序执行,不同分区并行。调整 worker 后同时核对 `vehicle_stat_config{setting="workers"}`、每个 `vehicle_stat_worker_active{worker}`、MySQL 连接数、write p99 和 stat lag;worker 不应超过 fields topic 的有效分区并行度。
|
||||
|
||||
stat-writer 对死锁、锁等待和网络中断等瞬时 MySQL 故障按 `STATS_RETRY_ATTEMPTS` 有限重试,重试耗尽后保持失败关闭并等待存储恢复。只有 `NULL`、数值越界、类型转换、字段过长和 CHECK 约束这类明确的单消息数据错误,才会在同样的有限重试耗尽后写入 `STATS_QUARANTINE_DIR` 并提交该 Kafka 偏移量,避免一条坏消息阻塞整个分区;表结构、权限和未知程序错误不会被隔离。隔离文件包含原 topic、partition、offset、key、value 和错误,默认权限为 `0600`,处理前不得删除。
|
||||
|
||||
Reference in New Issue
Block a user