lingniu-vehicle-ingest/docs/ops/go-service-observability.md

# Go Service Observability

## ECS

Host: `115.29.187.205`

The Go services expose local-only health and metrics endpoints. They are intended for ECS-local checks, Prometheus scraping through an agent, or SSH troubleshooting. They are not public business APIs.

## Endpoints

| Service | systemd unit | Address | Health | Readiness | Metrics |
| --- | --- | --- | --- | --- | --- |
| Gateway | `lingniu-go-gateway.service` | `127.0.0.1:20211` | `/healthz` | `/readyz` | `/metrics` |
| History writer | `lingniu-go-history-writer.service` | `127.0.0.1:20212` | `/healthz` | `/readyz` | `/metrics` |
| Stat writer | `lingniu-go-stat-writer.service` | `127.0.0.1:20213` | `/healthz` | `/readyz` | `/metrics` |
| NATS Kafka bridge | `lingniu-go-nats-kafka-bridge.service` | `127.0.0.1:20214` | `/healthz` | `/readyz` | `/metrics` |
| Identity writer | `lingniu-go-identity-writer.service` | `127.0.0.1:20217` | `/healthz` | `/readyz` | `/metrics` |
| Realtime API | `lingniu-go-realtime-api.service` | `0.0.0.0:20200` | `/healthz` | `/readyz` | `/metrics` |

## Quick Checks

生产事故处理顺序和阈值解释见 [车辆接入生产运行手册](vehicle-ingest-runbook.md)。

```bash
curl -fsS http://127.0.0.1:20211/readyz
curl -fsS http://127.0.0.1:20211/metrics
curl -fsS http://127.0.0.1:20212/readyz
curl -fsS http://127.0.0.1:20213/readyz
curl -fsS http://127.0.0.1:20214/readyz
curl -fsS http://127.0.0.1:20217/readyz
curl -fsS http://127.0.0.1:20200/readyz
```

本机容量健康摘要：

```bash
/opt/lingniu-go-native/current/capacity-check
```

它会抓取 Gateway、History writer、Stat writer、Identity writer、NATS bridge、Realtime writer/API、NATS fast writer 的本地 `/metrics`，输出 JSON。退出码 `0` 表示当前关键 backlog 和拒绝计数正常，退出码 `2` 表示存在超阈值 pending、Kafka lag、连接拒绝或 metrics 抓取失败，适合接入 cron/告警。

Identity writer 必须消费 `vehicle.raw.go.jt808.v1`，且 Gateway 指标 `vehicle_gateway_jt808_registration_gateway_writes_enabled` 必须为 `0`。两者同时写 `jt808_registration` 会被 `capacity-check` 判为重复 writer。

Gateway 的 NATS 接受边界由 WAL outbox 提供。`vehicle_durable_outbox_backlog_records` 表示已经 `fsync` 但尚未 PubAck 的记录，`vehicle_durable_outbox_inflight` 表示正在等待 PubAck 的记录；健康时两者最终回到 `0`。`vehicle_durable_outbox_publish_total` 的 `submit_error`、`ack_error`、`remove_error`、`close_timeout` 默认按 outbox 错误门禁处理。稳定 event ID/NATS MsgId 负责崩溃恢复期间的 JetStream 去重。

默认还会用 histogram 估算链路 p99：Gateway frame `250ms`、Gateway identity `100ms`、async sink `100ms`、bridge batch `5000ms`、fast-writer stage `100ms`、history flush `1000ms`、realtime store `100ms`、stat write `100ms`；history/stat/realtime/identity 的 Kafka lag 总和超过 `100` 时降级。Gateway async sink 队列深度默认超过 `10000` 降级，可用 `-async-sink-queue-depth-max` 调整；Gateway 身份快照必须 ready 且最近成功刷新时间不能超过 `180s`，分别用 `-gateway-identity-snapshot-required`、`-gateway-identity-snapshot-max-age-seconds` 调整。Gateway 身份解析缓存默认容量阈值是 `300000`，Stat writer 统计缓存默认容量阈值是 `1000000`，Realtime API 的 MySQL 写入节流缓存和 VIN 车牌缓存默认容量阈值分别是 `1000000` 和 `200000`，可分别用 `-gateway-identity-cache-entries-max`、`-stat-cache-entries-max`、`-realtime-throttle-cache-entries-max`、`-realtime-plate-cache-entries-max` 调整，设为 `0` 表示关闭该项容量检查。history、realtime、stat、identity writer 的 worker 数默认都不得低于 `3`，分别由 `-history-workers-min`、`-realtime-workers-min`、`-stat-workers-min`、`-identity-workers-min` 检查。低样本数会跳过 p99 判断，阈值设为 `0` 可临时关闭对应延迟检查。

批处理内部 in-flight pending 默认允许 `100` 条窗口：`-fast-writer-batch-pending-max=100`、`-history-batch-pending-max=100`、`-history-rows-pending-max=100`、`-stat-batch-pending-max=100`。这类指标用于发现批写卡住，不再因为高频流量下几十条正在 flush/ack 的瞬时批次直接降级；NATS consumer pending、ack pending 和 Kafka lag 仍按独立阈值判断真实积压。

topic 流转检查默认只抓完全断层：history/realtime 必须消费三类 raw topic，stat 必须消费三类 fields topic；bridge 未配置 subject 的 `route_error` 默认超过 `0` 条就降级，可用 `-bridge-route-error-max=-1` 临时关闭；fast-writer `invalid_json` 默认超过 `0` 条就降级，可用 `-fast-writer-invalid-json-max=-1` 临时关闭；history-writer `invalid_json` 默认超过 `0` 条就降级，可用 `-history-invalid-json-max=-1` 临时关闭；realtime projector `invalid_json` 默认超过 `0` 条就降级，可用 `-realtime-invalid-json-max=-1` 临时关闭；stat-writer `invalid_json` 默认超过 `0` 条就降级，可用 `-stat-invalid-json-max=-1` 临时关闭；某协议 gateway raw publish 达到 `100` 条，扣除 `skipped_non_realtime` 后仍然没有 fields publish 时降级；实时 fields 缺失或 publish error 比例超过 `20%` 时也会降级；某协议 gateway raw/fields publish 达到 `1000` 条但 bridge 没有写对应 Kafka topic 时降级；某协议 gateway raw publish 达到 `1000` 条但 fast-writer 没有消费同 raw subject 并成功写 Redis/ack 时降级；某 raw topic bridge 写 Kafka 达到 `1000` 条但 history 未收到/写入或 realtime 未收到/更新时降级；某 fields topic bridge 写 Kafka 达到 `100` 条但 stat-writer 未配置或未收到该 topic 时降级。解析质量检查默认在单协议 gateway frame 样本达到 `100` 条后启用，`OK` 以外的 `PARTIAL/BAD_FRAME/other` 比例超过 `5%` 时降级，用于发现协议解析器或上游报文质量突然恶化。响应质量检查默认在单协议 gateway response 尝试样本达到 `100` 条后启用，`build_error/write_error` 比例超过 `5%` 时降级；`skipped` 表示该消息类型无需响应，不计入分母。身份解析质量检查默认在单协议 identity 样本达到 `100` 条后启用，`resolved` 以外的 `unresolved/error/timeout` 比例超过 `20%` 时降级，用于发现 VIN/车牌/phone 映射突然失效。history-writer 会对实时数据帧检查 raw envelope 是否携带 `parsed_fields`，默认单 topic/protocol 达到 `100` 帧后，缺失比例超过 `5%` 降级，用于发现历史证据层丢失扁平化解析字段。Redis 快路径字段质量检查默认在单 raw subject 写入字段达到 `1000` 个后启用，`skipped_stale/seen` 超过 `20%` 时降级，用于发现大面积旧帧补发或设备时间漂移。stat-writer 还会在单个 fields topic 成功 append 达到 `100` 条但未抽取到任何里程样本时降级，或 VIN 缺失、里程缺失、非正里程、source 缺失这类可行动跳过比例超过 `60%` 时降级，用于发现字段映射或身份/source 关联断裂；单 topic 找到 `100` 条里程样本但没有 source tracking，或 source endpoint 缺失比例超过 `60%` 时也会降级，用于发现 `vehicle_data_source` 来源管理断层；单 topic 已写入 `100` 条里程样本但最终 `vehicle_daily_mileage` 投影写入仍为 `0` 时也会降级，用于发现 source 候选层到最终查询层的断层；`skipped_same_mileage` 是重复总里程去重，不计入可行动跳过比例。已出现过的成功/received `*_last_*_unix_seconds` 如果超过 `300s` 未刷新，也会降级；刚启动尚未出现过该指标时不会因缺失 last activity 降级。`capacity-check` JSON 会额外派生 `*_last_*_age_seconds`，便于直接读取距今秒数。历史位置派生写入错误使用 `vehicle_history_last_location_write_unix_seconds{status="error"}` 判断是否仍在发生：默认只在最近 `300s` 内出现错误时降级，进程生命周期累计 counter 继续保留用于审计；可用 `-history-location-error-recent-seconds` 调整窗口，设为 `0` 时恢复累计值门禁。特殊环境可用 `-required-consumer-topics=''` 临时关闭消费 topic 合同检查。

`capacity-check` 还会默认请求 Realtime API 的 `/api/stats/daily-metrics/diagnostics/reasons`，把当天每日里程诊断聚合放到 JSON 的 `daily_mileage_diagnostics`。这项默认只展示 `vehicle_total`、`actionable_issue_total` 和原因分布，不会因为存在业务缺口直接退出 `2`；需要把缺口纳入发布/巡检阻断时，显式传 `-daily-mileage-diagnostics-max-actionable=0` 或其他阈值。诊断 API 不可访问会被视为观测能力故障并降级；特殊环境可用 `-daily-mileage-diagnostics-url=''` 关闭。

ECS 上通过 systemd timer 每分钟执行一次：

```bash
systemctl status lingniu-go-capacity-check.timer
systemctl list-timers lingniu-go-capacity-check.timer
journalctl -u lingniu-go-capacity-check.service --since '10 minutes ago' --no-pager
systemctl start lingniu-go-capacity-check.service
```

`lingniu-go-capacity-check.service` 是 oneshot 服务。容量健康时退出码为 `0`；不健康时退出码为 `2`，timer 会保留 failed 结果，JSON findings 会写入 journal。

## Core Counters

| Metric | Meaning |
| --- | --- |
| `vehicle_gateway_active_connections` | Current TCP connections by protocol. Labels: `protocol`. |
| `vehicle_gateway_connection_closes_total` | TCP connection closes by protocol and reason. Labels: `protocol`, `reason`. Reasons include `eof`, `read_timeout`, `read_error`, `extract_error`, `context_cancelled`, `max_connections`. |
| `vehicle_gateway_connection_rejections_total` | TCP connection rejections by protocol and reason. Labels: `protocol`, `reason`. |
| `vehicle_gateway_frames_total` | Protocol frames received and parsed by the gateway. Labels: `protocol`, `status`. |
| `vehicle_gateway_last_frame_unix_seconds` | Last observed gateway frame time by protocol and parse status. Labels: `protocol`, `status`. |
| `vehicle_gateway_frame_duration_ms` | Last observed Gateway frame handling duration and histogram buckets for p95/p99. Labels: `protocol`, `status`. |
| `vehicle_gateway_identity_total` | Gateway identity resolve results. Labels: `protocol`, `status`; status includes `resolved`, `unresolved`, `error`, `timeout`. |
| `vehicle_gateway_identity_duration_ms` | Last observed identity resolve duration and histogram buckets for p95/p99. Labels: `protocol`, `status`. |
| `vehicle_gateway_identity_cache_entries` | Gateway identity resolver cache entry count. Labels: `cache`; values include runtime `lookup`/`registration`/`source_code`/`location_touch` and `snapshot_binding`/`snapshot_identifier`/`snapshot_registration`/`snapshot_source`. |
| `vehicle_gateway_identity_cache_max_entries` | Gateway identity resolver cache entry cap per cache. Controlled by `IDENTITY_LOOKUP_CACHE_MAX_ENTRIES`, default `300000`. |
| `vehicle_gateway_identity_snapshot_ready` | `1` after at least one complete atomic identity snapshot refresh; `0` means frames continue ingesting but unknown JT808 identities cannot be resolved. |
| `vehicle_gateway_identity_snapshot_entries` | Current identity snapshot entry counts. Labels: `kind` = `binding`, `identifier`, `registration`, `source`. |
| `vehicle_gateway_identity_snapshot_refresh_total` | Snapshot refresh attempts. Labels: `status` = `ok` or `error`; an error keeps the last known good snapshot. |
| `vehicle_gateway_identity_snapshot_last_success_unix_seconds` | Last successful complete snapshot publication time. |
| `vehicle_gateway_publish_total` | Gateway publish/delegation result. In NATS mode raw is published and fields uses `status="delegated"`; unified appears only when explicitly enabled. Labels: `protocol`, `kind`, `status`. |
| `vehicle_gateway_last_publish_unix_seconds` | Last gateway publish time by protocol/kind/status, useful for protocol freshness checks. Labels: `protocol`, `kind`, `status`. |
| `vehicle_gateway_fields_total` | Gateway fields eligibility result. Labels: `protocol`, `status`; NATS mode uses `delegated_to_bridge`, while compatibility direct mode may use `published`/`publish_error`. |
| `vehicle_gateway_response_total` | Gateway protocol response build/write result. Labels: `protocol`, `message_id`, `status`; status includes `ok`, `skipped`, `build_error`, `write_error`. |
| `vehicle_gateway_last_response_unix_seconds` | Last gateway protocol response build/write result time. Labels: `protocol`, `message_id`, `status`. |
| `vehicle_gateway_response_e2e_recent_p99_ms` | Bounded recent p99 from frame receipt through successful protocol response. Labels: `protocol`; capacity target defaults to `100ms`. |
| `vehicle_gateway_response_e2e_recent_samples` | Samples in the bounded response-latency window. Labels: `protocol`. |
| `vehicle_gateway_authentication_total` | Protocol authentication decisions. Labels: `protocol`, `mode`, `source`, `status`; `source=configured` means a configured credential matched, `source=device` means the JT808 phone's snapshot token matched, and `source=none` means no credential matched. `observe` records mismatches without rejecting, while `enforce` returns a protocol failure and closes the connection. |
| `vehicle_gateway_authentication_mode` | Active authentication mode by protocol. Labels: `protocol`, `mode`. |
| `vehicle_gateway_authentication_credentials` | Configured account count, never credential values. Labels: `protocol`. |
| `vehicle_async_sink_publish_duration_ms_histogram` | Async sink publish duration histogram covering worker publish calls to the delegate sink. Labels: `sink`, `kind`, `status`. |
| `vehicle_async_sink_queue_capacity` | Gateway async sink queue capacity. Labels: `sink`. NATS uses `NATS_ASYNC_QUEUE_SIZE`; Kafka fallback uses `KAFKA_ASYNC_QUEUE_SIZE`. |
| `vehicle_bridge_messages_total` | NATS messages fetched by the bridge. Labels: `subject`, `status`. |
| `vehicle_bridge_fields_projection_total` | Fields projection result from canonical raw. Labels: `protocol`, `status`; `published` is healthy, error/skip labels explain why no fields event was emitted. |
| `vehicle_bridge_fields_projection_count` | Latest flattened field count emitted by bridge per protocol/status. |
| `vehicle_bridge_last_message_unix_seconds` | Last NATS message fetched by subject/status. Labels: `subject`, `status`. |
| `vehicle_bridge_kafka_writes_total` | Bridge writes to Kafka. Labels: `topic`, `status`. |
| `vehicle_bridge_last_kafka_write_unix_seconds` | Last bridge Kafka write by topic/status. Labels: `topic`, `status`. |
| `vehicle_bridge_nats_acks_total` | NATS ack results after Kafka write or unrouted-subject isolation. Labels: `subject`, `status`; status includes `ok`, `error`, and `dropped_route_error`. |
| `vehicle_bridge_last_ack_unix_seconds` | Last NATS ack by subject/status. Labels: `subject`, `status`. |
| `vehicle_bridge_nats_consumer_pending` | JetStream messages pending for the bridge durable consumer. |
| `vehicle_bridge_nats_consumer_ack_pending` | JetStream messages delivered to bridge but not yet acked. |
| `vehicle_bridge_nats_consumer_waiting` | Pull requests waiting on the bridge durable consumer. |
| `vehicle_bridge_batch_pending_messages` | NATS messages fetched by the bridge but not yet written to Kafka and acked. |
| `vehicle_bridge_batch_pending_kafka_messages` | Kafka messages prepared for the current bridge batch. |
| `vehicle_bridge_batch_duration_ms_histogram` | Bridge batch duration histogram covering Kafka write and NATS ack. Labels: `status`. |
| `vehicle_history_batch_flush_duration_ms_histogram` | History writer TDengine batch flush duration histogram. Labels: `status`. |
| `vehicle_realtime_store_update_duration_ms_histogram` | Realtime MySQL/Redis projection duration histogram. Labels: `store`, `protocol`, `status`. |
| `vehicle_stat_write_duration_ms_histogram` | Stat writer MySQL write duration histogram. Labels: `topic`, `status`. |
| `vehicle_stat_samples_total` | Stat writer mileage sample results. Labels: `topic`, `protocol`, `status`; status includes `found`, `written`, `skipped_missing_fields`, `skipped_missing_vin`, `skipped_missing_mileage`, `skipped_non_positive_mileage`, `skipped_same_mileage`, `skipped_missing_source`, `event_time_future_adjusted`. `skipped_missing_fields` means the fields topic carried an envelope without flattened fields and should be treated as a stream-contract issue. The last status means device event time was more than 10 minutes ahead of received time and stats used received time instead. |
| `vehicle_fast_writer_nats_consumer_pending` | JetStream messages pending for the fast writer durable consumer. |
| `vehicle_fast_writer_nats_consumer_ack_pending` | JetStream messages delivered to fast writer but not yet acked. |
| `vehicle_fast_writer_nats_consumer_waiting` | Pull requests waiting on the fast writer durable consumer. |
| `vehicle_fast_writer_batch_pending_messages` | NATS fast writer messages fetched but not yet written to TDengine/Redis and acked. |
| `vehicle_fast_writer_batch_pending_envelopes` | Valid parsed envelopes in the current NATS fast writer batch. |
| `vehicle_fast_writer_messages_total` | Fast writer message results by NATS subject. Labels: `subject`, `status`; status includes `ok`, `error`, `invalid_json`, and `ack_error`. `invalid_json` is acked and isolated so it will not block the raw fast path, but it indicates the stream contract is polluted. |
| `vehicle_fast_writer_stage_duration_ms_histogram` | Fast writer stage duration histogram for TDengine, Redis, and NATS ack. The Redis stage is one batch pipeline when the repository supports batch updates. Labels: `subject`, `stage`, `status`. |
| `vehicle_fast_writer_redis_envelopes_total` | Redis realtime projection envelope result from the NATS fast path. Labels: `subject`, `status`; status includes `seen`, `updated`, `skipped_non_realtime`, `skipped_missing_vin`, `skipped_missing_vehicle_key`, `skipped_missing_fields`. Missing fields is an ingress contract failure: RAW is retained, but Redis online/KV projection is skipped and never re-flattened downstream. |
| `vehicle_fast_writer_redis_fields_total` | Redis realtime KV field write result from the NATS fast path. Labels: `subject`, `status`; status includes `seen`, `written`, `skipped_stale`. `skipped_stale` means an older event-time frame was allowed to add missing fields but was blocked from overwriting newer field values. |
| `vehicle_fast_writer_last_message_unix_seconds` | Last fast-writer message processing result by subject/status. Labels: `subject`, `status`. |
| `vehicle_fast_writer_last_stage_unix_seconds` | Last fast-writer stage completion by subject/stage/status. Labels: `subject`, `stage`, `status`. |
| `vehicle_history_writes_total` | TDengine history writes for valid envelopes only. Labels: `topic`, `status`; invalid JSON messages are committed after `vehicle_history_kafka_messages_total{status="invalid_json"}` and are not counted as writes. |
| `vehicle_history_parsed_fields_total` | Realtime raw envelopes that did or did not carry precomputed `parsed_fields` before TDengine history write. Labels: `topic`, `protocol`, `status`; status includes `present`, `missing`. |
| `vehicle_history_last_message_unix_seconds` | Last history Kafka message by topic/status. Labels: `topic`, `status`. |
| `vehicle_history_last_write_unix_seconds` | Last history TDengine write result by topic/status. Labels: `topic`, `status`. |
| `vehicle_history_last_commit_unix_seconds` | Last history Kafka commit by topic/status. Labels: `topic`, `status`. |
| `vehicle_history_batch_pending_messages` | Messages fetched by history writer but not yet flushed and committed. |
| `vehicle_history_batch_pending_rows` | Parsed raw envelopes waiting in the current TDengine batch. |
| `vehicle_history_retry_pending_messages` | Fetched messages retained in memory behind a failed TDengine write or Kafka commit. Healthy value is `0`; the writer does not fetch a new batch while this is non-zero. |
| `vehicle_history_batch_retries_total` | Failure-closed history batch retries. Labels: `reason`; `write_error` retries only the uncommitted suffix and `commit_error` retries commit without replaying TDengine writes. |
| `vehicle_history_batch_flush_duration_ms` | Last TDengine batch flush duration. Labels: `status`. |
| `vehicle_history_config{setting="workers"}` | Configured in-process history Kafka consumer count; default and capacity minimum are `3`. |
| `vehicle_history_worker_active` | Active history consumer gauge by `worker`; all configured workers should be `1`. |
| `vehicle_stat_writes_total` | MySQL metric writes. Labels: `topic`, `status`. |
| `vehicle_stat_kafka_messages_total` | Stat writer Kafka message results by fields topic. Labels: `topic`, `status`; `invalid_json` messages are committed and isolated before MySQL mileage statistics. |
| `vehicle_stat_last_message_unix_seconds` | Last stat Kafka message by topic/status. Labels: `topic`, `status`. |
| `vehicle_stat_last_write_unix_seconds` | Last stat MySQL append result by topic/status. Labels: `topic`, `status`. |
| `vehicle_stat_last_commit_unix_seconds` | Last stat Kafka commit by topic/status. Labels: `topic`, `status`. |
| `vehicle_stat_write_duration_ms_histogram` | MySQL stat writer append duration histogram. Labels: `topic`, `status`. |
| `vehicle_stat_config{setting="workers"}` | Configured in-process Kafka consumer count. Default and capacity-check minimum are `3`. |
| `vehicle_stat_worker_active` | Active stat consumer gauge by `worker`; all configured workers should remain `1` while the service is running. |
| `vehicle_stat_batch_pending_messages` | Messages fetched by stat writer but not yet appended to MySQL and committed. Controlled by `STATS_BATCH_SIZE` and `STATS_BATCH_WAIT_MS`; capacity check default threshold is `1000`. |
| `vehicle_stat_retry_pending_messages` | Fetched messages retained in memory behind a failed MySQL write or Kafka commit. Healthy value is `0`. |
| `vehicle_stat_batch_retries_total` | Failure-closed stat batch retries. Labels: `reason`; commit-only retries do not append the mileage sample twice in the same process. |
| `vehicle_stat_cache_entries` | Stat writer in-memory cache entries by cache type. Labels: `cache`; cache includes `last_total_mileage`, `source_seen`, `projection`, and `baseline`. Runtime cap is controlled by `STATS_CACHE_MAX_ENTRIES` and defaults to `1000000`. |
| `vehicle_stat_cache_max_entries` | Stat writer configured per-cache entry cap. `0` means unlimited and should not be used in production without an external memory guard. |
| `vehicle_stat_cache_evictions_total` | Cumulative stat writer cache evictions caused by capacity pressure. Labels: `cache`. |
| `vehicle_stat_samples_total` | Mileage samples extracted and written by stat writer. Use this with `vehicle_stat_writes_total`: write `ok` only means the append call succeeded, while `status="written"` proves a sample reached `vehicle_daily_mileage_source`. |
| `vehicle_stat_sources_total` | Stat writer source tracking results for `vehicle_data_source`. Labels: `topic`, `protocol`, `status`; status includes `attempted`, `written`, `skipped_throttled`, `skipped_missing_endpoint`. This intentionally avoids source IP labels to keep metrics low-cardinality. |
| `vehicle_stat_projections_total` | Final daily mileage projection results from `vehicle_daily_mileage_source` to `vehicle_daily_mileage`. Labels: `topic`, `protocol`, `status`; status includes `attempted`, `written`, `skipped_throttled`. |
| `vehicle_realtime_updates_total` | Redis/MySQL realtime projector updates. Labels: `topic`, `status`. |
| `vehicle_realtime_kafka_messages_total` | Realtime projector Kafka message results by raw topic. Labels: `topic`, `status`; `invalid_json` messages are committed and isolated before Redis/MySQL projection. |
| `vehicle_realtime_last_message_unix_seconds` | Last realtime Kafka message by topic/status. Labels: `topic`, `status`. |
| `vehicle_realtime_last_update_unix_seconds` | Last realtime projection result by topic/status. Labels: `topic`, `status`. |
| `vehicle_realtime_last_commit_unix_seconds` | Last realtime Kafka commit by topic/status. Labels: `topic`, `status`. |
| `vehicle_realtime_store_update_duration_ms_histogram` | Redis/MySQL store update duration histogram. Labels: `store`, `protocol`, `status`. |
| `vehicle_realtime_config{setting="workers"}` | Configured in-process realtime Kafka consumer count; default and capacity minimum are `3`. |
| `vehicle_realtime_worker_active` | Active realtime consumer gauge by `worker`; all configured workers should be `1`. |
| `vehicle_realtime_async_queue_total` | Realtime API async secondary projection queue events for MySQL snapshot/location. Labels: `store`, `protocol`, `status`; status includes `queued`, `dropped`, `closed`. |
| `vehicle_realtime_async_queue_depth` | Realtime API async MySQL projection queue depth. Labels: `store`, `protocol`. |
| `vehicle_realtime_async_queue_capacity` | Realtime API async MySQL projection queue capacity. Labels: `store`; controlled by `MYSQL_REALTIME_ASYNC_QUEUE_SIZE`. |
| `vehicle_realtime_retry_pending_messages` | Fetched messages retained in memory behind a failed realtime projection or Kafka commit. Healthy value is `0`. |
| `vehicle_realtime_batch_retries_total` | Failure-closed realtime batch retries. Labels: `reason`; only the uncommitted suffix is projected again after a write failure. |
| `vehicle_identity_writer_retry_pending_messages` | JT808 raw messages retained behind a failed registration MySQL transaction or Kafka commit. Healthy value is `0`; identity writer does not fetch newer offsets while non-zero. |
| `vehicle_identity_writer_config{setting="workers"}` | Configured identity Kafka consumer count; default and capacity minimum are `3`. |
| `vehicle_identity_writer_worker_active` | Active identity consumer gauge by `worker`; every configured worker should remain `1`. |
| `vehicle_identity_writer_batch_retries_total` | Failure-closed identity retries. Labels: `reason`; `write_error` retries the transaction and `commit_error` retries only Kafka commit. |
| `vehicle_realtime_plate_cache_entries` | Realtime API VIN-to-plate binding cache entry count. Default runtime cap is controlled by `PLATE_CACHE_MAX_ENTRIES` and defaults to `200000`. |
| `vehicle_realtime_plate_cache_max_entries` | Realtime API configured VIN-to-plate cache entry cap. `0` means unlimited and should not be used in production without an external memory guard. |
| `vehicle_realtime_plate_cache_evictions_total` | Cumulative VIN-to-plate cache evictions caused by TTL expiry cleanup or capacity pressure. |
| `vehicle_history_kafka_lag` | Estimated Kafka lag for history writer by topic and partition. |
| `vehicle_stat_kafka_lag` | Estimated Kafka lag for stat writer by topic and partition. |
| `vehicle_realtime_kafka_lag` | Estimated Kafka lag for realtime projector by topic and partition. |

## First Principle

These metrics are operational telemetry. They should stay out of MySQL and TDengine business tables unless a product query explicitly requires persisted aggregates. Persisted aggregate metrics should be derived from Kafka replay rather than by widening realtime or raw tables.