lingniu-vehicle-ingest/docs/ops/go-service-observability.md

Go Service Observability

ECS

Host: 115.29.187.205

The Go services expose local-only health and metrics endpoints. They are intended for ECS-local checks, Prometheus scraping through an agent, or SSH troubleshooting. They are not public business APIs.

Endpoints

Service	systemd unit	Address	Health	Readiness	Metrics
Gateway	lingniu-go-gateway.service	127.0.0.1:20211	/healthz	/readyz	/metrics
History writer	lingniu-go-history-writer.service	127.0.0.1:20212	/healthz	/readyz	/metrics
Stat writer	lingniu-go-stat-writer.service	127.0.0.1:20213	/healthz	/readyz	/metrics
NATS Kafka bridge	lingniu-go-nats-kafka-bridge.service	127.0.0.1:20214	/healthz	/readyz	/metrics
Realtime API	lingniu-go-realtime-api.service	0.0.0.0:20200	/healthz	/readyz	/metrics

Quick Checks

生产事故处理顺序和阈值解释见 车辆接入生产运行手册。

curl -fsS http://127.0.0.1:20211/readyz
curl -fsS http://127.0.0.1:20211/metrics
curl -fsS http://127.0.0.1:20212/readyz
curl -fsS http://127.0.0.1:20213/readyz
curl -fsS http://127.0.0.1:20214/readyz
curl -fsS http://127.0.0.1:20200/readyz

本机容量健康摘要：

/opt/lingniu-go-native/current/capacity-check

它会抓取 Gateway、History writer、Stat writer、NATS bridge、Realtime API、NATS fast writer 的本地 /metrics，输出 JSON。退出码 0 表示当前关键 backlog 和拒绝计数正常，退出码 2 表示存在 pending、Kafka lag、连接拒绝或 metrics 抓取失败，适合接入 cron/告警。

ECS 上通过 systemd timer 每分钟执行一次：

systemctl status lingniu-go-capacity-check.timer
systemctl list-timers lingniu-go-capacity-check.timer
journalctl -u lingniu-go-capacity-check.service --since '10 minutes ago' --no-pager
systemctl start lingniu-go-capacity-check.service

lingniu-go-capacity-check.service 是 oneshot 服务。容量健康时退出码为 0；不健康时退出码为 2，timer 会保留 failed 结果，JSON findings 会写入 journal。

Core Counters

Metric	Meaning
vehicle_gateway_active_connections	Current TCP connections by protocol. Labels: protocol.
vehicle_gateway_connection_closes_total	TCP connection closes by protocol and reason. Labels: protocol, reason. Reasons include eof, read_timeout, read_error, extract_error, context_cancelled, max_connections.
vehicle_gateway_connection_rejections_total	TCP connection rejections by protocol and reason. Labels: protocol, reason.
vehicle_gateway_frames_total	Protocol frames received and parsed by the gateway. Labels: protocol, status.
vehicle_gateway_frame_duration_ms	Last observed Gateway frame handling duration and histogram buckets for p95/p99. Labels: protocol, status.
vehicle_gateway_publish_total	Gateway publish result for raw events. Unified appears only when the compatibility stream is explicitly enabled. Labels: protocol, kind, status.
vehicle_async_sink_publish_duration_ms_histogram	Async sink publish duration histogram covering worker publish calls to the delegate sink. Labels: sink, kind, status.
vehicle_bridge_messages_total	NATS messages fetched by the bridge. Labels: subject, status.
vehicle_bridge_kafka_writes_total	Bridge writes to Kafka. Labels: topic, status.
vehicle_bridge_nats_acks_total	NATS ack results after Kafka write. Labels: subject, status.
vehicle_bridge_nats_consumer_pending	JetStream messages pending for the bridge durable consumer.
vehicle_bridge_nats_consumer_ack_pending	JetStream messages delivered to bridge but not yet acked.
vehicle_bridge_nats_consumer_waiting	Pull requests waiting on the bridge durable consumer.
vehicle_bridge_batch_pending_messages	NATS messages fetched by the bridge but not yet written to Kafka and acked.
vehicle_bridge_batch_pending_kafka_messages	Kafka messages prepared for the current bridge batch.
vehicle_bridge_batch_duration_ms_histogram	Bridge batch duration histogram covering Kafka write and NATS ack. Labels: status.
vehicle_fast_writer_nats_consumer_pending	JetStream messages pending for the fast writer durable consumer.
vehicle_fast_writer_nats_consumer_ack_pending	JetStream messages delivered to fast writer but not yet acked.
vehicle_fast_writer_nats_consumer_waiting	Pull requests waiting on the fast writer durable consumer.
vehicle_fast_writer_batch_pending_messages	NATS fast writer messages fetched but not yet written to TDengine/Redis and acked.
vehicle_fast_writer_batch_pending_envelopes	Valid parsed envelopes in the current NATS fast writer batch.
vehicle_fast_writer_stage_duration_ms_histogram	Fast writer stage duration histogram for TDengine, Redis, and NATS ack. The Redis stage is one batch pipeline when the repository supports batch updates. Labels: subject, stage, status.
vehicle_history_writes_total	TDengine history writes. Labels: topic, status.
vehicle_history_batch_pending_messages	Messages fetched by history writer but not yet flushed and committed.
vehicle_history_batch_pending_rows	Parsed raw envelopes waiting in the current TDengine batch.
vehicle_history_batch_flush_duration_ms	Last TDengine batch flush duration. Labels: status.
vehicle_stat_writes_total	MySQL metric writes. Labels: topic, status.
vehicle_stat_write_duration_ms_histogram	MySQL stat writer append duration histogram. Labels: topic, status.
vehicle_realtime_updates_total	Redis/MySQL realtime projector updates. Labels: topic, status.
vehicle_realtime_store_update_duration_ms_histogram	Redis/MySQL store update duration histogram. Labels: store, protocol, status.
vehicle_history_kafka_lag	Estimated Kafka lag for history writer by topic and partition.
vehicle_stat_kafka_lag	Estimated Kafka lag for stat writer by topic and partition.
vehicle_realtime_kafka_lag	Estimated Kafka lag for realtime projector by topic and partition.

First Principle

These metrics are operational telemetry. They should stay out of MySQL and TDengine business tables unless a product query explicitly requires persisted aggregates. Persisted aggregate metrics should be derived from Kafka replay rather than by widening realtime or raw tables.