92 lines
6.5 KiB
Markdown
92 lines
6.5 KiB
Markdown
# Go Service Observability
|
||
|
||
## ECS
|
||
|
||
Host: `115.29.187.205`
|
||
|
||
The Go services expose local-only health and metrics endpoints. They are intended for ECS-local checks, Prometheus scraping through an agent, or SSH troubleshooting. They are not public business APIs.
|
||
|
||
## Endpoints
|
||
|
||
| Service | systemd unit | Address | Health | Readiness | Metrics |
|
||
| --- | --- | --- | --- | --- | --- |
|
||
| Gateway | `lingniu-go-gateway.service` | `127.0.0.1:20211` | `/healthz` | `/readyz` | `/metrics` |
|
||
| History writer | `lingniu-go-history-writer.service` | `127.0.0.1:20212` | `/healthz` | `/readyz` | `/metrics` |
|
||
| Stat writer | `lingniu-go-stat-writer.service` | `127.0.0.1:20213` | `/healthz` | `/readyz` | `/metrics` |
|
||
| NATS Kafka bridge | `lingniu-go-nats-kafka-bridge.service` | `127.0.0.1:20214` | `/healthz` | `/readyz` | `/metrics` |
|
||
| Realtime API | `lingniu-go-realtime-api.service` | `0.0.0.0:20200` | `/healthz` | `/readyz` | `/metrics` |
|
||
|
||
## Quick Checks
|
||
|
||
生产事故处理顺序和阈值解释见 [车辆接入生产运行手册](vehicle-ingest-runbook.md)。
|
||
|
||
```bash
|
||
curl -fsS http://127.0.0.1:20211/readyz
|
||
curl -fsS http://127.0.0.1:20211/metrics
|
||
curl -fsS http://127.0.0.1:20212/readyz
|
||
curl -fsS http://127.0.0.1:20213/readyz
|
||
curl -fsS http://127.0.0.1:20214/readyz
|
||
curl -fsS http://127.0.0.1:20200/readyz
|
||
```
|
||
|
||
本机容量健康摘要：
|
||
|
||
```bash
|
||
/opt/lingniu-go-native/current/capacity-check
|
||
```
|
||
|
||
它会抓取 Gateway、History writer、Stat writer、NATS bridge、Realtime API、NATS fast writer 的本地 `/metrics`，输出 JSON。退出码 `0` 表示当前关键 backlog 和拒绝计数正常，退出码 `2` 表示存在 pending、Kafka lag、连接拒绝或 metrics 抓取失败，适合接入 cron/告警。
|
||
|
||
ECS 上通过 systemd timer 每分钟执行一次：
|
||
|
||
```bash
|
||
systemctl status lingniu-go-capacity-check.timer
|
||
systemctl list-timers lingniu-go-capacity-check.timer
|
||
journalctl -u lingniu-go-capacity-check.service --since '10 minutes ago' --no-pager
|
||
systemctl start lingniu-go-capacity-check.service
|
||
```
|
||
|
||
`lingniu-go-capacity-check.service` 是 oneshot 服务。容量健康时退出码为 `0`；不健康时退出码为 `2`，timer 会保留 failed 结果，JSON findings 会写入 journal。
|
||
|
||
## Core Counters
|
||
|
||
| Metric | Meaning |
|
||
| --- | --- |
|
||
| `vehicle_gateway_active_connections` | Current TCP connections by protocol. Labels: `protocol`. |
|
||
| `vehicle_gateway_connection_closes_total` | TCP connection closes by protocol and reason. Labels: `protocol`, `reason`. Reasons include `eof`, `read_timeout`, `read_error`, `extract_error`, `context_cancelled`, `max_connections`. |
|
||
| `vehicle_gateway_connection_rejections_total` | TCP connection rejections by protocol and reason. Labels: `protocol`, `reason`. |
|
||
| `vehicle_gateway_frames_total` | Protocol frames received and parsed by the gateway. Labels: `protocol`, `status`. |
|
||
| `vehicle_gateway_frame_duration_ms` | Last observed Gateway frame handling duration and histogram buckets for p95/p99. Labels: `protocol`, `status`. |
|
||
| `vehicle_gateway_publish_total` | Gateway publish result for raw events. Unified appears only when the compatibility stream is explicitly enabled. Labels: `protocol`, `kind`, `status`. |
|
||
| `vehicle_async_sink_publish_duration_ms_histogram` | Async sink publish duration histogram covering worker publish calls to the delegate sink. Labels: `sink`, `kind`, `status`. |
|
||
| `vehicle_bridge_messages_total` | NATS messages fetched by the bridge. Labels: `subject`, `status`. |
|
||
| `vehicle_bridge_kafka_writes_total` | Bridge writes to Kafka. Labels: `topic`, `status`. |
|
||
| `vehicle_bridge_nats_acks_total` | NATS ack results after Kafka write. Labels: `subject`, `status`. |
|
||
| `vehicle_bridge_nats_consumer_pending` | JetStream messages pending for the bridge durable consumer. |
|
||
| `vehicle_bridge_nats_consumer_ack_pending` | JetStream messages delivered to bridge but not yet acked. |
|
||
| `vehicle_bridge_nats_consumer_waiting` | Pull requests waiting on the bridge durable consumer. |
|
||
| `vehicle_bridge_batch_pending_messages` | NATS messages fetched by the bridge but not yet written to Kafka and acked. |
|
||
| `vehicle_bridge_batch_pending_kafka_messages` | Kafka messages prepared for the current bridge batch. |
|
||
| `vehicle_bridge_batch_duration_ms_histogram` | Bridge batch duration histogram covering Kafka write and NATS ack. Labels: `status`. |
|
||
| `vehicle_fast_writer_nats_consumer_pending` | JetStream messages pending for the fast writer durable consumer. |
|
||
| `vehicle_fast_writer_nats_consumer_ack_pending` | JetStream messages delivered to fast writer but not yet acked. |
|
||
| `vehicle_fast_writer_nats_consumer_waiting` | Pull requests waiting on the fast writer durable consumer. |
|
||
| `vehicle_fast_writer_batch_pending_messages` | NATS fast writer messages fetched but not yet written to TDengine/Redis and acked. |
|
||
| `vehicle_fast_writer_batch_pending_envelopes` | Valid parsed envelopes in the current NATS fast writer batch. |
|
||
| `vehicle_fast_writer_stage_duration_ms_histogram` | Fast writer stage duration histogram for TDengine, Redis, and NATS ack. The Redis stage is one batch pipeline when the repository supports batch updates. Labels: `subject`, `stage`, `status`. |
|
||
| `vehicle_history_writes_total` | TDengine history writes. Labels: `topic`, `status`. |
|
||
| `vehicle_history_batch_pending_messages` | Messages fetched by history writer but not yet flushed and committed. |
|
||
| `vehicle_history_batch_pending_rows` | Parsed raw envelopes waiting in the current TDengine batch. |
|
||
| `vehicle_history_batch_flush_duration_ms` | Last TDengine batch flush duration. Labels: `status`. |
|
||
| `vehicle_stat_writes_total` | MySQL metric writes. Labels: `topic`, `status`. |
|
||
| `vehicle_stat_write_duration_ms_histogram` | MySQL stat writer append duration histogram. Labels: `topic`, `status`. |
|
||
| `vehicle_realtime_updates_total` | Redis/MySQL realtime projector updates. Labels: `topic`, `status`. |
|
||
| `vehicle_realtime_store_update_duration_ms_histogram` | Redis/MySQL store update duration histogram. Labels: `store`, `protocol`, `status`. |
|
||
| `vehicle_history_kafka_lag` | Estimated Kafka lag for history writer by topic and partition. |
|
||
| `vehicle_stat_kafka_lag` | Estimated Kafka lag for stat writer by topic and partition. |
|
||
| `vehicle_realtime_kafka_lag` | Estimated Kafka lag for realtime projector by topic and partition. |
|
||
|
||
## First Principle
|
||
|
||
These metrics are operational telemetry. They should stay out of MySQL and TDengine business tables unless a product query explicitly requires persisted aggregates. Persisted aggregate metrics should be derived from Kafka replay rather than by widening realtime or raw tables.
|