59 lines
3.6 KiB
Markdown
59 lines
3.6 KiB
Markdown
# Go Service Observability
|
|
|
|
## ECS
|
|
|
|
Host: `115.29.187.205`
|
|
|
|
The Go services expose local-only health and metrics endpoints. They are intended for ECS-local checks, Prometheus scraping through an agent, or SSH troubleshooting. They are not public business APIs.
|
|
|
|
## Endpoints
|
|
|
|
| Service | systemd unit | Address | Health | Readiness | Metrics |
|
|
| --- | --- | --- | --- | --- | --- |
|
|
| Gateway | `lingniu-go-gateway.service` | `127.0.0.1:20211` | `/healthz` | `/readyz` | `/metrics` |
|
|
| History writer | `lingniu-go-history-writer.service` | `127.0.0.1:20212` | `/healthz` | `/readyz` | `/metrics` |
|
|
| Stat writer | `lingniu-go-stat-writer.service` | `127.0.0.1:20213` | `/healthz` | `/readyz` | `/metrics` |
|
|
| NATS Kafka bridge | `lingniu-go-nats-kafka-bridge.service` | `127.0.0.1:20214` | `/healthz` | `/readyz` | `/metrics` |
|
|
| Realtime API | `lingniu-go-realtime-api.service` | `0.0.0.0:20200` | `/healthz` | `/readyz` | `/metrics` |
|
|
|
|
## Quick Checks
|
|
|
|
生产事故处理顺序和阈值解释见 [车辆接入生产运行手册](vehicle-ingest-runbook.md)。
|
|
|
|
```bash
|
|
curl -fsS http://127.0.0.1:20211/readyz
|
|
curl -fsS http://127.0.0.1:20211/metrics
|
|
curl -fsS http://127.0.0.1:20212/readyz
|
|
curl -fsS http://127.0.0.1:20213/readyz
|
|
curl -fsS http://127.0.0.1:20214/readyz
|
|
curl -fsS http://127.0.0.1:20200/readyz
|
|
```
|
|
|
|
## Core Counters
|
|
|
|
| Metric | Meaning |
|
|
| --- | --- |
|
|
| `vehicle_gateway_active_connections` | Current TCP connections by protocol. Labels: `protocol`. |
|
|
| `vehicle_gateway_frames_total` | Protocol frames received and parsed by the gateway. Labels: `protocol`, `status`. |
|
|
| `vehicle_gateway_frame_duration_ms` | Last observed Gateway frame handling duration and histogram buckets for p95/p99. Labels: `protocol`, `status`. |
|
|
| `vehicle_gateway_publish_total` | Gateway publish result for raw events. Unified appears only when the compatibility stream is explicitly enabled. Labels: `protocol`, `kind`, `status`. |
|
|
| `vehicle_bridge_messages_total` | NATS messages fetched by the bridge. Labels: `subject`, `status`. |
|
|
| `vehicle_bridge_kafka_writes_total` | Bridge writes to Kafka. Labels: `topic`, `status`. |
|
|
| `vehicle_bridge_nats_acks_total` | NATS ack results after Kafka write. Labels: `subject`, `status`. |
|
|
| `vehicle_bridge_nats_consumer_pending` | JetStream messages pending for the bridge durable consumer. |
|
|
| `vehicle_bridge_nats_consumer_ack_pending` | JetStream messages delivered to bridge but not yet acked. |
|
|
| `vehicle_bridge_nats_consumer_waiting` | Pull requests waiting on the bridge durable consumer. |
|
|
| `vehicle_history_writes_total` | TDengine history writes. Labels: `topic`, `status`. |
|
|
| `vehicle_history_batch_pending_messages` | Messages fetched by history writer but not yet flushed and committed. |
|
|
| `vehicle_history_batch_pending_rows` | Parsed raw envelopes waiting in the current TDengine batch. |
|
|
| `vehicle_history_batch_flush_duration_ms` | Last TDengine batch flush duration. Labels: `status`. |
|
|
| `vehicle_stat_writes_total` | MySQL metric writes. Labels: `topic`, `status`. |
|
|
| `vehicle_realtime_updates_total` | Redis/MySQL realtime projector updates. Labels: `topic`, `status`. |
|
|
| `vehicle_history_kafka_lag` | Estimated Kafka lag for history writer by topic and partition. |
|
|
| `vehicle_stat_kafka_lag` | Estimated Kafka lag for stat writer by topic and partition. |
|
|
| `vehicle_realtime_kafka_lag` | Estimated Kafka lag for realtime projector by topic and partition. |
|
|
|
|
## First Principle
|
|
|
|
These metrics are operational telemetry. They should stay out of MySQL and TDengine business tables unless a product query explicitly requires persisted aggregates. Persisted aggregate metrics should be derived from Kafka replay rather than by widening realtime or raw tables.
|