lingniu-vehicle-ingest/docs/ops/go-service-observability.md

Go Service Observability

ECS

Host: 115.29.187.205

The Go services expose local-only health and metrics endpoints. They are intended for ECS-local checks, Prometheus scraping through an agent, or SSH troubleshooting. They are not public business APIs.

Endpoints

Service	systemd unit	Address	Health	Readiness	Metrics
Gateway	lingniu-go-gateway.service	127.0.0.1:20211	/healthz	/readyz	/metrics
History writer	lingniu-go-history-writer.service	127.0.0.1:20212	/healthz	/readyz	/metrics
Stat writer	lingniu-go-stat-writer.service	127.0.0.1:20213	/healthz	/readyz	/metrics
NATS Kafka bridge	lingniu-go-nats-kafka-bridge.service	127.0.0.1:20214	/healthz	/readyz	/metrics
Identity writer	lingniu-go-identity-writer.service	127.0.0.1:20217	/healthz	/readyz	/metrics
Realtime API	lingniu-go-realtime-api.service	0.0.0.0:20200	/healthz	/readyz	/metrics

Quick Checks

生产事故处理顺序和阈值解释见 车辆接入生产运行手册。

curl -fsS http://127.0.0.1:20211/readyz
curl -fsS http://127.0.0.1:20211/metrics
curl -fsS http://127.0.0.1:20212/readyz
curl -fsS http://127.0.0.1:20213/readyz
curl -fsS http://127.0.0.1:20214/readyz
curl -fsS http://127.0.0.1:20217/readyz
curl -fsS http://127.0.0.1:20200/readyz

本机容量健康摘要：

/opt/lingniu-go-native/current/capacity-check

它会抓取 Gateway、History writer、Stat writer、Identity writer、NATS bridge、Realtime writer/API、NATS fast writer 的本地 /metrics，输出 JSON。退出码 0 表示当前关键 backlog 和拒绝计数正常，退出码 2 表示存在超阈值 pending、Kafka lag、连接拒绝或 metrics 抓取失败，适合接入 cron/告警。

Identity writer 必须消费 vehicle.raw.go.jt808.v1，且 Gateway 指标 vehicle_gateway_jt808_registration_gateway_writes_enabled 必须为 0。两者同时写 jt808_registration 会被 capacity-check 判为重复 writer。

Gateway 的 NATS 接受边界由 WAL outbox 提供。vehicle_durable_outbox_backlog_records 表示已经 fsync 但尚未 PubAck 的记录，vehicle_durable_outbox_inflight 表示正在等待 PubAck 的记录；健康时两者最终回到 0。vehicle_durable_outbox_publish_total 的 submit_error、ack_error、remove_error、close_timeout 默认按 outbox 错误门禁处理。稳定 event ID/NATS MsgId 负责崩溃恢复期间的 JetStream 去重。

默认还会用 histogram 估算链路 p99：Gateway frame 250ms、Gateway identity 100ms、async sink 100ms、bridge batch 5000ms、fast-writer stage 100ms、history flush 1000ms、realtime store 100ms、stat write 100ms；history/stat/realtime/identity 的 Kafka lag 总和超过 100 时降级。Gateway async sink 队列深度默认超过 10000 降级，可用 -async-sink-queue-depth-max 调整；Gateway 身份快照必须 ready 且最近成功刷新时间不能超过 180s，分别用 -gateway-identity-snapshot-required、-gateway-identity-snapshot-max-age-seconds 调整。Gateway 身份解析缓存默认容量阈值是 300000，Stat writer 统计缓存默认容量阈值是 1000000，Realtime API 的 MySQL 写入节流缓存和 VIN 车牌缓存默认容量阈值分别是 1000000 和 200000，可分别用 -gateway-identity-cache-entries-max、-stat-cache-entries-max、-realtime-throttle-cache-entries-max、-realtime-plate-cache-entries-max 调整，设为 0 表示关闭该项容量检查。history、realtime、stat、identity writer 的 worker 数默认都不得低于 3，分别由 -history-workers-min、-realtime-workers-min、-stat-workers-min、-identity-workers-min 检查。低样本数会跳过 p99 判断，阈值设为 0 可临时关闭对应延迟检查。

批处理内部 in-flight pending 默认允许 100 条窗口：-fast-writer-batch-pending-max=100、-history-batch-pending-max=100、-history-rows-pending-max=100、-stat-batch-pending-max=100。这类指标用于发现批写卡住，不再因为高频流量下几十条正在 flush/ack 的瞬时批次直接降级；NATS consumer pending、ack pending 和 Kafka lag 仍按独立阈值判断真实积压。

topic 流转检查默认只抓完全断层：history/realtime 必须消费三类 raw topic，stat 必须消费三类 fields topic；bridge 未配置 subject 的 route_error 默认超过 0 条就降级，可用 -bridge-route-error-max=-1 临时关闭；fast-writer invalid_json 默认超过 0 条就降级，可用 -fast-writer-invalid-json-max=-1 临时关闭；history-writer invalid_json 默认超过 0 条就降级，可用 -history-invalid-json-max=-1 临时关闭；realtime projector invalid_json 默认超过 0 条就降级，可用 -realtime-invalid-json-max=-1 临时关闭；stat-writer invalid_json 默认超过 0 条就降级，可用 -stat-invalid-json-max=-1 临时关闭；某协议 gateway raw publish 达到 100 条，扣除 skipped_non_realtime 后仍然没有 fields publish 时降级；实时 fields 缺失或 publish error 比例超过 20% 时也会降级；某协议 gateway raw/fields publish 达到 1000 条但 bridge 没有写对应 Kafka topic 时降级；某协议 gateway raw publish 达到 1000 条但 fast-writer 没有消费同 raw subject 并成功写 Redis/ack 时降级；某 raw topic bridge 写 Kafka 达到 1000 条但 history 未收到/写入或 realtime 未收到/更新时降级；某 fields topic bridge 写 Kafka 达到 100 条但 stat-writer 未配置或未收到该 topic 时降级。解析质量检查默认在单协议 gateway frame 样本达到 100 条后启用，OK 以外的 PARTIAL/BAD_FRAME/other 比例超过 5% 时降级，用于发现协议解析器或上游报文质量突然恶化。响应质量检查默认在单协议 gateway response 尝试样本达到 100 条后启用，build_error/write_error 比例超过 5% 时降级；skipped 表示该消息类型无需响应，不计入分母。身份解析质量检查默认在单协议 identity 样本达到 100 条后启用，resolved 以外的 unresolved/error/timeout 比例超过 20% 时降级，用于发现 VIN/车牌/phone 映射突然失效。history-writer 会对实时数据帧检查 raw envelope 是否携带 parsed_fields，默认单 topic/protocol 达到 100 帧后，缺失比例超过 5% 降级，用于发现历史证据层丢失扁平化解析字段。Redis 快路径字段质量检查默认在单 raw subject 写入字段达到 1000 个后启用，skipped_stale/seen 超过 20% 时降级，用于发现大面积旧帧补发或设备时间漂移。stat-writer 还会在单个 fields topic 成功 append 达到 100 条但未抽取到任何里程样本时降级，或 VIN 缺失、里程缺失、非正里程、source 缺失这类可行动跳过比例超过 60% 时降级，用于发现字段映射或身份/source 关联断裂；单 topic 找到 100 条里程样本但没有 source tracking，或 source endpoint 缺失比例超过 60% 时也会降级，用于发现 vehicle_data_source 来源管理断层；单 topic 已写入 100 条里程样本但最终 vehicle_daily_mileage 投影写入仍为 0 时也会降级，用于发现 source 候选层到最终查询层的断层；skipped_same_mileage 是重复总里程去重，不计入可行动跳过比例。已出现过的成功/received *_last_*_unix_seconds 如果超过 300s 未刷新，也会降级；刚启动尚未出现过该指标时不会因缺失 last activity 降级。capacity-check JSON 会额外派生 *_last_*_age_seconds，便于直接读取距今秒数。历史位置派生写入错误使用 vehicle_history_last_location_write_unix_seconds{status="error"} 判断是否仍在发生：默认只在最近 300s 内出现错误时降级，进程生命周期累计 counter 继续保留用于审计；可用 -history-location-error-recent-seconds 调整窗口，设为 0 时恢复累计值门禁。特殊环境可用 -required-consumer-topics='' 临时关闭消费 topic 合同检查。

capacity-check 还会默认请求 Realtime API 的 /api/stats/daily-metrics/diagnostics/reasons，把当天每日里程诊断聚合放到 JSON 的 daily_mileage_diagnostics。这项默认只展示 vehicle_total、actionable_issue_total 和原因分布，不会因为存在业务缺口直接退出 2；需要把缺口纳入发布/巡检阻断时，显式传 -daily-mileage-diagnostics-max-actionable=0 或其他阈值。诊断 API 不可访问会被视为观测能力故障并降级；特殊环境可用 -daily-mileage-diagnostics-url='' 关闭。

ECS 上通过 systemd timer 每分钟执行一次：

systemctl status lingniu-go-capacity-check.timer
systemctl list-timers lingniu-go-capacity-check.timer
journalctl -u lingniu-go-capacity-check.service --since '10 minutes ago' --no-pager
systemctl start lingniu-go-capacity-check.service

lingniu-go-capacity-check.service 是 oneshot 服务。容量健康时退出码为 0；不健康时退出码为 2，timer 会保留 failed 结果，JSON findings 会写入 journal。

Core Counters

Metric	Meaning
vehicle_gateway_active_connections	Current TCP connections by protocol. Labels: protocol.
vehicle_gateway_connection_closes_total	TCP connection closes by protocol and reason. Labels: protocol, reason. Reasons include eof, read_timeout, read_error, extract_error, context_cancelled, max_connections.
vehicle_gateway_connection_rejections_total	TCP connection rejections by protocol and reason. Labels: protocol, reason.
vehicle_gateway_frames_total	Protocol frames received and parsed by the gateway. Labels: protocol, status.
vehicle_gateway_last_frame_unix_seconds	Last observed gateway frame time by protocol and parse status. Labels: protocol, status.
vehicle_gateway_frame_duration_ms	Last observed Gateway frame handling duration and histogram buckets for p95/p99. Labels: protocol, status.
vehicle_gateway_identity_total	Gateway identity resolve results. Labels: protocol, status; status includes resolved, unresolved, error, timeout.
vehicle_gateway_identity_duration_ms	Last observed identity resolve duration and histogram buckets for p95/p99. Labels: protocol, status.
vehicle_gateway_identity_cache_entries	Gateway identity resolver cache entry count. Labels: cache; values include runtime lookup/registration/source_code/location_touch and snapshot_binding/snapshot_identifier/snapshot_registration/snapshot_source.
vehicle_gateway_identity_cache_max_entries	Gateway identity resolver cache entry cap per cache. Controlled by IDENTITY_LOOKUP_CACHE_MAX_ENTRIES, default 300000.
vehicle_gateway_identity_snapshot_ready	1 after at least one complete atomic identity snapshot refresh; 0 means frames continue ingesting but unknown JT808 identities cannot be resolved.
vehicle_gateway_identity_snapshot_entries	Current identity snapshot entry counts. Labels: kind = binding, identifier, registration, source.
vehicle_gateway_identity_snapshot_refresh_total	Snapshot refresh attempts. Labels: status = ok or error; an error keeps the last known good snapshot.
vehicle_gateway_identity_snapshot_last_success_unix_seconds	Last successful complete snapshot publication time.
vehicle_gateway_publish_total	Gateway publish/delegation result. In NATS mode raw is published and fields uses status="delegated"; unified appears only when explicitly enabled. Labels: protocol, kind, status.
vehicle_gateway_last_publish_unix_seconds	Last gateway publish time by protocol/kind/status, useful for protocol freshness checks. Labels: protocol, kind, status.
vehicle_gateway_fields_total	Gateway fields eligibility result. Labels: protocol, status; NATS mode uses delegated_to_bridge, while compatibility direct mode may use published/publish_error.
vehicle_gateway_response_total	Gateway protocol response build/write result. Labels: protocol, message_id, status; status includes ok, skipped, build_error, write_error.
vehicle_gateway_last_response_unix_seconds	Last gateway protocol response build/write result time. Labels: protocol, message_id, status.
vehicle_gateway_response_e2e_recent_p99_ms	Bounded recent p99 from frame receipt through successful protocol response. Labels: protocol; capacity target defaults to 100ms.
vehicle_gateway_response_e2e_recent_samples	Samples in the bounded response-latency window. Labels: protocol.
vehicle_gateway_authentication_total	Protocol authentication decisions. Labels: protocol, mode, source, status; source=configured means a configured credential matched, source=device means the JT808 phone's snapshot token matched, and source=none means no credential matched. observe records mismatches without rejecting, while enforce returns a protocol failure and closes the connection.
vehicle_gateway_authentication_mode	Active authentication mode by protocol. Labels: protocol, mode.
vehicle_gateway_authentication_credentials	Configured account count, never credential values. Labels: protocol.
vehicle_async_sink_publish_duration_ms_histogram	Async sink publish duration histogram covering worker publish calls to the delegate sink. Labels: sink, kind, status.
vehicle_async_sink_queue_capacity	Gateway async sink queue capacity. Labels: sink. NATS uses NATS_ASYNC_QUEUE_SIZE; Kafka fallback uses KAFKA_ASYNC_QUEUE_SIZE.
vehicle_bridge_messages_total	NATS messages fetched by the bridge. Labels: subject, status.
vehicle_bridge_fields_projection_total	Fields projection result from canonical raw. Labels: protocol, status; published is healthy, error/skip labels explain why no fields event was emitted.
vehicle_bridge_fields_projection_count	Latest flattened field count emitted by bridge per protocol/status.
vehicle_bridge_last_message_unix_seconds	Last NATS message fetched by subject/status. Labels: subject, status.
vehicle_bridge_kafka_writes_total	Bridge writes to Kafka. Labels: topic, status.
vehicle_bridge_last_kafka_write_unix_seconds	Last bridge Kafka write by topic/status. Labels: topic, status.
vehicle_bridge_nats_acks_total	NATS ack results after Kafka write or unrouted-subject isolation. Labels: subject, status; status includes ok, error, and dropped_route_error.
vehicle_bridge_last_ack_unix_seconds	Last NATS ack by subject/status. Labels: subject, status.
vehicle_bridge_nats_consumer_pending	JetStream messages pending for the bridge durable consumer.
vehicle_bridge_nats_consumer_ack_pending	JetStream messages delivered to bridge but not yet acked.
vehicle_bridge_nats_consumer_waiting	Pull requests waiting on the bridge durable consumer.
vehicle_bridge_batch_pending_messages	NATS messages fetched by the bridge but not yet written to Kafka and acked.
vehicle_bridge_batch_pending_kafka_messages	Kafka messages prepared for the current bridge batch.
vehicle_bridge_batch_duration_ms_histogram	Bridge batch duration histogram covering Kafka write and NATS ack. Labels: status.
vehicle_history_batch_flush_duration_ms_histogram	History writer TDengine batch flush duration histogram. Labels: status.
vehicle_realtime_store_update_duration_ms_histogram	Realtime MySQL/Redis projection duration histogram. Labels: store, protocol, status.
vehicle_stat_write_duration_ms_histogram	Stat writer MySQL write duration histogram. Labels: topic, status.
vehicle_stat_samples_total	Stat writer mileage sample results. Labels: topic, protocol, status; status includes found, written, skipped_missing_fields, skipped_missing_vin, skipped_missing_mileage, skipped_non_positive_mileage, skipped_same_mileage, skipped_missing_source, event_time_future_adjusted. skipped_missing_fields means the fields topic carried an envelope without flattened fields and should be treated as a stream-contract issue. The last status means device event time was more than 10 minutes ahead of received time and stats used received time instead.
vehicle_fast_writer_nats_consumer_pending	JetStream messages pending for the fast writer durable consumer.
vehicle_fast_writer_nats_consumer_ack_pending	JetStream messages delivered to fast writer but not yet acked.
vehicle_fast_writer_nats_consumer_waiting	Pull requests waiting on the fast writer durable consumer.
vehicle_fast_writer_batch_pending_messages	NATS fast writer messages fetched but not yet written to TDengine/Redis and acked.
vehicle_fast_writer_batch_pending_envelopes	Valid parsed envelopes in the current NATS fast writer batch.
vehicle_fast_writer_messages_total	Fast writer message results by NATS subject. Labels: subject, status; status includes ok, error, invalid_json, and ack_error. invalid_json is acked and isolated so it will not block the raw fast path, but it indicates the stream contract is polluted.
vehicle_fast_writer_stage_duration_ms_histogram	Fast writer stage duration histogram for TDengine, Redis, and NATS ack. The Redis stage is one batch pipeline when the repository supports batch updates. Labels: subject, stage, status.
vehicle_fast_writer_redis_envelopes_total	Redis realtime projection envelope result from the NATS fast path. Labels: subject, status; status includes seen, updated, skipped_non_realtime, skipped_missing_vin, skipped_missing_vehicle_key, skipped_missing_fields. Missing fields is an ingress contract failure: RAW is retained, but Redis online/KV projection is skipped and never re-flattened downstream.
vehicle_fast_writer_redis_fields_total	Redis realtime KV field write result from the NATS fast path. Labels: subject, status; status includes seen, written, skipped_stale. skipped_stale means an older event-time frame was allowed to add missing fields but was blocked from overwriting newer field values.
vehicle_fast_writer_last_message_unix_seconds	Last fast-writer message processing result by subject/status. Labels: subject, status.
vehicle_fast_writer_last_stage_unix_seconds	Last fast-writer stage completion by subject/stage/status. Labels: subject, stage, status.
vehicle_history_writes_total	TDengine history writes for valid envelopes only. Labels: topic, status; invalid JSON messages are committed after vehicle_history_kafka_messages_total{status="invalid_json"} and are not counted as writes.
vehicle_history_parsed_fields_total	Realtime raw envelopes that did or did not carry precomputed parsed_fields before TDengine history write. Labels: topic, protocol, status; status includes present, missing.
vehicle_history_last_message_unix_seconds	Last history Kafka message by topic/status. Labels: topic, status.
vehicle_history_last_write_unix_seconds	Last history TDengine write result by topic/status. Labels: topic, status.
vehicle_history_last_commit_unix_seconds	Last history Kafka commit by topic/status. Labels: topic, status.
vehicle_history_batch_pending_messages	Messages fetched by history writer but not yet flushed and committed.
vehicle_history_batch_pending_rows	Parsed raw envelopes waiting in the current TDengine batch.
vehicle_history_retry_pending_messages	Fetched messages retained in memory behind a failed TDengine write or Kafka commit. Healthy value is 0; the writer does not fetch a new batch while this is non-zero.
vehicle_history_batch_retries_total	Failure-closed history batch retries. Labels: reason; write_error retries only the uncommitted suffix and commit_error retries commit without replaying TDengine writes.
vehicle_history_batch_flush_duration_ms	Last TDengine batch flush duration. Labels: status.
vehicle_history_config{setting="workers"}	Configured in-process history Kafka consumer count; default and capacity minimum are 3.
vehicle_history_worker_active	Active history consumer gauge by worker; all configured workers should be 1.
vehicle_stat_writes_total	MySQL metric writes. Labels: topic, status.
vehicle_stat_kafka_messages_total	Stat writer Kafka message results by fields topic. Labels: topic, status; invalid_json messages are committed and isolated before MySQL mileage statistics.
vehicle_stat_last_message_unix_seconds	Last stat Kafka message by topic/status. Labels: topic, status.
vehicle_stat_last_write_unix_seconds	Last stat MySQL append result by topic/status. Labels: topic, status.
vehicle_stat_last_commit_unix_seconds	Last stat Kafka commit by topic/status. Labels: topic, status.
vehicle_stat_write_duration_ms_histogram	MySQL stat writer append duration histogram. Labels: topic, status.
vehicle_stat_config{setting="workers"}	Configured in-process Kafka consumer count. Default and capacity-check minimum are 3.
vehicle_stat_worker_active	Active stat consumer gauge by worker; all configured workers should remain 1 while the service is running.
vehicle_stat_batch_pending_messages	Messages fetched by stat writer but not yet appended to MySQL and committed. Controlled by STATS_BATCH_SIZE and STATS_BATCH_WAIT_MS; capacity check default threshold is 1000.
vehicle_stat_retry_pending_messages	Fetched messages retained in memory behind a failed MySQL write or Kafka commit. Healthy value is 0.
vehicle_stat_batch_retries_total	Failure-closed stat batch retries. Labels: reason; commit-only retries do not append the mileage sample twice in the same process.
vehicle_stat_cache_entries	Stat writer in-memory cache entries by cache type. Labels: cache; cache includes last_total_mileage, source_seen, projection, and baseline. Runtime cap is controlled by STATS_CACHE_MAX_ENTRIES and defaults to 1000000.
vehicle_stat_cache_max_entries	Stat writer configured per-cache entry cap. 0 means unlimited and should not be used in production without an external memory guard.
vehicle_stat_cache_evictions_total	Cumulative stat writer cache evictions caused by capacity pressure. Labels: cache.
vehicle_stat_samples_total	Mileage samples extracted and written by stat writer. Use this with vehicle_stat_writes_total: write ok only means the append call succeeded, while status="written" proves a sample reached vehicle_daily_mileage_source.
vehicle_stat_sources_total	Stat writer source tracking results for vehicle_data_source. Labels: topic, protocol, status; status includes attempted, written, skipped_throttled, skipped_missing_endpoint. This intentionally avoids source IP labels to keep metrics low-cardinality.
vehicle_stat_projections_total	Final daily mileage projection results from vehicle_daily_mileage_source to vehicle_daily_mileage. Labels: topic, protocol, status; status includes attempted, written, skipped_throttled.
vehicle_realtime_updates_total	Redis/MySQL realtime projector updates. Labels: topic, status.
vehicle_realtime_kafka_messages_total	Realtime projector Kafka message results by raw topic. Labels: topic, status; invalid_json messages are committed and isolated before Redis/MySQL projection.
vehicle_realtime_last_message_unix_seconds	Last realtime Kafka message by topic/status. Labels: topic, status.
vehicle_realtime_last_update_unix_seconds	Last realtime projection result by topic/status. Labels: topic, status.
vehicle_realtime_last_commit_unix_seconds	Last realtime Kafka commit by topic/status. Labels: topic, status.
vehicle_realtime_store_update_duration_ms_histogram	Redis/MySQL store update duration histogram. Labels: store, protocol, status.
vehicle_realtime_config{setting="workers"}	Configured in-process realtime Kafka consumer count; default and capacity minimum are 3.
vehicle_realtime_worker_active	Active realtime consumer gauge by worker; all configured workers should be 1.
vehicle_realtime_async_queue_total	Realtime API async secondary projection queue events for MySQL snapshot/location. Labels: store, protocol, status; status includes queued, dropped, closed.
vehicle_realtime_async_queue_depth	Realtime API async MySQL projection queue depth. Labels: store, protocol.
vehicle_realtime_async_queue_capacity	Realtime API async MySQL projection queue capacity. Labels: store; controlled by MYSQL_REALTIME_ASYNC_QUEUE_SIZE.
vehicle_realtime_retry_pending_messages	Fetched messages retained in memory behind a failed realtime projection or Kafka commit. Healthy value is 0.
vehicle_realtime_batch_retries_total	Failure-closed realtime batch retries. Labels: reason; only the uncommitted suffix is projected again after a write failure.
vehicle_identity_writer_retry_pending_messages	JT808 raw messages retained behind a failed registration MySQL transaction or Kafka commit. Healthy value is 0; identity writer does not fetch newer offsets while non-zero.
vehicle_identity_writer_config{setting="workers"}	Configured identity Kafka consumer count; default and capacity minimum are 3.
vehicle_identity_writer_worker_active	Active identity consumer gauge by worker; every configured worker should remain 1.
vehicle_identity_writer_batch_retries_total	Failure-closed identity retries. Labels: reason; write_error retries the transaction and commit_error retries only Kafka commit.
vehicle_realtime_plate_cache_entries	Realtime API VIN-to-plate binding cache entry count. Default runtime cap is controlled by PLATE_CACHE_MAX_ENTRIES and defaults to 200000.
vehicle_realtime_plate_cache_max_entries	Realtime API configured VIN-to-plate cache entry cap. 0 means unlimited and should not be used in production without an external memory guard.
vehicle_realtime_plate_cache_evictions_total	Cumulative VIN-to-plate cache evictions caused by TTL expiry cleanup or capacity pressure.
vehicle_history_kafka_lag	Estimated Kafka lag for history writer by topic and partition.
vehicle_stat_kafka_lag	Estimated Kafka lag for stat writer by topic and partition.
vehicle_realtime_kafka_lag	Estimated Kafka lag for realtime projector by topic and partition.

First Principle

These metrics are operational telemetry. They should stay out of MySQL and TDengine business tables unless a product query explicitly requires persisted aggregates. Persisted aggregate metrics should be derived from Kafka replay rather than by widening realtime or raw tables.