feat: build vehicle data platform and production pipeline
@@ -39,12 +39,38 @@ If a keyword cannot be resolved to a VIN, data APIs must not fabricate a VIN. Th
|
||||
|
||||
## Core Query APIs
|
||||
|
||||
### Unified Metric Catalog
|
||||
|
||||
```http
|
||||
GET /api/v2/metrics
|
||||
```
|
||||
|
||||
Returns the server-owned metric whitelist used by rule configuration and future telemetry/history surfaces. Production definitions come from `vehicle_metric_definition` plus `vehicle_metric_protocol_mapping`; migration `005_metric_catalog.sql` only seeds missing rows, so later database configuration is not overwritten by releases. Each metric includes `key`, localized `label`, `unit`, `category`, `valueType`, supported `protocols`, per-protocol `sourceFields`, and `searchable/chartable/alertable` capabilities. Alert rule writes reject unknown, disabled, non-alertable, type-mismatched, or evaluator-unsupported metrics; clients must not invent metric keys or derive SQL fields from user input. `/api/v2/history/metrics` remains the category/column compatibility catalog for the current history page.
|
||||
|
||||
### Latest vehicle telemetry
|
||||
|
||||
`GET /api/v2/vehicles/{vin}/telemetry/latest`
|
||||
|
||||
The path accepts the same VIN, plate or terminal identity resolution used by the vehicle service. After resolving identity once, the server reads `GB32960`, `JT808` and `YUTONG_MQTT` independently and concurrently, at most five newest frames per protocol and 15 frames overall, without running pagination counts. It then returns the newest scalar value for every unified metric or remaining source field. A field that matches `/api/v2/metrics.sourceFields` uses the unified `key`, localized label, unit, type, description and category while retaining its exact `sourceField`. Manufacturer/protocol extensions remain visible under a server-owned dynamic category instead of being discarded or classified by the browser.
|
||||
|
||||
Every value includes `protocol`, optional `sourceEndpoint`, `frameId`, device and receive times, `freshnessSeconds`, optional `dataDelaySeconds`, and `quality=good|stale|warning` with an explicit `qualityReason`. `good` requires a successful parser status and a receive time within five minutes; `stale` means the newest value is older than five minutes; parse failures, missing/implausibly future receive times or device/receive divergence over five minutes are `warning`. The response also returns populated categories with counts, `scannedFrames`, `staleAfterSeconds`, `asOf` and selection evidence. Object/array fields are deliberately excluded from the compact latest-value surface and remain available through RAW evidence APIs.
|
||||
|
||||
### Dashboard Summary
|
||||
|
||||
```http
|
||||
GET /api/dashboard/summary
|
||||
```
|
||||
|
||||
### V2 Global Monitor
|
||||
|
||||
```http
|
||||
GET /api/v2/monitor/summary?keyword=粤A&protocol=JT808&status=driving
|
||||
GET /api/v2/monitor/map?zoom=5
|
||||
GET /api/v2/monitor/map?zoom=13&bounds=113,22,114,24&status=online
|
||||
```
|
||||
|
||||
`summary` and `map` share keyword, protocol and `online|offline|driving|idle` status semantics. `map` accepts `bounds=minLongitude,minLatitude,maxLongitude,maxLatitude`; malformed, non-finite, reversed or out-of-world bounds return `MONITOR_BOUNDS_INVALID`. Below zoom 11 the response remains clustered or mixed. At zoom 11 and above it returns lightweight MassMarks only when the visible point set is at most 2,000; otherwise it adaptively coarsens cells until no more than 2,000 clusters remain. It never returns full telemetry JSON in a point. The browser keeps the normal list at 200 rows, requests map data independently, waits 300ms after `moveend/zoomend`, and drops stale viewport bounds when a direct vehicle search must locate and focus its unique result.
|
||||
|
||||
Returns vehicle service KPIs, source distribution, vehicle service status distribution, link health, and realtime backlog.
|
||||
|
||||
### Vehicle Service
|
||||
@@ -55,6 +81,24 @@ GET /api/vehicle-service?keyword=粤AG18312
|
||||
|
||||
Returns one vehicle service view with identity, realtime summary, source coverage, history preview, RAW preview, mileage preview, and quality issues.
|
||||
|
||||
The response also carries `profile`, the supplemental business master record from `vehicle_profile`. Gateway-owned VIN, plate and OEM identity remain authoritative and are never overwritten by this record. Profile completeness is calculated across model, vehicle type, company, operation status, access provider, first access time and cumulative runtime.
|
||||
|
||||
```http
|
||||
GET /api/v2/vehicles/{vin}/profile
|
||||
PUT /api/v2/vehicles/{vin}/profile
|
||||
POST /api/v2/vehicle-profiles/sync
|
||||
```
|
||||
|
||||
Profile reads require `viewer`; manual writes and bulk sync require `admin`. A manual write uses the returned `version` for optimistic concurrency, rejects unknown VINs and invalid status/time/runtime values, changes the source to `manual`, and creates a versioned `vehicle_profile_audit` snapshot. Supported operation statuses are `unknown`, `active`, `inactive`, `maintenance`, and `retired`.
|
||||
|
||||
The controlled sync endpoint accepts 1–500 complete supplemental profiles per request with required `sourceSystem` and `sourceVersion`, `conflictPolicy=preserve|overwrite`, and `dryRun`. VIN identity must already exist in the gateway-owned `vehicle_identity_binding`; missing identities are reported per item and never invented. The default `preserve` policy skips manual records and records owned by another external system. `overwrite` is an explicit administrator takeover. Repeating the same source/version and identical payload is `unchanged`; changing a payload under the same source/version is `conflict_source_version`, preventing a mutable upstream snapshot from silently defeating idempotency. Created and updated profiles receive `source_system`, `source_version`, `synced_at`, optimistic profile version, authenticated actor, and an immutable `sync_created` or `sync_updated` audit snapshot. `dryRun=true` uses the same identity, ownership, validation and locking path but rolls back all writes.
|
||||
|
||||
The admin UI accepts UTF-8 CSV with the exact header below, parses quoted fields and validates duplicate VIN, status and integer runtime before upload. It renders only aggregate counts plus the first 20 conflicts, so a 500-row batch does not create a large React table.
|
||||
|
||||
```text
|
||||
vin,modelName,vehicleType,companyName,operationStatus,accessProvider,firstAccessAt,runtimeSeconds
|
||||
```
|
||||
|
||||
`/api/vehicles/detail` remains available as a compatibility alias. New UI and integrations should use `/api/vehicle-service` to make the vehicle-first boundary explicit.
|
||||
|
||||
`sourceConsistency` is the vehicle-level diagnosis across GB32960, JT808, and Yutong MQTT sources. The three protocols are source evidence for one vehicle service, so callers should display the diagnosis result instead of asking users to compare protocol rows manually.
|
||||
@@ -107,6 +151,8 @@ GET /api/history/locations?keyword=粤AG18312&protocol=JT808&dateFrom=2026-07-03
|
||||
|
||||
Returns historical location points from TDengine, enriched with plate where possible.
|
||||
|
||||
Location points include `socPercent`, `directionDeg` and `alarmFlag` when the source protocol supplies them. `socAvailable` distinguishes a real zero from an unavailable historical value. Device and receive timestamps are returned as absolute RFC3339 instants; query bounds must include an explicit offset (for example `+08:00`) so TDengine session timezone cannot shift the requested window.
|
||||
|
||||
For trajectory replay, the BFF should eventually return two shapes from the same evidence set:
|
||||
|
||||
- `items`: paginated table rows for audit and export.
|
||||
@@ -114,6 +160,34 @@ For trajectory replay, the BFF should eventually return two shapes from the same
|
||||
|
||||
This keeps table pagination precise while preventing large routes from overloading the browser map.
|
||||
|
||||
### History Series And Controlled Export
|
||||
|
||||
```http
|
||||
GET /api/v2/history/series?keyword=粤AG18312&dateFrom=2026-07-14T00:00:00%2B08:00&dateTo=2026-07-14T06:00:00%2B08:00&targetPoints=240
|
||||
|
||||
POST /api/v2/exports
|
||||
Content-Type: application/json
|
||||
|
||||
{
|
||||
"keywords": ["粤AG18312"],
|
||||
"category": "location",
|
||||
"protocol": "JT808",
|
||||
"dateFrom": "2026-07-14T00:00:00+08:00",
|
||||
"dateTo": "2026-07-14T06:00:00+08:00",
|
||||
"metrics": ["speed_kmh", "total_mileage_km"],
|
||||
"format": "csv"
|
||||
}
|
||||
|
||||
GET /api/v2/exports
|
||||
GET /api/v2/exports/{id}/download
|
||||
```
|
||||
|
||||
The series endpoint returns server-aggregated location telemetry with explicit bucket coverage and missingness. Requests are limited to 31 days and 60–600 target points; timestamps must identify an absolute window rather than relying on a database session timezone.
|
||||
|
||||
Export creation and listing require `operator` or `admin`. A task accepts 1–5 vehicles, `location`, `raw`, or `mileage`, CSV format, at most 31 days and 32 metrics. The service runs one export at a time and enforces a 1,000,000-row and 30-minute ceiling. Location and RAW data use stable forward-only cursors; the implementation does not use growing `OFFSET` scans.
|
||||
|
||||
Job status values are `queued`, `running`, `completed`, and `failed`. During execution, clients should display the server-owned `processedRows`, `totalRows`, `progress`, and `evidence` fields rather than estimate progress from elapsed time. On completion, `rowCount`, `processedRows`, and `totalRows` are equal, `fileSizeBytes` and `completedAt` are populated, and `downloadUrl` becomes available. The final CSV is exposed only after its temporary `.part` file has been flushed, synchronized, closed, and atomically renamed. Completed jobs survive API restarts; jobs interrupted while queued or running become explicit failures and may be recreated.
|
||||
|
||||
### RAW Frames
|
||||
|
||||
Short GET query:
|
||||
@@ -151,6 +225,8 @@ GET /api/mileage/summary?keyword=粤AG18312&protocol=JT808&dateFrom=2026-07-01&d
|
||||
|
||||
Returns daily mileage rows and aggregate mileage summary.
|
||||
|
||||
The production projection table stores `daily_mileage_km` and `latest_total_mileage_km`; it deliberately no longer stores a duplicate first-total column. The API derives `startMileageKm = latestTotalMileageKm - dailyMileageKm` and normalizes a missing latest value to zero, matching the current gateway storage contract.
|
||||
|
||||
### Alert Events
|
||||
|
||||
Current API surface:
|
||||
@@ -173,14 +249,29 @@ Initial alert types:
|
||||
- `FIELD_MISSING`
|
||||
- `CAPACITY_RISK`
|
||||
|
||||
Future durable alert records should extend this surface with event IDs, manual status, assignee, acknowledgement, escalation state, and notification logs:
|
||||
These legacy aliases remain read-only quality projections. The durable V2 alert center is a separate truth surface:
|
||||
|
||||
```http
|
||||
GET /api/alert-events/{id}
|
||||
PATCH /api/alert-events/{id}
|
||||
GET /api/alert-events/{id}/notifications?limit=20&offset=0
|
||||
POST /api/v2/alerts/summary
|
||||
POST /api/v2/alerts/events
|
||||
GET /api/v2/alerts/events/{id}
|
||||
POST /api/v2/alerts/events/{id}/actions
|
||||
GET /api/v2/alerts/rules
|
||||
POST /api/v2/alerts/rules
|
||||
PUT /api/v2/alerts/rules/{id}
|
||||
PUT /api/v2/alerts/rules/{id}/enabled
|
||||
GET /api/v2/alerts/notifications?unreadOnly=true&limit=20&offset=0
|
||||
POST /api/v2/alerts/notifications/read
|
||||
```
|
||||
|
||||
Rule operators are type-aware: numeric metrics support `gt/gte/lt/lte/eq/neq/between/outside`; Boolean metrics support `eq/neq/changed`. Range rules carry `threshold` and `thresholdHigh`. Scopes support `scopeProtocols`, `scopeVins`, authoritative `scopeOems`, and `scopeModels/scopeCompanies` joined from `vehicle_profile`. Each scope list is capped at 500 normalized values with a 128-character per-value bound, protecting evaluator latency. Vehicles without the requested master-data dimension do not match a model/company-scoped rule. `changed` rules persist the previous Boolean state per rule/VIN/protocol, and repeat suppression uses `repeatIntervalSec` against the latest event fingerprint.
|
||||
|
||||
The rule editor reads its selectable metrics from `GET /api/v2/metrics`. The API repeats the same catalog validation on every create/update, so a stale or manipulated client cannot configure an unsupported evaluator field.
|
||||
|
||||
`summary` and `events` accept the same JSON filter: `keyword, severity, status, ruleId, protocol, dateFrom, dateTo, limit, offset`. Event states are `unprocessed, processing, recovered, closed, ignored`; event changes require the current `version` and actions are `acknowledge, close, ignore`. Every action writes an immutable timeline item with actor, before/after state, note, and timestamp. Stale event or rule writes return HTTP 409 with a `*_VERSION_CONFLICT` code.
|
||||
|
||||
Rules support numeric/Boolean values, bounded duration and repeat interval, recovery hysteresis, protocol/VIN scope, enable state, and a versioned audit snapshot. For telemetry metrics, duration is the difference between distinct, monotonic source event observations rather than time spent rereading one MySQL snapshot; `freshness_sec` is the explicit exception because staleness changes with platform time. Duplicate and late observations are counted but cannot advance a candidate. The evaluator deduplicates active rule+vehicle+protocol fingerprints and automatically recovers only when the configured recovery condition is true. Disabling a rule atomically clears its candidate and Boolean state together with the versioned disable audit. Station notifications have real unread/read state. `sms`, `email`, and `wecom` records are explicitly `reserved`; they are not reported as sent.
|
||||
|
||||
### Online And Completeness Statistics
|
||||
|
||||
Planned API surface:
|
||||
@@ -193,6 +284,49 @@ GET /api/statistics/source-consistency?keyword=粤AG18312&dateFrom=2026-07-01&da
|
||||
|
||||
These APIs should link back to `/api/history/locations` or `/api/history/raw-frames/query` evidence. The UI must not show statistics as untraceable numbers.
|
||||
|
||||
### V2 Access Management
|
||||
|
||||
```http
|
||||
POST /api/v2/access/summary
|
||||
Content-Type: application/json
|
||||
|
||||
{"protocol":"JT808","onlineState":"offline","delayState":"abnormal"}
|
||||
|
||||
POST /api/v2/access/vehicles
|
||||
Content-Type: application/json
|
||||
|
||||
{"keyword":"粤AG18312","protocol":"JT808","model":"ZK6122","provider":"G7s","firstSeenFrom":"2026-07-14T00:00:00+08:00","latestSeenFrom":"2026-07-14T08:00:00+08:00","limit":50,"offset":0}
|
||||
|
||||
POST /api/v2/access/unresolved-identities
|
||||
Content-Type: application/json
|
||||
|
||||
{"keyword":"","protocol":"JT808","limit":20,"offset":0}
|
||||
|
||||
GET /api/v2/access/thresholds
|
||||
PUT /api/v2/access/thresholds
|
||||
```
|
||||
|
||||
`summary` and `vehicles` consume the same filter shape, so KPI, protocol/OEM distribution, and rows share one scope. Besides keyword, protocol, OEM and status, both accept substring filters `model` and `provider`, plus paired `firstSeenFrom`/`firstSeenTo` and `latestSeenFrom`/`latestSeenTo` bounds. A supplied pair must be ordered; malformed or reversed timestamps return `ACCESS_TIME_INVALID` or `ACCESS_TIME_RANGE_INVALID`. `onlineState` values are `online`, `offline`, `never_reported`, and `unknown`. Online state is calculated on every request from latest receive/update freshness and the current global/protocol threshold version; the stored snapshot boolean is not reused as the answer.
|
||||
|
||||
`unresolved-identities` returns terminals that have real access evidence but no authoritative VIN binding. The current implementation reads JT808 registration evidence and excludes terminals already present in `vehicle_identity_binding`. It exposes only a stable hashed ID and masked terminal identifier, never the raw phone number. Resolve `missing_vin_jt808` by verifying registration evidence and maintaining an authoritative `phone -> VIN` binding; do not infer or fabricate a VIN.
|
||||
|
||||
Access rows expose both event and receive time, `dataDelaySec`, `freshnessSec`, the applied `thresholdSec`, latest realtime message semantics, event ID, source table, model/company master data, and evidence notes. The gateway snapshot upsert maintains `access_first_seen_at`, `access_previous_received_at`, `access_latest_received_at`, `access_report_interval_ms`, `access_sample_count`, and the latest received event ID independently from device event-time ordering. A strictly newer receipt with a different event ID advances the projection atomically; duplicates, equal timestamps, replay and older receipts do not change the interval or sample count. The API rounds milliseconds to `reportIntervalSec` and returns `reportSampleCount`, `firstSeenSource`, `firstSeenEvidence`, and `reportIntervalEvidence`. `live_writer` means first observation after the writer began maintaining the row. `snapshot_backfill` is only a deployment baseline reconstructed from the previous current-state row and must not be described as the vehicle's historical first-ever access.
|
||||
|
||||
Threshold configuration is stored in `vehicle_access_threshold_config`; every successful update increments `version`, requires the caller's previous version, and writes `vehicle_access_threshold_audit`. Stale updates return `ACCESS_THRESHOLD_VERSION_CONFLICT`. Valid ranges are bounded server-side.
|
||||
|
||||
## Compatibility Rule
|
||||
|
||||
Existing callers may still send `vin`. The BFF treats `vin` as a vehicle lookup value for compatibility, so `vin=粤AG18312` still resolves through identity binding. New code should use `keyword` to avoid implying the value is already a real VIN.
|
||||
## Track Replay Evidence
|
||||
|
||||
`GET /api/v2/tracks` accepts `keyword|vin`, optional `protocol`, `dateFrom`, `dateTo` and `maxPoints`.
|
||||
|
||||
- If either time bound is provided, both are required, the end must be later than the start, and the range is limited to seven days. Omitting both retains the bounded latest-slice compatibility mode.
|
||||
- The TDengine source read remains capped at 5,000 latest points per request. `coverage.complete=false` and `coverage.evidence` explicitly state when summary metrics describe only that slice rather than the whole requested window.
|
||||
- `coverage` reports total, fetched, processed and returned point counts plus `latest_source_slice`, `quality_filter` and `map_sampling` reasons.
|
||||
- `quality` reports invalid coordinates, exact duplicates, short-interval >220 km/h implied drift, source switches and gaps over ten minutes. Filtered points never silently disappear without these counters.
|
||||
- When multiple protocols report the same VIN, coordinate quality is evaluated within each protocol and the map selects one primary source (explicit `protocol`, otherwise highest point count then latest observation). Alternate valid points remain visible in `sources`, `quality.alternateSourcePoints`, and the `primary_source_projection` coverage reason; parallel source coordinates are never stitched into one route.
|
||||
- `segments` are inferred as `moving`, `stopped` or `gap`. A stop requires at least three minutes of consecutive <=3 km/h and <=150 m interval movement. The evidence explicitly says this is GPS inference and not ignition state.
|
||||
- Positive subsecond intervals remain continuous telemetry even when their displayed whole-second duration rounds to zero; only invalid/non-increasing timestamps or intervals over ten minutes become `gap`.
|
||||
- Sampling preserves the route endpoints, event points, stop centers and every retained segment boundary. `sampledIndex` fields point directly into the returned `points` array; clients must not approximate an original index by ratio when this field is present.
|
||||
- Playback points and events carry synchronized speed, SOC availability/value, direction and alarm fields. The browser resolves only the paused/current coordinate through the cached reverse-geocode endpoint; playback never triggers one address request per point.
|
||||
|
||||
@@ -10,6 +10,11 @@ pnpm --dir apps/web install
|
||||
pnpm run web:build
|
||||
cd apps/api
|
||||
GOOS=linux GOARCH=amd64 CGO_ENABLED=0 go build -trimpath -ldflags='-s -w' -o ../../dist/platform-api ./cmd/platform-api
|
||||
GOOS=linux GOARCH=amd64 CGO_ENABLED=0 go build -trimpath -ldflags='-s -w' -o ../../dist/alert-evaluator ./cmd/alert-evaluator
|
||||
GOOS=linux GOARCH=amd64 CGO_ENABLED=0 go build -trimpath -ldflags='-s -w' -o ../../dist/alert-stream-evaluator ./cmd/alert-stream-evaluator
|
||||
GOOS=linux GOARCH=amd64 CGO_ENABLED=0 go build -trimpath -ldflags='-s -w' -o ../../dist/platform-migrate ./cmd/platform-migrate
|
||||
# Optional release/performance gate; this binary is run on demand, not installed as a service.
|
||||
GOOS=linux GOARCH=amd64 CGO_ENABLED=0 go build -trimpath -ldflags='-s -w' -o ../../dist/alert-benchmark ./cmd/alert-benchmark
|
||||
```
|
||||
|
||||
## ECS Paths
|
||||
@@ -34,7 +39,11 @@ TDENGINE_DRIVER=taosWS
|
||||
TDENGINE_DSN=root:***@ws(172.17.111.57:6041)/
|
||||
TDENGINE_DATABASE=lingniu_vehicle_ts
|
||||
CAPACITY_CHECK_BIN=/opt/lingniu-go-native/current/capacity-check
|
||||
AUTH_TOKEN=***
|
||||
DATA_MODE=production
|
||||
EXPORT_DIR=/opt/lingniu-vehicle-platform/data/exports
|
||||
AUTH_MODE=enforce
|
||||
# JSON array with viewer/operator/admin principals. Keep this file mode 0600.
|
||||
AUTH_TOKENS_JSON=[{"token":"<at-least-16-random-characters>","name":"ecs-admin","role":"admin"}]
|
||||
REQUEST_TIMEOUT_MS=5000
|
||||
AMAP_WEB_JS_KEY=***
|
||||
AMAP_SECURITY_JS_CODE=***
|
||||
@@ -42,16 +51,170 @@ AMAP_SECURITY_SERVICE_HOST=/_AMapService
|
||||
# Optional reserved server-side key for AMap REST/track services.
|
||||
AMAP_API_KEY=***
|
||||
PLATFORM_RELEASE=platform-YYYYMMDDHHMMSS
|
||||
ALERT_EVALUATION_INTERVAL_SEC=10
|
||||
ALERT_STREAM_MODE=active
|
||||
# Defaults to KAFKA_BROKERS from the shared ingest environment when omitted.
|
||||
ALERT_STREAM_KAFKA_BROKERS=<kafka-host>:9092
|
||||
ALERT_STREAM_KAFKA_TOPICS=vehicle.fields.go.gb32960.v1,vehicle.fields.go.jt808.v1,vehicle.fields.go.yutong-mqtt.v1
|
||||
ALERT_STREAM_KAFKA_GROUP=vehicle-alert-stream-shadow-v1
|
||||
ALERT_STREAM_BATCH_SIZE=200
|
||||
ALERT_STREAM_BATCH_WAIT_MS=100
|
||||
ALERT_STREAM_LATENESS_SEC=120
|
||||
```
|
||||
|
||||
`ALERT_STREAM_MODE=shadow` only validates envelopes and advances checkpoints; `active` additionally evaluates every non-`freshness_sec` rule in the same MySQL transaction. Keep the existing consumer group when switching so active resumes the shadow-proven offsets instead of replaying retained history. In active mode the snapshot evaluator retains exclusive ownership of `freshness_sec` and skips all dynamic telemetry rules. Roll back the mode to `shadow` and restart the API plus both evaluators if active batches fail; transaction rollback prevents a failed rule effect from advancing the authoritative checkpoint.
|
||||
|
||||
`AMAP_WEB_JS_KEY` is served to the browser through `/app-config.js` so the same static build can be reused across environments. For production, set both `AMAP_SECURITY_JS_CODE` and `AMAP_SECURITY_SERVICE_HOST=/_AMapService`; the API service keeps the security code on the server and proxies AMap requests with `jscode` appended. Only omit `AMAP_SECURITY_SERVICE_HOST` for controlled debugging where exposing `AMAP_SECURITY_JS_CODE` to the browser is acceptable. `AMAP_API_KEY` is reserved for backend-only AMap service APIs such as geocoding, route planning, geofence, or trajectory service integration.
|
||||
|
||||
`PLATFORM_RELEASE` is surfaced by `/api/ops/health.runtime.platformRelease` so operators can confirm which ECS release is currently active after a deployment.
|
||||
|
||||
All three platform systemd units first load `/opt/lingniu-go-native/env/base.env` for the existing MySQL, Redis, TDengine and Kafka connection settings, then load `platform.env` for platform-specific overrides. `DATA_MODE=production` is mandatory on ECS: a missing or unreachable MySQL connection returns `DATA_STORE_UNAVAILABLE` instead of silently serving demonstration data. Use `DATA_MODE=mock` only for local development.
|
||||
|
||||
`EXPORT_DIR` must point outside the release symlink. The API writes CSV files and an atomic `jobs.json` index there; completed tasks survive API restarts, while interrupted queued/running jobs are marked failed and can be recreated. Exports run one at a time, use 5,000-row forward-only TDengine cursors instead of `OFFSET`, stream into a `.part` file, and atomically publish the final CSV only after flush, `fsync`, and close succeed. A task is limited to five vehicles, 31 days, 32 metrics, 1,000,000 rows, and 30 minutes. Both the initial count and observed rows enforce the row cap so late-arriving data cannot bypass it. Keep the directory mode `0750` and include it in retention/backup policy. Export list and download APIs require `operator` or `admin`.
|
||||
|
||||
Release verification for this path includes the opt-in synthetic million-row gate:
|
||||
|
||||
```bash
|
||||
cd vehicle-data-platform/apps/api
|
||||
EXPORT_MILLION_TEST=1 go test ./internal/platform -run '^TestHistoryExportMillionRows$' -count=1 -v
|
||||
```
|
||||
|
||||
Production smoke must create an export for an active VIN, wait for `completed`, verify `rowCount == processedRows == totalRows`, compare the downloaded SHA-256 and byte count before and after one API restart, and confirm that no `.part` file remains. CSV files begin with query-range and metric/unit metadata followed by the data header.
|
||||
|
||||
`AUTH_MODE=enforce` is mandatory on ECS. Tokens are stored as SHA-256 comparisons in memory and sent as Bearer credentials; the browser stores the entered token only in `sessionStorage`. Roles are cumulative:
|
||||
|
||||
| Role | Permissions |
|
||||
| --- | --- |
|
||||
| `viewer` | Read pages, query data and inspect evidence |
|
||||
| `operator` | Viewer permissions plus exports, alert actions and notification reads |
|
||||
| `admin` | Operator permissions plus rule and access-threshold configuration |
|
||||
|
||||
After editing the environment file, run `chmod 600 /opt/lingniu-vehicle-platform/env/platform.env`. Never put a real token in Git, static JavaScript, shell history or deployment logs.
|
||||
|
||||
## Forward Migration
|
||||
|
||||
Before switching the API symlink, apply the idempotent access-threshold migration with a MySQL account allowed to create/alter platform-owned tables:
|
||||
|
||||
```bash
|
||||
export MYSQL_DSN="$(sed -n 's/^MYSQL_DSN=//p' /opt/lingniu-go-native/env/base.env | tail -1)"
|
||||
test -n "$MYSQL_DSN"
|
||||
/opt/lingniu-vehicle-platform/releases/$PLATFORM_RELEASE/platform-migrate \
|
||||
/opt/lingniu-vehicle-platform/releases/$PLATFORM_RELEASE/deploy/migrations/001_access_thresholds.sql \
|
||||
/opt/lingniu-vehicle-platform/releases/$PLATFORM_RELEASE/deploy/migrations/002_alert_center.sql \
|
||||
/opt/lingniu-vehicle-platform/releases/$PLATFORM_RELEASE/deploy/migrations/003_alert_rule_advanced.sql \
|
||||
/opt/lingniu-vehicle-platform/releases/$PLATFORM_RELEASE/deploy/migrations/004_alert_repeat_index.sql \
|
||||
/opt/lingniu-vehicle-platform/releases/$PLATFORM_RELEASE/deploy/migrations/005_metric_catalog.sql \
|
||||
/opt/lingniu-vehicle-platform/releases/$PLATFORM_RELEASE/deploy/migrations/006_vehicle_profile.sql \
|
||||
/opt/lingniu-vehicle-platform/releases/$PLATFORM_RELEASE/deploy/migrations/007_alert_master_data_scope.sql \
|
||||
/opt/lingniu-vehicle-platform/releases/$PLATFORM_RELEASE/deploy/migrations/008_access_projection.sql \
|
||||
/opt/lingniu-vehicle-platform/releases/$PLATFORM_RELEASE/deploy/migrations/009_alert_stream_checkpoint.sql \
|
||||
/opt/lingniu-vehicle-platform/releases/$PLATFORM_RELEASE/deploy/migrations/010_alert_stream_metric_mapping.sql \
|
||||
/opt/lingniu-vehicle-platform/releases/$PLATFORM_RELEASE/deploy/migrations/011_alert_stream_invalid_evidence.sql
|
||||
```
|
||||
|
||||
The API guards the access-threshold tables for compatibility, while alert APIs deliberately require the alert migrations to exist. Run every numbered migration explicitly before switching traffic so DDL permission and index creation failures are caught early. The migration journal records filename and SHA-256 and refuses a changed file; duplicate forward `ADD COLUMN` and `CREATE INDEX` statements are tolerated only when resuming partially executed MySQL DDL. Full-line SQL comments are removed before statement splitting, so punctuation in a comment cannot become executable SQL. Migration `008` adds forward-compatible access evidence columns and an index to the gateway-owned realtime snapshot table without changing its `(protocol, vin)` primary key. Migration `009` creates the per-group/topic/partition event-time checkpoint used to make MySQL effects authoritative before Kafka offsets are committed.
|
||||
|
||||
Install and start the evaluator as a separate unit only after API smoke checks and a small enabled-rule review:
|
||||
|
||||
```bash
|
||||
sudo cp deploy/systemd/lingniu-vehicle-alert-evaluator.service /etc/systemd/system/
|
||||
sudo cp deploy/systemd/lingniu-vehicle-alert-stream-evaluator.service /etc/systemd/system/
|
||||
sudo systemctl daemon-reload
|
||||
sudo systemctl enable --now lingniu-vehicle-platform lingniu-vehicle-alert-evaluator lingniu-vehicle-alert-stream-evaluator
|
||||
sudo systemctl status --no-pager lingniu-vehicle-alert-evaluator
|
||||
sudo systemctl status --no-pager lingniu-vehicle-alert-stream-evaluator
|
||||
sudo journalctl -u lingniu-vehicle-alert-evaluator -n 100 --no-pager
|
||||
sudo journalctl -u lingniu-vehicle-alert-stream-evaluator -n 100 --no-pager
|
||||
```
|
||||
|
||||
## Health
|
||||
|
||||
```bash
|
||||
curl -fsS http://127.0.0.1:20300/api/ops/health
|
||||
curl -fsS 'http://127.0.0.1:20300/api/vehicles?limit=5'
|
||||
read -rsp 'Platform admin token: ' PLATFORM_TOKEN; echo
|
||||
AUTH_HEADER="Authorization: Bearer $PLATFORM_TOKEN"
|
||||
curl -fsS -H "$AUTH_HEADER" http://127.0.0.1:20300/api/v2/session
|
||||
curl -fsS -H "$AUTH_HEADER" http://127.0.0.1:20300/api/ops/health
|
||||
curl -fsS -H "$AUTH_HEADER" http://127.0.0.1:20300/api/v2/metrics
|
||||
curl -fsS -H "$AUTH_HEADER" 'http://127.0.0.1:20300/api/v2/monitor/summary?limit=200'
|
||||
curl -fsS -H "$AUTH_HEADER" -H 'Content-Type: application/json' \
|
||||
-d '{"keyword":"LNXNEGRR7SR318212","status":"active","limit":20,"offset":0}' \
|
||||
http://127.0.0.1:20300/api/v2/alerts/events
|
||||
curl -fsS -H "$AUTH_HEADER" http://127.0.0.1:20300/api/v2/vehicles/LNXNEGRR7SR318212/telemetry/latest
|
||||
curl -fsS -H "$AUTH_HEADER" 'http://127.0.0.1:20300/api/vehicles?limit=5'
|
||||
curl -fsS -H "$AUTH_HEADER" \
|
||||
'http://127.0.0.1:20300/api/v2/history/series?keyword=LNXNEGRR7SR318212&dateFrom=2026-07-14T00%3A00&dateTo=2026-07-14T06%3A00&targetPoints=240'
|
||||
curl -fsS -X POST -H "$AUTH_HEADER" -H 'Content-Type: application/json' -d '{"limit":5,"offset":0}' \
|
||||
http://127.0.0.1:20300/api/v2/access/vehicles
|
||||
curl -fsS -X POST -H "$AUTH_HEADER" -H 'Content-Type: application/json' -d '{"limit":20,"offset":0}' \
|
||||
http://127.0.0.1:20300/api/v2/access/unresolved-identities
|
||||
curl -fsS -H "$AUTH_HEADER" http://127.0.0.1:20300/api/v2/access/thresholds
|
||||
curl -fsS -X POST -H "$AUTH_HEADER" -H 'Content-Type: application/json' \
|
||||
-d '{"sourceSystem":"release-smoke","sourceVersion":"dry-run-1","conflictPolicy":"preserve","dryRun":true,"items":[{"vin":"RELEASE_SMOKE_MISSING","operationStatus":"unknown"}]}' \
|
||||
http://127.0.0.1:20300/api/v2/vehicle-profiles/sync
|
||||
curl -fsS -X POST -H "$AUTH_HEADER" -H 'Content-Type: application/json' -d '{"limit":5,"offset":0}' \
|
||||
http://127.0.0.1:20300/api/v2/alerts/events
|
||||
curl -fsS -H "$AUTH_HEADER" http://127.0.0.1:20300/api/v2/alerts/rules
|
||||
curl -fsS -H "$AUTH_HEADER" 'http://127.0.0.1:20300/api/v2/alerts/notifications?unreadOnly=true&limit=5'
|
||||
curl -fsS http://127.0.0.1:20300/app-config.js
|
||||
curl -fsS http://127.0.0.1:20300/ | grep -q '<div id="root"></div>'
|
||||
MAIN_ASSET="$(sed -n 's/.*src="\([^"]*\.js\)".*/\1/p' /opt/lingniu-vehicle-platform/current/web/index.html | head -1)"
|
||||
curl -fsS "http://127.0.0.1:20300$MAIN_ASSET" >/dev/null
|
||||
unset PLATFORM_TOKEN AUTH_HEADER
|
||||
```
|
||||
|
||||
The release gate must include both the root document and its hashed main asset. API-only smoke checks are insufficient because an incomplete archive can leave the service healthy while the browser returns 404. Before upload, verify the archive contains `web/index.html`, `platform-api`, both evaluator binaries, `platform-migrate`, every numbered migration and all three platform systemd units.
|
||||
|
||||
When `alertStream.lastInvalidCode` reports a recent `missing_vin_jt808`, query `/api/v2/access/unresolved-identities` as a viewer and hand the masked evidence to the identity owner. The response must not contain raw `phone` or unmasked identifier fields. Confirm the evidence against the GPS provider or vehicle owner, then maintain the authoritative `vehicle_identity_binding`; never derive VIN from the terminal number. The queue excludes a terminal after a valid binding exists. Retain the invalid counter as audit evidence and verify the recent warning clears after five minutes without another invalid frame.
|
||||
|
||||
The history-series gate must use an active production VIN and a current local-time window. Verify that `rawPointCount > 0`, every returned series remains within the requested point budget, `dateFrom`/`dateTo` and point timestamps describe the same absolute window, and the browser renders `Asia/Shanghai` labels. TDengine timestamp literals for this endpoint include an explicit RFC3339 offset; bare UTC-looking strings are unsafe because the server session may interpret them again in its local timezone.
|
||||
|
||||
Track smoke must also select the final playback event and verify synchronized SOC/direction/alarm availability plus a resolved current address. For high-frequency sources, confirm positive subsecond samples do not become alternating zero-second `数据间隔` segments; only non-increasing timestamps or intervals over ten minutes are gaps. Access smoke should exercise `model`/`provider` and one paired first/latest receive-time range, then confirm the filter survives a shareable URL and every returned row remains in scope.
|
||||
|
||||
Global-monitor smoke must verify `zoom=5` returns clusters, a city `bounds` at `zoom=13` returns at most 2,000 lightweight points, and invalid/reversed bounds return HTTP 400. In the browser, the rail renders at most 200 vehicle rows while the legend reports the independent map mode. Selecting `driving` or `idle` must constrain both the server-side count and every loaded row. A unique keyword result must discard the previous viewport and focus at point zoom. The synthetic release gate is `go test ./internal/platform -run TestMonitorMapTenThousandVehicles -count=1` plus `go test ./internal/platform -run '^$' -bench BenchmarkMonitorMapTenThousandVehicles -benchtime=30x -benchmem`.
|
||||
|
||||
Latest-telemetry smoke must use an actively reporting VIN and verify that populated categories exactly sum to `values.length`, catalog-mapped values retain both the unified `key` and exact `sourceField`, every value has frame/time/protocol evidence, and the quality reason agrees with freshness and parser status. `scannedFrames` must remain at or below 15: the implementation resolves identity once, queries the three supported protocol tables concurrently with five rows each, and never runs a pagination count. The browser must display source, compact device time and quality without deriving labels, units or categories from RAW keys. The bounded synthetic gate is `go test ./internal/platform -run 'TestBuildLatestTelemetryResponse|TestLatestTelemetryQuality|TestLatestTelemetryBoundsProtocolReads' -count=1` plus `go test ./internal/platform -run '^$' -bench BenchmarkLatestTelemetryHundredFrames -benchmem`.
|
||||
|
||||
Alert evaluation is intentionally not exposed as an HTTP route because it mutates candidate/event state. Production evaluation is owned by the evaluator unit. Confirm its `rules/vehicles/candidates_advanced/duplicate_observations/late_observations/stale_evidence_skipped/opened/recovered` log line and verify the resulting event through the read APIs. A release gate with an active scoped rule must survive at least one persisted candidate reload and one persisted last-trigger reload; MySQL `DATETIME(3)` values are scanned directly as times so millisecond values are never forced through an integer `UNIX_TIMESTAMP` conversion.
|
||||
|
||||
The Kafka stream evaluator accepts only explicit `shadow` and `active` modes. A new group starts at the latest retained offset, then resumes committed offsets. Each batch validates the canonical `FIELDS` envelope, protocol/topic pairing, field namespace, identity and event/receive time. In active mode it locks enabled dynamic rules, reads only state belonging to the current batch's rules/VINs/protocols, applies candidate/event/action/notification changes, and updates the MySQL checkpoint in one transaction; Kafka is committed only afterward. Replayed offsets below the database checkpoint are skipped. Evidence beyond the configured receive/event lateness window can still drive `data_delay_sec`, but cannot open or recover speed/SOC/alarm rules. `/api/ops/health.alertStream` and the `Alert Kafka stream` link expose mode, partitions, checkpoint lag, processed/valid/invalid/late/replay counts, last invalid code/time and last update. A recent `missing_vin_<protocol>` warning means a real terminal is still reporting without an authoritative VIN; resolve it through access/identity operations instead of fabricating a binding. Historical invalid totals remain audit evidence but stop holding the link in warning five minutes after the last invalid message. Release requires lag below the configured gate, no unexplained invalid messages and an active-rule canary with exactly-once event evidence.
|
||||
|
||||
## Alert Candidate Write Benchmark
|
||||
|
||||
`alert-benchmark` measures the evaluator's seven-column candidate batch shape against the configured MySQL/RDS connection. It never writes `vehicle_alert_candidate` itself and only accepts two compiled-in benchmark table names.
|
||||
|
||||
- The default `temporary` mode creates a connection-scoped temporary table. It is the lowest-risk network, SQL and index-path check.
|
||||
- `durable` mode requires `--confirm-durable-write`, takes a zero-wait MySQL advisory lock, creates a dedicated physical `vehicle_alert_candidate_benchmark_durable` table with `LIKE vehicle_alert_candidate`, commits one transaction, verifies the row count, drops the table and verifies its removal through `information_schema`. A stale table from an interrupted prior run is removed only after the lock is held.
|
||||
- Both modes are bounded to 100,000 rows and 1,000 rows per insert. The process uses an independent cleanup timeout so normal error returns still clean up the benchmark table.
|
||||
|
||||
```bash
|
||||
export MYSQL_DSN="$(sed -n 's/^MYSQL_DSN=//p' /opt/lingniu-go-native/env/base.env | tail -1)"
|
||||
/opt/lingniu-vehicle-platform/current/alert-benchmark --mode temporary --rows 10000 --batch-size 500
|
||||
/opt/lingniu-vehicle-platform/current/alert-benchmark --mode durable --confirm-durable-write --rows 10000 --batch-size 500
|
||||
/opt/lingniu-vehicle-platform/current/alert-benchmark --mode durable --confirm-durable-write --rows 100000 --batch-size 500
|
||||
unset MYSQL_DSN
|
||||
```
|
||||
|
||||
ECS baseline on 2026-07-14 against production RDS MySQL 8.0.36. Every row count matched and every dedicated table cleanup was verified:
|
||||
|
||||
| Mode | Rows | Batch | Transactions | Duration | Throughput | Observed redo delta | Cleanup |
|
||||
| --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: |
|
||||
| temporary | 10,000 | 500 | 1 | 192ms | 51,883 rows/s | n/a | verified |
|
||||
| temporary | 100,000 | 500 | 1 | 1,765ms | 56,646 rows/s | n/a | verified |
|
||||
| durable | 10,000 | 500 | 1 | 195ms | 51,234 rows/s | 3,121,664 bytes | 3ms, verified |
|
||||
| durable | 100,000 | 500 | 1 | 1,971ms | 50,713 rows/s | 31,037,440 bytes | 3ms, verified |
|
||||
|
||||
`globalStatusDelta` is sampled from MySQL instance-global counters, including `Innodb_os_log_written`, `Binlog_cache_use`, `Binlog_cache_disk_use` and `Com_commit`. These values prove redo/binlog machinery was active around the durable run, but they are shared with concurrent RDS traffic and are not an exact per-benchmark attribution. The durable result closes the committed physical-table throughput gap without polluting business data; it still does not replace long-running evaluator observation with active rules, lock-contention tests, RDS Performance Insights, or binlog/redo monitoring over a representative peak window.
|
||||
|
||||
## Rollback
|
||||
|
||||
Keep the previous release directory until post-release monitoring is complete. Database changes are forward-compatible and are not rolled back. If API or browser checks fail, atomically restore the old symlink and restart both units:
|
||||
|
||||
```bash
|
||||
PREVIOUS_RELEASE=/opt/lingniu-vehicle-platform/releases/<previous-release>
|
||||
sudo ln -sfn "$PREVIOUS_RELEASE" /opt/lingniu-vehicle-platform/current
|
||||
sudo systemctl restart lingniu-vehicle-platform
|
||||
sudo systemctl stop lingniu-vehicle-alert-evaluator
|
||||
sudo systemctl status --no-pager lingniu-vehicle-platform
|
||||
```
|
||||
|
||||
Do not roll back the symlink if the old binary cannot tolerate the forward schema; validate this before every migration. The current `001`–`011` migrations are forward-only additions of tables, columns and indexes; the old realtime writer ignores the nullable/defaulted `008` columns and the prior API does not select them. Migration `003` once stopped after its first two MySQL `ALTER` statements because `last_value` was a reserved identifier; the final immutable migration uses `observed_value`, and the journaled runner safely resumed only the duplicate schema objects before creating the missing state table. Do not edit an applied migration—add the next numbered migration for future schema changes.
|
||||
|
||||
|
After Width: | Height: | Size: 110 KiB |
|
After Width: | Height: | Size: 30 KiB |
|
After Width: | Height: | Size: 83 KiB |
|
After Width: | Height: | Size: 172 KiB |
|
After Width: | Height: | Size: 110 KiB |
|
After Width: | Height: | Size: 88 KiB |
|
After Width: | Height: | Size: 30 KiB |
|
After Width: | Height: | Size: 1.2 MiB |
|
After Width: | Height: | Size: 1.3 MiB |
|
After Width: | Height: | Size: 1.3 MiB |
|
After Width: | Height: | Size: 1.2 MiB |
|
After Width: | Height: | Size: 1.3 MiB |
|
After Width: | Height: | Size: 1.3 MiB |
@@ -32,6 +32,19 @@
|
||||
- 里程统计页要像客户报告工作台:先确认统计口径和闭合状态,再进入轨迹复核、历史明细、CSV 导出和交付结论。
|
||||
- 内部可观测性继续保留,但默认折叠在运维/依据层,避免客户把系统理解为“协议接入平台”。
|
||||
|
||||
## 2026-07-14 Replay 深化对标
|
||||
|
||||
- Verizon Connect Reveal 的官方 Replay 说明默认展示当天,并把多日回放限制为最多 7 天;多日先按天呈现 Journey,再进入具体行程。这支持本项目采用“默认今天 + 单次最多 7 天 + 明确覆盖边界”,避免把无限时间窗的最新切片误读成全量。参考:https://reveal-help.verizonconnect.com/hc/en-us/articles/1500004436922-View-a-vehicle-or-asset-s-journey-in-Replay
|
||||
- Reveal 的时间轴按 Moving、Stopped、Idling 分段,并在详情中展示时间、时长、行驶距离、地址和驾驶事件。本项目 `vehicle_locations` 没有 ignition,所以只把低速小位移称为“GPS 推断停车”,不冒充熄火/怠速;数据间隔作为独立分段。参考:https://reveal-help.verizonconnect.com/hc/en-us/articles/360010566999-View-all-of-your-vehicles-and-assets-in-Replay
|
||||
- Reveal Spotlight 的 Journey Summary 同时展示距离、行驶时间、停车次数、停车时间,详细时间轴能下钻到停车位置、移动段和超速点。因此本项目轨迹摘要增加移动/停车时长、停车/分段数,关键事件与分段边界必须在地图抽稀后仍能精确定位。参考:https://reveal-help.verizonconnect.com/hc/en-ca/articles/1500003198622-Review-a-vehicle-or-asset-s-activity-with-Replay-in-Spotlight
|
||||
- 停车时长阈值是产品配置而非自然事实。Reveal 报表允许隐藏小于指定分钟数的停车;本项目一期先采用 3 分钟固定门槛并在 API 证据中公开,后续再进入可版本化配置。参考:https://reveal-help.verizonconnect.com/hc/en-us/articles/360050801533-Travel-and-stops-report
|
||||
|
||||
## 2026-07-14 大数据导出对标
|
||||
|
||||
- Geotab 官方 `GetFeed` 用持久化 `toVersion/fromVersion` 游标连续读取数据,并明确要求客户端保存版本令牌以便停止后无缝继续;单次通常最多 50,000 条。这验证了大历史数据应采用单调游标而不是越来越慢的 `OFFSET`。本项目一期对 TDengine 采用 `(ts, protocol, frame_id)` 复合前向游标和 5,000 行批次,先保证单 ECS 的稳定内存边界;多实例阶段再把任务与游标迁移到数据库队列。参考:https://developers.geotab.com/myGeotab/guides/dataFeed/index.html 和 https://developers.geotab.com/myGeotab/apiReference/methods/GetFeed/index.html
|
||||
- Geotab 还建议“批次达到上限就立即继续、少量或空批次再退避”,说明进度必须来自服务端已处理记录而不是前端估算。本项目任务卡展示 `processedRows/totalRows`,每个成功批次才持久化进度。
|
||||
- Samsara 的车队报表 API 把报表类型、时间范围和 RFC3339 时区作为显式合同,而不是导出当前页面偶然加载的行。本项目 CSV 因此写入查询起止时间、车辆范围、数据类型、协议和指标单位元数据,并使用带时区的绝对时间查询 TDengine。参考:https://developers.samsara.com/reference/getfuelenergyvehiclereports
|
||||
|
||||
## 本项目采用的产品原则
|
||||
|
||||
- 三个接入来源只是数据通道,最终服务对象是车辆。
|
||||
|
||||