shishengliang/lingniu-vehicle-ingest
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat: expand vehicle data platform capabilities

Browse Source
This commit is contained in:
lingniu
2026-07-27 16:46:15 +08:00
parent e3a1f80f86
commit 3c4bece72c
650 changed files with 62155 additions and 2552 deletions
Download Patch File Download Diff File Expand all files Collapse all files

97
vehicle-data-platform/docs/deployment.md
View File

@@ -8,13 +8,18 @@ Run from repository root:
cd vehicle-data-platform
pnpm --dir apps/web install
pnpm run web:build
pnpm --dir apps/open-portal install
pnpm run open-portal:build
cd apps/api
GOOS=linux GOARCH=amd64 CGO_ENABLED=0 go build -trimpath -ldflags='-s -w' -o ../../dist/platform-api ./cmd/platform-api
GOOS=linux GOARCH=amd64 CGO_ENABLED=0 go build -trimpath -ldflags='-s -w' -o ../../dist/open-platform-api ./cmd/open-platform-api
GOOS=linux GOARCH=amd64 CGO_ENABLED=0 go build -trimpath -ldflags='-s -w' -o ../../dist/alert-evaluator ./cmd/alert-evaluator
GOOS=linux GOARCH=amd64 CGO_ENABLED=0 go build -trimpath -ldflags='-s -w' -o ../../dist/alert-stream-evaluator ./cmd/alert-stream-evaluator
GOOS=linux GOARCH=amd64 CGO_ENABLED=0 go build -trimpath -ldflags='-s -w' -o ../../dist/alert-notification-dispatcher ./cmd/alert-notification-dispatcher
GOOS=linux GOARCH=amd64 CGO_ENABLED=0 go build -trimpath -ldflags='-s -w' -o ../../dist/platform-migrate ./cmd/platform-migrate
GOOS=linux GOARCH=amd64 CGO_ENABLED=0 go build -trimpath -ldflags='-s -w' -o ../../dist/oneos-scope-sync ./cmd/oneos-scope-sync
GOOS=linux GOARCH=amd64 CGO_ENABLED=0 go build -trimpath -ldflags='-s -w' -o ../../dist/reconciliation-evaluator ./cmd/reconciliation-evaluator
GOOS=linux GOARCH=amd64 CGO_ENABLED=0 go build -trimpath -ldflags='-s -w' -o ../../dist/open-platform-stat ./cmd/open-platform-stat
# Optional release/performance gate; this binary is run on demand, not installed as a service.
GOOS=linux GOARCH=amd64 CGO_ENABLED=0 go build -trimpath -ldflags='-s -w' -o ../../dist/alert-benchmark ./cmd/alert-benchmark
```
@@ -25,8 +30,16 @@ GOOS=linux GOARCH=amd64 CGO_ENABLED=0 go build -trimpath -ldflags='-s -w' -o ../
/opt/lingniu-vehicle-platform/current
/opt/lingniu-vehicle-platform/releases
/opt/lingniu-vehicle-platform/env/platform.env
/opt/lingniu-vehicle-open-platform/current
/opt/lingniu-vehicle-open-platform/releases
/opt/lingniu-vehicle-open-platform/env/open-platform.env
```
The internal platform remains on `20300`. The partner-facing portal, portal
session APIs, Swagger/OpenAPI assets, and `/api/v1/vehicles/*` data APIs run in
the independent `lingniu-vehicle-open-platform` service on `20310`.
## Browser-safe release switch
The web application splits every major route into a hashed lazy-loaded asset. An already-open browser tab can therefore request the previous release's route asset after `current` has switched. Keep exactly one previous generation of original build assets in the new release before the atomic symlink switch:
@@ -67,7 +80,7 @@ PREVIOUS_WEB=/opt/lingniu-vehicle-platform/releases/$PREVIOUS_RELEASE/web
"$PREVIOUS_WEB/.release-assets"
```
`deploy/install-web-release.sh` also accepts an optional third argument containing a newly built `platform-api`, an optional fourth argument containing a newly built `oneos-scope-sync`, and an optional fifth argument containing a newly built `reconciliation-evaluator`. When provided, it atomically publishes those binaries with the Web. The installer inherits every runtime binary that exists in the previous release (`alert-evaluator`, `alert-stream-evaluator`, `platform-migrate`, `oneos-scope-sync`, `reconciliation-evaluator` and the optional benchmark) before switching the symlink, so a later systemd restart cannot fail because a Web-oriented release omitted an unchanged binary:
`deploy/install-web-release.sh` also accepts optional runtime binaries after the Web archive: `platform-api`, `oneos-scope-sync`, `reconciliation-evaluator`, `open-platform-stat`, `alert-evaluator`, `alert-stream-evaluator`, and `alert-notification-dispatcher` (arguments 3–9 in that order). When provided, it atomically publishes those binaries with the Web. The installer also inherits every runtime binary that exists in the previous release, including the notification dispatcher, before switching the symlink, so a later systemd restart cannot fail because a Web-oriented release omitted an unchanged binary:
```bash
deploy/install-web-release.sh \
@@ -75,7 +88,11 @@ deploy/install-web-release.sh \
"/tmp/$PLATFORM_RELEASE-web.tar.gz" \
"/tmp/$PLATFORM_RELEASE-platform-api" \
"/tmp/$PLATFORM_RELEASE-oneos-scope-sync" \
"/tmp/$PLATFORM_RELEASE-reconciliation-evaluator"
"/tmp/$PLATFORM_RELEASE-reconciliation-evaluator" \
"/tmp/$PLATFORM_RELEASE-open-platform-stat" \
"/tmp/$PLATFORM_RELEASE-alert-evaluator" \
"/tmp/$PLATFORM_RELEASE-alert-stream-evaluator" \
"/tmp/$PLATFORM_RELEASE-alert-notification-dispatcher"
```
## Environment
@@ -124,9 +141,23 @@ ALERT_STREAM_MODE=active
ALERT_STREAM_KAFKA_BROKERS=<kafka-host>:9092
ALERT_STREAM_KAFKA_TOPICS=vehicle.fields.go.gb32960.v1,vehicle.fields.go.jt808.v1,vehicle.fields.go.yutong-mqtt.v1
ALERT_STREAM_KAFKA_GROUP=vehicle-alert-stream-shadow-v1
ALERT_NOTIFICATION_TARGETS_JSON=[{"id":"night-shift","label":"夜班负责人","channels":["sms","wecom"]},{"id":"data-platform","label":"数据平台组","channels":["email","wecom"]}]
ALERT_NOTIFICATION_SMS_URL=https://trusted-notification-gateway.example.internal/sms
ALERT_NOTIFICATION_SMS_SECRET=***
ALERT_NOTIFICATION_EMAIL_URL=https://trusted-notification-gateway.example.internal/email
ALERT_NOTIFICATION_EMAIL_SECRET=***
ALERT_NOTIFICATION_WECOM_URL=https://trusted-notification-gateway.example.internal/wecom
ALERT_NOTIFICATION_WECOM_SECRET=***
ALERT_NOTIFICATION_BATCH_SIZE=20
ALERT_NOTIFICATION_POLL_INTERVAL_MS=1000
ALERT_NOTIFICATION_LEASE_SEC=30
ALERT_NOTIFICATION_TIMEOUT_MS=5000
ALERT_STREAM_BATCH_SIZE=200
ALERT_STREAM_BATCH_WAIT_MS=100
ALERT_STREAM_LATENESS_SEC=120
HISTORY_EXPORT_CLEANUP_AUTOMATION_ENABLED=true
HISTORY_EXPORT_CLEANUP_POLL_SEC=60
HISTORY_EXPORT_CLEANUP_LEASE_SEC=300
```
`ALERT_STREAM_MODE=shadow` only validates envelopes and advances checkpoints; `active` additionally evaluates every non-`freshness_sec` rule in the same MySQL transaction. Keep the existing consumer group when switching so active resumes the shadow-proven offsets instead of replaying retained history. In active mode the snapshot evaluator retains exclusive ownership of `freshness_sec` and skips all dynamic telemetry rules. Roll back the mode to `shadow` and restart the API plus both evaluators if active batches fail; transaction rollback prevents a failed rule effect from advancing the authoritative checkpoint.
@@ -143,6 +174,8 @@ All three platform systemd units first load `/opt/lingniu-go-native/env/base.env
`EXPORT_DIR` must point outside the release symlink. The API writes mode-`0640` CSV files and an atomic `jobs.json` index there; completed tasks survive API restarts, while interrupted queued/running jobs are marked failed and can be recreated. Every job persists its owner account, role/user type, customer/tenant references, resolved VINs and per-vehicle time scope. Non-admin principals can list and download only their own jobs; customers additionally pass a live account-status and vehicle-grant check before execution, every 5,000-row batch and every download. Old ownerless jobs remain visible only to administrators. The browser downloads through an authenticated Blob request rather than a direct anchor, so the Bearer credential is always present.
`HISTORY_EXPORT_CLEANUP_AUTOMATION_ENABLED=true` starts the approval-gated cleanup coordinator inside each API instance. Every instance may poll, but `cleanup-automation.lock` under `EXPORT_DIR` admits only one scheduler tick at a time; a claimed execution also carries an expiring lease so process termination is recoverable without duplicate cleanup. The coordinator only creates review batches until an administrator approves one in the history task center. Keep `EXPORT_DIR` on a filesystem that provides process-shared POSIX advisory locks, and do not place it on an object-store mount that ignores `flock`. The policy, run history, approval evidence and leases persist in `cleanup-automation.json`; back it up with `jobs.json` and `cleanup-audit.json`. A production canary should run two API instances against the same export directory, verify only one creates/claims a due batch, terminate the lease owner during a disposable approved batch, and confirm the survivor resumes only after the recorded lease expiry.
Vehicle authorization dates must be changed through account management, which atomically updates the active `platform_user_vehicle` projection and its matching `platform_user_vehicle_grant_history` interval. Do not edit only one table. After saving, verify both timestamps match through `GET /api/v2/admin/users`; the response includes current `vehicles[].validFrom/validTo` and `grantHistory[]`. A start time after local midnight intentionally makes daily mileage visible from the next complete `Asia/Shanghai` day, while track/location/RAW data begins at the exact timestamp.
Exports run one at a time, use 5,000-row forward-only TDengine cursors instead of `OFFSET`, stream into a `.part` file, and atomically publish the final CSV only after flush, `fsync`, and close succeed. A task is limited to five vehicles, 31 days, 32 metrics, 1,000,000 rows, and 30 minutes. Both the initial count and observed rows enforce the row cap so late-arriving data cannot bypass it. Keep the directory mode `0750`, preserve `jobs.json` and completed CSVs across releases, and include the directory in retention/backup policy. Export routes require an authenticated viewer-capable principal; ownership and current Scope are enforced inside the export service instead of relying only on role middleware.
@@ -209,10 +242,60 @@ test -n "$MYSQL_DSN"
/opt/lingniu-vehicle-platform/releases/$PLATFORM_RELEASE/deploy/migrations/019_vehicle_source_provider.sql \
/opt/lingniu-vehicle-platform/releases/$PLATFORM_RELEASE/deploy/migrations/020_vehicle_profile_brand.sql \
/opt/lingniu-vehicle-platform/releases/$PLATFORM_RELEASE/deploy/migrations/021_protocol_metric_catalog.sql \
/opt/lingniu-vehicle-platform/releases/$PLATFORM_RELEASE/deploy/migrations/022_gb32960_reference_catalog.sql
/opt/lingniu-vehicle-platform/releases/$PLATFORM_RELEASE/deploy/migrations/022_gb32960_reference_catalog.sql \
/opt/lingniu-vehicle-platform/releases/$PLATFORM_RELEASE/deploy/migrations/023_vehicle_open_platform.sql \
/opt/lingniu-vehicle-platform/releases/$PLATFORM_RELEASE/deploy/migrations/024_vehicle_open_portal.sql \
/opt/lingniu-vehicle-platform/releases/$PLATFORM_RELEASE/deploy/migrations/025_vehicle_open_platform_admin.sql \
/opt/lingniu-vehicle-platform/releases/$PLATFORM_RELEASE/deploy/migrations/026_hydrogen_stream_state.sql \
/opt/lingniu-vehicle-platform/releases/$PLATFORM_RELEASE/deploy/migrations/027_hydrogen_rate_stream_state.sql \
/opt/lingniu-vehicle-platform/releases/$PLATFORM_RELEASE/deploy/migrations/029_hydrogen_pressure_capacity.sql \
/opt/lingniu-vehicle-platform/releases/$PLATFORM_RELEASE/deploy/migrations/030_hydrogen_pressure_hysteresis.sql \
/opt/lingniu-vehicle-platform/releases/$PLATFORM_RELEASE/deploy/migrations/031_alert_automation_triggers.sql \
/opt/lingniu-vehicle-platform/releases/$PLATFORM_RELEASE/deploy/migrations/032_reconciliation_ownership.sql \
/opt/lingniu-vehicle-platform/releases/$PLATFORM_RELEASE/deploy/migrations/033_open_mileage_range_snapshot.sql \
/opt/lingniu-vehicle-platform/releases/$PLATFORM_RELEASE/deploy/migrations/034_alert_notification_retry.sql \
/opt/lingniu-vehicle-platform/releases/$PLATFORM_RELEASE/deploy/migrations/035_alert_notification_dispatch.sql \
/opt/lingniu-vehicle-platform/releases/$PLATFORM_RELEASE/deploy/migrations/036_alert_rule_archive.sql
```
The API guards the access-threshold tables for compatibility, while alert and reconciliation APIs deliberately require their migrations to exist. Run every numbered migration explicitly before switching traffic so DDL permission and index creation failures are caught early. The migration journal records filename and SHA-256 and refuses a changed file; duplicate forward `ADD COLUMN` and `CREATE INDEX` statements are tolerated only when resuming partially executed MySQL DDL. Full-line SQL comments are removed before statement splitting, so punctuation in a comment cannot become executable SQL. Migration `008` adds forward-compatible access evidence columns and an index to the gateway-owned realtime snapshot table without changing its `(protocol, vin)` primary key. Migration `009` creates the per-group/topic/partition event-time checkpoint used to make MySQL effects authoritative before Kafka offsets are committed. Migration `012` creates the atomic, versioned business Scope projection; its reserved `row_number` column is quoted for production MySQL compatibility. Migration `014` backfills active vehicle-grant start times, creates the grant-interval history table and adds the active time lookup index; apply it before starting an API binary that writes grant history. Migration `015` adds platform-owned optimistic versions and immutable audits for per-vehicle location-source policy changes. It does not alter the gateway election SQL or expose the gateway `source_key`; the API resolves an opaque `sourceRef` server-side and the gateway applies the saved policy on the next valid vehicle report. Migration `016` adds customer name, department and responsible-person dimensions plus bounded lookup indexes to the platform-owned Scope projection. Migration `017` creates the reconciliation run, issue and immutable action tables; the unique fingerprint index is the database-level duplicate-work-item guard. Migration `018` records every OneOS-sourced brand fill before the platform identity OEM field changes; it does not grant access to OneOS or run a cross-database sync. Migration `019` stores administrator-maintained provider names by exact vehicle/protocol/source key and writes an immutable versioned audit. The browser still receives only the SHA-256 `sourceRef`; raw source keys remain server-side. Migration `020` separates the actual vehicle brand in `vehicle_profile.brand_name` from T-Box/GPS provider identity. Migration `021` adds the cross-protocol core metric catalog and correct source paths for JT808 GPS mileage, GB32960/YUTONG instrument mileage and GB32960 fuel-cell/engine telemetry. Migration `022` applies the authoritative GB/T 32960 Chinese terminology, changes business hydrogen-concentration metrics and alert thresholds from ppm to percent, and leaves the ppm field only as disabled historical evidence.
Migration `023` creates vehicle open-platform appKey records, per-vehicle grant intervals, precomputed daily energy, and immutable API/admin audits. Apply it before serving `/api/v1/vehicles/*` or enabling `lingniu-vehicle-open-stat.timer`.
Migration `029` creates the local VIN-to-tank-capacity projection and pressure-calculation evidence columns. Migration `030` adds the per-refuel-cycle low-water mark used to prevent pressure and temperature oscillation from being counted repeatedly. The stat writer synchronizes capacity from `ln_asset_management.vehicle_info.vehicle_model_id → vehicle_model.tank_capacity` at startup and every six hours, then serves frame-time lookups from process memory.
Migration `038` adds daily pure-hydrogen mileage to the elected mileage table and its per-source evidence table. Deploy the migration before the updated stat writer and API so GB32960 `engine_work_state=2` and Yutong `TRIANGLE_STATE=4/11` intervals can be accumulated and returned as `pureHydrogenMileageKm`.
Migration `031` adds explicit metric, geofence, stationary and offline automation triggers. It must be applied before the API and both alert evaluators are restarted; geofence rules deliberately require one positioning protocol to avoid multi-source coordinate drift and duplicate boundary events.
Migration `035` adds rule recipient-group references plus notification leases and dispatch indexes. Apply it before publishing an API/evaluator that writes `notification_targets_json` or enabling `lingniu-vehicle-alert-notification-dispatcher`. The dispatcher must remain disabled until every configured gateway has passed a signed canary that returns a non-empty provider message ID.
Migration `036` adds reversible rule-archive metadata, lifecycle query indexes, and audit reasons. Apply it before publishing an API or Web build that calls `/api/v2/alerts/rules/library`, `/archive`, or `/restore`, and before restarting either alert evaluator. Archive is accepted only for disabled rules and removes live candidate/state rows; restore always returns a rule disabled. During rollback, older evaluators still ignore archived rules because the archive transaction has already disabled them, but operators must not use the new lifecycle UI against an API that predates migration `036`.
Migration `024` creates partner users, role-based app membership, expiring
sessions, login lockout state, and immutable user-operation audits. Apply it
before starting the standalone open-platform service.
Migration `025` lets the standalone service issue its own session after
validating an enabled local `platform_user` administrator. It does not copy or
store a second administrator password. Apply it before deploying a portal that
shows the platform-management workspace.
The standalone API binary embeds its version-matched OpenAPI assets and serves
the compiled partner portal. After switching a release, verify
`http://127.0.0.1:20310/`, `/healthz`, `/portal-api/catalog`,
`/open-api/swagger/`, and `/open-api/openapi.yaml`. Install the unit and deploy
an atomic release with:
```bash
sudo cp deploy/systemd/lingniu-vehicle-open-platform.service /etc/systemd/system/
sudo systemctl daemon-reload
COPYFILE_DISABLE=1 tar -C apps/open-portal/dist -czf /tmp/open-portal-web.tar.gz .
sudo deploy/install-open-platform-release.sh \
"$OPEN_PLATFORM_RELEASE" \
/tmp/open-portal-web.tar.gz \
dist/open-platform-api
```
The API guards the access-threshold tables for compatibility, while alert and reconciliation APIs deliberately require their migrations to exist. Run every numbered migration explicitly before switching traffic so DDL permission and index creation failures are caught early. The migration journal records filename and SHA-256 and refuses a changed file; duplicate forward `ADD COLUMN` and `CREATE INDEX` statements are tolerated only when resuming partially executed MySQL DDL. Full-line SQL comments are removed before statement splitting, so punctuation in a comment cannot become executable SQL. Migration `008` adds forward-compatible access evidence columns and an index to the gateway-owned realtime snapshot table without changing its `(protocol, vin)` primary key. Migration `009` creates the per-group/topic/partition event-time checkpoint used to make MySQL effects authoritative before Kafka offsets are committed. Migration `012` creates the atomic, versioned business Scope projection; its reserved `row_number` column is quoted for production MySQL compatibility. Migration `014` backfills active vehicle-grant start times, creates the grant-interval history table and adds the active time lookup index; apply it before starting an API binary that writes grant history. Migration `015` adds platform-owned optimistic versions and immutable audits for per-vehicle location-source policy changes. It does not alter the gateway election SQL or expose the gateway `source_key`; the API resolves an opaque `sourceRef` server-side and the gateway applies the saved policy on the next valid vehicle report. Migration `016` adds customer name, department and responsible-person dimensions plus bounded lookup indexes to the platform-owned Scope projection. Migration `017` creates the reconciliation run, issue and immutable action tables; the unique fingerprint index is the database-level duplicate-work-item guard. Migration `018` records every OneOS-sourced brand fill before the platform identity OEM field changes; it does not grant access to OneOS or run a cross-database sync. Migration `019` stores administrator-maintained provider names by exact vehicle/protocol/source key and writes an immutable versioned audit. The browser still receives only the SHA-256 `sourceRef`; raw source keys remain server-side. Migration `020` separates the actual vehicle brand in `vehicle_profile.brand_name` from T-Box/GPS provider identity. Migration `021` adds the cross-protocol core metric catalog and correct source paths for JT808 GPS mileage, GB32960/YUTONG instrument mileage and GB32960 fuel-cell/engine telemetry. Migration `022` applies the authoritative GB/T 32960 Chinese terminology, changes business hydrogen-concentration metrics and alert thresholds from ppm to percent, and leaves the ppm field only as disabled historical evidence. Migration `032` adds structured reconciliation responsibility, assignment audit metadata, explicit due times, and owner/SLA lookup indexes; apply it before deploying an API binary that selects or writes assignment fields.
`docs/oneos-brand-backfill.sql` is an explicit one-time calibration, not a service dependency. It reads OneOS vehicle/model master data, inserts immutable source evidence, and fills only empty `vehicle_identity_binding.oem` values. Review the pre-run count, apply migration `018`, run the script once through the migration runner or a transaction-capable MySQL client, and verify its final audit/missing counts. Never overwrite a non-empty platform brand and never schedule this script; future updates belong in the formal OneOS API.
@@ -231,16 +314,19 @@ Install and start the evaluator as a separate unit only after API smoke checks a
```bash
sudo cp deploy/systemd/lingniu-vehicle-alert-evaluator.service /etc/systemd/system/
sudo cp deploy/systemd/lingniu-vehicle-alert-stream-evaluator.service /etc/systemd/system/
sudo cp deploy/systemd/lingniu-vehicle-alert-notification-dispatcher.service /etc/systemd/system/
sudo cp deploy/systemd/lingniu-vehicle-oneos-scope-sync.service /etc/systemd/system/
sudo cp deploy/systemd/lingniu-vehicle-oneos-scope-sync.timer /etc/systemd/system/
sudo cp deploy/systemd/lingniu-vehicle-reconciliation-evaluator.service /etc/systemd/system/
sudo cp deploy/systemd/lingniu-vehicle-reconciliation-evaluator.timer /etc/systemd/system/
sudo systemctl daemon-reload
sudo systemctl enable --now lingniu-vehicle-platform lingniu-vehicle-alert-evaluator lingniu-vehicle-alert-stream-evaluator
sudo systemctl enable --now lingniu-vehicle-platform lingniu-vehicle-alert-evaluator lingniu-vehicle-alert-stream-evaluator lingniu-vehicle-alert-notification-dispatcher
sudo systemctl status --no-pager lingniu-vehicle-alert-evaluator
sudo systemctl status --no-pager lingniu-vehicle-alert-stream-evaluator
sudo systemctl status --no-pager lingniu-vehicle-alert-notification-dispatcher
sudo journalctl -u lingniu-vehicle-alert-evaluator -n 100 --no-pager
sudo journalctl -u lingniu-vehicle-alert-stream-evaluator -n 100 --no-pager
sudo journalctl -u lingniu-vehicle-alert-notification-dispatcher -n 100 --no-pager
# Only after a successful manual OneOS API sync and projection audit:
sudo systemctl enable --now lingniu-vehicle-oneos-scope-sync.timer
sudo systemctl status --no-pager lingniu-vehicle-oneos-scope-sync.timer
@@ -281,6 +367,7 @@ curl -fsS -X POST -H "$AUTH_HEADER" -H 'Content-Type: application/json' \
curl -fsS -X POST -H "$AUTH_HEADER" -H 'Content-Type: application/json' -d '{"limit":5,"offset":0}' \
http://127.0.0.1:20300/api/v2/alerts/events
curl -fsS -H "$AUTH_HEADER" http://127.0.0.1:20300/api/v2/alerts/rules
curl -fsS -H "$AUTH_HEADER" 'http://127.0.0.1:20300/api/v2/alerts/rules/library?lifecycle=current&status=all&limit=10&offset=0'
curl -fsS -H "$AUTH_HEADER" 'http://127.0.0.1:20300/api/v2/alerts/notifications?unreadOnly=true&limit=5'
curl -fsS -H "$AUTH_HEADER" 'http://127.0.0.1:20300/api/v2/reconciliation/summary?days=30'
curl -fsS -X POST -H "$AUTH_HEADER" -H 'Content-Type: application/json' \