shishengliang/lingniu-vehicle-ingest
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat(platform): consolidate production vehicle data workflows

Browse Source
This commit is contained in:
lingniu
2026-07-15 23:26:29 +08:00
parent 6cddc0a43d
commit 3fabcf181a
59 changed files with 6849 additions and 3532 deletions
Download Patch File Download Diff File Expand all files Collapse all files

24
vehicle-data-platform/docs/deployment.md
View File

@@ -13,6 +13,7 @@ GOOS=linux GOARCH=amd64 CGO_ENABLED=0 go build -trimpath -ldflags='-s -w' -o ../
GOOS=linux GOARCH=amd64 CGO_ENABLED=0 go build -trimpath -ldflags='-s -w' -o ../../dist/alert-evaluator ./cmd/alert-evaluator
GOOS=linux GOARCH=amd64 CGO_ENABLED=0 go build -trimpath -ldflags='-s -w' -o ../../dist/alert-stream-evaluator ./cmd/alert-stream-evaluator
GOOS=linux GOARCH=amd64 CGO_ENABLED=0 go build -trimpath -ldflags='-s -w' -o ../../dist/platform-migrate ./cmd/platform-migrate
GOOS=linux GOARCH=amd64 CGO_ENABLED=0 go build -trimpath -ldflags='-s -w' -o ../../dist/oneos-scope-sync ./cmd/oneos-scope-sync
# Optional release/performance gate; this binary is run on demand, not installed as a service.
GOOS=linux GOARCH=amd64 CGO_ENABLED=0 go build -trimpath -ldflags='-s -w' -o ../../dist/alert-benchmark ./cmd/alert-benchmark
```
@@ -44,6 +45,10 @@ EXPORT_DIR=/opt/lingniu-vehicle-platform/data/exports
AUTH_MODE=enforce
# JSON array with viewer/operator/admin principals. Keep this file mode 0600.
AUTH_TOKENS_JSON=[{"token":"<at-least-16-random-characters>","name":"ecs-admin","role":"admin"}]
ONEOS_MYSQL_DSN=vehicle_scope_reader:***@tcp(rm-bp179zbv481rnw3e2.mysql.rds.aliyuncs.com:3306)/ln_asset_management?parseTime=true&loc=Asia%2FShanghai
ONEOS_SCOPE_SYNC_TIMEOUT_SEC=60
ONEOS_SCOPE_MAX_REJECTED=100
ONEOS_SCOPE_MAX_REJECT_RATIO=0.10
REQUEST_TIMEOUT_MS=5000
AMAP_WEB_JS_KEY=***
AMAP_SECURITY_JS_CODE=***
@@ -91,6 +96,14 @@ Production smoke must create an export for an active VIN, wait for `completed`,
After editing the environment file, run `chmod 600 /opt/lingniu-vehicle-platform/env/platform.env`. Never put a real token in Git, static JavaScript, shell history or deployment logs.
`ONEOS_MYSQL_DSN` must use a dedicated account with direct table-level `SELECT` grants only. The sync binary runs `SHOW GRANTS FOR CURRENT_USER` before every read and accepts only global `USAGE` plus `SELECT` on the seven tables used by its query. It refuses database/global reads, `ALL PRIVILEGES`, DML, DDL, PROCESS, replication, roles, `SHOW VIEW`, or any other privilege. It then opens a repeatable-read, read-only transaction, applies a 10-second statement timeout, classifies invalid customer/contract relationships, and atomically publishes a content-addressed local snapshot. Never point it at the existing `ln-bi` account: production audit showed that account still has broad write and replication privileges.
Have an RDS administrator review and run `docs/oneos-scope-reader-provision.sql` separately. It restricts the login source to the verified ECS private address and grants `SELECT` on only the seven source tables used by the query. Do not include that DDL in application deployment or migration automation. Store `platform.env` as root-owned mode `0600`, and verify `SHOW GRANTS` before enabling the timer.
The active lifecycle comes from OneOS delivery and return task facts: the newest active delivery must have `delivery_status IN (2,3)`, and it must not have an active return task with `status IN (2,3,5)`. The sync intentionally does not use `vehicle_lease_order_record.last_return_time`, because OneOS currently writes that aggregate field while a return form is still a draft. The aggregate record remains a separate customer-ownership cross-check; missing, duplicate, or conflicting ownership fails closed.
The default two-minute timer reads one bounded current-scope result set (639 rows at the 2026-07-14 transactional-fact baseline), so it avoids per-request access to OneOS and adds negligible RDS load. A run is rejected before publication if it has no accepted rows, more than 100 rejected rows, or a rejected ratio above 10%. Current baseline is 613 accepted and 26 quarantined. An unchanged content hash records a successful run without duplicating the snapshot.
## Forward Migration
Before switching the API symlink, apply the idempotent access-threshold migration with a MySQL account allowed to create/alter platform-owned tables:
@@ -109,7 +122,8 @@ test -n "$MYSQL_DSN"
/opt/lingniu-vehicle-platform/releases/$PLATFORM_RELEASE/deploy/migrations/008_access_projection.sql \
/opt/lingniu-vehicle-platform/releases/$PLATFORM_RELEASE/deploy/migrations/009_alert_stream_checkpoint.sql \
/opt/lingniu-vehicle-platform/releases/$PLATFORM_RELEASE/deploy/migrations/010_alert_stream_metric_mapping.sql \
/opt/lingniu-vehicle-platform/releases/$PLATFORM_RELEASE/deploy/migrations/011_alert_stream_invalid_evidence.sql
/opt/lingniu-vehicle-platform/releases/$PLATFORM_RELEASE/deploy/migrations/011_alert_stream_invalid_evidence.sql \
/opt/lingniu-vehicle-platform/releases/$PLATFORM_RELEASE/deploy/migrations/012_business_scope_projection.sql
```
The API guards the access-threshold tables for compatibility, while alert APIs deliberately require the alert migrations to exist. Run every numbered migration explicitly before switching traffic so DDL permission and index creation failures are caught early. The migration journal records filename and SHA-256 and refuses a changed file; duplicate forward `ADD COLUMN` and `CREATE INDEX` statements are tolerated only when resuming partially executed MySQL DDL. Full-line SQL comments are removed before statement splitting, so punctuation in a comment cannot become executable SQL. Migration `008` adds forward-compatible access evidence columns and an index to the gateway-owned realtime snapshot table without changing its `(protocol, vin)` primary key. Migration `009` creates the per-group/topic/partition event-time checkpoint used to make MySQL effects authoritative before Kafka offsets are committed.
@@ -119,12 +133,16 @@ Install and start the evaluator as a separate unit only after API smoke checks a
```bash
sudo cp deploy/systemd/lingniu-vehicle-alert-evaluator.service /etc/systemd/system/
sudo cp deploy/systemd/lingniu-vehicle-alert-stream-evaluator.service /etc/systemd/system/
sudo cp deploy/systemd/lingniu-vehicle-oneos-scope-sync.service /etc/systemd/system/
sudo cp deploy/systemd/lingniu-vehicle-oneos-scope-sync.timer /etc/systemd/system/
sudo systemctl daemon-reload
sudo systemctl enable --now lingniu-vehicle-platform lingniu-vehicle-alert-evaluator lingniu-vehicle-alert-stream-evaluator
sudo systemctl enable --now lingniu-vehicle-platform lingniu-vehicle-alert-evaluator lingniu-vehicle-alert-stream-evaluator lingniu-vehicle-oneos-scope-sync.timer
sudo systemctl status --no-pager lingniu-vehicle-alert-evaluator
sudo systemctl status --no-pager lingniu-vehicle-alert-stream-evaluator
sudo journalctl -u lingniu-vehicle-alert-evaluator -n 100 --no-pager
sudo journalctl -u lingniu-vehicle-alert-stream-evaluator -n 100 --no-pager
sudo systemctl status --no-pager lingniu-vehicle-oneos-scope-sync.timer
sudo journalctl -u lingniu-vehicle-oneos-scope-sync.service -n 100 --no-pager
```
## Health
@@ -162,7 +180,7 @@ curl -fsS "http://127.0.0.1:20300$MAIN_ASSET" >/dev/null
unset PLATFORM_TOKEN AUTH_HEADER
```
The release gate must include both the root document and its hashed main asset. API-only smoke checks are insufficient because an incomplete archive can leave the service healthy while the browser returns 404. Before upload, verify the archive contains `web/index.html`, `platform-api`, both evaluator binaries, `platform-migrate`, every numbered migration and all three platform systemd units.
The release gate must include both the root document and its hashed main asset. API-only smoke checks are insufficient because an incomplete archive can leave the service healthy while the browser returns 404. Before upload, verify the archive contains `web/index.html`, `platform-api`, both evaluator binaries, `platform-migrate`, `oneos-scope-sync`, every numbered migration and all platform systemd units.
When `alertStream.lastInvalidCode` reports a recent `missing_vin_jt808`, query `/api/v2/access/unresolved-identities` as a viewer and hand the masked evidence to the identity owner. The response must not contain raw `phone` or unmasked identifier fields. Confirm the evidence against the GPS provider or vehicle owner, then maintain the authoritative `vehicle_identity_binding`; never derive VIN from the terminal number. The queue excludes a terminal after a valid binding exists. Retain the invalid counter as audit evidence and verify the recent warning clears after five minutes without another invalid frame.