shishengliang/lingniu-vehicle-ingest
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat(operations): add automated reconciliation center

Browse Source
This commit is contained in:
lingniu
2026-07-16 19:14:53 +08:00
parent bbab018d55
commit d67f42b4f4
25 changed files with 1697 additions and 39 deletions
Download Patch File Download Diff File Expand all files Collapse all files

25
vehicle-data-platform/docs/deployment.md
View File

@@ -14,6 +14,7 @@ GOOS=linux GOARCH=amd64 CGO_ENABLED=0 go build -trimpath -ldflags='-s -w' -o ../
GOOS=linux GOARCH=amd64 CGO_ENABLED=0 go build -trimpath -ldflags='-s -w' -o ../../dist/alert-stream-evaluator ./cmd/alert-stream-evaluator
GOOS=linux GOARCH=amd64 CGO_ENABLED=0 go build -trimpath -ldflags='-s -w' -o ../../dist/platform-migrate ./cmd/platform-migrate
GOOS=linux GOARCH=amd64 CGO_ENABLED=0 go build -trimpath -ldflags='-s -w' -o ../../dist/oneos-scope-sync ./cmd/oneos-scope-sync
GOOS=linux GOARCH=amd64 CGO_ENABLED=0 go build -trimpath -ldflags='-s -w' -o ../../dist/reconciliation-evaluator ./cmd/reconciliation-evaluator
# Optional release/performance gate; this binary is run on demand, not installed as a service.
GOOS=linux GOARCH=amd64 CGO_ENABLED=0 go build -trimpath -ldflags='-s -w' -o ../../dist/alert-benchmark ./cmd/alert-benchmark
```
@@ -66,14 +67,15 @@ PREVIOUS_WEB=/opt/lingniu-vehicle-platform/releases/$PREVIOUS_RELEASE/web
"$PREVIOUS_WEB/.release-assets"
```
`deploy/install-web-release.sh` also accepts an optional third argument containing a newly built `platform-api` and an optional fourth argument containing a newly built `oneos-scope-sync`. When provided, it atomically publishes those binaries with the Web. The installer inherits every runtime binary that exists in the previous release (`alert-evaluator`, `alert-stream-evaluator`, `platform-migrate`, `oneos-scope-sync` and the optional benchmark) before switching the symlink, so a later systemd restart cannot fail because a Web-oriented release omitted an unchanged binary:
`deploy/install-web-release.sh` also accepts an optional third argument containing a newly built `platform-api`, an optional fourth argument containing a newly built `oneos-scope-sync`, and an optional fifth argument containing a newly built `reconciliation-evaluator`. When provided, it atomically publishes those binaries with the Web. The installer inherits every runtime binary that exists in the previous release (`alert-evaluator`, `alert-stream-evaluator`, `platform-migrate`, `oneos-scope-sync`, `reconciliation-evaluator` and the optional benchmark) before switching the symlink, so a later systemd restart cannot fail because a Web-oriented release omitted an unchanged binary:
```bash
deploy/install-web-release.sh \
"$PLATFORM_RELEASE" \
"/tmp/$PLATFORM_RELEASE-web.tar.gz" \
"/tmp/$PLATFORM_RELEASE-platform-api" \
"/tmp/$PLATFORM_RELEASE-oneos-scope-sync"
"/tmp/$PLATFORM_RELEASE-oneos-scope-sync" \
"/tmp/$PLATFORM_RELEASE-reconciliation-evaluator"
```
## Environment
@@ -108,6 +110,7 @@ ONEOS_SCOPE_API_MAX_ATTEMPTS=3
ONEOS_SCOPE_SYNC_TIMEOUT_SEC=60
ONEOS_SCOPE_MAX_REJECTED=100
ONEOS_SCOPE_MAX_REJECT_RATIO=0.10
RECONCILIATION_TIMEOUT_SEC=300
REQUEST_TIMEOUT_MS=5000
AMAP_WEB_JS_KEY=***
AMAP_SECURITY_JS_CODE=***
@@ -198,10 +201,13 @@ test -n "$MYSQL_DSN"
/opt/lingniu-vehicle-platform/releases/$PLATFORM_RELEASE/deploy/migrations/013_platform_identity_access.sql \
/opt/lingniu-vehicle-platform/releases/$PLATFORM_RELEASE/deploy/migrations/014_customer_vehicle_grant_time.sql \
/opt/lingniu-vehicle-platform/releases/$PLATFORM_RELEASE/deploy/migrations/015_vehicle_source_policy_audit.sql \
/opt/lingniu-vehicle-platform/releases/$PLATFORM_RELEASE/deploy/migrations/016_business_scope_dimensions.sql
/opt/lingniu-vehicle-platform/releases/$PLATFORM_RELEASE/deploy/migrations/016_business_scope_dimensions.sql \
/opt/lingniu-vehicle-platform/releases/$PLATFORM_RELEASE/deploy/migrations/017_reconciliation_center.sql
```
The API guards the access-threshold tables for compatibility, while alert APIs deliberately require the alert migrations to exist. Run every numbered migration explicitly before switching traffic so DDL permission and index creation failures are caught early. The migration journal records filename and SHA-256 and refuses a changed file; duplicate forward `ADD COLUMN` and `CREATE INDEX` statements are tolerated only when resuming partially executed MySQL DDL. Full-line SQL comments are removed before statement splitting, so punctuation in a comment cannot become executable SQL. Migration `008` adds forward-compatible access evidence columns and an index to the gateway-owned realtime snapshot table without changing its `(protocol, vin)` primary key. Migration `009` creates the per-group/topic/partition event-time checkpoint used to make MySQL effects authoritative before Kafka offsets are committed. Migration `012` creates the atomic, versioned business Scope projection; its reserved `row_number` column is quoted for production MySQL compatibility. Migration `014` backfills active vehicle-grant start times, creates the grant-interval history table and adds the active time lookup index; apply it before starting an API binary that writes grant history. Migration `015` adds platform-owned optimistic versions and immutable audits for per-vehicle location-source policy changes. It does not alter the gateway election SQL or expose the gateway `source_key`; the API resolves an opaque `sourceRef` server-side and the gateway applies the saved policy on the next valid vehicle report. Migration `016` adds customer name, department and responsible-person dimensions plus bounded lookup indexes to the platform-owned Scope projection.
The API guards the access-threshold tables for compatibility, while alert and reconciliation APIs deliberately require their migrations to exist. Run every numbered migration explicitly before switching traffic so DDL permission and index creation failures are caught early. The migration journal records filename and SHA-256 and refuses a changed file; duplicate forward `ADD COLUMN` and `CREATE INDEX` statements are tolerated only when resuming partially executed MySQL DDL. Full-line SQL comments are removed before statement splitting, so punctuation in a comment cannot become executable SQL. Migration `008` adds forward-compatible access evidence columns and an index to the gateway-owned realtime snapshot table without changing its `(protocol, vin)` primary key. Migration `009` creates the per-group/topic/partition event-time checkpoint used to make MySQL effects authoritative before Kafka offsets are committed. Migration `012` creates the atomic, versioned business Scope projection; its reserved `row_number` column is quoted for production MySQL compatibility. Migration `014` backfills active vehicle-grant start times, creates the grant-interval history table and adds the active time lookup index; apply it before starting an API binary that writes grant history. Migration `015` adds platform-owned optimistic versions and immutable audits for per-vehicle location-source policy changes. It does not alter the gateway election SQL or expose the gateway `source_key`; the API resolves an opaque `sourceRef` server-side and the gateway applies the saved policy on the next valid vehicle report. Migration `016` adds customer name, department and responsible-person dimensions plus bounded lookup indexes to the platform-owned Scope projection. Migration `017` creates the reconciliation run, issue and immutable action tables; the unique fingerprint index is the database-level duplicate-work-item guard.
Production release `reconciliation-summary-fix-20260716191153` applied migration `017`, deployed the API, Web and reconciliation evaluator, and enabled the daily timer. The initial run detected 1,622 active differences in about 1.43 seconds; an immediate repeat completed in about 0.88 seconds with `new=0`. The authenticated smoke verified summary/list/detail for administrator and operator, viewer denial, optimistic-version conflict handling, evidence-key redaction, 50-row pagination, responsive desktop/mobile rendering and no browser console errors.
Release `oneos-api-client-20260716184601` applied migrations `012` and `016` and deployed the switchable API/database sync binary. The projection is intentionally empty (`active_version` unset) until a verified OneOS endpoint is configured. A production fail-closed smoke with `ONEOS_SCOPE_SOURCE=api` and no endpoint exited nonzero without publishing any row. Do not install or enable the timer until the endpoint, service token, signing secret, source-IP restriction and a representative snapshot have passed the contract checks below.
@@ -214,6 +220,8 @@ sudo cp deploy/systemd/lingniu-vehicle-alert-evaluator.service /etc/systemd/syst
sudo cp deploy/systemd/lingniu-vehicle-alert-stream-evaluator.service /etc/systemd/system/
sudo cp deploy/systemd/lingniu-vehicle-oneos-scope-sync.service /etc/systemd/system/
sudo cp deploy/systemd/lingniu-vehicle-oneos-scope-sync.timer /etc/systemd/system/
sudo cp deploy/systemd/lingniu-vehicle-reconciliation-evaluator.service /etc/systemd/system/
sudo cp deploy/systemd/lingniu-vehicle-reconciliation-evaluator.timer /etc/systemd/system/
sudo systemctl daemon-reload
sudo systemctl enable --now lingniu-vehicle-platform lingniu-vehicle-alert-evaluator lingniu-vehicle-alert-stream-evaluator
sudo systemctl status --no-pager lingniu-vehicle-alert-evaluator
@@ -224,6 +232,11 @@ sudo journalctl -u lingniu-vehicle-alert-stream-evaluator -n 100 --no-pager
sudo systemctl enable --now lingniu-vehicle-oneos-scope-sync.timer
sudo systemctl status --no-pager lingniu-vehicle-oneos-scope-sync.timer
sudo journalctl -u lingniu-vehicle-oneos-scope-sync.service -n 100 --no-pager
# Run once before enabling the timer; a failure must leave the timer disabled.
sudo systemctl start lingniu-vehicle-reconciliation-evaluator.service
sudo journalctl -u lingniu-vehicle-reconciliation-evaluator.service -n 100 --no-pager
sudo systemctl enable --now lingniu-vehicle-reconciliation-evaluator.timer
sudo systemctl list-timers lingniu-vehicle-reconciliation-evaluator.timer --no-pager
```
## Health
@@ -256,6 +269,10 @@ curl -fsS -X POST -H "$AUTH_HEADER" -H 'Content-Type: application/json' -d '{"li
http://127.0.0.1:20300/api/v2/alerts/events
curl -fsS -H "$AUTH_HEADER" http://127.0.0.1:20300/api/v2/alerts/rules
curl -fsS -H "$AUTH_HEADER" 'http://127.0.0.1:20300/api/v2/alerts/notifications?unreadOnly=true&limit=5'
curl -fsS -H "$AUTH_HEADER" 'http://127.0.0.1:20300/api/v2/reconciliation/summary?days=30'
curl -fsS -X POST -H "$AUTH_HEADER" -H 'Content-Type: application/json' \
-d '{"status":"active","limit":50,"offset":0}' \
http://127.0.0.1:20300/api/v2/reconciliation/issues
curl -fsS http://127.0.0.1:20300/app-config.js
curl -fsS http://127.0.0.1:20300/ | grep -q '<div id="root"></div>'
MAIN_ASSET="$(sed -n 's/.*src="\([^"]*\.js\)".*/\1/p' /opt/lingniu-vehicle-platform/current/web/index.html | head -1)"