shishengliang/lingniu-vehicle-ingest
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat(operations): add auditable source provider maintenance

Browse Source
This commit is contained in:
lingniu
2026-07-16 20:01:08 +08:00
parent c270140006
commit 9a5e6e0c4f
19 changed files with 430 additions and 67 deletions
Download Patch File Download Diff File Expand all files Collapse all files

7
vehicle-data-platform/docs/deployment.md
View File

@@ -203,10 +203,11 @@ test -n "$MYSQL_DSN"
/opt/lingniu-vehicle-platform/releases/$PLATFORM_RELEASE/deploy/migrations/015_vehicle_source_policy_audit.sql \
/opt/lingniu-vehicle-platform/releases/$PLATFORM_RELEASE/deploy/migrations/016_business_scope_dimensions.sql \
/opt/lingniu-vehicle-platform/releases/$PLATFORM_RELEASE/deploy/migrations/017_reconciliation_center.sql \
/opt/lingniu-vehicle-platform/releases/$PLATFORM_RELEASE/deploy/migrations/018_vehicle_oem_audit.sql
/opt/lingniu-vehicle-platform/releases/$PLATFORM_RELEASE/deploy/migrations/018_vehicle_oem_audit.sql \
/opt/lingniu-vehicle-platform/releases/$PLATFORM_RELEASE/deploy/migrations/019_vehicle_source_provider.sql
```
The API guards the access-threshold tables for compatibility, while alert and reconciliation APIs deliberately require their migrations to exist. Run every numbered migration explicitly before switching traffic so DDL permission and index creation failures are caught early. The migration journal records filename and SHA-256 and refuses a changed file; duplicate forward `ADD COLUMN` and `CREATE INDEX` statements are tolerated only when resuming partially executed MySQL DDL. Full-line SQL comments are removed before statement splitting, so punctuation in a comment cannot become executable SQL. Migration `008` adds forward-compatible access evidence columns and an index to the gateway-owned realtime snapshot table without changing its `(protocol, vin)` primary key. Migration `009` creates the per-group/topic/partition event-time checkpoint used to make MySQL effects authoritative before Kafka offsets are committed. Migration `012` creates the atomic, versioned business Scope projection; its reserved `row_number` column is quoted for production MySQL compatibility. Migration `014` backfills active vehicle-grant start times, creates the grant-interval history table and adds the active time lookup index; apply it before starting an API binary that writes grant history. Migration `015` adds platform-owned optimistic versions and immutable audits for per-vehicle location-source policy changes. It does not alter the gateway election SQL or expose the gateway `source_key`; the API resolves an opaque `sourceRef` server-side and the gateway applies the saved policy on the next valid vehicle report. Migration `016` adds customer name, department and responsible-person dimensions plus bounded lookup indexes to the platform-owned Scope projection. Migration `017` creates the reconciliation run, issue and immutable action tables; the unique fingerprint index is the database-level duplicate-work-item guard. Migration `018` records every OneOS-sourced brand fill before the platform identity OEM field changes; it does not grant access to OneOS or run a cross-database sync.
The API guards the access-threshold tables for compatibility, while alert and reconciliation APIs deliberately require their migrations to exist. Run every numbered migration explicitly before switching traffic so DDL permission and index creation failures are caught early. The migration journal records filename and SHA-256 and refuses a changed file; duplicate forward `ADD COLUMN` and `CREATE INDEX` statements are tolerated only when resuming partially executed MySQL DDL. Full-line SQL comments are removed before statement splitting, so punctuation in a comment cannot become executable SQL. Migration `008` adds forward-compatible access evidence columns and an index to the gateway-owned realtime snapshot table without changing its `(protocol, vin)` primary key. Migration `009` creates the per-group/topic/partition event-time checkpoint used to make MySQL effects authoritative before Kafka offsets are committed. Migration `012` creates the atomic, versioned business Scope projection; its reserved `row_number` column is quoted for production MySQL compatibility. Migration `014` backfills active vehicle-grant start times, creates the grant-interval history table and adds the active time lookup index; apply it before starting an API binary that writes grant history. Migration `015` adds platform-owned optimistic versions and immutable audits for per-vehicle location-source policy changes. It does not alter the gateway election SQL or expose the gateway `source_key`; the API resolves an opaque `sourceRef` server-side and the gateway applies the saved policy on the next valid vehicle report. Migration `016` adds customer name, department and responsible-person dimensions plus bounded lookup indexes to the platform-owned Scope projection. Migration `017` creates the reconciliation run, issue and immutable action tables; the unique fingerprint index is the database-level duplicate-work-item guard. Migration `018` records every OneOS-sourced brand fill before the platform identity OEM field changes; it does not grant access to OneOS or run a cross-database sync. Migration `019` stores administrator-maintained provider names by exact vehicle/protocol/source key and writes an immutable versioned audit. The browser still receives only the SHA-256 `sourceRef`; raw source keys remain server-side.
`docs/oneos-brand-backfill.sql` is an explicit one-time calibration, not a service dependency. It reads OneOS vehicle/model master data, inserts immutable source evidence, and fills only empty `vehicle_identity_binding.oem` values. Review the pre-run count, apply migration `018`, run the script once through the migration runner or a transaction-capable MySQL client, and verify its final audit/missing counts. Never overwrite a non-empty platform brand and never schedule this script; future updates belong in the formal OneOS API.
@@ -214,6 +215,8 @@ Production release `reconciliation-summary-fix-20260716191153` applied migration
Production release `vehicle-count-authority-exact-20260716193428` made `vehicle_identity_binding` the authoritative denominator for vehicle coverage, coverage summaries, missing-source statistics and service readiness. Authenticated production smoke returned 1,024 vehicles from vehicle query, monitor, access management and source readiness, while keeping the 11 unbound realtime-only VINs as a separate `identityRequiredVehicles` operational metric and 11 `UNBOUND_SOURCE` reconciliation issues. Do not reintroduce the realtime snapshot union into formal vehicle totals; an unbound source becomes a vehicle only after the identity-binding workflow succeeds.
Production release `source-provider-collation-hotfix-20260716195830` applied migration `019` and added platform-owned provider maintenance by opaque `sourceRef`. Provider changes require a separate authoritative evidence note, use the same optimistic vehicle-source version, preserve the existing gateway-policy remark, and write an immutable provider audit. Diagnostic and access projections prefer the verified override while raw `source_key` remains server-side. The first release exposed a production-only collation difference between gateway-owned source tables and platform-owned tables; the hotfix uses exact binary VIN/protocol/source-key joins, avoiding fuzzy matching and restoring both source diagnostic and access detail. Authenticated smoke on `LB9A32A21R0LS1464` verified rejected missing evidence without a version change, admin-only writes, operator read-only access, viewer denial, `v1→v2`, unchanged policy remark, immediate `JT808 / G7s` access evidence, and source-key redaction. The platform, both alert evaluators and reconciliation timer remained active.
Release `oneos-api-client-20260716184601` applied migrations `012` and `016` and deployed the switchable API/database sync binary. The projection is intentionally empty (`active_version` unset) until a verified OneOS endpoint is configured. A production fail-closed smoke with `ONEOS_SCOPE_SOURCE=api` and no endpoint exited nonzero without publishing any row. Do not install or enable the timer until the endpoint, service token, signing secret, source-IP restriction and a representative snapshot have passed the contract checks below.
Production release `source-diagnosis-stable-20260716173740` applied migration `015` before switching API traffic. The release gate verified 23 current assets and 42 compatibility assets. The authenticated diagnostic smoke used a real multi-source vehicle, confirmed that `source_key` was absent, and exercised the admin PUT route with values identical to the current policy; version, audit count and recommended source remained unchanged. Viewer/operator/admin access returned 403/200/200, median response time across 20 reads was approximately 70 ms (P95 79 ms), and the platform plus both alert evaluators remained active.