shishengliang/lingniu-vehicle-ingest
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat(history): scope export tasks to owners

Browse Source
This commit is contained in:
lingniu
2026-07-16 17:00:33 +08:00
parent a2722e4afd
commit 38b056f5f1
18 changed files with 528 additions and 100 deletions
Download Patch File Download Diff File Expand all files Collapse all files

12
vehicle-data-platform/docs/deployment.md
View File

@@ -129,7 +129,9 @@ ALERT_STREAM_LATENESS_SEC=120
All three platform systemd units first load `/opt/lingniu-go-native/env/base.env` for the existing MySQL, Redis, TDengine and Kafka connection settings, then load `platform.env` for platform-specific overrides. `DATA_MODE=production` is mandatory on ECS: a missing or unreachable MySQL connection returns `DATA_STORE_UNAVAILABLE` instead of silently serving demonstration data. Use `DATA_MODE=mock` only for local development.
`EXPORT_DIR` must point outside the release symlink. The API writes CSV files and an atomic `jobs.json` index there; completed tasks survive API restarts, while interrupted queued/running jobs are marked failed and can be recreated. Exports run one at a time, use 5,000-row forward-only TDengine cursors instead of `OFFSET`, stream into a `.part` file, and atomically publish the final CSV only after flush, `fsync`, and close succeed. A task is limited to five vehicles, 31 days, 32 metrics, 1,000,000 rows, and 30 minutes. Both the initial count and observed rows enforce the row cap so late-arriving data cannot bypass it. Keep the directory mode `0750` and include it in retention/backup policy. Export list and download APIs require `operator` or `admin`.
`EXPORT_DIR` must point outside the release symlink. The API writes mode-`0640` CSV files and an atomic `jobs.json` index there; completed tasks survive API restarts, while interrupted queued/running jobs are marked failed and can be recreated. Every job persists its owner account, role/user type, customer/tenant references, resolved VINs and per-vehicle time scope. Non-admin principals can list and download only their own jobs; customers additionally pass a live account-status and vehicle-grant check before execution, every 5,000-row batch and every download. Old ownerless jobs remain visible only to administrators. The browser downloads through an authenticated Blob request rather than a direct anchor, so the Bearer credential is always present.
Exports run one at a time, use 5,000-row forward-only TDengine cursors instead of `OFFSET`, stream into a `.part` file, and atomically publish the final CSV only after flush, `fsync`, and close succeed. A task is limited to five vehicles, 31 days, 32 metrics, 1,000,000 rows, and 30 minutes. Both the initial count and observed rows enforce the row cap so late-arriving data cannot bypass it. Keep the directory mode `0750`, preserve `jobs.json` and completed CSVs across releases, and include the directory in retention/backup policy. Export routes require an authenticated viewer-capable principal; ownership and current Scope are enforced inside the export service instead of relying only on role middleware.
Release verification for this path includes the opt-in synthetic million-row gate:
@@ -138,14 +140,14 @@ cd vehicle-data-platform/apps/api
EXPORT_MILLION_TEST=1 go test ./internal/platform -run '^TestHistoryExportMillionRows$' -count=1 -v
```
Production smoke must create an export for an active VIN, wait for `completed`, verify `rowCount == processedRows == totalRows`, compare the downloaded SHA-256 and byte count before and after one API restart, and confirm that no `.part` file remains. CSV files begin with query-range and metric/unit metadata followed by the data header.
Production smoke must create an export for an active VIN, wait for `completed`, verify `rowCount == processedRows == totalRows`, and assert that owner, role, resolved VINs and time scope are present. Download with the same Bearer principal, confirm the CSV contains export audit, vehicle Scope, query-range, metric/unit and quality metadata, and verify a guessed task ID returns 404. Compare the downloaded SHA-256 and byte count before and after one API restart, and confirm that no `.part` file remains. The automated gate must separately cover two mutually isolated customer principals plus account/vehicle revocation during execution.
`AUTH_MODE=enforce` is mandatory on ECS. Tokens are stored as SHA-256 comparisons in memory and sent as Bearer credentials; the browser stores the entered token only in `sessionStorage`. Roles are cumulative:
| Role | Permissions |
| --- | --- |
| `viewer` | Read pages, query data and inspect evidence |
| `operator` | Viewer permissions plus exports, alert actions and notification reads |
| `viewer` | Read pages, query data and inspect evidence; may create/download only its own history exports when the menu is available |
| `operator` | Viewer permissions plus alert actions and notification reads; export access remains owner-bound |
| `admin` | Operator permissions plus rule and access-threshold configuration |
After editing the environment file, run `chmod 600 /opt/lingniu-vehicle-platform/env/platform.env`. Never put a real token in Git, static JavaScript, shell history or deployment logs.
@@ -246,6 +248,8 @@ The history-series gate must use an active production VIN and a current local-ti
The history-query gate must also request both `location` and `raw` through `/api/v2/history/query`. Every returned row must contain a non-empty `qualityReason`; RAW columns must include fields discovered from the bounded fetched scope rather than only the visible page. In the browser, verify field-name/key search, multi-select, the single visual time-range control, selected-field consistency across table/trend/export, and absence of a RAW-evidence placeholder when `evidenceId` is unavailable. Keep the existing maximum of five vehicles, 200 rows per page and 1,000 fetched rows for interactive field discovery.
The history-export gate must verify owner-bound listing and authenticated Blob download. A customer A task must be absent from customer B's list and return 404 when B guesses its ID; an administrator may audit it. Persisted job JSON and CSV metadata must retain the owner and exact per-VIN time scope after an API restart. Disabling the customer or revoking any required vehicle grant must stop an incomplete task and deny download of a previously completed file.
Track smoke must also select the final playback event and verify synchronized SOC/direction/alarm availability plus a resolved current address. For high-frequency sources, confirm positive subsecond samples do not become alternating zero-second `数据间隔` segments; only non-increasing timestamps or intervals over ten minutes are gaps. Access smoke should exercise `model`/`provider` and one paired first/latest receive-time range, then confirm the filter survives a shareable URL and every returned row remains in scope.
Global-monitor smoke must verify `zoom=5` returns clusters, a city `bounds` at `zoom=13` returns at most 2,000 lightweight points, and invalid/reversed bounds return HTTP 400. In the browser, the rail renders at most 200 vehicle rows while the legend reports the independent map mode. Selecting `driving` or `idle` must constrain both the server-side count and every loaded row. A unique keyword result must discard the previous viewport and focus at point zoom. The synthetic release gate is `go test ./internal/platform -run TestMonitorMapTenThousandVehicles -count=1` plus `go test ./internal/platform -run '^$' -bench BenchmarkMonitorMapTenThousandVehicles -benchtime=30x -benchmem`.