feat(history): scope export tasks to owners
This commit is contained in:
@@ -1,14 +1,17 @@
|
||||
# 车辆数据中台客户 Scope API 覆盖矩阵
|
||||
|
||||
更新时间:2026-07-14
|
||||
更新时间:2026-07-16
|
||||
|
||||
状态:安全审计与实施设计;尚未开放客户主体。
|
||||
状态:本地客户主体与车辆/时间 Scope 已实施;OneOS 联邦身份仍按后续接口契约接入。
|
||||
|
||||
## 1. 当前结论
|
||||
|
||||
车辆中台当前的 Bearer Token 只表达 `viewer/operator/admin` 功能角色,不表达客户或车辆数据范围。
|
||||
因此,即使 OneOS 已能登录客户,也不能直接把客户 token 映射成 `viewer`:这会让客户读取全量车辆、轨迹、
|
||||
告警、统计及其他人的导出文件。
|
||||
车辆中台已支持本地 `admin/customer` 账号、菜单权限、车辆 VIN Scope 和授权有效时间。认证中间件把
|
||||
`subjectId/userType/customerRef/tenantRef/authProvider` 以及当前车辆授权注入 `Principal`;车辆、监控、轨迹、
|
||||
历史、里程和导出入口均在服务端校验主体与时间边界。客户不能通过 Query、Header 或 Body 自行覆盖 customer ID。
|
||||
|
||||
OneOS 身份尚不能直接映射为本地 `viewer`。后续只有在 OneOS 提供签名、受众、身份版本和客户关联等稳定接口后,
|
||||
才能映射为同一强类型 Principal;在此之前继续使用车辆中台本地客户账号,不读取或修改 OneOS 业务逻辑。
|
||||
|
||||
2026-07-14 已只读核验 ECS 当前运行状态:
|
||||
|
||||
@@ -17,7 +20,7 @@
|
||||
- `/opt/lingniu-vehicle-platform/env/platform.env` 权限为 `0600 root:root`;
|
||||
- `lingniu-vehicle-platform` 服务已启用且正在运行。
|
||||
|
||||
该配置能阻止匿名访问普通平台 API,但不能提供客户级隔离。
|
||||
该配置负责内部静态 token;客户级隔离由平台账号、会话、菜单和车辆授权表共同提供。
|
||||
|
||||
## 2. 独立发现:高德服务端接口绕过平台鉴权
|
||||
|
||||
@@ -126,26 +129,21 @@ VIN 集合,设置数量上限后再生成参数化过滤。不能先从 TDengi
|
||||
|
||||
禁止应在服务端 capability 层返回 403,不能只隐藏前端菜单。
|
||||
|
||||
### 5.3 导出必须单独重构后才能开放
|
||||
### 5.3 导出 owner、异步 Scope 与审计(已实施)
|
||||
|
||||
当前导出实现有四个 P0 问题:
|
||||
|
||||
1. `CreateHistoryExport` 没有 `context`,创建时无法固化主体;
|
||||
2. `HistoryExportJob` 没有 owner/customer/scope version;
|
||||
3. `ListHistoryExports` 返回全局任务;
|
||||
4. `HistoryExportFile(id)` 只校验 ID 和完成状态,知道 ID 即可下载。
|
||||
|
||||
整改要求:
|
||||
release `export-owner-scope-20260716165737` 已完成:
|
||||
|
||||
- `Create/List/Download` 全部接收 context 和 Principal;
|
||||
- Job 固化 `owner_subject_id`、`customer_id`、`scope_version`、已解析 VIN、时间范围和审计 request ID;
|
||||
- 创建任务时校验每个 VIN,异步执行前再次校验;
|
||||
- 列表只返回当前 owner;下载同时校验 owner 和当前授权,越权返回 404;
|
||||
- 客户停用、车辆授权撤销或 Scope 失效时停止未完成任务并拒绝下载;
|
||||
- 文件名和 CSV 元数据不得包含其他客户信息;文件按 owner 隔离目录并设置 `0640`;
|
||||
- 定期清理过期文件与 jobs 记录。
|
||||
- Job 固化 owner ID、账号、角色/用户类型、认证来源、客户/租户、已解析 VIN 和逐车时间范围;
|
||||
- 创建时同步校验每个 VIN 和历史授权时间;异步开始、每 5,000 行及下载前重新校验客户账号与车辆授权;
|
||||
- 非管理员列表只返回当前 owner;越权或猜测下载 ID 返回 404;管理员保留审计视角;
|
||||
- 客户停用、车辆授权撤销或授权区间变化会停止未完成任务并拒绝历史文件下载;
|
||||
- CSV 文件使用 `0640`,任务索引原子持久化;CSV 元数据包含创建账号、角色、客户、车辆 Scope、查询条件与生成时间;
|
||||
- Web 使用带 Bearer Token 的 Blob 下载,不再通过无 Authorization 的直接链接;
|
||||
- 服务重启后 owner、Scope 和完成文件仍可复核,执行中任务安全失败并要求重新创建。
|
||||
|
||||
在上述改造完成前,客户主体不授予 `export:create` 或 `export:download`。
|
||||
自动化已覆盖两个互斥客户、管理员审计、IDOR、撤权停止和持久化元数据。生产真实导出在重启前后文件
|
||||
SHA-256 一致,伪造 ID 返回 404,且无 `.part` 文件残留。
|
||||
|
||||
## 6. 当前 Scope 与历史 Scope 的边界
|
||||
|
||||
@@ -189,6 +187,6 @@ customer_id, vin, contract_id, valid_from, valid_to, source_version
|
||||
2. 引入强类型 QueryScope,先改车辆解析与单车详情;
|
||||
3. 改所有列表、汇总、地图、统计和 TDengine 查询;
|
||||
4. 改告警详情 IDOR、逆地理编码鉴权限流;
|
||||
5. 重构导出 owner 与异步 Scope;
|
||||
6. 完成双客户自动化测试和生产影子查询;
|
||||
7. 最后为测试客户启用 capabilities,默认保持客户入口关闭。
|
||||
5. 导出 owner 与异步 Scope:已完成;
|
||||
6. 双客户导出自动化与生产管理员实车导出:已完成;完整客户 API 矩阵继续补齐;
|
||||
7. 本地测试客户可按菜单和车辆授权启用;OneOS 联邦客户入口保持关闭,等待正式身份接口。
|
||||
|
||||
@@ -129,7 +129,9 @@ ALERT_STREAM_LATENESS_SEC=120
|
||||
|
||||
All three platform systemd units first load `/opt/lingniu-go-native/env/base.env` for the existing MySQL, Redis, TDengine and Kafka connection settings, then load `platform.env` for platform-specific overrides. `DATA_MODE=production` is mandatory on ECS: a missing or unreachable MySQL connection returns `DATA_STORE_UNAVAILABLE` instead of silently serving demonstration data. Use `DATA_MODE=mock` only for local development.
|
||||
|
||||
`EXPORT_DIR` must point outside the release symlink. The API writes CSV files and an atomic `jobs.json` index there; completed tasks survive API restarts, while interrupted queued/running jobs are marked failed and can be recreated. Exports run one at a time, use 5,000-row forward-only TDengine cursors instead of `OFFSET`, stream into a `.part` file, and atomically publish the final CSV only after flush, `fsync`, and close succeed. A task is limited to five vehicles, 31 days, 32 metrics, 1,000,000 rows, and 30 minutes. Both the initial count and observed rows enforce the row cap so late-arriving data cannot bypass it. Keep the directory mode `0750` and include it in retention/backup policy. Export list and download APIs require `operator` or `admin`.
|
||||
`EXPORT_DIR` must point outside the release symlink. The API writes mode-`0640` CSV files and an atomic `jobs.json` index there; completed tasks survive API restarts, while interrupted queued/running jobs are marked failed and can be recreated. Every job persists its owner account, role/user type, customer/tenant references, resolved VINs and per-vehicle time scope. Non-admin principals can list and download only their own jobs; customers additionally pass a live account-status and vehicle-grant check before execution, every 5,000-row batch and every download. Old ownerless jobs remain visible only to administrators. The browser downloads through an authenticated Blob request rather than a direct anchor, so the Bearer credential is always present.
|
||||
|
||||
Exports run one at a time, use 5,000-row forward-only TDengine cursors instead of `OFFSET`, stream into a `.part` file, and atomically publish the final CSV only after flush, `fsync`, and close succeed. A task is limited to five vehicles, 31 days, 32 metrics, 1,000,000 rows, and 30 minutes. Both the initial count and observed rows enforce the row cap so late-arriving data cannot bypass it. Keep the directory mode `0750`, preserve `jobs.json` and completed CSVs across releases, and include the directory in retention/backup policy. Export routes require an authenticated viewer-capable principal; ownership and current Scope are enforced inside the export service instead of relying only on role middleware.
|
||||
|
||||
Release verification for this path includes the opt-in synthetic million-row gate:
|
||||
|
||||
@@ -138,14 +140,14 @@ cd vehicle-data-platform/apps/api
|
||||
EXPORT_MILLION_TEST=1 go test ./internal/platform -run '^TestHistoryExportMillionRows$' -count=1 -v
|
||||
```
|
||||
|
||||
Production smoke must create an export for an active VIN, wait for `completed`, verify `rowCount == processedRows == totalRows`, compare the downloaded SHA-256 and byte count before and after one API restart, and confirm that no `.part` file remains. CSV files begin with query-range and metric/unit metadata followed by the data header.
|
||||
Production smoke must create an export for an active VIN, wait for `completed`, verify `rowCount == processedRows == totalRows`, and assert that owner, role, resolved VINs and time scope are present. Download with the same Bearer principal, confirm the CSV contains export audit, vehicle Scope, query-range, metric/unit and quality metadata, and verify a guessed task ID returns 404. Compare the downloaded SHA-256 and byte count before and after one API restart, and confirm that no `.part` file remains. The automated gate must separately cover two mutually isolated customer principals plus account/vehicle revocation during execution.
|
||||
|
||||
`AUTH_MODE=enforce` is mandatory on ECS. Tokens are stored as SHA-256 comparisons in memory and sent as Bearer credentials; the browser stores the entered token only in `sessionStorage`. Roles are cumulative:
|
||||
|
||||
| Role | Permissions |
|
||||
| --- | --- |
|
||||
| `viewer` | Read pages, query data and inspect evidence |
|
||||
| `operator` | Viewer permissions plus exports, alert actions and notification reads |
|
||||
| `viewer` | Read pages, query data and inspect evidence; may create/download only its own history exports when the menu is available |
|
||||
| `operator` | Viewer permissions plus alert actions and notification reads; export access remains owner-bound |
|
||||
| `admin` | Operator permissions plus rule and access-threshold configuration |
|
||||
|
||||
After editing the environment file, run `chmod 600 /opt/lingniu-vehicle-platform/env/platform.env`. Never put a real token in Git, static JavaScript, shell history or deployment logs.
|
||||
@@ -246,6 +248,8 @@ The history-series gate must use an active production VIN and a current local-ti
|
||||
|
||||
The history-query gate must also request both `location` and `raw` through `/api/v2/history/query`. Every returned row must contain a non-empty `qualityReason`; RAW columns must include fields discovered from the bounded fetched scope rather than only the visible page. In the browser, verify field-name/key search, multi-select, the single visual time-range control, selected-field consistency across table/trend/export, and absence of a RAW-evidence placeholder when `evidenceId` is unavailable. Keep the existing maximum of five vehicles, 200 rows per page and 1,000 fetched rows for interactive field discovery.
|
||||
|
||||
The history-export gate must verify owner-bound listing and authenticated Blob download. A customer A task must be absent from customer B's list and return 404 when B guesses its ID; an administrator may audit it. Persisted job JSON and CSV metadata must retain the owner and exact per-VIN time scope after an API restart. Disabling the customer or revoking any required vehicle grant must stop an incomplete task and deny download of a previously completed file.
|
||||
|
||||
Track smoke must also select the final playback event and verify synchronized SOC/direction/alarm availability plus a resolved current address. For high-frequency sources, confirm positive subsecond samples do not become alternating zero-second `数据间隔` segments; only non-increasing timestamps or intervals over ten minutes are gaps. Access smoke should exercise `model`/`provider` and one paired first/latest receive-time range, then confirm the filter survives a shareable URL and every returned row remains in scope.
|
||||
|
||||
Global-monitor smoke must verify `zoom=5` returns clusters, a city `bounds` at `zoom=13` returns at most 2,000 lightweight points, and invalid/reversed bounds return HTTP 400. In the browser, the rail renders at most 200 vehicle rows while the legend reports the independent map mode. Selecting `driving` or `idle` must constrain both the server-side count and every loaded row. A unique keyword result must discard the previous viewport and focus at point zoom. The synthetic release gate is `go test ./internal/platform -run TestMonitorMapTenThousandVehicles -count=1` plus `go test ./internal/platform -run '^$' -bench BenchmarkMonitorMapTenThousandVehicles -benchtime=30x -benchmem`.
|
||||
|
||||
@@ -299,6 +299,14 @@ Production reconciliation proved that the three visible vehicle counts represent
|
||||
|
||||
High-volume history rows, aggregated series, track playback and daily-mileage matrices consume the request `AbortSignal` and use zero inactive-cache retention. Small vehicle-option and summary results retain only a bounded 30–60 second cache. This preserves responsive repeated lookups without retaining multiple large arbitrary date-window payloads after their observer is gone.
|
||||
|
||||
## 2026-07-16: authenticated owner-bound history exports
|
||||
|
||||
Release `export-owner-scope-20260716165737` removes the last direct download link from the history page. Completed files are now fetched through the API client with the current session's Bearer token, converted to a short-lived object URL and downloaded with the server-provided business filename. The server encodes the Chinese filename in `X-Export-Name` and RFC 5987 `Content-Disposition`, so HTTP headers remain valid across Go, proxies and browsers.
|
||||
|
||||
The export workspace is owner-aware: each row shows the creating account, authorized vehicle count and query window. A customer sees only their own tasks; an administrator can audit all tasks. Frontend tests cover customer visibility and authenticated download headers, while backend tests cover customer A/B isolation, guessed-ID denial, live grant revocation and persisted owner/Scope metadata.
|
||||
|
||||
The production gate exported 1,274 location rows for one real VIN. The 215,212-byte CSV contained export audit, vehicle Scope and quality-reason metadata. A guessed ID returned 404, the API restarted successfully, and the file SHA-256 remained `00a4b1a877364f048a06e8bf6d80828e0dacaddd2f946b39b6cd00cfb2e9013e` with no partial file left behind. The release passed all 241 production frontend tests and the TypeScript/Vite build gate.
|
||||
|
||||
## Release evidence template
|
||||
|
||||
- Commit and release identifier
|
||||
|
||||
@@ -190,7 +190,18 @@
|
||||
|
||||
### P0-05 导出任务账号归属与审计
|
||||
|
||||
状态:`待开发`
|
||||
状态:`已完成`
|
||||
|
||||
已上线结果(release `export-owner-scope-20260716165737`):
|
||||
|
||||
- 创建、列表和下载接口均使用当前认证主体;任务固化创建账号、显示名、角色/用户类型、认证来源、客户/租户、已授权 VIN、逐车时间范围和创建/完成时间。
|
||||
- 客户和内部普通账号只能看到、下载自己的任务;管理员保留全局审计视角。旧版无 owner 的历史任务不会暴露给非管理员,猜测或复用其他任务 ID 统一按不存在处理。
|
||||
- 客户创建任务时同步解析车牌/VIN 并校验车辆与历史时间授权;异步任务开始、每 5,000 行批次及下载前再次查询账号启用状态和车辆授权。账号停用、车辆撤权或授权区间变化后,未完成任务停止,已完成文件拒绝下载。
|
||||
- 导出任务持久化保存 owner 与车辆时间 Scope;服务重启后继续按同一主体授权下载。排队或执行中的任务在重启后安全标记失败,不会以无主体状态恢复。
|
||||
- CSV 增加导出账号、角色、客户、创建/生成时间、车辆 Scope、查询条件、字段/单位及质量原因;任务名和下载文件名包含业务时间及创建者,中文文件名通过 RFC 5987/URL 编码安全传递。
|
||||
- Web 下载改为带 Bearer Token 的受控 Blob 请求,不再使用绕过请求头的直接 `<a>` 链接;客户历史页可创建自己的导出,列表展示创建者、车辆数和时间范围。
|
||||
- 自动化覆盖客户 A/B 任务隔离、管理员审计、越权下载、撤权停止、持久化审计元数据和前端 Bearer 下载;本地后端全量测试及前端 241 项测试通过。
|
||||
- ECS 真实车辆 `LNXNEGRR7SR318212` 在 `2026-07-14 00:00–06:00` 导出 1,274 行,`rowCount = processedRows = totalRows`;下载文件 215,212 字节,包含审计/Scope/质量元数据。伪造任务 ID 返回 404;API 重启前后 SHA-256 均为 `00a4b1a877364f048a06e8bf6d80828e0dacaddd2f946b39b6cd00cfb2e9013e`,且无 `.part` 残留。
|
||||
|
||||
目标:
|
||||
|
||||
|
||||
Reference in New Issue
Block a user