diff --git a/docs/superpowers/specs/2026-07-08-multi-source-mileage-stats-design.md b/docs/superpowers/specs/2026-07-08-multi-source-mileage-stats-design.md new file mode 100644 index 00000000..41417bed --- /dev/null +++ b/docs/superpowers/specs/2026-07-08-multi-source-mileage-stats-design.md @@ -0,0 +1,161 @@ +# Multi-Source Mileage Statistics Design + +## Goal + +Vehicle mileage statistics must keep every protocol source that reports data, while still exposing one reliable default result for BI and API consumers. The design separates source facts from elected business results so multi-source JT808, GB32960, and Yutong MQTT data can be audited, corrected, and re-elected without losing raw source evidence. + +## Current Problem + +The current `vehicle_daily_mileage` table stores one row per `vin + stat_date + protocol`. That prevents duplicate counting, but it also forces source election too early. When multiple source platforms report the same VIN, mixing them can create false mileage jumps; selecting only one source can undercount when a reliable alternative exists. + +The observed example was `LA9GG64L7PBAF4001` on `2026-07-08` for JT808. Two source IPs reported different total mileage ranges. The correct model is to retain each source as its own candidate and then elect the recommended candidate for default queries. + +## Tables + +### `vehicle_data_source` + +Stores source platform metadata discovered from incoming data and later maintained by operators. + +Identity: + +- `protocol` +- `source_ip` + +Fields: + +- `id` +- `protocol` +- `source_ip` +- `latest_source_endpoint` +- `platform_name` +- `trust_priority` +- `enabled` +- `first_seen_at` +- `latest_seen_at` +- `remark` +- `created_at` +- `updated_at` + +Rules: + +- `protocol + source_ip` is unique. +- Ports are not part of source identity because sender ports change often. +- Runtime may update `latest_source_endpoint`, `first_seen_at`, and `latest_seen_at`. +- Runtime must not overwrite manually maintained `platform_name`, `trust_priority`, `enabled`, or `remark`. + +### `vehicle_daily_mileage_source` + +Stores one daily candidate per source. + +Identity: + +- `vin` +- `stat_date` +- `protocol` +- `source_key` + +Fields: + +- `vin` +- `stat_date` +- `protocol` +- `source_key` +- `source_ip` +- `source_endpoint` +- `phone` +- `device_id` +- `platform_name` +- `first_total_mileage_km` +- `latest_total_mileage_km` +- `daily_mileage_km` +- `sample_count` +- `first_event_time` +- `latest_event_time` +- `quality_status` +- `quality_reason` +- `is_selected` +- `created_at` +- `updated_at` + +Rules: + +- Candidate rows are facts by source. They should not be deleted just because they are not selected. +- `source_key` is normalized as protocol-specific device identity plus source IP. For JT808 that is usually `phone@source_ip`. +- A candidate can be marked invalid without being removed. +- Candidate statistics never mix different `source_key` values. + +### `vehicle_daily_mileage` + +Keeps the existing default query surface for BI and APIs. + +Rules: + +- One row per `vin + stat_date + protocol`. +- The row is a projection from one selected `vehicle_daily_mileage_source` row. +- It stores selected source metadata for fast default queries. + +## Election + +The default selected source is chosen from candidate rows. Selection priority: + +1. Source must be enabled in `vehicle_data_source`. +2. Prefer lower `trust_priority` when configured. +3. Prefer candidates with a valid same-source previous-day baseline. +4. Reject impossible deltas, such as negative mileage or mileage above the configured daily maximum. +5. Prefer more samples when priority and quality are tied. +6. Prefer later `latest_event_time` when still tied. + +If trusted source A is missing on a day, source B can be selected only if B has its own previous-day baseline. The system must not calculate `B today - A yesterday`. + +## Data Flow + +1. Gateway/history/realtime parsing continues to emit protocol fields without non-protocol synthetic fields. +2. Stats writer extracts protocol-specific total mileage from the existing fields event. +3. Stats writer upserts `vehicle_data_source` by `protocol + source_ip`. +4. Stats writer upserts `vehicle_daily_mileage_source` by `vin + stat_date + protocol + source_key`. +5. Election updates `vehicle_daily_mileage` from candidates. +6. History backfill can rebuild candidates from TDengine raw frames and then rebuild final projections. + +## API Behavior + +Default daily mileage API continues to read `vehicle_daily_mileage`. + +Future query extension: + +- `includeCandidates=true` returns all candidates for each selected row. +- Candidate rows include `platform_name`, `source_ip`, `source_endpoint`, `quality_status`, and `quality_reason`. + +## Edge Cases + +- Multiple ports from the same platform source are collapsed by source IP. +- A new source with no previous-day baseline is stored as a candidate with `NO_PREVIOUS_BASELINE` and is not selected by default. +- A source that reports a total-mileage reset is stored as a candidate with an invalid quality status until a clear policy is applied. +- If all candidates are invalid, no final row is projected for that VIN/day/protocol. +- Manual platform names and priority changes should allow re-election without reparsing raw data. + +## Performance + +- Real-time writes are small upserts keyed by indexed dimensions. +- Default BI/API reads stay fast because they read the final projection table. +- Candidate/audit queries are opt-in. +- Backfill aggregates by source and date instead of writing every raw frame to MySQL. + +## Testing + +Automated tests should cover: + +- Source IP extraction and endpoint normalization. +- Source table upsert preserving manually maintained fields. +- Candidate stats never mixing different source keys. +- Election choosing the continuous source for `LA9GG64L7PBAF4001`-style data. +- Election skipping `B today - A yesterday`. +- Backfill rebuilding candidates and final projection. + +## Rollout + +1. Add schemas and repository helpers. +2. Write candidate rows in stats writer and backfill. +3. Add election projection into `vehicle_daily_mileage`. +4. Rebuild `2026-07-08` JT808 statistics and verify `LA9GG64L7PBAF4001`. +5. Deploy to ECS and monitor stat-writer lag and write errors. +