lingniu-vehicle-ingest/docs/superpowers/specs/2026-07-08-multi-source-mileage-stats-design.md

Multi-Source Mileage Statistics Design

Goal

Vehicle mileage statistics must keep every protocol source that reports data, while still exposing one reliable default result for BI and API consumers. The design separates source facts from elected business results so multi-source JT808, GB32960, and Yutong MQTT data can be audited, corrected, and re-elected without losing raw source evidence.

Current Problem

The current vehicle_daily_mileage table stores one row per vin + stat_date + protocol. That prevents duplicate counting, but it also forces source election too early. When multiple source platforms report the same VIN, mixing them can create false mileage jumps; selecting only one source can undercount when a reliable alternative exists.

The observed example was LA9GG64L7PBAF4001 on 2026-07-08 for JT808. Two source IPs reported different total mileage ranges. The correct model is to retain each source as its own candidate and then elect the recommended candidate for default queries.

Tables

vehicle_data_source

Stores source platform metadata discovered from incoming data and later maintained by operators.

Identity:

• protocol
• source_ip

Fields:

• id
• protocol
• source_ip
• latest_source_endpoint
• platform_name
• trust_priority
• enabled
• first_seen_at
• latest_seen_at
• remark
• created_at
• updated_at

Rules:

• protocol + source_ip is unique.
• Ports are not part of source identity because sender ports change often.
• Runtime may update latest_source_endpoint, first_seen_at, and latest_seen_at.
• Runtime must not overwrite manually maintained platform_name, trust_priority, enabled, or remark.

vehicle_daily_mileage_source

Stores one daily candidate per source.

Identity:

• vin
• stat_date
• protocol
• source_key

Fields:

• vin
• stat_date
• protocol
• source_key
• source_ip
• source_endpoint
• phone
• device_id
• platform_name
• first_total_mileage_km
• latest_total_mileage_km
• daily_mileage_km
• sample_count
• first_event_time
• latest_event_time
• quality_status
• quality_reason
• is_selected
• created_at
• updated_at

Rules:

• Candidate rows are facts by source. They should not be deleted just because they are not selected.
• source_key is normalized as protocol-specific device identity plus source IP. For JT808 that is usually phone@source_ip.
• A candidate can be marked invalid without being removed.
• Candidate statistics never mix different source_key values.

vehicle_daily_mileage

Keeps the existing default query surface for BI and APIs.

Rules:

• One row per vin + stat_date + protocol.
• The row is a projection from one selected vehicle_daily_mileage_source row.
• It stores selected source metadata for fast default queries.

Election

The default selected source is chosen from candidate rows. Selection priority:

1. Source must be enabled in vehicle_data_source.
2. Prefer lower trust_priority when configured.
3. Prefer candidates with a valid same-source previous-day baseline.
4. Reject impossible deltas, such as negative mileage or mileage above the configured daily maximum.
5. Prefer more samples when priority and quality are tied.
6. Prefer later latest_event_time when still tied.

If trusted source A is missing on a day, source B can be selected only if B has its own previous-day baseline. The system must not calculate B today - A yesterday.

Data Flow

1. Gateway/history/realtime parsing continues to emit protocol fields without non-protocol synthetic fields.
2. Stats writer extracts protocol-specific total mileage from the existing fields event.
3. Stats writer upserts vehicle_data_source by protocol + source_ip.
4. Stats writer upserts vehicle_daily_mileage_source by vin + stat_date + protocol + source_key.
5. Election updates vehicle_daily_mileage from candidates.
6. History backfill can rebuild candidates from TDengine raw frames and then rebuild final projections.

API Behavior

Default daily mileage API continues to read vehicle_daily_mileage.

Future query extension:

• includeCandidates=true returns all candidates for each selected row.
• Candidate rows include platform_name, source_ip, source_endpoint, quality_status, and quality_reason.

Edge Cases

• Multiple ports from the same platform source are collapsed by source IP.
• A new source with no previous-day baseline is stored as a candidate with NO_PREVIOUS_BASELINE and is not selected by default.
• A source that reports a total-mileage reset is stored as a candidate with an invalid quality status until a clear policy is applied.
• If all candidates are invalid, no final row is projected for that VIN/day/protocol.
• Manual platform names and priority changes should allow re-election without reparsing raw data.

Performance

• Real-time writes are small upserts keyed by indexed dimensions.
• Default BI/API reads stay fast because they read the final projection table.
• Candidate/audit queries are opt-in.
• Backfill aggregates by source and date instead of writing every raw frame to MySQL.

Testing

Automated tests should cover:

• Source IP extraction and endpoint normalization.
• Source table upsert preserving manually maintained fields.
• Candidate stats never mixing different source keys.
• Election choosing the continuous source for LA9GG64L7PBAF4001-style data.
• Election skipping B today - A yesterday.
• Backfill rebuilding candidates and final projection.

Rollout

1. Add schemas and repository helpers.
2. Write candidate rows in stats writer and backfill.
3. Add election projection into vehicle_daily_mileage.
4. Rebuild 2026-07-08 JT808 statistics and verify LA9GG64L7PBAF4001.
5. Deploy to ECS and monitor stat-writer lag and write errors.