refactor: store jt808 mileage as metric diff

This commit is contained in:
lingniu
2026-07-01 02:32:46 +08:00
parent a5f5cc94ee
commit b94780b48c
12 changed files with 73 additions and 40 deletions

View File

@@ -610,10 +610,10 @@ flowchart LR
</tr> </tr>
<tr> <tr>
<td>modules/services/vehicle-stat-service</td> <td>modules/services/vehicle-stat-service</td>
<td>消费 Kafka 全字段事件,抽取统计源点,按车辆规则计算每日里程等日统计;生产消费可使用 <code>tryIngest</code> 和自动装配的 <code>EnvelopeConsumerProcessor</code> 隔离坏 protobuf缺少 telemetry_snapshot 的消息会标记为 SKIPPED 并进入死信出口。</td> <td>消费 Kafka 全字段事件,808 每日里程只使用 0x0200 附加项 0x01 上报的总里程做当日首末差值;生产消费可使用 <code>tryIngest</code> 和自动装配的 <code>EnvelopeConsumerProcessor</code> 隔离坏 protobuf缺少 telemetry_snapshot 的消息会标记为 SKIPPED 并进入死信出口。</td>
<td>Kafka Protobuf Envelope、车辆统计规则</td> <td>Kafka Protobuf Envelope、808 <code>total_mileage_km</code></td>
<td><code>vin + stat_date</code> 保存的日统计结果</td> <td><code>vehicle_stat_metric</code><code>metric_key=daily_mileage_km</code><code>calculation_method=JT808_TOTAL_MILEAGE_DIFF</code></td>
<td><span class="tag green">每日里程</span><span class="tag">规则配置</span></td> <td><span class="tag green">每日里程</span><span class="tag">指标表</span></td>
</tr> </tr>
<tr> <tr>
<td>modules/apps/command-gateway</td> <td>modules/apps/command-gateway</td>

View File

@@ -12,6 +12,7 @@ The JT808 daily-mileage value is calculated from the GPS total mileage reported
```text ```text
daily_mileage_km = last_total_mileage_km - first_total_mileage_km daily_mileage_km = last_total_mileage_km - first_total_mileage_km
calculation_method = JT808_TOTAL_MILEAGE_DIFF
``` ```
The metric is saved only after at least two ordered JT808 location points with valid `total_mileage_km` have been received for the same vehicle and local day. If the difference is negative, no metric row is saved for that state update. The metric is saved only after at least two ordered JT808 location points with valid `total_mileage_km` have been received for the same vehicle and local day. If the difference is negative, no metric row is saved for that state update.

View File

@@ -8,11 +8,11 @@ Consume JT808 Kafka location events, keep rolling daily state, and write the der
- Source topic: `vehicle.event.jt808.v1` - Source topic: `vehicle.event.jt808.v1`
- Runtime app: `vehicle-analytics-app` - Runtime app: `vehicle-analytics-app`
- State: Redis or in-memory `Jt808MileageStateStore` - Runtime state: Redis `Jt808MileageStateStore`
- Metric output: `VehicleStatRepository.saveDailyStat(...)` - Metric output: `VehicleStatRepository.saveDailyStat(...)`
- Production metric storage: JDBC/MySQL `vehicle_stat_metric` - Production metric storage: JDBC/MySQL `vehicle_stat_metric`
- Local fallback storage: `VEHICLE_STAT_FILE_PATH/daily-stats.tsv`
- Date boundary: `Asia/Shanghai` - Date boundary: `Asia/Shanghai`
- Calculation method: `JT808_TOTAL_MILEAGE_DIFF`
JT808 daily mileage is calculated only from the GPS total mileage reported by JT808 location additional information: JT808 daily mileage is calculated only from the GPS total mileage reported by JT808 location additional information:

View File

@@ -269,28 +269,20 @@ MYSQL_USERNAME
MYSQL_PASSWORD MYSQL_PASSWORD
``` ```
`vehicle_daily_stat` Daily derived metrics are stored in the common metric table, not in a
protocol-specific daily-mileage table.
```sql ```sql
CREATE TABLE IF NOT EXISTS vehicle_daily_stat ( CREATE TABLE IF NOT EXISTS vehicle_stat_metric (
vin VARCHAR(32) NOT NULL, vin VARCHAR(64) NOT NULL,
stat_date DATE NOT NULL, stat_date DATE NOT NULL,
platform_account VARCHAR(64), metric_key VARCHAR(64) NOT NULL,
total_mileage_km DECIMAL(14,3), metric_value DECIMAL(18,6) NULL,
daily_mileage_km DECIMAL(14,3), metric_unit VARCHAR(16) NOT NULL,
daily_power_kwh DECIMAL(14,3), calculation_method VARCHAR(64) NOT NULL,
daily_hydrogen_kg DECIMAL(14,4), created_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
daily_hydrogen_kg_per_100km DECIMAL(14,4), updated_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
first_event_time DATETIME(3), PRIMARY KEY (vin, stat_date, metric_key)
last_event_time DATETIME(3),
first_total_mileage_km DECIMAL(14,3),
last_total_mileage_km DECIMAL(14,3),
last_soc_percent DECIMAL(8,3),
sample_count BIGINT NOT NULL DEFAULT 0,
late_sample_count BIGINT NOT NULL DEFAULT 0,
calculation_quality VARCHAR(32) NOT NULL,
updated_at DATETIME(3) NOT NULL,
PRIMARY KEY (vin, stat_date)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4; ) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4;
``` ```

View File

@@ -159,15 +159,16 @@ Responsibilities:
- Consume normalized telemetry events from Kafka. - Consume normalized telemetry events from Kafka.
- Maintain day-level vehicle point state needed for statistics. - Maintain day-level vehicle point state needed for statistics.
- Run scheduled and event-triggered calculations. - Run scheduled and event-triggered calculations.
- Store daily results in a statistics database. - Store derived results in the common metric table.
- Support per-vehicle calculation rules.
Initial daily mileage strategies: Current JT808 daily mileage strategy:
| Strategy | Formula | | Strategy | Formula |
|---|---| |---|---|
| `CURRENT_LAST_MINUS_PREVIOUS_LAST` | Current day last valid total mileage minus previous day last valid total mileage | | `JT808_TOTAL_MILEAGE_DIFF` | Current local day last valid JT808 GPS total mileage minus first valid JT808 GPS total mileage |
| `DAY_MAX_MINUS_DAY_MIN` | Current day max valid total mileage minus current day min valid total mileage |
The derived value is stored as `metric_key = daily_mileage_km` in
`vehicle_stat_metric`; there is no protocol-specific JT808 daily-mileage table.
Hydrogen and electricity statistics use internal fields from the same event Hydrogen and electricity statistics use internal fields from the same event
contract. Refueling and charging resets must be handled in statistics logic, not contract. Refueling and charging resets must be handled in statistics logic, not

View File

@@ -1,6 +1,16 @@
package com.lingniu.ingest.vehiclestat; package com.lingniu.ingest.vehiclestat;
public enum DailyMileageStrategy { public enum DailyMileageStrategy {
CURRENT_LAST_MINUS_PREVIOUS_LAST, JT808_TOTAL_MILEAGE_DIFF;
DAY_MAX_MINUS_DAY_MIN
public static DailyMileageStrategy fromStorage(String value) {
if (value == null || value.isBlank()) {
return JT808_TOTAL_MILEAGE_DIFF;
}
String normalized = value.trim();
if ("CURRENT_LAST_MINUS_PREVIOUS_LAST".equals(normalized)) {
return JT808_TOTAL_MILEAGE_DIFF;
}
return DailyMileageStrategy.valueOf(normalized);
}
} }

View File

@@ -57,7 +57,7 @@ public final class JdbcVehicleStatMetricRepository implements VehicleStatReposit
rs.getBigDecimal("metric_value") == null rs.getBigDecimal("metric_value") == null
? OptionalDouble.empty() ? OptionalDouble.empty()
: OptionalDouble.of(rs.getBigDecimal("metric_value").doubleValue()), : OptionalDouble.of(rs.getBigDecimal("metric_value").doubleValue()),
DailyMileageStrategy.valueOf(rs.getString("calculation_method"))), DailyMileageStrategy.fromStorage(rs.getString("calculation_method"))),
normalizedVin, Date.valueOf(statDate), DAILY_MILEAGE_KEY); normalizedVin, Date.valueOf(statDate), DAILY_MILEAGE_KEY);
return rows.stream().findFirst(); return rows.stream().findFirst();
} }

View File

@@ -19,7 +19,7 @@ public record VehicleDailyStatResult(String vin,
dailyMileageKm = OptionalDouble.empty(); dailyMileageKm = OptionalDouble.empty();
} }
if (dailyMileageStrategy == null) { if (dailyMileageStrategy == null) {
dailyMileageStrategy = DailyMileageStrategy.CURRENT_LAST_MINUS_PREVIOUS_LAST; dailyMileageStrategy = DailyMileageStrategy.JT808_TOTAL_MILEAGE_DIFF;
} }
} }
} }

View File

@@ -90,7 +90,7 @@ public final class Jt808DailyMileageState {
statVehicleId(), statVehicleId(),
statDate, statDate,
dailyMileage, dailyMileage,
DailyMileageStrategy.CURRENT_LAST_MINUS_PREVIOUS_LAST)); DailyMileageStrategy.JT808_TOTAL_MILEAGE_DIFF));
} }
private void updateOdometer(Jt808LocationPoint point) { private void updateOdometer(Jt808LocationPoint point) {

View File

@@ -23,12 +23,12 @@ class JdbcVehicleStatMetricRepositoryTest {
"VIN001", "VIN001",
LocalDate.of(2026, 6, 30), LocalDate.of(2026, 6, 30),
OptionalDouble.of(12.345), OptionalDouble.of(12.345),
DailyMileageStrategy.CURRENT_LAST_MINUS_PREVIOUS_LAST)); DailyMileageStrategy.JT808_TOTAL_MILEAGE_DIFF));
repository.saveDailyStat(new VehicleDailyStatResult( repository.saveDailyStat(new VehicleDailyStatResult(
"VIN001", "VIN001",
LocalDate.of(2026, 6, 30), LocalDate.of(2026, 6, 30),
OptionalDouble.of(15.5), OptionalDouble.of(15.5),
DailyMileageStrategy.CURRENT_LAST_MINUS_PREVIOUS_LAST)); DailyMileageStrategy.JT808_TOTAL_MILEAGE_DIFF));
assertThat(repository.findDailyStat("VIN001", LocalDate.of(2026, 6, 30))) assertThat(repository.findDailyStat("VIN001", LocalDate.of(2026, 6, 30)))
.isPresent() .isPresent()
@@ -40,5 +40,33 @@ class JdbcVehicleStatMetricRepositoryTest {
assertThat(jdbcTemplate.queryForObject( assertThat(jdbcTemplate.queryForObject(
"SELECT metric_key FROM vehicle_stat_metric WHERE vin = 'VIN001'", "SELECT metric_key FROM vehicle_stat_metric WHERE vin = 'VIN001'",
String.class)).isEqualTo("daily_mileage_km"); String.class)).isEqualTo("daily_mileage_km");
assertThat(jdbcTemplate.queryForObject(
"SELECT calculation_method FROM vehicle_stat_metric WHERE vin = 'VIN001'",
String.class)).isEqualTo("JT808_TOTAL_MILEAGE_DIFF");
assertThat(jdbcTemplate.queryForObject("""
SELECT COUNT(*)
FROM information_schema.tables
WHERE table_name = 'vehicle_daily_stat'
""", Integer.class)).isZero();
}
@Test
void readsLegacyCalculationMethodAsTotalMileageDifference() {
JdbcTemplate jdbcTemplate = new JdbcTemplate(new DriverManagerDataSource(
"jdbc:h2:mem:vehicle_stat_metric_legacy;MODE=MySQL;DATABASE_TO_LOWER=TRUE;DB_CLOSE_DELAY=-1",
"sa",
""));
JdbcVehicleStatMetricRepository repository = new JdbcVehicleStatMetricRepository(jdbcTemplate);
jdbcTemplate.update("""
INSERT INTO vehicle_stat_metric
(vin, stat_date, metric_key, metric_value, metric_unit, calculation_method)
VALUES ('VIN001', DATE '2026-06-30', 'daily_mileage_km', 8.5, 'km', 'CURRENT_LAST_MINUS_PREVIOUS_LAST')
""");
assertThat(repository.findDailyStat("VIN001", LocalDate.of(2026, 6, 30)))
.isPresent()
.get()
.extracting(VehicleDailyStatResult::dailyMileageStrategy)
.isEqualTo(DailyMileageStrategy.JT808_TOTAL_MILEAGE_DIFF);
} }
} }

View File

@@ -21,14 +21,15 @@ class VehicleStatControllerTest {
"VIN001", "VIN001",
LocalDate.parse("2026-06-22"), LocalDate.parse("2026-06-22"),
OptionalDouble.of(35.5), OptionalDouble.of(35.5),
DailyMileageStrategy.CURRENT_LAST_MINUS_PREVIOUS_LAST)); DailyMileageStrategy.JT808_TOTAL_MILEAGE_DIFF));
MockMvc mvc = standaloneSetup(new VehicleStatController(repository)).build(); MockMvc mvc = standaloneSetup(new VehicleStatController(repository)).build();
mvc.perform(get("/api/vehicle-stat/VIN001/daily").param("date", "2026-06-22")) mvc.perform(get("/api/vehicle-stat/VIN001/daily").param("date", "2026-06-22"))
.andExpect(status().isOk()) .andExpect(status().isOk())
.andExpect(jsonPath("$.vin").value("VIN001")) .andExpect(jsonPath("$.vin").value("VIN001"))
.andExpect(jsonPath("$.statDate").value("2026-06-22")) .andExpect(jsonPath("$.statDate").value("2026-06-22"))
.andExpect(jsonPath("$.dailyMileageKm").value(35.5)); .andExpect(jsonPath("$.dailyMileageKm").value(35.5))
.andExpect(jsonPath("$.dailyMileageStrategy").value("JT808_TOTAL_MILEAGE_DIFF"));
} }
private static final class InMemoryVehicleStatRepository implements VehicleStatRepository { private static final class InMemoryVehicleStatRepository implements VehicleStatRepository {

View File

@@ -36,7 +36,7 @@ class Jt808MileageStreamProcessorTest {
assertThat(latest.vin()).isEqualTo("VIN001"); assertThat(latest.vin()).isEqualTo("VIN001");
assertThat(latest.statDate()).isEqualTo(LocalDate.of(2026, 6, 30)); assertThat(latest.statDate()).isEqualTo(LocalDate.of(2026, 6, 30));
assertThat(latest.dailyMileageKm()).hasValue(1.0); assertThat(latest.dailyMileageKm()).hasValue(1.0);
assertThat(latest.dailyMileageStrategy()).isEqualTo(DailyMileageStrategy.CURRENT_LAST_MINUS_PREVIOUS_LAST); assertThat(latest.dailyMileageStrategy()).isEqualTo(DailyMileageStrategy.JT808_TOTAL_MILEAGE_DIFF);
} }
@Test @Test