# Go Vehicle Ingest Redesign Phase 1 Implementation Plan > **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. **Goal:** Build the first production-capable Go runtime for GB32960, JT808, and Yutong MQTT ingestion with unified Kafka, TDengine, MySQL statistics, and Redis realtime state boundaries. **Architecture:** Create a new single Go module at `go/vehicle-gateway` and migrate only the useful pieces from the existing `go/ingest-edge` and `go/vehicle-state` prototypes. The gateway produces unified envelopes to Kafka; independent consumers write TDengine history, MySQL daily metrics, and Redis realtime state. **Tech Stack:** Go 1.26+, Kafka (`segmentio/kafka-go`), Redis (`redis/go-redis`), MySQL (`go-sql-driver/mysql`), TDengine official Go connector, MQTT (`eclipse/paho.mqtt.golang`), standard library TCP. --- ## File Structure - Create: `go/vehicle-gateway/go.mod` - Create: `go/vehicle-gateway/cmd/gateway/main.go` - Create: `go/vehicle-gateway/cmd/history-writer/main.go` - Create: `go/vehicle-gateway/cmd/stat-writer/main.go` - Create: `go/vehicle-gateway/cmd/realtime-api/main.go` - Create: `go/vehicle-gateway/internal/envelope/envelope.go` - Create: `go/vehicle-gateway/internal/envelope/envelope_test.go` - Create: `go/vehicle-gateway/internal/protocol/jt808/*` - Create: `go/vehicle-gateway/internal/protocol/gb32960/*` - Create: `go/vehicle-gateway/internal/protocol/yutongmqtt/*` - Create: `go/vehicle-gateway/internal/gateway/*` - Create: `go/vehicle-gateway/internal/identity/*` - Create: `go/vehicle-gateway/internal/eventbus/*` - Create: `go/vehicle-gateway/internal/history/*` - Create: `go/vehicle-gateway/internal/stats/*` - Create: `go/vehicle-gateway/internal/realtime/*` - Create: `go/vehicle-gateway/internal/observability/*` - Modify: `README.md` - Modify: `docs/target-architecture.md` - Create: `deploy/portainer/docker-compose-go.yml` The existing `go/ingest-edge` and `go/vehicle-state` directories are migration sources only. After phase 1 is verified, remove or mark them superseded. --- ### Task 1: Create Single Go Module **Files:** - Create: `go/vehicle-gateway/go.mod` - Create: `go/vehicle-gateway/internal/observability/logger.go` - Create: `go/vehicle-gateway/cmd/gateway/main.go` - [ ] **Step 1: Create module manifest** Add `go/vehicle-gateway/go.mod`: ```go module lingniu-vehicle-ingest/go/vehicle-gateway go 1.26 require ( github.com/eclipse/paho.mqtt.golang v1.5.1 github.com/go-sql-driver/mysql v1.9.3 github.com/redis/go-redis/v9 v9.17.2 github.com/segmentio/kafka-go v0.4.49 github.com/taosdata/driver-go/v3 v3.8.1 ) ``` - [ ] **Step 2: Add logger helper** Add `go/vehicle-gateway/internal/observability/logger.go`: ```go package observability import ( "log/slog" "os" ) func NewLogger(service string) *slog.Logger { handler := slog.NewJSONHandler(os.Stdout, &slog.HandlerOptions{AddSource: true}) return slog.New(handler).With("service", service) } ``` - [ ] **Step 3: Add temporary gateway entrypoint** Add `go/vehicle-gateway/cmd/gateway/main.go`: ```go package main import "lingniu-vehicle-ingest/go/vehicle-gateway/internal/observability" func main() { logger := observability.NewLogger("vehicle-gateway") logger.Info("vehicle gateway scaffold started") } ``` - [ ] **Step 4: Verify scaffold builds** Run: ```bash cd go/vehicle-gateway go mod tidy go test ./... go build ./cmd/gateway ``` Expected: all commands exit `0`. --- ### Task 2: Define Unified Envelope **Files:** - Create: `go/vehicle-gateway/internal/envelope/envelope.go` - Create: `go/vehicle-gateway/internal/envelope/envelope_test.go` - [ ] **Step 1: Write envelope tests** Add `go/vehicle-gateway/internal/envelope/envelope_test.go`: ```go package envelope import "testing" func TestFrameEnvelopeVehicleKeyPrefersVIN(t *testing.T) { e := FrameEnvelope{Protocol: ProtocolJT808, VIN: "LNBVIN00000000001", Phone: "013307795425"} if got := e.VehicleKey(); got != "LNBVIN00000000001" { t.Fatalf("VehicleKey() = %q", got) } } func TestFrameEnvelopeVehicleKeyFallsBackToPhone(t *testing.T) { e := FrameEnvelope{Protocol: ProtocolJT808, Phone: "013307795425"} if got := e.VehicleKey(); got != "JT808:013307795425" { t.Fatalf("VehicleKey() = %q", got) } } func TestFrameEnvelopeEventIDStable(t *testing.T) { e := FrameEnvelope{ Protocol: ProtocolJT808, MessageID: "0x0200", Phone: "013307795425", Sequence: 1, EventTimeMS: 1782745114000, ReceivedAtMS: 1782745114999, RawHex: "7e02000000ff7e", } a := e.StableEventID() b := e.StableEventID() if a == "" || a != b { t.Fatalf("event id must be non-empty and stable: %q %q", a, b) } } ``` - [ ] **Step 2: Run tests and confirm failure** Run: ```bash cd go/vehicle-gateway go test ./internal/envelope ``` Expected: fail because `FrameEnvelope` is not defined. - [ ] **Step 3: Implement envelope** Add `go/vehicle-gateway/internal/envelope/envelope.go`: ```go package envelope import ( "crypto/sha256" "encoding/hex" "encoding/json" "fmt" "strings" ) type Protocol string const ( ProtocolGB32960 Protocol = "GB32960" ProtocolJT808 Protocol = "JT808" ProtocolYutongMQTT Protocol = "YUTONG_MQTT" ) type ParseStatus string const ( ParseOK ParseStatus = "OK" ParsePartial ParseStatus = "PARTIAL" ParseBadFrame ParseStatus = "BAD_FRAME" ) type FrameEnvelope struct { EventID string `json:"event_id"` TraceID string `json:"trace_id"` Protocol Protocol `json:"protocol"` MessageID string `json:"message_id"` Sequence uint16 `json:"sequence"` VIN string `json:"vin,omitempty"` VehicleKeyHint string `json:"vehicle_key,omitempty"` Phone string `json:"phone,omitempty"` DeviceID string `json:"device_id,omitempty"` Plate string `json:"plate,omitempty"` SourceEndpoint string `json:"source_endpoint,omitempty"` EventTimeMS int64 `json:"event_time_ms"` ReceivedAtMS int64 `json:"received_at_ms"` RawHex string `json:"raw_hex,omitempty"` RawText string `json:"raw_text,omitempty"` Parsed map[string]any `json:"parsed,omitempty"` Fields map[string]any `json:"fields,omitempty"` ParseStatus ParseStatus `json:"parse_status"` ParseError string `json:"parse_error,omitempty"` } func (e FrameEnvelope) VehicleKey() string { if key := strings.TrimSpace(e.VIN); key != "" { return key } if key := strings.TrimSpace(e.VehicleKeyHint); key != "" { return key } if key := strings.TrimSpace(e.Phone); key != "" { return string(e.Protocol) + ":" + key } if key := strings.TrimSpace(e.DeviceID); key != "" { return string(e.Protocol) + ":" + key } return string(e.Protocol) + ":unknown" } func (e FrameEnvelope) StableEventID() string { if strings.TrimSpace(e.EventID) != "" { return e.EventID } input := fmt.Sprintf("%s|%s|%s|%d|%d|%s", e.Protocol, e.MessageID, e.VehicleKey(), e.Sequence, e.EventTimeMS, e.RawHex) sum := sha256.Sum256([]byte(input)) return hex.EncodeToString(sum[:16]) } func (e FrameEnvelope) MarshalJSONBytes() ([]byte, error) { if e.EventID == "" { e.EventID = e.StableEventID() } if e.ParseStatus == "" { e.ParseStatus = ParseOK } return json.Marshal(e) } ``` - [ ] **Step 4: Verify tests pass** Run: ```bash cd go/vehicle-gateway go test ./internal/envelope ``` Expected: pass. --- ### Task 3: Implement JT808 Frame and 0200 Parser **Files:** - Create: `go/vehicle-gateway/internal/protocol/jt808/parser.go` - Create: `go/vehicle-gateway/internal/protocol/jt808/parser_test.go` - [ ] **Step 1: Add sample frame tests** Use the production sample provided in the thread: ```text 7E020000320133077954250001000000000048000301D2C4C707376139000A00E6004F26063016235701040001900C2504000000000202000030011F31010F867E ``` Expected assertions: - message id is `0x0200` - phone keeps normalized BCD string `013307795425` - sequence is `1` - `fields.total_mileage_km` is parsed from additional item `0x01` - latitude, longitude, speed, direction, alarm, status, and device time are present - [ ] **Step 2: Implement parser** Implementation rules: - strip `0x7e` start/end delimiters - unescape `0x7d 0x02 -> 0x7e` - unescape `0x7d 0x01 -> 0x7d` - verify XOR checksum - parse 2011/2013 common header - parse BCD phone from 6 bytes and keep both raw and normalized values in `parsed.header` - parse 0200 fixed body - parse additional item list as `id,length,value_hex` - parse additional `0x01` as `total_mileage_km = uint32 / 10` - [ ] **Step 3: Verify** Run: ```bash cd go/vehicle-gateway go test ./internal/protocol/jt808 ``` Expected: pass. --- ### Task 4: Implement GB32960 Frame and Data Unit Parser **Files:** - Create: `go/vehicle-gateway/internal/protocol/gb32960/parser.go` - Create: `go/vehicle-gateway/internal/protocol/gb32960/parser_test.go` - [ ] **Step 1: Add parser tests** Tests must cover: - `##` frame boundary - command id - response flag - 17-byte VIN - encryption flag - payload length - BCC verification - realtime data command `0x02` - reissue data command `0x03` - data unit `0x01` vehicle status fields - data unit `0x05` position fields - [ ] **Step 2: Implement parser** Implementation rules: - parse header without allocating large temporary buffers - keep RAW as hex in envelope - write all recognized data units to `Parsed` - write core fields to `Fields` - unsupported data unit stays in `Parsed["unknown_units"]` - [ ] **Step 3: Verify** Run: ```bash cd go/vehicle-gateway go test ./internal/protocol/gb32960 ``` Expected: pass. --- ### Task 5: Implement Kafka Sink **Files:** - Create: `go/vehicle-gateway/internal/eventbus/kafka_sink.go` - Create: `go/vehicle-gateway/internal/eventbus/kafka_sink_test.go` - [ ] **Step 1: Add sink interface** Create a sink interface: ```go package eventbus import ( "context" "lingniu-vehicle-ingest/go/vehicle-gateway/internal/envelope" ) type Sink interface { PublishRaw(context.Context, envelope.FrameEnvelope) error PublishUnified(context.Context, envelope.FrameEnvelope) error Close() error } ``` - [ ] **Step 2: Implement Kafka topic routing** Rules: - `GB32960 -> vehicle.raw.gb32960.v1` - `JT808 -> vehicle.raw.jt808.v1` - `YUTONG_MQTT -> vehicle.raw.yutong-mqtt.v1` - unified topic is `vehicle.event.unified.v1` - message key is `env.VehicleKey()` - [ ] **Step 3: Verify routing with unit tests** Run: ```bash cd go/vehicle-gateway go test ./internal/eventbus ``` Expected: pass. --- ### Task 6: Implement TDengine History Writer **Files:** - Create: `go/vehicle-gateway/internal/history/schema.go` - Create: `go/vehicle-gateway/internal/history/writer.go` - Create: `go/vehicle-gateway/internal/history/writer_test.go` - Create: `go/vehicle-gateway/cmd/history-writer/main.go` - [ ] **Step 1: Add schema bootstrap SQL** Implement schema strings for: - database `lingniu_vehicle_ts` - stable `raw_frames` - stable `vehicle_locations` - stable `vehicle_mileage_points` - [ ] **Step 2: Implement writer** Rules: - `AppendRawFrame` always writes one raw row. - `AppendLocation` writes only when longitude and latitude exist. - `AppendMileagePoint` writes only when `total_mileage_km` exists. - child table name is deterministic hash of `protocol + vehicle_key`. - escape tag values. - [ ] **Step 3: Use TDengine official driver** Import WebSocket driver in command: ```go import _ "github.com/taosdata/driver-go/v3/taosWS" ``` Default driver name: ```text TDENGINE_DRIVER=taosWS ``` Short-term compatibility: ```text TDENGINE_DRIVER=taosSql ``` Only use `taosSql` when ECS TDengine WebSocket is not available. - [ ] **Step 4: Verify** Run: ```bash cd go/vehicle-gateway go test ./internal/history go build ./cmd/history-writer ``` Expected: pass. --- ### Task 7: Implement MySQL Daily Metric Writer **Files:** - Create: `go/vehicle-gateway/internal/stats/schema.go` - Create: `go/vehicle-gateway/internal/stats/daily_metric.go` - Create: `go/vehicle-gateway/internal/stats/daily_metric_test.go` - Create: `go/vehicle-gateway/cmd/stat-writer/main.go` - [ ] **Step 1: Add schema bootstrap** Implement `vehicle_daily_metric` schema from the design spec. - [ ] **Step 2: Add metric derivation tests** Test cases: - no `total_mileage_km` produces no metric - one sample produces: - `daily_mileage_km = 0` - `daily_total_mileage_km = sample` - later larger sample updates: - `latest_total_mileage_km` - `daily_mileage_km` - `daily_total_mileage_km` - out-of-order smaller sample updates: - `first_total_mileage_km` - `daily_mileage_km` - [ ] **Step 3: Implement MySQL upsert** Use one table and one idempotent upsert: ```sql INSERT INTO vehicle_daily_metric (vin, stat_date, protocol, metric_key, metric_value, metric_unit, first_total_mileage_km, latest_total_mileage_km, sample_count, calculation_method) VALUES (?, ?, ?, ?, ?, 'km', ?, ?, 1, 'TOTAL_MILEAGE_DIFF') ON DUPLICATE KEY UPDATE first_total_mileage_km = LEAST(first_total_mileage_km, VALUES(first_total_mileage_km)), latest_total_mileage_km = GREATEST(latest_total_mileage_km, VALUES(latest_total_mileage_km)), metric_value = CASE WHEN metric_key = 'daily_mileage_km' THEN GREATEST(latest_total_mileage_km, VALUES(latest_total_mileage_km)) - LEAST(first_total_mileage_km, VALUES(first_total_mileage_km)) WHEN metric_key = 'daily_total_mileage_km' THEN GREATEST(latest_total_mileage_km, VALUES(latest_total_mileage_km)) ELSE VALUES(metric_value) END, sample_count = sample_count + 1, updated_at = CURRENT_TIMESTAMP ``` - [ ] **Step 4: Verify** Run: ```bash cd go/vehicle-gateway go test ./internal/stats go build ./cmd/stat-writer ``` Expected: pass. --- ### Task 8: Implement Redis Realtime API **Files:** - Create: `go/vehicle-gateway/internal/realtime/repository.go` - Create: `go/vehicle-gateway/internal/realtime/repository_test.go` - Create: `go/vehicle-gateway/internal/realtime/http.go` - Create: `go/vehicle-gateway/cmd/realtime-api/main.go` - [ ] **Step 1: Add repository contract** Methods: - `Update(ctx, envelope.FrameEnvelope) error` - `GetMerged(ctx, vin string) (Snapshot, error)` - `GetProtocol(ctx, vin string, protocol envelope.Protocol) (Snapshot, error)` - `IsOnline(ctx, vin string) (OnlineStatus, error)` - [ ] **Step 2: Implement merge logic** Rules: - update `vehicle:latest:{vin}:{protocol}` - update `vehicle:latest:{vin}` with newest fields - update `vehicle:online:{vin}` - update sorted set `vehicle:last_seen` - [ ] **Step 3: Implement HTTP API** Routes: - `GET /api/realtime/vehicles/{vin}` - `GET /api/realtime/vehicles/{vin}/online` - `GET /api/realtime/vehicles/{vin}/protocols/{protocol}` - [ ] **Step 4: Verify** Run: ```bash cd go/vehicle-gateway go test ./internal/realtime go build ./cmd/realtime-api ``` Expected: pass. --- ### Task 9: Wire Gateway Runtime **Files:** - Create: `go/vehicle-gateway/internal/gateway/tcp_server.go` - Create: `go/vehicle-gateway/internal/gateway/mqtt_client.go` - Modify: `go/vehicle-gateway/cmd/gateway/main.go` - [ ] **Step 1: Implement TCP server** Rules: - one goroutine per accepted connection - bounded max connections - read timeout and idle timeout - protocol-specific frame extractor - structured peer endpoint - graceful shutdown on SIGTERM - [ ] **Step 2: Implement MQTT client** Rules: - connect with official production config from environment - subscribe configured topic list - convert each message into envelope - publish raw and unified events - reconnect with backoff - [ ] **Step 3: Verify local JSON mode** Run without Kafka: ```bash cd go/vehicle-gateway GB32960_TCP_ADDR=:132960 JT808_TCP_ADDR=:18080 go run ./cmd/gateway ``` Expected: service starts and logs configured listeners. --- ### Task 10: Docker and ECS Deployment **Files:** - Create: `go/vehicle-gateway/Dockerfile` - Create: `deploy/portainer/docker-compose-go.yml` - Modify: `docs/operations/current-ecs-deployment.md` - [ ] **Step 1: Add multi-stage Dockerfile** Build all commands: - `gateway` - `history-writer` - `stat-writer` - `realtime-api` - [ ] **Step 2: Add Portainer compose** Services: - `go-vehicle-gateway` - `go-history-writer` - `go-stat-writer` - `go-realtime-api` Each service must include: - restart policy - memory limit - Kafka env - MySQL/TDengine/Redis env as needed - logging options - [ ] **Step 3: Verify Linux build** Run: ```bash cd go/vehicle-gateway GOOS=linux GOARCH=amd64 go build ./cmd/gateway GOOS=linux GOARCH=amd64 go build ./cmd/history-writer GOOS=linux GOARCH=amd64 go build ./cmd/stat-writer GOOS=linux GOARCH=amd64 go build ./cmd/realtime-api ``` Expected: all commands exit `0`. --- ### Task 11: Production Verification **Files:** - Create: `docs/operations/go-vehicle-gateway-verification.md` - [ ] **Step 1: Record test commands** Document commands to verify: - gateway process health - Kafka topic consumption - TDengine row counts - MySQL daily metric rows - Redis realtime lookup - [ ] **Step 2: Validate real traffic** Evidence required: - one real 32960 VIN with RAW, location, mileage point, daily metric, Redis snapshot - one real JT808 phone/VIN with RAW, location, mileage point, daily metric, Redis snapshot - one real Yutong MQTT VIN with RAW and Redis snapshot - [ ] **Step 3: Keep old Java services until evidence is captured** Only disable Java equivalents after the evidence file contains successful command outputs and timestamps. --- ## Self-Review Checklist - The plan creates one new Go module instead of extending scattered prototypes. - The plan covers GB32960, JT808, and Yutong MQTT ingress. - The plan covers Kafka, TDengine, MySQL, and Redis. - The plan includes 32960 and 808 daily mileage and daily total mileage. - The plan includes local and ECS verification. - The plan does not restore Xinda Push. - The plan keeps Java services untouched until Go evidence exists.