box-box/documentations/refactor/02-backend-architecture.md

Backend Architecture

Summary

The backend should move from direct page handlers calling OpenF1 into a layered local-first architecture. Source clients fetch data, ingestion persists it, store/query packages expose domain reads, and Web handlers return read models with source and freshness metadata.

Proposed Package Boundaries

internal/store

Owns SQLite as the local domain database.

Responsibilities:

• Schema creation and migrations.
• Typed upsert methods for domain records.
• Typed read methods for screens and backend services.
• Raw payload storage.
• Ingestion metadata and provenance.
• Transactions and batch writes.

Non-goals:

• Calling OpenF1 directly.
• Knowing Web UI route behavior.
• Rendering derived frontend-specific structures unless they are shared read models.

internal/ingest

Coordinates backfill, refresh, and opportunistic fetches.

Responsibilities:

• Ingest year, meeting, or session.
• Fetch required endpoints through source clients.
• Persist raw payloads and normalized rows.
• Track partial successes and failures.
• Support resumable, idempotent runs.
• Respect rate limits and free-tier constraints.

Default ingestion modes:

• CLI bulk ingestion for years, meetings, and sessions.
• Opportunistic small fetches in Web mode when a user opens missing data.
• Explicit refresh mode for completed data when needed.

Rate-limit defaults:

• Bulk ingestion must be resumable and idempotent.
• Bulk ingestion should default to conservative sequential fetching with a delay between OpenF1 requests.
• Failed requests should use bounded exponential backoff with jitter.
• HTTP 429 and live-session lockout should pause or stop the current run rather than tight-loop retries.
• --dry-run should show planned datasets and estimated request count before a large ingest.

OpenF1 Source Client Layer

The current internal/api client can remain, but it should become one source adapter rather than the main application data layer.

Responsibilities:

• Build OpenF1 URLs.
• Apply auth headers when OPENF1_API_KEY exists.
• Decode source payloads into source/domain structs.
• Preserve stale fallback behavior where useful.

Future direction:

• Make source fetches observable by ingestion metadata.
• Avoid direct UI route dependency on source calls.

Live Timing Bridge

The current live parser should be extracted out of internal/ui into reusable backend/domain logic.

Responsibilities:

• Connect to official F1 SignalR.
• Parse topic payloads into typed live events/state.
• Maintain current live snapshot.
• Broadcast snapshots to Web clients through SSE.
• Feed TUI live mode without coupling parser code to Bubble Tea.
• Persist live events/snapshots as an append-only stream once the bridge is extracted.

Persistence policy:

• Live SignalR data should be stored separately from normalized post-session OpenF1 records.
• Live data represents what was broadcast at the time, not necessarily the corrected final historical record.
• A later reconciliation step can compare live stream data with OpenF1 post-session records.

Web API Read Models

Web handlers should become thin adapters from query services to JSON.

Responsibilities:

• Validate route/query parameters.
• Call local-first query/read services.
• Return consistent response envelopes.
• Include source/freshness metadata.

Suggested response metadata:

• source: local, api, cache, live, or missing.
• last_ingested_at.
• is_stale.
• missing_datasets.
• errors where partial data is returned.

CLI Ingestion Commands

CLI commands should make bulk ingestion explicit and user-controlled.

Candidate commands/flags:

• --ingest-year 2024
• --ingest-meeting <meeting_key>
• --ingest-session <session_key>
• --refresh
• --dry-run

CLI output should include:

• What will be fetched.
• What is already local.
• What succeeded.
• What failed.
• Whether the run is resumable.

Local-First Read Behavior

Default rule:

1. Read from local domain DB.
2. If missing and request scope is small, optionally fetch from OpenF1.
3. Persist successful fetches.
4. Return local/read-model data with metadata.
5. If OpenF1 is unavailable, return partial local data and clear missing/stale metadata rather than an empty page.

Examples:

• Opening a completed race with all local data should perform no OpenF1 calls.
• Opening a completed race with missing weather may opportunistically fetch only weather.
• Opening a whole season should not silently trigger a large backfill.
• During live-session lockout, historical pages should still render from local data.

Opportunistic Fetch Policy

Allowed by default:

• Single meeting sessions.
• Single session results/grid/weather/race control.
• Small metadata gaps needed to render a screen.

Not allowed by default:

• Full season backfills.
• High-volume telemetry/location/car data.
• Repeated refresh loops during API lockout.
• Silent destructive refresh of completed local data.

Migration Strategy

The existing SQLite HTTP cache should remain operational during the refactor. The new domain database should be introduced without requiring users to delete their current cache.

Default migration stance:

• Keep the current cache tables and stale fallback behavior intact.
• Introduce domain tables through internal/store.
• Prefer a separate domain database file at first if it materially reduces migration risk; using the same SQLite file remains acceptable if table names and migrations are carefully isolated.
• Do not attempt to transform arbitrary URL-keyed cache entries into domain rows automatically.
• New ingestion runs should populate domain tables from fresh source fetches or explicitly supported raw payloads.
• Web routes can migrate endpoint by endpoint from source-first to local-first.

Failure Modes

The backend should explicitly represent:

• Local data available.
• Local data partial.
• Local data missing.
• OpenF1 locked/unavailable.
• Stale cache fallback used.
• Live feed connected/disconnected.
• Ingestion partial failure.

The Web UI should be able to show these states without guesswork.

Open Questions

• Should API response envelopes be introduced globally or per endpoint during migration?
• How should source schema drift be detected and surfaced?
• What is the minimum dataset required for a Race Hub to be considered complete?