mirror of
https://github.com/AmanTahiliani/box-box.git
synced 2026-08-07 19:56:18 -04:00
Add paddock briefing RSS backend spike
This commit is contained in:
110
documentations/refactor/27-phase-19b-paddock-briefing-rss.md
Normal file
110
documentations/refactor/27-phase-19b-paddock-briefing-rss.md
Normal file
@@ -0,0 +1,110 @@
|
|||||||
|
# Phase 19B Paddock Briefing RSS Backend Spike
|
||||||
|
|
||||||
|
## Goal
|
||||||
|
|
||||||
|
Prototype the backend foundation for a future fan-facing Paddock Briefing module
|
||||||
|
without scraping article pages or touching Race Hub UI. The spike keeps news as a
|
||||||
|
local-first cache: feeds can be fetched and normalized by backend code, stored in
|
||||||
|
SQLite, and read through a small API shape.
|
||||||
|
|
||||||
|
## Source Evaluation
|
||||||
|
|
||||||
|
Recommended first-party or publisher-owned feeds:
|
||||||
|
|
||||||
|
- FIA official RSS, `https://www.fia.com/rss/news`: keep as the official source
|
||||||
|
for federation announcements and regulatory context.
|
||||||
|
- BBC Sport F1, `https://feeds.bbci.co.uk/sport/formula1`: strong free headline
|
||||||
|
source for UK-centered coverage.
|
||||||
|
- Autosport F1, `https://www.autosport.com/rss/f1/news/`: useful motorsport
|
||||||
|
specialist feed; retain summaries only when provided by the feed.
|
||||||
|
- RaceFans F1, `https://www.racefans.net/category/f1-news/feed/`: useful
|
||||||
|
independent specialist feed with a clean WordPress RSS surface.
|
||||||
|
- Guardian Formula One,
|
||||||
|
`https://www.theguardian.com/sport/formulaone/rss`: broad editorial coverage
|
||||||
|
and stable RSS conventions.
|
||||||
|
|
||||||
|
Optional sources to evaluate before shipping:
|
||||||
|
|
||||||
|
- RACER F1, `https://racer.com/f1/feed`: reasonable supplemental specialist
|
||||||
|
feed.
|
||||||
|
- Formula 1 YouTube Atom,
|
||||||
|
`https://www.youtube.com/feeds/videos.xml?channel_id=UCB_qr75-ydFVKSF9Dmo6izg`:
|
||||||
|
video-only briefing cards, separate from article news.
|
||||||
|
- Motorsport.com F1, `https://www.motorsport.com/rss/f1/news/`: not included in
|
||||||
|
the default prototype list until terms and caching expectations are reviewed.
|
||||||
|
|
||||||
|
Avoid Formula1.com scraping or hidden endpoints, X/Twitter scraping, Reddit as a
|
||||||
|
primary news source, and feed aggregator products such as RSS.app or Feedspot.
|
||||||
|
|
||||||
|
## Legal And Product Caveats
|
||||||
|
|
||||||
|
Only fetch publisher-provided RSS/Atom XML. Do not fetch article bodies, bypass
|
||||||
|
paywalls, scrape Open Graph metadata, or store full article content. The product
|
||||||
|
surface should show source, title, canonical URL, publish time, category, and a
|
||||||
|
short feed-provided summary/snippet when available. Each card should link users
|
||||||
|
to the publisher site for the article.
|
||||||
|
|
||||||
|
Before enabling a source by default, review the publisher feed terms, robots/TOS
|
||||||
|
language around caching, and whether feed summaries are intended for display.
|
||||||
|
Keep TTLs conservative and make source attribution visible in the UI.
|
||||||
|
|
||||||
|
## Implemented Proof
|
||||||
|
|
||||||
|
This spike adds:
|
||||||
|
|
||||||
|
- `internal/news`: a standard-library RSS/Atom parser and polite fetch helper
|
||||||
|
with a 10-second default timeout and a box-box User-Agent.
|
||||||
|
- URL-based deduplication with UTM parameter stripping.
|
||||||
|
- `news_sources` and `news_items` tables in SQLite migration `003_news.sql`.
|
||||||
|
- Store/query methods for upserting cached feed metadata/items and listing
|
||||||
|
newest cached items.
|
||||||
|
- `GET /api/v1/news`, with optional `limit` and `source` query params.
|
||||||
|
- Unit tests using local XML fixtures only.
|
||||||
|
|
||||||
|
The endpoint is intentionally read-only against the local SQLite cache. It does
|
||||||
|
not fetch feeds during web requests, avoiding unexpected network work in the
|
||||||
|
product UI path. A later ingestion command can call `internal/news.Fetch`, upsert
|
||||||
|
sources/items, and mark `fetched_at`/`expires_at` according to a TTL policy.
|
||||||
|
|
||||||
|
## API Shape
|
||||||
|
|
||||||
|
`GET /api/v1/news?limit=25&source=racefans-f1`
|
||||||
|
|
||||||
|
Response:
|
||||||
|
|
||||||
|
```json
|
||||||
|
[
|
||||||
|
{
|
||||||
|
"source": "racefans-f1",
|
||||||
|
"title": "Example headline",
|
||||||
|
"url": "https://publisher.example/story",
|
||||||
|
"published_at": "2026-05-25T14:00:00Z",
|
||||||
|
"summary": "Feed-provided snippet",
|
||||||
|
"category": "news",
|
||||||
|
"fetched_at": "2026-05-25T14:10:00Z"
|
||||||
|
}
|
||||||
|
]
|
||||||
|
```
|
||||||
|
|
||||||
|
Default limit is 25; maximum accepted limit is 100.
|
||||||
|
|
||||||
|
## Follow-Up Frontend Plan
|
||||||
|
|
||||||
|
Add a Paddock Briefing surface outside Race Hub while Race Hub redesign work is
|
||||||
|
active. Recommended first UI slice:
|
||||||
|
|
||||||
|
- Query `/api/v1/news?limit=12`.
|
||||||
|
- Group by recency with source badges and external-link treatment.
|
||||||
|
- Show snippets only when present, with clear publisher attribution.
|
||||||
|
- Add source filters after the cache refresh command exists.
|
||||||
|
- Treat video feed items as a separate rail or filter, not mixed into hard-news
|
||||||
|
headlines by default.
|
||||||
|
|
||||||
|
## Open Questions
|
||||||
|
|
||||||
|
- Should news refresh live behind an explicit CLI command, opportunistic startup
|
||||||
|
refresh, or a manual button in an admin/data-health screen?
|
||||||
|
- What default TTL should each source use? A 15-30 minute TTL is reasonable for
|
||||||
|
race weekends; longer may be enough outside live sessions.
|
||||||
|
- Should the cache keep historical briefing items indefinitely, or prune after a
|
||||||
|
rolling window such as 30-90 days?
|
||||||
@@ -92,6 +92,9 @@ not implementation tickets yet.
|
|||||||
- [25 Phase 18 Fan Command Center](25-phase-18-fan-command-center.md): reworks
|
- [25 Phase 18 Fan Command Center](25-phase-18-fan-command-center.md): reworks
|
||||||
`/` around race-weekend identity and splits ingestion/admin concerns into
|
`/` around race-weekend identity and splits ingestion/admin concerns into
|
||||||
the new `/admin` (Data Health) route.
|
the new `/admin` (Data Health) route.
|
||||||
|
- [27 Phase 19B Paddock Briefing RSS](27-phase-19b-paddock-briefing-rss.md):
|
||||||
|
backend spike for publisher-owned RSS/Atom feeds, local SQLite caching, and a
|
||||||
|
future fan-facing briefing API.
|
||||||
|
|
||||||
## External References
|
## External References
|
||||||
|
|
||||||
|
|||||||
260
internal/news/news.go
Normal file
260
internal/news/news.go
Normal file
@@ -0,0 +1,260 @@
|
|||||||
|
package news
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"encoding/xml"
|
||||||
|
"fmt"
|
||||||
|
"io"
|
||||||
|
"net/http"
|
||||||
|
"net/url"
|
||||||
|
"sort"
|
||||||
|
"strings"
|
||||||
|
"time"
|
||||||
|
)
|
||||||
|
|
||||||
|
const UserAgent = "box-box/phase-19b-rss-spike"
|
||||||
|
|
||||||
|
// Source describes a feed that can be fetched and normalized.
|
||||||
|
type Source struct {
|
||||||
|
ID string
|
||||||
|
Name string
|
||||||
|
URL string
|
||||||
|
Category string
|
||||||
|
}
|
||||||
|
|
||||||
|
// Item is the normalized shape consumed by storage and API layers.
|
||||||
|
type Item struct {
|
||||||
|
Source string
|
||||||
|
Title string
|
||||||
|
URL string
|
||||||
|
PublishedAt time.Time
|
||||||
|
Summary string
|
||||||
|
Category string
|
||||||
|
FetchedAt time.Time
|
||||||
|
}
|
||||||
|
|
||||||
|
// DefaultSources are free RSS/Atom feeds worth using for the Paddock Briefing spike.
|
||||||
|
var DefaultSources = []Source{
|
||||||
|
{ID: "fia", Name: "FIA", URL: "https://www.fia.com/rss/news", Category: "official"},
|
||||||
|
{ID: "bbc-f1", Name: "BBC Sport F1", URL: "https://feeds.bbci.co.uk/sport/formula1", Category: "news"},
|
||||||
|
{ID: "autosport-f1", Name: "Autosport F1", URL: "https://www.autosport.com/rss/f1/news/", Category: "news"},
|
||||||
|
{ID: "racefans-f1", Name: "RaceFans F1", URL: "https://www.racefans.net/category/f1-news/feed/", Category: "news"},
|
||||||
|
{ID: "guardian-f1", Name: "Guardian Formula One", URL: "https://www.theguardian.com/sport/formulaone/rss", Category: "news"},
|
||||||
|
{ID: "racer-f1", Name: "RACER F1", URL: "https://racer.com/f1/feed", Category: "news"},
|
||||||
|
{ID: "f1-youtube", Name: "Formula 1 YouTube", URL: "https://www.youtube.com/feeds/videos.xml?channel_id=UCB_qr75-ydFVKSF9Dmo6izg", Category: "video"},
|
||||||
|
}
|
||||||
|
|
||||||
|
// Fetch retrieves and parses one RSS or Atom feed with the provided HTTP client.
|
||||||
|
func Fetch(ctx context.Context, client *http.Client, source Source) ([]Item, error) {
|
||||||
|
if client == nil {
|
||||||
|
client = &http.Client{Timeout: 10 * time.Second}
|
||||||
|
}
|
||||||
|
req, err := http.NewRequestWithContext(ctx, http.MethodGet, source.URL, nil)
|
||||||
|
if err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
req.Header.Set("User-Agent", UserAgent)
|
||||||
|
|
||||||
|
resp, err := client.Do(req)
|
||||||
|
if err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
defer resp.Body.Close()
|
||||||
|
if resp.StatusCode < 200 || resp.StatusCode >= 300 {
|
||||||
|
return nil, fmt.Errorf("fetch %s: status %d", source.ID, resp.StatusCode)
|
||||||
|
}
|
||||||
|
return Parse(source, resp.Body, time.Now().UTC())
|
||||||
|
}
|
||||||
|
|
||||||
|
// Parse normalizes RSS 2.0 or Atom XML into Items.
|
||||||
|
func Parse(source Source, r io.Reader, fetchedAt time.Time) ([]Item, error) {
|
||||||
|
payload, err := io.ReadAll(io.LimitReader(r, 2<<20))
|
||||||
|
if err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
|
||||||
|
var rss rssFeed
|
||||||
|
if err := xml.Unmarshal(payload, &rss); err == nil && len(rss.Channel.Items) > 0 {
|
||||||
|
return normalizeRSS(source, rss.Channel.Items, fetchedAt), nil
|
||||||
|
}
|
||||||
|
|
||||||
|
var atom atomFeed
|
||||||
|
if err := xml.Unmarshal(payload, &atom); err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
if len(atom.Entries) == 0 {
|
||||||
|
return nil, fmt.Errorf("parse %s: no RSS items or Atom entries found", source.ID)
|
||||||
|
}
|
||||||
|
return normalizeAtom(source, atom.Entries, fetchedAt), nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// DeduplicateByURL keeps the newest instance of each canonical URL.
|
||||||
|
func DeduplicateByURL(items []Item) []Item {
|
||||||
|
byURL := make(map[string]Item, len(items))
|
||||||
|
for _, item := range items {
|
||||||
|
key := canonicalURL(item.URL)
|
||||||
|
if key == "" {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
item.URL = key
|
||||||
|
if existing, ok := byURL[key]; !ok || item.PublishedAt.After(existing.PublishedAt) {
|
||||||
|
byURL[key] = item
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
out := make([]Item, 0, len(byURL))
|
||||||
|
for _, item := range byURL {
|
||||||
|
out = append(out, item)
|
||||||
|
}
|
||||||
|
sort.Slice(out, func(i, j int) bool {
|
||||||
|
return out[i].PublishedAt.After(out[j].PublishedAt)
|
||||||
|
})
|
||||||
|
return out
|
||||||
|
}
|
||||||
|
|
||||||
|
type rssFeed struct {
|
||||||
|
Channel struct {
|
||||||
|
Items []rssItem `xml:"item"`
|
||||||
|
} `xml:"channel"`
|
||||||
|
}
|
||||||
|
|
||||||
|
type rssItem struct {
|
||||||
|
Title string `xml:"title"`
|
||||||
|
Link string `xml:"link"`
|
||||||
|
GUID string `xml:"guid"`
|
||||||
|
PubDate string `xml:"pubDate"`
|
||||||
|
Description string `xml:"description"`
|
||||||
|
Categories []string `xml:"category"`
|
||||||
|
}
|
||||||
|
|
||||||
|
type atomFeed struct {
|
||||||
|
Entries []atomEntry `xml:"entry"`
|
||||||
|
}
|
||||||
|
|
||||||
|
type atomEntry struct {
|
||||||
|
Title string `xml:"title"`
|
||||||
|
ID string `xml:"id"`
|
||||||
|
Updated string `xml:"updated"`
|
||||||
|
Published string `xml:"published"`
|
||||||
|
Summary string `xml:"summary"`
|
||||||
|
Content string `xml:"content"`
|
||||||
|
Links []atomLink `xml:"link"`
|
||||||
|
Categories []struct {
|
||||||
|
Term string `xml:"term,attr"`
|
||||||
|
Label string `xml:"label,attr"`
|
||||||
|
} `xml:"category"`
|
||||||
|
}
|
||||||
|
|
||||||
|
type atomLink struct {
|
||||||
|
Href string `xml:"href,attr"`
|
||||||
|
Rel string `xml:"rel,attr"`
|
||||||
|
}
|
||||||
|
|
||||||
|
func normalizeRSS(source Source, raw []rssItem, fetchedAt time.Time) []Item {
|
||||||
|
items := make([]Item, 0, len(raw))
|
||||||
|
for _, entry := range raw {
|
||||||
|
link := strings.TrimSpace(entry.Link)
|
||||||
|
if link == "" {
|
||||||
|
link = strings.TrimSpace(entry.GUID)
|
||||||
|
}
|
||||||
|
items = append(items, Item{
|
||||||
|
Source: source.ID,
|
||||||
|
Title: cleanText(entry.Title),
|
||||||
|
URL: link,
|
||||||
|
PublishedAt: parseFeedTime(entry.PubDate),
|
||||||
|
Summary: cleanText(entry.Description),
|
||||||
|
Category: firstNonEmpty(entry.Categories, source.Category),
|
||||||
|
FetchedAt: fetchedAt,
|
||||||
|
})
|
||||||
|
}
|
||||||
|
return DeduplicateByURL(items)
|
||||||
|
}
|
||||||
|
|
||||||
|
func normalizeAtom(source Source, raw []atomEntry, fetchedAt time.Time) []Item {
|
||||||
|
items := make([]Item, 0, len(raw))
|
||||||
|
for _, entry := range raw {
|
||||||
|
category := source.Category
|
||||||
|
if len(entry.Categories) > 0 {
|
||||||
|
category = firstNonEmpty([]string{entry.Categories[0].Label, entry.Categories[0].Term}, source.Category)
|
||||||
|
}
|
||||||
|
items = append(items, Item{
|
||||||
|
Source: source.ID,
|
||||||
|
Title: cleanText(entry.Title),
|
||||||
|
URL: atomEntryURL(entry),
|
||||||
|
PublishedAt: parseFeedTime(firstNonEmpty([]string{entry.Published, entry.Updated}, "")),
|
||||||
|
Summary: cleanText(firstNonEmpty([]string{entry.Summary, entry.Content}, "")),
|
||||||
|
Category: category,
|
||||||
|
FetchedAt: fetchedAt,
|
||||||
|
})
|
||||||
|
}
|
||||||
|
return DeduplicateByURL(items)
|
||||||
|
}
|
||||||
|
|
||||||
|
func atomEntryURL(entry atomEntry) string {
|
||||||
|
for _, link := range entry.Links {
|
||||||
|
if link.Rel == "" || link.Rel == "alternate" {
|
||||||
|
return strings.TrimSpace(link.Href)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if len(entry.Links) > 0 {
|
||||||
|
return strings.TrimSpace(entry.Links[0].Href)
|
||||||
|
}
|
||||||
|
return strings.TrimSpace(entry.ID)
|
||||||
|
}
|
||||||
|
|
||||||
|
func parseFeedTime(value string) time.Time {
|
||||||
|
value = strings.TrimSpace(value)
|
||||||
|
if value == "" {
|
||||||
|
return time.Time{}
|
||||||
|
}
|
||||||
|
layouts := []string{
|
||||||
|
time.RFC1123Z,
|
||||||
|
time.RFC1123,
|
||||||
|
time.RFC3339,
|
||||||
|
time.RFC3339Nano,
|
||||||
|
"Mon, 02 Jan 2006 15:04:05 -0700",
|
||||||
|
"Mon, 2 Jan 2006 15:04:05 -0700",
|
||||||
|
}
|
||||||
|
for _, layout := range layouts {
|
||||||
|
if ts, err := time.Parse(layout, value); err == nil {
|
||||||
|
return ts.UTC()
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return time.Time{}
|
||||||
|
}
|
||||||
|
|
||||||
|
func cleanText(value string) string {
|
||||||
|
value = strings.TrimSpace(value)
|
||||||
|
value = strings.ReplaceAll(value, "\n", " ")
|
||||||
|
value = strings.ReplaceAll(value, "\t", " ")
|
||||||
|
return strings.Join(strings.Fields(value), " ")
|
||||||
|
}
|
||||||
|
|
||||||
|
func firstNonEmpty(values []string, fallback string) string {
|
||||||
|
for _, value := range values {
|
||||||
|
if strings.TrimSpace(value) != "" {
|
||||||
|
return strings.TrimSpace(value)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return fallback
|
||||||
|
}
|
||||||
|
|
||||||
|
func canonicalURL(raw string) string {
|
||||||
|
raw = strings.TrimSpace(raw)
|
||||||
|
if raw == "" {
|
||||||
|
return ""
|
||||||
|
}
|
||||||
|
parsed, err := url.Parse(raw)
|
||||||
|
if err != nil || parsed.Scheme == "" || parsed.Host == "" {
|
||||||
|
return raw
|
||||||
|
}
|
||||||
|
parsed.Fragment = ""
|
||||||
|
q := parsed.Query()
|
||||||
|
for key := range q {
|
||||||
|
if strings.HasPrefix(strings.ToLower(key), "utm_") {
|
||||||
|
q.Del(key)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
parsed.RawQuery = q.Encode()
|
||||||
|
return parsed.String()
|
||||||
|
}
|
||||||
72
internal/news/news_test.go
Normal file
72
internal/news/news_test.go
Normal file
@@ -0,0 +1,72 @@
|
|||||||
|
package news
|
||||||
|
|
||||||
|
import (
|
||||||
|
"strings"
|
||||||
|
"testing"
|
||||||
|
"time"
|
||||||
|
)
|
||||||
|
|
||||||
|
func TestParseRSSDeduplicatesByURL(t *testing.T) {
|
||||||
|
fixture := `<?xml version="1.0"?>
|
||||||
|
<rss version="2.0">
|
||||||
|
<channel>
|
||||||
|
<item>
|
||||||
|
<title>First story</title>
|
||||||
|
<link>https://example.com/f1/story?utm_source=rss</link>
|
||||||
|
<pubDate>Mon, 25 May 2026 10:00:00 GMT</pubDate>
|
||||||
|
<description>Latest from the paddock</description>
|
||||||
|
<category>Formula 1</category>
|
||||||
|
</item>
|
||||||
|
<item>
|
||||||
|
<title>Duplicate story newer</title>
|
||||||
|
<link>https://example.com/f1/story</link>
|
||||||
|
<pubDate>Mon, 25 May 2026 11:00:00 GMT</pubDate>
|
||||||
|
<description>Updated headline</description>
|
||||||
|
</item>
|
||||||
|
</channel>
|
||||||
|
</rss>`
|
||||||
|
source := Source{ID: "example", Category: "news"}
|
||||||
|
items, err := Parse(source, strings.NewReader(fixture), time.Unix(100, 0).UTC())
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("Parse() error = %v", err)
|
||||||
|
}
|
||||||
|
if len(items) != 1 {
|
||||||
|
t.Fatalf("items len = %d, want 1", len(items))
|
||||||
|
}
|
||||||
|
if items[0].Title != "Duplicate story newer" {
|
||||||
|
t.Fatalf("title = %q, want newer duplicate", items[0].Title)
|
||||||
|
}
|
||||||
|
if items[0].URL != "https://example.com/f1/story" {
|
||||||
|
t.Fatalf("url = %q, want canonical URL", items[0].URL)
|
||||||
|
}
|
||||||
|
if items[0].PublishedAt.IsZero() {
|
||||||
|
t.Fatal("PublishedAt was not parsed")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestParseAtom(t *testing.T) {
|
||||||
|
fixture := `<?xml version="1.0" encoding="UTF-8"?>
|
||||||
|
<feed xmlns="http://www.w3.org/2005/Atom">
|
||||||
|
<entry>
|
||||||
|
<title>Formula 1 video</title>
|
||||||
|
<link rel="alternate" href="https://www.youtube.com/watch?v=abc123"/>
|
||||||
|
<published>2026-05-25T12:30:00Z</published>
|
||||||
|
<summary>Highlights from the weekend</summary>
|
||||||
|
<category term="video"/>
|
||||||
|
</entry>
|
||||||
|
</feed>`
|
||||||
|
source := Source{ID: "f1-youtube", Category: "video"}
|
||||||
|
items, err := Parse(source, strings.NewReader(fixture), time.Unix(200, 0).UTC())
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("Parse() error = %v", err)
|
||||||
|
}
|
||||||
|
if len(items) != 1 {
|
||||||
|
t.Fatalf("items len = %d, want 1", len(items))
|
||||||
|
}
|
||||||
|
if items[0].Source != "f1-youtube" || items[0].Category != "video" {
|
||||||
|
t.Fatalf("item = %+v, want source/category preserved", items[0])
|
||||||
|
}
|
||||||
|
if items[0].URL != "https://www.youtube.com/watch?v=abc123" {
|
||||||
|
t.Fatalf("url = %q, want alternate link", items[0].URL)
|
||||||
|
}
|
||||||
|
}
|
||||||
51
internal/query/news.go
Normal file
51
internal/query/news.go
Normal file
@@ -0,0 +1,51 @@
|
|||||||
|
package query
|
||||||
|
|
||||||
|
import (
|
||||||
|
"time"
|
||||||
|
|
||||||
|
"github.com/AmanTahiliani/box-box/internal/store"
|
||||||
|
)
|
||||||
|
|
||||||
|
// NewsItem is the frontend-facing cached briefing shape.
|
||||||
|
type NewsItem struct {
|
||||||
|
Source string `json:"source"`
|
||||||
|
Title string `json:"title"`
|
||||||
|
URL string `json:"url"`
|
||||||
|
PublishedAt *time.Time `json:"published_at,omitempty"`
|
||||||
|
Summary string `json:"summary,omitempty"`
|
||||||
|
Category string `json:"category,omitempty"`
|
||||||
|
FetchedAt time.Time `json:"fetched_at"`
|
||||||
|
}
|
||||||
|
|
||||||
|
// ListNews returns cached briefing items.
|
||||||
|
func (s *Service) ListNews(limit int, source string) ([]NewsItem, error) {
|
||||||
|
rows, err := s.store.ListNewsItems(limit, source)
|
||||||
|
if err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
out := make([]NewsItem, 0, len(rows))
|
||||||
|
for _, row := range rows {
|
||||||
|
out = append(out, NewsItem{
|
||||||
|
Source: row.Source,
|
||||||
|
Title: row.Title,
|
||||||
|
URL: row.URL,
|
||||||
|
PublishedAt: row.PublishedAt,
|
||||||
|
Summary: row.Summary,
|
||||||
|
Category: row.Category,
|
||||||
|
FetchedAt: row.FetchedAt,
|
||||||
|
})
|
||||||
|
}
|
||||||
|
return out, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func NewsItemToStore(item NewsItem) store.NewsItem {
|
||||||
|
return store.NewsItem{
|
||||||
|
URL: item.URL,
|
||||||
|
Source: item.Source,
|
||||||
|
Title: item.Title,
|
||||||
|
PublishedAt: item.PublishedAt,
|
||||||
|
Summary: item.Summary,
|
||||||
|
Category: item.Category,
|
||||||
|
FetchedAt: item.FetchedAt,
|
||||||
|
}
|
||||||
|
}
|
||||||
24
internal/store/migrations/003_news.sql
Normal file
24
internal/store/migrations/003_news.sql
Normal file
@@ -0,0 +1,24 @@
|
|||||||
|
CREATE TABLE IF NOT EXISTS news_sources (
|
||||||
|
source TEXT PRIMARY KEY,
|
||||||
|
name TEXT NOT NULL,
|
||||||
|
feed_url TEXT NOT NULL,
|
||||||
|
category TEXT,
|
||||||
|
enabled INTEGER NOT NULL DEFAULT 1,
|
||||||
|
fetched_at INTEGER,
|
||||||
|
expires_at INTEGER,
|
||||||
|
updated_at INTEGER NOT NULL
|
||||||
|
);
|
||||||
|
|
||||||
|
CREATE TABLE IF NOT EXISTS news_items (
|
||||||
|
url TEXT PRIMARY KEY,
|
||||||
|
source TEXT NOT NULL,
|
||||||
|
title TEXT NOT NULL,
|
||||||
|
published_at INTEGER,
|
||||||
|
summary TEXT,
|
||||||
|
category TEXT,
|
||||||
|
fetched_at INTEGER NOT NULL,
|
||||||
|
FOREIGN KEY (source) REFERENCES news_sources(source)
|
||||||
|
);
|
||||||
|
|
||||||
|
CREATE INDEX IF NOT EXISTS idx_news_items_published ON news_items (published_at DESC);
|
||||||
|
CREATE INDEX IF NOT EXISTS idx_news_items_source_published ON news_items (source, published_at DESC);
|
||||||
@@ -28,6 +28,29 @@ type IngestionRun struct {
|
|||||||
SummaryJSON string
|
SummaryJSON string
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// NewsSource stores RSS/Atom feed metadata for local-first briefing reads.
|
||||||
|
type NewsSource struct {
|
||||||
|
Source string
|
||||||
|
Name string
|
||||||
|
FeedURL string
|
||||||
|
Category string
|
||||||
|
Enabled bool
|
||||||
|
FetchedAt *time.Time
|
||||||
|
ExpiresAt *time.Time
|
||||||
|
UpdatedAt time.Time
|
||||||
|
}
|
||||||
|
|
||||||
|
// NewsItem stores a normalized feed item deduplicated by URL.
|
||||||
|
type NewsItem struct {
|
||||||
|
URL string
|
||||||
|
Source string
|
||||||
|
Title string
|
||||||
|
PublishedAt *time.Time
|
||||||
|
Summary string
|
||||||
|
Category string
|
||||||
|
FetchedAt time.Time
|
||||||
|
}
|
||||||
|
|
||||||
// Meeting is a race weekend record.
|
// Meeting is a race weekend record.
|
||||||
type Meeting struct {
|
type Meeting struct {
|
||||||
MeetingKey int
|
MeetingKey int
|
||||||
|
|||||||
134
internal/store/news.go
Normal file
134
internal/store/news.go
Normal file
@@ -0,0 +1,134 @@
|
|||||||
|
package store
|
||||||
|
|
||||||
|
import (
|
||||||
|
"database/sql"
|
||||||
|
"fmt"
|
||||||
|
"time"
|
||||||
|
)
|
||||||
|
|
||||||
|
// UpsertNewsSource inserts or updates RSS/Atom feed metadata.
|
||||||
|
func (s *Store) UpsertNewsSource(src NewsSource) error {
|
||||||
|
updatedAt := src.UpdatedAt
|
||||||
|
if updatedAt.IsZero() {
|
||||||
|
updatedAt = time.Now().UTC()
|
||||||
|
}
|
||||||
|
_, err := s.db.Exec(`
|
||||||
|
INSERT INTO news_sources (
|
||||||
|
source, name, feed_url, category, enabled, fetched_at, expires_at, updated_at
|
||||||
|
) VALUES (?, ?, ?, ?, ?, ?, ?, ?)
|
||||||
|
ON CONFLICT(source) DO UPDATE SET
|
||||||
|
name = excluded.name,
|
||||||
|
feed_url = excluded.feed_url,
|
||||||
|
category = excluded.category,
|
||||||
|
enabled = excluded.enabled,
|
||||||
|
fetched_at = excluded.fetched_at,
|
||||||
|
expires_at = excluded.expires_at,
|
||||||
|
updated_at = excluded.updated_at
|
||||||
|
`,
|
||||||
|
src.Source,
|
||||||
|
src.Name,
|
||||||
|
src.FeedURL,
|
||||||
|
nullString(src.Category),
|
||||||
|
boolInt(src.Enabled),
|
||||||
|
nullableTime(src.FetchedAt),
|
||||||
|
nullableTime(src.ExpiresAt),
|
||||||
|
updatedAt.Unix(),
|
||||||
|
)
|
||||||
|
if err != nil {
|
||||||
|
return fmt.Errorf("upsert news source: %w", err)
|
||||||
|
}
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// UpsertNewsItem inserts or updates a normalized feed item.
|
||||||
|
func (s *Store) UpsertNewsItem(item NewsItem) error {
|
||||||
|
_, err := s.db.Exec(`
|
||||||
|
INSERT INTO news_items (
|
||||||
|
url, source, title, published_at, summary, category, fetched_at
|
||||||
|
) VALUES (?, ?, ?, ?, ?, ?, ?)
|
||||||
|
ON CONFLICT(url) DO UPDATE SET
|
||||||
|
source = excluded.source,
|
||||||
|
title = excluded.title,
|
||||||
|
published_at = excluded.published_at,
|
||||||
|
summary = excluded.summary,
|
||||||
|
category = excluded.category,
|
||||||
|
fetched_at = excluded.fetched_at
|
||||||
|
`,
|
||||||
|
item.URL,
|
||||||
|
item.Source,
|
||||||
|
item.Title,
|
||||||
|
nullableTime(item.PublishedAt),
|
||||||
|
nullString(item.Summary),
|
||||||
|
nullString(item.Category),
|
||||||
|
item.FetchedAt.Unix(),
|
||||||
|
)
|
||||||
|
if err != nil {
|
||||||
|
return fmt.Errorf("upsert news item: %w", err)
|
||||||
|
}
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// ListNewsItems returns newest cached news items, optionally filtered by source.
|
||||||
|
func (s *Store) ListNewsItems(limit int, source string) ([]NewsItem, error) {
|
||||||
|
if limit <= 0 || limit > 100 {
|
||||||
|
limit = 25
|
||||||
|
}
|
||||||
|
|
||||||
|
query := `
|
||||||
|
SELECT url, source, title, published_at, summary, category, fetched_at
|
||||||
|
FROM news_items
|
||||||
|
`
|
||||||
|
var args []any
|
||||||
|
if source != "" {
|
||||||
|
query += ` WHERE source = ?`
|
||||||
|
args = append(args, source)
|
||||||
|
}
|
||||||
|
query += ` ORDER BY COALESCE(published_at, fetched_at) DESC, fetched_at DESC LIMIT ?`
|
||||||
|
args = append(args, limit)
|
||||||
|
|
||||||
|
rows, err := s.db.Query(query, args...)
|
||||||
|
if err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
defer rows.Close()
|
||||||
|
|
||||||
|
var out []NewsItem
|
||||||
|
for rows.Next() {
|
||||||
|
var item NewsItem
|
||||||
|
var published sql.NullInt64
|
||||||
|
var summary, category sql.NullString
|
||||||
|
var fetched int64
|
||||||
|
if err := rows.Scan(
|
||||||
|
&item.URL,
|
||||||
|
&item.Source,
|
||||||
|
&item.Title,
|
||||||
|
&published,
|
||||||
|
&summary,
|
||||||
|
&category,
|
||||||
|
&fetched,
|
||||||
|
); err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
item.PublishedAt = nullTimePtr(published)
|
||||||
|
item.Summary = summary.String
|
||||||
|
item.Category = category.String
|
||||||
|
item.FetchedAt = time.Unix(fetched, 0).UTC()
|
||||||
|
out = append(out, item)
|
||||||
|
}
|
||||||
|
return out, rows.Err()
|
||||||
|
}
|
||||||
|
|
||||||
|
func nullableTime(v *time.Time) any {
|
||||||
|
if v == nil || v.IsZero() {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
return v.Unix()
|
||||||
|
}
|
||||||
|
|
||||||
|
func nullTimePtr(v sql.NullInt64) *time.Time {
|
||||||
|
if !v.Valid {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
t := time.Unix(v.Int64, 0).UTC()
|
||||||
|
return &t
|
||||||
|
}
|
||||||
@@ -28,8 +28,8 @@ func TestOpenAppliesMigrations(t *testing.T) {
|
|||||||
if err != nil {
|
if err != nil {
|
||||||
t.Fatalf("SchemaVersion() error = %v", err)
|
t.Fatalf("SchemaVersion() error = %v", err)
|
||||||
}
|
}
|
||||||
if version != 2 {
|
if version != 3 {
|
||||||
t.Fatalf("SchemaVersion() = %d, want 2", version)
|
t.Fatalf("SchemaVersion() = %d, want 3", version)
|
||||||
}
|
}
|
||||||
|
|
||||||
tables := []string{
|
tables := []string{
|
||||||
@@ -48,6 +48,8 @@ func TestOpenAppliesMigrations(t *testing.T) {
|
|||||||
"race_control",
|
"race_control",
|
||||||
"weather",
|
"weather",
|
||||||
"laps",
|
"laps",
|
||||||
|
"news_sources",
|
||||||
|
"news_items",
|
||||||
}
|
}
|
||||||
for _, table := range tables {
|
for _, table := range tables {
|
||||||
var name string
|
var name string
|
||||||
@@ -84,6 +86,12 @@ func TestMigrationsAreIdempotent(t *testing.T) {
|
|||||||
if count != 1 {
|
if count != 1 {
|
||||||
t.Fatalf("schema_migrations v2 count = %d, want 1", count)
|
t.Fatalf("schema_migrations v2 count = %d, want 1", count)
|
||||||
}
|
}
|
||||||
|
if err := s.db.QueryRow(`SELECT COUNT(*) FROM schema_migrations WHERE version = 3`).Scan(&count); err != nil {
|
||||||
|
t.Fatalf("count schema_migrations v3: %v", err)
|
||||||
|
}
|
||||||
|
if count != 1 {
|
||||||
|
t.Fatalf("schema_migrations v3 count = %d, want 1", count)
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestRawPayloadInsertAndRead(t *testing.T) {
|
func TestRawPayloadInsertAndRead(t *testing.T) {
|
||||||
@@ -597,6 +605,58 @@ func TestAnalyticsUpsertRead(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func TestNewsUpsertRead(t *testing.T) {
|
||||||
|
s := openTestStore(t)
|
||||||
|
now := time.Unix(1800000000, 0).UTC()
|
||||||
|
expires := now.Add(30 * time.Minute)
|
||||||
|
|
||||||
|
if err := s.UpsertNewsSource(NewsSource{
|
||||||
|
Source: "bbc-f1",
|
||||||
|
Name: "BBC Sport F1",
|
||||||
|
FeedURL: "https://feeds.bbci.co.uk/sport/formula1",
|
||||||
|
Category: "news",
|
||||||
|
Enabled: true,
|
||||||
|
FetchedAt: &now,
|
||||||
|
ExpiresAt: &expires,
|
||||||
|
UpdatedAt: now,
|
||||||
|
}); err != nil {
|
||||||
|
t.Fatalf("UpsertNewsSource() error = %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
published := now.Add(-time.Hour)
|
||||||
|
item := NewsItem{
|
||||||
|
URL: "https://example.com/f1/story",
|
||||||
|
Source: "bbc-f1",
|
||||||
|
Title: "Paddock update",
|
||||||
|
PublishedAt: &published,
|
||||||
|
Summary: "Short briefing text",
|
||||||
|
Category: "news",
|
||||||
|
FetchedAt: now,
|
||||||
|
}
|
||||||
|
if err := s.UpsertNewsItem(item); err != nil {
|
||||||
|
t.Fatalf("UpsertNewsItem() error = %v", err)
|
||||||
|
}
|
||||||
|
updated := item
|
||||||
|
updated.Title = "Paddock update revised"
|
||||||
|
if err := s.UpsertNewsItem(updated); err != nil {
|
||||||
|
t.Fatalf("second UpsertNewsItem() error = %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
items, err := s.ListNewsItems(10, "bbc-f1")
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("ListNewsItems() error = %v", err)
|
||||||
|
}
|
||||||
|
if len(items) != 1 {
|
||||||
|
t.Fatalf("items len = %d, want 1", len(items))
|
||||||
|
}
|
||||||
|
if items[0].Title != updated.Title {
|
||||||
|
t.Fatalf("title = %q, want %q", items[0].Title, updated.Title)
|
||||||
|
}
|
||||||
|
if items[0].PublishedAt == nil || !items[0].PublishedAt.Equal(published) {
|
||||||
|
t.Fatalf("published_at = %v, want %v", items[0].PublishedAt, published)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
func TestWithTxRollback(t *testing.T) {
|
func TestWithTxRollback(t *testing.T) {
|
||||||
s := openTestStore(t)
|
s := openTestStore(t)
|
||||||
|
|
||||||
|
|||||||
@@ -115,6 +115,24 @@ func (s *Server) handleSessions(w http.ResponseWriter, r *http.Request) {
|
|||||||
writeJSON(w, sessions)
|
writeJSON(w, sessions)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// --- /api/v1/news ---
|
||||||
|
|
||||||
|
func (s *Server) handleNews(w http.ResponseWriter, r *http.Request) {
|
||||||
|
if !s.hasLocalQuery() {
|
||||||
|
writeJSON(w, []query.NewsItem{})
|
||||||
|
return
|
||||||
|
}
|
||||||
|
|
||||||
|
limit, _ := strconv.Atoi(r.URL.Query().Get("limit"))
|
||||||
|
source := strings.TrimSpace(r.URL.Query().Get("source"))
|
||||||
|
items, err := s.query.ListNews(limit, source)
|
||||||
|
if err != nil {
|
||||||
|
writeError(w, err, http.StatusInternalServerError, false)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
writeJSON(w, items)
|
||||||
|
}
|
||||||
|
|
||||||
// --- /api/v1/drivers ---
|
// --- /api/v1/drivers ---
|
||||||
|
|
||||||
func (s *Server) handleDrivers(w http.ResponseWriter, r *http.Request) {
|
func (s *Server) handleDrivers(w http.ResponseWriter, r *http.Request) {
|
||||||
|
|||||||
@@ -192,6 +192,52 @@ func TestHandleSeasonsWithData(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func TestHandleNewsWithLocalData(t *testing.T) {
|
||||||
|
st := openTestStore(t)
|
||||||
|
now := time.Unix(1800000000, 0).UTC()
|
||||||
|
published := now.Add(-30 * time.Minute)
|
||||||
|
if err := st.UpsertNewsSource(store.NewsSource{
|
||||||
|
Source: "racefans-f1",
|
||||||
|
Name: "RaceFans F1",
|
||||||
|
FeedURL: "https://www.racefans.net/category/f1-news/feed/",
|
||||||
|
Category: "news",
|
||||||
|
Enabled: true,
|
||||||
|
UpdatedAt: now,
|
||||||
|
}); err != nil {
|
||||||
|
t.Fatalf("UpsertNewsSource() error = %v", err)
|
||||||
|
}
|
||||||
|
if err := st.UpsertNewsItem(store.NewsItem{
|
||||||
|
URL: "https://example.com/story",
|
||||||
|
Source: "racefans-f1",
|
||||||
|
Title: "RaceFans story",
|
||||||
|
PublishedAt: &published,
|
||||||
|
Summary: "Brief summary",
|
||||||
|
Category: "news",
|
||||||
|
FetchedAt: now,
|
||||||
|
}); err != nil {
|
||||||
|
t.Fatalf("UpsertNewsItem() error = %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
srv := testServer(t, st)
|
||||||
|
req := httptest.NewRequest(http.MethodGet, "/api/v1/news?source=racefans-f1&limit=5", nil)
|
||||||
|
rec := httptest.NewRecorder()
|
||||||
|
srv.handleNews(rec, req)
|
||||||
|
|
||||||
|
if rec.Code != http.StatusOK {
|
||||||
|
t.Fatalf("status = %d, want 200", rec.Code)
|
||||||
|
}
|
||||||
|
var items []query.NewsItem
|
||||||
|
if err := json.Unmarshal(rec.Body.Bytes(), &items); err != nil {
|
||||||
|
t.Fatalf("decode response: %v", err)
|
||||||
|
}
|
||||||
|
if len(items) != 1 {
|
||||||
|
t.Fatalf("items len = %d, want 1", len(items))
|
||||||
|
}
|
||||||
|
if items[0].Source != "racefans-f1" || items[0].Title != "RaceFans story" {
|
||||||
|
t.Fatalf("items[0] = %+v, want seeded item", items[0])
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
func TestHandleWeekendNotFound(t *testing.T) {
|
func TestHandleWeekendNotFound(t *testing.T) {
|
||||||
st := openTestStore(t)
|
st := openTestStore(t)
|
||||||
srv := testServer(t, st)
|
srv := testServer(t, st)
|
||||||
|
|||||||
@@ -64,6 +64,7 @@ func (s *Server) routes() (http.Handler, error) {
|
|||||||
mux.HandleFunc("/api/v1/race-hub", s.handleRaceHub)
|
mux.HandleFunc("/api/v1/race-hub", s.handleRaceHub)
|
||||||
mux.HandleFunc("/api/v1/seasons", s.handleSeasons)
|
mux.HandleFunc("/api/v1/seasons", s.handleSeasons)
|
||||||
mux.HandleFunc("/api/v1/weekend", s.handleWeekend)
|
mux.HandleFunc("/api/v1/weekend", s.handleWeekend)
|
||||||
|
mux.HandleFunc("/api/v1/news", s.handleNews)
|
||||||
mux.HandleFunc("/api/v1/meetings", s.handleMeetings)
|
mux.HandleFunc("/api/v1/meetings", s.handleMeetings)
|
||||||
mux.HandleFunc("/api/v1/sessions", s.handleSessions)
|
mux.HandleFunc("/api/v1/sessions", s.handleSessions)
|
||||||
mux.HandleFunc("/api/v1/drivers", s.handleDrivers)
|
mux.HandleFunc("/api/v1/drivers", s.handleDrivers)
|
||||||
|
|||||||
Reference in New Issue
Block a user