Feature Deep-Dive: Sources (RSS & X)
What it does
Gives readers a single place to curate their signal. Adding or removing sources updates Supabase, enqueues a job, and kicks off ingestion with optimistic UI responses.
Why it matters
Without clean source management, the HUD becomes static. This flow handles validation, deduping, and job logging so the ingestion workers know exactly what to fetch next.
Design
sequenceDiagram
participant UI
participant API as Next API (/api/sources)
participant DB as Supabase (sources, jobs)
participant EF as Edge Function (rss-fetch/twitter-fetch)
UI->>API: POST { type, url|handle }
API->>DB: insert source + enqueue job
API->>EF: POST /functions/v1/(rss|twitter)-fetch
EF->>DB: upsert items, update job
UI->>API: GET /api/feed
Tech choices & trade-offs
| Option | Perf | Complexity | Cost | DX | Why we picked/not |
|---|---|---|---|---|---|
| Edge Functions ingest | High | Med | $ | Great | Chosen — near DB, serverless |
| Next API ingest | Med | Low | $ | Good | More load on web tier |
Implementation walkthrough
web/src/app/api/sources/route.ts handles both listing and creation. Noteworthy moves:
- Validation via zod: we guard against malformed URLs and rewrite handles with
normalizeTwitterHandle. - Profile bootstrap: first-time users get a
profilesrow so preferences have a home. - Job enqueue: new sources trigger an insert into
public.jobs, which ingestion functions read later. - Immediate fetch: we call the relevant Edge Function (
rss-fetchortwitter-fetch) using the service role, so users see content without waiting for cron.
Deletion lives in web/src/app/api/sources/[id]/route.ts and enforces ownership using the Supabase session.
Smart bits (gotchas)
- We keep a unique constraint on
(user_id, url)and(user_id, handle)so the same feed can’t be added twice. - Display names are auto-generated: RSS grabs the hostname, Twitter prettifies the handle via
formatTwitterHandle. - The job table records every enqueue attempt; if the Edge Function fails, the
errorcolumn tells you why.
Testing
We typically:
- Add a new RSS source, assert the response includes
status: 'queued', then poll/api/feedto confirm items arrive. - Attempt to add the same handle twice and expect a 500 from Supabase (handled with user-friendly messaging client-side).
Operational notes
- If you rotate your Supabase service-role key, update the environment variables on Vercel immediately—otherwise source creation will succeed but ingestion won’t fire.
- Rate-limit responses from Twitter are logged in
jobs.error; keep an eye on repeatedrate_limitedentries.
What we’d do next
- Offer recommended starter sources so brand-new users aren’t staring at an empty HUD.
- Let readers tag sources with their own labels, feeding future personalization.