Feature Deep-Dive: Sources (RSS & X)

What it does

Gives readers a single place to curate their signal. Adding or removing sources updates Supabase, enqueues a job, and kicks off ingestion with optimistic UI responses.

Why it matters

Without clean source management, the HUD becomes static. This flow handles validation, deduping, and job logging so the ingestion workers know exactly what to fetch next.

Design

sequenceDiagram
  participant UI
  participant API as Next API (/api/sources)
  participant DB as Supabase (sources, jobs)
  participant EF as Edge Function (rss-fetch/twitter-fetch)
  UI->>API: POST { type, url|handle }
  API->>DB: insert source + enqueue job
  API->>EF: POST /functions/v1/(rss|twitter)-fetch
  EF->>DB: upsert items, update job
  UI->>API: GET /api/feed

Tech choices & trade-offs

OptionPerfComplexityCostDXWhy we picked/not
Edge Functions ingestHighMed$GreatChosen — near DB, serverless
Next API ingestMedLow$GoodMore load on web tier

Implementation walkthrough

web/src/app/api/sources/route.ts handles both listing and creation. Noteworthy moves:

  • Validation via zod: we guard against malformed URLs and rewrite handles with normalizeTwitterHandle.
  • Profile bootstrap: first-time users get a profiles row so preferences have a home.
  • Job enqueue: new sources trigger an insert into public.jobs, which ingestion functions read later.
  • Immediate fetch: we call the relevant Edge Function (rss-fetch or twitter-fetch) using the service role, so users see content without waiting for cron.

Deletion lives in web/src/app/api/sources/[id]/route.ts and enforces ownership using the Supabase session.

Smart bits (gotchas)

  • We keep a unique constraint on (user_id, url) and (user_id, handle) so the same feed can’t be added twice.
  • Display names are auto-generated: RSS grabs the hostname, Twitter prettifies the handle via formatTwitterHandle.
  • The job table records every enqueue attempt; if the Edge Function fails, the error column tells you why.

Testing

We typically:

  • Add a new RSS source, assert the response includes status: 'queued', then poll /api/feed to confirm items arrive.
  • Attempt to add the same handle twice and expect a 500 from Supabase (handled with user-friendly messaging client-side).

Operational notes

  • If you rotate your Supabase service-role key, update the environment variables on Vercel immediately—otherwise source creation will succeed but ingestion won’t fire.
  • Rate-limit responses from Twitter are logged in jobs.error; keep an eye on repeated rate_limited entries.

What we’d do next

  • Offer recommended starter sources so brand-new users aren’t staring at an empty HUD.
  • Let readers tag sources with their own labels, feeding future personalization.