Frontier Digest

AI-lab updates are spread across separate sites. Frontier Digest brings articles into one searchable collection and lets a reader inspect the sources behind an answer.

This was my second retrieval project. I used it to work on collecting live articles and serving answers through a web interface and Telegram — the parts a first retrieval project skips. It retrieves full article bodies, not listing summaries, so answers can draw on the actual content.

A job runs once a day, on its own, with no laptop involved:

1. 1. Scrape. It visits ~18 lab blogs, finds new articles, and extracts each body to a file.
2. 2. Summarise. Each new article goes to Claude Haiku, which returns a short “nugget,” a topic, and a 1–5 importance signal in one call.
3. 3. Digest. It drops low-signal items, caps each lab, and emails the top nuggets (email delivery is currently test-only) and broadcasts them to Telegram subscribers.
4. 4. Commit. The fresh data is committed back to GitHub, which wakes the host to rebuild the search index and the site.

When you ask a question, the system embeds it, retrieves the closest article chunks, and answers from only those chunks — returning the answer with source links, and saying “I don't know” when the answer is not in the articles. Citation support and behaviour when evidence is missing still need formal evaluation.

The Telegram bot @JiGyasaaBOT is not a second copy of the logic — Telegram pushes each message to a webhook on the same hosted app, which runs the same retrieval-and-answer core and replies with sources. /start subscribes you to the daily message,/stop removes you, and /latest lists the week's posts. A bot can only message people who wrote to it first, so every subscriber opted in — the safest broadcast model, enforced by the platform.

• Git is the glue. The daily job writes fresh data into the repo; the web host reads that same repo to rebuild. The two halves never talk directly, so the system needs no database and no shared disk for its content — a simpler, more reliable design.
• A tooling constraint became the cleanest seam. The scraper must run in its own isolated environment, so the build split cleanly along a file boundary: the scraper writes files under one interpreter, the RAG reads them under another.
• Webhook, not polling, for the bot. Telegram pushes updates to the already-hosted app, so there is no extra always-on process, and the webhook is authenticated with a constant-time secret compare.
• Migrate by write-concurrency, not importance. Only data that a live web request writes (the subscriber list) went into a real database; single-writer state stays as JSON in git.
• Keep lookup and search separate. The typed box does semantic search; the lab chips and stats do plain lookup. Mixing the two is exactly what once produced wrong-lab sources.

Two decoupled halves, joined only through the git repo. The daily job (GitHub Actions) scrapes with Scrapling in its own environment, summarises with Claude Haiku, sends email via Resend, broadcasts to Telegram, and commits the data back. The web app (FastAPI on Render's free plan) rebuilds a Chroma index at deploy and serves the question, feed, stats, and webhook endpoints. Search uses Voyage embeddings; the subscriber list lives in Supabase, called directly over HTTP. One language — Python — runs the whole thing.

For three days the daily run failed silently. It turned out to be three bugs stacked on top of each other: a revoked API key, a missing database credential in one step, and a table that had never been created (and then needed a permission grant once it was). Because every pipeline step is set to continue on error, the run only went red at the final health check — and the real error was buried in a step log.

I had first guessed the cause was an out-of-memory crash from a “50-second-then-500” symptom; the logs showed a fast 401, and the 50 seconds was just a cold start. The lesson stuck: read the step logs, don't infer — and when code starts depending on a new table or environment variable, create it, grant it, and wire the job's environment in the same change.

• The demo runs on a free tier that sleeps after idle, so the first request after a nap can take tens of seconds to wake.
• It answers from the articles and says “I don't know” when the answer is not in them — but retrieval and citation quality are not yet quantified.
• Email delivery is currently test-only, pending a verified sending domain.
• Scraping is best-effort per site, and some pages with no readable body are skipped by design.