The Popular Archive Pipeline

The Popular Archive is a content discovery and transformation pipeline. It finds interesting content from Reddit and the web, rewrites it with a specific editorial voice -- Bill Bryson meets Schott's Miscellany -- and generates whimsical pencil-sketch illustrations. The stack is OpenRouter routing to Claude for the writing, fal.ai for image generation, and ScrapingBee for capturing source material. The hard part isn't any individual piece. It's making the output feel like a human editor curated it.

Prompt engineering for editorial voice is surprisingly nuanced. "Write like Bill Bryson" produces pastiche -- obvious, try-hard imitation. Instead, I decomposed the voice into specific qualities: parenthetical asides that add genuine context, a sense of wonder at mundane facts, dry humour that never signals it's being funny, and the confidence to tell you exactly why something matters. The system randomly selects between prose, annotated lists, and structured tables for each piece, which breaks the monotony that makes AI content feel robotic. A reader shouldn't be able to tell which format is coming next, just like a real magazine varies its editorial approach.

The self-improving discovery loop is where the system gets interesting. As the pipeline processes content and scores it for quality -- engagement signals, source authority, editorial fit -- it generates new search prompts based on what performed well. If articles about obscure Victorian inventions consistently score high, the system generates more specific discovery prompts in that space. If a topic cluster saturates, it pivots. The quality feedback loop means the discovery gets more targeted over time without manual curation. After three months, the hit rate for "content worth publishing" went from roughly 12% to around 40%.

The illustration pipeline uses fal.ai with a consistent pencil-sketch style prompt. Every article gets a header illustration that's thematically relevant but stylistically consistent across the entire archive. SHA-256 deduplication prevents the system from processing the same source material twice, and a quality gate rejects generated images that don't meet a minimum aesthetic threshold. The whole pipeline runs on a schedule -- discover, filter, rewrite, illustrate, publish -- with human review as an optional approval step rather than a bottleneck. The goal is editorial quality at algorithmic scale, and the gap between those two things is where all the interesting engineering lives.