AI Quality Scoring

Curating content at scale means you need a way to decide what's worth showing. I built a weighted scoring system that uses LLM evaluation to rate content on an integer 1-10 scale across multiple dimensions. It runs in both Siteinspire and Popular Archive, though the dimensions differ. Siteinspire scores on visual design, technical execution, and layout innovation. Popular Archive scores on editorial quality, factual depth, and topical relevance. The weights shift depending on the content type, but the mechanism is the same.

The scoring prompt is deliberately constrained. I ask for integer scores only, no decimals, no ranges. Each dimension gets a brief rubric so the LLM has concrete anchors: a 3 in visual design means "functional but unremarkable," a 7 means "distinctive, cohesive design language," a 10 means "sets a new standard." Without these anchors, scores drift over time as the model interprets "good" differently across sessions. The rubric keeps it grounded. I also enforce a JSON response format so parsing is deterministic. No free-text reasoning in the scoring response, just numbers.

The composite score is a weighted sum, normalized back to 1-10. For Siteinspire, visual design carries 40% weight, technical execution 35%, layout innovation 25%. These weights came from manual calibration: I scored 50 sites by hand, then adjusted weights until the automated scores matched my rankings within one point for 80% of cases. It took three rounds of adjustment to get there. The remaining 20% divergence is mostly on sites where I have personal taste preferences the model doesn't share, which is fine. Consistency matters more than perfect agreement with my own judgment.

The real insight is that the score itself isn't the product. What matters is having a consistent evaluative lens applied uniformly across thousands of items. Human curation is better for any individual piece, but it doesn't scale. A systematic scoring approach lets me filter 500 new sites per week down to the 30 worth featuring, and the threshold is applied the same way every time. The system doesn't replace taste. It operationalizes it.