Verification and the Quality Layer
Defining what good looks like . . .
*This piece is part of How to Design an Agentic System, a series on the design decisions required to specify an internal agentic system.
Your agent produces output. How do you know if it’s good enough? If you’re reviewing every result manually, you haven’t built an autonomous system. You’ve built a drafting tool with extra steps. Verification is how you earn the right to let the agent ship without you watching.
What Verification Means Here
Traditional testing checks deterministic code against expected outputs. The same input always produces the same output, and either it matches or it doesn’t.
Agentic verification is different. The output varies every run. Two executions of the same task will produce different code, different text, different decisions. What stays constant is the standard: does the output meet the quality criteria? Tests pass. Schema validates. Style scores above a threshold. Facts check out. The evaluation is stable even when the output is not.
This distinction matters because it changes what you’re building. You’re not writing assertions against specific values. You’re defining what “good” looks like, then checking whether the output meets that bar.
The Ralph Loop
Geoffrey Huntley’s creation is a surprisingly powerful (and simple) technique. Attempt, verify, iterate. The agent tries something. A separate process evaluates the result. If it fails, the agent gets feedback and tries again. If it passes, the output moves forward.
This works when two conditions hold: you can define clearly what “good” looks like, and the task is valuable enough to justify burning tokens on iteration. Code is the obvious fit. The test suite either passes or it doesn’t. The linter either clears or it doesn’t. Run the agent, run the tests, feed failures back, let the agent fix and retry.
But the Ralph Loop is a technique, not a requirement. An executive assistant that drafts your emails doesn’t need a formal verification loop. A research agent summarizing a paper probably doesn’t either. The technique applies when the reward signal is clear and the cost of iteration is lower than the cost of a bad output.
Generator/Verifier Separation
Here’s the constraint that makes verification work: the thing that produces the output cannot be the thing that evaluates it.
An agent asked to write code and then assess whether that code is correct will almost always find it acceptable. It just wrote it. It has every reason to believe it’s good. This isn’t a flaw in the model. It’s a structural problem. Self-evaluation biases toward confirmation.
The fix is separation. The generator produces output. A different process, often a different model or a deterministic check, evaluates it. Stripe runs CI as the verifier. StrongDM runs a full system replica (their “Digital Twin Universe”) against every change. Superpowers requires TDD before any code generation begins, so the verification criteria exist before the generator even starts.
The strongest verification systems make this separation architectural. The generator never sees the verifier’s internals. The verifier never modifies the output. They communicate through a defined interface: output in, pass/fail (plus feedback) out.
Verification Approaches
Different domains call for different techniques. The common thread is that the verifier is always a separate process from the generator.
Tests and CI. The most mature approach. Your test suite, linter, type checker, and CI pipeline all serve as verifiers. The agent generates code, the pipeline evaluates it. This works because code has decades of verification infrastructure already built. Stripe caps agents at two CI iterations before escalating to a human. Bounded loops prevent runaway token spend.
LLM-as-judge. A second model evaluates the output of the first. Useful for content, research summaries, email drafts. The judge uses different criteria (accuracy, tone, completeness) and ideally a different model to avoid correlated failures. Less reliable than deterministic checks, but applicable to domains where deterministic checks don’t exist.
Statistical validation. For data extraction and transformation: values in expected ranges, required fields populated, distributions matching historical patterns. Deterministic, fast, and effective for structured output.
Human review as a formal gate. Not “I’ll glance at it.” A defined checkpoint where a human evaluates against explicit criteria before the output proceeds. This is verification, not just oversight. The distinction: a formal gate has criteria. Casual review does not.
Council-of-elders. Multiple models evaluate the same output independently. Different models catch different failure modes. Agreement across models is a stronger signal than any single model’s judgment. Useful for high-stakes decisions where one verifier isn’t enough.
Building Verification for New Domains
Code has tests, linters, type checkers, and CI pipelines. That tooling took decades to build. Most other domains have nothing comparable. But that’s an investment gap, not a fundamental limitation.
You can build verification for any domain where you can articulate what “good” looks like.
For content: style scoring against reference samples, fact-checking against source documents, readability metrics, voice consistency checks. For data extraction: schema validation, value range checks, cross-reference against known records. For research: citation verification, sample-size checks, methodology assessment (my research agent Elise learned this one the hard way). For customer communication: sentiment analysis, compliance keyword checks, response-time verification.
None of these are as mature as a test suite. All of them are better than nothing. The question isn’t “is my domain verifiable?” It’s “how much am I willing to invest in verification infrastructure for this domain?”
When Verification Compounds
The payoff from verification is non-linear. Early investment feels expensive. You’re building infrastructure that slows down each individual run. But over time, the feedback loop tightens the entire system.
Devin went from merging 34% of its PRs to 67% over 18 months, while also getting 4x faster and 2x more efficient. That improvement didn’t come from a better model alone. It came from tightening the verification loop: better tests, better feedback, better iteration criteria. Each cycle taught the system something about what “good” looks like.
The counterpoint is equally instructive. Jellyfish analyzed 20 million PRs and found that AI-coauthored PRs have approximately 1.7x more issues than human-only PRs. Scale without verification doesn’t just fail to improve. It compounds failures. More output means more bad output means more downstream problems.
The compounding works in both directions. Invest in verification and quality improves over time. Skip it and quality degrades as you scale.
When It’s NOT Worth It
Not everything needs a formal verification loop. The decision comes down to three factors.
Stakes. If the cost of a bad output is low (a draft email you’ll read before sending, a research summary you’ll scan before citing), the verification infrastructure costs more than just fixing the mistakes.
Frequency. One-off tasks don’t justify building a verifier. If you’ll run this workflow a hundred times, invest. If you’ll run it once, review it yourself.
Definability. Some quality criteria are genuinely hard to formalize. “Does this sound like me?” is a real requirement for content, but building a reliable automated check for it is harder than just reading the output. When the quality bar is subjective and you can’t articulate clear pass/fail criteria, human judgment is the right verifier.
Verification is an investment. Like any investment, it has a break-even point. For high-frequency, high-stakes, clearly definable quality criteria, that break-even comes fast. For low-frequency, low-stakes, subjective quality, it may never arrive.