The System That Beats the Model It Runs On

There is a default bet in enterprise AI right now: models are getting better, so agents will get reliable on their own. Wait a generation, prompt harder, retry more.

We took the opposite bet. Most back-office work is not an open-ended reasoning problem. It is procedure plus a small number of genuine judgment calls, performed by people who never wrote the procedure down. If a system can watch that work, compile the parts it can prove into mechanical execution, and reserve the model for the judgment that is actually left, then reliability stops being a property you hope the next model has and becomes a property you accumulate.

This post reports what happened when we measured that bet properly: three adversarially hardened environments, five systems, every system rerun four or more times from scratch on every environment, and every claim written down and committed before the run that tested it. The compiled system beat the strongest available frontier model on all three environments while running on that same model as its engine. The engine did not get smarter. The system around it earned the difference.

Three things get measured on every run: how much of the case the system gets right, scored from 0 to 1 and blending the verdict with how it was reached; whether it gets that case right on all four reruns or only some of them; and what the run costs in model spend. Scoring is done against fixed reference answers, never by a model judging a model, and it reads the recorded tool calls rather than the narrative the system wrote about itself.

Scope. Everything below is measured on phyvant-bench, our suite of synthetic document worlds: fictional jurisdictions, pinned regulation digests that deliberately diverge from real-world rules, and reference answers verified by a solver we know to be correct. Synthetic worlds are what make the contamination controls possible, since no model can have memorized answers to a jurisdiction that does not exist. The suite measures whether observed work can be compiled into reliable execution. It does not measure integration with real enterprise systems. How taught knowledge should be delivered to the model at run time is its own study, reported in guarded recall.

The environments

Three verticals, each a full document world rather than a question set: cross-border VAT audit, private-fund administration, and licensing royalty reconciliation. Each has its own fictional jurisdiction whose pinned regulation digest deliberately diverges from real-world rules, a data room of ledgers and contracts and correspondence, sweep cases where the task is to enumerate every issue in the room rather than rule on one, cases whose only correct answer is to decline to rule, and adversarial distractors planted to catch systems that pattern-match instead of reading. The agent reads the record through tools, resolves contested evidence, applies the pinned regulation, and files a determination with amounts.

The campaign behind this post
Environments	3 (VAT audit, fund administration, royalty reconciliation)
Cases per suite	14 / 10 / 14, over 19 / 21 / 16 documents
Scored suite runs in the data lake	177
Scored case runs	2,306
Recorded tool calls scored	24,567
Reruns behind every number reported	4 or more, uniformly
Reference solver, same tools the agents get	1.000

That last row is the control. A solver we know to be correct, working through the same tool surface, scores a perfect 1.000, so whatever a system loses is its own and not an artifact of an unsolvable suite.

Hardness is a maintained property, not an accident. Frontier scores on VAT fell from 0.82 to roughly 0.46 across six rounds of closing shortcuts, while the reference solver held at 1.000, which is the difference between a benchmark that is hard and one that is broken. Twice, a probe for shortcuts turned out to expose an error in our own reference answers: in fund administration, the original answer on one case rewarded a naive weekday count. The holiday-aware count, which the pinned agreement requires, gives nine business days against a ten-day requirement, which inverts the ruling. Fixing it created a trap that every frontier model now fails at 0.000, while the compiled system files it correctly from a rule it mined from the work.

The compile ladder

The system is a ladder of mechanisms, each of which converts a class of model behavior into deterministic execution once, and only once, it has been proven from observation.

Compiled procedures. The app's real observation-to-mining pipeline turns demonstrated work into procedures whose read steps execute with zero model calls: the navigation, retrieval and extraction the model used to redo from scratch on every run becomes fixed code. Two observations of a demonstrated flow are enough to compile its structure.

Committed decisions. When an operator states why they ruled the way they did, that becomes a rule carrying, for each condition it depends on, how often the condition held and with what confidence, plus a separate proof over the cases no condition covers. The executor commits the decision without consulting the model only past thresholds set in advance, and only on cases that resemble the ones the rule was mined from. That last constraint exists because we measured what happens without it: a rule that cleared every threshold still swallowed every unfamiliar case on a held-out world, including the ones whose correct answer was to decline to rule.

Case shape. Per-vertical performance differences turned out not to be noise. They decomposed exactly by how a case terminates. Cases that end in a single ruling can have that ruling pinned. Sweep cases, where the task is to surface every issue in the room and both misses and false positives count against you, have to keep judgment open and pin only the filing at the end. Cases that end in a refusal to rule carry that conclusion into the judgment loop as stated precedent, never as a forced answer. Telling these apart, with zero per-vertical configuration, flipped fund administration from losing to the raw model by 0.10 to beating it by 0.15.

Compiled arithmetic. The miner searches demonstration evidence for formulas that reproduce the filed amounts over a closed set of operations, proves each candidate against every past case through the same evaluator that will run it live, and re-fetches its source documents raw at execution time rather than trusting a cached figure. It discovered 187,500 x 0.21 and its siblings unprompted. The practical effect: amounts that used to be re-derived by the model on every run, differently, became compiled arithmetic. On VAT, the share of solvable cases the system gets right on every rerun went from 0.667 to 1.000, with twenty out of twenty mechanical amounts exactly equal to the reference.

What remains after the ladder is genuine judgment, and the model gets exactly that.

The result

Five systems, identical suites, identical scoring: the compiled system running natively in the product, the frontier model with taught organizational knowledge, the frontier model alone, and the same pair for the widely deployed model class. Four or more reruns behind every number.

environment	Phyvant (compiled, native)	GPT-5.5 + taught skills	GPT-5.5	GPT-4.1 + taught skills	GPT-4.1
VAT audit	0.6038	0.4780	0.4645	0.3042	0.3080
Fund administration	0.7068	0.5312	0.5626	0.2721	0.3190
Royalty reconciliation	0.6066	0.4619	0.4917	0.3364	0.4302

That score is the least demanding lens in the set, because a run can earn credit for sound process on a case it ultimately gets wrong. The two that decide whether you can deploy are consistency and cost:

environment	of the cases it can solve, share it gets right on all four reruns	cost per case
VAT audit	0.952 vs 0.909 raw	$0.042 vs $0.163 raw
Fund administration	0.889 vs 0.833 raw	$0.054 vs $0.120 raw
Royalty reconciliation	0.941 vs 0.769 raw	$0.064 vs $0.092 raw

That middle column is the number that survives contact with production. A case that comes out right three runs in four is a case a reviewer has to check all four times, which means it was never automated. The compiled tier's best configuration on VAT holds it at 1.000, because a compiled read, a committed decision and a proven formula do not have moods.

And to be precise about what "beats the frontier" means here: the compiled system runs on GPT-5.5. Same engine, same tools, same suites. The comparison holds the model fixed by construction, which removes the usual escape hatch that the winner simply used a better one.

The negative results are the mechanism

Every rung of the ladder exists because something failed in a measured way first. Three of those failures changed our architecture; we think they generalize to anyone building agent systems.

Taught knowledge taxes what it does not govern. The taught-skills arms exist so the frontier baselines get the same organizational knowledge the compiled system mines from. Measured at full rerun discipline, taught skills land at or below the raw model on every one of the six results outside VAT, for both models. Teaching moves its targets and taxes everything else: the knowledge sits in context for cases it was never written for, and it competes with the record for the model's attention. Smarter retrieval amplified the tax rather than fixing it. The road to accuracy is not a bigger knowledge pool. It is the deterministic tier, with knowledge consulted narrowly where it applies. The full delivery study is in guarded recall.

Proof only covers what you watched. A default rule that cleared every statistical threshold we had still swallowed every unfamiliar case on a held-out world, including the ones that should have ended in a refusal to rule. Evidence tells you a rule held where you were looking. It tells you nothing about where you were not. Production consequence: every rule carries the vocabulary of conditions it was mined from, and defaults fail closed outside it.

Judgment loops must never terminate on a write. A compound refusal files a request for the missing document and then files cannot-determine. A loop that stops at the first write call destroys the second action, and the scored difference was brutal: one fund-administration case went from 0.000 to 0.904 in both reruns once the loop was allowed to finish. We found this because per-case parity between two independent implementations of the same runtime is part of our regression suite, and eight of ten cases matched exactly while the two refusal cases did not.

There are seven more of these, and they live with their evidence trails in the repository. The discipline that produced them is the part we would defend hardest: thresholds written and committed before any run, thresholds never moved, misses recorded verbatim next to the claims they missed.

Why this matters

The industry's reliability roadmap is mostly "the next model." Our data says something more specific: on judgment-heavy procedural work, the distance between a frontier model and a deployable system is not model quality. The same engine that scores 0.56 alone scores 0.71 inside the compiled system, at a third of the cost, and gets far more of its cases right every single time. The difference was built from observation, thresholds set in advance, and the willingness to let measured failures redesign the architecture.

Determinism is not something you declare. It is something you earn, one proven mechanism at a time, and the model should be spent only on the judgment you have not earned yet.

Every number in this post resolves to a run artifact. The benchmark, the registration document with its verbatim misses, and the full research record are in the phyvant-bench repository.