By Anthony Skinner, CEO and Co-Founder, and the Pressfit.ai Engineering Team
Summary
Companies are being told to put agentic AI everywhere. Many will hit a wall: the token bill climbs and the headcount stays where it was, because someone still has to check every output. We hit that wall ourselves, then measured it.
The problem is not the model. A model on its own is a probabilistic system, and business processes need predictable execution. Prompts, skills, MCP tools and human review all help, but they are instructions. A model reads instructions. It doesn't execute them the way code does.
We compared four ways of running the same work: Claude alone, a structured template, skills with hooks, and our business process engine built on a graph. In the graph, code runs the process and the model only reads, judges and writes.
- Short jobs: Claude alone gave 7 different ranking orders in 10 runs of the same job. Every constrained setup gave 1. The engine ran about 45% faster than Claude alone, at 2.8 minutes against 5.1.
- Long jobs: at 300 steps, the graph got 97.5% of results right. An agent with a template and hooks got 85%, and a prompt-only agent got 70%.
- Tokens: on script-heavy jobs, the graph used 3 to 27 times fewer tokens, and about the same amount every run. On one long review it used more tokens than a plain brief, but the same amount each time.
- What it didn't fix: in every setup, one free model step still chose the main recommendation, so that recommendation still moved between runs.
The number to manage is not tokens per run. It is cost per accepted deliverable: what it takes to get one output you can use, reruns and review included.
Why agents drift
Microsoft Research and Salesforce Research ran more than 200,000 simulated conversations across 15 large language models. When a task was spread over several turns instead of given all at once, performance dropped 39% on average. Aptitude fell about 16%. Unreliability, how much results vary from run to run, rose 112%. Once a model takes a wrong turn, it tends to build on it and doesn't recover. (LLMs Get Lost in Multi-Turn Conversation)
Our own work showed the same pattern. Over long jobs, information got dropped or compressed, working files changed, and the same instructions were read differently from one run to the next. Having a document available did not mean the model used the right part of it at the right moment.
Google has drawn the same line. Its Agent Development Kit builds agents into workflows to "ensure tasks run in the required order or pattern consistently," and points new code to graph-based workflows. (Google ADK workflows)
How we got here
Our Playbook reads a customer's website and several competitors' sites, compares their content and search keywords, and builds a plan to be found in search and in AI answers. For large customers, one job could run up to 10 hours.
- Laptops and a human in the loop. Skills on Claude, with a person reviewing every deliverable. Quality was good, but nothing scaled while someone read every output.
- Autonomous on desktops. We took the human out. When a 10-hour job failed, there was no way to recover. We started over.
- Claude headless in the cloud. Skills in containers, tools and data through MCP, checkpoints and phase retries. Recovery improved, but we were managing APIs, containers and context instead of the process.
- Graphs. We split the work into code, content and decisions, and moved the order, state, math and checks into code. The model kept the reading, judging and writing.
It is an old idea. On Boeing's DCAC/MRM program in the 1990s, computer-aided process planning documented how airplanes were built and encoded the steps. Nobody asked a system to improvise the assembly.
Short jobs: four setups, 40 runs
One job: compare a fictional company's competitors, rank them by threat, and recommend its one big bet. Eight fictional competitors, 42 frozen source files (about 21,000 words) and 24 planted facts. Same brief, same model, web search off. Ten runs per setup.
| Measure | Claude alone | Template | Skills + hooks | Graph |
|---|---|---|---|---|
| Different ranking orders in 10 runs | 7 | 1 | 1 | 1 |
| Comparison table agreement | 0.88 | 0.98 | 0.99 | 0.99 |
| Pairs of runs with the same top 3 | 53% | 100% | 100% | 100% |
| Planted facts right (none stated wrongly) | 99% | 100% | 100% | 100% |
| Average minutes per run | 5.1 | 2.9 | 2.9 | 2.8 |
| Blind quality score, out of 5 | 4.84 | 3.87 | 3.77 | 3.70 |
Facts were not the problem. Claude alone got 99% of the planted facts right and stated none wrongly. Judgment moved. Fixed rules did most of the work on repeatability: the template alone stopped the ranking from moving, and the hooks never had to block anything.
Our Red Dev Test team set two bars high on purpose, and we report both results. We set a bar before looking at the data: the graph had to beat Claude alone by 0.15 on the comparison table and the ranking. It beat it by 0.11 and 0.07, so it didn't clear that bar on this job. And Claude alone scored higher on blind quality. This was a test, so we built in real-world variables, based on what we had learned: the template we built for the test capped the summary at 200 words with no room for uncertainty, and every constrained setup worked inside those limits.
Long jobs
Each long test was a follow-up to the one before. Our hypothesis: a prompt-only agent isn't great even on short runs and gets worse as runs get longer; a template and hooks help; and as runs get longer, the graph is the place to go. Every long-job result here is early, one to a few runs per setup, not yet counted runs.
20 steps against 300 steps
| Setup | Correct at 20 steps | Correct at 300 steps | Error rate at 300 steps |
|---|---|---|---|
| Graph | 100% | 97.5% | 2.5% |
| Agent, template and hooks | 95% | 85% | 15% |
| Agent, prompt only | 82.5% | 70% | 30% |
The prompt-only agent was weakest at both lengths. The template and hooks helped, but couldn't stop the slide. At 300 steps, the agent with template and hooks got six times as many results wrong as the graph, and the prompt-only agent twelve times as many. In the graph, code holds the order and the state, so the length of the job barely moved the result.
Two-hour payment job
70 steps across five regions, each region starting from the last one's result, with 25 planted failures: scripts that hang, crash, finish late, write half a result, or lose the reply after money moved. Six runs per setup.
| Claude, plain brief | Graph | |
|---|---|---|
| Results right | 60 of 60, every run | 60 of 60, every run |
| Planted failures handled | 25 of 25 | 25 of 25 |
| What ran the job | A new program each run, 344–552 lines | The same 162 lines every run |
| Minutes per run | 112–117 | 125 |
| Tokens per run | 1.7M–5.7M (3.4x swing) | 0.56M–0.63M (within 13%) |
Both got everything right. Claude was a little faster. The difference was sameness: the graph ran the same code each time and used roughly 3 to 10 times fewer tokens. On a 14-step, 30-minute job, the gap was 27 times: 0.12M tokens for the graph against 3.2M.
Annual vendor review
64 vendors, about 480,000 words, 28 policy rulings (12 of them with no right answer), and a board decision at the end to release or stop payments. About a 10-hour job at real speed, sped up to about 45 minutes for testing. Five runs per setup.
| Setup | Agreement on open rulings | Board decision | Tokens per run |
|---|---|---|---|
| Claude, plain brief | 0.78 | not reported | 5.6M–13.0M |
| Script with the rulings prompt fixed | 0.97 | Split 4 to 1 | About half the engine's |
| Graph engine | 0.97 | Same in 5 of 5 | 16.4M–16.6M |
Here the engine used about 1.8 times the tokens of a plain brief, but the same amount every run. The fixed-prompt script matched it on the rulings at half the tokens, but its board decision still split, because a later free judgment went its own way. Controlling one judgment does not control the ones after it.
Other long tests
- Killed partway and restarted: Claude recovered 4 of 4; the graph 4 of 5, with the one it missed traced back to our graph code, which is what the Red Dev Test team is there to find.
- Failures nothing announces: both handled 5 of 5. Claude named all five in its report; the graph named one.
- 20 dependent steps, 301 records judged: both 20 of 20. Claude split the work across 30 subagents and finished in 8 minutes, against 29 for the graph.
- Board decision by five votes: Claude made the same pick 6 of 6 times; the graph 5 of 6.
A real job: one phase for a national equipment rental company
In October, Claude drove the discovery phase of a Playbook from our skill. In November, the same phase ran through the graph. Both runs stop at the first approval gate.
| Measure | October: Claude driving the skill | November: graph |
|---|---|---|
| Ranked keywords pulled | 12,000 | 20,000 |
| Keywords left undecided | 4,484 (36.1%) | 68 (0.35%) |
| Competitors rated as direct | 11 | 19 |
| Non-rental sites approved | 7 | 2 |
| Start to approval gate | 4 h 47 m | 6 h 17 m |
| Claude usage | not recorded | $248.44 |
Judging competitors alone took 51 copies of Claude running side by side for about three hours. The graph decided almost every keyword and replaced directory sites like Yelp, MapQuest and Zillow with real rental rivals.
The test also showed us where to go next. Where both runs made a call, they agreed 62.5% of the time. The graph judged the company's own brand name as not relevant, treated nearly identical portable toilet terms differently, and left the gate's "approved by" field empty. The runs were two weeks apart and search data moved in between, and Claude still makes the keyword and competitor judgments in both. That is the next thing we are fixing.
Inside a graph
Our discovery graph, built from our client-discovery skill and its real runs, has 135 steps. 103 are plain code, 13 are AI agent steps, 15 are stops the graph answers on its own, and 4 fan out with one agent per item. After a few rounds of fixes, it ran unattended to the first approval gate for four held-back clients. Seventeen steps never ran in those runs; most are recovery paths for failures that didn't happen.
- Code holds the order. The model never decides what runs next.
- Code holds the state. Each step reads from it and writes back to it, so nothing important lives only in the model's memory.
- Code does the math. Scores, rankings and totals come out the same for the same inputs.
- Every model step is checked. A failed check retries once, then stops with a clear error.
- Free judgments are the risk. Find the steps where the model still decides freely, and constrain them with rules, a rubric or independent votes.
Cost per accepted deliverable
Tokens spent are a bill, not a productivity number. A run that has to be redone means another full run, plus someone to decide which answer is right.
cost per accepted deliverable = cost per run ÷ (1 − redo rate)
An illustration, not a result from our tests: at a 30% redo rate, every accepted deliverable takes about 1.43 runs. At 5%, it takes about 1.05, roughly 26% fewer runs before any gain inside each run. That is why a defined path can cost more per run and still cost less per accepted deliverable. We have not yet measured cost per accepted deliverable across counted runs; that is the next test.
What we are testing next
- Constraining the free recommendation step with independent votes and explicit criteria.
- Giving the summary room to say what it doesn't know.
- Counted, full-speed long runs against Claude alone, skills with hooks and the graph, reporting cost per accepted deliverable.
- A 10-hour job with branches, loops and judgment calls, where a graph should be strongest.
Build your own. The starter kit has everything in this report you can put to work: the spec template, acceptance criteria, graph rules, a LangGraph skeleton and a test plan.
Need help putting it to work? Contact our team at redevteam@pressfit.ai.
Sources
pressfit.ai testing, 2 to 11 November 2026. Long-job runs are early and not yet counted.
Laban, Hayashi, Zhou and Neville. LLMs Get Lost in Multi-Turn Conversation. Microsoft Research and Salesforce Research, 2025. https://arxiv.org/abs/2505.06120
Google Agent Development Kit, workflows. https://adk.dev/workflows/
The blog post: https://pressfit.ai/blog/stop-prompting-and-start-engineering