Skip to main content
AI

Stop Prompting and Start Engineering (Part 2): Why Prompt-Driven Agent Orchestration Breaks Down at Scale

Anthony Skinner, CEO and Co-Founder, and the Pressfit.ai Engineering Team20 min read

Anthropic just laid out an updated Claude Code playbook. We held its recommendations up against our 300-node pass^k reliability curves and a live enterprise production run. The short version: agent orchestration, where agents manage other agents, is fast to set up and falls apart over long jobs. A coded workflow graph gave the same result on every run, and it let a cheaper model do the judging.

What Anthropic's New Claude Code Playbook Recommends

New here? Start with Part 1: Stop Prompting and Start Engineering.

New to the jargon? Harness, hook, pass^k and the rest are defined in plain English in our glossary.

Right after we published Stop Prompting and Start Engineering, Anthropic's Lydia Hallie gave her talk Rethinking Claude Code Best Practices at AI Engineer.

In it, she walked through how Anthropic engineers work today with frontier models like Opus 5.5 and Fable 5.2. Her advice? Step out of the loop, run dozens of sessions in parallel from your terminal or the cloud, and let an orchestrator session spawn, track and verify the sub-agents doing the work.

It sounds compelling in a developer demo. In production, running repeatable business workflows this way, by prompting agents to manage other agents from a desktop, is a fool's errand.

Prefer the numbers at a glance? View Part 2 as an infographic.

The talk is about moving faster and staying safe. What it doesn't promise is repeatability: the same job giving the same answer every run.

SWE-bench Pro scores from Opus 4.5 to Opus 5.5, as reported by Anthropic
SWE-bench Pro scores as reported by Anthropic, with test settings varying by model. The models keep getting better at solving coding tasks. Solving a task once isn't the same as running a business process the same way every time. Screenshot: Lydia Hallie, Anthropic, "Rethinking Claude Code Best Practices", AI Engineer, November 2026.

You don't need the latest, most expensive frontier model to get there. Whether you build on Google's Agent Development Kit (ADK), LangGraph or our own Business Process Engine, real engineering, not desktop prompt wrangling, is what gets you off your laptop and into production at a fraction of the cost.

A Quick Refresher: The Four Ways We Tested This

If you want the full breakdown of our 40-run lab benchmark, the 70-step payment stress test and the vendor review modeled on a 10-hour job, read Part 1: Stop Prompting and Start Engineering.

For a quick refresher, we tested four ways of building AI workflows on identical frozen inputs:

  1. Unconstrained prompting (A0, prompt only): Give the model a brief and tools, and let it decide the steps, write throwaway scripts and decide when it is finished.
  2. Structured markdown templates: Lock the output into rigid templates while letting the model navigate the underlying work.
  3. Skills with programmatic hooks (A-Full, all features): Give the model structured skills, sub-agents and code hooks that block bad outputs and force retries.
  4. Deterministic graph process engine (G and G1): Compile the process into an explicit state machine. Pre-written code owns the control flow, state, math and acceptance gates; the LLM only makes bounded calls inside individual steps.

Our conclusion in Part 1 was blunt: on short, three-minute tasks, templates and hooks looked fine. As workflows stretched across dozens or hundreds of steps, probabilistic control slid.

The Math: Why Multi-Agent Orchestration Falls Apart at 300 Nodes

To see why multi-agent orchestration, "agents orchestrating agents," breaks down in enterprise production, look at reliability across workflow depth: 20-node pipelines against 300-node pipelines.

Part 1 reported the share of results correct, including the 20-step and 300-step accuracy curve in the research infographic. Since then, we've run the graph again and again, to see whether it gives the same results every time and whether it learns from run to run. These newer runs use a stricter measure. First, S_det: the share of runs (40 runs per bar: 8 different jobs, each repeated 5 times) that passed every deterministic code check on the first try.

Architecture20-node pipeline300-node pipelineWhat it means in production
Unconstrained prompt (A0)70.0%40.0%60% of runs fail at scale
Skills + hooks + all features (A-Full)70.0%55.0%Nearly half of all runs fail checks
Deterministic graph engine (G and G1)100.0%100.0%No structural drift across 160 runs

A 40% to 55% single-run pass rate is already bad. Now apply the metric that actually governs business operations, pass^k. We repeated each of the 8 jobs five times, and pass^k is the chance that all k of those repeats succeed without a human rescue. Four out of four (k=4) is our primary enterprise reliability bar. The G and G1 rows cover two versions of our graph, each run 40 times at both 20 and 300 nodes, 160 graph runs in all.

Setupk=1k=2k=3k=4 (primary bar)k=5
Graph engine (G and G1, 20 and 300 nodes)1.0001.0001.0001.000 (100%)1.000 (100%)
Prompt only (A0, 20 nodes)0.7000.5630.4500.350 (35%)0.250 (25%)
All features and hooks (A-Full, 20 nodes)0.7000.5250.4130.325 (32.5%)0.250 (25%)
All features and hooks (A-Full, 300 nodes)0.5500.2630.1000.025 (2.5%)0.000 (0%)
Prompt only (A0, 300 nodes)0.4000.1630.0500.000 (0%)0.000 (0%)

Note: values are rounded to three decimal places from exact ratios across the 8 jobs (for example, 45/80 = 0.5625). The error bars in our S_det chart are 95% confidence intervals clustered by job, which is why A-Full at 300 nodes (2 to 4 passes on every job) shows narrower bars than A0 at 20 nodes (anywhere from 0 to 5 passes per job). A-Full tested skills, sub-agents and programmatic hooks, not the Claude Code Projects orchestrator from the talk, which was in beta at the time.

The number that matters: at 300 nodes, even with every skill, hook and prompt feature turned on (A-Full), the chance of passing four consecutive runs drops to 2.5%, and it hits 0.000 at five runs. The deterministic graph holds 1.000 at both lengths.

This is why chaining autonomous agents to manage other autonomous agents compounds failure. Each step can look solid on its own, but chaining probabilistic decisions without a fixed state graph drives multi-run reliability toward zero.

Claude Code Best Practices vs. Production Reality

Listen closely to the talk and you will hear the tension engineers are running into:

  1. Humans become full-time managers of parallel bots. The talk describes Anthropic developers moving from correcting one session to managing dozens at once. Builders watching it noticed. From the comments: "Humans aren't made to review that much in parallel," and "I will never know enough to set this up safely."
An orchestrating session starting one session per task, each ending in a pull request
One orchestrating session starts a session for each of the week's tasks. Each opens a pull request; some auto-merge, and the auth migration waits for a person. Elapsed time: 5 hours 30 minutes. Screenshot: Lydia Hallie, Anthropic, "Rethinking Claude Code Best Practices", AI Engineer, November 2026.
  1. Even the safety nets leak. Claude Code's auto mode uses a separate classifier to approve most commands. In the August study cited in the talk, it blocked 89% of dangerous commands, which still lets roughly one in ten through. Human reviewers in the same study caught only about 14%. When the talk needed something that can't be talked out of, it reached for deterministic mechanisms: deny rules checked before the classifier, an operating-system sandbox, hooks that return exit code 2 to block a step, and CI that runs the same checks on every pull request.
  2. Frontier models cost real money to supervise other frontier models. The talk's example setup uses Fable 5.2 for the orchestrating session that plans and verifies, and Opus 5.5 for the threads that write the code. Builders in the comments are already feeling the margin squeeze: "I wish the orchestrator were cheaper."
DimensionAnthropic's Claude Code playbookPressfit Business Process Engine (BPE)
The core problemSame prompt, different runs. Each looks reasonable.Seven answers in ten runs, and one answer with a coded path.
How you fix driftA bigger model, more sessions, checks after the work.Code sets the path. The model gets one bounded step.
VerificationTests and CI catch drift after it happens.Acceptance tests gate every deliverable before it ships.
Who manages itYou manage dozens of agents, or an agent manages them.You manage one process, reviewed once.
Model choiceA top-tier model to orchestrate and a top-tier model to do the work.The cheapest model that gets it right. Sonnet 5.5 gave the same answer twice 97.9% of the time (95.2% agreement with Opus 5.5).
Compute profileDozens of sessions running for hours.13 of 135 steps (9.6%) call the model; code runs the rest.
Anthropic's advice vs. Pressfit

An Agent Harness Isn't the Same as Engineering the Process

Claude Code is an agent harness: the program around the model that decides what it can see and what it can touch. Much of the talk is about tuning that harness, with permission rules, auto mode, hooks and the sandbox.

Look at what the sandbox actually covers. Bash and its child processes run inside it. Claude's own Read, Edit and WebFetch, MCP servers and hooks run outside it. Each safety layer covers part of the surface, so you stack them: deny rules, then the classifier, then the sandbox, then hooks, then CI.

Claude Code's sandbox covers Bash and its child processes; Read, Edit, WebFetch, MCP servers and hooks run outside it
What Claude Code's sandbox covers on your machine, and what it doesn't. Screenshot: Lydia Hallie, Anthropic, "Rethinking Claude Code Best Practices", AI Engineer, November 2026.

A harness answers one question: what is the agent allowed to do? It doesn't answer the question that decides whether a business process is repeatable: what happens next? Inside even the best harness, the agent still picks its own path, so two runs of the same job can take two different paths, safely.

A harness keeps the agent safe. A graph makes it repeatable.

Context Engineering: The Other Half of the Harness

The other half of harnessing is context engineering: deciding exactly what goes into the model's context window at each step. Long contexts fail in four well-documented ways: poisoning, where a mistake gets into the context and keeps getting reused; distraction, where the model leans so hard on a long context that it neglects what it already knows; confusion, where extra content, like tools it doesn't need, drags down the answer; and clash, where new information and tools conflict with what's already there (How Long Contexts Fail). Chroma tested 18 models and found they grow less reliable as the input gets longer, even on simple tasks (Context Rot). LangChain sorts the fixes into four strategies: write context outside the window, select only what the step needs, compress what's left and isolate work in separate contexts (Context Engineering for Agents).

A graph is where that discipline lives in code. Our rebuilt judging step does all four:

  • Write: State lives in the graph, not in the prompt. The next runs settle the client's offering list once, so no call has to rediscover it.
  • Select: Each call gets one keyword and the examples it needs, not 350 keywords and an 851-line skill.
  • Compress: The 851-line prompt became a short XML prompt, and about 93% of it is read from the cache on every call.
  • Isolate: Every keyword is judged in its own call and returns a typed JSON answer that code checks.

That is what moved the same-answer rate from 87.7% to 99.1% on Opus and 97.9% on Sonnet, with zero keywords lost. The harness keeps the agent safe. The graph and the engineered context make it repeatable. You need both.

Field Update: The National Equipment Rental Client

In Part 1, we introduced our live production deployment for a national equipment rental client. The job: judge nearly 20,000 search keywords (19,593 at Gate 1 in Graph Run 1) to decide whether each relates to equipment the client rents or sells, and bring its top 20 online competitors to the first approval gate, Gate 1.

In October, we ran it the desktop-agent way, letting Claude read our research skill and drive the execution. It took three attempts, and two stalled on usage limits, about 1 hour and 40 minutes of waiting. It stopped with 4,484 keywords undecided, 36.1% of the 12,422 keywords at the gate, and approved 7 non-competitors in the top 20, including Yelp, Zillow, U-Haul and Lowe's.

In November, we moved that same research skill onto fixed rails inside our Business Process Engine: 135 fixed steps, made up of 103 code steps (76.3%, using zero LLM tokens), 13 AI steps (9.6%), 15 decision points the graph answers on its own (11.1%) and 4 fan-outs with one agent per item (3.0%). Coded loops ran at most 6 rounds, governed by a readiness check. Claude was limited to judging the data.

At Gate 1October (Claude driving the skill)November (Pressfit BPE graph)Change
Keywords at the gate12,42219,593+57.7%
Keywords decided (yes or no)7,93819,525+146%
Keywords left undecided4,484 (36%)68 (0.35%)98.5% fewer (a 99% lower rate)
Non-competitors in the top 207 (Yelp, Zillow, Lowe's and others)271.4% fewer
Competitors rated direct11198 more (+72.7%)
Elapsed time4 h 49 m (incl. 1 h 40 m stalled)6 h 17 m30.4% longer
Claude cost$56.14 or more (first attempt not recorded)$248.44Up to 4.4x more

Note: Gate 1 brings a shortlist of 20 competitors. In November, 2 of those 20 were still non-competitors, while 19 candidates overall were rated direct. Graph Run 1 also judged the client's own brand name as not relevant.

The graph gets better every run: October against November at Gate 1

The graph was more complete and cleaner, deciding nearly 2.5 times as many keywords, but it wasn't cheaper or faster, largely because of that volume. That is why the next round of work went into the judging step itself.

Why Putting It in a Graph Isn't Enough, and Where the Cost Comes Down

Here is what we found after Part 1. When we ran the exact same equipment rental graph a second time from scratch to check repeatability, it ran for 7.3 hours and $299.68 of Claude, against Run 1's 6.3 hours and $248.44. Each run pulls fresh search data, so the keyword count moves a little: Run 2 worked from 19,544 keywords against Run 1's 19,593, and 99.7% of them were the same in both runs, with identical search volumes. It stopped three times and exposed three failure modes:

  1. Silent truncation (after about $206): We fed Claude batches of 350 keywords at a time. On one pass, Claude returned 349 verdicts and quietly dropped a row.
  2. A loop that never ended ($3.72 more): One keyword could never be saved to the database, so the readiness check kept looping until we coded an exception rule.
  3. Prioritization blindness ($72.81 more): Each round judged the 400 highest-volume searches instead of the roughly 130 blocking keywords needed to clear Gate 1. Once we fixed that routing rule in code, the final stretch finished in 27 minutes for $17.12 and passed the readiness check in one round.

Even inside a fixed graph, Run 1 and Run 2 agreed on 86.4% of keyword decisions, meaning 13.6%, about 1 in 7, still flipped on the same keyword with the same data. The two runs also shared only 11 of their 20 Gate 1 competitors, up from 9 of 20 between October and Run 1. Why? Because we were still letting Claude judge 350 keywords at once from an 851-line skill prompt, with no strict output schema. In Run 1, Claude listed 8 core offerings for the client; in Run 2, it listed 10. Every keyword is judged against that list, so downstream judgments drifted with the shifting baseline.

So we rebuilt the judging step from the ground up:

  • Replaced the 851-line prompt: We wrote a short, structured XML system prompt with 72 labeled examples (36 yes, 36 no) covering every core offering.
  • Eliminated batching: We switched from 350-item batches to one keyword per call behind a strict JSON schema. Across 6,000 test calls, one answer went missing (0.017%), and our deterministic check caught it and re-asked automatically.
  • Pinned and cached: We pinned the exact model ID and effort, and about 93% of every call's prompt was read from the cache.

Then we ran that rebuilt step twice on the same 1,000 keywords across three tiers of Claude models, for $21.26 in total:

Same 1,000 keywords, run twiceOld graph stepOpus 5.5 (optimized)Sonnet 5.5 (optimized)Haiku 5.5 (optimized)
Same answer both times87.7%99.1%97.9%96.1%
Agrees with Opus 5.588% to 92%–95.2%90.8%
Cost per 1,000 keywords–$6.93$3.47$0.23
Keywords silently lost1 in 350000

Old graph step: "88% to 92%" is the range for the two unoptimized graph runs compared against the optimized Opus 5.5. "1 in 350" is the one keyword silently dropped from a 350-keyword batch in Graph Run 2.

Look at those numbers. When you engineer the graph and bound the step, Sonnet 5.5 gives the same answer 97.9% of the time and agrees with Opus 95.2% of the time, at half the cost ($3.47 against $6.93, 49.9% less). Haiku 5.5 gives the same answer 96.1% of the time for $0.23, 96.7% less than Opus and 8.4 points more repeatable than the old graph step. One caution: agreeing with Opus isn't the same as being right. Haiku says no more often than Opus, and a hand check of those keywords will decide which is right before we pick a model for production.

Google ADK, LangGraph or Your Own Engine: It's Engineering, Not Prompting

You don't need to build everything from scratch, and you don't need to stay stuck in a desktop terminal.

Look at modern agent frameworks like Google's Agent Development Kit (ADK) or LangGraph. ADK lets you compose structured multi-agent workflows from sequential, parallel and loop steps, with explicit state and guardrails, so you can get off your laptop and deploy scalable services.

Let's be honest about the trade-off: setting up a production workflow graph in LangGraph or Google ADK takes real software engineering. It takes time, architectural skill and rigorous acceptance testing to make sure every node, schema and retry loop holds up under load.

That is the entire point.

Prompting an agent to manage other agents feels fast on Tuesday afternoon and turns into an unauditable, token-burning mess by Friday. Engineering a deterministic workflow takes discipline up front, but once the rails are built:

  1. You review the path once, not every output. Deterministic code holds the order, the state and the acceptance gates.
  2. You scale off the desktop. Parallel worker nodes run thousands of bounded calls in the cloud without human babysitting.
  3. You stop paying the frontier tax. On a bounded step, mid-tier models like Sonnet and lightweight models like Haiku gave the same answer 96.1% to 97.9% of the time, at a fraction of Opus pricing.
You can't manage agents at scale. You can manage a process.

The big platforms ship both styles. Microsoft's Copilot Studio leans on generative orchestration, where the platform decides at runtime which topics, tools and sub-agents to call, using the conversation so far. Microsoft's own documentation notes that the same question can get a different response in a fresh test conversation than in a longer Teams chat. Meanwhile, Microsoft's Agent Framework 1.0, generally available since April 2026, recommends graph-based workflows whenever execution order matters. Google's ADK puts graph workflows front and center, and ADK 2.x deprecates its older SequentialAgent, ParallelAgent and LoopAgent classes in favor of its graph-based Workflow. The real divide isn't Microsoft against Google. It's letting a model decide the path at runtime against coding the path in advance.

When you measure multi-step reliability, the workflow-first model wins.

Architectural Comparison: Generative Orchestration vs. Coded Workflows

DimensionGenerative orchestration (for example, Copilot Studio)Coded workflows (for example, Google ADK)
Core philosophy"Digital coworkers": persistent agents that interpret intent and compose plans on the fly."Graph engineering": explicit graphs where code defines the steps and the edges between them.
Control flow and routingProbabilistic: the orchestrator picks tools, topics and sub-agents from natural-language descriptions and context.Deterministic: explicit edges and conditional routes in code decide what runs next, without asking an LLM.
Tool and function executionInvoked when the model decides a tool fits the prompt, for example through MCP.Wrapped as typed function nodes with input and output schemas that run plain code with zero LLM tokens.
Verification and error handlingTranscript review, activity-map debugging and model-driven retries.Retries, timeouts, schema checks and human-in-the-loop pauses built into the graph's nodes.
Primary deployment surfaceUser-facing productivity tools (Teams, Microsoft 365) and low-code makers.Backend services built for cloud-scale execution.

Generative Orchestration: The Unstructured Trap

Copilot Studio's generative orchestration prioritizes flexibility and low-code access. Instead of hard-coding logic trees, makers write natural-language descriptions of topics, knowledge sources and sub-agents, and the orchestrator assembles a plan at runtime.

Where this breaks down in production:

  • Probabilistic routing drift: Because the orchestrator chooses which sub-agent or tool to call from descriptions and conversation history, small changes in phrasing or upstream context can send the work down different paths.
  • Multi-turn degradation: When a manager agent delegates across several sub-agents and the transcript keeps growing, early wrong turns stick. Researchers at Microsoft and Salesforce ran more than 200,000 simulated conversations and found performance dropped 39% on average when a task was spread over several turns (LLMs Get Lost in Multi-Turn Conversation).
  • Governance and token overhead: Unstructured triggers and multi-agent handoffs need ongoing transcript audits and can create unpredictable usage spikes, such as retry loops with no coded stop condition.

Coded Workflows: Engineering Deterministic Paths

Google's ADK treats agents not as free-roaming coworkers but as bounded nodes inside a software-defined graph.

In ADK, developers define workflows with explicit edges (start, then node A, then node B), typed input and output schemas, and prebuilt flow patterns for sequential, parallel and loop steps:

  • Zero-token code nodes: Plain functions handle data fetching, math, filtering and API calls without calling an LLM.
  • Typed contracts between steps: Instead of passing long chat transcripts from agent to agent, steps pass structured results through named state keys.
  • Isolated LLM calls: Model steps (LlmAgent) are limited to the nodes where extraction, classification or writing is actually needed.

Why a Workflow Gets Better Results

Moving from generative orchestration to a coded workflow addresses the three biggest failure modes of enterprise AI:

It stops compounding error (pass^k reliability). In an unstructured multi-step process, even a 95% success rate per step compounds downward over dozens of steps. In our 300-node runs, the prompt-only and hook-assisted setups dropped to 0% and 2.5% chances of passing four consecutive runs. By locking step order, state transitions and validation gates into the graph's edges, the workflow held 100% structural consistency at both lengths.

It cuts compute cost and makes smaller models viable. Unstructured setups need expensive frontier models just to figure out what to do next, and in Part 1, rewriting code and retrying swung token use by about 3.4 times across identical runs (1.7 million to 5.7 million tokens) on the 70-step job, and from 5.6 million to 13.0 million on the vendor review, where the graph held a flat 16.4 million to 16.6 million. When the graph sets the path and checks a JSON schema at every node, you no longer need a frontier model to manage control flow. On our rebuilt judging step, Sonnet cost half as much as Opus per 1,000 keywords, and Haiku 96.7% less.

It makes the economics predictable. Unstructured agents are quick to set up and shift the cost downstream to the reviewers who inspect transcripts and re-run failed outputs. A coded workflow takes up-front engineering to map the edges and schemas, but once deployed, it catches silent truncation, endless retry loops and missing answers in code instead of in human review. Our next measure is cost per accepted deliverable across counted runs.

The Iterative Evolution Across Runs

Moving a complex enterprise workflow from an autonomous agent to a production-grade LangGraph state machine isn't a one-and-done switch. It is an iterative engineering process, where each run exposes a specific failure mode that gets locked down in code.

In our field deployment for the national equipment rental client, judging nearly 20,000 search keywords and identifying its top 20 competitors, we went through distinct engineering iterations. Each one peeled another decision away from the LLM and encoded it in deterministic Python inside LangGraph.

IterationControl architectureWhat happenedKey engineering improvement
Baseline (October)An unconstrained Claude session driving a research skillStalled twice on usage limits (1 h 40 m of waiting); stopped with 4,484 keywords undecided; approved 7 non-competitors.Replaced model-driven control flow with an explicit 135-step LangGraph graph.
Graph Run 1 (Nov 9)135-step LangGraph graph (103 code steps, 13 AI steps, 15 automatic decision points, 4 fan-outs)Reached Gate 1 with 19,593 keywords; cut undecided keywords to 68 (0.35%) and non-competitors to 2.Fixed step order and loop limits in code; Claude could no longer skip steps or decide it was finished.
Graph Run 2 (Nov 10 to 12)The identical graph, run from scratch to test repeatabilityStopped 3 times (batch truncation, an unsaveable record, poor routing); $299.68 of Claude in all; 13.6% (about 1 in 7) of keyword decisions flipped against Run 1; 11 of 20 competitors in common.Added 3 permanent code fixes; the final stretch cleared Gate 1 in 27 minutes for $17.12.
Optimized step (1,000-keyword test)Rebuilt judging step (1 keyword per call, cached XML prompt, strict JSON schema)Repeatability rose from 87.7% to 99.1% (Opus) and 97.9% (Sonnet); 0 keywords dropped.Eliminated 350-item batches and the 851-line prompt; labeled examples cover every offering. Graph Runs 3 and 4 will use this step.

Step-by-Step Breakdown: How Each Run Improved the Graph

Baseline to Graph Run 1: Replacing Prompt Navigation with a 135-Step Graph

In the October baseline, a single Claude session read the research skill and decided which tools to run, in what order and when the job was finished. It stopped with over a third of the keywords at the gate (4,484) undecided and approved directory and retail sites like Yelp and Zillow as competitors in the final list.

The LangGraph fix:

  • We compiled the research skill into 135 fixed steps. 103 of them are plain Python code, using zero LLM tokens, that pull search data, do the counting and run the checks.
  • Conditional graph edges enforce a coded loop, at most 6 rounds, governed by a readiness check. Claude was stripped of control flow and limited to judging keyword relevance and competitors inside parallel worker nodes.

Graph Run 1 to Graph Run 2: Hardening Edge Cases in Code

When we ran the exact same graph a second time from scratch, it hit three real-world snags. Instead of asking an LLM to figure out how to recover, we turned each stop into a permanent code path in the graph:

  • Stop 1, silent batch truncation: We passed batches of 350 keywords into a node, and Claude returned 349 verdicts, quietly dropping one.
    • Code fix: The graph compares the keywords it sent with the verdicts it got back, and any missing keyword is judged in the next round.
  • Stop 2, an endless loop on an unsaveable record: One malformed keyword failed database validation, so the readiness check never passed and the loop never ended.
    • Code fix: Keywords that can't be saved no longer count against the readiness check.
  • Stop 3, prioritization blindness: On each retry, the graph judged the 400 highest-volume keywords rather than the roughly 130 blocking keywords needed to pass Gate 1.
    • Code fix: We rewrote the selection step to judge blocking keywords first. With that fix, the final pass took 27 minutes and $17.12 and cleared the readiness check in one round instead of failing six.

Graph Run 2 to an Optimized Step: Eliminating Drift Inside the Step

Graph Run 1 and Graph Run 2 agreed on 86.4% of keyword decisions, against 62.5% between the October agent run and Graph Run 1. The October comparison spans two methods run two weeks apart, so it isn't a like-for-like test, but two graph runs clearly land much closer together. Still, 13.6% of keyword decisions, about 1 in 7, flipped on identical data.

Comparing the runs showed why: inside the LLM step, we were still feeding 350 keywords at once alongside an 851-line prompt. In Run 1, Claude inferred 8 core equipment offerings for the client; in Run 2, it inferred 10. Because every keyword was judged against that shifting list, downstream verdicts drifted.

The LangGraph fix:

  • Settle the inputs once: The next graph runs settle the client's offering list once, so every worker judges against the same baseline.
  • One item per call, strict schema: We eliminated 350-item batches and fanned out one keyword per call. Each answer is checked against a strict JSON schema; across 6,000 test calls, the one response that went missing was caught by the check and re-asked automatically.
  • Prompt caching and a smaller model: With a short XML prompt holding 72 labeled examples (36 yes, 36 no) and about 93% of each prompt cached, repeatability rose from 87.7% to 99.1% on Opus 5.5 and 97.9% on Sonnet 5.5, with zero dropped keywords. Per 1,000 keywords, Opus cost $6.93, Sonnet $3.47 and Haiku $0.23.

How LangGraph Enforces Deterministic Code Paths Under the Hood

LangGraph separates control flow (Python code) from inference (bounded LLM calls) through four core mechanisms:

  • Typed state objects (TypedDict or Pydantic): Instead of passing chat transcripts from agent to agent, every node reads from and writes to a central, typed state. Each node returns only the keys it updates, and code decides what happens with them.
  • Deterministic conditional edges (add_conditional_edges): Routing decisions, such as whether to loop back for missing keywords, send a case to human review or advance to Gate 1, are made by plain Python functions that inspect state, never by asking an LLM, "Are you done yet?"
  • Node-level validation and isolated retries: When an LLM returns malformed JSON or drops a field, a validation step catches it and re-runs only that call with a clean context, preventing context bloat and drift.
  • Checkpointed persistence: Every state transition is saved by a checkpointer. If a run stops midway because of an API outage or an edge case, engineers patch the Python logic and resume from the step that stopped, without re-running or paying for completed work. That is how Graph Run 2 finished: it resumed from each stop rather than starting over.

Learning Graphs Are How You Get Reliable Agentic Workflows

An unconstrained prompt-loop agent isn't true enterprise agency. It is probabilistic improvisation. Every run starts from zero, rediscovers edge cases the hard way, burns tokens rewriting throwaway scripts and forgets rules when its context is compacted.

A learning graph, a deterministic state machine (like LangGraph or Google ADK) backed by cached ground-truth examples, typed state and automated regression tests, is what actually delivers reliable autonomous capability:

  1. Defined paths eliminate structural drift. When orchestration is compiled into a graph, the system never spends intelligence guessing which step comes next. Code handles sequencing, joins and loop limits, which kept structural execution at 100% consistency across 300 nodes, where the unconstrained setup fell to a 0% four-run pass rate.
  2. It learns from workload data without retraining the model. Instead of fragile prompt tweaks or expensive fine-tuning, the graph learns from production runs. Every edge case, like silent batch truncation, unsaveable database records or a shifting offering list, becomes a permanent rule in code or a labeled example in the step's example bank.
  3. Prompt caching turns domain knowledge into cheap memory. Pinning the fixed instructions and 72 verified examples in a cached prefix (about 93% of each prompt read from the cache) gives every bounded call the same institutional memory at a fraction of the cost. That let mid-tier Sonnet (97.9% repeatability) and Haiku (96.1%) beat the old graph step's 87.7%.
  4. Automated test gates enforce self-correction. True autonomy requires knowing an output is wrong before a person sees it. Hard-coded schema checks, ID comparisons and acceptance gates catch malformed or missing answers immediately (one missing response caught out of 6,000 calls) and retry only the step that failed.

FAQ

What is agent orchestration?

Agent orchestration is when one AI agent plans the work, starts other agents to do the pieces, and checks their results. It is quick to set up, but the orchestrating agent picks the path on every run, so two runs of the same job can take two different paths. In our 300-node tests, the prompt-only setup never passed four runs in a row, and the setup with every skill and hook on passed 2.5% of the time.

What is pass^k?

pass^k is the chance that all k repeats of the same job succeed without a person stepping in. We repeated each of 8 jobs five times and use four out of four (k=4) as our main reliability bar. It is stricter than a single-run pass rate, because one good run says nothing about the next one.

What is the difference between an agent harness and a workflow graph?

An agent harness, like Claude Code, controls what the agent is allowed to see and touch: permission rules, a sandbox, hooks. A workflow graph controls what happens next: code sets the order of the steps, and the model only makes bounded calls inside them. The harness keeps the agent safe; the graph makes it repeatable. You need both.

What is context engineering?

Context engineering is deciding exactly what goes into the model's context window at each step. Long contexts fail in predictable ways, and models get less reliable as the input grows. Our rebuilt judging step gives each call one keyword and the examples it needs, which moved the same-answer rate from 87.7% to 99.1% on Opus 5.5.

Do you need a frontier model to run AI agents in production?

Not when code sets the path. On our rebuilt judging step, Sonnet 5.5 gave the same answer twice 97.9% of the time at half the cost of Opus 5.5, and Haiku 5.5 gave the same answer 96.1% of the time for 96.7% less. We still check that agreement against hand-judged answers before choosing a model.

The Research Behind This Series

Need help putting it to work? Get in touch, or email our team at redevteam@pressfit.ai.

See where you and your competitors rank in search and in AI answers.

Get your Playbook to outrank them. Paste your URL, and we do the hard work. Your Playbook arrives with your first report within 48 hours.

Get Onboarded

Newsletter

Prefer to read? Get new posts in your inbox — no spam, unsubscribe anytime.