Stop Prompting and Start Engineering (Part 2)
Why prompt-driven agent orchestration breaks down at scale: holding Anthropic's updated Claude Code playbook up against 300-node pass^k reliability curves and a live 19,593-keyword enterprise production run.
The graph (G and G1) holds 1.000 at k=4, against 2.5% for A-Full and 0% for A0.
Per 1,000 keywords: $6.93 on Opus 5.5 to $0.23 on Haiku 5.5, still beating the old graph step's repeatability (96.1% vs. 87.7%).
Opus 5.5 on the rebuilt step (97.9% on Sonnet 5.5 at half the cost): one keyword per call, 72 examples, about 93% of the prompt cached.
From 36.1% (4,484 keywords, desktop agent run) to 0.35% (68 of 19,593) in the LangGraph engine.
The Math: Why Prompt-Driven Control Falls Apart at 300 Nodes
Anthropic's updated Claude Code playbook runs dozens of parallel sessions managed by an orchestrator session. Frontier models like Opus 5.5 solve single coding tasks faster, but solving a task once isn't running a repeatable business workflow. Across 8 jobs repeated 5 times each (40 runs per bar; 160 graph runs across G and G1), chaining probabilistic decisions drives multi-run reliability toward zero.
First-Try Code-Check Pass Rate (S_det)
Share of runs passing every deterministic code check on the first try. At 300 nodes, prompt-only A0 fails 60% of runs, and A-Full, with every skill, sub-agent and hook turned on, still fails 45%.
Key takeaway: Skills and hooks lift 300-node S_det from 40% to 55%. Compiling the workflow into a graph holds it at 100%.
Repeatability Curve (pass^k, k = 1 to 5)
The chance that all k of 5 repeats succeed without a human rescue. At 300 nodes, A-Full falls to 2.5% at k=4 and 0% at k=5. Four out of four is the primary bar.
Precision note: values at k=2 and k=3 are exact ratios across the 8 jobs (45/80, 33/80, 21/80, 13/80), rounded to three places. A-Full tested skills, sub-agents and programmatic hooks, not the Claude Code Projects orchestrator from the talk.
Harnessing Isn't the Same as Engineering the Process
A harness like Claude Code answers what the agent is allowed to do, with permission rules, a sandbox and a classifier. A graph answers what happens next, and engineers exactly what enters the context window at every step, guarding against long-context poisoning, distraction, confusion and clash.
Inside the 135-Step Graph
Only 13 of 135 steps (9.6%) are AI steps. Code steps, automatic decision points and fan-outs make up the rest.
Coded guardrails: conditional edges cap the judging loop at 6 rounds, governed by a readiness check written in Python.
Safety Boundary vs. a Fixed Path
Walled off from the rest of the machine. Backed by deny rules (checked first) and the auto mode classifier, which blocked 89% of dangerous commands in Anthropic's study; human reviewers caught about 14%.
Claude's own file tools, MCP servers and hooks run outside the sandbox. Even inside a safe harness, an agent can take two different paths on identical inputs.
Context Engineering in the Judging Step
Same answer twice rose from 87.7% to 99.1%.
State lives in the graph, not the prompt. The next runs settle the client's offering list once, so no call has to rediscover it.
Each call sees one keyword and 72 labeled examples (36 yes, 36 no) instead of 350 keywords, with about 93% of the prompt read from cache.
An 851-line skill prompt became a short, pinned XML system prompt with strict decision rules.
Every keyword runs in its own call and returns strict JSON checked by code: 1 missing answer caught across 6,000 calls (0.017%).
Field Update: National Equipment Rental Client (Gate 1)
The job: judge nearly 20,000 search keywords (19,593 at Gate 1 in Graph Run 1) and bring the top 20 online competitors to Gate 1. Moving from an autonomous desktop Claude session (October) to our 135-step LangGraph engine (November) cut undecided keywords by 98.5%, and a repeat run exposed three edge cases that we fixed permanently in code.
Gate 1 Keyword Coverage and Competitor Quality
October's agent run stalled twice on usage limits (1 h 40 m idle) and left 4,484 keywords undecided (36.1%). Graph Run 1 decided 19,525 of 19,593 keywords (99.65%) and cut non-competitors in the top 20 from 7 (Yelp, Zillow, U-Haul, Lowe's) to 2.
Graph Run 2: What It Took to Finish ($299.68 total)
Re-running the identical graph from scratch (Nov 10 to 12) surfaced three edge cases. Run 2 worked from 19,544 keywords against Run 1's 19,593, and 99.7% were the same in both. Because LangGraph saves a checkpoint at every step, we patched the code and resumed without paying again for finished work:
Claude got 350 keywords and returned 349 verdicts. Fix: the graph compares sent and returned IDs and judges any missing ones next round.
One malformed keyword could never be saved, so the readiness check never passed. Fix: unsaveable keywords no longer count against readiness.
Each round judged the 400 highest-volume keywords instead of the ~130 blocking ones. Fix: judge blocking keywords first.
Passed the readiness check in one round instead of failing six. (The stop amounts sum to $299.65 against $299.68 because Stop 1 is rounded.)
Why drift still happened: Runs 1 and 2 batched 350 keywords with an 851-line prompt. Claude inferred 8 core offerings in Run 1 and 10 in Run 2, flipping 13.6% (about 1 in 7) of keyword decisions.
Rebuilding the Judging Step: 99.1% Repeatability, 96.7% Lower Cost
With 350-keyword batches gone, a pinned XML prompt holding 72 labeled examples, strict JSON and about 93% prompt caching, we ran the same 1,000 keywords twice on three Claude tiers ($21.26 in total). On a bounded step, smaller models beat the old graph step's repeatability at a fraction of the price.
Same Answer Twice and Agreement with Opus 5.5
Every optimized tier rose from the old graph step's 87.7% to 96.1% to 99.1%, with 0 keywords silently lost.
Method note: the old step's agreement with Opus spans 88% to 92% (plotted at 90%). Haiku 5.5 says no more often than Opus 5.5; a hand check against ground truth decides the production tier.
Cost per 1,000 Keywords (USD)
Sonnet 5.5 agrees with Opus 5.5 on 95.2% of verdicts and gives the same answer twice 97.9% of the time at 49.9% lower cost ($3.47 vs. $6.93). Haiku 5.5 costs 96.7% less ($0.23).
The shift: instead of paying frontier models to supervise other frontier models for hours, caching and typed schemas let mid-tier and lightweight models run reliably in the cloud.
Generative Orchestration vs. Coded Workflow Graphs
Enterprise platforms ship both styles: generative orchestration (Anthropic's multi-session playbook, Microsoft Copilot Studio) and explicit graph runtimes (LangGraph, Google ADK 2.x's Workflow and FunctionNode, Microsoft Agent Framework 1.0's workflows). In multi-turn conversations, Microsoft and Salesforce researchers (arXiv:2505.06120) found an average 39% performance drop across 15 LLMs, driven mostly by a 112% rise in unreliability.
| Dimension | Generative orchestration (Claude Code playbook, Copilot Studio) | Coded workflow graphs (LangGraph, Google ADK 2.x, Agent Framework 1.0) |
|---|---|---|
| Control flow and routing | Probabilistic: the orchestrator picks tools, topics and sub-agents from descriptions and conversation history. | Deterministic: Python edges and conditional routes decide what runs next without asking an LLM. |
| Verification and retries | Transcript review, CI after the work, and model-driven retries that risk context bloat. | Schema checks, ID reconciliation, quarantine rules and human approval gates built into the steps. |
| Token and compute profile | Volatile: a 3.4x token swing (1.7M to 5.7M) on the 70-step test; frontier models supervising frontier models for hours. | Predictable: a flat 16.4M to 16.6M band on the 64-vendor review; only 9.6% of steps call an LLM; Sonnet and Haiku viable. |
| The human's role at scale | A full-time manager reviewing dozens of parallel bot outputs and pull requests. | An architect who reviews the process once and approves deliverables at coded gates. |
Why Learning Graphs Deliver Reliable Agentic Behavior
Turning probabilistic models into auditable systems without retraining the model:
Code owns sequencing, joins and loop limits, holding structural execution at 100% across 300 nodes.
Production edge cases, like batch truncation, malformed rows and shifting baselines, become permanent rules in code.
Pinning 72 verified examples (about 93% cached) let Haiku 5.5 (96.1%) beat the old graph step (87.7%).
Typed JSON checks catch missing or malformed fields immediately and retry only the one call that failed.