pressfit.ai Research
Empirical AI Research Benchmark | Deterministic Compute vs Probabilistic Agents

Stop Prompting and Start Engineering

Why enterprise autonomous AI agents hit an architectural wall, and how replacing probabilistic process control with deterministic graph workflow engines stabilizes token spend, restores execution accuracy, and eliminates output volatility.

By Anthony Skinner, CEO and Co-Founder, and the Pressfit.ai Engineering Team

Variance Reduction

85.7%

7 distinct ranking orders dropped to 1 in constrained graph execution.

Execution Speed

45% Faster

2.8 min average run time for engine vs 5.1 min for unconstrained Claude.

Token Savings

3x - 27x

Fewer tokens on script-heavy jobs by replacing dynamic code churn with static execution.

300-Step Accuracy

97.5%

Graph engine accuracy vs 70% for prompt-only agents over extended task horizons.

Tool Creation Options Evaluated in Research

The benchmark study analyzed four distinct engineering methodologies for building AI tooling and managing enterprise workflows. Here is how each option constructs tools, manages context, and enforces workflow execution.

Option 1 Highest Variance

Unconstrained Prompting (Dynamic Ephemeral Code)

The LLM is prompted to write ephemeral execution scripts (Python/Bash) on the fly, execute them in a sandbox, and dynamically recover from errors during multi-turn chats.

Tool Creation: Dynamic AI Code Generation
Process Control: Model Decides Next Step
Output Variance: 7 Ranking Orders / 10 Runs
Token Volatility: 3.4x Swing Between Runs
Option 2 Moderate Control

Structured Markdown Templates & Prompts

Human engineers write static, highly detailed system runbooks and markdown templates. The LLM strictly formats outputs according to specified markdown schemas.

Tool Creation: Static Prompt Schemas
Process Control: Model Follows Text Guide
Output Variance: 1 Ranking Order
Token Volatility: High Context Overhead
Option 3 High Tool Security

Skills with Programmatic Hooks (Tool Calling)

Tools are exposed as typed function schemas (JSON/OpenAPI) with programmatic pre/post execution validation hooks. The model selects when to invoke tools.

Tool Creation: JSON Schema APIs + Code Hooks
Process Control: Model Invokes Registered APIs
Output Variance: 1 Ranking Order
Token Volatility: Moderate Stability
Recommended
Option 4 Deterministic Control

Deterministic Graph Process Engine (pressfit.ai)

Workflow orchestration is compiled as an explicit state machine DAG in code. Pre-written code handles state, tool execution, API calls, and retry loops—calling LLMs strictly for semantic judgment.

Tool Creation: Compiled Code Graph Nodes
Process Control: Deterministic State Machine
Output Variance: 1 Ranking Order
Token Volatility: <1.3% Token Variance

Measured Results Across Paradigms

Real scores from our tests, as percentages. The first four come from the 40-run short test; blind quality is the average score out of 5. Option 2 was not run at 300 steps.

Executive Paradigm Matrix: Architectural Selection Guide

How to select the right tool creation approach based on your enterprise SLA requirements, compliance needs, and task horizon complexity.

Methodology Process Control 300-Step Trajectory Decay Token Spend Volatility Enterprise SLA Suitability Ideal Use Case
Option 1: Unconstrained Prompts Probabilistic (Model decided) High (70.0% accuracy) Extreme (3.4x swing) Unsuitable for production Ad-hoc research, interactive single-user exploration
Option 2: Markdown Templates Guided text schemas Not tested at 300 steps High prompt overhead Low-risk internal tools Structured document formatting, simple reporting
Option 3: Tool Calling Hooks Hybrid (Model selects API) Moderate (85.0% accuracy) Controlled tool inputs Bounded task automation Single API integrations, isolated transactional tools
Option 4: Graph Engine (pressfit) Deterministic State Machine Minimal (97.5% accuracy) <1.3% Volatility Mission-Critical Enterprise SLA Multi-step enterprise workflows, compliance-critical pipelines

1. Short-Horizon Benchmark: Variance & Speed Analysis

In a controlled short-horizon test using 8 fictional competitors, 42 frozen source documents (~21,000 words), and 24 planted facts across 40 scored runs (10 runs per approach), pressfit.ai evaluated four paradigms: Claude alone, structured markdown templates, skills with programmatic hooks, and our business process engine.

Output Variance: Distinct Ranking Orders

Number of unique competitive ranking results generated across 10 identical runs (lower is better).

Key Takeaway: Unconstrained Claude yielded 7 different ranking orders on identical input data. All constrained setups forced total output determinism (1 distinct order).

Execution Speed: Average Minutes per Run

Time required to complete the complete analytical task across 10 runs (lower is faster).

Key Takeaway: Moving orchestration into code reduced execution time from 5.1 minutes to 2.8 minutes—a 45.1% processing speedup over unconstrained LLM execution.

2. Architecture Breakdown: Visualizing a 135-Step Deterministic DAG

Rather than letting an unconstrained LLM navigate step sequencing dynamically, the Graph Engine compiles business logic into an explicit Directed Acyclic Graph (DAG). Pre-compiled code nodes wrap around micro-LLM calls strictly for localized judgment.

Graph Orchestration Visual Workflow Map

Code Node
1. Input Ingestion

0 LLM Tokens Spent

Parallel Fan-Out
2. Itemized Workers

4 Fan-Outs, One Agent per Item

Code Node
3. Gate Validator

Automated Retry Logic

103
Deterministic Code Steps

Zero-token computations, joins & state validations.

13
Bounded AI Calls

Isolated LLM invocations for synthesis & semantic judgment.

15
Automated Stops

Internal queries resolved by pre-compiled logic.

4
Parallel Fan-Out Agents

Parallel worker threads per discrete itemized state.

3. Multi-Step Trajectory Decay: 20 Steps vs 300 Steps

Research from Microsoft Research and Salesforce confirms that multi-turn model interactions drop performance by an average of 39%, driven by unreliability rather than lost skill. To measure this, we tested task accuracy across short (20 steps) and extended (300 steps) execution horizons.

Execution Accuracy Across Workflow Length

Percentage of task outputs verified correct at 20 steps vs 300 steps.

20 Steps 300 Steps

Prompt-Only Agent Decay

Accuracy fell from 82.5% to 70.0% (30% failure rate). Trajectory drift compounds with every turn.

Template + Hooks Decay

Accuracy fell from 95.0% to 85.0% (15% failure rate). Output checks slowed but could not stop drift.

Deterministic Graph Engine

Accuracy held at 97.5% at 300 steps (2.5% error rate). Code maintains control flow regardless of length.

4. Long-Horizon Stress Tests & Token Volatility

Evaluating extended operational runs reveals a major hidden cost of autonomous agents: dynamic code generation causes extreme token consumption variance.

Payment Run: Token Consumption Volatility

Token usage across 6 identical runs of a 70-step payment job with 25 injected faults.

Insight: Unconstrained Claude dynamically rewrote code every run (344-552 lines), causing tokens to swing 3.4x (1.7M to 5.7M). The Graph Engine ran static 162-line code, reducing tokens to 0.56M - 0.63M (up to 10x savings).

Vendor Review: Token Budget Range

Min vs Max token spend (in millions) across 5 runs of a 64-vendor review (480,000 words).

Insight: Plain brief token usage more than doubled, from 5.6M to 13.0M tokens between runs. The Engine consumed a stable 16.4M–16.6M tokens (<1.3% variance), enabling reliable enterprise financial budgeting.

5. Field Validation: National Equipment Rental Production Case Study

Comparing live production deployment metrics for a national equipment rental client: October (Claude driving skill directly) vs November (LangGraph Business Process Engine).

Undecided Keyword Rate

36.1% (Oct) → 0.35% (Nov)

October left 4,484 keywords unclassified. Graph engine reduced dropped items to 68.

Irrelevant Approved Domains

7 Sites (Oct) → 2 Sites (Nov)

October approved non-rental entities (Yelp, Zillow, MapQuest). Engine filtered hallucinations.

6. Financial Framework: Cost per Accepted Deliverable ($C_{AD}$)

Tracking API token billing alone is misleading. The true financial metric for enterprise AI is Cost per Accepted Deliverable ($C_{AD}$), which factors in output rejections and mandatory human re-run overhead:

CAD = Crun / (1 - predo)

Where Crun = Direct execution cost (Tokens + Compute), and predo = Failure / Rejection probability requiring re-run.

Interactive Scenario Simulator

Base Execution Run Cost ($C_{run}$): $10.00
Unconstrained Agent Rejection Rate (predo): 30%
pressfit.ai Engine Rejection Rate (predo): 5.0%

Effective Financial Comparison

Unconstrained Agent Realized $C_{AD}$: $14.29
pressfit.ai Engine Realized $C_{AD}$: $10.53
Net Efficiency Savings per Output: 26.3%

Enterprise AI Workflow Health Checklist

Evaluate whether your engineering team is hitting the probabilistic wall. Check the items below to diagnose risk in your AI architecture: Note: the percentages in this checklist are rules of thumb, not results from our testing.

Strategic Synthesis for Enterprise Leaders

1. Stop Relying on Prompts Alone

Prompt engineering alone cannot guarantee business control flow. LLMs are probabilistic inference engines; asking them to manage multi-step processes leads to inevitable trajectory drift.

2. Encode Methodology in Code

Use graph-orchestration engines to hold explicit state, manage retry logic, and route tasks. Restrict LLMs to localized semantic tasks where judgment is required.

3. Measure Cost per Accepted Output

Stop tracking simple token spend. Factor in failure rates, re-runs, and human inspection overhead to optimize true enterprise operational efficiency ($C_{AD}$).

Source Data: pressfit.ai Research Benchmarks & Enterprise Field Deployment Reports Back to Top ↑