Skip to main content
AI-visibility

Beyond Analytics: What Firewall and Edge Logs Reveal About AI Demand

Anthony Skinner, CEO and Co-Founder, and the Pressfit.ai Engineering Team14 min read

Google Analytics told us AI barely sends Pressfit.ai any traffic. Our firewall told a different story: AI and search crawlers hit the site 2.71 times more often than people did, and 187 of those requests were live fetches triggered by someone asking an AI a question. The short version: client-side analytics can't see most AI demand, so you have to read it at the edge, separate live human fetches from background crawling, and tie it back to pipeline.

New here? Start with Part 1: Measuring AIO and GEO.

Prefer the numbers at a glance? View the AI Search Attribution Gap infographic.

Why Google Analytics Misses AI Traffic

If your marketing agency is still hosting webinars about "AI attribution" by staring at referral traffic inside Google Analytics, you are getting sold a rearview-mirror report.

Client-side analytics, the tracking scripts that run in a visitor's browser, only measure human sessions that allow those scripts to fire and keep the HTTP referrer, the header that says where a visit came from. Today, privacy browsers block scripts, mobile LLM apps strip referrers, and users research inside AI interfaces without clicking blue links. To see what is actually happening, technical marketing teams drop down a layer to web application firewall (WAF), CDN and edge network logs.

The Pressfit.ai 30-Day Telemetry Reality Check

Look at our own 30-day telemetry at Pressfit.ai. If you only looked at our traditional web analytics, you would see about 6,500 human visitors with almost no visible AI attribution:

  • Direct traffic: 6,460 visits (99.38%), absorbing dark social, stripped LLM app referrers and cookie-blocked browsers.
  • Syndicated Google search: 20 visits (0.31%).
  • Grok referral: 10 visits (0.15%), the only major LLM that passed a clean referrer in our report.
  • Paid social (Facebook ads): 10 visits (0.15%).

A surface-level marketer looking at that dashboard would conclude that ChatGPT, Claude and Meta AI aren't sending traffic, and might even question whether creating structured content for LLMs is worth the effort.

Then you open the firewall and network logs. Over that exact same 30-day window, core AI and search crawlers hit Pressfit.ai 17,630 times, 2.71 times our total human visitor count (and 22,692 total bot requests when you include AdsBot's 5,002 and 60 from secondary Google services). Even more telling, AI and answer-engine bots alone made 7,321 requests, 1.13 times our human traffic.

AI Crawlers in Our Edge Logs: 7,321 Requests

Every one of these crawlers represents an AI platform, retrieval API or autonomous agent ingesting Pressfit.ai, yet almost none of them pass a clean referrer to Google Analytics:

AI provider30-day requestsShare of AI botsUser agents identified at the edgeRole in AI discovery
Meta AI2,28031.1%Meta-WebIndexer, Meta-ExternalAgentOur top AI crawler by volume: indexing (1,764 Meta-WebIndexer) and fetching web answers (516 Meta-ExternalAgent) for Meta AI across WhatsApp, Instagram and Facebook.
OpenAI / ChatGPT1,78924.4%OAI-SearchBot, GPTBot, ChatGPT-User, GPT Actions, OperatorSearch indexing (1,401) alongside live user-triggered lookups (127) and autonomous agent actions (129).
Amazon1,23416.9%AmazonbotFeeds Alexa answers, Rufus shopping retrieval and AWS Bedrock knowledge tools.
Parallel6138.4%ShapBotAI web search and retrieval infrastructure behind real-time agent APIs.
Anthropic / Claude5807.9%ClaudeBot, Claude-UserModel ingestion (523) plus live, human-prompted page fetches (57) during vendor research.
Exa4516.2%ExaSearchBot, ExaUserNeural search API behind developer-built AI apps, deep-research agents and enterprise copilots.
Perplexity3464.7%PerplexityBotAnswer-engine indexing and citation retrieval.
Common Crawl170.2%CCBotOpen-web corpus used in open-source and enterprise LLM training.
DuckDuckGo AI110.2%DuckAssistBotPrivacy-first AI answers.

Search Crawlers That Feed AI Overviews: 10,309 Requests

Traditional search crawlers remain the foundational feeding tube for Google AI Overviews, Microsoft Copilot and Apple Intelligence. We leave out paid-ad verification bots like AdsBot, so paid media sweeps don't inflate organic GEO counts:

Search crawler30-day requestsShare of search crawlersEcosystem powered
Googlebot6,48662.9%Core Google Search index and Google AI Overviews / AI Mode grounding
Bingbot1,39313.5%Bing Search, Microsoft Copilot and third-party LLM search APIs
GoogleOther1,08710.5%Research, specialized Google indexes and Gemini grounding
PetalBot7046.8%Huawei Petal Search and mobile AI assistants
Baiduspider4053.9%Baidu Search and the ERNIE AI ecosystem
DuckDuckBot1051.0%DuckDuckGo search and private AI chat grounding
Applebot790.8%Siri, Spotlight and Apple Intelligence web answers
YandexBot500.5%Yandex Search and Alice AI

Live User Fetches vs. Background Crawling

One of the biggest mistakes teams make when they first open edge logs is treating every AI bot hit as a live human prompt. If OAI-SearchBot hits your site 1,401 times, that does not mean 1,401 people asked ChatGPT about your company. It means OpenAI's search indexer crawled 1,401 pages to refresh its retrieval index.

To turn network telemetry into an executive-ready signal, split AI bot requests into live user-triggered fetches (a person or an active user workflow triggered a lookup right now) and background indexing and model training (automated fleet sweeps):

Signal (Pressfit.ai 30-day edge logs)Intent categoryWhat it proves
ChatGPT-User: 127Live, user-triggeredThe strongest OpenAI signal of real people asking ChatGPT a question that required fetching your URL to verify facts or compare vendors.
Claude-User: 57Live, user-triggeredThe strongest Anthropic signal of user-requested retrieval during an active Claude session.
ExaUser: 3Live, user-triggeredReal-time, user- or app-triggered retrieval through Exa's neural search API.
OAI-SearchBot: 1,401Search indexingOpenAI keeping ChatGPT Search citations current, not 1,401 individual prompts.
Meta-WebIndexer: 1,764Search indexingMeta AI search indexing (alongside 516 Meta-ExternalAgent hits), not direct prompt volume.
PerplexityBot: 346Search indexingAnswer-engine crawling; confirms index eligibility, not exact question or citation counts.
ClaudeBot: 523; GPTBot: 132Background trainingFoundation model training collection, not direct evidence of live buyer prompting.

Why this distinction matters: Across the user agents above, 187 requests (4.30%) were verified, live user-triggered fetches (127 ChatGPT-User, 57 Claude-User and 3 ExaUser), on top of 129 autonomous agent actions (127 GPT Actions and 2 Operator). The remaining 4,166 requests (95.70%) were search indexing (3,511) and background training (655). Google Analytics showed zero referrals from ChatGPT, Claude or Exa, yet our edge logs confirm 187 live, human-initiated retrieval events on our pages in 30 days.

A note on telemetry hygiene. Self-declared user-agent strings can be spoofed by scrapers, so mature edge pipelines validate high-volume bots against official IP lists, reverse DNS or cryptographic Web Bot Auth signatures (IETF RFC 9421), which Exa, Parallel's ShapBot and OpenAI's ChatGPT agent support. They also check for HTTP 200 status codes, so silent 403 WAF blocks or 499 cache timeouts don't masquerade as healthy indexing. And a bot hit proves retrieval, not a citation or a sale.

How to Measure Modern Search: The Four-Layer Attribution Stack

If your agency still measures SEO and GEO only by referral traffic in an analytics dashboard, it is ignoring 17,630 or more machine retrieval events on your domain every month. Real SEO and GEO measurement connects all four layers of data, using tools most marketing and engineering teams already own.

Layer 1: Deterministic Content and Site Engineering

  • What it measures: Crawl readiness, semantic HTML delivery, entity coverage and query fan-out alignment.
  • How to do it: Render priority commercial pages on the server (never behind client-only JavaScript shells), put a concise, direct answer in the first 30% of the page, consolidate fan-out subtopics into authoritative hub pages instead of thin doorway pages, and refresh pages quarterly so they don't suffer citation decay.

Layer 2: Edge and Firewall Telemetry

Separate bot fetches from user fetches at the edge.

Cloudflare (AI Crawl Control):

  1. In your Cloudflare dashboard, select your domain and open AI Crawl Control.
  2. In the Crawlers tab, filter by category (Search, Training, Agent or user-triggered) and operator (OpenAI, Anthropic, Meta, Perplexity, Exa) to separate OAI-SearchBot from ChatGPT-User.
  3. Check the status code column to make sure WAF rules aren't sending 403 blocks or 429 rate limits to legitimate search or user-triggered bots, and that your crawler rules match your visibility goals.
  4. Optional: stream raw HTTP logs with Analytics & Logs → Logpush to BigQuery, S3 or Datadog, grouped by user agent and response status.

AWS CloudFront or AWS WAF:

  1. Turn on AWS WAF Bot Control for your distribution or load balancer.
  2. Query your WAF log group in CloudWatch Logs Insights, filtering the User-Agent request header to count ChatGPT-User, Claude-User, OAI-SearchBot and GPTBot.

Vercel or an Nginx origin:

  • Vercel: open Observability (or Firewall → Bot Management) and filter by user agent for ChatGPT-User, Claude-User and OAI-SearchBot.
  • Nginx: run a one-line user-agent count against your access log. CDN-cached hits only appear at the edge, so expect your origin to undercount.
grep -E "ChatGPT-User|Claude-User|OAI-SearchBot|GPTBot|ClaudeBot|Meta-WebIndexer|PerplexityBot|ExaUser" /var/log/nginx/access.log | awk -F'"' '{print $6}' | sort | uniq -c | sort -nr

Layer 3: GA4, Search Console and Direct-Lift Correlation

Since May 13, 2026, GA4 has a built-in AI Assistant default channel (medium = ai-assistant) covering ChatGPT, Gemini, DeepSeek, Copilot and Grok. It is not retroactive and it leaves out Perplexity.

  1. See the built-in traffic: go to Reports → Acquisition → Traffic acquisition, set the primary dimension to Session default channel group, and add Session source / medium as a secondary dimension.
  2. Build a fuller channel: go to Admin → Data display → Channel groups → Create new channel group, and add a channel called AI Assistants (Full) where Session source matches this regex. Drag it above Referral and save.
chatgpt\.com|chat\.openai\.com|claude\.ai|gemini\.google\.com|perplexity\.ai|grok\.com|x\.ai|copilot\.microsoft\.com|deepseek\.com|meta\.ai
  1. Correlate direct lift: open Reports → Engagement → Landing page, filter to Direct, and watch for articles (not the homepage) whose Direct sessions jump right after ChatGPT-User or Claude-User spikes at the edge.

Google Search Console: open Performance → Search results, click + New → Search appearance, and filter by AI Overviews or AI Mode (AI_MODE in the API) to track generative visibility alongside classic click-through.

Layer 4: CRM and Self-Reported Attribution

Add a required "How did you first hear about us?" field to every demo, contact and trial form. Set up a CRM workflow (HubSpot, Salesforce) that tags free-text answers mentioning ChatGPT, GPT, Claude, Gemini, Perplexity, Grok or AI into an AI Discovery (Self-Reported) source, and join lead IDs back to GA4 Direct conversions. That ties unattributed AI discovery to closed revenue.

At Pressfit, we don't rely on guesswork, manual prompting or broken cookie reports. We engineer the deterministic playbooks that get your brand retrieved by AI engines, and connect the network, analytics and pipeline telemetry to prove it works. Get your Playbook.

FAQ

How do I see AI crawler traffic on my website?

Read your edge, CDN or firewall logs, not just Google Analytics. Cloudflare AI Crawl Control, AWS WAF Bot Control with CloudWatch Logs Insights, Vercel Observability or a grep of your Nginx access log will show requests by user agent. On Pressfit.ai that view showed 7,321 AI and answer-engine requests in 30 days.

What is the difference between ChatGPT-User and OAI-SearchBot?

ChatGPT-User fetches a page live because a person asked ChatGPT something that needed it. OAI-SearchBot crawls pages to keep ChatGPT Search's index current. In our logs, 127 ChatGPT-User requests were live fetches, while 1,401 OAI-SearchBot requests were indexing, not prompts.

Why doesn't Google Analytics show ChatGPT or Claude referrals?

Mobile AI apps, logged-in sessions and copied links often drop the referrer, so those visits land in Direct. GA4's AI Assistant channel only catches sessions that keep a recognized referrer. Our GA4 showed zero ChatGPT or Claude referrals while our edge logs recorded 184 live fetches from them.

Can AI crawler user agents be faked?

Yes. Validate high-volume bots against the official IP lists, reverse DNS or Web Bot Auth signatures (IETF RFC 9421), and check status codes so blocked or timed-out requests aren't counted as healthy indexing.

How do I track AI traffic in GA4?

Check the built-in AI Assistant channel under Traffic acquisition, then build a custom channel group with a regex that adds Perplexity, Claude and Meta AI, and place it above Referral. Pair it with direct-lift analysis on landing pages that get live AI fetches.

The Research Behind This Post

Need help putting it to work? Get in touch, or email our team at redevteam@pressfit.ai.

See where you and your competitors rank in search and in AI answers.

Get your Playbook to outrank them. Paste your URL, and we do the hard work.

Get Onboarded

We love helping marketers like you.

Sign up for the Deterministic Compute newsletter.

Need help building it? See our deterministic AI engines

Get your Playbook.

See where you and your competitors rank in search and in AI answers. Need it built? We build deterministic AI engines for your business.