State of AI Brand Crawling — March 2026
March 2026 Baseline
AI crawling is the mechanism through which Brand Authority signals are collected — and the volume is growing fast.
111,286
Total requests/week
19,454
AI crawler requests
17.5%
AI share of traffic
6,730
Search engine crawlers
4.3:1
AI-to-Google ratio
5,829
Brands in directory
1,604
New profiles this week
8,008
AI representation score changes
9
Brands self-claimed
This is month one. Future reports will include month-over-month trends.
Platform Breakdown
| Platform | Total Requests | Primary Crawler | % of AI Traffic |
|---|---|---|---|
| OpenAI | 10,816 | GPTBot/1.3 (8,159) | 55.6% |
| Anthropic | 4,669 | ClaudeBot/1.0 (4,235) | 24.0% |
| Amazon | 4,366 | Amzn-SearchBot (2,885) | 22.4% |
| Perplexity | 1,699 | PerplexityBot/1.0 | 8.7% |
| ByteDance | 15 | Bytespider | 0.1% |
Note: percentages exceed 100% because some requests serve multiple purposes across platform variants.
What the Crawlers Are Actually Requesting
AI crawlers don't uniformly crawl the directory. Some patterns:
- •Brand profile pages receive the highest crawl frequency — these are the primary entity pages that AI models use to build brand representations.
- •Category pages receive moderate crawl frequency, but our living taxonomy system returns 503s for categories still being computed — AI crawlers handle this gracefully and retry.
- •The brand directory index receives consistent crawls from all platforms — it's the discovery page that leads crawlers to individual profiles.
- •Blog content receives lower AI crawler frequency but higher search-retrieval frequency (OAI-SearchBot, Claude-SearchBot).
Directory Growth
The directory grew by approximately 1,000 brands in the past 3 weeks, reaching 5,829 total profiles.
1,604
Profiles generated this week
642
Peak day (March 25)
2,443
Cap-blocked (awaiting processing)
9
Self-claimed brands
Organic traction: 9 brands have self-claimed their profiles with zero outbound sales. They found their profile in the directory, reviewed the AI representation data, and claimed ownership.
What This Means for Brands
The headline takeaway from March 2026: AI models are the primary non-human consumers of brand information on the web. On our directory, AI crawlers outpace Googlebot by 4.3:1. OpenAI alone sends more requests than all search engines combined.
For brand marketers: the data you serve to AI crawlers is no longer a secondary concern. It's the primary interface between your brand and the AI-mediated decision layer. If your robots.txt blocks AI crawlers, if your structured data is incomplete, if your authoritative sources disagree with each other — that's not a hypothetical future problem. It's a measurable, present-tense problem that we can quantify in our directory right now.
Next month's report will include trend data comparing April to March.
Related
Data source & limitations: Cloudflare server logs and Optimly directory data, week of March 22–28, 2026. AI crawler classification is based on user-agent strings. Brands in the directory represent a subset of the total market. Monthly comparisons will begin in the April 2026 report.