Cloudflare Year in Review: AI Bots Drive Traffic, Post-Quantum Encryption Surpasses 50%

AI Bot Crawling and Post-Quantum Encryption Adoption Surge

Cloudflare’s sixth annual Radar Year in Review highlights significant shifts in internet traffic patterns, particularly the impact of AI bots and the growing adoption of post-quantum encryption. The report shows a 19% yearly growth in global internet traffic, with Googlebot remaining the dominant crawler and Go-based clients doubling their share of automated API requests to over 20%.

Why This Matters

Current AI model training relies heavily on web scraping, creating a disparity between content consumption and attribution. This “crawl-to-refer” imbalance, with ratios reaching 500,000:1 for some AI platforms, threatens content creators and the open web. The increasing adoption of post-quantum encryption, now at 50% of human web traffic, is a proactive measure against future decryption of currently intercepted data, mitigating a potentially catastrophic data breach scenario.

Key Insights

• Googlebot dominance: Responsible for 4.5% of all HTML requests in 2025.
• AI Crawl-to-Refer Ratios: Anthropic’s ratio peaked at 500,000:1, highlighting aggressive content consumption without reciprocal traffic.
• Go API Client Growth: Adoption increased from 12% to 20% year-over-year, indicating growing popularity for building automated systems.

Practical Applications

• Use Case: Content creators can utilize Cloudflare Radar to identify disproportionate crawling activity from AI bots and potentially implement rate limiting or access controls.
• Pitfall: Ignoring the increasing crawl-to-refer ratios of AI bots may lead to bandwidth exhaustion and reduced revenue for content providers.

References:

• https://www.infoq.com/news/2025/12/cloudflare-2025-ai-bots/
• https://blog.cloudflare.com/cloudflare-radar-2025-year-in-review/