Optimizing llms.txt: Avoiding Common Anti-Patterns for AI Crawlers

The five anti-patterns

Engineer Ken Imoto audited 30 production llms.txt files from industry leaders like Stripe, Vercel, and Anthropic. He discovered that 24 of the 30 files exhibited at least one of five recurring technical failures.

Why This Matters

While adoption of the llms.txt standard is growing—with some estimates citing 844,000 sites as of May 2026—implementation quality is lagging. Technical contradictions between robots.txt and llms.txt, combined with a failure to provide Markdown versions of content, create a gap where AI agents can find a page but cannot efficiently parse its data within context window budgets.

Key Insights

• Adoption scale: A March 2026 SE Ranking study found roughly 10% adoption across 300,000 domains.
• Context budget constraints: The recommended file size is 10KB to ensure LLMs can read the directory without exhausting context windows needed for the actual query.
• Parsing efficiency: Using the .md companion pattern (as seen with Stripe) allows crawlers to access clean Markdown instead of JavaScript-heavy HTML.
• Maintenance risk: Many files suffer from ‘staleness,’ containing 404 links or outdated product names because they are hand-curated rather than automated.

Practical Applications

References:

• https://dev.to/kenimo49/i-audited-30-llmstxt-files-in-the-wild-5-anti-patterns-are-already-forming-18h4