Free AI.txt & LLMs.txt Validator & AI Crawler Checker
Validate AI.txt, LLMs.txt, and llms-full.txt files instantly. Detect syntax errors, crawler compatibility issues, GEO readiness, AI discoverability, and optimize your website for modern AI search engines—all inside your browser.
Your ai.txt file is specification compliant and optimized for AI crawlers.
All AI.txt, LLMs.txt, and Robots.txt validation, scoring, and auto-fix logic run 100% locally inside your web browser.
No policy text, domain credentials, or site parameters are uploaded to external tracking servers. Enterprise safe.
1. What is AI.txt?
AI.txt is an open web specification hosted at the root directory of a domain (https://yourdomain.com/ai.txt) that enables website owners, publishers, and enterprise engineering teams to specify explicit permission policies for Artificial Intelligence (AI) web crawlers, data harvesters, and foundation model pretraining bots.
As artificial intelligence models require vast amounts of text and code data for neural network pretraining, web creators need a fine-grained, machine-readable standard to declare which AI agents may crawl their content, which paths are disallowed, and where commercial licensing terms are documented.
2. What is LLMs.txt?
LLMs.txt is a standardized Markdown documentation index hosted at https://yourdomain.com/llms.txt that provides Large Language Models (LLMs), AI assistants, and Retrieval-Augmented Generation (RAG) agents with structured, high-signal documentation links and concise project summaries.
While traditional HTML web pages contain navigation menus, sidebars, cookie banners, and visual layouts designed for human eyes, LLMs.txt strips away HTML noise to deliver clean, token-efficient Markdown indexes optimized for AI context window consumption.
3. What is llms-full.txt?
llms-full.txt is an expanded companion file to LLMs.txt hosted at https://yourdomain.com/llms-full.txt. Whereas llms.txt provides a concise summary and curated list of links, llms-full.txt consolidates the complete text of an entire documentation library or project codebase into a single, comprehensive Markdown document.
Modern frontier LLMs featuring context windows of 128k to 2M tokens can ingest llms-full.txt in a single prompt call, giving AI agents complete contextual knowledge of your product architecture, API definitions, and code examples.
4. AI.txt vs robots.txt
Comparison between traditional search indexing manifests and modern AI crawling policy manifests:
| Feature / Purpose | robots.txt (Legacy Search) | AI.txt (AI Training & Licensing) |
|---|---|---|
| Primary Objective | Control search engine SERP indexing | Manage AI pretraining & LLM data harvesting |
| Target Crawlers | Googlebot, Bingbot, YandexBot | GPTBot, ClaudeBot, Google-Extended, CCBot |
| Impact on Organic Search | Direct impact on Google SERP rankings | Zero impact on Google organic search rankings |
| Licensing & Terms Support | Not Supported | Supported via License: & Policy: directives |
| File Location | /robots.txt | /ai.txt |
5. AI.txt vs LLMs.txt
Understanding the distinction between permission policy files and content documentation indexes:
| Attribute | AI.txt (Permissions & Policy) | LLMs.txt (Content & Context) |
|---|---|---|
| Core Purpose | Defines who can crawl and train on data | Provides what the site/product is about |
| File Format | Plain Text (Directive: Value) | Markdown (.md syntax) |
| Key Elements | User-agent, Allow, Disallow, License | H1 Title, Blockquote summary, Bullet links |
| Target Audience | AI crawler bots & data scraping engines | LLM context windows & RAG search agents |
6. How AI Crawlers Work
AI crawlers operate in two distinct modes: Model Pretraining and Real-Time Search Retrieval.
Model Pretraining Crawlers
Bots such as GPTBot, ClaudeBot, Google-Extended, and CCBot fetch billions of web pages to train foundational LLM neural network weights. They check /ai.txt and /robots.txt before downloading content. Blocking these bots prevents your site from being ingested into future model checkpoints.
Real-Time Search & RAG Crawlers
Bots such as PerplexityBot, ChatGPT-User, OAI-SearchBot, and Claude-Web execute live web fetches when users submit questions to AI search engines. They fetch /llms.txt to quickly locate authoritative documentation sources and cite your website in generative answer cards.
7. Supported AI Crawlers Directory
ToolMono detects, parses, and validates user-agent directives across 20+ major foundation model crawlers:
| User-Agent Name | Organization | Crawler Category | Primary Purpose |
|---|---|---|---|
| GPTBot | OpenAI | Training | Pretrains OpenAI GPT-4o and future foundation models |
| ChatGPT-User | OpenAI | User Search | Real-time web browsing triggered by ChatGPT queries |
| OAI-SearchBot | OpenAI | Search Index | Web indexing for OpenAI Search products |
| ClaudeBot | Anthropic | Training | Pretrains Anthropic Claude 3.5 Sonnet and Opus models |
| Claude-Web | Anthropic | User Search | Real-time web retrieval for Claude user prompts |
| PerplexityBot | Perplexity | Search Index | Real-time answer engine fetching and RAG indexing |
| Google-Extended | Training | Trains Gemini models without affecting Google SERP rankings | |
| CCBot | Common Crawl | Open Web Dump | Non-profit web archiving used in 60%+ open-source LLM datasets |
| Meta-ExternalAgent | Meta | Training | Pretrains Meta Llama 3 open-weights foundation models |
| DeepSeekBot | DeepSeek | Training | Pretrains DeepSeek V3 and R1 reasoning models |
8. How ToolMono Validates Files
ToolMono evaluates policy text using a 4-stage browser parser pipeline:
Lexical & Encoding Check
Verifies UTF-8 encoding, strips BOM headers, and checks control characters.
AST Directive Parser
Parses Directive: Value pairs, matches User-agent groups, and verifies syntax.
Crawler Matrix Engine
Maps policy rules against 20+ AI crawlers to compute Allowed/Blocked matrices.
GEO Scorecard
Calculates 7 multi-dimensional AI Discoverability and GEO Readiness metrics.
9. Common Validation Errors
Line lacks a colon (':') separator (e.g. 'User-agent GPTBot')
Directives must follow the 'Directive: Value' format. Use One-Click Auto-Fix to format colons automatically.
Disallow path 'admin' does not begin with '/' or '*'
Paths must be absolute relative paths (e.g. 'Disallow: /admin').
LLMs.txt is missing a top-level '# Project Title' H1 header
Add a single top-level '# Project Name' header at line 1 of your LLMs.txt file.
LLMs.txt link lacks a colon summary description
Format bullet links as '- Title: Brief summary text' to maximize LLM comprehension.
10. Understanding AI Discoverability
AI Discoverabilitymeasures how effectively an AI search engine or LLM assistant can locate, parse, comprehend, and cite your website's documentation and product data.
Higher AI Discoverability scores correlate directly with increased brand visibility inside Perplexity answer cards, ChatGPT web search citations, and Gemini AI Overviews.
11. Generative Engine Optimization (GEO)
Generative Engine Optimization (GEO) is the next evolution of Technical SEO. While SEO focuses on winning blue-link SERP positions in traditional search engines, GEO focuses on optimizing content structure, entity declarations, and LLMs.txt feeds so generative AI models synthesize accurate, cited answers referencing your domain.
12. AI Search Optimization (AISO)
AI Search Optimization (AISO) aligns your technical web infrastructure with real-time AI answer engines. Key AISO tactics include maintaining accessible /ai.txt permissions, serving high-signal /llms.txt indexes, and validating HTTP headers.
13. How to Deploy AI.txt & LLMs.txt
Production deployment snippets across top web frameworks, CMS platforms, and CDN hosting providers:
Next.js App Router (app/ai.txt/route.ts)
import { NextResponse } from 'next/server';
export async function GET() {
const content = `User-agent: *\nAllow: /\n\nSitemap: https://example.com/sitemap.xml`;
return new NextResponse(content, {
headers: { 'Content-Type': 'text/plain; charset=utf-8' },
});
}Nginx Web Server Configuration
location = /ai.txt {
default_type text/plain;
add_header Access-Control-Allow-Origin "*";
}
location = /llms.txt {
default_type text/markdown;
add_header Access-Control-Allow-Origin "*";
}14. Best Practices Checklist
- Explicitly define directives for key AI agents (GPTBot, ClaudeBot, Google-Extended, PerplexityBot, CCBot).
- Host your AI.txt file at the root domain path (/ai.txt) served with 'Content-Type: text/plain; charset=utf-8'.
- Format LLMs.txt with a single top-level '# Project Title' header and concise bullet links with colon summaries.
- Provide a valid contact email or policy link (e.g. 'Contact: ai-policy@yourdomain.com') for licensing inquiries.
- Regularly validate your policy files against updated specification standards using ToolMono.
15. Troubleshooting Common Issues
Issue: AI Crawlers are ignoring my Disallow directives
Solution: Ensure your /ai.txt file is hosted at the exact root domain, returns HTTP status 200 OK, and is not blocked by CDN firewall rules or Cloudflare bot fight mode.
Issue: CORS blocked URL validation in ToolMono
Solution: Add the HTTP response header Access-Control-Allow-Origin: * to your server response, or paste the file text directly into the editor.
16. Frequently Asked Questions (30 FAQs)
17. Official References & Specification Standards
Authoritative specifications, web standards bodies, and official AI crawler documentation:
LLMs.txt Specification Standard
Official specification proposal for /llms.txt and /llms-full.txt Markdown documentation indexes.
Spawning AI & AI.txt Documentation
Spawning AI organization standard for web creator consent and AI training opt-out specifications.
Google Search Central: AI Crawlers
Google's official developer documentation on Google-Extended, GoogleOther, and Gemini training control.
OpenAI GPTBot Documentation
OpenAI's official developer guide on managing GPTBot, ChatGPT-User, and OAI-SearchBot crawler access.
Anthropic ClaudeBot Documentation
Anthropic's guide on configuring web crawling policies for ClaudeBot and Claude-Web.
Common Crawl CCBot Documentation
Common Crawl's official guide on CCBot user-agent directives and data extraction opt-outs.
IETF RFC 9309: Robots Exclusion Protocol
Official IETF internet standard governing robots.txt directive syntax and parser behavior.
MDN Web Docs: Web Crawlers & Robots
Mozilla Developer Network guide to web crawling standards, user-agents, and search indexing.