robots.txt & Sitemap Tool
Build crawl rules, test paths live, block AI scrapers, and validate XML sitemaps locally.
HTML & URL Utilities
Encode query parameters, inspect URLs, and debug HTTP redirects cleanly.
SSL Certificate Decoder
Audit HTTPS domain SANs, expiry dates, and key strength before launch.
The Crawl vs. Indexing Paradox: The Most Expensive SEO Mistake
In technical search engine optimization, few configuration files wield as much power—or cause as much unintended catastrophe—as /robots.txt.
A single misplaced slash can wipe out millions of impressions overnight. Yet the most frequent disaster stems from a fundamental conceptual confusion: crawling is not indexing.
What robots.txt Actually Controls
robots.txtis an advisory protocol governing server crawl bandwidth. It tells compliant search engine bots (Googlebot, Bingbot, YandexBot) which URLs they are permitted to fetch from your origin server.- It does NOT prevent a page from being indexed. If high-authority external sites link to your
/internal/billing-termspage, Google knows the URL exists. Because yourrobots.txtprohibits Googlebot from fetching the page, the bot cannot download the HTML to inspect a<meta name="robots" content="noindex">tag. - Result: Google indexes the URL anyway, displaying the raw URL in search results with a generic snippet reading "No information is available for this page."
To permanently remove a private page from search results:
- Allow crawling in
robots.txt. - Add
<meta name="robots" content="noindex, follow">in the page<head>or emit anX-Robots-Tag: noindexHTTP header. - Allow Googlebot to crawl the page once, read the directive, and purge the document from its index.
Technical SEO Crawl Control Pipeline (RFC 9309 Architecture)
Modern search engines and AI agents process web discovery through a rigorous multi-stage decision pipeline.
Technical SEO Crawl Control & Search Indexing Architecture (RFC 9309)
From bot identification and longest-prefix rule matching to on-page meta tags, XML sitemap validation, and IndexNow broadcast.
The Algorithmic Rules of RFC 9309: Precedence Explained
Published in 2022, RFC 9309 (Robots Exclusion Protocol) standardized the exact precedence logic used by Google, Microsoft Bing, and other modern web crawlers:
- User-Agent Record Selection: A bot matches only the single most specific user-agent group that contains its name. If a
robots.txtfile contains bothUser-agent: *andUser-agent: Googlebot, Googlebot evaluates only theGooglebotsection and completely ignores rules declared under*. - Longest Matching Path Prefix: Within the selected User-Agent group, rules are sorted by the character length of the path. The longest matching rule wins:
For the URLUser-agent: * Disallow: /catalog/ Allow: /catalog/featured//catalog/featured/laptop:Disallow: /catalog/matches 9 characters.Allow: /catalog/featured/matches 18 characters.- Result: ALLOWED. The 18-character rule wins.
- Equal Length Ties Default to Allow: If an
AllowandDisallowrule match the exact same number of characters for a URL, search engines give precedence toAllow. - Wildcards (
*) and End-of-Path ($):Disallow: /*.pdf$blocks all URLs ending in.pdf.Disallow: /search?*q=blocks internal faceted search result parameters.
The 2026 AI Bot Taxonomy: Blocking Scrapers While Preserving Organic Search
AI foundation models consume vast amounts of server bandwidth scraping website documentation, blog content, and product catalogs. Fortunately, major AI providers now support dedicated user-agent tokens:
Taxonomy of Prominent AI User-Agents
| Crawler Name | Parent Company | Purpose | Impact on Search Rankings If Blocked |
|---|---|---|---|
| GPTBot | OpenAI | GPT-4 / GPT-5 Model Training | Zero impact on Google or Bing |
| ChatGPT-User | OpenAI | Live user requests inside ChatGPT | User cannot browse your link in chat |
| OAI-SearchBot | OpenAI | ChatGPT Search (Web Search Index) | Reduces referral traffic from ChatGPT Search |
| CCBot | Common Crawl | Public web scrape for AI models | Zero impact on commercial search engines |
| ClaudeBot | Anthropic | Claude model training & evaluation | Zero impact on commercial search engines |
| Bytespider | ByteDance / TikTok | AI model training & Douyin search | Zero impact on Google or Bing |
| PerplexityBot | Perplexity AI | Perplexity Answer Engine index | Decreases citations in Perplexity AI answers |
| Google-Extended | Gemini & Vertex AI training | Zero impact on Googlebot or Google Search! | |
| Applebot-Extended | Apple | Apple Intelligence training | Zero impact on Apple Search / Spotlight |
Production AI-Blocker Configuration Template
# Standard Search Engine Crawlers
User-agent: *
Allow: /
Disallow: /admin/
Disallow: /api/private/
Disallow: /checkout/
# Block AI Training Harvesters Without Affecting Search
User-agent: GPTBot
Disallow: /
User-agent: CCBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Bytespider
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: Applebot-Extended
Disallow: /
# Reference Spec-Compliant XML Sitemap
Sitemap: https://yourdomain.com/sitemap.xml
Test any URL path against any User-Agent using the Live Path Tester in the KitWharf robots.txt Studio.
Spec-Compliant XML Sitemaps: Formatting & Crawl Budget Rules
A sitemap is an explicit invitation for search engines to discover your canonical URLs. Adhere to these strict rules:
- Protocol Limits: Never exceed 50,000 URLs or 50MB uncompressed in a single file. For large sites, declare a Sitemap Index:
<?xml version="1.0" encoding="UTF-8"?> <sitemapindex xmlns="http://www.sitemaps.org/schemas/sitemap/0.9"> <sitemap> <loc>https://example.com/sitemap-tools.xml</loc> <lastmod>2026-11-19T00:00:00Z</lastmod> </sitemap> <sitemap> <loc>https://example.com/sitemap-blog.xml</loc> <lastmod>2026-11-19T00:00:00Z</lastmod> </sitemap> </sitemapindex> - Canonical Only: Never include 301 redirects, 404 dead links, or non-canonical trailing-slash variants.
- Escaped Entities: Always escape
&into&,<into<, and>into>.
Paste your raw URL list into the KitWharf XML Sitemap Generator to format, sort, validate, and download valid XML files in seconds.