KitWharfFREEFast & Private Web Apps

robots.txt & XML Sitemap Generator / Validator

Build and test robots.txt crawl rules (including AI bot blocks) or generate and validate spec-compliant sitemap.xml files locally.

Rule Group #1
Generated robots.txt
Live Path Tester
BLOCKED — Disallow: /admin/
Technical SEO Workflow

How to Configure Crawl Rules & Sitemaps Safely

1

Pick a Crawl Preset

Start from Allow All, Block AI Crawlers (GPTBot/CCBot/ClaudeBot), or Staging Block All.

2

Test Paths Live

Use the Live Path Tester to verify whether any URL path is ALLOWED or BLOCKED for a bot.

3

Validate Sitemap URLs

Paste full https:// URLs in the XML Sitemap tab—invalid URLs are flagged automatically.

4

Export to Site Root

Download robots.txt and sitemap.xml and deploy both to your domain root (/robots.txt).

Frequently Asked Questions

Modern crawlers (Googlebot, Bingbot) evaluate the most specific matching path prefix (longest matching rule). If an Allow and Disallow rule have equal length, Allow typically wins.
Explore More Utilities

Complementary Tools

Other fast, free, and zero-upload utilities built for your workflow.

View all 109 utilities