Robots.txt Generator & Tester
FreeThe Robots.txt Generator helps you build a robots.txt file with allow and disallow rules, and includes a tester to check whether any URL would be blocked from crawling before you deploy it.
Test a URL against your rules
How to use the Robots.txt Generator & Tester
-
1
Set the user-agent
Use * to target all crawlers, or name a specific bot.
-
2
Add allow/disallow rules
List the paths you want to block or explicitly allow.
-
3
Add your sitemap
Optionally include your sitemap URL so crawlers can find it.
-
4
Generate and test
Generate the file, then use the tester to confirm specific URLs are allowed or blocked.
Features
- Build allow / disallow rules visually
- Add your sitemap URL
- Test whether a path is blocked
- Standards-compliant robots.txt output
When to use it
- Block staging or admin paths from crawlers
- Advertise your sitemap location
- Verify a rule before deploying it live
Frequently asked questions
robots.txt is a plain-text file at the root of your domain that tells search-engine crawlers which paths they may or may not request. It is the first file most crawlers fetch.
No. Disallow only stops crawling, not indexing — a blocked URL can still appear in search results without a snippet. To prevent indexing, use a noindex meta tag instead.
It must be at the domain root, e.g. https://example.com/robots.txt. It does not work in a subfolder.
Yes — adding a Sitemap: line helps search engines discover your sitemap even if you have not submitted it in Search Console.
Want more daily uses?
Create a free account for higher daily limits, or upgrade to Pro for unlimited access to every SEO tool.
Related free tools
Keep going with these tools: