About the Robots.txt Checker
Load any domain's robots.txt, see its rules grouped by user-agent, and test whether a given path is allowed or blocked for a crawler.
How to use it
- Enter the domain.
- Click Fetch robots.txt.
- Test important paths to make sure they are not blocked by mistake.
- Free, no sign-up
- Live results from our server
- Works on mobile and desktop
Frequently asked questions
Is the Robots.txt Checker free?
Yes. The Robots.txt Checker is completely free to use, with no sign-up or usage limits beyond fair-use protection.
Is my data private?
This tool fetches only the public URL or domain you enter, from our server. Results are not stored.