Robots Txt Tester

Robots.txt rules can block a page by accident, especially when a broad disallow rule covers a path you wanted indexed. Testing rules before you upload them helps you catch those mistakes early.

This tool simulates the rules you paste. Enter a URL, and it tells you whether the longest matching rule allows or blocks that path, using the standard longest match idea, with Allow winning a tie.

How to Use This Tool

1. Paste your robots rules: paste the full rules you plan to upload.
2. Enter a URL or path to test: use a full URL or just the path part.
3. Click Test URL: read whether the path is allowed or blocked.
4. Test more paths: test important pages, not just one, before you upload.

Good Practice Tips

• Test your key pages: a rule that blocks a whole folder may also block a key page inside it.
• Remember this is a simulation: it reads only the pasted rules. It does not fetch a live file or guarantee how every crawler behaves.
• Keep rules simple: simple rules are easier to test and harder to get wrong.
• Test again after upload: open the live file and retest, to confirm the uploaded rules match the tested rules.