Robots.txt Validator and Testing Tool
- Paste or fetch the robots.txt By domain or by hand. Only the root file counts, and “Fetch” asks for it right there.
- Pick the crawler and the URLs One URL per line. The verdict updates as you type.
- Read what decides, and why Applied group, AI-training preference, warnings about what Google ignores and the exact line of each rule.
📄 robots.txt
the whole file, as the domain serves itPaste a robots.txt or fetch it by domain to test it here.
You can also drag the file onto the text box or .
Reading the file
what applies to that crawler, what to watch out for and the line of each thingFurther reading: How to write a good robots.txt · Directives Google does not read