
Free tools
robots.txt tester
Load a live robots.txt or paste a draft, choose a user agent and test any path. The matching follows Google's published rules, including wildcards.
How to use it
How to use it
- 01
Enter a domain and load its robots.txt, or paste your own rules
- 02
Pick a user agent and type the path you want to test
- 03
Read which rule decided the result and edit the rules to try changes safely
What you should know
What you should know
Google picks the most specific group for its user agent, then applies the rule with the longest matching path. When an allow and a disallow rule match with the same length, the less restrictive allow rule wins. The * wildcard matches any sequence of characters and $ marks the end of the address.
robots.txt controls crawling, not indexing. A blocked page can still appear in results if other sites link to it. To keep a page out of Google, allow crawling and use a noindex robots meta tag or header. Google reads up to 500 KiB of the file, and a missing file (404) means everything may be crawled.
FAQ
Frequently asked questions
Should I block AI crawlers?
That is a business decision. Many AI crawlers, such as GPTBot and Google-Extended, respect robots.txt. Blocking Google-Extended does not affect normal Google Search.
Why is my page indexed even though robots.txt blocks it?
Google can index an address it is not allowed to crawl if it finds links to it. Remove the block and add noindex, then wait for Google to crawl the page again.
Is the result exactly what Google does?
The tester follows Google's documented rules, but Search Console's URL Inspection tool remains the final word for your own site.
Free tools
More free tools

Contact
Prefer that we fix it for you?
Send a short brief and we will reply with a plan and a fixed quote within one working day.

















