robots.txt tester

Free tools

robots.txt tester

Load a live robots.txt or paste a draft, choose a user agent and test any path. The matching follows Google's published rules, including wildcards.

How to use it

How to use it

  1. 01

    Enter a domain and load its robots.txt, or paste your own rules

  2. 02

    Pick a user agent and type the path you want to test

  3. 03

    Read which rule decided the result and edit the rules to try changes safely

What you should know

What you should know

Google picks the most specific group for its user agent, then applies the rule with the longest matching path. When an allow and a disallow rule match with the same length, the less restrictive allow rule wins. The * wildcard matches any sequence of characters and $ marks the end of the address.

robots.txt controls crawling, not indexing. A blocked page can still appear in results if other sites link to it. To keep a page out of Google, allow crawling and use a noindex robots meta tag or header. Google reads up to 500 KiB of the file, and a missing file (404) means everything may be crawled.

FAQ

Frequently asked questions

Should I block AI crawlers?

That is a business decision. Many AI crawlers, such as GPTBot and Google-Extended, respect robots.txt. Blocking Google-Extended does not affect normal Google Search.

Why is my page indexed even though robots.txt blocks it?

Google can index an address it is not allowed to crawl if it finds links to it. Remove the block and add noindex, then wait for Google to crawl the page again.

Is the result exactly what Google does?

The tester follows Google's documented rules, but Search Console's URL Inspection tool remains the final word for your own site.

Contact

Prefer that we fix it for you?

Send a short brief and we will reply with a plan and a fixed quote within one working day.

Start a project
Start a project