Discuss your project

TOOLBOX · SEO CHECK

robots.txt and Sitemap.

Does your robots.txt block more than you think, and do search engines find your sitemap? Enter a domain and see what the file does for Google, Bing and AI crawlers.

The bot check protects the toolbox against automated abuse.

We load /robots.txt and at most one sitemap. We do not fetch the paths from the robots.txt or the pages of the sitemap. We do not store results permanently.

WHAT IS CHECKED

One Text File.
Big Impact.

The robots.txt decides which pages search engines and other crawlers may fetch. A typo or a server error can take a whole website out of search. The benchmark is the standard RFC 9309.

  • 01

    Reachability

    200, 404 or a server error: this decides whether crawlers follow your rules, fetch everything or consider everything blocked.

  • 02

    Format and Size

    A text file as text/plain in UTF-8, at most 500 KiB. If the server delivers an HTML page instead, no rules apply.

  • 03

    Syntax

    Every line with a field name and colon, groups start with User-agent, paths with /. The report shows typos such as “Dissallow” with the line number.

  • 04

    Effect per Crawler

    For Googlebot, Bingbot, GPTBot, ClaudeBot and Google-Extended: is the home page allowed? Under RFC 9309, the longest matching rule wins.

  • 05

    Internal Paths

    Paths such as /backup/ or /internal/ in the robots.txt make areas known in the first place. We point this out and do not fetch them.

  • 06

    Sitemap

    Reachable, in the right format, with full addresses on your own host, without addresses that the robots.txt blocks at the same time.

CRAWLERS IN THE REPORT

Who Reads
Along?

The product tokens come from the providers’ documentation. You decide which crawlers you allow; the report only shows what your file does.

  • Googlebot

    Crawler of Google Search, also for images, News and Discover.

    Source: Google Search Central

  • Bingbot

    Crawler of Bing Search. If there is a group of its own for bingbot, only its rules apply to it.

    Source: Bing Webmaster Blog

  • GPTBot

    Crawler of OpenAI for training generative AI models.

    Source: OpenAI

  • ClaudeBot

    Crawler of Anthropic that can collect content for training models.

    Source: Anthropic

  • Google-Extended

    Not a crawler of its own but a token for use by Gemini. No influence on Google Search.

    Source: Google Search Central

  • All Others (*)

    Crawlers without a group of their own follow the group User-agent: *. If there is none, no rules apply to them.

    Source: RFC 9309, Section 2.2.1

EXAMPLE

Clearly Structured.
With a Sitemap.

One group per crawler or list of crawlers, with the rules below it. Several user-agent lines in a row share the same rules (RFC 9309, Section 2.2.1).

  • robots.txt to Adapt

    The paths are examples. Only block what should not appear in search engines, and protect internal content with a sign-in.

    # For all crawlers: exclude search and shopping cart
User-agent: *
Disallow: /search
Disallow: /cart/
Allow: /search/help

# Exclude AI training, allow search
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
Disallow: /

Sitemap: https://www.example.com/sitemap.xml

  • What the Check Does Not Do

    Limits you should know:

    • We never fetch the paths from the robots.txt.
    • We load exactly one sitemap, no sub-sitemaps of an index and no pages from it.
    • Nobody can check from outside whether crawlers follow your rules.
    • We compare sitemap and robots.txt for the first 500 addresses.

More checks of the same domain: the website security check with an overall grade and the security.txt check for the reporting channel for vulnerabilities.

QUESTIONS

Good to Know.

Does the robots.txt protect areas of my website?

No. It is a request to crawlers, not an access restriction: RFC 9309 states explicitly that the protocol is not a substitute for security measures and that paths in the robots.txt become publicly known through it (Section 3). You protect internal areas with a sign-in. The check points out paths that sound like internal areas but never fetches them.

What is worse: no robots.txt or an error?

An error. If /robots.txt answers with 404 or another 4xx status, crawlers may fetch everything under RFC 9309 (Section 2.3.1.3). If it answers with a server error (5xx) or not at all, crawlers must assume that the whole website is blocked (Section 2.3.1.4). Google also treats 429 (too many requests) like a server error. A firewall or bot protection that blocks /robots.txt can take a website out of search this way.

Do Crawl-delay and Noindex work in the robots.txt?

Not reliably. They are not part of RFC 9309; crawlers may evaluate such fields but do not have to (Section 2.2.4). Google only evaluates user-agent, allow, disallow and sitemap; Google removed the code for unsupported rules such as noindex, nofollow and crawl-delay as of 1 September 2019. If a page should not appear in the search results, a noindex belongs on the page itself.

How do I block AI crawlers without dropping out of search?

With groups of their own for their product tokens, such as GPTBot (OpenAI), ClaudeBot (Anthropic) or Google-Extended. Google-Extended is not a crawler of its own but a token that controls the use of your content for Gemini; according to Google, it has no influence on Google Search. For every crawler, the group with its name applies, otherwise the group “*” (RFC 9309, Section 2.2.1). The check shows for the most important crawlers whether the home page is allowed or blocked.

What must a sitemap meet?

According to sitemaps.org, at most 50,000 addresses and 50 MB per file, UTF-8, full addresses with https:// and XML escaping (& as &). Larger websites split the sitemap with a sitemap index. In the robots.txt, you name the sitemap with Sitemap: and the full address. The check loads exactly one sitemap, the first one from the robots.txt or otherwise /sitemap.xml, at most 1 MB; it does not fetch the pages in it.

Does TOMORIS store my input or the result?

No, not permanently. Our server fetches the robots.txt and at most one sitemap; the website checked sees our server, not your IP address. The result stays in memory for five minutes so that repeated requests do not put load on the website again, and it is deleted after six minutes at the latest. Details are in the privacy statement.

TOOLBOX

More
Tools.

YOUR NEXT STEP

You Have the Result.
What Comes Next?

We assess findings, prioritize by risk and put changes into practice on request: as a project with EXPERTS or as ongoing operations with GRID.

Discuss the Result