The Cobbler and His Sitemap: We Ran robotcheck.coffee Through robotcheck
A tool that grades other websites on machine readability should pass its own test. Ours did, with 8.6 out of 10. The more interesting mistakes, however, never showed up in the score.
There is an uncomfortable question anyone who hands out grades should ask themselves: what would mine look like? On a Sunday in September we asked it and sent robotcheck.coffee through robotcheck.coffee. The result was 8.6 out of 10, grade A. One could stop there, pat oneself on the back and move on to the next topic.
We kept going, though not with the analyzer. We used something more old-fashioned: opening every URL in our own sitemap, one by one, and looking at what came back. This post is a first-hand account of what we found. It is about a respectable grade, eight dead links, and the question of what a score actually measures.
What the sitemap promised
The sitemap of robotcheck.coffee dated from the early days of the project. It listed the home page and the check page in fourteen languages, including Danish, Finnish, Norwegian and Romanian. Those four languages do not exist on the site and never have. Each of these eight URLs returned a 404. At the same time, five languages that do exist were missing: Russian, Japanese, Chinese, Korean and Turkish. So were the pricing page and every blog article.
A sitemap is something like the table of contents you hand to a crawler. Ours pointed to chapters that did not exist and left out a third of the chapters that did. A crawler that trusted it got a wrong picture of the site.
Our own analyzer had awarded this sitemap 0.75 out of 1 point. It checks whether a valid sitemap exists and counts its entries. It does not check whether those entries can actually be reached. After the fix, the sitemap contains more than a hundred URLs, all reachable, each linked to its language versions. The score rises to the maximum as a result. Not because the sitemap is now correct, though, but because it is longer. We will come back to that.
The detour via /de
The second mistake was subtler. The German version of robotcheck.coffee lives at the main address, without a language prefix. Anyone requesting /de is redirected to the main address with a 307. The hreflang annotations in the HTML, which tell search engines and crawlers where a page's language versions live, pointed to exactly that /de for German. Every page was declaring that its German version sat at an address which itself only points somewhere else.
For a human this makes no difference; the browser follows the redirect without anyone noticing. For a system that evaluates hreflang, it is a contradiction: the canonical address of the German home page and the address given in hreflang differ. The English home page had a similar problem, its canonical pointing to /en/ with a trailing slash, an address that redirects in turn. And a few hours earlier we had noticed that the pricing page and the changelog declared the home page as their canonical, effectively telling every crawler they were duplicates.
None of these mistakes moved the grade noticeably. The hreflang check establishes that hreflang annotations exist. Where they lead is not followed.
What we changed, and what was merely cosmetic
Honesty requires naming the changes that achieved less than they appear to. Our robots.txt did not mention Claude-SearchBot or Claude-User. Anthropic's crawler documentation now distinguishes three agents: ClaudeBot collects training data, Claude-SearchBot indexes for search, and Claude-User fetches pages when a person asks Claude something. According to Anthropic, all three respect robots.txt, including the non-standard Crawl-delay directive.
We added both agents. The catch: a crawler that is not named in robots.txt falls back to the general group for all user agents, and ours already allowed everything except the API. Claude-SearchBot and Claude-User were therefore allowed in before. What changed is how readable the file is for humans who want to know our intent, and the score under our own ruleset. Crawler behaviour stays the same.
Much the same goes for the llms.txt, which now has a description for every link, a list of the blog articles and a statement of the available languages. In an earlier post we analysed our server logs: over 22 days, not a single AI crawler requested the file. A better llms.txt is sound craftsmanship whose value, for now, lies mainly in being there if that changes.
The obvious objection
You could call all of this trivia. The site had an A, and AI crawlers were visiting regularly before any of the fixes. Eight dead links in a sitemap do not bring a website down, and Google and other search systems are well practised at silently repairing redirects and contradictory hreflang.
That is true, and it is still the wrong argument. That the large search engines tolerate mistakes does not mean every system reading web pages today does. The landscape of AI crawlers is young, their heuristics are undocumented, and nobody outside the vendors knows how forgiving a particular agent is with contradictions. Relying on the other side to understand what you meant shifts the work onto a system whose patience you do not know. A site that says nothing contradictory is the only bet that pays off either way.
What a score does not see
The truly important finding of this self-test is not about our website but about our tool. robotcheck checks whether something is present and whether it has the right form. In many cases it does not check whether it is true. A sitemap with dead entries passes; a longer sitemap with dead entries passes better. An hreflang annotation that leads nowhere counts the same as a correct one.
This is not peculiar to robotcheck. It applies to any automated rating system that is supposed to judge an entire website in a few seconds. But it is a reason to read a good grade as a progress report rather than as proof. For us it is a to-do: the sitemap check should actually request a sample of the listed URLs, and the hreflang check should detect redirects. Both are now on the list.
Until then, the advice stands that we gave ourselves that Sunday. Take the grade, then open your own sitemap and click through a few entries. It takes ten minutes. For us, those ten minutes found more than the score did.
— Andreas Kurt Peter Reuter