Measured Fairly: What robotcheck Does Better After the Overhaul with Fable 5.2

A measuring tool has only one asset: that people believe its numbers. In November we had to admit that our own numbers were not quite deserved in one respect. A website that did everything right could not reach 10 out of 10 on robotcheck. Not because anything was missing, but because our rules did not allow for it.

This post tells how we found and fixed that, and what changes for everyone who has a website checked. It is a report from our own workshop, backed by the project's change logs, not a study of other people's systems.

How we found the weakness

The analysis and the concept for the overhaul came from Fable 5.2. The question was simple: where does our score penalise a page for something that is not a flaw at all? The answers were uncomfortable because they were small and numerous. A single-language website got only 0.3 points for hreflang, although it could do nothing wrong there. A correct HTTPS canonical reached 0.7 at most. The performance rating never rose above 0.85 on a page without images. And for llms.txt, it cost points not to implement two optional recommendations that hardly any site implements. stripe.com showed it: points deducted without a real flaw.

None of these errors was dramatic on its own. Together they made a system that quietly marked down honest pages. That is the most dangerous kind of error, because nobody who only looks at the grade notices it.

What has changed

The concept rests on a simple idea: what we cannot determine, or what does not concern a page at all, does not count as a flaw. Such rules are now treated as not applicable, carry no weight and deduct nothing. The overall score is the weighted average of the criteria that apply to that page in the first place. The result is the one we wanted: a flawless page can now reach the full score. In addition, the evaluation of robots.txt by groups according to RFC 9309 became more precise, and the Schema.org check now covers nested structures it used to miss.

The second idea concerns which rules may cost points at all. Previously an automatic mechanism decided based on a rule's severity. Today an explicit list says what counts: nine rules from the llms.txt specification and seven rules on whether search and user bots such as Googlebot and the bots of OpenAI, Anthropic and Perplexity are locked out. Everything else appears in the report as a hint without any deduction. Anyone who wants to add something to the score has to change that list deliberately. That makes every change to the score a decision you can look up, instead of a side effect.

The third idea is the one you see most in the interface. Every score rule now has an understandable hint text, written in all 15 languages of the website. And the report no longer contradicts itself: no duplicates, no mixing of languages, no recommendation for a rule the page has passed.

Implementation and control

Claude Opus 5.5 and Sonnet 5.5 did the implementation, in small steps, each with its own description in the log. The part that gives us the most peace of mind is not visible: before the weekly regenerated checks are released, the old and the new version run against fixed snapshots of eight comparison domains, among them heise.de, stripe.com, python.org and robotcheck.coffee itself. If the reference domain fails a score rule, nothing is released and the previous checks stay active. Every rule that changes its verdict this way appears in the weekly report.

The rules themselves are also admitted more strictly. A proposal for a new rule needs a verbatim quote that can be found in the source text, and a requirement level. There are at most three new rules per source and week, and changed rules may not be rated more strictly than before. An example file from Anthropic that used to be treated as a specification has been retired as a source, because it is only an example. We also made our own llms.txt conform to the specification again along the way.

The obvious objection

You can read all this cynically: a rating system becomes more generous, and suddenly everyone looks better. The suspicion is justified, and we do not want to talk it away. But it only holds if generosity were the goal. The goal was accuracy. A page with a real flaw, such as a blocked search-bot group in robots.txt or a broken llms.txt, still loses points. What no longer happens is that it loses points for things that are not flaws. That the difference stays verifiable is the regression test's job. A verdict cannot be loosened silently without showing up in the weekly report.

One honest caveat remains. Even a fair set of rules only measures what can be measured from outside. It does not say whether an AI system will actually cite a page in the end. The score is an orientation, not a promise.

What we take from this

That Fable 5.2 listed the weaknesses so soberly helped us, because it turned the problem from a vague worry into a list of concrete corrections. A list can be worked through. Afterwards, what should hold for every checking tool holds: it must be able to check itself before it checks others.

If you want to know how your own site fares now, simply run the check again. The grade may have changed, and if so, for a reason the report names.

— Andreas Kurt Peter Reuter

Check your own website →