What the Crawler Really Sees: Server Logs and the View Without JavaScript

A score is a claim about a website. It says: if a crawler comes by, it will find clean HTML, a permissive robots.txt, understandable markup. What it cannot say is whether anyone comes by at all, and what that someone actually takes away. There are two tools for that, older than any AI debate and still barely used: your own server logs, and a fetch of the page in which no JavaScript runs.

This post describes how we combine the two. It is a how-to, not a study. The commands run on an ordinary Nginx server and need nothing that isn't already installed.

Why JavaScript is the decisive boundary here

Modern websites often deliver only an empty shell and assemble the actual content in the browser with JavaScript. For people, that makes no difference. For the crawlers behind ChatGPT and Claude, it makes all the difference.

The most solid measurement on this comes from Vercel and MERJ, from December 2024. They analyzed crawler traffic across Vercel's network; in a single month that came to 569 million requests from GPTBot and 370 million from ClaudeBot. The result was unambiguous: none of the major AI crawlers executed JavaScript, neither OpenAI's three bots nor ClaudeBot. The detail behind it is remarkable. GPTBot downloaded JavaScript files in about 11.5 percent of its requests, ClaudeBot in almost 24 percent, but neither of them executed them. The exception is Gemini, which builds on Google's crawling infrastructure and therefore renders.

The study is almost two years old, and crawlers change. We haven't found a newer measurement that contradicts it, but that is exactly why it pays not to rely on other people's numbers and to check your own site yourself.

Step one: fetch the page the way a crawler sees it

The simplest test is a fetch with curl. Curl doesn't execute JavaScript; it retrieves exactly the HTML the server delivers. So that the server behaves as it would toward the real bot, we pass GPTBot's user agent:

curl -s -A "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.2; +https://openai.com/gptbot" \
  https://example.com/my-page > raw.html

Then we search that file for a sentence from the page's main content, ideally from the middle of the text, not from the headline:

grep -c "a sentence from the middle of the article" raw.html

A zero means: for GPTBot and ClaudeBot, this text does not exist. It is only generated in the browser.

For a rougher comparison, you can render the same page once with a headless browser and compare the amount of text:

chromium --headless --dump-dom https://example.com/my-page > rendered.html

sed 's/<[^>]*>//g' raw.html | wc -w
sed 's/<[^>]*>//g' rendered.html | wc -w

The count is sloppy; it also includes inline scripts and style declarations. But if the first number is 300 and the second 3,000, you don't need precision anymore.

Step two: check the logs for who actually comes

The curl test shows what a crawler would see. The logs show whether it was there, which pages it wanted and what it got. First we filter out every line whose user agent belongs to a known AI crawler. The command also reads rotated and compressed logs:

zcat -f /var/log/nginx/access.log* \
  | grep -E 'GPTBot|OAI-SearchBot|ChatGPT-User|ClaudeBot|Claude-SearchBot|Claude-User|PerplexityBot|CCBot' \
  > ai-bots.log

One name missing from this list is Google-Extended. It isn't a crawler of its own, just a control token for robots.txt. Google keeps crawling as Googlebot, which is why Google-Extended never shows up in the logs.

How many requests per bot:

grep -oE 'GPTBot|OAI-SearchBot|ChatGPT-User|ClaudeBot|Claude-SearchBot|Claude-User|PerplexityBot|CCBot' ai-bots.log \
  | sort | uniq -c | sort -rn

And which paths they requested, with which status code and response size. The example assumes Nginx's default combined format:

awk -F'"' '{split($2,r," "); split($3,s," "); print s[1], s[2], r[2]}' ai-bots.log \
  | sort | uniq -c | sort -rn | head -30

The columns are count, status code, bytes delivered and path. Many 404s mean crawlers are requesting old or wrong URLs, often from an outdated sitemap. Many requests for .js files say nothing about whether the bot executes them. The Vercel study showed that.

Step three: tell real bots from disguises

A user agent is a self-declaration. Anyone can call themselves GPTBot; it takes exactly as long as typing the curl command above. Our own tests from step one end up in these logs too. Anyone who wants to publish numbers has to filter them out.

OpenAI publishes the IP ranges of its crawlers as JSON files, for GPTBot at openai.com/gptbot.json, for the other bots in separate files. A short Python script checks which addresses in our logs really belong:

curl -s https://openai.com/gptbot.json -o gptbot.json
grep 'GPTBot' ai-bots.log | awk '{print $1}' | sort -u > gptbot-ips.txt

import ipaddress, json

networks = [ipaddress.ip_network(p.get("ipv4Prefix") or p.get("ipv6Prefix"))
            for p in json.load(open("gptbot.json"))["prefixes"]]

for line in open("gptbot-ips.txt"):
    ip = ipaddress.ip_address(line.strip())
    genuine = any(ip in n for n in networks)
    print(ip, "genuine" if genuine else "NOT in the published ranges")

Not every provider makes it this easy. For crawlers without published ranges, what remains is reverse DNS lookup or, if that is missing too, the honest statement that their origin cannot be verified.

What the combination shows that neither part shows alone

Only together do the two methods become meaningful. The logs reveal which pages GPTBot actually requests and how many bytes it gets for them. The curl test reveals how much content is in those bytes. If a bot fetches a product page twenty times a month, gets 4 kilobytes each time, and the curl test shows an empty shell for exactly that page, the diagnosis is clear: the crawler is welcome, it comes regularly, and it leaves empty-handed every time.

No score from the outside can determine that, ours included. An analysis tool sees the page, but not the server. It can check whether robots.txt lets GPTBot in. Whether GPTBot then actually comes is only in the logs.

The obvious objection

You could object that all of this is too much effort for most site owners. Anyone on a website builder has no access to Nginx logs, and anyone who does doesn't want to fiddle with awk. That's true. But the first step, the curl test, needs no server access and takes a minute. On its own it already answers the most important question: whether the content is there at all without JavaScript. The logs are the deeper dive for those who want to know whether their assumptions about crawlers match reality. Ours, for example, showed that of 1,252 AI bot requests in three weeks, not a single one was for llms.txt, whatever GEO marketing promises about that file.

The lesson behind it is simple. Between what you believe about your website and what is in the logs, there is usually a surprise. It pays to find it yourself, before a language model quietly finds it for you.

— Andreas Kurt Peter Reuter

Check your own website →