Reese Calder
I measure which websites AI search crawlers can actually reach, and I publish what I find for free. Each claim below links to the data or the page it comes from, so you can check it instead of taking my word for it.
The census
In September 2026 I read the robots.txt of the top 5,000 sites in the Tranco ranking and counted which ones block each AI crawler. Robots.txt is only half of it: a firewall or bot rule can refuse the crawler behind ChatGPT search or Perplexity while robots.txt welcomes it. I found 56 of the top 5,000 sites doing exactly that. The data is free under CC BY 4.0 and has a citable DOI, 10.5281/zenodo.22729641. Read the census.
Where it has been cited
IT-Administrator, a German IT trade magazine from Heinemann Verlag, updated its article on Cloudflare's September 15 change to Managed robots.txt with my measurement of what happened to the sites that used it, and links to my write-up.
A test of the free checkers
I ran six free AI crawler checkers against a test site that allows every AI crawler in robots.txt and refuses eight of them at the server. All six printed an all-clear for crawler access. The test and the captures.
Monitoring
The free checker tells you where a site stands today. Monitoring re-runs the checks every day for one site, $19 a month: whether the AI search crawlers are allowed in robots.txt and actually served, whether the homepage has readable text without JavaScript, and whether an llms.txt is there. Results go on a private status page. What it covers, and a live example.
Things I've built
- AI visibility checker
Paste a website address and see whether the crawlers behind ChatGPT, Perplexity and Google's AI answers are allowed to fetch it, and whether the page sends real text or an empty JavaScript shell.
- robots.txt user-agent checker
Finds user-agent lines that look fine but match nothing, because the robots.txt standard does not allow digits or dots inside a crawler name.
- llms.txt generator and validator
Builds an llms.txt from the pages a site actually has, or checks an existing one against the spec.
- AI crawler census
An open measurement of the top 5,000 sites: which AI crawlers each one allows or blocks, how many ship an llms.txt, and how many homepages send real text without JavaScript. Published as JSON under CC BY 4.0.
- Every site in the census that blocks AI search
The full named list, with what each one blocks, linking through to that site's own live result.
- Who blocks AI search: tracked board
A smaller panel of recognisable sites, re-checked over time, so a site changing its mind shows up with a date.
Recent work on open source
six merged. Each line links to the commit that shipped, so you can read the change.
- sebastienros/jint#35934.7k starsmerged
This JavaScript engine returned Infinity for some very large numbers that were actually finite. Its parser capped the exponent too early. I raised the cap. (a JavaScript engine for .NET)
- velopack/velopack#10472.3k starsmerged
Signing a Windows installer through Azure left the app's name out of the install prompt, so people saw a random filename instead. The signing command was missing the description field, so I added it.
- Koenkk/zigbee-herdsman-converters#130651.2k starsmerged
A ubisys device would let you write its input configuration but gave you no way to read it back. I added the read.
- Koenkk/zigbee-herdsman-converters#130641.2k starsmerged
A Legrand dimmer ignored the fade time you set, because another handler was quietly overriding it. I removed the override.
- Koenkk/zigbee-herdsman-converters#130661.2k starsmerged
An empty catch block was swallowing the reason some reads failed, so nobody could tell what went wrong. I added a log line.
- Koenkk/zigbee-herdsman-converters#130601.2k starsmerged
A power meter measured its AC frequency internally but never actually reported it. I wired it up.
If you need something fixed
Have a bug you can't pin down, or a library you depend on that nobody maintains anymore? Send it over. I'll dig in and usually send back a fix you can read in a couple of minutes.
The first look is free. If there turns out to be real work in it, we sort out money then.