AI Visibility Audit

Common Crawl is a nonprofit that has archived the open web since 2008. Its crawler goes through the web every month, and the results are free for anyone to download. That archive became one of the main data sources used to train large language models. If your pages are in it, your content can end up in what AI models know. If they are not, every model that trains on this archive learns the web without you.

No, these are separate systems. Google runs its own crawler and its own index. AI training runs on different crawlers and different datasets, and those can be blocked while Google gets through fine. A site can hold the top Google spot and still be missing from every AI answer, because the AI crawlers were never let in. Gemini is a partial exception: for live answers it can pull from Google's search index, though its training data is still separate. That mismatch is exactly what this audit finds.

The crawler has not found you yet. Common Crawl does not try to visit every domain. It follows links, and it spends its monthly budget on sites close to the well-connected core of the web. New sites and sites with few inbound links can stay outside the archive for months even when nothing blocks the crawler. The fastest way in is links from well-connected sites that are already crawled. One mention on a busy, well-linked site does more than many links from quiet corners of the web. Once the crawler discovers you, your pages can appear in the next monthly snapshot.

Harmonic centrality measures how connected your site is to the rest of the web. The web has a core, a cluster of strongly connected sites that everything else links toward, and your rank is your distance from it. The fewer link hops it takes to reach your site from that core, the better your rank. A short distance means the crawler visits you deeper and more often. Link quality beats link count here. One link from a well-connected site helps your rank more than many links from sites with no links of their own.

Usually the block comes from a CDN or firewall setting that was on by default and nobody checked. Some CDNs reject AI crawlers at the edge, or quietly rewrite your robots.txt with disallow rules. Your robots.txt looks clean, but the firewall answers the bot with a 403 first, so your server never knows the bot came. The audit tests this live by fetching your site with the AI crawlers' full user agent strings. One limit worth knowing: those test requests come from our server, not from OpenAI's or Anthropic's real machines. A firewall that decides by IP address instead of user agent can treat the real bots differently, so a block found here is a strong signal, not final proof. If you want certainty, look at your own server logs.

If the block is in robots.txt, remove the disallow lines for the AI user agents you want to allow. If robots.txt is clean but bots still get errors, the block lives in your CDN or firewall dashboard, look for settings like "block AI bots" or bot fight mode and switch them off for the crawlers you want. The fix is usually one toggle, not a site rebuild.

There are two speeds. AI tools that search the web live, like Perplexity or ChatGPT with search, can pick up your pages within days once your bots are unblocked. The training path is slower. Common Crawl captures pages monthly, so the next snapshot is your first chance to enter the archive, and models train on that data in later cycles that can be months apart. Opening access today starts both clocks. The search payoff comes fast. The training payoff waits for the next model release, and nobody can speed that up.

That choice is fine, and the audit helps here too. The same checks that show you are visible can show that your block actually works. This is worth testing, because opt-outs often fail without anyone noticing, and a rule that blocks one crawler can miss the next one. Common Crawl also has an opt-out registry. Recording your choice there keeps you out of future crawls. Removal from already-published archives is a separate request, and reporting suggests those removals have been slow and incomplete, so use this audit to verify your block actually works.

From Common Crawl's own published datasets, processed into a fast lookup copy: the URL index for monthly page counts, and the Web Graph domain rankings, 94 to 209 million domains per release depending on the snapshot, from 2023 to 2026. The access, structured data, and rendering checks run live against your site when you hit the button. The five-check method comes from the AI Visibility Audit field guide by Stephen Burns of the Common Crawl Foundation, published under CC BY 4.0.

Free, no signup. You can audit up to three domains at once by separating them with spaces, which is handy for comparing against competitors. Every run checks your site live, so if you fix something and audit again you see the change straight away. There is a fair-use daily limit across all users to keep the tool healthy.