AI Visibility Audit

Common Crawl is a nonprofit that has crawled the open web every month since 2008 and publishes the results as a free archive. That archive became one of the main data sources used to train large language models. If your pages are in it, your content can end up in what AI models know. If they are not, it cannot.

No, these are separate systems. Google runs its own crawler and its own index. AI training runs on different crawlers and different datasets, and those can be blocked while Google gets through fine. A page can sit at position 1 in Google and still be invisible to ChatGPT, Claude, Gemini, and Perplexity. That mismatch is exactly what this audit finds.

It measures how close your site sits to the well-connected core of the web. Think of it as distance, not popularity: how few link-hops does it take to get from your site to everywhere else? The crawler uses this to decide who gets crawled deeply and often. One link from a strongly connected site moves you closer to the core than dozens of links from isolated ones.

Usually it is a default setting nobody reviewed. Some CDNs and firewalls ship with an option that rejects AI crawlers at the edge, or quietly rewrite your robots.txt with disallow rules. Your own files look clean, but the bot gets a 403 before your server ever sees the request. The audit tests this live by fetching your site with real AI crawler user agents.

If the block is in robots.txt, remove the disallow lines for the AI user agents you want to allow. If robots.txt is clean but bots still get errors, the block lives in your CDN or firewall dashboard, look for settings like "block AI bots" or bot fight mode and switch them off for the crawlers you want. The fix is usually one toggle, not a site rebuild.

Slowly, and that is normal. Common Crawl captures pages monthly, so the next snapshot is your first chance to enter the archive. AI models then train on that data in later cycles, which can be months apart. Opening access today starts the clock; the payoff arrives on the model's schedule, not yours. That is also why opening up early matters.

That is a legitimate choice, and this audit works in both directions. The same checks that confirm you are visible can confirm you are actually excluded, which is worth verifying because opt-outs are often misconfigured, a rule that stops one bot may do nothing to another. Common Crawl also runs an opt-out registry where you can register your preference, including for content in older crawls.

From Common Crawl's own published datasets, processed into a fast lookup copy: the URL index for monthly page counts, and the Web Graph domain rankings covering roughly 160 million domains per release, from 2023 to 2026. The access, structured data, and rendering checks run live against your site when you hit the button.

Free, no signup. You can audit up to three domains at once by separating them with spaces, which is handy for comparing against competitors. Results for a domain are cached for a few hours, and there is a fair-use daily limit across all users to keep the tool healthy.