95 Top SaaS Sites: Who Did They Tell AI It Can Read Them?

AI answers now pull from the open web the same way search does — but only from pages their crawlers are allowed to read. I checked the robots.txt of 95 leading B2B SaaS companies to see which AI crawlers each one allows or blocks. The result: four out of five have made no decision at all.

What I measured

For each domain I fetched /robots.txt and classified 13 AI crawlers — the OpenAI, Anthropic, Perplexity, Google, and training-scraper bots — as one of three states:

• Blocked — the crawler has its own rule group telling it to stay out (Disallow: /).
• Allowed — the crawler has its own group that doesn't fully disallow it.
• Default — no rule of its own, so it falls under the site's general User-agent: * rules.

95 of 100 sites were readable (5 refused the fetch or redirected to a duplicate). The full list is at the bottom.

The finding: most sites never decided

Four out of five leading SaaS sites have never named a single AI crawler. Their content is open or closed to AI purely by whatever their platform or CDN set as a default. Nobody chose. And of the sites that did act, almost all of them open the door — the blocks that exist land on bulk training scrapers, not the engines that drive AI answers.

Where blocks do land, by crawler: Bytespider (4 sites), CCBot (3), Amazonbot (2), GPTBot (1), ClaudeBot (1), PerplexityBot (0). The answer engines are almost never the ones blocked.

Why it matters

robots.txt is now the gate to AI search, not just Google. If a site blocks GPTBot or ClaudeBot, that content can't be pulled into ChatGPT or Claude answers — so when a buyer asks AI "best tools for X," a blocked site can't be named. Most teams never set this on purpose; a security setting or a one-click "block AI bots" toggle flips it, and marketing never finds out.

The full list — 95 SaaS sites

Method notes: robots.txt fetched October 2026. Crawlers checked: GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, anthropic-ai, PerplexityBot, Perplexity-User, Google-Extended, CCBot, Bytespider, Amazonbot, Applebot-Extended, Meta-ExternalAgent. A site is counted as "No rules" when it names none of these. Measures stated instructions, not crawler behaviour.