We checked 1,039 tourism websites. Here is how many block AI

One in 22 US tourism board websites blocks at least one AI crawler, and 71 refuse automated visitors entirely. What that means and how to check yours.

5 min read

Of 930 US tourism websites with a readable robots.txt file, 42 block at least one AI crawler and none block the crawlers ChatGPT and Claude use to search while they answer.

Every AI assistant that answers a travel question has to read the web first. Before ChatGPT recommends a weekend in your town, something had to fetch your pages. Whether it can do that is decided by a small text file most marketing teams never look at: robots.txt.

In November 2026 we read the robots.txt file of 1,039 US tourism board websites, from state tourism offices to small-town visitor bureaus. Here is what they tell AI.

What we found

  • 930 sites had a robots.txt file we could read.
  • 42 of them (4.5%) block at least one AI crawler. Almost all of these block the same three: OpenAI's GPTBot, Anthropic's ClaudeBot and Google-Extended.
  • 3 sites block PerplexityBot, which Perplexity uses to read pages for its answers. One is the visitor bureau of a major metro.
  • Not one site blocks the crawlers ChatGPT and Claude use for search (OAI-SearchBot and Claude-SearchBot).
  • 71 sites refused our request entirely, including 17 state tourism offices. That usually means a security firewall that turns away automated visitors. It may turn some AI assistants away too.
  • 133 sites (13%) publish an llms.txt, a short summary written for AI assistants. Seven of them also block AI training crawlers, which sends AI two opposite signals and is probably a vendor template.

Blocking is not a big-city habit. More than half of the 42 are small cities, counties and historic towns, the places that most need AI to know they exist.

Can AI read the homepage at all?

Being allowed in is one thing. Finding words on the page is another. We also requested each site's homepage once, the way an automated reader does, without running the page's scripts:

  • 28 homepages showed their text only after scripts ran. To a simple automated reader they look almost empty. They include the official tourism sites of a state, a major US city and several well-known resort areas.
  • 56 homepages refused the request outright (a "403 Forbidden" error), the sign of a security firewall.
  • 34 came back empty.

That's about 1 in 9 tourism homepages that an automated reader can't read properly. The big AI assistants run more capable readers than ours, and some do run scripts. But every barrier makes it less likely your own page is the one an assistant reads and quotes.

Two kinds of AI crawler

This is where it gets confusing, because the AI companies send out more than one crawler, and they do different jobs.

Training crawlers collect pages to train future AI models. GPTBot, ClaudeBot, Google-Extended and Common Crawl's CCBot are the best known. Blocking them is mostly a content-rights choice: you are saying "don't learn from my pages".

Search and answer crawlers fetch pages when an assistant is answering a question right now. OAI-SearchBot and ChatGPT-User (OpenAI), Claude-SearchBot and Claude-User (Anthropic) and PerplexityBot and Perplexity-User (Perplexity) do this job. Block these and the assistant cannot read your page when a traveler asks about you. It will answer from someone else's page instead.

Google is a special case. Google-Extended controls whether Gemini can use your pages, but Google's AI Overviews in search read the web through the ordinary Googlebot. A site that blocks Google-Extended still shows up in AI Overviews.

So the good news is that no tourism site we checked blocks the search crawlers of ChatGPT or Claude. The 42 that block training crawlers are mostly keeping their pages out of future models, not out of today's answers. The three that block PerplexityBot are a different story: Perplexity can't read them when it answers.

Why it may still matter

Training data shapes what an AI model "knows" before it searches. When an assistant answers from memory, without searching, it leans on what it learned in training. In our weekly checks, a large share of answers about travel cite no source at all. Those answers come from training. A destination that blocked training crawlers for the past two years is a little thinner in that memory.

There is no single right answer here. Some boards block training crawlers on purpose, to protect photography, partner content or paid research. The point is to make that choice on purpose, not by accident.

How these blocks happen

In most cases nobody at the tourism board decided anything. Blocks come from:

  • A website vendor's default. Many of the 42 sites block exactly the same three crawlers, which suggests a shared template.
  • A security setting. Firewalls and bot protection often treat every automated visitor as a threat.
  • An old decision. In 2023 many publishers blocked GPTBot in a hurry. Some tourism sites copied the same lines.

What to check on your own site

You don't need a developer to look. Open your browser and go to your site's address with /robots.txt on the end, for example visityourtown.com/robots.txt.

  1. Look for "User-agent: GPTBot", "ClaudeBot", "PerplexityBot", "Google-Extended" or "CCBot" followed by "Disallow: /". Each one means that crawler is shut out of the whole site.
  2. Check that nothing blocks the search crawlers: OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User, PerplexityBot and Perplexity-User. If any of these are disallowed, fix that first.
  3. Look at "User-agent: *". If it says "Disallow: /", every crawler is blocked unless it is named and allowed elsewhere.
  4. Ask your web vendor about the firewall. If your robots.txt loads in a browser but an automated tool gets "403 Forbidden", your site may be turning AI assistants away too.
  5. Decide on training crawlers on purpose, with your communications and legal team, and write the decision down.

How Radar uses this

Radar by SPLASHY checks every week whether AI assistants name your destination and cite your website. When your site is missing from answers, the first question is whether AI can read it at all. Radar tells you which AI crawlers your site lets in, and which pages AI couldn't read.

You can run a free sample sweep to see what AI tells travelers about your destination today.

How we did it: in November 2026 we requested robots.txt and the homepage from 1,039 US tourism board websites, one request each, and read the rules for 13 AI and search crawlers. A crawler counts as blocked when the rules shut it out of the whole site. Sites that partly restrict a crawler are counted as open.

See what AI says about your destination

Ten travel questions, put to five AI assistants, turned into data your team can work from. Free, no card, a few minutes.

Run a free sample sweep

All posts