AI answers change every week. Here is why one check is not enough

The same question can get a different answer every time you ask, so a tourism board needs a steady weekly count, not a screenshot.

4 min read

An example with made-up numbers from the article: how often AI named an invented town on discovery questions over four weeks, 31%, 26%, 34% and 30%, with nothing changed in between.

Ask ChatGPT for the best beach towns for a family weekend. Then ask again. There is a good chance the second list is different. Ask next week and it may change again.

For a tourism board, this matters. If you check once, take a screenshot and put it in a board report, you may be reporting a good day or a bad one. One answer is a snapshot. It is not a measure.

How much answers move

Here is a sample, with made-up numbers, for Port Juniper (a town we invented for examples). It shows how often AI named Port Juniper on discovery questions, the ones that describe a trip and never name the town:

WeekNamed in
Week 131% of answers
Week 226% of answers
Week 334% of answers
Week 430% of answers

Nothing changed in Port Juniper over those four weeks. No campaign, no new page, no press. The number still moved eight points from its low to its high. If the marketing team had checked only in week 2, it would have looked like a slump. In week 3 it would have looked like a win.

Why AI answers change

There are a few reasons, and most of them have nothing to do with you.

AI does not give one fixed answer. Assistants write each answer fresh, with some chance built in. The same question can produce a list of five towns one time and a different five the next, especially when several places are a close call.

Assistants search the web as they answer. Many of them look things up before replying. The pages they find can differ from one day to the next, and the answer follows the pages.

The models change. The companies behind these assistants update them often, sometimes without notice. A new version can favor different sources or different places.

The web changes. A new magazine roundup, a viral forum thread or a review site's updated list can shift answers within weeks.

The seasons change. Ask about beach weekends in January and in June and you may get different towns, even with the same wording.

What one check misses

Because answers move this much, a single look can mislead you in both directions:

  • You can miss a real problem. If a rival has been gaining for a month, one good week can hide it.
  • You can chase a false one. A dip that comes back by itself next week does not need a new campaign.
  • You cannot tell if your work paid off. If you rewrite a page in March, you need March, April and May on the same footing to see whether answers changed.

How to tell a real change from noise

You do not need statistics training. You need a few habits.

  1. Ask the same questions, worded the same way, every time. If the wording changes, you cannot tell whether the answer changed or the question did.
  2. Ask each question more than once. A count across several answers is steadier than any single answer.
  3. Ask more than one assistant. They read different sites and move at different times.
  4. Compare like with like. If an assistant was missing from one week's check, leave it out of the comparison.
  5. Wait for a trend. A change that holds for three or four checks in a row is worth acting on. A change that lasts one week usually is not.

How Radar handles it

Radar by SPLASHY is built around these habits, so the numbers your team sees are ones you can act on.

  • The same questions every week. Your questions stay fixed. If you reword one, Radar starts it fresh as a new question, so a change in wording never looks like a change in what AI says.
  • Several answers per question. ChatGPT, Claude, Gemini and Perplexity are each asked every question three times per check. Google AI Overviews is asked once.
  • Small moves are labelled as small. On the Overview, a change of less than two points is shown as about the same, because answers wobble that much on their own.
  • Fair comparisons. When an assistant fails or is added, Radar compares only the assistants that were checked both times, and says so.
  • History you can see. On the Prompts tab, every question shows its last six checks, so you can see at a glance whether a question is steady, climbing or slipping.

The result is a weekly number you can put in a report and defend, and a record that shows whether your work moved it.

Start with a baseline

The best time to start counting is before you need the number. A few weeks of steady data gives you a baseline, and every campaign after that has something to be measured against.

You can run a free sample sweep to see what AI says about your destination today: ten questions, five assistants, in a few minutes.

See what AI says about your destination

Ten travel questions, put to five AI assistants, turned into data your team can work from. Free, no card, a few minutes.

Run a free sample sweep

All posts