Your hotel's AI visibility score is mostly noise

Ask AI the same question and the brands it names change more than 99% of the time — yet the dashboard sold to your hotel reports a single reading as fact

by Markus Busch

The number on your dashboard. A new category of tool pitches hotels a tidy figure: your AI visibility score, your citation share, your rank against the property down the road. It arrives clean, to one decimal place, and it invites you to act on it. That decimal is the problem.

What it actually is. A generative model adds randomness to every answer. Each citation it hands back is one URL out of many it could have pulled, so a single reading is one draw from a distribution, not a fact about the world. Researchers at the University of St. Gallen ran the same queries over and over and measured it: answer content changed roughly 70% of the time, only about a third of brands survived from one run to the next, and once they drew proper confidence intervals, many of the gaps between a brand and its rival sat inside the noise. SparkToro, testing thousands of prompts, found the model returned a different list of brands more than 99% of the time you asked it the same thing.

How much before it means anything. Enough sampling can pull signal out of noise, but the bill is steep. Across 30 platform-and-topic tests, a ranking needed between 33 and 94 answers-with-citations before the order held still and the leaders separated by more than the margin of error. Three of the thirty never separated at all, even after 125 questions. And the number you need isn't the one you'd guess: some engines stack citations onto the same few sites, so each answer carries less than its raw count suggests. A clean single figure, then, is a warning sign. It means someone measured once and rounded. (Those thresholds come from a vendor that sells sampling software; the independent work is what confirms the shape.)

What stays still. Under all that churn, one thing holds: the set of brands in contention. The exact order flickers, but the same familiar names keep turning up — in one test, four known brands filled 55% to 77% of the answers. Read that from a boutique's chair. You are not losing a close ranking to the hotel next door. You are missing from the set the machine returns at all, and more sampling will not find you in it. What survives the noise is being already known — which is its own argument, for another day.

What to do Monday. One question sorts a measurement tool from theater: ask the vendor to show their math. The honest ones sample the same query many times, report a range instead of a decimal, and say "not enough data" when that's the truth. Treat the top of any list as real and the middle and tail as a coin flip, and never take a single before-and-after as proof a content change worked — measure each side more than once. A clean rank from one reading is a guess with a decimal point.

The stake. These studies ran on running gear and headphones, not hotels, and two are preprints — but the noise is baked into how the model answers, whatever the topic. So stop paying for the number. It moves every time the model runs, and the position that outlasts the churn is the one no dashboard can sell you.

Enjoying this analysis? Hospitality.today delivers daily insights on hotel distribution, AI trends, and travel commerce — straight to your inbox. Subscribe for free at Hospitality.today →