AI answer variability describes the instability of AI-generated answers. Ask the same question twice and you may get different wording, different cited sources and a different list of recommended products. This comes from sampling randomness in the model, changing search results, personalization, location, model routing and frequent model updates. It is a basic fact of measuring AI visibility and the main reason single screenshots are poor evidence.
Sources of variation
Several layers combine, and products do not document how much each contributes.
- Sampling: models choose tokens with some randomness.
- Retrieval: search results and selected passages shift.
- Context: memory, chat history, location and language.
- Routing: products may send similar prompts to different models.
- Updates: models, indexes and system prompts change often.
Measuring through the noise
Treat each prompt as a sample, not an answer. Run it several times, in clean sessions, from the locations that matter, and report rates: the share of runs that mention you, the share that cite your domain, and the average position when you appear. Compare periods using the same method. Large, sustained shifts are meaningful; small swings between two runs usually are not.
When reporting to leadership, show ranges or rates rather than a single 'we rank first in ChatGPT' claim, which may not hold on the next run.
Example
From screenshots to rates
Ironvale, a fictional field service platform, used to celebrate screenshots where it topped an assistant's list. Running its key prompts ten times each over a week shows it is named in roughly half of runs for one prompt and nearly all runs for another. The team now reports those rates monthly and investigates only changes that persist across several weeks.
Common mistakes
Reacting to one answer
A single run can be unrepresentative in either direction.
Testing from one logged-in account
Personalization skews results. Use clean sessions and vary location where relevant.
Questions
It is partly by design: some randomness makes outputs less repetitive. Combined with changing sources, it means answers are never fully fixed.
There is no universal number. More runs give steadier rates; pick a count your budget supports and keep it constant.
