Inference is what happens every time someone uses an AI model: the trained model takes an input, processes it and generates an output. Training builds the model once, at great expense; inference runs it millions of times a day. Inference cost and speed shape many product decisions in AI search, including how many sources a system retrieves, how long answers are, and when a slower reasoning mode is used.
Training versus inference
During training, a model's weights change as it learns from data. During inference, the weights are fixed and the model only computes outputs. A model does not learn from your conversation during inference, although a product may store conversations and use them for future training if its policies allow.
Why inference economics matter for search
Each AI answer costs compute in proportion to the tokens read and written. Providers balance answer quality against that cost. That pressure helps explain why systems retrieve a limited number of passages, why they may truncate long pages, and why deeper modes such as deep research are rate-limited or reserved for paid tiers.
Reasoning models spend extra inference compute thinking before they answer, sometimes called test-time compute. They can be more accurate on hard questions and slower on easy ones.
- Input tokens: the prompt, history and retrieved sources
- Output tokens: the answer, plus any hidden reasoning tokens
- Latency: how long the user waits
- Cost: what the provider pays per answer
Example
Budgeting an in-app assistant
Lanternmoss, a fictional knowledge base tool, prices out an AI search feature. Retrieving ten long passages per question makes answers slightly better but multiplies inference cost. Retrieving five shorter, well-chunked passages gives nearly the same quality for less. The same trade-off, at much larger scale, shapes public AI search products.
Common mistakes
Thinking the model learns from each chat
Inference does not update weights. Changes to model knowledge come from new training or retrieval.
Ignoring output length
Output tokens often cost more than input tokens. Long default responses add up in your own AI features.
Questions
A company that runs models and sells access to them through an API, whether it trained the model or hosts someone else's.
Inference is running the model. Reasoning is a style of inference where the model generates intermediate steps before the final answer.
