Inference (running a model)
- AI basics
- Standard term
Inference is using an AI model to produce a result from information you give it. A model is the trained AI inside an app. Results might include a reply or a document category such as “invoice.”
The model draws on patterns learned during training to answer your request. Writing the answer doesn't itself retrain it. If you want an earlier correction to affect a later reply, the app needs to include that correction in the new request, perhaps from the chat history or a saved note.
Where you’ll meet it
You tell your assistant that your company's fiscal year starts in April, and it adjusts a report correctly. In a new chat the following week, with no saved note or instructions, it assumes the year starts in January again.
What to do
Save the April start date in instructions or a file the app reads for financial work. Before the next report, check that the new chat includes that rule; the correction from last week's conversation may not be available there.
Easy to mix up
| Inference (running a model) | Training |
|---|---|
| No; the model uses what it has already learned | Training adjusts the numbers that control how the model handles information |
Check yourself
A company introduces a refund policy after its assistant was trained. The app attaches the policy to a customer's question, and the assistant uses it. How can it do this?
Go deeperProcessing input and generating output involve different computational work
OpenAI's July 2026 engineering account of inference separates processing the input from generating output. When the model first reads a request, it saves results from its calculations. As it writes each new token, it uses those saved results and adds new ones. This lets it reuse work from earlier text as it continues.
The same account describes scheduling, caching, and changes to the software that runs models on hardware. Those changes can improve hardware use and speed up responses even when the trained model stays the same.
Sources
- Machine Learning Glossary: inference, Google for Developers
- How GPT-5.6 fuses frontier intelligence with frontier efficiency, OpenAI, July 2026