Pretraining
- Training
- Standard term
Pretraining is the first, broad stage of training an AI model on a very large collection of data, before it is adapted to follow instructions, match preferences, or do narrower tasks.
The model reads text a token (a word or part of a word) at a time and learns to predict the next one. Doing that well across trillions of words takes grammar, facts, and patterns of reasoning, so the model picks those up. No one labels the text, so the collection can be enormous.
What it learns stops at the date the data was gathered, and it learns common things better than rare ones, since each subject leaves only as much trace as people wrote about it.
Where you’ll meet it
Shopping for a phone, you ask a chat assistant to compare two models, one of them released last month. It describes last year's model under the new name, and its specs for the new phone match the old one's.
What to do
Check whether the reply cites any web pages. If it cites none, it may well have answered from training alone. Turn on web search, or paste the phone company's spec page into the chat, and ask again.
Easy to mix up
Post-training data is made on purpose, by people or models writing examples and ratings. Pretraining data was mostly written by people for their own reasons, in web pages and books.
Check yourself
A lab releases a model that has had pretraining only. You type "What is the capital of France?" and it replies with three more quiz questions. Why?
Go deeperDevelopers choose the data mix and seldom publish it
Developers filter the raw data, remove duplicates, and decide how much of each kind of source goes in, including how much code. Those choices decide what the model learns as much as the total size does. Most developers publish only broad categories, so outsiders usually can't check whether a given topic was covered.
Asked for its own cutoff date, a model often gives a wrong one, and part of the reason is in the data. Writing about any month keeps piling up for years afterward, so at collection time the last few months are thinly covered and easy for the model to miss.