Next-token prediction (LLMs)

Large language models are next-token (next-word) predictors. A sequence goes in as the input, and the model decides which token should come next (optionally sampling from the top few candidates for variety rather than always picking the argmax). Once a token is chosen, it gets added to the input context and fed back into the model to generate the next token.

When input–output relationships are highly complex or high-dimensional, linear models fail; neural networks scale to arbitrarily non-linear relationships.

A natural-language lexicon is huge (~tens of thousands of “classes”); code vocabularies are smaller, so code LLMs can feel disproportionately capable at similar parameter scales.