Token
The smallest unit into which a language model splits text; it can be a word, part of a word or a punctuation mark.
// EXPLANATION
In agglutinative languages such as Turkish a word is often several tokens. The model produces the next token at every step; context window, speed and pricing are all measured in tokens.
// ALSO KNOWN AS
tokentokens
// RELATED TERMS
LLM (large language model)A neural network trained on very large amounts of text that generates text by probabilistically predicting the next piece (token) of a sequence.
InferenceThe stage in which a trained model produces output for new input; the phase that serves users after training.
Transformer architectureThe neural-network architecture introduced in 2017 that computes relationships between all words in a text in parallel using an attention mechanism.
