AI 4UAnalyze my business

Plain-language AI glossary

Term 03FundamentalsMeaning / context / connections

Fundamentals / Definition

Tokenization

The process of breaking text into smaller units (tokens) that an AI model can process, typically subwords or word pieces.

Browse all termsUnderstand the idea
03of 75
01

MeaningThe one-sentence definition.

02

ContextHow the idea works in practice.

03

UsesWhere the concept becomes useful.

01 / Plain-language context

How Tokenization works.

Before a language model can process text, it converts the text into tokens. A token may be a whole word, part of a word, punctuation, or another learned unit. The exact count varies by language and tokenizer. Tokenization matters because context limits, usage, and many API prices are measured in tokens rather than words.

02 / Practical uses

Where it helps.

  1. 01API cost estimation
  2. 02Context window management
  3. 03Multilingual processing
  4. 04Text preprocessing
03 / Keep exploring
Large Language Model (LLM)A neural network trained on massive text datasets that can generate, understand, and reason about human language.Context WindowThe maximum amount of text (measured in tokens) that an AI model can process in a single request, including both input and output.EmbeddingsNumerical vector representations of text that capture semantic meaning, enabling similarity search and clustering.