AI 4UAnalyze my business

Plain-language AI glossary

Term 38InfrastructureMeaning / context / connections

Infrastructure / Definition

Latency

The time delay between sending a request to an AI model and receiving the response, critical for real-time user-facing applications.

Browse all termsUnderstand the idea
38of 75
01

MeaningThe one-sentence definition.

02

ContextHow the idea works in practice.

03

UsesWhere the concept becomes useful.

01 / Plain-language context

How Latency works.

AI latency is usually measured through time to first token, total generation time, and end-to-end task completion. The number that matters depends on the experience: a conversation benefits from early feedback, while a background workflow may care more about total completion and reliability.

Latency changes with model size, prompt length, reasoning settings, output length, provider load, network distance, tool calls, and retries. Improve it by measuring the complete user journey, selecting the smallest model that passes quality tests, streaming when partial output is useful, caching safe repeated work, running independent steps in parallel, and keeping unnecessary context out of the request.

02 / Practical uses

Where it helps.

  1. 01Optimizing chat response times
  2. 02Choosing between model tiers
  3. 03Real-time feature design
  4. 04User experience benchmarking
03 / Keep exploring
InferenceThe process of running a trained AI model to generate predictions or outputs from new inputs, as opposed to training the model.StreamingA method of receiving AI model output token-by-token in real time as it is generated, rather than waiting for the complete response.Model ServingThe infrastructure and process of hosting a trained AI model and exposing it as an API endpoint for real-time or batch inference.GPU / TPUSpecialized processors designed for the parallel mathematical operations that AI models require for training and inference.