AI 4UAnalyze my business

Plain-language AI glossary

Term 33InfrastructureMeaning / context / connections

Infrastructure / Definition

Model Serving

The infrastructure and process of hosting a trained AI model and exposing it as an API endpoint for real-time or batch inference.

Browse all termsUnderstand the idea
33of 75
01

MeaningThe one-sentence definition.

02

ContextHow the idea works in practice.

03

UsesWhere the concept becomes useful.

01 / Plain-language context

How Model Serving works.

Model serving is what happens behind the scenes when you call an AI API. The provider runs your input through the model on GPU hardware and returns the output. When self-hosting open-source models like Llama, you need to set up your own model serving infrastructure.

Popular serving frameworks include vLLM (high-throughput serving with PagedAttention), TGI (Hugging Face's Text Generation Inference), and Triton (NVIDIA's inference server). These handle batching (combining multiple requests for GPU efficiency), memory management, and concurrent request handling. Cloud platforms like AWS SageMaker, Google Vertex AI, and Replicate abstract away serving complexity.

For most builders, model serving is handled by the API provider. You only need to think about it when self-hosting for cost, privacy, or customization reasons. The key metrics are: latency (time to first token), throughput (requests per second), and cost per token. Optimizations like quantization and continuous batching significantly improve all three.

02 / Practical uses

Where it helps.

  1. 01Self-hosting open-source models
  2. 02High-throughput inference pipelines
  3. 03Custom model deployment
  4. 04On-premise AI for regulated industries
03 / Keep exploring
InferenceThe process of running a trained AI model to generate predictions or outputs from new inputs, as opposed to training the model.GPU / TPUSpecialized processors designed for the parallel mathematical operations that AI models require for training and inference.QuantizationA technique that reduces AI model size and memory requirements by using lower-precision numbers to represent model weights, trading a small accuracy loss for major efficiency gains.LatencyThe time delay between sending a request to an AI model and receiving the response, critical for real-time user-facing applications.