AI 4UAnalyze my business

Plain-language AI glossary

Term 36InfrastructureMeaning / context / connections

Infrastructure / Definition

Quantization

A technique that reduces AI model size and memory requirements by using lower-precision numbers to represent model weights, trading a small accuracy loss for major efficiency gains.

Browse all termsUnderstand the idea
36of 75
01

MeaningThe one-sentence definition.

02

ContextHow the idea works in practice.

03

UsesWhere the concept becomes useful.

01 / Plain-language context

How Quantization works.

AI models store their parameters as floating-point numbers, typically in 16-bit (FP16) or 32-bit (FP32) precision. Quantization converts these to lower precision: 8-bit (INT8), 4-bit (INT4), or even 2-bit. A 70B-parameter model at FP16 requires ~140GB of memory. At 4-bit quantization, it fits in ~35GB, making it runnable on consumer hardware.

Quantization techniques range from simple (round weights to nearest lower-precision value) to sophisticated (GPTQ, AWQ, GGUF formats that minimize accuracy loss). Tools like llama.cpp and GGUF format have made quantized models accessible. You can run a quantized Llama model on a MacBook with 16GB RAM.

For builders, quantization is relevant when self-hosting models or deploying to edge devices. The accuracy tradeoff is usually small: 4-bit quantized models retain 95-99% of the original model's quality for most tasks. The main impact is on complex reasoning and nuanced language understanding. Always benchmark quantized models on your specific task before deploying.

02 / Practical uses

Where it helps.

  1. 01Running LLMs on consumer hardware
  2. 02Mobile and edge AI deployment
  3. 03Reducing inference costs
  4. 04Fitting larger models in limited GPU memory
03 / Keep exploring
LlamaMeta's open-weight model family, available for download and deployment across cloud, data-center, edge, and local environments under its license.InferenceThe process of running a trained AI model to generate predictions or outputs from new inputs, as opposed to training the model.Edge AI / On-Device AIRunning AI models directly on user devices (phones, laptops, IoT) rather than sending data to cloud servers for processing.DistillationA technique where a smaller "student" model is trained to replicate the behavior of a larger "teacher" model, achieving comparable quality at lower cost.