Transformer Architecture (Detailed)

The Transformer architecture, introduced in 2017, is the backbone of all modern AI. Understanding its components helps you reason about model capabilities and limitations. **Self-Attention**: The core innovation. For each token, the model computes how much "attention" to pay to every other token in the sequence. This is done through Query, Key, and Value matrices. The attention score between two tokens is the dot product of their Query and Key vectors, normalized and applied to Value vectors. This allows the model to relate distant tokens ("The cat sat on the mat because it was tired" — attention connects "it" to "cat"). **Multi-Head Attention**: Instead of one attention calculation, the model runs multiple in parallel (e.g., 96 heads). Each head can learn different relationship types — one might track grammar, another semantics, another long-range dependencies. **Positional Encoding**: Since attention processes all tokens simultaneously (no inherent order), position information is added via sinusoidal functions or learned embeddings. Modern models use RoPE (Rotary Position Embeddings) for better handling of varying sequence lengths. **Feed-Forward Layers**: After attention, each token passes through a feed-forward network that transforms its representation. This is where much of the model's "knowledge" is stored. **Modern Variants**: Decoder-only (GPT, Claude, Llama — used for generation), Encoder-only (BERT — used for classification/embeddings), Encoder-Decoder (T5, original Transformer — used for translation).