Attention Mechanism

Attention is the core innovation that makes transformers work. When generating the next word, the model assigns "attention scores" to every previous token, focusing more on the ones that are contextually relevant. For example, when completing "The cat sat on the ___", the model attends strongly to "cat" and "sat" to predict "mat." Self-attention (used in transformers) lets every token in a sequence attend to every other token. This is powerful but computationally expensive: attention cost scales quadratically with sequence length. This is why longer context windows (like Claude's 1M tokens) are technically challenging and more expensive. Techniques like Flash Attention, sparse attention, and sliding window attention reduce this cost. For builders, attention explains why context window limits exist, why longer prompts cost more, and why models sometimes lose track of information in very long contexts (the "lost in the middle" problem). It also explains why RAG works: by placing relevant information near the end of the prompt, you ensure the model attends to it strongly.