Attention lets every position look at every other position and take a weighted average of what it finds, with weights computed from the data. It's the core operation of Transformers.

The analogy is a soft dictionary lookup. Each token produces a query (what am I looking for?), a key (what do I contain?) and a value (what I'll hand over). A query is compared with every key; the softmax of those scores says how much of each value to take.

Attention(Q, K, V) = softmax( Q Kᵀ / √d_k ) V

Project
Q = X W_Q, K = X W_K, V = X W_V — three learned linear maps of the same input (that's what makes it self-attention).
Score
Q Kᵀ — dot product of every query with every key: an (n × n) matrix.
Scale
Divide by √d_k so the scores don't grow with the vector size.
Mask (optional)
Set forbidden scores to −∞ — future tokens in a decoder, padding.
Softmax
Row-wise: each token's weights over all tokens sum to 1.
Mix
Multiply by V: each output is a weighted average of value vectors.

From scratch

Python

import numpy as np

def softmax(z):
    e = np.exp(z - z.max(axis=-1, keepdims=True))
    return e / e.sum(axis=-1, keepdims=True)

def attention(Q, K, V, mask=None):
    scores = Q @ K.T / np.sqrt(K.shape[-1])
    if mask is not None:
        scores = np.where(mask, scores, -np.inf)
    weights = softmax(scores)
    return weights @ V, weights

rng = np.random.default_rng(0)
n, d_model, d_k = 4, 8, 8                      # 4 tokens
X = rng.normal(size=(n, d_model))
W_q, W_k, W_v = (rng.normal(0, d_model ** -0.5, (d_model, d_k)) for _ in range(3))
Q, K, V = X @ W_q, X @ W_k, X @ W_v

out, w = attention(Q, K, V)
print(out.shape, w.shape)                      # (4, 8) (4, 4) ← one output per token; n × n weights
print(w.sum(axis=1).round(6).tolist())         # [1.0, 1.0, 1.0, 1.0] ← each row sums to 1

causal = np.tril(np.ones((n, n), dtype=bool))  # token i may only look at tokens ≤ i
_, w_causal = attention(Q, K, V, mask=causal)
print(w_causal[2].round(2).tolist())           # [0.31, 0.34, 0.35, 0.0] ← token 2 sees tokens 0–2; token 3 is masked
print(w_causal[0].round(2).tolist())           # [1.0, 0.0, 0.0, 0.0] ← token 0 can only attend to itself

Python

import numpy as np
import torch
import torch.nn.functional as F

torch.manual_seed(0)
Q, K, V = torch.randn(1, 4, 8), torch.randn(1, 4, 8), torch.randn(1, 4, 8)   # (batch, tokens, d_k)

manual = torch.softmax(Q @ K.transpose(-2, -1) / 8 ** 0.5, dim=-1) @ V
fused = F.scaled_dot_product_attention(Q, K, V)          # PyTorch's optimised kernel (FlashAttention on GPUs)
print(torch.allclose(manual, fused, atol=1e-6))          # True
first = F.scaled_dot_product_attention(Q, K, V, is_causal=True)[0, 0]
print(torch.allclose(first, V[0, 0]))                    # True ← causal: the first token can only copy its own value

Why divide by √d_k?

Python

import numpy as np

rng = np.random.default_rng(0)
for_d = {}
for d in (4, 64, 1024):
    q, k = rng.normal(size=(10000, d)), rng.normal(size=(10000, d))
    dots = (q * k).sum(axis=1)
    for_d[d] = (round(float(dots.std()), 1), round(float((dots / np.sqrt(d)).std()), 2))
print(for_d)       # {4: (2.0, 1.0), 64: (8.1, 1.01), 1024: (32.0, 1.0)} ← raw std ≈ √d; scaled std ≈ 1

Dot products of random d-dimensional vectors have standard deviation √d. Large scores push softmax into saturation — nearly one-hot, with vanishing gradients. Dividing by √d_k keeps the scores at unit scale.

Variant	Q comes from	K, V come from	Used in
Self-attention	The sequence	The same sequence	Encoders (BERT), decoders (GPT)
Masked (causal) self-attention	The sequence	Earlier positions only	Decoders / LLMs — no peeking at the future
Cross-attention	The decoder	The encoder's output	Translation, image captioning, diffusion text conditioning

Explain self-attention.

Why is attention better than an RNN for long sequences?