The Transformer (“Attention Is All You Need”, 2017) drops recurrence entirely: stacks of self-attention and feed-forward layers, with residual connections and layer norm. It powers modern language, vision and audio models.

Tokens → Embeddings + positional encoding → N × [Multi-head attention → Add & Norm → Feed-forward → Add & Norm] → Output head

Encoder, decoder, or both

Type	Attention	Trained to	Examples	Good for
Encoder-only	Bidirectional self-attention	Fill in masked tokens	BERT, RoBERTa	Classification, embeddings, extraction
Decoder-only	Causal self-attention	Predict the next token	GPT, LLaMA, Claude	Text generation, chat — today's LLMs
Encoder–decoder	Encoder self-attention + decoder cross-attention	Map input sequence → output sequence	Original Transformer, T5, Whisper	Translation, summarisation, speech-to-text

The GenAI notes cover the LLM side (GPT vs BERT, tokens, context windows, sampling); here we look at the machinery.

Positional encoding

Attention is permutation-invariant — shuffle the tokens and each output is just shuffled too — so order must be injected. The original paper adds sinusoids of different frequencies to the embeddings; many models learn position embeddings, and modern LLMs use rotary embeddings (RoPE) applied inside attention.

PE(pos, 2i) = sin(pos / 10000^(2i/d)) PE(pos, 2i+1) = cos(pos / 10000^(2i/d))

Python

import numpy as np

def positional_encoding(n_positions, d):
    pos = np.arange(n_positions)[:, None]
    i = np.arange(0, d, 2)[None, :]
    angle = pos / 10000 ** (i / d)
    pe = np.zeros((n_positions, d))
    pe[:, 0::2], pe[:, 1::2] = np.sin(angle), np.cos(angle)
    return pe

pe = positional_encoding(50, 16)
print(pe.shape)                          # (50, 16)
print(pe[0, :4].tolist())                # [0.0, 1.0, 0.0, 1.0] ← position 0: sin 0, cos 0, …
print(pe[1, :4].round(3).tolist())       # [0.841, 0.54, 0.311, 0.95]
near, far = pe[10] @ pe[11], pe[10] @ pe[40]
print(near > far)                        # True ← nearby positions have more similar codes

Multi-head attention

Instead of one attention with d_model-wide vectors, split into h heads of size d_model / h, attend in each independently, concatenate and project. Different heads learn different relations — one follows syntax, another coreference, another nearby tokens — at the same total cost.

MultiHead(X) = Concat(head₁, …, head_h) W_O headᵢ = Attention(X W_Qⁱ, X W_Kⁱ, X W_Vⁱ)

Python

import torch
from torch import nn

torch.manual_seed(0)
mha = nn.MultiheadAttention(embed_dim=64, num_heads=8, batch_first=True)   # 8 heads of size 8
x = torch.randn(2, 10, 64)                       # (batch, tokens, d_model)
out, weights = mha(x, x, x)                      # self-attention: query = key = value = x
print(out.shape, weights.shape)                  # torch.Size([2, 10, 64]) torch.Size([2, 10, 10]) ← weights averaged over heads
print(sum(p.numel() for p in mha.parameters()))  # 16640 ← 4 projections of 64×64 (+ biases)

A Transformer block

Python

import torch
from torch import nn

class TransformerBlock(nn.Module):
    """Pre-LN block, as in GPT-2 and most modern models."""
    def __init__(self, d_model=64, n_heads=4, d_ff=256, dropout=0.1):
        super().__init__()
        self.ln1, self.ln2 = nn.LayerNorm(d_model), nn.LayerNorm(d_model)
        self.attn = nn.MultiheadAttention(d_model, n_heads, dropout=dropout, batch_first=True)
        self.ff = nn.Sequential(nn.Linear(d_model, d_ff), nn.GELU(), nn.Linear(d_ff, d_model), nn.Dropout(dropout))

    def forward(self, x, causal=True):
        n = x.shape[1]
        mask = torch.triu(torch.ones(n, n, dtype=torch.bool), diagonal=1) if causal else None   # True = blocked
        h = self.ln1(x)
        x = x + self.attn(h, h, h, attn_mask=mask, need_weights=False)[0]   # residual 1
        x = x + self.ff(self.ln2(x))                                         # residual 2
        return x

torch.manual_seed(0)
block = TransformerBlock()
x = torch.randn(2, 10, 64)
print(block(x).shape)                                   # torch.Size([2, 10, 64]) ← same shape in and out: blocks stack
print(sum(p.numel() for p in block.parameters()))       # 49984

layer = nn.TransformerEncoderLayer(d_model=64, nhead=4, dim_feedforward=256, batch_first=True, norm_first=True)
encoder = nn.TransformerEncoder(layer, num_layers=6, enable_nested_tensor=False)   # built-in, 6 deep
print(encoder(x).shape)                                 # torch.Size([2, 10, 64])

Component	Role
Multi-head self-attention	Tokens exchange information
Feed-forward network (FFN)	Processes each token independently; usually 4× wider; holds most parameters
Residual connections	Gradient highway; each sub-layer adds a refinement
LayerNorm (Pre-LN)	Stable training of deep stacks
Output head	Linear layer to vocabulary logits (often tied to the input embedding matrix)

Why Transformers won

Parallel training over all positions — unlike RNNs — so they use GPUs efficiently.
Direct long-range interactions between any two tokens.
Scale predictably: loss keeps falling as data, parameters and compute grow (scaling laws).
General: the same block works on text tokens, image patches (ViT), audio frames and more.

Walk through a Transformer layer.

Why do Transformers need positional encodings?

What is the purpose of multiple attention heads?