Contrasting the two dominant architectural paths spawned by the original Transformer design.
| Feature | GPT (Generative Pre-trained Transformer) | BERT (Bidirectional Encoder Representations from Transformers) |
|---|---|---|
| Architecture | Decoder-only (stacked masked self-attention) | Encoder-only (stacked bidirectional self-attention) |
| Context Look | Left-to-right (causal masking hides future tokens) | Bidirectional (looks left and right simultaneously) |
| Pre-training Objective | Next-token prediction (Autoregressive Generation) | Masked Language Model (MLM) & Next Sentence Prediction (NSP) |
| Primary Utility | Creative generation, dialog, coding, text synthesis | Search queries, sentiment analysis, NER, semantic representation |
| Generation Style | Autoregressive (outputs token, appends to prompt, repeats) | Non-generative (outputs representation vectors per token) |
Autoregressive Loop Visualized
Step 1: Input: "Prompt: The sky is" -> Predict: "blue" (Probability: 85%)Step 2: Input: "Prompt: The sky is blue" -> Predict: "and" (Probability: 40%)Step 3: Input: "Prompt: The sky is blue and" -> Predict: "clear" (Probability: 60%)