Measure on data the model never trained on, in eval mode, with the metric that matches the real goal. Loss tells you how training is going; task metrics tell you whether the model is useful.

Three splits

Split	Used for	Touched
Training	Computing gradients	Every step
Validation	Early stopping, hyper-parameters, model selection	Every epoch
Test	Final, unbiased estimate	Once, at the end

From logits to metrics

Python

import torch

logits = torch.tensor([[2.0, 0.1, -1.0], [0.2, 1.5, 0.3], [1.2, 0.8, 0.1],
                       [-0.5, 0.2, 2.2], [0.1, 2.0, -0.3], [0.3, 0.1, 0.9]])
y_true = torch.tensor([0, 1, 1, 2, 1, 0])

probs = torch.softmax(logits, dim=1)
y_pred = probs.argmax(dim=1)
print(y_pred.tolist())                                   # [0, 1, 0, 2, 1, 2]
print(round((y_pred == y_true).float().mean().item(), 3))   # 0.667 ← accuracy

k = 3
cm = torch.zeros(k, k, dtype=torch.long)                 # rows = true class, cols = predicted
for t, p in zip(y_true, y_pred):
    cm[t, p] += 1
print(cm.tolist())                                       # [[1, 0, 1], [1, 2, 0], [0, 0, 1]]

tp = cm.diag().float()
precision = tp / cm.sum(dim=0).clamp(min=1)              # per predicted column
recall = tp / cm.sum(dim=1).clamp(min=1)                 # per true row
print([round(v, 2) for v in precision.tolist()], [round(v, 2) for v in recall.tolist()])   # [0.5, 1.0, 0.5] [0.5, 0.67, 1.0]

top2 = logits.topk(2, dim=1).indices
print(round((top2 == y_true[:, None]).any(dim=1).float().mean().item(), 3))   # 1.0 ← top-2 accuracy

Task	Metrics
Classification	Accuracy, precision/recall/F1 (per class, macro/weighted), ROC-AUC, top-k accuracy, confusion matrix
Regression	MAE, RMSE, R²
Detection / segmentation	mAP (mean average precision), IoU, Dice
Language models	Perplexity = `exp(mean cross-entropy)`, plus task benchmarks
Generation	FID (images), BLEU/ROUGE (text), human or LLM-judge evaluation

Python

import math
mean_ce = 2.0                     # average cross-entropy per token, in nats
print(round(math.exp(mean_ce), 2))   # 7.39 ← perplexity: as unsure as choosing among ~7.4 tokens

Beyond one number

Calibration — does 90% confidence mean right 90% of the time? Deep nets are often over-confident; temperature scaling on the validation set fixes much of it.
Slices — accuracy per class, per demographic, per data source; averages hide failures.
Error analysis — look at the worst mistakes (see Error Analysis in the Machine Learning notes).
Variance — rerun with a few seeds; differences smaller than seed-to-seed noise aren't real.
Cost — latency, memory and throughput at inference matter as much as accuracy in production.

Why is accuracy misleading on imbalanced data?

What is perplexity?