A multi-layer perceptron (MLP) stacks fully-connected layers with activations in between. The forward pass is just matrix multiplies — computed for a whole batch at once.

Input layer (features) → Hidden layer 1 + ReLU → Hidden layer 2 + ReLU → Output layer (logits)

Shapes and parameter counts

parameters of a dense layer = n_in × n_out + n_out (weights + biases)

Python

import numpy as np

rng = np.random.default_rng(0)
sizes = [4, 16, 8, 3]                  # 4 features → 16 → 8 → 3 classes
params = [(rng.normal(0, 0.5, (a, b)), np.zeros(b)) for a, b in zip(sizes, sizes[1:])]

def forward(X):
    h = X
    for i, (W, b) in enumerate(params):
        h = h @ W + b
        if i < len(params) - 1:       # ReLU on hidden layers only
            h = np.maximum(0, h)
    return h                           # logits

X = rng.normal(size=(5, 4))            # a batch of 5 samples
print(forward(X).shape)                # (5, 3)
print(sum(W.size + b.size for W, b in params))   # 243 ← (4·16+16) + (16·8+8) + (8·3+3)

A hidden layer solves XOR

With hand-picked weights, two ReLU units are enough: h₁ = relu(x₁ + x₂) counts the ones, h₂ = relu(x₁ + x₂ − 1) fires only when both are 1, and the output is h₁ − 2h₂.

Python

import numpy as np

X = np.array([[0, 0], [0, 1], [1, 0], [1, 1]])
W1 = np.array([[1, 1], [1, 1]]); b1 = np.array([0, -1])
W2 = np.array([1, -2]);          b2 = 0

H = np.maximum(0, X @ W1 + b1)
print(H.tolist())                  # [[0, 0], [1, 0], [1, 0], [2, 1]] ← the hidden representation
print((H @ W2 + b2).tolist())      # [0, 1, 1, 0] ← XOR

The same network in PyTorch

Python

import torch
from torch import nn

torch.manual_seed(0)
model = nn.Sequential(
    nn.Linear(4, 16), nn.ReLU(),
    nn.Linear(16, 8), nn.ReLU(),
    nn.Linear(8, 3),                  # raw logits — the loss applies softmax
)
X = torch.randn(5, 4)
print(model(X).shape)                                    # torch.Size([5, 3])
print(sum(p.numel() for p in model.parameters()))       # 243

Universal approximation

Choice	Rule of thumb
Hidden layers	1–3 for tabular data; deeper nets need tricks (normalisation, residuals)
Width	Powers of two (64, 128, 256…) — wide enough to overfit a small batch
Hidden activation	ReLU / GELU
Output layer	Linear, with `n_out` = 1 (regression / binary) or number of classes

What happens in a forward pass?

How many parameters does a Linear(512, 256) layer have?