Feature engineering creates inputs that make the pattern easier to learn; feature selection removes inputs that add noise, cost or leakage. On tabular data, good features usually beat a fancier algorithm.

Engineering ideas

Idea	Example
Ratios & differences	`price_per_item = total / items`, `debt_to_income`
Aggregates over history	Purchases in the last 7 / 30 days, average basket size per user
Date & time parts	Hour, weekday, month, is_weekend, days since signup
Cyclical encoding	`sin(2π·hour/24)`, `cos(2π·hour/24)` so 23:00 is next to 00:00
Binning	Age bands — can help linear models capture non-linear effects
Interactions & polynomials	`area × quality`, `x²` for linear models (`PolynomialFeatures`)
Text	Length, keyword flags, TF-IDF (see Data Science)
Domain knowledge	BMI from height and weight; distance from coordinates

Python

import numpy as np
import pandas as pd

df = pd.DataFrame({
    "ts": pd.to_datetime(["2024-01-05 09:15", "2024-01-06 23:40", "2024-01-08 00:20"]),
    "total": [120.0, 80.0, 300.0],
    "items": [4, 1, 6],
})
df["hour"] = df["ts"].dt.hour
df["is_weekend"] = df["ts"].dt.dayofweek >= 5          # Monday = 0 … Sunday = 6
df["price_per_item"] = df["total"] / df["items"]
print(df[["hour", "is_weekend", "price_per_item"]].values.tolist())   # [[9, False, 30.0], [23, True, 80.0], [0, False, 50.0]]

angle = 2 * np.pi * df["hour"] / 24
pts = np.c_[np.sin(angle), np.cos(angle)]
print(abs(23 - 0), round(np.linalg.norm(pts[1] - pts[2]), 2))   # 23 0.26 ← raw gap vs cyclical distance

Selecting features

Family	How it works	scikit-learn	Trade-off
Filter	Score each feature against the target, independently of any model	`VarianceThreshold`, `SelectKBest(f_classif / chi2 / mutual_info_classif)`	Fast; misses interactions
Wrapper	Search subsets by repeatedly training a model	`RFE`, `RFECV`, `SequentialFeatureSelector`	Model-aware; expensive, can overfit the selection
Embedded	Selection happens during training	L1 (Lasso) coefficients, tree importances via `SelectFromModel`	Cheap; tied to one model's view

Python

import numpy as np
from sklearn.feature_selection import RFE, SelectFromModel, SelectKBest, f_classif
from sklearn.linear_model import LogisticRegression

rng = np.random.default_rng(0)
X = rng.normal(size=(400, 10))                                    # 10 features …
y = (2 * X[:, 0] - 1.5 * X[:, 1] + X[:, 2] + rng.normal(0, 0.5, 400) > 0).astype(int)   # … only 0, 1, 2 matter

print(SelectKBest(f_classif, k=3).fit(X, y).get_support(indices=True))                     # [0 1 2] ← filter
print(RFE(LogisticRegression(), n_features_to_select=3).fit(X, y).get_support(indices=True))    # [0 1 2] ← wrapper
l1 = LogisticRegression(l1_ratio=1, solver="liblinear", C=0.05)                            # L1 penalty
print(SelectFromModel(l1).fit(X, y).get_support(indices=True))                            # [0 1 2] ← embedded

Why bother selecting?

Fewer irrelevant features → less overfitting, especially with few rows (see Curse of Dimensionality in the k-NN topic).
Faster training and prediction; fewer data pipelines to maintain in production.
Easier explanations.
Removing features that are available in training but not at prediction time prevents leakage.

Filter vs wrapper vs embedded feature selection?

How would you encode the hour of day?