A model can only be as good as its data. The training set must represent the population the model will serve, and the labels must be consistent enough to learn from.

Where data comes from

Source	Strengths	Watch out for
Product logs & databases	Real, plentiful, matches production	Logging bugs, missing fields, only sees what the current system shows users
Public datasets	Fast to start, benchmarks	Different population; license terms
APIs & web scraping	Broad coverage	Terms of service, copyright, personal data, rate limits
Sensors / IoT	High volume, time-stamped	Drift, calibration errors, gaps
Human annotation	Labels exactly what you need	Cost, inconsistency, annotator bias
Synthetic data	Rare cases, privacy	May not reflect real-world messiness

Representativeness and sampling bias

Selection bias — the sample isn't random: a loan model trained only on approved applicants never sees how rejected ones would have repaid.
Survivorship bias — only the survivors are recorded (products still on sale, companies still trading).
Coverage gaps — a speech model trained on one accent, a vision model trained on daytime photos.
Time mismatch — data from last year when behaviour has since changed.

Labeling

Write guidelines
Exact definitions and edge cases (“is a sarcastic review negative?”) with examples.
Pilot and measure agreement
Several annotators label the same items; compute inter-annotator agreement.
Label at scale with checks
Gold questions with known answers, multiple labels per item, majority vote or a label model.
Audit
Review disagreements and model–label conflicts; they often reveal unclear guidelines.

Raw percent agreement is inflated by chance. Cohen's kappa corrects for the agreement two annotators would reach by guessing with their own label frequencies:

κ = (pₒ − pₑ) / (1 − pₑ) pₒ = observed agreement, pₑ = agreement expected by chance

Python

from sklearn.metrics import cohen_kappa_score

a = ["spam", "ham", "spam", "spam", "ham", "ham", "spam", "ham", "ham", "ham"]
b = ["spam", "ham", "ham", "spam", "ham", "ham", "spam", "ham", "spam", "ham"]

p_o = sum(x == y for x, y in zip(a, b)) / len(a)
p_e = 0.4 * 0.4 + 0.6 * 0.6            # both say spam 40% of the time, ham 60%
print(p_o, round(p_e, 2))              # 0.8 0.52
print(round((p_o - p_e) / (1 - p_e), 3), round(cohen_kappa_score(a, b), 3))   # 0.583 0.583 ← same number

Rough reading (Landis & Koch): < 0.4 poor to fair, 0.4–0.6 moderate, 0.6–0.8 substantial, > 0.8 near-perfect. Use Fleiss' kappa or Krippendorff's alpha for more than two annotators.

Getting labels cheaply

Python

import numpy as np
from sklearn.datasets import make_classification
from sklearn.linear_model import LogisticRegression

X, y = make_classification(n_samples=500, random_state=0)
labeled, pool = np.arange(20), np.arange(20, 500)        # we paid for 20 labels so far

model = LogisticRegression().fit(X[labeled], y[labeled])
p = model.predict_proba(X[pool])[:, 1]
most_uncertain = np.argsort(np.abs(p - 0.5))[:5]           # probabilities closest to 0.5
print(pool[most_uncertain])                               # [393 398 245 464 413] ← send these rows to annotators
print(p[most_uncertain].round(2))                         # [0.5  0.5  0.51 0.51 0.49]

How do you handle noisy labels?

What is active learning?