AIbeginner

What Machine Learning Does: Learning a Function From Examples

Machine learning replaces hand-written rules with a function fitted to examples. Learn the supervised setup, what a loss function measures, how a linear regression is actually fitted, why a baseline comes first, and when a problem is better solved without machine learning at all.

Updated Sep 27, 2026 · AI-authored, review-gated · how lessons are made

machine-learning supervised-learning linear-regression loss-functions scikit-learn baselines

Part of:Machine Learning Fundamentals: From Examples to Trustworthy Models

Not signed in: your progress and quiz score won't be saved.

Progress1 / 10

Rules versus examples

A spam filter can be built two ways. You can write the rules yourself: block messages containing certain phrases, from certain domains, with too many links. Or you can collect thousands of messages already marked spam or not spam and let an algorithm find the rule that separates them.

	Traditional programming	Machine learning
Input	Data + rules written by a person	Data + the answers for that data
Output	Answers	A rule (the model)
Breaks when	The world outgrows the rules	The new data stops resembling the old
Improved by	Editing code	Better or more data, or a better model family

The second approach wins when the rule exists but is too tangled to write down, like recognising a handwritten 7, pricing a house from forty attributes, or spotting fraud in a pattern of card payments.

Full lesson text

All 10 steps on one page, for reading, reference, and search.

Show

1. Rules versus examples

	Traditional programming	Machine learning
Input	Data + rules written by a person	Data + the answers for that data
Output	Answers	A rule (the model)
Breaks when	The world outgrows the rules	The new data stops resembling the old
Improved by	Editing code	Better or more data, or a better model family

2. The supervised learning setup

Most practical machine learning is supervised: every training example comes with the answer you want the model to reproduce.

Definition: A supervised learning problem has examples $(x_i, y_i)$ , where $x_i$ is a vector of features (the inputs you can observe) and $y_i$ is the label (the answer). The goal is a function $f$ such that $f(x) \approx y$ on examples the model has never seen.

Two cases cover most problems:

Regression: $y$ is a number. Predict a house price from floor area, location and age.
Classification: $y$ is a category. Predict whether a transaction is fraudulent, or which of ten digits an image shows.

The last words of the definition carry the whole field. Reproducing the training labels is easy; a lookup table does it perfectly. The only thing worth paying for is accuracy on the next example, which is why the rest of this course keeps returning to data the model did not train on.

3. Three kinds of learning

Supervised learning is one of three broad families. They differ in what the data tells the learner.

Family	What the data contains	Typical task	Example
Supervised	Inputs with correct answers	Predict the answer for a new input	Credit default from loan history
Unsupervised	Inputs only	Find structure: groups, directions, anomalies	Segmenting customers by behaviour
Reinforcement	Actions and delayed rewards	Choose actions that maximise reward over time	A robot learning to grasp

The boundaries blur in practice. Self-supervised learning, which trains language models, manufactures labels from the data itself, for instance by hiding a word and predicting it. It has its own course on AnyLearn, as does reinforcement learning. This course stays with supervised learning on tables of features, because that is where most deployed models in business live and where the core ideas are easiest to see.

4. Loss: turning good into a number

To search for a good $f$ , the algorithm needs a score for how wrong a candidate is. That score is the loss. For regression the standard choice is the mean squared error over the $n$ training examples:

\text{MSE}(f) = \frac{1}{n}\sum_{i=1}^{n}\big(y_i - f(x_i)\big)^2

Training means picking, from a family of candidate functions $\mathcal{F}$ , the one with the lowest average loss on the training data:

\hat{f} = \arg\min_{f \in \mathcal{F}} \frac{1}{n}\sum_{i=1}^{n} L\big(y_i, f(x_i)\big)

This is called empirical risk minimisation. "Empirical" is the warning label: you minimise error on the sample you have, while hoping to minimise error on the population you will meet. Squaring also has consequences. One error of 10 costs as much as a hundred errors of 1, so a few outliers can drag the whole fit toward them.

5. Fitting a line by hand

Take three flats, with floor area $x$ in square metres and price $y$ in thousands:

Area $x$	Price $y$
50	150
80	210
110	300

Linear regression looks for $f(x) = w x + b$ with the lowest MSE. For one feature the answer has a closed form: the slope is the covariance of $x$ and $y$ divided by the variance of $x$ .

w = \frac{\sum (x_i-\bar{x})(y_i-\bar{y})}{\sum (x_i-\bar{x})^2} = \frac{(-30)(-70) + 0 + (30)(80)}{900 + 0 + 900} = \frac{4500}{1800} = 2.5

The intercept makes the line pass through the means: $b = \bar{y} - w\bar{x} = 220 - 2.5 \times 80 = 20$ . The fitted model predicts 145, 220 and 295. The errors are 5, minus 10 and 5, so the training MSE is $(25 + 100 + 25)/3 = 50$ .

Predict first

What does this model predict for a 100 square metre flat?

6. The same fit in scikit-learn

Every scikit-learn model follows the same two-call interface: fit learns from features and labels, predict applies what was learned.

import numpy as np
from sklearn.linear_model import LinearRegression

X = np.array([[50], [80], [110]])   # features: one row per flat, one column per feature
y = np.array([150, 210, 300])       # labels: price in thousands

model = LinearRegression().fit(X, y)
print(model.coef_, model.intercept_)  # [2.5] 20.0
print(model.predict([[100]]))         # [270.]

Two details matter as soon as the toy grows up. X is always two-dimensional, rows are examples and columns are features, even with a single feature. And swapping LinearRegression for a random forest or a gradient-boosted model changes one line, because the interface is shared. That uniformity is why the hard part of machine learning is rarely the call to fit. It is deciding what goes into X, what y really measures, and how to check the result honestly.

7. The workflow around the model

The fit is one box among seven. Most failures happen in the boxes around it.

flowchart LR
  A["Define the question and the label"] --> B["Collect and clean data"]
  B --> C["Split: train and held-out"]
  C --> D["Fit a baseline"]
  D --> E["Fit and tune candidate models"]
  E --> F["Evaluate once on held-out data"]
  F --> G["Deploy and monitor"]
  G --> B

8. A model can only learn what it can express

The family $\mathcal{F}$ you search over is a set of assumptions, sometimes called the model's inductive bias. A straight line assumes each extra square metre adds the same amount. If price actually rises faster for large flats, no amount of data will make a linear model bend.

Model family	What it assumes	What it struggles with
Linear model	Effects add up, each feature at a constant rate	Curves and interactions, unless you engineer them as features
Decision tree	The answer changes in steps at thresholds	Smooth trends, extrapolating beyond the data
Nearest neighbours	Similar inputs have similar answers	Many irrelevant features, which make everything look far apart
Neural network	Little, given enough data and tuning	Small datasets, and explaining itself

Gotcha: A more flexible family is not automatically better. Flexibility lets a model fit the noise in its training data as readily as the signal, which is the subject of the next lesson.

9. Baseline first

Before any model, compute the score of the dumbest reasonable prediction. For regression that is predicting the average label for every example. For the three flats the average is 220, the errors are minus 70, minus 10 and 80, and the baseline MSE is $(4900 + 100 + 6400)/3 = 3800$ .

The standard summary, $R^2$ , is exactly the comparison with this baseline:

R^2 = 1 - \frac{\text{MSE}_{\text{model}}}{\text{MSE}_{\text{predict the mean}}} = 1 - \frac{50}{3800} \approx 0.987

An $R^2$ of 0 means the model is no better than the average; a negative value means it is worse.

In practice: Baselines expose problems early. If a sophisticated model barely beats the mean, the features probably do not contain the signal, and more tuning will not put it there. For classification the equivalent baseline is always predicting the most common class, which, as a later lesson shows, can look embarrassingly strong.

10. When not to use machine learning

Google's Rules of Machine Learning, an engineering guide written from its production experience, opens with Rule #1: "Don't be afraid to launch a product without machine learning." Its reason is blunt: "Machine learning is cool, but it requires data." Rule #4 adds: "Keep the first model simple and get the infrastructure right."

A problem is a good candidate when all four hold:

A pattern exists that links inputs to the answer, even if nobody can state it.
Labelled examples exist, or can be collected, in enough volume.
Errors are tolerable. A model will be wrong some fraction of the time, so the cost of a wrong answer must be bounded or caught downstream.
Rules would be too complex to write and maintain, or the pattern drifts faster than people can update them.

Key idea: If a short rule gets you 90% of the value, ship the rule, log its mistakes, and use those logs as the labelled data for a model later.

Check your understanding

The lesson ends with a 5-question quiz. Take it in the player above to see your score.

In supervised learning, what is the goal of training?
- Reproduce every training label exactly
- Find groups in unlabelled data
- Find a function that predicts labels well on examples it has not seen
- Maximise a delayed reward through actions
A linear regression fitted to three flats gives slope 2.5 and intercept 20 (price in thousands, area in square metres). What does it predict for 60 square metres?
- 170
- 150
- 190
- 140
Why can a few outliers distort a model trained with mean squared error?
- MSE ignores large errors
- MSE only works for classification
- Outliers are removed automatically by scikit-learn
- Squaring makes one large error cost as much as many small ones
A model has an R squared of 0 on held-out data. What does that mean?
- It predicts perfectly
- It does no better than predicting the average label
- It is worse than random guessing on every example
- It has no parameters
According to Google's Rules of Machine Learning, what should the first model be?
- The most flexible model available
- A neural network, so it can be extended later
- Simple, with the effort going into getting the infrastructure right
- Whatever wins the most benchmarks

Related lessons

intermediate

Decision Trees, Random Forests and Gradient Boosting

Tree ensembles remain the default for tabular data. Learn how a decision tree picks its splits, why one deep tree memorises, how random forests average away variance and gradient boosting corrects errors in sequence, and how to read feature importance without being misled.

11 steps·~17 min

intermediate

Generalization: Measuring What a Model Actually Learned

Training error flatters every model. Learn how train, validation and test splits work, watch overfitting happen in computed numbers, and see how regularization, cross-validation and leak-proof pipelines give you a score you can trust on data the model has never seen.

10 steps·~15 min

intermediate

Classification: Logistic Regression and Metrics That Do Not Lie

A classifier outputs a score; you choose where to cut it. Learn how logistic regression and log loss work, why 99% accuracy can be worthless on rare events, how precision, recall and PR curves expose it, and how to pick a threshold from the real costs of each mistake.

10 steps·~15 min

Math

intermediate

Gradients, Jacobians, and Hessians: Calculus in Many Dimensions

One derivative becomes three objects once a function has many inputs and many outputs. This lesson builds the gradient, the Jacobian and the Hessian, shows what each one actually tells you, and explains why curvature decides how many steps an optimiser needs and why nobody ever writes the Hessian down.

10 steps·~15 min