Christian Igel
@christian_igel

The Learning Problem

Christian’s slides are “pretty self-contained”. They should work as course notes

Supervised Learning Scenario

Learning is goal-directed changingof behaviour based on experience

Binary Classification

Two classes: {apples, pears}, {-1, 1}

Supervised learning is to uncover an unknown relation between the input and output space

Based on sample input-output data

S = {(x_1, y_1),…,(x_N,y_N)}

l.i.d.: identically independently distributed

assumtion: Each pattern in the distribution is drawn independently from a stable distribution $𝑝$ over $\left(\mathrm{𝒳︀},\mathrm{𝒴︀}\right)$

$$𝑝\left(\left(𝑥,𝑦\right)\right)=𝑝\left(𝑥\right)·𝑝\left(𝑦|𝑥\right)$$

for example:

• The probability of a (red, 8cm, round) fruit is always the same for each sample
• Probability of being an apple is the same for each fruit that is (red, 8cm, round)

A hypothesis $ℎ:\mathrm{𝒳︀}\to \mathrm{𝒴︀}$ maps inputs to outputs. A hypothesis class $\mathrm{\mathscr{H}︀}$ is a set of such functions

Applying a learning algorithm means finding a hypothesis given sample data. i.e. it maps from $\mathrm{𝒳︀}\times \mathrm{𝒴︀}$ to $\mathrm{\mathscr{H}︀}$

Loss/risk function: numberically show how bad a certain output is. $\mathrm{𝓁︀}:\mathrm{𝒴︀}\times \mathrm{𝒴︀}\to [0,\infty [$, where $\mathrm{𝓁︀}\left(𝑦,𝑦\right)=0$

Given sample data, we want to find hypothesis showing good average performance

I.e. we want to minimize the risk:

$${\mathrm{ℛ︀}}_{𝑝}\left(ℎ\right)={𝔼}_{\left(𝑥,𝑦\right)\sim 𝑝}\left\{\mathrm{𝓁︀}\left(𝑦,ℎ\left(𝑥\right)\right)\right\}$$

Empirical risk of $ℎ$ on a sample data $𝑆$ ($𝑁≔|𝑆|$) is:

$${\mathrm{ℛ︀}}_{𝑝}\left(ℎ\right)=\frac{1}{𝑁}\sum _{𝑖=1}^{𝑁}\mathrm{𝓁︀}\left({𝑦}_{𝑖},ℎ\left({𝑥}_{𝑖}\right)\right)$$

If we define the loss function to be: $\mathrm{𝓁︀}\left(𝑦,ℎ\left(𝑥\right)\right)=𝕀\left(ℎ\left(𝑥\right)\ne 𝑦\right)$, then the empirical risk is:

$${\mathrm{ℛ︀}}_{𝑠}\left(ℎ\right)=\frac{1}{𝑁}\sum _{𝑖=1}^{𝑁}𝕀\left\{ℎ\left({𝑥}_{𝑖}\right)\ne {𝑦}_{𝑖}\right\}$$

(Indicator function $𝕀\{·\}$ is $1$ if it’s argument is true, otherwise 0)

Regression: Under the squared error $\mathrm{𝓁︀}\left(𝑦,\widehat{𝑦}\right)={\left(𝑦-\widehat{𝑦}\right)}^2$ the empirical risk corresesponds to the sum-of-squares error

Minimum risk over all measrable functions is the Bayes risk

$${\mathrm{ℛ︀}}_{𝑝}^{\text{Bayes}}={\mathrm{ℛ︀}}_{𝑝}\left({ℎ}^{\text{Bayes}}\right)=\underset{ℎ}{inf}{\mathrm{ℛ︀}}_{𝑝}\left(ℎ\right)$$

usually differs from ${\mathrm{ℛ︀}}_{𝑝}^{\ast }={\mathrm{ℛ︀}}_{𝑝}\left({ℎ}^{\ast }\right)={inf}_{ℎ\in \mathrm{\mathscr{H}︀}}{\mathrm{ℛ︀}}_{𝑝}\left(ℎ\right)$

For $\mathrm{𝒴︀}=\left\{-1,1\right\}$, the bayes risk is: ${ℎ}^{\text{Bayes}}\left(𝑥\right)=\text{sgn}\left[{𝔼}_{𝑝}\left(𝑌|𝑋=𝑥\right)\right]$. This can be done just by choosing the most likely one.

Algorithm $𝑎$ is consistent if ${lim}_{𝑁\to \infty }{\mathrm{ℛ︀}}_{𝑆}\left(𝑎\left(𝑆\right)\right)={𝑅}_{𝑝}^{\text{Bayes}}$

In many cases, $\mathrm{ℛ︀}\left({ℎ}^{\text{Bayes}}\right)=0$ can’t be reached

“No free lunch”

Prior knowledge can be incorporated by:

• restriction of the hypothesis space (inductive bias)
• a proper definition of neighbourhood/similarity (i.e. by defining some metric)
• penalising the “complexity” of classifiers (i.e. preferring simpler solutions)

Nearest Neighbor Classification

Every point is classified as the sample that’s nearest to it

Idea:

• given training data $𝑆=\left\{\left({𝑥}_1,{𝑦}_1\right),\dots ,\right\}$

“nearest” requires some understanding of “similarity”; a metric.

Metric or distance function on $\mathrm{𝒳︀}$ is a function $𝑑:\mathrm{𝒳︀}\times \mathrm{𝒳︀}\to ℝ$ with the following properties ($\forall 𝑥,𝑦\in \mathrm{𝒳︀}$):

• $𝑑\left(𝑥,𝑦\right)\ge 0$
• $𝑑\left(𝑥,𝑦\right)=0\iff 𝑥=𝑦$
• $𝑑\left(𝑥,𝑧\right)\le 𝑑\left(𝑥,𝑦\right)+𝑑\left(𝑦,𝑧\right)$ (triangle inequality)

1-Nearest neighbor (1-NN): just finding the nearest item from the training data K-Nearest neighbor (K-NN): just finding the nearest $𝑘$ items from the training data, and use the result that’s the most common from them.

Generalization and Regularization

1-NN will be (by definition) errorless on the training data

Minimizing empirical risk does not imply good generalization $\to$ overfitting

We can see, that training data is not good for evaluating a hypothesis. We need some test data.

$${𝔼}_{𝑆\stackrel{\text{iid}}{\sim }𝑝}\left[{\mathrm{ℛ︀}}_{𝑆\left(ℎ\right)}\right]={𝔼}_{\left(𝑥,𝑦\right)\sim 𝑝}\left\{\ell \left(𝑦,ℎ\left(𝑥\right)\right)\right\}={\mathrm{ℛ︀}}_{𝑝\left(ℎ\right)}$$

I.e. ${\mathrm{ℛ︀}}_{𝑆}\left(ℎ\right)$ (empirical risk over the test data) is an unbiased estimator of $\mathrm{ℛ︀}\left(ℎ\right)$ (risk of the hypothesis)

law of large numbers mentioned (larger testing set will improve risk estimate)

Selection of hypothesis (lecture note):

• Choosing the archer with the most bullseyes probably chooses the best archery player
• BUT if a room of 100 were to flip 1000 coins, and we choose the person who had the most tails, the expected number of tails is obviously over 0.5.
• if we were to apply this to finding minimum risk over some hypotheses, we’d find a good hypothesis, but an overoptimistic estimate of its risk
• $𝔼\left[max\left({𝑋}_1,\dots ,{𝑋}_{𝑀}\right)\right]\ge max\left(𝔼\left[{𝑋}_1\right],\dots ,𝔼\left[{𝑋}_{𝑀}\right]\right)$ (Jensen’s inequality)

Cross-validation for Hyperparameter Selection

How to chose $𝐾$ for K-NN

Algorithm parameters like $𝐾$ are called hyperparameters

Model selection: finding hyperparameters

Standard method for selecting from a few hyperparameters: $𝜅$-fold cross-validation

• Select a value for the hyperparameter
• Split $𝑆$ into $𝜅$ parts (hopefully even parts)
• Train $𝜅$ times on $𝜅-1$ parts and test the held-out part (test set)
• The estimate for that specific hyperparameter setting, is the mean of all $𝜅$ empirical risks
• Take hyperparameters maximising mean performance on the test set