During training, we can’t know how well the model will generalize to data outside of the training dataset. BUT we can calculate approximate bounds on the performance of the model.

$𝐿\left(ℎ\right)$ (basically risk) can’t be computed, since we don’t know the distribution. $\widehat{𝐿}$ the other hand CAN be calculated, although it’s accuracy depends on the used dataset. We need bounds to show how good it actually is in reality.

$\widehat{𝐿}$ is just the mean of the original loss function over a given set.

How different is $\widehat{𝐿}$ from $𝐿$?

Markov’s Inequality

Suppose $𝑋$ is a non-negative random variable. When, for all $𝜀>0$

$$\mathbb{P}\left(𝑋\ge 𝜀\right)\le \frac{𝔼\left[𝑥\right]}{𝜀}$$

X must be positive for this to work, if you can’t make sure of that, you should make modifications that keeps the probability like: $\mathbb{P}\left(𝑋<𝜀\right)=\mathbb{P}\left({𝑒}^{𝑋}<{𝑒}^{𝜀}\right)$ (because these share the probability, now the inequality can kick in)

Extended Markov

If $𝑓$ is monotonically increasing, such that $𝑓\left(𝑋\right)$ is non-negative, then for all $𝜀>0$

$$\mathbb{P}\left(𝑋\ge 𝑒\right)\le \frac{𝔼\left[𝑓\left(𝑋\right)\right]}{𝑓\left(𝜀\right)}$$

Chebyshev’s Inequality

Given any random variable $𝑋$, and $𝜀>0$

He uses squares that keeps probability then kall in Markov to get the inequality of ${\left(𝑋-𝔼\left[𝑋\right]\right)}^2\ge {𝜀}^2$ which then collapses to the variance by definition.

$$\mathbb{P}\left(\left|𝑋-𝔼\left[𝑋\right]\right|\ge 𝜀\right)\le \frac{\text{Var}\left(𝑋\right)}{{𝜀}^2}$$

For I.I.D variables

By setting $𝑋$ to the sample mean (${\widehat{𝜇}}_{𝑛}$) of some i.i.d random variables ${𝑋}_1,{𝑋}_2,\dots ,{𝑋}_{𝑛}$:

$$\mathbb{P}\left(\left|{\widehat{𝜇}}_{𝑛}-𝜇\right|\ge 𝜀\right)\le \frac{\text{Var}\left({𝑋}_1\right)}{𝑛{𝜀}^2}$$

Meaning that by increasing $𝑛$, the bound gets closer to 0, decaying at a rate $\frac{1}{𝑛}$

Hoeffding’s inequality

Full I.I.D. is not required, independence is sufficient here

${𝑋}_1,\dots ,{𝑋}_{𝑛}$ independent (not necessarily i.i.d) random variables, with support $\left[0,1\right]$ and ${𝑆}_{𝑛}={\sum }_{𝑖=1}^{𝑛}{𝑋}_{𝑖}$. Then for all $𝜀>0$

1. $$\mathbb{P}\left({𝑆}_{𝑛}-𝔼\left[{𝑆}_{𝑛}\right]\ge 𝜀\right)\le {𝑒}^{-2\frac{{𝜀}^2}{𝑛}}$$
2. $$\mathbb{P}\left({𝑆}_{𝑛}-𝔼\left[{𝑆}_{𝑛}\right]\le -𝜀\right)\le {𝑒}^{-2\frac{{𝜀}^2}{𝑛}}$$

Thus:

$$\mathbb{P}\left(\left|{𝑆}_{𝑛}-𝔼\left[{𝑆}_{𝑛}\right]\right|\ge 𝜀\right)\le 2{𝑒}^{-2\frac{{𝜀}^2}{𝑛}}$$

support: interval that a random variable can take numbers from $\left[0,1\right]$

This inequality implies:

$$\mathbb{P}\left(\left|\frac{{𝑆}_{𝑛}}{𝑛}-\frac{𝔼\left[{𝑆}_{𝑛}\right]}{𝑛}\right|\ge 𝜀\right)\le 2{𝑒}^{-2𝑛{𝜀}^2}$$

Because we substituted $𝑦=𝑛𝜀$ and $𝑦$ gets squared then divided by $𝑛$, the result is just ${𝑛}^2{𝜀}^2\frac{1}{𝑛}=𝑛{𝜀}^2$

$$\mathbb{P}\left(\left|{𝑆}_{𝑛}-𝔼\left[{𝑆}_{𝑛}\right]\right|\ge 𝑦\right)\le 2{𝑒}^{-2\frac{{𝑦}^2}{𝑛}}$$

Where $\frac{{𝑆}_{𝑛}}{𝑛}$ can be understood as ${\widehat{𝜇}}_{𝑛}$ and $\frac{𝔼\left[{𝑆}_{𝑛}\right]}{𝑛}$ as $𝜇$. Thus this is a bound on the error of the mean. This is similar to Chebyshev’s Inequality, BUT it decays exponentially.