Generalization bounds

Think of it as a confidence interval for empirical loss

Single hypothesis

$$𝐿\left(ℎ\right)={𝔼}_{\left(𝑋,𝑌\right)\sim 𝑝}\left[\ell \left(ℎ\left(𝑋\right),𝑌\right)\right]$$

Since we don’t know $𝑝$ precisely, we can’t know $𝐿\left(ℎ\right)$ precisely either.

Let’s collect some i.i.d. elements from $𝑝$ into $𝑆$

$$\underset{\text{Empirical loss}}{\underbrace{\widehat{𝐿}\left(ℎ,𝑆\right)}}=\frac{1}{𝑛}\sum _{𝑖=1}^{𝑛}\ell \left(ℎ\left({𝑋}_{𝑖}\right),{𝑌}_{𝑖}\right)$$

$\widehat{𝐿}$ is an unbiased estimator of $𝐿$

$$𝐿\left(ℎ\right)\le \widehat{𝐿}\left(ℎ,𝑆\right)+\sqrt{\frac{1}{2𝑛}log\left(\frac{1}{𝛿}\right)}\text{with probability}\ge 1-𝛿$$

It would also be possible to give a lower bound BUT!!!!

Giving a lower bound as well as an upper one decreases precision, without being necessarily useful (with what we have now, we can see the worst-case scenario loss with high probability)

Finite hypothesis class

Learning by selection

We fix a single sample set $𝑆$, but have multiple hypotheses: $\mathrm{\mathscr{H}︀}=\left\{{ℎ}_1,{ℎ}_2,\dots ,{ℎ}_{𝑀}\right\}$

We would like to find an ${ℎ}^{\ast }\in \mathrm{\mathscr{H}︀}$, that has minimal loss from all of $\mathrm{\mathscr{H}︀}$. Unfortunately, since we can’t calculate the loss, we have to use empirical loss instead:

$$\widehat{{ℎ}_{𝑆}^{\ast }}=\underset{ℎ\in \mathrm{\mathscr{H}︀}}{argmin}\widehat{𝐿}\left(ℎ,𝑆\right)$$