Non-linear Transformations

$\mathrm{\Phi }$ is a non-linear transformation, $\mathrm{\Phi }:\mathrm{𝒳︀}\to \mathrm{𝒵︀}$, where $\mathrm{𝒵︀}={ℝ}^{\widetilde{𝑑}}$ ($\widetilde{𝑑}$ just means, that this isn’t the $𝑑$ of the original input space)

Where $\mathrm{𝒵︀}$ is called feature space.

$$ℎ\left(𝑥\right)=\widetilde{ℎ}\left(\mathrm{\Phi }\left(𝑥\right)\right)={\widetilde{𝑤}}^{𝑇}\mathrm{\Phi }\left(𝑥\right)$$

Overfitting and regularization

In the underdetermined case ($𝑁\le 𝑑$) linear regression always finds a model with zero empirical risk.

Even if ${𝑥}_{𝑖}$ and ${𝑦}_{𝑖}$ are independent!!!

Overfitting occurs if the model faithfully reflects idiosyncrasies of the training data rather than the underlying process that has generated the data.

Let’s assume we use a polynomial non-linearity. In this case, we’d need to figure out what $𝑘$ degree to use. We can find it using cross-validation.

Two-norm regularization

We can restrict out hypothesis with a parameter $𝐶>0$, such that $\Vert 𝑤\Vert \le 𝐶$ (see presentation for proof)

This $𝐶$ can restrict how steep our hypothesis can be.

But doing optimization under a constraint gets hard. We can instead add another term to the empirical risk: $𝜆{𝑤}^{𝑇}𝑤$ ($𝜆$ is a parameter similar to $𝐶$)

Adding this still gets us an analytical solution:

$${𝑤}^{\star }={\left({𝑋}^{𝑇}𝑋+𝜆𝐼\right)}^{-1}{𝑋}^{𝑇}𝑦$$

If we set $𝜆=0$ then the regularization term disappears, and the whole formula becomes the original regularization formula.

If $𝜆\to \infty$, then the value under the matrix inverse becomes really large, thus its inverse converges to 0. I.e. the ${𝑤}^{\star }$ becomes zero as well.

Similarly, rather than two-norm, one can use one-norm for regularization. This method is called LASSO. It’s used, since one-norm regularization causes sparser models.

BUT in the real world, things are somewhat different. If we take a heavily overparameterized model, the errors suddenly start getting better (double descent). This is because in the overparameterized case, our goal (see: previous lecture) was to find the weights that go through every datapoint, while having the smallest weight norm. I.e. it does implicit two-norm regularization.