Linear regression

Derivatives recap

Multivariate chain rule

Linear regression

Predicting target variable $𝑦\in {ℝ}^{𝑚}$, based on $𝑥\in \mathrm{𝒳︀}={ℝ}^{𝑑}$ (for now, let $𝑚=1$)

$$\widehat{𝑦}=𝑓\left(𝑥\right)=\sum _{𝑖=1}^{𝑑}{𝑤}_{𝑖}{𝑥}_{𝑖}+𝑏={𝑤}^{𝑇}𝑥+𝑏$$

(where $𝑤\in {ℝ}^{𝑑}$ and $𝑏\in ℝ$)

We can get rid of the $𝑏$ parameter by defining: ${\widetilde{𝑥}}^{𝑇}=\left({𝑥}_1,{𝑥}_2,\dots ,{𝑥}_{𝑑},1\right)$ and ${\widetilde{𝑤}}^{𝑇}=\left({𝑤}_1,\dots ,{𝑤}_{𝑑},𝑏\right)$

Then:

$$\widehat{𝑦}=𝑓\left(\widetilde{𝑥}\right)={\widetilde{𝑤}}^{𝑇}\widetilde{𝑥}$$

(Basically add another dimension ranging from $1$ to $𝑏$ (from $𝑥$ to $𝑤$), so the equation works pretty much the same) what

Let’s omit the tilde for easier usage.

The training data can be gathered into a matrix:

$$𝑋=\left(\begin{array}{ccccc}{𝑥}_{11}& {𝑥}_{12}& \dots & {𝑥}_{1𝑑}& 1\\ {𝑥}_{21}& {𝑥}_{22}& \dots & {𝑥}_{2𝑑}& 1\\ ⋮& ⋮& \ddots & ⋮& ⋮\\ {𝑥}_{𝑁1}& {𝑥}_{𝑁2}& \dots & {𝑥}_{𝑁𝑑}& 1\end{array}\right)$$

You can index it with $𝑥$ (even partially). The notation ${\left[{𝑥}_{𝑖}\right]}_{𝑗}$ is to keep it ‘simple’.

Our goal is still to minimize risk.

Using squared loss: $\ell \left(𝑦,\widehat{𝑦}\right)={\left(𝑦-\widehat{𝑦}\right)}^2$ (Mean squared error)

If ${𝑋}^{𝑇}𝑋$ has full rank, optimal parameters are given by:

$${𝑤}^{\star }=\underset{{𝑋}^{†}}{\underbrace{{\left({𝑋}^{𝑇}𝑋\right)}^{-1}{𝑋}^{𝑇}}}𝑦$$

$\left({𝑋}^{†}\right)$ is the pseudoinverse

note: vectors won’t produce an outer product with full rank (just 1)

Deriving Linear Regression

Overdetermined case

Probably see the slides

Underdetermined case

At times the data points are fewer than the the amount of parameters available, the (resulting matrix) rank will not be full, and many solution can give us 0 error. In that case, minimization of the norm might be a good idea.

We want to find the weights with minimal norm, that still goes through every datapoint:

${𝑤}^{\star }={\text{argmin}}_{𝑤\in {ℝ}^{𝑑}}\Vert 𝑤{\Vert }^2$ (norm) such that $\forall 𝑖:{𝑤}^{𝑇}{𝑥}_{𝑖}-{𝑦}_{𝑖}=0$ (linear constraint)

In this case, the gradient at the optimal point must be a linear combination of the normal (perpendicular) vectors of the constraint.

$${𝑤}^{\star }=\underset{{𝑋}^{†}}{\underbrace{{𝑋}^{𝑇}{\left({𝑋}^{𝑇}𝑋\right)}^{-1}}}𝑦$$

Notice the order has changed

Singular value decomposition (SVD)

$$𝑋=𝑈\mathrm{\Gamma }{𝑉}^{𝑇}$$

for $𝑟=min\left\{𝑁,𝑑\right\}$

• $𝑈\in {ℝ}^{𝑁\times 𝑟}$ (semi-orthogonal)
• $\mathrm{\Gamma }=\text{diag}\left({𝛾}_1,\dots ,{𝛾}_{𝑟}\right)\in {ℝ}^{𝑟\times 𝑟}$ with ${𝛾}_1\ge {𝛾}_2\ge \dots \ge 0$
• $𝑉\in {ℝ}^{𝑑\times 𝑟}$ semi-orthogonal

Computation is not included… Exercise for the reader implementor…

$${𝑋}^{†}\stackrel{\begin{array}{c}\mathrm{✨️}\text{Magic}\mathrm{✨️}\\ \downarrow \end{array}}{=}𝑉{\mathrm{\Gamma }}^{-1}{𝑈}^{𝑇}$$

Likelihood function

Also see the presentation