Principal Component Analysis

• How to reduce to $𝑘<𝑑$ variables, such that as much information is preserved as possible?
• How many dimensions are required to capture a certain percentage of information?

Orthogonal matrix (iff. basis is orthonormal):

$${𝑈}^{𝑇}={𝑈}^{-1}$$

Semi-orthogonal: orthonormal in the lowest dimension

Semi-orthogonal matrices are semi-unitary:

$${𝑈}^{𝑇}𝑈=𝐼\vee 𝑈{𝑈}^{𝑇}=𝐼$$

Eigenvectors and Eigenvalues

$$𝑆{𝑢}_{𝑖}={𝜆}_{𝑖}{𝑢}_{𝑖}$$

We only care about Eigenvectors of unit length, and with the following ordering:

$𝑖<𝑗\Rightarrow {𝜆}_{𝑖}\ge {𝜆}_{𝑗}$

Eigencomposition of real symmetric matrix

For a real symmetric matrix $𝑀$, the decomposition is:

$$𝑀=𝑄\mathrm{\Lambda }{𝑄}^{𝑇}$$

Where:

• $𝑄\in {ℝ}^{𝑑\times 𝑑}$ is orthogonal
• $\mathrm{\Lambda }\in {ℝ}^{𝑑\times 𝑑}=\text{diag}\left({𝜆}_1,\dots ,{𝜆}_{𝑑}\right)$ where ${𝜆}_{𝑖}$ are the eithervalues
• The columns of $𝑄$ are corresponding eigenvectors

Singular value decomposition

For any(!) matrix $𝑋$:

$$𝑋=𝑈\mathrm{\Gamma }{𝑉}^{𝑇}$$

for $𝑟=min\left\{𝑁,𝑑\right\}$

• $𝑈\in {ℝ}^{𝑁\times 𝑟}$ Semi-orthogonal
• $\mathrm{\Gamma }=\text{diag}\left({𝛾}_1,\dots ,{𝛾}_{𝑟}\right)\in {ℝ}^{𝑟\times 𝑟}$ with ${𝛾}_1\ge \dots \ge {𝛾}_{𝑟}\ge 0$
• $𝑉\in {ℝ}^{𝑑\times 𝑟}$ Semi-orthogonal

${𝛾}_{𝑖}$ are NOT the eigenvectors!!! They are the singular values

PCA

Basic idea is to find an affine linear function that can map back to the dataset’s dimention as accurately as possible. (by selecting a smaller $𝑘$ as input dimension)

The linear function’s coefficient ${𝑈}_{𝑘}\in {ℝ}^{𝑑\times 𝑘}$ has pairwise orthogonal columns, all unit length

Reconstruction error is calculated by a sum-of-squares error

Formal goal is to find affine model and ${𝑧}_{𝑖}$ projected data points that minimizes sum-of-squares

Solution

For the $𝑏$ bias, use the empirical mean of the data points

Datapoints should be encoded as:

$${𝑧}_{𝑖}={𝑈}_{𝑘}^{𝑇\left({𝑥}_{𝑖}-\overline{𝑥}\right)}$$

The ${𝑈}_{𝑘}$ vector is made out of the first $𝑘$ eigenvectors of the data covariance matrix (S):

$$𝑆=\frac{1}{𝑁}\sum _{𝑖=1}^{𝑁}\left({𝑥}_{𝑖}-\overline{𝑥}\right){\left({𝑥}_{𝑖}-𝑥\right)}^{𝑇}$$

PCA assumes our distribution is Gaussian.

In short, PCA is a heuristic to not care about all the features, but the rest should be normalized for most algorithms