Linear Classification

$𝑎,𝑏\in {ℝ}^{𝑛}$

${𝑎}^{𝑇}𝑏={\sum }_{𝑖=1}^{𝑛}{\left[𝑎\right]}_{𝑖}{\left[𝑏\right]}_{𝑖}$

$⟨𝑎,𝑏⟩=\Vert 𝑎\Vert \Vert 𝑏\Vert cos\angle \left(𝑎,𝑏\right)$

(Affine) linear function

$𝑓\left(𝑧\right)={𝑤}^{𝑇}𝑧+𝑏$

Note that the Solid contour line (where $𝑓\left(𝑧\right)=0$) is linear as well.

$$𝑓\left(𝑥\right)=⟨𝑤,𝑥⟩$$

${𝑥}_{\perp }$ is the orthogonal projection of $𝑥$ to the contour line (plane or stg on higher dimensions)

Distance from the plane:

$$\frac{|𝑓\left(𝑥\right)|}{\Vert 𝑤\Vert }=\frac{|{𝑤}^{𝑇}𝑥+𝑏|}{\Vert 𝑤\Vert }$$

Note: signed distance

Decision functions

Typical classification between at least two label classes.

One approach is the ‘discriminant functions’

We’ll have $\sim$ as usual

We have a fun modified $\text{sgn}$ function:

$\text{sgn}\left(𝛿\left(𝑥\right)\right)=\{\begin{array}{c}1\text{if}𝛿\left(𝑥\right)>0\\ -1\text{otherwise}\end{array}$

$𝑃\left(𝑌={\mathrm{𝒞︀}}_{𝑘}|𝑋=𝑥\right)=\frac{𝑝\left(𝑋=𝑥|𝑌={\mathrm{𝒞︀}}_{𝑘}\right)𝑃\left(𝑌={\mathrm{𝒞︀}}_{𝑘}\right)}{𝑝\left(𝑋=𝑥\right)}$

Division by zero is only possible if the event is immesurable.

$𝑃$ vs $𝑝$:

Input space ($𝑋$) can be continous, while the label space ($𝑌$) is discrete
$𝑝$ is a probability distribution

Joint probailities

see the slides

left of the line is one class, other is to the left

The coloured areas are point of possible error

Sweet spot is their intersection

By moving the line up until the sweet spot, we can see that the error decreases. (Red area is gone)

Logistic regression

Logistic function $\sim$ Fermi function $\sim$ standard sigmoid

$𝜃\left(𝑠\right)\frac{1}{1+{𝑒}^{-𝑠}}$

Goal is (for two-point distribution):

$𝑓\left(𝑥\right)=𝑃\left(𝑌=1|𝑋=𝑥\right)$

$𝑃\left(𝑌=𝑦|𝑋=𝑥\right)=\{\begin{array}{c}𝑓\left(𝑥\right)\text{for}𝑦=1\\ 1-𝑓\left(𝑥\right)\text{for}𝑦=-1\end{array}$

For labels $\left\{0,1\right\}$, we have Bernouli.

$$𝑃\left(𝑌=𝑦|𝑋=𝑥;𝑤\right)=𝜃\left(𝑦{𝑤}^{𝑇}𝑥\right)$$

..because the sigmoid does the casing for $𝑦\in \left\{-1,1\right\}$

Example for diverging algorithm for finding a classification function with ever higher confidence:

Logistinc regression

• For every step, you calculate a sum of all datapoints.
• Mini-batch only takes a random subset (introducing some noise/uncertainty); one can take smaller steps to compensate that.
• Also, you can do the stochastic descent so you only take random points, und update the parameter until you meet the criteria.

Polinomial embedding:

You transform the data, so you can fit a function that separate the data points. (See the slides)

The slide first shows a higher dimensional representation so we can see the points being separate by adding an extra parameter (using $\sqrt{}$)

Later this parameter can be the transformation on the input so we arrive in the original dimensions.

Perceptron Learning and Margins

Larger distance to the decision boundary (margin) can withstand a bigger noise. (making it more robust). Margin should be measured in relation to the input data. (‘cos changing the scale may show a ‘bigger’ margin with the same performance)

The algorithm checks for misclassification (margin is negative :o).

Intuitively, if the point is misclassified, we add the vector to the boundary’s normal, and if we repeat this method, we’ll converge to a boundary where no errors are present. (Again, given that the two groups are separable)

This algorithm should terminate in ${\left(\frac{𝑅}{𝜌}\right)}^2$ steps.

where

$$𝑅\leftarrow \underset{1\le 𝑖\le 𝑁}{max}\Vert 𝑥\Vert$$

And we assume there is a ${𝑤}_{\text{opt}}$ that

$${𝑦}_{𝑖}{𝑤}_{\text{opt}}^{𝑇}{𝑥}_{𝑖}\ge 𝜌>0$$

Proof:

He jsut drops the $cos$ parf of the vector product, so the inequality holds (in one of the steps).

Summary

• Great method, ‘cos it does not tend to overfitting.
• Very linear, may need non-linear transformation to compensate
• Good for ‘Big Data’