Clustering

• Categorizing elements by a notion of similarity.
• We have no labels, the training will be unsuperwised
• Categories emerge from the similarity criteria
• good for $\_$ around and $\_$ out

K-means problem

• Finding the center point for each category (clusters)

$$𝜙\left(\mathrm{𝒳︀},𝐶\right)≔\sum _{𝑥\in \mathrm{𝒳︀}}\underset{𝑐\in 𝐶}{min}\Vert 𝑥-𝑐{\Vert }_2^2$$

${.}_2$ is for the Euclidian distance.

$$ is defined as the sum of (squared) distances between every point and their nearest point in $𝐶$

The problem itself is to find the two sets that minimizes the $𝜙$ function.

• Elements in $𝐶$ may not be elements of $\mathrm{𝒳︀}$
• The problem itself is NP-hard (even the approximation)
• we will treat the special case of $𝑘=1$

${min}_{𝐶=\left\{𝑐\right\}\in {ℝ}^{𝑚}}{\sum }_{𝑥\in \mathrm{𝒳︀}\Vert 𝑥-𝑐{\Vert }_2^2}$

solved when $𝑐=\frac{1}{|\mathrm{𝒳︀}|}{\sum }_{𝑥\in \mathrm{𝒳︀}}𝑥$ called a centroid (stands for center of mass geometrically speaking)

k-means algorithm

• Place random centerpoints to form the first clusters
• Re-calculate the center point to better fit that one partition. (doesn’t need to be globally correct)
• Rinse and repeat

• Should converge in finite iterations

  • … to a locally optimal clustering

Notes from Sadegh:

• Simple, and relatively efficient*
• Fast convergence in practice

• needs $𝑘$
• local optimum can be really bad
• outliers and heuristics matters a lot

seeding methods: k-means++

Common remedies for the root problems of k-means:

• run it multiple times, select the best

algorithm

• Place each starting point one-by-one:

  • Starting point is selected from the data at random
  • Each point has the probability of being selected equal to the distance from the closest center point
• Run the original k-means with this setup