Deep Learning

Multiple levels of non-linear transformations

Idea: extract more and more abstract features, learning more abstract representations

Convolutional neural network (CNN)

Elements:

• Convolution -> ReLU
• max polling
• fully connected -> ReLU
• softmax activation

Convolution

Notation: $\ast$

Example:

$$\text{image}\ast \left(\begin{array}{cc}-1& 1\\ -1& 1\end{array}\right)$$

Convolution make sense because:

• Sparsity: Kernel is usually smaller than input, i.e. increased memory and statistical efficiency
• Parameter sharing: Each neuron in a feature map has the same weight, again, increased memory and statistical efficiency
• Equivariance: Translation of input also translates output similarly

Pooling

Combine pixels based on maximum/average in a cluster (down-sampling)

stride: Number of pixels moved

Decreases dimensionality, increase scale, and support translation invariance

Typically we use multiple input channels and multiple feature maps

• Input channels are convoluted with individual kernels, and then the results are added together

“Tricks-of-the-Trade”

• Ensemble methods

• Dropout

  • During mini-batch training
  • Regard some nodes as non-existent in each batch during training
  • Each masked model can be taken as a different model
  • We can make an ensemble of the different models

• Residual learning/shortcuts

  • “smooths the error surface” (slides p. 37)

  • makes the first layers have a larger contribution to the output

    • since the first layers are “less-linear” it makes the output “less-linear” as well
    • slides p. 38
  • If input and output dimensionality is different, a linear transformation can be used

• Augmentation

  • Applying random transformations to an input when composing the training dataset
  • helps the model generalize to different transformations
  • learns invariance with respect to the specified transformations

• Batch normalization

  • “fixes on questionable layers”
  • Helps against vanishing gradients
  • Goal is to reach zero mean and unit variance
  • Can be applied inside the model
  • Each mini-batch the variance and mean is calculated and adjusted
  • During inference the variances and means are calculated as a moving average made during training

• Layer normalization

  • “I have no idea why anyone would do this, but they seem to be happy about it”
  • dual of batch normalization