Neural networks

Different ways to categorize neural networks:

• feed-forward NN recurrent networks
• supervised unsupervised

The function ${𝑧}_{𝑖\left(𝑥\right)}$ represents the output of neuron $𝑖$.

x is often omitted for simplicity

Modeling real world neurons, the idea is that neurons integrate their inputs, and fire over some threshold.

Pre-activation:

$${𝑎}_{𝑖}=\sum _{𝑗=1}^{𝑑}{𝑤}_{𝑖𝑗}{𝑥}_{𝑗}+{𝑤}_{𝑖0}$$

Then we apply some transfer function (activation function) $ℎ$ to it:

$${𝑧}_{𝑖}=ℎ\left({𝑎}_{𝑖}\right)=ℎ\left(\sum _{𝑗=1}^{𝑑}{𝑤}_{𝑖𝑗}{𝑥}_{𝑗}+{𝑤}_{𝑖0}\right)$$

One of the “default” choices for $ℎ$ is the logistic function (or any non-polynomial function)

A single neuron by itself is not very useful (can’t even compute XOR).

A neural network is a DAG network of neurons

$𝑑=\text{Number of input neurons}$
$𝑀=\text{Number of hidden neurons}$
$𝐾=\text{Number of output neurons}$

Activation function of neuron $𝑖$ is denoted ${ℎ}_{𝑖}$

Neuron $𝑖$ can only get input from neuron $𝑗<𝑖$, making sure the graph stays acyclic.

Pre-activation of neuron $𝑖>𝑑$ (non-input neuron):

$${𝑎}_{𝑖}=\sum _{\left(𝑗,𝑖\right)\in 𝐴}{𝑤}_{𝑖𝑗}{𝑧}_{𝑗}$$

${𝑤}_{𝑖𝑗}$ means “weight from $𝑗$ to $𝑖$”.

$${𝑧}_{𝑖}={ℎ}_{𝑖\left({𝑎}_{𝑖}\right)}$$

• ${𝑧}_0$ is always 1 since z_0 is associated with the bias parameter
• ${𝑧}_{𝑖}={𝑥}_{𝑖}\forall 𝑖\in {𝑉}_{\text{input}}$ (The results on the input neurons is exactly the input)

Gradient descent will return in Machine Learning A

We are going to use sum-of-squares error for regression

Classification

one-hot encoding

Class $𝑖$ from $𝑛$ can be encoded as: $\left(\begin{array}{cccccc}0& 0& \dots & 1& \dots & 0\end{array}\right)$

Binary Classification

If we assume the sum of $𝑓\left({𝑥}_{𝑖}\right)$ are 1, and are all non-negative (i.e. our $𝑓$ models probabilities):

$$𝑝\left(𝑥\text{belongs to class encoded by y}|𝑥\right)=\prod _{𝑘=1}^{𝐾}{\left[𝑓\left(𝑥\right)\right]}^{{𝑦}_{𝑘}}$$

i.e. the element of $𝑓\left(𝑥\right)$ that’s indexed by $𝑦$

Giving the negative log-likelihood (called cross-entropy):

$$-ln𝑝\left(𝑆\right)=-\sum _{𝑛=1}^{𝑁}\sum _{𝑘=1}^{𝐾}{\left[{𝑦}_{𝑛}\right]}_{𝑘}{ln\left(𝑓\left({𝑥}_{𝑛}\right)\right)}_{𝑘}$$

Multi-class Classification

We could use soft-max (still making sure of the sum to be 1)

$${𝑓\left(𝑥\right)}_{𝑗}=\frac{{𝑒}^{{𝑎}_{𝑀+𝑑+𝑗}}}{{\sum }_{𝑘=1}^{𝐾}{𝑒}^{{𝑎}_{𝑀+𝑑+𝑘}}}$$

Thus, we can have a neural network, with soft-max as its last activation, and then a cross-entropy layer for training.

In reality, the model would output the values BEFORE soft-max. During training we could use a loss combining soft-max and log-likelihood, and during inference just add a soft-max.

Layers

A layer is a collection of neurons, where each neuron has the same inputs, and does the same calculation (has the same activation)

Layers can be numbered $𝑙$, and each layer $𝑙$ only gets inputs from ${𝑙}^{\prime }<𝑙$

${ℎ}^{\left(𝑙\right)}$ is the activation of layer $𝑙$

Inputs of layer $𝑙$ is gathered into a vector ${𝑥}^{\left(𝑙\right)}$

We can denote the weights of the $𝑖$th neuron in layer $𝑙$ by ${𝑤}_{𝑖}^{\left(𝑙\right)}$, and the bias by ${𝑏}_{𝑖}^{\left(𝑙\right)}$. This way the result is:

$${ℎ}_{𝑖}^{\left(𝑙\right)}\left(𝑥\right)={ℎ}_{𝑖}^{\left(𝑙\right)}\left({\left[{𝑥}_{𝑖}^{\left(𝑙\right)}\right]}^{𝑇}𝑥+{𝑏}_{𝑖}^{\left(𝑙\right)}\right)$$

Or, in matrix form (assuming implicit bias):

$${𝐻}^{\left(𝑙\right)}\left({𝑋}^{\left(𝑙\right)}\right)={ℎ}^{\left(𝑙\right)}\left({𝑊}^{\left(𝑙\right)}{\left[{𝑋}^{\left(𝑙\right)}\right]}^{𝑇}\right)$$

(where ${ℎ}^{\left(𝑙\right)}$ acts component-wise)

Epoch: one pass through the training data