The purpose of this article is to hold your hand through the process of designing and training a neural network.

*Note that this article is Part 2 of Introduction to Neural Networks. R code for this tutorial is provided here in the Machine Learning Problem Bible.*

## Description of the problem

We start with a motivational problem. We have a collection of 2×2 grayscale images. We’ve identified each image as having a “stairs” like pattern or not. Here’s a subset of those.

Our goal is to build and train a neural network that can identify whether a new 2×2 image has the stairs pattern.

## Description of the network

Our problem is one of binary classification. That means our network could have a single output node that predicts the probability that an incoming image represents stairs. However, we’ll choose to interpret the problem as a multi-class classification problem – one where our output layer has two nodes that represent “probability of stairs” and “probability of something else”. This is unnecessary, but it will give us insight into how we could extend task for more classes. In the future, we may want to classify {“stairs pattern”, “floor pattern”, “ceiling pattern”, or “something else”}.

Our measure of success might be something like accuracy rate, but to implement backpropagation (the fitting procedure) we need to choose a convenient, differentiable loss function like cross entropy. We’ll touch on this more, below.

Our training dataset consists of grayscale images. Each image is 2 pixels wide by 2 pixels tall, each pixel representing an intensity between 0 (white) and 255 (black). If we label each pixel intensity as , , , , we can represent each image as a numeric vector which we can feed into our neural network.

ImageId | p1 | p2 | p3 | p4 | IsStairs |
---|---|---|---|---|---|

1 | 252 | 4 | 155 | 175 | TRUE |

2 | 175 | 10 | 186 | 200 | TRUE |

3 | 82 | 131 | 230 | 100 | FALSE |

… | … | … | … | … | … |

498 | 36 | 187 | 43 | 249 | FALSE |

499 | 1 | 160 | 169 | 242 | TRUE |

500 | 198 | 134 | 22 | 188 | FALSE |

For no particular reason, we’ll choose to include one hidden layer with two nodes. We’ll also include bias terms that feed into the hidden layer and bias terms that feed into the output layer. A rough sketch of our network currently looks like this.

Our goal is to find the best weights and biases that fit the training data. To make the optimization process a bit simpler, we’ll treat the bias terms as weights for an additional input node which we’ll fix equal to 1. Now we only have to optimize weights instead of weights *and* biases. This will reduce the number of objects/matrices we have to keep track of.

Finally, we’ll squash each incoming signal to the hidden layer with a sigmoid function and we’ll squash each incoming signal to the output layer with the softmax function to ensure the predictions for each sample are in the range [0, 1] and sum to 1.

Note here that we’re using the subscript to refer to the th training sample as it gets processed by the network. We use superscripts to denote the layer of the network. And for each weight matrix, the term represents the weight from the th node in the th layer to the th node in the th layer. Since keeping track of notation is tricky and critical, we will supplement our algebra with this sample of training data

ImageId | p1 | p2 | p3 | p4 | IsStairs |
---|---|---|---|---|---|

1 | 252 | 4 | 155 | 175 | TRUE |

2 | 175 | 10 | 186 | 200 | TRUE |

3 | 82 | 131 | 230 | 100 | FALSE |

4 | 115 | 138 | 80 | 88 | FALSE |

The matrices that go along with out neural network graph are

## Initializing the weights

Before we can start the gradient descent process that finds the *best* weights, we need to initialize the network with *random* weights. In this case, we’ll pick uniform random values between -0.01 and 0.01.

Is it possible to choose bad weights? Yes. Numeric stability often becomes an issue for neural networks and choosing bad weights can exacerbate the problem. There are methods of choosing good initial weights, but that is beyond the scope of this article. (See this for more details.)

## Forward Pass

Now let’s walk through the forward pass to generate predictions for each of our training samples.

**Step 1**

Compute the signal going into the hidden layer,

**Step 2**

Squash the signal to the hidden layer with the sigmoid function to determine the inputs to the output layer,

**Step 3**

Calculate the signal going into the output layer,

**Step 4**

Squash the signal to the output layer with the softmax function to determine the predictions,

Recall that the softmax function is a mapping from to . In other words, it takes a vector as input and returns an equal size vector as output. For the th element of the output,

In our model, we apply the softmax function to each vector of predicted probabilities. In other words, we apply the softmax function “row-wise” to .

Running the forward pass on our sample data gives

## Backpropagation

Our strategy to find the optimal weights is gradient descent. Since we have a set of initial predictions for the training samples we’ll start by measuring the model’s current performance using our loss function, cross entropy. The loss associated with the th prediction would be

where iterates over the target classes.

Note here that is only affected by the prediction value associated with the True instance. For example, if we were doing a 3-class prediction problem and = [0, 1, 0], then = [0, 0.5, 0.5] and = [0.25, 0.5, 0.25] would both have .

The cross entropy loss of our entire training dataset would then be the average over all samples. For our training data, after our initial forward pass we’d have

ImageId | p1 | p2 | p3 | p4 | IsStairs | Yhat_Stairs | Yhat_Else | CE |
---|---|---|---|---|---|---|---|---|

1 | 252 | 4 | 155 | 175 | TRUE | 0.49865 | 0.50135 | 0.6958 |

2 | 175 | 10 | 186 | 200 | TRUE | 0.49836 | 0.50174 | 0.6966 |

3 | 82 | 131 | 230 | 100 | FALSE | 0.49757 | 0.50253 | 0.6881 |

4 | 115 | 138 | 80 | 88 | FALSE | 0.49838 | 0.50172 | 0.6897 |

Next, we need to determine how a “small” change in each of the weights would affect our current loss. In other words, we want to determine , , … which is the gradient of with respect to each of the weight matrices, and .

To start, recognize that where is the rate of change of [ of the th sample] with respect to weight . In light of this, let’s concentrate on calculating , “How much will of the first training sample change with respect to a small change in ?”. If we can calculate this, we can calculate and so forth, and then average the partials to determine the overall expected change in with respect to a small change in .

Recall our network diagram.

**Step 1**

Determine

Recall

So

**Step 2**

Determine

We need to determine expressions for the elements of

Recall

We can make use of the quotient rule to show

.

Hence,

Now we have

**Step 3**

Determine

**Step 4**

Determine

**Step 5**

Determine

Where is the tensor product that does “element-wise” multiplication between matrices.

Next we’ll use the fact that to deduce that the expression above is equivalent to

**Step 6**

Determine

**Recapping** we have

Now we have expressions that we can easily use to compute how cross entropy of the first training sample should change with respect to a small change in each of the weights. These formulas easily generalize to let us compute the change in cross entropy for every training sample as follows.

Notice how convenient these expressions are. We already know , , , and , and we calculated and during the forward pass. This happens because we smartly chose activation functions such that their derivative could be written as a function of their current value.

Following up with our sample training data, we’d have

Now we can update the weights by taking a small step in the direction of the negative gradient. In this case, we’ll let stepsize = 0.1 and make the following updates

For our sample data…

The updated weights are not guaranteed to produce a lower cross entropy error. It’s possible that we’ve stepped too far in the direction of the negative gradient. It’s also possible that, by updating every weight simultaneously, we’ve stepped in a bad direction. Remember, is the instantaneous rate of change of with respect to **under the assumption that every other weight stays fixed**. However, we’re updating all the weights at the same time. In general this shouldn’t be a problem, but occasionally it’ll cause increases in our loss as we update the weights.

## Wrapping it up

We started with random weights, measured their performance, and then updated them with (hopefully) better weights. The next step is to do this again and again, either a fixed number of times or until some convergence criteria is met.

## Challenge

Try implementing this network in code. I’ve done it in R here.

hey nice to read, found a mistake back propagation second last line it should be

[-(y11-yhat11) -(y12-yhat12)] you missed the brackets, took me some time to figure out.

There are some errors in the array at the beginning of the “Backpropagation” section, in the “Yhat_stairs” column. The probabilities of the 3 last examples are 0.0001 too high, this might lead to some confusion.

Hey Ben, thanks for the great tutorial. Very clear, clean and detailed.

One clarification: tensor product is not the same as “element-wise” multiplication between matrices. Tensor product produces a tensor of NxN dimension, where N is the total number of elements in each multiplied tensor, while elementwise product does not change it, leaning it at N. Just call it element-wise product and use a simple “x” to denote it, while use “.” for the scalar product.

Thanks,

Lev