This is a blog on neural networks that explains the basic concepts of neural networks and how they operate.

What are Neural Networks?

Human brain is a very complex structure made up of several neurons that pass information to one another. Neural networks or artifical neural networks (ANNs) are an attempt to recreate this structure so that even computers can learn some things that we may even consider menial.

It is very useful when it comes to identifying patterns/relationships in any given dataset.

How do Neural Networks work?

A neural network is a collection of connected nodes called artificial neurons (nodes). A network made up of just one node is called a perceptron. Neural networks are loosely based on the neurons in a biological brain as you can see in the image below.

Each connection or edge in a network, like the synapses in a biological brain, can transmit a signal to other neurons. A node that receives a signal then processes it and can signal other nodes connected to it.

Nodes and edges have weights attached to them which are initially randomly set but later adjusted based on their ‘learning’ of the network.

The ‘signal’ at each edge is a real number and it’s output at each node is computed using a mathematical formula:

$$\displaystyle y = \displaystyle\sum_{i=0}^n {w_ix_i} + bias$$
$$\displaystyle \begin{array}\\ \text{Here,}\\ n & = & \text{number of input edges}\\ w & = & \text{weight of each input edge}\\ x & = & \text{input value of each edge}\\ bias & = & \text{constant added to make the neural network more flexible}\\ \end{array}$$

Based on this formula, decisions of each individual neuron are made.

$$\displaystyle \begin{array}{ll} y = \begin{cases} 1 & {\text{if}}\ \sum{w_ix_i} + b > 0\\ 0 & {\text{if}}\ \sum{w_ix_i} + b \leq 0\\ \end{cases} \end{array}$$

If y = 1 then the neuron is activated and the value is send forward to the next neuron. If a neuron is an output neuron, it will predict if the output true or not. For example if a perceptron is meant to detect if it will rain or not based on the wind speed, humiity and temperature, it will take these three values as input and use the formula to get the value of y. If y > 0 then the perceptron predicts that it will rain or else it predicts that it will not rain.

Bigger neural networks are formed with multiple neurons as can be seen in the image below:

Neural Network

These typically have 3 parts - Input layer, Hidden layers and Output layer. Th input layer and the output layer consists of a single layer that is defined by the programmer. The number of nodes in the input layer correspond to the number of inputs given to the network whereas the output layer corresponds to the number of outputs expected (The number of prediction classes). The hidden layers on the other hand are the layer where all the computation takes place.

While training neural networks, there are two major steps involved:

  1. Forward Propogation
  2. Back Propogation

Forward Propogation

This is the processes by which the input data is fed foward to the nuerons of the hidden layer. At the first run, the weights are randomly assigned. Hence, based on the weights assigned and the input data, the output nodes are predicted.

Back Propogation

Most times, the output formed in the very first computation will not be correct as the weights are randomly set. Hence, in back propogation the difference between the output generated by network and the actual outpt is calculated. This is called loss and is calculated by a loss function. The network now sends back this loss value and recalculates all the weights of the network. This process is called backpropogation. This is how the neural network ‘learns’ form the data - by adjusting it’s weights based on the dataset.

Back Propogation

Both these processes are occur one after the other several times until an optimal loss is reached or a fixed time has passed. Once either of these two conditions are reached, training of the network is complete. The weights of the network can be saved for furture use.