Dealing with Complex Problems through Neural Networks

Neural Networks as a subject was the most difficult one to learn when I started taking interest in AI. Although, Internet was full of NN tutorials but they all seemed cryptic and too much mathematical. Anything which was available was hard to digest for a beginner in this field. I finally understood them and their practical applications. I immediately decided to write code in Visual Basic (my favourite language). After an hour of writing code and few more for tweaking it, I finally produced something which I had only dreamt of. There isn't a single Neural Network tutorial available for Visual Basic Programmers. That is why I decided to write this tutorial

Introduction

Neural Network or more appropriately Artificial Neural Network is basically a mathematical model of what goes in our mind (or brain). The brain of all the advanced living creatures consists of neurons, a basic cell, which when interconnected produces what we call Neural Network. The sole purpose of a Neuron is to receive electrical signals, accumulate them and see further if they are strong enough to pass forward. So simple in its basic functionality but the interconnections of these produces beings (me, u and others) capable of writing about them. The real thing lies not in neurons but the complex pattern in which they are interconnected. Neural Networks are just like a game of chess, easy to learn but hard to master. As the moves of chess are simple, yet the succession of moves is what makes the game complex and fun to play. Imagine a chess game in which you are allowed only one single move. Would that game be fun to play?

In the same way, a single neuron is useless. Well, practically useless. It is the complex connection between them and values attached with them which makes brains capable of thinking and having a sense of consciousness.

Basic Working

As explained earlier, a neuron is basically a cell which accumulates electrical signals with different strengths. What it does more is that it compares the accumulated signal with one predefined value unique to every neuron. This value is called bias. Well, now I think is time to explain with an image. So here is it:

The circles in the image represent neurons. This network or more appropriately this network topology is called feed-forward multi layered neural network. It is the most basic and most widely used network. I will be explaining this network only.
The network is called multi layered because it consists of more than two layers. The neurons are arranged in a number of layers, generally three. They are input, hidden/middle and output layers. The names signify the function of the layer.
This network is feed-forward, means the values are propagated in one direction only. There are many other topologies in which values can be looped or move in both forward and backward direction. But, this network allows the movement of values only from input layer to output layer. The functions of various layers are explained below:

·  Input layer: As it says, this layer takes the inputs (the values you pass) and forwards it to hidden layer. You can just imagine input layer as a group of neurons whose sole task is to pass the numeric inputs to the next level. Well it depends on the implementation but, in my implementation the numeric input has to be in range 0 and 1 (both inclusive). The larger the number, greater its strength. E.g. 0.51 is stronger than 0.39 but 0.93412 is stronger still. But, the interpretation of this strength depends upon the implementation and the problem NN is required to solve. E.g. for an OCR you connect every pixel with its respective input neuron and darker the pixel, higher the signal/input strength. In short shades of gray would correspond to input strength. But at last again, remember: Input layer never processes data, it just hands over it.

·  Middle layer: This layer is the real thing behind the network. Without this layer, network would not be capable of solving complex problems. There can be any number or middle or hidden layers. But, for most of the tasks, one is sufficient. The number of neurons in this layer is crucial. There is no formula for calculating the number, just hit and trial works. 
This layer takes the input from input layer, does some calculations and forwards to the next layer, in most cases it is the output layer. 

·   Output layer: This layer consists of neurons which output the result to you. This layer takes the value from the previous layer, does calculations and gives the final result. Basically, this layer is just like hidden layer but instead of passing values to the next layer, the values are treated as output. 

·   Dendrites: No! it is not some creature from X-Files, but a name given to straight lines joining two neurons of consecutive layers, which you can see in the image. They are just a passage (or method) through which values are passed from one layer to the next. There is a value attached with dendrite called weight. The weight associated with dendrites basically determines the importance of incoming value. A weight with larger value determines that the value from that particular neuron is of higher significance. To achieve this, what we need to do is multiply the incoming value with weight. So no matter how high the value is, if the weight is low the multiplication yields the final low value. I know it is hard to explain and understand, but take a look at code and you'll understand this concept more clearly. 
So now let’s get to the code and define the basic elements we'll need in NNs

‘Don’t forget to write option base 1 into the code
‘or else this net will not work           
Option Base 1
Option Explicit

Const e = 2.7183 'Mathematical const, used in sigmod function
'Dendrite connects one neuron to another and allows signal to pass from it

Private Type Dendrite 
Weight As Double 'Weight it has
End Type

Private Type Neuron 'The main thing
Dendrites() As Dendrite 'Array of Dendrites
DendriteCount As Long 'Number of dendrites
Bias As Double 'The bias
Value As Double 'The value to be passed to next layer of neurons
Delta As Double 'The delta of neuron (used while learning)
End Type

Private Type Layer 'Layer containing number of neurons
Neurons() As Neuron 'Neurons in the layer
NeuronCount As Long 'Number of neurons
End Type

Private Type NeuralNetwork
Layers() As Layer 'Layers in the network
LayerCount As Long 'Number of layers
LearningRate As Double 'The learning rate of the network
End Type

Dim Network As NeuralNetwork 'Our main network

The code above is pretty self explanatory. And for the things you haven't understood, don't worry that belongs to belong to the training section, which I would be explaining a little later on. The only thing that you need to understand now is bias. Bias is just another value or parameter associated with a neuron. In my and most of the implementations this value is added to the accumulated incoming value. You will better understand this when we see code for running the NN. For now see the code for creating the net.

'0 = Unsuccessful and 1 = Successful
Function CreateNet(LearningRate As Double, ArrayOfLayers As Variant) As Integer
Dim i, j, k As Integer
Network.LayerCount = UBound(ArrayOfLayers) 'Init number of layers
If Network.LayerCount < 2 Then 'Input and output layers must be there
CreateNet = 0 'Unsuccessful
Exit Function
End If

Network.LearningRate = LearningRate 'The learning rate
ReDim Network.Layers(Network.LayerCount) As Layer 'Redim the layers variable 
For i = 1 To UBound(ArrayOfLayers) ' Initialize all layers
DoEvents
Network.Layers(i).NeuronCount = ArrayOfLayers(i)
ReDim Network.Layers(i).Neurons(Network.Layers(i).NeuronCount) As Neuron
For j = 1 To ArrayOfLayers(i) 'Initialize all neurons
DoEvents

'We will not init dendrites for it because output layers do not have any

 If i = UBound(ArrayOfLayers) Then 
  Network.Layers(i).Neurons(j).Bias = GetRand 'Set the bias to random value
  Network.Layers(i).Neurons(j).DendriteCount = ArrayOfLayers(i - 1)
  ReDim Network.Layers(i).Neurons(j).Dendrites(Network.Layers(i).Neurons(j).DendriteCount) As Dendrite  
 'Redim the dendrite var

For k = 1 To ArrayOfLayers(i - 1)
DoEvents
Network.Layers(i).Neurons(j).Dendrites(k).Weight = GetRand 'Set the weight of each dendrite

Next k
ElseIf i = 1 Then 'Only init dendrites not bias
DoEvents 'Do nothing coz it is input layer

Else
 Network.Layers(i).Neurons(j).Bias = GetRand 'Set the bias to random value
 Network.Layers(i).Neurons(j).DendriteCount = ArrayOfLayers(i - 1)
 ReDim    Network.Layers(i).Neurons(j).Dendrites(Network.Layers(i).Neurons(j).DendriteCount) As Dendrite  
  'Redim the dendrite var

For k = 1 To ArrayOfLayers(i - 1)
DoEvents
Network.Layers(i).Neurons(j).Dendrites(k).Weight = GetRand 'Set the weight of each dendrite
Next k
End If
Next j
Next i
CreateNet = 1
End Function

Basically, what we are doing above is: 

· Creating an array of Neurons and dendrites. 
· Looping through all the neurons and assigning their respective biases a random value. 
· Looping through all the dendrites and assigning their respective weights a random value. 

Now let’s see, how we run a neural network:

What we have done above may be summed up as: 

· Assigning all the input neurons the values provided in the 'ArrayOfInputs' array. For every other type of neuron, the value assigned is 0 because we have to calculate that value. 
· Now for every hidden layer (also for output layer), the value is sum of values of previous layers multiplied with the weight of dendrites. 
· To the resulting value, add up the bias value. 
· Pass the resulting value through Activation function. 
· Give back the values of output neurons as final results. 

I would like to explain that activation function. What this function does is that, no matter what number you pass, it always returns a number between 0 and 1. This is because we want the final value of every neuron between 0 and 1, but what if it comes to be something like 2.5 or 1.25, etc. Thus, in order to avoid this incompatibility with standards we use the activation function. The function is something like:

Private Function Activation(Value As Double) 

'To crunch a number between 0 and 1

Activation = (1 / (1 + Exp(Value * -1)))
End Function

I will suppose you wouldn't like to get into the mathematical details. In case you like, you can easily google it and find out.

Training:

Training is the most important part of a neural network and the one consisting of the most mathematics. We'll be using Backpropagation method for training the NN. I would be explaining only the basic idea how it is done and not all the details. And you need not worry, just apply the formulae written in the code and you can get the training up and ready. Always remember, it is not necessary to know the details of any system in order to get something practical and working out of it. The best example illustrating this principle is Charles Darwin. Yes, at the time when he wrote 'On the Origin of Species', DNA was not known. So, he propounded the evolution without even knowing the method of how it is done i.e. how traits are passed on from parents to offspring. Back to NN training, first let’s see the code:

Function SupervisedTrain(inputdata As Variant, outputdata As Variant) As Integer '0=unsuccessful and 1 = successful

Dim i, j, k As Integer
If UBound(inputdata) <> Network.Layers(1).NeuronCount Then 
'Check if correct amount of input is given
 SupervisedTrain = 0
 Exit Function
End If

If UBound(outputdata) <> Network.Layers(Network.LayerCount).NeuronCount Then 
'Check if correct amount of output is given
SupervisedTrain = 0
Exit Function
End If

Call Run(inputdata) 'Calculate values of all neurons and set the input Calculate delta's
For i = 1 To Network.Layers(Network.LayerCount).NeuronCount
DoEvents

'Deltas of Output layer

Network.Layers(Network.LayerCount).Neurons(i).Delta = Network.Layers(Network.LayerCount).Neurons(i).Value *
 (contd)* (1 - Network.Layers(Network.LayerCount).Neurons(i).Value) * (contd)* (outputdata(i) - 
Network.Layers(Network.LayerCount).Neurons(i).Value) 

For j = Network.LayerCount - 1 To 2 Step -1
DoEvents

For k = 1 To Network.Layers(j).NeuronCount
DoEvents

Network.Layers(j).Neurons(k).Delta = Network.Layers(j).Neurons(k).Value * (contd)
* (1 - Network.Layers(j).Neurons(k).Value) * Network.Layers(j + 1).Neurons(i).Dendrites(k).Weight * (contd)
* Network.Layers(j + 1).Neurons(i).Delta 

'Deltas of Hidden Layers

Next k
Next j
Next i

For i = Network.LayerCount To 2 Step -1
DoEvents

For j = 1 To Network.Layers(i).NeuronCount
DoEvents

Network.Layers(i).Neurons(j).Bias = Network.Layers(i).Neurons(j).Bias + (contd)+ 
(Network.LearningRate * 1 * Network.Layers(i).Neurons(j).Delta)  'Calculate new bias

For k = 1 To Network.Layers(i).Neurons(j).DendriteCount
DoEvents

'Calculate new weights

Network.Layers(i).Neurons(j).Dendrites(k).Weight = Network.Layers(i).Neurons(j).Dendrites(k).Weight + 
(contd)(Network.LearningRate * Network.Layers(i - 1).Neurons(k).Value * Network.Layers(i).Neurons(j).Delta) 

Next k
Next j
Next i
SupervisedTrain = 1
End Function

So let’s see what we have here in arguments, inputdata a sample input array and outputdata, its corresponding output which we require the NN to produce. 

· Run the NN using inputdata as inputs. Now, we have all the values of individual neurons. 
· Loop through last layer to first layer. 
· Calculate the difference between output values desired and output values produced. 
· Using that difference, adjust the values of bias and weights accordingly. 
This training procedure must be repeated for larger number of samples so that our NN can produce accurate results for untrained input samples. 

Practical Applications

There are numerous applications of NNs limited only by your imagination. Innovation is the key to success, so use NNs to create something which will revolutionize the world! 
A few applications are given below: 

Ø Optical Character Recognization 
Ø Stock market prediction 
Ø Creating new art forms 
Ø Modeling human behavior 
Ø Loan risk analysis 
Ø Classification of patterns, Image analysis, music generation, etc, etc.
Ø E-mail spam filtering


Conclusion

There is more theory in this world about NNs than you can read in your lifetime. There are Hopfield networks, Recurrent networks, NeuroGenetics, etc. They are for people who really want to do serious research in this field. But in case, you want to read further, you can. Internet is a great free information highway. You can get anything you want on the internet if you know how to search effectively. Even if you don't, train a Neural Network to fetch NN tutorials for you :-)