Developing ANN in Java Framework

The term neural network is usually meant to refer to artificial neural network (ANN). An artificial neural network attempts to simulate the real neural networks that are contained in the brains of all animals. Neural networks were introduced in the 1950’s and have experienced numerous setbacks, and have yet to deliver on the promise of simulating human thought. For processes that cannot easily be broken into a finite number of steps, a neural network can be an ideal solution .This article attempts to explore neural network programming through two mechanisms. First the reader is shown how to create a reusable neural network package that could be used in any Java program. Second, this reusable neural network package is applied to several real world problems that are commonly faced by IS programmers.

Introduction 

Artificial Intelligence (AI) is the field of Computer Science that attempts to give computers humanlike abilities. One of the primary means by which computers are endowed with humanlike abilities, is through the use of a neural network. The human brain is the ultimate example of a neural network. The human brain consists of a network of over a hundred billion interconnected neurons. Neurons are individual cells that can process small amounts of information and then activate other neurons to continue the process. 

The term neural network, as it is normally used, is actually a misnomer. Computers attempt to simulate an artificial neural network. However, most publications use the term “neural network” rather than “artificial neural network.” 

Constructing Biological Neural Network 

To construct a computer capable of “human like thought,” researchers used the only working model they had available—the human brain. To construct an artificial neural network, the brain is not considered as a whole. Taking the human brain as a whole would be far too complex. Rather, the individual cells that make up the human brain are studied. At the most basic level, the human brain is composed primarily of neuron cells. 
A neuron cell, as seen in Figure-1, is the basic building block of the human brain. It accepts signals from the dendrites. When a neuron accepts a signal, that neuron may fire. When a neuron fires, a signal is transmitted over the neuron’s axon. Ultimately the signal will leave the neuron as it travels to the axon terminals. The signal is then transmitted to other neurons or nerves. 

This signal, transmitted by the neuron is an analog signal. Most modern computers are digital machines, and thus require a digital signal. A digital computer processes information as either on or off. This is the basis of the binary digits zero and one. The presence of an electric signal represents a value of one, whereas the absence of an electrical signal represents a value of zero. Figure -2 shows a digital signal. 



Some of the early computers were analog rather than digital. An analog computer uses a much greater range of values than zero or one. This greater range is achieved as by increasing or decreasing the voltage of the signal. Figure-3 shows an analog signal. Though analog computers are useful for certain simulation activates, they are not suited to processing the large volumes of data that digital computers typically process. Because of this, nearly every computer in use today is digital. 



Biological Neural Networks are analog. Simulating analog neural networks on a digital computer can present some challenges. Neurons accept an analog signal through their dendrites. Because this signal is analog the voltage of this signal will vary. If the voltage is within a certain range, the neuron will fire. When a neuron fires, a new analog signal is transmitted from the firing neuron to other neurons. This signal is conducted over the firing neuron’s axon. The regions of input and output are called synapses. 

By firing or not firing, a neuron is making a decision. These are extremely low level decisions. It takes the decisions of a large number of such neurons to read this sentence. Higher level decisions are the result of the collective input and output of many neurons. 

These decisions can be represented graphically by charting the input and output of neurons. Figure-4 shows the input and output of a particular neuron.. As you can see from the graph shown in Figure-4, this neuron will fire at any input greater than 0.5 volts. 

As you can see, a biological neuron is capable of making basic decisions. This model is what artificial neural networks are based on. You will now be shown how this model is simulated using a digital computer. 

Simulating a Biological Neural Network with a Computer

Solving Problems with Neural Networks

As a programmer of neural networks you must know what problems are adaptable to neural networks. You must also be aware of what problems are not particularly well suited to neural networks. Like most computer technologies and techniques often the most important thing learned is when to use the technology and when not to. Neural networks are no different. 

A significant goal of this article is not only to show you how to construct neural networks, but also when to use neural networks. An effective neural network programmer knows what neural network structure, if any, is most applicable to a given problem. First, the problems that are not conducive to a neural network solution will be examined. 

Problems Not Suited to a Neural Network

Programs that are easily written out as a flowchart are an example of programs that are not well suited to neural networks. If your program consists of well defined steps, normal programming techniques will suffice. 

Another criterion to consider is whether the logic of your program is likely to change. The ability for a neural network to learn is one of the primary features of the neural network. If the algorithm used to solve your problem is an unchanging business rule there is no reason to use a neural network. It might be detrimental to your program if the neural network attempts to find a better solution, and begins to diverge from the expected output of the program. 

Finally, neural networks are often not suitable for problems where you must know exactly how the solution was derived. A neural network can become very useful for solving the problem for which the neural network was trained. But the neural network can not explain its reasoning. The neural network knows because it was trained to know. The neural network cannot explain how it followed a series of steps to derive the answer. 
Problems Suited to a Neural Network

Although there are many problems that neural networks are not suited for there are also many problems that a neural network is quite useful for solving. In addition, neural networks can often solve problems with fewer lines of code than a traditional programming algorithm. It is important to understand what these problems are.

Neural networks are particularly useful for solving problems that cannot be expressed as a series of steps, such as recognizing patterns, classifying into groups, series prediction and data mining. 

Pattern recognition is perhaps the most common use for neural networks. The neural network is presented a pattern. This could be an image, a sound, or any other sort of data. The neural network then attempts to determine if the input data matches a pattern that the neural network has memorized. 

Classification is a process that is closely related to pattern recognition. A neural network trained for classification is designed to take input samples and classify them into groups. These groups may be fuzzy, without clearly defined boundaries. These groups may also have quite rigid boundaries. 

Training Neural Networks

The individual neurons that make up a neural network are interconnected through the synapses. These connections allow the neurons to signal each other as information is processed. Not all connections are equal. Each connection is assigned a connection weight. If there is no connection between two neurons, then their connection weight is zero. These weights are what determine the output of the neural network. Therefore, it can be said that the connection weights form the memory of the neural network. 

Training is the process by which these connection weights are assigned. Most training algorithms begin by assigning random numbers to the weight matrix. Then the validity of the neural network is examined. Next, the weights are adjusted based on how valid the neural network performed. This process is repeated until the validation error is within an acceptable limit. There are many ways to train neural networks. Neural network training methods generally fall into the categories of supervised, unsupervised and various hybrid approaches. 
Supervised training is accomplished by giving the neural network a set of sample data along with the anticipated outputs from each of these samples. Supervised training is the most common form of neural network training. As supervised training proceeds, the neural network is taken through several iterations, or epochs, until the actual output of the neural network matches the anticipated output, with a reasonably small error. Each epoch is one pass through the training samples. 

Unsupervised training is similar to supervised training except that no anticipated outputs are provided. Unsupervised training usually occurs when the neural network is to classify the inputs into several groups. The training progresses through many epochs, just as in supervised training. As training progresses the classification groups are “discovered” by the neural network. 

There are several hybrid methods that combine several aspects of supervised and unsupervised training. One such method is called reinforcement training. In this method the neural network is provided with sample data that does not contain anticipated outputs, as is done with unsupervised training. However, for each output, the neural network is told whether the output was right or wrong given the input. 
It is very important to understand how to properly train a neural network. 

Validating Neural Networks

Once a neural network has been trained it must be evaluated to see if it is ready for actual use. This final step is important so that it can be determined if additional training is required. To correctly validate a neural network, validation data must be set aside that is completely separate from the training data. 

First training a neural network with a given sample set and also using this same set to predict the anticipated error of the neural network a new arbitrary set, will surely lead to bad results. The error achieved using the training set will almost always be substantially lower than the error on a new set of sample data. The integrity of the validation data must always be maintained. 

If the validation is performing badly this most likely means that there was data present in the validation set that was not available in the training data. The way that this situation should be solved is by trying a different, more random, way of separating the data into training and validation sets. If this fails, you must combine the training and validation sets into one large training set. Then new data must be acquired to serve as the validation data. 

For some situations it may be impossible to gather additional data to use as either training or validation data. If this is the case then you are left with no other choice but to combine all or part of the validation set with the training set. While this approach will forgo the security of a good validation, if additional data cannot be acquired this may be your only alterative. 

Programming ANN in Java Using the JOONE Editor

This section will show you how to create neural networks using the Java programming language. You will be introduced to the Java Object Oriented Neural Engine (JOONE). JOONE is an open source neural network engine written completely in Java. JOONE is distributed under Lesser GNU Public License, or LGPL. The lesser GNU public license means that JOONE may be freely used in both commercial and non-commercial projects without royalties, so long as you mention that you used JOONE. 

Setting Up Your Environment

Before you can begin using JOONE editor or engine you must make sure that the Java environment is correctly setup on your computer, and that JOONE is installed. The instructions for setting up your environment differ greatly depending on what sort of operating system you are running. JOONE consists of several components. At the heart of everything is the JOONE engine. The JOONE engine has no user interface and implements all of the low level functions of the network. If you write programs that make use of the JOONE system, these programs will most likely make use of the engine. JOONE also contains a GUI editor that can be used to quickly design and test neural networks. Figure -5 shows a UML diagram of some of the main JOONE Engine components. 

Regardless of if you are using the JOONE Editor or the JOONE Engine you will be using these components. These components are listed briefly here. As you build the neural networks in this section you will see how many of these objects are actually used.

· FileInputSymapse A synapse that is designed to input from a disk file. 
· FileOutputSynapse A synapse that is designed to output to a disk file. 
· InputSynapse A interface that defines a input synapse, or data source. 
· InputTokenizer A tokenizer us uses to transform underlying input formats into arrays of floating point numbers that will be fed to the neural network. 
· Layer The layer object is the basic element in JOONE. A layer transfers the input pattern to the output pattern while executing a transfer function. The output pattern is then sent to all output synapses that are attached to the layer. 
· LinerLayer A neuron layer that uses a simple linear function to activate 
· Monitor The Monitor is used to control the behavior of the neural net. It controls the start/stop actions and permit to set the parameters of the net. Each layer and synapse is connected to a monitor. 
· PatternTokenizer A interface that is used to define a tokenizer. 
· SigmoidLayer neuron layer that uses a sigmoid function to activate. 
· SimpleLayer This abstract class represents layers that are made up of neurons that implement some transfer function. 
· StreamInputSynapse This abstract class forms the base for any stream based input synapses. 
· StreamOutputSynapse This abstract class forms the base for any stream based output synapses. 
· Synapse The Synapse is the connection element between two Layer objects. Each connection between neurons in these layers is represented by a weight. The values of these weights are the memory of the neural network. 
· TanhLayer A neuron layer that uses the hyperbolic tangent to activate 
· URLInputSynapse A synapse that is designed to input from a HTTP connection. 

Hopfield Neural Network Example

The example given in this section implements the entire neural network. More complex neural network examples will often use JOONE. The classes used to create the Hopfield example are shown in Figure-6. 



Using the Hopfield Network

You will now be shown a Java program that implements a 4-neuron Hopfield neural network. This simple program is implemented as a Swing Java Application. Figure -7 shows the application as it appears when it initially starts up. Initially, the network activation weights are all zero. The network has learned no patterns at this point.



We will begin by teaching it to recognize the pattern 0101. Enter 0101 under the “Input pattern to run or train”. Click the “Train” button. Notice the weight matrix adjust to absorb the new knowledge. You should now see the same connection weight matrix as Figure-8. 



Now you will test it. Enter the pattern 0101 into the “Input pattern to run or train”(it should still be there from your training). The output will be “0101”. This is an autoassociative network, therefore it echoes the input if it recognizes it. 

Now you should try something that does not match the training pattern exactly. Enter the pattern “0100” and click “Run”. The output will now be “0101”. The neural network did not recognize “0100”, but the closest thing it knew was “0101”. It figured you made an error typing and attempted a correction. 

Now lets test the side effect mentioned previously. Enter “1010”, which is the binary inverse of what the network was trained with (“0101”). Hopfield networks always get trained for the binary inverse too. So if you enter “0101”, the network will recognize it. 
We will try one final test. Enter “1111”, which is totally off base and not close to anything the neural network knows. The neural network responds with “0000”, it did not try to correct, it has no idea what you mean. You can play with the network more. It can be taught more than one pattern. As you train new patterns it builds upon the matrix already in memory. Pressing “Clear” clears out the memory.

Constructing the Hopfield Example

Code-1: The Hopfield Application (Hopfield.java)

import java.awt.*;
import javax.swing.*;
import java.awt.event.*;

/**
* Example: The Hopfield Neural Network
* 
* This is an example that implements a Hopfield neural
* network. This example network contains four fully 
* connected neurons. This file, Hopfield, implements a
* Swing interface into the other two neural network
* classes: Layer and Neuron.

*/
public class Hopfield extends JFrame implements ActionListener {

/**
* The number of neurons in this neural network.
*/
public static final int NETWORK_SIZE = 4;

/**
* The weight matrix for the four fully connected
* neurons.
*/
JTextField matrix[][] = 
new JTextField[NETWORK_SIZE][NETWORK_SIZE];


/**
* The input pattern, used to either train
* or run the neural network. When the network
* is being trained, this is the training 
* data. When the neural network is to be ran
* this is the input pattern.
*/
JComboBox input[] = new JComboBox[NETWORK_SIZE];

/**
* The output from each of the four neurons.
*/
JTextField output[] = new JTextField[NETWORK_SIZE];

/**
* The clear button. Used to clear the weight
* matrix.
*/
JButton btnClear = new JButton("Clear");

/**
* The train button. Used to train the
* neural network.
*/
JButton btnTrain = new JButton("Train");

/**
* The run button. Used to run the neural 
* network.
*/
JButton btnRun = new JButton("Run");

/**
* Constructor, create all of the components and position
* the JFrame to the center of the screen.
*/
public Hopfield()
{ 
setTitle("Hopfield Neural Network");

// create connections panel
JPanel connections = new JPanel();
connections.setLayout( 
new GridLayout(NETWORK_SIZE,NETWORK_SIZE) );
for ( int row=0;row<NETWORK_SIZE;row++ ) {
for ( int col=0;col<NETWORK_SIZE;col++ ) {
matrix[row][col] = new JTextField(3);
matrix[row][col].setText("0");
connections.add(matrix[row][col]);
}
}

Container content = getContentPane();

GridBagLayout gridbag = new GridBagLayout();
GridBagConstraints c = new GridBagConstraints();
content.setLayout(gridbag);

c.fill = GridBagConstraints.NONE;
c.weightx = 1.0;

// Weight matrix label
c.gridwidth = GridBagConstraints.REMAINDER; //end row
c.anchor = GridBagConstraints.NORTHWEST;
content.add(
new JLabel(
"Weight Matrix for the Hopfield Neural Network:"),c); 

// Weight matrix
c.anchor = GridBagConstraints.CENTER;
c.gridwidth = GridBagConstraints.REMAINDER; //end row
content.add(connections,c);
c.gridwidth = 1;

// Input pattern label
c.anchor = GridBagConstraints.NORTHWEST;
c.gridwidth = GridBagConstraints.REMAINDER; //end row
content.add(
new JLabel(
"Click \"Train\" to train the following pattern:"),c); 

// Input pattern

String options[] = { "0","1"};

JPanel inputPanel = new JPanel();
inputPanel.setLayout(new FlowLayout());
for ( int i=0;i<NETWORK_SIZE;i++ ) {
input[i] = new JComboBox(options);
inputPanel.add(input[i]);

}

c.gridwidth = GridBagConstraints.REMAINDER; //end row
c.anchor = GridBagConstraints.CENTER;
content.add(inputPanel,c); 

// Output pattern label
c.anchor = GridBagConstraints.NORTHWEST;
c.gridwidth = GridBagConstraints.REMAINDER; //end row
content.add(
new JLabel("Click \"Run\" to see output pattern:"),c); 

// Output pattern

JPanel outputPanel = new JPanel();
outputPanel.setLayout(new FlowLayout());
for ( int i=0;i<NETWORK_SIZE;i++ ) {
output[i] = new JTextField(3);
output[i].setEditable(false);
outputPanel.add(output[i]);
}
c.gridwidth = GridBagConstraints.REMAINDER; //end row
c.anchor = GridBagConstraints.CENTER;
content.add(outputPanel,c);

// Buttons

JPanel buttonPanel = new JPanel();
btnClear = new JButton("Clear");
btnTrain = new JButton("Train");
btnRun = new JButton("Run");
btnClear.addActionListener(this);
btnTrain.addActionListener(this);
btnRun.addActionListener(this);
buttonPanel.setLayout(new FlowLayout());
buttonPanel.add(btnClear);
buttonPanel.add(btnTrain);
buttonPanel.add(btnRun);
content.add(buttonPanel,c);

// adjust size and position 
pack();
Toolkit toolkit = Toolkit.getDefaultToolkit();
Dimension d = toolkit.getScreenSize();
setLocation(
(int)(d.width-this.getSize().getWidth())/2,
(int)(d.height-this.getSize().getHeight())/2 );
setDefaultCloseOperation(WindowConstants.DISPOSE_ON_CLOSE);
setResizable(false);


}

/**
* Used to dispatch events from the buttons
* to the handler methods.
* 
* @param e The event
*/
public void actionPerformed(ActionEvent e)
{
if ( e.getSource()==btnRun )
run();
else if ( e.getSource()==btnClear )
clear();
else if ( e.getSource()==btnTrain )
train();
}

/**
* Called when the neural network is to be ran against
* the input.
*/
protected void run()
{
boolean pattern[] = new boolean[NETWORK_SIZE];
int wt[][] = new int[NETWORK_SIZE][NETWORK_SIZE];

for ( int row=0;row<NETWORK_SIZE;row++ )
for ( int col=0;col<NETWORK_SIZE;col++ )
wt[row][col]=Integer.parseInt(matrix[row][col].getText());
for ( int row=0;row<NETWORK_SIZE;row++ ) {
int i = input[row].getSelectedIndex();
if ( i==0 )
pattern[row] = false;
else
pattern[row] = true;
}

Layer net = new Layer(wt);
net.activation(pattern);

for ( int row=0;row<NETWORK_SIZE;row++ ) {
if ( net.output[row] )
output[row].setText("1");
else
output[row].setText("0");
if ( net.output[row]==pattern[row] )
output[row].setBackground(java.awt.Color.green);
else
output[row].setBackground(java.awt.Color.red);
}

}

/**
* Called to clear the weight matrix.
*/
protected void clear()
{
for ( int row=0;row<NETWORK_SIZE;row++ )
for ( int col=0;col<NETWORK_SIZE;col++ )
matrix[row][col].setText("0"); 
}

/**
* Called to train the weight matrix based on the
* current input pattern.
*/
protected void train()
{
int work[][] = new int[NETWORK_SIZE][NETWORK_SIZE];
int bi[] = new int[NETWORK_SIZE];

for ( int x=0;x<NETWORK_SIZE;x++ ) {
if ( input[x].getSelectedIndex()==0 )
bi[x] = -1;
else
bi[x] = 1;
}

for ( int row=0;row<NETWORK_SIZE;row++ )
for ( int col=0;col<NETWORK_SIZE;col++ ) {
work[row][col] = bi[row]*bi[col];
}

for ( int x=0;x<NETWORK_SIZE;x++ )
work[x][x] =-1;

for ( int row=0;row<NETWORK_SIZE;row++ )
for ( int col=0;col<NETWORK_SIZE;col++ ) {
int i = Integer.parseInt(matrix[row][col].getText());
matrix[row][col].setText( "" + (i+work[row][col]));
}

}

/**
* Main program entry point, display the 
* frame.
* 
* @param args Command line arguments are not used
*/
static public void main(String args[])
{
JFrame f = new Hopfield();
f.show(); 
}
}


Code-2: The Layer Class (Layer.java)

public class Layer {

/**
* An array of neurons.
*/
protected Neuron neuron[] = new Neuron[4];

/**
* The output of the neurons.
*/
protected boolean output[] = new boolean[4];

/**
* The number of neurons in this layer. And because this is a
* single layer neural network, this is also the number of
* neurons in the network.
*/
protected int neurons;

/**
* A constant to multiply against the threshold function.
* This is not used, and is set to 1.
*/
public static final double lambda = 1.0;

/**
* The constructor. The weight matrix for the
* neurons must be passed in. Because this is
* a single layer network the weight array should
* always be perfectly square(i.e. 4x4). These
* weights are used to initialize the neurons.
* 
* @param weights A 2d array that contains the weights 
* between each
* neuron and the other neurons.
*/
Layer(int weights[][])
{
neurons = weights[0].length;

neuron = new Neuron[neurons];
output = new boolean[neurons];

for ( int i=0;i<neurons;i++ )
neuron[i]=new Neuron(weights[i]);
}

/**
* The threshold method is used to determine if the neural
* network will fire for a given pattern. This threshold
* uses the hyperbolic tangent (tanh).
* 
* @param k The product of the neuron weights and the input
* pattern.
* @return Whether to fire or not to fire.
*/
public boolean threshold(int k)
{ 
double kk = k * lambda;
double a = Math.exp( kk );
double b = Math.exp( -kk );
double tanh = (a-b)/(a+b);
return(tanh>=0);
}

/**
* This method is called to actually run the neural network.
* 
* @param pattern The input pattern to present to the 
* neural network.
*/
void activation(boolean pattern[])
{
int i,j;
for ( i=0;i<4;i++ ) {
neuron[i].activation = neuron[i].act(pattern);
output[i] = threshold(neuron[i].activation);
}
}
}


Code-3: The Neuron Class (Neuron.java)
public class Neuron {

/**
* The weights between this neuron and the other neurons on
* the layer.
*/
public int weightv[];

/**
* Activation results for this neuron.
*/
public int activation;

/**
* The constructor. The weights between this neuron and 
* every other neuron(including itself) is passed in as
* an array. Usually the weight between this neuron and
* itself is zero.
* 
* @param in The weight vector.
*/
public Neuron(int in[])
{
weightv = in;
}

/**
* This method is called to determine if the neuron would
* activate, or fire.
* 
* @param x Neuron input
* @return If the neuron would activate, or fire
*/
public int act(boolean x[] )
{
int i;
int a=0;

for ( i=0;i<x.length;i++ )
if ( x[i] )
a+=weightv[i];
return a;
}

}


Code-3: The Neuron Class (Neuron.java)

public class Neuron {

/**
* The weights between this neuron and the other neurons on
* the layer.
*/
public int weightv[];

/**
* Activation results for this neuron.
*/
public int activation;

/**
* The constructor. The weights between this neuron and 
* every other neuron(including itself) is passed in as
* an array. Usually the weight between this neuron and
* itself is zero.
* 
* @param in The weight vector.
*/
public Neuron(int in[])
{
weightv = in;
}

/**
* This method is called to determine if the neuron would
* activate, or fire.
* 
* @param x Neuron input
* @return If the neuron would activate, or fire
*/
public int act(boolean x[] )
{
int i;
int a=0;

for ( i=0;i<x.length;i++ )
if ( x[i] )
a+=weightv[i];
return a;
}

}


Conclusion

JOONE includes a graphical editor that allows you visually create a neural network. This editor works fine for prototyping networks but neural networks created by the editor cannot easily be incorporated into an actual Java program. To incorporate JOONE neural networks into an actual Java program the JOONE Engine must be used.

About Authors:

Sunil Kr.Pandey 
1-Department of Computer Science,SMS,Varanasi,India
E-mail:sunilmca5@rediffmail.com

R.B.Mishra
2-Department of Computer Engineering ,IT,BHU,Varanasi,India
E-mail:rbmbhu@yahoo.com