History of neural network and the way it works

“We are making the historic transition from the age of scientific discovery to the

age of scientific mastery in which we will be able to manipulate and mold the nature

almost to our wishes.”

These are the words of Michio Kaku, (an American theoretical physicist

specializing in string field theory, and a futurist) in his movie “Vision of the future”. I

dare to use his words as an introduction since they really fit with the topic of our

paper.

In order to understand how neural network works now, one should know where

all this started from. The history of neural network is older than one can think. The

first to reveal to the world a theory about neural networking was Alexander Bain

(1818 - 1903) of the United Kingdom in his 1873 book entitled "Mind and Body. The

Theories of Their Relation". As stated in the paper of David D. Olmsted “History

and Principles of Neural Networks to 1960” Bain claimed that memory is actually a

“set of nerve currents weaker than that produced by the original stimulus” Further he

investigated the behavior of memory in connection with the neurons and discovered

that in order to recall a memory there should be a connection of neurons, that link

different memories one to another. In order to understand his idea, let’s take a simple

example. Let’s imagine an infant that hears the sound of a bell for the first time in his

life; at that time he wouldn’t be able to recognize that this is the sound of a bell,
simply because he doesn’t know what a bell is. Later, when that infant will grow up,

he will hear again that sound and see the bell, and by that time he will have some

other memories with the bell (for example from seeing it in some picture, or by being

told by someone what a bell is) and all this memories linked together in a very

complex process would allow his to recognize that this is a sound of a bell. This is

basically the Bain’s concept about how the memory works in complex

interconnections of the neurons. Later, based on the upper assumptions, Bain

developed a decision making model of a human brain. In his model he presented an

interaction between different ideas of the brain.

Simply speaking, Bain concludes that if our sensory fibers would have a limited

number of connection they would respond the same way to the external stimuli of the

same type; but in real life we don’t react the same to the external stimuli or when

making a decision. Taking a simple example we can be excited by both seeing a

person that we strongly like, and seeing a person that we strongly dislike. But we

wouldn’t behave the same way to the person that we dislike, as to the one that we

like.
 Because the hardwiring system that Bain proposed was too complex, and at

times impractical, being almost impossible the existence of that many connections in

the human brain, he proposed the idea that actually there is the matter of the length of

the connection , that makes us act differently. His discovery was called Adaptive

Rule. All this understanding of how human brain works, was and is very important if

one wants to make the machines act like people. What make people superior to

machines are the emotions, so in order to make machines act like people one should

implement emotions into machines. Bain, even though he started his researches far

before the machines, simulating humans, appeared, he knew very well that it is

extremely important to understand human emotions.

Later in 1890, the psychologist William James proposed that the actual behavior

of the neurons is described by a flow moving from excessively electrically charged

cells to those that have an electrical deficit, as stated in the same paper of David D.

Olmsted. William’s idea is very alike with the Bain’s Adaptive rule, James claiming:

"When two elementary brain-processes have been active together or in immediate

succession, one of them, on reoccurring, tends to propagate its excitement into the

other." (James 1890, p566) And the key concept in his theory was that memory could

be “created” by repetition of the same process, this being a very important fact for

later explanation of the neural network mechanism.

But this was not the end of the discoveries. The theories exposed earlier served as

a basis for further discoveries. In 1943 the neurophysiologist and cybernetician,

Warren Sturgis McCulloch, and the logician Walter Pitts, presented the model of

threshold logic. “In 1943 Warren McCulloch and Walter Pitts realized that the natural

consequence of the standard neuron model's threshold in combination with binary

action potentials produced another type of logic called threshold logic. Since each

action potential pulse is an all or nothing binary event a threshold value of 2 defines

an AND operation” as from the same paper of David D. Olmsted.

But all of the above, were theories or models of how brain works from inside,

and they weren’t exposed to practice. In order to develop a simulation of the human

brain, people needed to make sure that if implementing the above models into a non-

human, that non-human will behave like a human. So in 1954, with the launch of the

computer (in this case serving as the non-human) the first Randomly Connected

Reverberatory Networks was simulated. Based on the assumption proposed by the

professor Donald Hebb, that repetition is the key for a reflected action, or a driven-

back action, Belmont Greenlee Farley and Wesley Allison Clark were the first

ones to simulate a neural network system, using a computer. Their system was

composed of 4 nodes, which were actually simulating the neurons, in which inputs

were being summed up, and then generating an output. The system was divided in 4

quarters so the output from one would become the input for the other. “They

succeeded in running the first artificial neural network... They were

able to train their networks to recognize simple patterns. In

addition, they discovered that the random destruction of 10 percent

of the neurons did not affect the network's performance—a feature

that is reminiscent of the brain's ability to tolerate limited damage.”

As stated on the official site of The American Psychological Society.

A series of astonishing discoveries were made one after

another, some completing each other, and other’s overthrowing the

previous. In 1956 Nathan Rochester, who created a slightly different model of

neurons connections, concluded that neurons in our brain are randomly connected,

and thus, it’s almost impossible to imitate them. In 1958, Frank Rosenblatt claimed

“memory was simply a change in the relation between some input and some output

which changed with regular use”

The first neural pattern arrives in 1960, with the ADALINE concept developed

by Bernard Widrow and Marcian Hoff. It is mainly a system consisting of “a weight a

bias and a summation function” as by the data provided in the paper of David D.

Olmsted.

This is all about history, but how does it actually work nowadays? How does

the artificial neural network look like? As by the definition, the artificial network is a

simulation of something natural, and in this case it is the simulation of the biological

neural network. The biological neuron is made of the cell/body, which has 2

offshoots- dendrites and axon. A biological neuron has many dendrites that receive

signals from other neurons, while each neuron has only one axon, which is coming

out of the axon hillock and generates electrical signals out of the hillock, and these

signals are called action potentials. All these action potentials are similar electrical
signals which transmit information to the brain. The way brain differentiates them is

by the path that those signals took. The axon is insulated by a fatty pattern called

Myelin, but there are some parts of the axon which are uninsulated, and those parts

are called nodes. Within those nodes, the signal is regenerated, ensuring that the

signal is constant. The neurons connect with each other thru synapses, which is

basically a form of chemical matter through which the signal is transmitted from one

neuron to another. The artificial neuron also has synapses, but in its case they are

called weights. “The strength of the connection between an input and a neuron is

noted by the value of the weight.” There is a linear combination activity, when all the

inputs that are generated to the neuron are summed up by an adder, according to their

weights. In other words, one can imagine a packing automatic line, where there are

ribs packed and labeled with their weight, expiration date etc. So as they come up on

the line an automatic scale weighs them and issues a label with the amount of pounds.

In the artificial network the weights stand for scales, that measure the power of the

input, and then all the inputs are summed up by the adder. “Finally, an activation

function controls the amplitude of the output of the neuron. An acceptable range of

output is usually between 0 and 1, or -1 and 1” as stated on the site of Artificial

Neural Networks.

Basically the artificial neural network is a set of processing units that interact

with each other “over a large number of weighted connections”. Each unit receives a

certain input from external or neighboring units, or sends output and by this

assumption one distinguishes, input units (receives inputs from outside of the
neuron), output units (sends outputs outside of the neuron), and hidden units (receives

inputs from inside and sends outputs inside the neuron). Another function that is

processed inside the system is the adjustments of the weights, so to say, the

recognition of the power of the input. The system is basically a parallel one, or if one

neuron breaks down, the others will continue working, just as in the case of human

brain. The connection between the units can be feed-forward (no feedback is expected

back) and recurrent (connections involving feedback from each other).

In order this system to work; one should train it, so that it gives the expected

output to certain inputs. There are several ways of doing this. First, the weights can be

set properly, in accordance with the prior data. More simply it will sound to say that

the weights should be set to the amount that was reached in other examples of the

same input experiment. Another way of making it work is training it “by feeding it

teaching patterns and letting it change its weights according to some learning rule.”

As mentioned on the same site of Artificial Neural Networks. There are three types of

training:
• Supervised – match the input to the output

• Unsupervised – in this case the system should develop it’s own output as a

reaction to certain inputs

• Reinforcement- this is more linked to the assumption of William James that

repetition is the key of the neural process. This method predicts that the system

should be allowed to repeatedly stimulate some output and based on the feedback

it receives from the environment it should auto create some patterns.

 Sources

1 Michio Kaku, “Vision of the future”, BBC documentary, UK, 2007

2 “History and Principles of Neural Networks to 1960” , David D. Olmsted,

(Copyright - 1998, 1999, 2006. Free to use for personal and educational purposes)

Last revised August 26, 2006, http://neurocomputing.org/NNHistoryTo1960.aspx

3. The site of American Psychological Society

 http://www.the-aps.org/membership/obituaries/belmont_farley.htm

4. http://en.wikipedia.org/wiki/ADALINE

5. The site of Artificial Neural Networks

<b><a href='http://www.learnartificialneuralnetworks.com'>Neural Networks -

Training Models and algorithms</a></b><br>

A guide to neural networks, complete explanations about training models and

algorithms, the biological model and mathematical model.

http://www.learnartificialneuralnetworks.com/#Intro