Autonomous Land Vehicle In a

Neural Network
Seminar report submitted in partial fulfillment of the requirements for the award

of degree

of

BACHELOR OF TECHNOLOGY

IN

COMPUTER SCIENCE ENGINEERING

(2006-2010)

Submitted By

M.GIRIDHAR.GOWTHAM
06B11A0550

DEPARTMENT OF COMPUTER SCIENCE AND ENGINEERING

AL-AMEER COLLEGE OF ENGINEERING & INFORMATION TECHNOLOGY

(Affiliated to Jawaharlal Nehru Technological University, Kakinada, A.P)

Gudilova, Anandapuram, Visakhapatnam-531173

Abstract :

ALVINN (Autonomous Land Vehicle in a Neural Network) is a back propagation based neural

architecture Which has been used to successfully drive a car along a highway at highway speeds .

The input to ALVINN is a 30x32 images taken by a camera connected to a digitizer. The output is a

steering direction, which is accomplished By a motor connected to the steering column. One of the

main concerns with ALVINN has been the Speed at which it learns to drive in new situations.

While it is capable of on-line learning, the training time is Usually on the order of 2-3 minutes. The

speed is influenced by many factors: the training algorithm, the training set size, the dynamic

training set management, and the computational power available. In the 5 years since the original

ALVINN work was performed, not only has computational power increased, other learning

methods have also been developed which may allow a speedup in learning.

The motivation for looking at this is to see whether it is viable to

return to actively using ALVINN as a supplement to the RALPH (Rapid Adaptive Lateral Position

Handler) lane tracking system on Navlab . One of the main reasons ALVINN was replaced was the

long learning time when dealing with new road conditions,coupled with the requirement of driver

intervention during that period. It may be possible to remove both constraints by using faster

learning algorithms, and by using RALPH to train ALVINN when road conditions change. After

RALPH trained ALVINN, the neural net’s generalization capabilities would ensure that we don’t

have simply two copies of RALPH running.

To this end, we measure the performance of three learning algorithms on learning ALVINN.

The algorithms are quickprop, cascade correlation, and cascade2. The performance of these three

algorithms is compared against a backprop baseline. In addition, we also use hidden unit analysis to

determine how the network learns.

INDEX
Contents: Page No

1. Overview 1

2. On the Road with Alvinn 3

3. Technical Specification 6

Overview:
 This sensor generates heading information required to steer a robotic vehicle by

"watching" the road. The processing performed on chip is ALVINN (Autonomous Land Vehicle In

a Neural Network), a neural network trained to drive without human intervention on public

highways. Circuitry for neural computations is integrated with a photo sensor array using VLSI in

order to directly sense road-image information.

Image-based control of a vehicle at high speeds is a demanding real-time task. While an image

sensor generates vast amounts of data, only a small fraction of the information is relevant. Human

drivers use their experience to extract needed information from what they see. The ALVINN neural

network provides a similar capability, extracting information required to stay on the road from

converted intensity images. Through a training process, the network learns to filter out image

details not relevant to driving. However, current implementations of ALVINN rely on conventional

sense-then-process vision methods that must needlessly digitize, transfer and process full video

frames.

VLSI technology provides the opportunity to integrate the imaging and computation required by the

ALVINN task. The resulting computational sensor intelligently extracts relevant information from

raw image input at the point of sensing. The bottleneck between image input and computer, present

in traditional system implementations, is eliminated. Local processing of image information reduces

system latency while increasing data throughput --- meeting the fundamental requirements of real-

time robotic-vision tasks. In addition, computational sensors are compact, rugged and cost-effective

because they are implemented on a monolithic silicon substrate.

Prior to ALVINN-on-a-chip, significant bandwidth and computation were wasted transferring and

processing image data from video cameras. As a result, system throughput was limited to only 10
frames / second. Much higher frame rates are required to obtain further gains in the speed and

performance of the driving task. Latency is another serious problem alleviated by a VLSI

implementation. Applications, like ALVINN, are sensitive to the real-time nature of the images,

and excessive latency limits system stability. When video cameras and frame stores are used, the

image data available to update vehicle heading is that taken by the camera several frames back.

While pipelining can improve system throughput, the latency in an imaging system built around a

frame store cannot be eliminated.

VLSI integration of the ALVINN system provides a practical, yet challenging, application which

combines and builds on our expertise in computational sensors, real-time connectionist image

processing and autonomous mobile systems. An intelligent, rapidly programmable sensor for

neural-network based imaging that is fast, cost-effective, and compact will be the result. Our

strategy is to simultaneously advance the technology of neural-network based imaging as we further

investigate the potential of VLSI-based computational sensors.

ON THE ROAD WITH ALVINN

The divided highway lances through the beauties of a Pennsylvania dawn, morning frost glinting

across the hills below. It's hard to enjoy the natural wonders, though, when ALVINN is behind the

wheel, doing 88 km/h in the fast lane. First he lurches right, crossing both lanes of the blacktop and

alarming bleary-eyed commuters trying to share the road. Then he careens to the left, skidding onto

the gravel shoulder and nearly into a ditch. Finally Todd Jochem, 27, a graduate student at

Pittsburgh's Carnegie Mellon University, wrests the wheel of the four-wheel-drive Humvee from

ALVINN while Dean Pomerleau, a C.M.U. robotics research scientist, makes excuses for their

friend's driving. ``I guess he's a little confused,'' says Pomerleau. ``We'll let him try again in a

minute.'' Still, ALVINN is a marvel--a road-hogging computer able to drive like a human. Well,

almost.

3
ALVINN, short for Autonomous Land Vehicle in the Neural Network, originated in 1985 as a

research project by C.M.U.'s Robotics Institute for the U.S. military. Early progress aroused the

U.S. Department of Transportation and U.S. automakers to the fascinating possibilities of fully

automated passenger vehicles. The lean-back comfort of a train meets the door-to-door convenience

of a car. Automobile travelers would be able to key in their destination, flip open a magazine and

leave the actual driving to ALVINN. Last October the DOT committed $160 million to an

Automated Highway System research consortium, which includes the C.M.U. team. A European

coalition is working on similar technology called Prometheus; Japanese automakers are also

tinkering with prototypes.

At the moment, ALVINN can go only where he has gone before. The technology-- cameras, laser

range finders, sonar and motors--works in three stages. First ALVINN ``sees'' the road by taping it

as well as sending out laser and sonar waves as Jochem drives. Then the computer goes into

training, replaying the tape thousands of times, studying the details--road signs, lane markers, speed

limits. Finally ALVINN drives a stretch of road, handling his own acceleration, braking and

steering.

Right now ALVINN is barely a student driver. Small but unexpected events cause him to swerve

and shake in gigabyte panic. His performance may improve this month when ``smart car'' map

scanning is integrated into the programming. ``There should be a substantial difference,'' promises

Pomerleau. The researchers say they are only a couple of years from achieving a preliminary goal,

4
perfecting ALVINN as an anti-collision device to jog sleepy drivers before they run off the road.

When a car drifts dangerously close to the border of its lane, a road-watching camera would do

something like ring an alarm bell or shake the driver's seat. ``Fully automated driving may be a

ways off,'' says Pomerleau, ``but in the near term we're doing work to save lives.'' For now,

however, if you see ALVINN coming at you on the road, you might want to keep a respectful

distance.

Training the vehicle on a fly over

5
Technical Specifications:

ALVINN (Autonomous Land Vehicle In a Neural Network)is a perception system which learns to

control the NAVLAB (National Autonomous Vehicles Lab) vehicles by watching a person drive.

ALVINN's architecture consists of a single hidden layer back-propagation network. The input layer

of the network is a 30x32 unit two dimensional ``retina'' which receives input from the vehicles

video camera. Each input unit is fully connected to a layer of five hidden units which are in turn

fully connected to a layer of 30 output units. The output layer is a linear representation of the

direction the vehicle should travel in order to keep the vehicle on the road.

6
To drive the vehicle, a video image from the onboard camera is injected into the input layer.

Activation is passed forward through the network and a steering command is read off the output

layer. The most active output unit determines the direction in which to steer.

To teach the network to steer, ALVINN is shown video images from the onboard camera as a

person drives, and told it should output the steering direction in which the person is currently

steering. The back-propagation algorithm alters the strengths of connections between the units so

that the network produces the appropriate steering response when presented with a video image of

the road ahead of the vehicle. After about 3 minutes of watching a person drive, ALVINN is able to

take over and continue driving on its own.

Because it is able to learn what image features are important for particular driving situations,

ALVINN has been successfully trained to drive in a wider variety of situations than other

autonomous navigation systems which require fixed, predefined features (like the road's center line)

for accurate driving. The situations ALVINN networks have been trained to handle include single

lane dirt roads, single lane paved bike paths, two lane suburban neighborhood streets, and lined

divided highways. In this last domain, ALVINN has successfully driven autonomously at speeds of

up to 70 mph, and for distances of over 90 miles on a public highway north of Pittsburgh.

Specialized networks are trained for each new road type. The networks are trained not only to

output the correct direction to steer, but an estimate of its reliability. ALVINN uses these reliability

estimates to select the most appropriate network for the current road type, and to switch networks as

the road type changes.

7
The current challenge for vision based on-road navigation researchers is to create systems that

maintain the performance of the existing lane keeping systems, while adding the ability to execute

tactical level driving tasks like lane transition and intersection detection and navigation.

There are many ways to add tactical functionality to a driving system. Solutions range from

developing task specific software modules to grafting additional functionality onto a basic lane

keeping system. Solutions like these are problematic because they either make reuse of acquired

knowledge difficult or impossible, or preclude the use of alternative lane keeping systems.

A more desirable solution is to develop a robust, lane keeper independent control scheme that

provides the functionality to execute tactical actions. Based on this hypothesis, techniques that are

used to execute tactical level driving tasks should:

• Be based on a single framework that is applicable to a variety of tactical level actions,

• Be extensible to other vision based lane keeping systems, and

• Require little or no modification of the lane keeping system with which it is being used.

A framework, called Virtual Active Vision, which provides this functionality through intelligent

control of the visual information presented to the lane keeping system, has been developed. Novel

solutions based on this framework for two classes of tactical driving tasks, lane transition and

intersection detection and traversal, are presented in detail. Specifically, algorithms which allow the

ALVINN lane keeping system to robustly execute lane transition maneuvers like lane changing,

entrance and exit ramp detection and traversal, and obstacle avoidance have been tested.

Additionally, with the aid of active camera control, the ALVINN system enhanced with Virtual

Active Vision tools can successfully detect and navigate basic road intersections.
Stimulating the steering error