Controlling robot through gesture recognition

CHAPTER -1
 Controlling robot through gesture recognition

CHAPTER -1
INTRODUCTION

The project mainly focuses on the basis to implement hand gesture recognition technology
i.e., using Image processing using Raspberry pi board and control the dc motors according to
the data obtained from the images which are captured using a webcam. This model gives an
overview of Raspberry pi Arm-11 based processor board. The main features of Raspberry pi
are Broadcom BCM2835 Arm-11 processor (700 MHz), 512MB RAM, on board USB 2.0
ports. Providing a wide range of processors based on a common architecture that delivers
high performance and cost efficiency. These processor boards are capable to execute complex
operating systems such as Linux and can support complex graphic user interfaces
Gesture recognition is the process of recognizing and interpreting a stream of
continuous sequential gesture from the given set of input data. Gestures are non-verbal
information which is used to make computers understand human language and develop a user
friendly human computer interface. Human gestures are perceived through vision and this paper
aims to use computer vision to analyze different sets of gestures using human fingers and
interpret them in order to control the Robot.
Raspberry Pi is a Linux based platform which uses Python as the main programming language and
software development on Linux is easy as it is an open source code development environment. The
system consist of a Raspbian camera which continuously monitors and tracks the gestures.
The main aim is to create a framework with low cost and effective gesture interpretation system
which uses computer vision to analyze different sets of gestures or actions done using the human
fingers. Hand gesture recognition embedded system can be used as an interfacing medium between the
computer and human using different hand gestures in order to control the computer. Many approaches
have been made using cameras and computer vision algorithms to interpret sign language. However, the
identification and recognition of posture, gait, proxemics, and human behaviors is also the subject of
gesture recognition techniques.
 Controlling robot through gesture recognition

1.1 Raspberry Pi3

Raspberry pi is a Credit-card sized mini microprocessor that plugs into any monitor
(HDMI). A keyboard and mouse can be interfaced with the microprocessor with USB 2.0.
This microprocessor has a SD slot extendable up to 32 GB in which OS can be stored and
booted accordingly, we have used Raspbian OS. Ethernet Port or a USB Wi-Fi adaptor is
used to connect to the Internet [4]. GPIO pins allow us to control and interact with real world
sensors or component in this case motor driver (H-Bridge), ZigBee, etc.

Several generations of Raspberry Pis have been

released. The first generation (Raspberry Pi 1 Model B)
was released in February 2012. It was followed by a
simpler and inexpensive model Model A. In 2014, the
foundation released a board with an improved design
in Raspberry Pi 1 Model B+. These boards are
approximately credit-card sized and represent the
standard mainline form-factor. Improved A+ and B+
models were released a year later. A "compute
module" was released in April 2014 for embedded applications, and a Raspberry Pi Zero with smaller
size and reduced input/output (I/O) and general-purpose input/output (GPIO) capabilities was
released in November 2015 for US$5. The Raspberry Pi 2 which added more RAM was released in
February 2015. Raspberry Pi 3 Model B released in February 2016, is bundled with on-board Wi-
Fi, Bluetooth and USB boot capabilities.[11]As of January 2017, Raspberry Pi 3 Model B is the
newest mainline Raspberry Pi. Raspberry Pi boards are priced between US$535

The Raspberry Pi 3 has been announced, with the third iteration of the hugely popular
mini-computer boasting 50% more processing power. The latest Raspberry Pi is based on a
Broadcom BCM2837 System on a Chip , which includes the 64-bit quad-core ARM Cortex
A53 processor, giving the Pi 3 its power boost. The third iteration also adds built-in Wi-Fi
and Bluetooth - both highly-requested features - while sticking at a low price of $35
(30).There's also an integrated Video Core multimedia coprocessor which will allow for a
far more powerful graphical offering. The Raspberry Pi is a credit-card sized computer
which can be used for many of the things that your desktop PC does, like word-processing
and games. However one key aspect that makes the Raspberry Pi so brilliant for schools is its
ability to execute Python coded programmes.
 Controlling robot through gesture recognition

1.2 Python

Python is a wonderful and powerful programming language thats easy to use(easy to read and
write) and with Raspberry pi lets you connect to the project to the real world.Python is a widely
used high-level programming language for general-purpose programming, created by Guido van
Rossum and first released in 1991. An interpreted language, Python has a design philosophy which
emphasizes code readability (notably using whitespace indentation to delimit code blocks rather than
curly brackets or keywords), and a syntax which allows programmers to express concepts in fewer lines
of code than possible in languages such as C++ or Java.[22][23] The language provides constructs intended
to enable writing clear programs on both a small and large scale.

1.2 OpenCV

OpenCV-Python is a library of Python bindings designed to solve computer vision

problems. Python is a general purpose programming language started by Guido van Rossum that
became very popular very quickly, mainly because of its simplicity and code readability.

OpenCV was started at Intel in 1999 by Gary Bradsky, and the first release came out in
2000. Vadim Pisarevsky joined Gary Bradsky to manage Intel's Russian software OpenCV team. In
2005, OpenCV was used on Stanley, the vehicle that won the 2005 DARPA Grand Challenge. Later, its
active development continued under the support of Willow Garage with Gary Bradsky and Vadim
Pisarevsky leading the project.

OpenCV now supports a multitude of algorithms

related to Computer Vision and Machine Learning
and is expanding day by day.OpenCV supports a
wide variety of programming languages such as
C++, Python, Java, etc., and is available on
different platforms including Windows, Linux, OS
X, Android, and iOS. Interfaces for high-speed
GPU operations based on CUDA and OpenCV are
also under active development. OpenCV-Python is
the Python API for OpenCV, combining the best
qualities of the OpenCV C++ API and the Python language.
Controlling robot through gesture recognition

CHAPTER -2
 Controlling robot through gesture recognition

CHAPTER -2
LITERATURE SURVEY

We propose a system, using which the user can navigate the robot in the
environment using various gestures commands. The main objective is to provide reliable and
a more natural technique for the user to navigate controlled robot in the environment using
gestures.
In this system, user operates the robot from a control station that can be a laptop or a PC with
a good quality in-built webcam or external webcam. This webcam is used to capture real time
video stream of hand gestures to generate commands for the robot. Gesture commands are
given using hand palm. Mainly two kinds of gestures are used which are explained further.
Irrespective of the gesture technique used, robot is moved in all possible directions in the
environment using four possible types of commands which are Forward, Backward, Right
and Left. Image frame is taken as an input and processed using Image Processing. Processed
image is then used to extract the gesture command. This gesture command can have one of
the four possible commands as specified. From this generated gesture command, signal is
generated to pass the given command to the robot.
Generated signal is stored in the file at the control station. Raspberry Pi Ethernet on
robot accesses this file to transmit the signals from the control station to the robot. As soon as
the gets command from the control station, it is passed to the Raspberry pi. Raspberry pi
takes this signal as input from our pc or laptop and generates some output signals that are
passed to the motor driver. This output signal generation depends on the gesture input, for
every four possible gesture input, different output signal is generated. The motor driver is
used to drive the DC motors of the robot. It takes digital signals as the input from the
Raspberry Pi and gives these signals as an output to the DC motors. Once a command signal
is given to the robot, it continues to move in that direction till the next command is given or
any obstacle comes in the path.

The Raspberry Pi processes the received serial data and drives the DC motor. Webcam
mounted on the Robot captures the streaming video and uploads it to the local sever using the
Wi-Pi adaptor. This live streaming of the video can be viewed in the computer or mobile by
addressing the particular IP.
 Controlling robot through gesture recognition

Gesture are usually understood as hand and body movement which can pass information
from one to another. Since we are interested in hand gesture and so the term Gesture is
always referred to the hand gesture recognition in this paper. A Gesture is a sequence of
postures connected by motions over a short time span. Usually a gesture consists of one or
more posture sequentially occurring on the time axis. In this paper , Representing the
semantic component in sign language, GESTURE word can vocabulary are all equivalent
for a convenience of description. Furthermore , a sentence is define as sequence of gesture in
this paper.
 Controlling robot through gesture recognition

WORKING & PRINCIPLE

DC MOTOR 1

Raspberry Pi
MOTOR DRIVER Raspberry Pi 3 SERVO
WEB
L293D MOTOR CAM

DC MOTOR 2

Fig. Block Diagram Of Robot Section

IMPLEMENTATION

After we go through the literature survey we understand that there were so many ways to
recognize the hand gesture but the proposed system is mobile and effective technique to
recognize the hand gesture through web camera and Raspberry pi Microcontroller.

DESCRIPTION

Block diagram consists of Raspberry pi, Web camera,L293D and two DC motors as shown in Fig..
Web camera interfaced with Raspberry Pi through USB port and used to capture images. The captured
images will be processed through python OpenCV library. Interface DC motors to Raspberry Pi
Through an Motor Driver L293D. DC motors will move in Forward, Backward, Left and Right
according to number of fingers web camera captures.

INTRODUCTION OF RASPBERRY PI 3
 Controlling robot through gesture recognition

GESTURE IDENTIFICATION PROCESS

Heart of the system is to detect the gesture through image processing. We have to introduce
the new gesture reorganisation system using open CV software to detect the convexity of
hand. This following flowchart shows the new technique for Finger Identification. Steps for
the Finger reorganisation as given below

Configure the open CV with python in windows

Capturing the frames and displaying
Extract the ROI (Region of interest) from input frames or background subtraction
Find out the contour draw the convex hull
Find the convexity defects depending upon the number of defects and find out the
gesture