HUMAN COMPUER INTERACTION: BY VISION

A Mini Project Report

On

HUMAN COMPUER INTERACTION: BY VISION

Submitted to

VISVESVARAYA TECHNOLOGICAL UNIVERSITY,

BELGAUM

In partial fulfillment of the completion of Fifth semester B.E course

Bachelor of Engineering
in
Computer Science and Engineering

by

PAVAN SHINDE 2BV08CS057

PRAMOD RAMDURG 2BV08CS062
PRASANNA GADDANAKERI 2BV08CS063
PRASANNA BHAT 2BV08CS064

Under the guidance of

Prof. Arjun D.

B.V.B.College of Engineering & Technology, Hubli

.

DEPARTMENT OF COMPUTER SCIENCE AND ENGINEERING

B.V.B.COLLEGE OF ENGINEERING & TECHNOLOGY, HUBLI
Academic year 2009-10
 Visvesvaraya Technological University
Belgaum

CERTIFICATE

This is to certify that Project entitled HUMAN COMPUER INTERACTION: BY VISION

Carried out by Pavan Shinde, Pramod D Ramdurg, Prasanna M G ,Prasanna Bhat

USN2BV08CS057,2BV08CS062,2BV08CS063,2BV08CS064 to the Visvesvaraya Technological

University, Belgaum, for the award of the Bachelor degree in Computer Science and Engineering

during the year 2010-2011, a bonafide work carried out by them.

The Project report has been approved as it satisfied the academic requirements in the respect

of project work prescribed for the above said course.

Project Guide Head of Department

Signature of Examiners

Name Signature with date

1.
2.

K.L.E. Society’s
B.V.BhoomaraddiCollege of Engineering & Technology, Hubli-580 031
Department of Computer Science and Engineering
ABSTRACT
This report describes a project that explores computervision based analysis for developing
newforms of Human Computer Interface (HCI). Intention of the project is to use computer vision to
complement existing modes ofhuman-interaction by making them more intuitive. Thus by doing
this system is made human like.
The camera acts as input device for the system by giving continuous images. And these
images are processed through different algorithms to recognise a particular activity. Once this is
done the system is made to take a particular action.
A prototype for interacting with 3D GRAPHICAL EDITORS by using hand gestures and
Trackpenhas been developed.The MULTI-SKETCHapplication allows multiple users to paint in the
same windowsimultaneously.Another application, EASY TRANSFER,can transfer data from
system to cell phone just by gestures and Trackpen.
ACKNOWELEDGEMENT:

We are grateful to Prof K.R. Biradar, Head of the Department of Computer science and
Engineering and our guide Prof Karuna Gull, Prof Arjun Gotur, Prof Mahadev for providing
adequate laboratory facilities and for their time to time suggestion which has helped to improve this
project.
We may take this opportunity to thank all the members of the teaching and non-teaching staff
of the department of Computer science and Engineering for their cooperation in completing this
mini project.
It would be the incomplete if we don’t thank to our friends and colleagues whose critical
appreciation and camaraderie helped as to stay put with the project till the end.
Dept of Computer Science and Engineering.

PAVAN SHINDE
PRAMOD RAMDURG
PRASANNA GADDANAKERI
PRASANNA BHAT
DATE: - 11/25/2010
CONTENTS
1 INTRODUCTION ----------------------------------- 1
1.1Problem statement 3
1.2Features 3
1.3Scope of project 3
1.4Limitation 3

2REQUIREMENT ANALYSISAND
SPECIFICATION------------------------------------- 4
2.1 Functional & nonfunctional requirement 4
2.1.1 Functional requirements
2.1.2 Non-functional requirements
2.2 Hardware & software requirements 4
2.3 software process model 5
2.4 Use cases for the system 5

3 DESIGN &IMPLEMENTATION ---------------- 6

3.1 Block & circuit Diagram 6
3.2 Dataflow diagram 7
3.2.1 DFD for Graphical editor 7
3.2.2 DFD for Multisketch 8
3.2.3 DFD for easy data transfer 9
3.3 Algorithms 10
3.4 Data structure description 12

4 TESTING & OUTPUT ----------------------------- 14

4.1 Test Plan and Test Case 14
4.2 Outputs 15
4.2.1 Tracking of Hand Gestures 15
4.2.2 Multisketch & Data Transfer 16

5 CONCLUSION & FUTURE WORK ------------ 17

5.1 Conclusion 17
5.2 Future work 17

6 REFERENCES ---------------------------------------- 18

7 APPENDIX----------------------------------------------- 19
HCI:BY VISION

Chapter 1 INTRODUCTION

1.1 Overview

1.1.1 Human Computer Interaction: By Vision

Utilizing computers had always begged the question of interfacing. The methods by which
human has been interacting with computers has travelled a long way. The journey still continues
and new designs of technologies and systems appear more and more every day and the research in
this area has been growing very fast in the last few decades. Domestic environments requires new
Human-computer interaction styles those are natural, convenient and efficient. They should be less
like the current desktop paradigm and more like the way we interact in the real world, using speech
and gesture.

1.1.2 Existing system and its disadvantages

In recent years many technological breakthroughs have occurred, but computers are still
limited in their multimedia understanding. This means that we have increased the effective
bandwidth of information from computers to humans, by sending audio, images, graphics, but the
same rate of improvement has not happened in computer understanding. Most computers still
receive input from low bandwidth devices like keyboards or mouse. Only few interfaces are able to
understand application related domains of audio, visual or haptic information.

1.1.3 Proposed System

The proposed system emphasizes on increasing bandwidth of information in computer

understanding.

Frames captured by the camera are taken as input. After processing those frames, system
will recognize the valid movements of hand or trackpen. In case of trackpen, blue colored LED is
tracked and its mapped coordinates are sent to mouse function of system. And in case of hand
gestures difference between two consecutive frames is calculated and binary image is generated out
of it, which gives the region of pixels in motion. Depending on the type of motion, desired task is
done. Block diagram of proposed system is given below.

5th Sem Mini Project Page 1

HCI:BY VISION

DESIRED TASK

GESTURES/
TRACKPEN CAMERA SYSTEM

Fig: Abstract diagram of the proposed system

Advantages of proposed system

 More intuitive than existing input devices.

 Natural way of playing games.

 Easy to draw pictures.

 Multi users can draw at same time.

 Can be used in the areas where there is need of motion sensing.

5th Sem Mini Project Page 2

HCI:BY VISION

1.1 Statement of problem

To design human computer interface, by giving vision to computer.

1.2 Features
 Allows users to interact with computer in a natural way.
 Multi sketch allows to draw multi user at a time.
 Data transfer allows easy data transfer to Bluetooth device.
 Track pen can be used as pointing device instead of mouse.

1.3 Scope of project

The main aim of our project is to simplify the use of computer. In this project we are
making input device more human friendly, so that it is more intuitive to use. This prototype can be
used in Gaming, 3D modelling, Personal computers, Presentations etc..

1.4 Limitation
 Does not work in absence of light or dim light.

5th Sem Mini Project Page 3

HCI:BY VISION

Chapter 2 REQUIREMENT ANALYSIS AND

SPECIFICATION
2.1 Functional & Non-functional Requirements

2.1.1 Functional Requirements

 User shall be able to perform the mouse operations by Track pen.

 User shall be able to operate graphical editors by hand gestures and track pen.
 User shall be able to play specified games using hand gestures without using mouse.
 User shall be given freedom to switch between mouse and virtual mouse.
 Multiple users can draw images at a time in Multisketch.
 User can transfer data to his mobile, by just selecting file and showing mobile to camera.

2.1.2 Non-functional requirements

 Hand gestures of the user should match the gestures which are fed to the software in code.
 User should operate within limited area in front of the camera.
 Control on the virtual mouse should not be affected if another hand appears within the range
of the camera.

2.2 Hardware & Software Requirements

2.2.1 Hardware requirements
System integrated with webcam, or usual webcam for desktop.

Hardware requirements for trackpen

 Red and blue LEDs
 10kΩ resistance pot.
 Switches.
 Marker pen.
 3V Batteries(4).
 Wires etc.

5th Sem Mini Project Page 4

HCI:BY VISION

2.2.2 Software requirements

 Microsoft visual C++ 2008 Express edition.
 Open source computer vision library.
 ACA colour picker

2.3 Software process model

 Project is implemented using Spiral development model.

2.4 USE Cases for the system

PRIMARY ACTOR: -User

PRECONDITION: - User performs gestures in front of camera.

POST CONDITION: - Particular action will be performed by the computer.

MAIN SUCCESS SCENARIO: - The user is allowed to do gestures.

ALTERNATE SCENARIO:- Not a valid gesture, action is not performed.

USE CASE DIAGRAM: -

Gesture Enable Display the out

System put

5th Sem Mini Project Page 5

HCI:BY VISION

Chapter 3 DESIGN AND IMPLEMENTATION

3.1 Block & Circuit Diagram

3.1.1 Block diagram

DESIRED TASK

GESTURES/
TRACKPEN CAMERA SYSTEM

3.1.2 Circuit diagram of Trackpen

Switch
Switch

Blue LED 3V 3V
Red LED
10kΩ 10kΩ

3.1.3 Implemented Trackpen

10kΩ variable
resistance

Blue LED’s
switch
Blue LED

Red LED

Red LED’s
switch

5th Sem Mini Project Page 6

HCI:BY VISION

3.2 Data flow diagram

3.2.1 Data flow diagram for graphical editor

User Gesture info

gestures

validity gesture

Track pen Gesture

recognition

Valid gesture

Graphical
editor window

Valid gesture

Select option
Draw using tracking pen
rectangle Other
Circle Line

Draw selected
option

Rotated camera view

Drawing Rotate camera

window

display

Monitor

5th Sem Mini Project Page 7

HCI:BY VISION

3.2.2 Data flow diagram for Multisketch

Gesture info

Gesture
recognition

gestures
gestures
gestures

User1 User2 User3

Colour 2
Colour 1 Colour 3

Painting
window Save painting
Select to save

display
display saved painting

Monitor

5th Sem Mini Project Page 8

HCI:BY VISION

3.2.3 Data flow diagram for easy data transfer

User

Select a file
by using track pen

Data
Transfer

Mobile in front
Of camera
Transfer from
computer

Mobile
PC suite
From PC suite
To mobile

5th Sem Mini Project Page 9

HCI:BY VISION

3.3 Algorithms
3.3.1 Algorithm for Graphical Editor

Begin(graphical editor){
frameNo=0;
trackPEN=false;
while(true){
srcImage=getNextImage();
trackPEN=isTrackingTrackPEN(sourceImage);
if(trackPEN)
trackTheTrackPEN(sourceImage)
else{
diffImage=frameDifference(sourceImage);
identify those pixels which are moving in diffImage and put them in binaryImage.
if(frameNo%3==0)
recogniseGesture(binaryImage);
}
frameNo++;
}//end while( true );
}
end //graphical editor

boolisTrackingTrackPEN( IplImage* sourceImage){

Search for blue color in the sourceImage;
if( found )
return true;
else
return false;

}//end

void trackTheTrackPEN(IplImage*sourceImage){
the blue color led is being tracked;
assign the x and y co-ordinates of this led to the mouse co-ordinates;
now in the vicinity of point(x,y) search for red led
if( found )
execute left click operation of mouse
}//end trackTheTrackPEN

IplImage*frameDifference(IplImage*sourceImage){
if(frameNo==1 )
tempImage=sourceImage;
else{
calculate the frame difference between tempImageand sourceImage;
put this difference in diffImage;
}
return DiffImage;
}

5th Sem Mini Project Page 10

HCI:BY VISION

3.3.2 Algorithm for transferring data to cell phone from PC

Begin (data transfer) {

Connect the pc and cell via Bluetooth
Point to the file which is to be transferred to cellphone by trackPEN
Bring the cellphone in the view of camera, selected file will be trasnsferred to cellphone
}
end //data transfer

3.3.3 Algorithm for Multisketch

Begin(multisketch){
Create new image called DrawImage
while(true){
sourceImage=getnextImage();
search for Red,Blue,Greencolours in the sourceImage;
get the co-ordinates of each color;
draw with these colors and co-ordinates in the DrawImage
}//end while
}
end //multisketch

5th Sem Mini Project Page 11

HCI:BY VISION

3.4 Data structure description

The main data structures used are IplImage, CvScalar and CvSize

IplImage
typedefstruct _IplImage{
intnSize;
int ID;
intnChannels;
intalphaChannel;
int depth;
char colorModel[4];
char channelSeq[4];
intdataOrder;
int origin;
int align;
int width;
int height;
struct _IplROI *roi;
struct _IplImage *maskROI;
void *imageId;
struct _IplTileInfo *tileInfo;
intimageSize;
char *imageData;
intwidthStep;
intBorderMode[4];
intBorderConst[4];
char *imageDataOrigin;
}
IplImage;

5th Sem Mini Project Page 12

HCI:BY VISION

CvScalar
A container for 1-,2-,3- or 4-tuples of doubles.
typedefstructCvScalar{
double val[4];
}
CvScalar;

CvSize
Pixel-accurate size of a rectangle.
typedefstructCvSize{
int width;
int height;
}
CvSize;

5th Sem Mini Project Page 13

HCI:BY VISION

Chapter 4 TESTING & OUTPUT

4.1 Test Plan and Test Case
Test Description I/p and expected Actual result Bug Description
case result
id
1 Functioning Each LED’s color All LEDs are visible as The camera is sensitive to
of TrackPen appears same in the white instead its particular both visible & IR
camera, as its color. radiation. Hence all
original color. different LEDs appeared
white since they emit IR
radiation.
2 Functioning A filter is used on LED tracking is working, Since visible radiation is
of TrackPen the lens of camera to but all visible radiations needed for hand
block unwanted were blocked by filter. recognition, the idea of
radiation. Because of which using filter is dropped.
recognize hand gestures
cannot be done.

3 Functioning LED bulb’s surface Proper Result -

of TrackPen is made rough so
that the visible light
from it is scattered
so that the actual
LED’s color can be
tracked in camera.
4 Hand Recognition of Proper Result -
gesture horizontal and
recognition. vertical movement
of hands.
5 Taking Movement of Proper Result -
action on camera in 3D
recognized modeling.
hand
gesture.

5th Sem Mini Project Page 14

HCI:BY VISION

4.2 Outputs

4.2.1 Tracking of Hand Gestures

Frame1
Frame2

FrameDifference=Frame2-Frame1 Binary image generated from

FrameDifference

Dilated Image Eroded Image

5th Sem Mini Project Page 15

HCI:BY VISION

4.2.2 Multisketch & Data Transfer

Output image: red, green

Input Image for tracking blue sketch pens are
detected, and its
coordinates are saved.

Tracked points are painted Data transfer by showing

paint window. Mobile: Mobile tracked in
the image

5th Sem Mini Project Page 16

HCI:BY VISION

Chapter 5 CONCLUSION & FUTURE WORK

5.1 Conclusion

This is project may lead to the new era of interaction between human and computers.
Handling computer will be very easy, user can handle computer in natural way without using
traditional mouse and keyboard in the fields like 3D modeling, gaming etc.

Especially for children, learning of computer becomes very joyful, interesting and easy.
In the field of gaming they can play games in a natural way, as they play with other humans. So this
project will be helpful in the coming future.

5.2 Future work

This is on-going work to make recognition of gestures more position independent, to

increase the tolerance for varying lighting conditions, and to increase recognition performance with
complex backgrounds.

5th Sem Mini Project Page 17

HCI:BY VISION

Chapter 6 REFERENCES

 OpenCV Reference manual.

 Learning OpenCV by O’Reily
 http://msdn.microsoft.com/en-us/library/default.aspx

5th Sem Mini Project Page 18

HCI:BY VISION

Chapter 7 APPENDIX

Introduction to new technologies

OpenCV 2.10

OpenCV means Open Source Computer Vision Library. It is a collection of C functions and
a few C++ classes that implement many popular Image Processing and Computer Vision
algorithms.

© 2010 by Pavan Shinde, Pramod R, Prasanna G, Prasanna Bhat.

5th Sem Mini Project Page 19