The Eye Writer Project

Mansoura University
Faculty of Engineering
Department of Electronics and
Communication Engineering
The Eye-Writer Project

Theoretical study & practical application of image processing
Under supervision of Dr. Ahmed Mohammed Abu-Taleb
Introduced to you by:

Mahmoud Adel Mahmoud
Hussein Zaghloul Mohammed
Sameh Nabil Elsharawy
Abdel Rahman Mohammed Abdel Halim
Ahmed Abdel Rahman Mutwally

Contents
Introduction ………………………………………………………………………………………………………………………001
0.1 Requirements to get through that course ……………………………………………………………………002

0.2 Programs to run software ……………………………………………………………………………………………004
Chapter 1
ALS ………………………………………………………………………………….…………………………………………………005
1.1 what is ALS ……………………………………………………………………………………………………………….….005

1.2 who gets ALS ………………………………………………………………………………………………………………..006
1.3 people who got ALS ……………………………………………………………………………………………………..007
Chapter 2
Overview of Eye-Writer ………………………………………………………………………………………………………..011
2.1 What is Eye-Writer ………………………………………………………………………………………………………………………011
2.2 Why Eye-Writer …………………………………………………………………………………………………………………………..011
Chapter 3
Hardware and interface of the Eye-writer ………………………………………………………………………….012
3.1 Functional design specifications …………………………………………………………………………………..012
3.2 Parts and tools …………………………………………………………………………………………………….……….013
3.3 Steps of interfacing and hardware solid work …………………………………………………………………………….021
3.4 Connect to the Eye-Writer Software …………………………………………………………………………………………..038
3.5 PS3 Eye drivers & QT components ……………………………………………………………………………………………..039
3.6 Video Capture …………………………………………………………………………………………………………………………….039
3.7 Eye-Tracking Software ………………………………………………………………………………………………………………..039
3.8 Eye-Drawing Software ………………………………………………………………………………………………………………..039
3.9 Eye-Writer installation ………………………………………………………………………………………………………………..040
3.10 On-Screen Drawing …………………………………………………………………………………………………………………..040

3.11 Co-located and Projected Drawing …………………………………………………………………………………………….041
3.12 Remote Drawing and Projection ………………………………………………………………………………………………..041
Chapter 4
Theoretical background about C++ ……………………………………………………………………………………..042
4.1 What is C++ …………………………………………………………………………………………………………………..042
4.2 Download, install, launch Code::Blocks …………………………………………………………………………042
4.3 Main structure of the program ……………………………………………………………………………………..046
4.4 Fundamental data types ……………………………………………………………………………………………………………..049
4.5 Declaration of variables ……………………………………………………………………………………………………………….049
4.6 Initialization of variables ……………………………………………………………………………………………………………..050
4.7 Operators ……………………………………………………………………………………………………………………………………051
4.8 Conditional structure: if and else ………………………………………………………………………………………………..057
4.9 Iteration structures (loops) …………………………………………………………………………………………………………058
4.10 Jump statements ………………………………………………………………………………………………………………………061
4.11 The selective structure: switch ………………………………………………………………………………………………….064
4.12 Functions (I) ………………………………………………………………………………………………………………………………064
4.13 Functions with no type: void functions ……………………………………………………………………………………..066
4.14 Functions (II) ……………………………………………………………………………………………………………………………..067
4.15 Declaring functions …………………………………………………………………………………………………………………….070
4.16 Arrays …………………………………………………………………………………………………………………………………………073
4.17 Character Sequences ………………………………………………………………………………………………………………….076
4.18 Pointers ………………………………………………………………………………………………………………………………………078
4.19 Classes ………………………………………………………………………………………………………………………………………..079
4.20 Friendship and inheritance …………………………………………………………………………………………………………081
4.21 Separating C++ program into 3 files …………………………………………………………………………………………….089

Chapter 5
Theoretical background about OpenCV …………………………………………………………………………….091
5.1 What is OpenCV …………………………………………………………………………………………………………..091
5.2 Building OpenCV from Source Using CMake, Using the Command Line ……………………….091
5.3 Installation in Windows …………………………………………………………………………………………………………094
5.4 Image Watch: viewing in-memory images in the Visual Studio debugger ………………………………….097
5.5 Load and Display an Image ………………………………………………………………………………………………………..097
5.6 Load, Modify, and Save an Image ………………………………………………………………………………………………098
5.7 How the image matrix is stored in the memory …………………………………………………………………………100
5.8 Discrete Fourier Transform ………………………………………………………………………………………………..………..103
5.9 Video Input with OpenCV and similarity measurement ………………………………………………………...……..106
5.10 How to read a video stream (online-camera or offline-file) ……………………………………………..…….…..113
5.11 Camera calibration With OpenCV ……………………………………………………………………………………………….114
5.12 The calibration and save ……………………………………………………………………………………………………………..121
Chapter 6
Theoretical background about OpenFrameworks …………..………………………………………………….. 124
6.1 What is openFrameworks ………………………………………..…………………………………………………….124
6.2 Functionality of openFrameworks ……………………………………………..…………………………………..124
6.3 Operating systems supported by openFrameworks …………………………………………………………124
6.4 Basics of openFrameworks ………………………………………………………….………………………………………………..125
6.5 Graphics ……………………………………………………………………………………………………………………………………….143
6.6 Video (load and play) …………………………………………………………………………………………………….151
6.7 Sound (load and play) …………………………………………………………………………………………………...153
6.8 Loading and 3-D modeling …………………………………………………………………………………………….155

6.9 Listening to events ………………………………………………………………………………………………………..155
Chapter 7
Software and codes …………………………………………………………………………………………………………158
7.1 Main file ……………………………………………………….……………………………………………………………158
7.2 Other implementation files ………………………….…………………………………………………………….158
7.3 Class files ………………………………………………..……………………..…………………………………………..168
7.4 Header files ………………………….…………………….…………………………………..………………………….191
7.5 User’s interface ……………………………………………….………………………………………………………….193
7.6 Final formula ……………………………………….………………………………………………………………………196
Source …………………………………………………………………..……………………………………………………........194
Introduction
Introduction
___________________________________________________________________
Introduction
Image processing is an important field of technology that is almost involved in all

aspects of modern life technologies, as image processing is concerned with
visualizing the world around us and changing the nature of its views into a better
formula for both human eye and machine perspective.
It’s a field where it’s more than enough to study its very basics and start
making creative projects based upon them, and by the time you keep getting in
depth you’ll find that results and applications are getting more and more complex
but more and more fun and useful.
Image processing has a lot of applications especially in biomedical field,

including visualizing human cells, tumors and body organs using different
techniques, and analyzing visualized images to get information. It helps people
with disabilities or incurable diseases to reduce the difficulties of their critical
conditions in terms of motion as in case of losing limbs and replacing them with
artificial limbs, or difficulties of communicating, as in case of patients with ALS or
blind people and equipping them with special devices to create a new window of
communication with the out world.
1
Introduction
___________________________________________________________________
In this book we’re going to show how to help people with ALS (amyotrophic
lateral sclerosis) who mainly lost the ability to move or speak, we’ll equip them
with special glasses attached with a camera reading the motion of the eye and
we’ll use our own software to process the stream of images taken from eye
motion and create our new communication window so that patient can speak
with the surroundings, use his computer or mobile phone, explore the internet,
and almost do everything a hale guy can do concerning computer and
communication field.
0.1 Requirements to get through that course
To get through that course you need to build a background about image
processing basics specially dealing with important programming languages like
C++, so that you get to know how to make a program, define variables, do
arithmetic and logic operations, use conditional states, and create classes and
functions, and divide your own program into three separated files header files, …
etc.
You also need to study openCV which is an open source computer vision and
machine learning software library. OpenCV was built to provide a common
infrastructure for computer vision applications and to accelerate the use of
machine perception in the commercial products. Being a BSD-licensed product,
OpenCV makes it easy for businesses to utilize and modify the code.
OpenCV course shows you how to modify the nature of image, how to load
videos saved on your computer, or taken from a live capture device, or live from a
website, and stream it into your own program so that you can do live processing.
You’ll learn how to get DFT (Discrete Fourier Transform) of an image so that you
2
Introduction
can process it more easily, how to separate a colored image into three gray-scale
images representing red, blue, and green components of the original image, how
to merge them again into one single image or merge two of them into one image
with three channels. OpenCV course includes the way you setup OpenCV into
your visual Studio program, how to add libraries and separated classes that have
special uses in image processing field. You’ll learn how to calibrate camera.
Camera calibration is very vital for any image processing application is real life,
also principles of how to detect specific objects and determine the 3 main
dimensions (x, y, z) for Aruco markers you insert to your program.
The third branch you need to study before you can start developing software
of the Eye-Writer is openFrameworks which is an open source C++ toolkit
designed to assist the creative process by providing a simple and intuitive
framework for experimentation.
OpenFrameworks is designed to work as general purpose glue, and wraps

together several commonly used libraries, including:
 OpenGL, GLEW, GLUT, libtess2 and cairo for graphics

 rtAudio, PortAudio, OpenAL and Kiss FFT or FMOD for audio input, output
and analysis
 FreeType for fonts
 FreeImage for image saving and loading
 Quicktime, GStreamer and videoInput for video playback and grabbing
 Poco for a variety of utilities
 OpenCV for computer vision
 Assimp for 3D model loading
The code is written to be massively cross-compatible as openFrameworks

supports five operating systems (Windows, OSX, Linux, iOS, Android) and four
IDEs (Xcode, Code::Blocks, and Visual Studio and Eclipse).
3
Introduction
0.2 Programs to run software
First of all you need to have either Code::Blocks or Visual studio as an IDE
(Integrated development environment) . They both help you to create your own
C++ project and of course include special libraries like openCV, also have no worry
about paying money as Code::Blocks is totally free to download and run and it’s
available as an open-source IDE.
4
Chapter 1
ALS
Chapter1
Chapter1
ALS
1.1 What is ALS ?

Amyotrophic lateral sclerosis (ALS) is a group of rare neurological diseases that mainly
involve the nerve cells (neurons) responsible for controlling voluntary muscle
movement. Voluntary muscles produce movements like chewing, walking, and talking.
The disease is progressive, meaning the symptoms get worse over time. Currently,
there is no cure for ALS and no effective treatment to halt, or reverse, the progression of
the disease.
ALS belongs to a wider group of disorders known as motor neuron diseases, which are
caused by gradual deterioration (degeneration) and death of motor neurons. Motor
neurons are nerve cells that extend from the brain to the spinal cord and to muscles
throughout the body. These motor neurons initiate and provide vital communication links
between the brain and the voluntary muscles.
Messages from motor neurons in the brain (called upper motor neurons) are transmitted
to motor neurons in the spinal cord and to motor nuclei of brain (called lower motor
neurons) and from the spinal cord and motor nuclei of brain to a particular muscle or
muscles.
In ALS, both the upper motor neurons and the lower motor neurons degenerate or die,
and stop sending messages to the muscles. Unable to function, the muscles gradually
weaken, start to twitch (called fasciculations), and waste away (atrophy). Eventually, the
brain loses its ability to initiate and control voluntary movements.
Early symptoms of ALS usually include muscle weakness or stiffness. Gradually all
muscles under voluntary control are affected, and individuals lose their strength and the
ability to speak, eat, move, and even breathe.
Most people with ALS die from respiratory failure, usually within 3 to 5 years from when
the symptoms first appear. However, about 10 percent of people with ALS survive for 10
______________________________________________________________________
5
Chapter1
or more years.
1.2 Who gets ALS?

In 2016 the Centers for Disease Control and Prevention estimated that between 14,000
– 15,000 Americans have ALS. ALS is a common neuromuscular disease worldwide. It
affects people of all races and ethnic backgrounds.
There are several potential risk factors for ALS including:
 Age. Although the disease can strike at any age, symptoms most commonly develop
between the ages of 55 and 75.
 Gender. Men are slightly more likely than women to develop ALS. However, as we age the
difference between men and women disappears.
 Race and ethnicity. Most likely to develop the disease are Caucasians and non-Hispanics.
Some studies suggest that military veterans are about 1.5 to 2 times more likely to
develop ALS. Although the reason for this is unclear, possible risk factors for veterans
include exposure to lead, pesticides, and other environmental toxins. ALS is recognized
as a service-connected disease by the U.S. Department of Veterans Affairs.
Sporadic ALS
The majority of ALS cases (90 percent or more) are considered sporadic. This means
the disease seems to occur at random with no clearly associated risk factors and no
family history of the disease. Although family members of people with sporadic ALS are
at an increased risk for the disease, the overall risk is very low and most will not develop
ALS.
Familial (Genetic) ALS
About 5 to 10 percent of all ALS cases are familial, which means that an individual
inherits the disease from his or her parents. The familial form of ALS usually only
requires one parent to carry the gene responsible for the disease. Mutations in more
than a dozen genes have been found to cause familial ALS. About 25 to 40 percent of
all familial cases (and a small percentage of sporadic cases) are caused by a defect in a
gene known as “chromosome 9 open reading frame 72,” or C9ORF72. Interestingly, the
same mutation can be associated with atrophy of frontal-temporal lobes of the brain
causing frontal-temporal lobe dementia. Some individuals carrying this mutation may
show signs of both motor neuron and dementia symptoms (ALS-FTD). Another 12 to 20
percent of familial cases result from mutations in the gene that provides instructions for
the production of the enzyme copper-zinc superoxide dismutase 1 (SOD1).
______________________________________________________________________
6
Chapter1
1.3 people who got ALS

Knowing that a lot of people suffered from this incurable disease is a sad thing. There’s
a long list of famous people: writers, athletes, scientists, and artists who got prevented
from continuing their normal life as it used to be. Thanks to modern technologies like
speech-generating devices (SGD) they could go on in their journey. One of these
fighters was Stephen Hawking who recorded the longest time a man can get ALS and
be still alive, as he lived for 55 years after being diagnosed with ALS since 1963.
And that’s list of other celebrities who got diagnosed by the same disease:
 Raymond Abrashkin – author

 Zeca Afonso – Portuguese folk singer and anti-fascist politician
 Derek Bailey – British avant-garde guitar virtuoso
 Jason Becker – American guitar virtuoso
 Lead Belly – blues singer and guitarist
 Stefano Borgonovo – Italian football player
 Rob Borsellino – Des Moines Register columnist and author of So I’m Talkin’ to This Guy…
 Scott Brazil – American television producer and director
 O.J. Brigance – American football player and Advisor
 Harry Browne – best-selling author and 2-time Libertarian U.S. presidential candidate
________________________________________________________________________________
7
Chapter1
 Ben Byer – American playwright and subject of the film Indestructible, documenting his life post-
diagnosis
 Jeff Capel II – American collegiate and professional basketball coach[1]
 Paul Cellucci – politician and diplomat; 69th Governor of Massachusetts and U.S. Ambassador to
Canada
 Ezzard Charles – boxer; former world heavyweight champion
 Leonard Cheshire – notable RAF pilot and charity worker
 Marián Čišovský – Slovak football player[2]
 Dwight Clark – American football player[3]
 Preston Cloud – eminent American earth scientist
 Sid Collins – radio personality; radio voice of the Indianapolis 500
 Luca Coscioni – Italian researcher, political activist and advocate for euthanasia
 Ronnie Corbett – British comedian and actor
 Neale Daniher – former AFL player (Essendon) & coach (Melbourne)
 Dennis Day – singer, comedian, actor
 Dieter Dengler – Vietnam era Air Force pilot who escaped from Laotian POW camp
 Michael Donnelly – Gulf War veteran
 Peter Doohan – Australian tennis player
 Ann Downer – Author of books for children and teenagers
 Constantinos Apostolou Doxiadis – Greek architect, urban planner and visionary
 John Drury – longtime ABC7 Chicago news anchor
 Bruce Edwards – PGA Tour caddie for golfer Tom Watson
 Jenifer Estess – theatre producer; star of HBO documentary Three Sisters, subject of HBO
film Jennifer; founding member of Project ALS
 Hal Finney – computer scientist
 Jay S. Fishman – Chairman of the Board and former CEO of The Travelers Companies
 Roberto Fontanarrosa – Argentine cartoonist
 Pete Frates – former Boston College baseball star, founder and inspiration behind the viral ALS
Ice Bucket Challenge (Summer 2014)
 Steven Gey- law professor and expert on the separation of church and state and freedom of
speech; former on-air analyst for ABC during the 2000 presidential recount
 Lou Gehrig – baseball player, after whom the disease is commonly referred
 Richard Glatzer – writer and director; director of Still Alice
 Steve Gleason – American football player for the New Orleans Saints 2000-2007
 Jérôme Golmard – French tennis player
 Stanislav Gross – former Prime Minister of the Czech Republic
 Marc Harrison – designer
 Pro Hart – Australian painter
________________________________________________________________________________
8
Chapter1
 Stephen Hawking – theoretical physicist and author of several books on astrophysics,

including A Brief History of Time
 Bob Haymes – actor, singer, pianist and songwriter of the Great American
Songbook ballad “That’s All”
 Stephen Heywood – carpenter; subject of So Much So Fast and His Brother’s Keeper
 Stephen Hillenburg – marine biologist and cartoonist; creator of SpongeBob SquarePants
 Jim “Catfish” Hunter – baseball player
 Jörg Immendorff, German painter
 Jacob K. Javits, U.S. Senator from New York
 Axel Jensen – writer
 Jimmy Johnstone, Scottish international footballer
 Tony Judt – historian and writer
 Hans Keller – Austrian-born British musicologist and music critic.
 Suna Kıraç, Turkish businesswoman and philanthropist
 Dan Klein – Singer of The Frightnrs
 Denny Miller- actor
 Charles Mingus – jazz bass player
 Glenn Montgomery – NFL football player for the Houston Oilers and Seattle Seahawks
 Augie Nieto – fitness guru; founder and retired chief executive of Life Fitness and the chairman
of Octane Fitness
 David Niven – actor
 Krzysztof Nowak, Polish footballer
 Richard K. Olney – neurologist; ALS physician and researcher
 Sidney Preston Osborn – former governor of Arizona
 Neon Park – American artist
 Mike Porcaro- American bassist, Toto
 Diane Pretty – British “right to die” advocate
 Tony Proudfoot – CFL player, teacher, coach, broadcaster and journalist.
 Don Revie – English football player and manager
 Fernando Ricksen – Dutch football player
 Sue Rodriguez – Canadian “right to die” advocate
 Franz Rosenzweig – philosopher and religious thinker
 Ayan Sadakov – Bulgarian football player and manager
 Stanley Sadie – British musicologist, music critic and editor of the New Grove Dictionary of
Music and Musicians
 Ed Sadowski – baseball catcher and coach
 Washington César Santos – Brazilian Footballer.
________________________________________________________________________________
9
Chapter1
 Michael Schwartz – key conservative political strategist in the U.S. Congress; American “right to
life” advocate; chief of staff to U.S. Senator Tom Coburn, M.D. (R-Okla.)
 Morrie Schwartz – educator
 Raúl Sendic – Uruguayan Marxist and leader of the Tupamaros
 Sam Shepard – American actor and playwright[5]
 Gianluca Signorini – Italian football player
 Lane Smith – actor
 Konrad Spindler – archaeologist, involved in the analysis of the Ötzi glacier mummy
 Jon Stone – creator of Sesame Street
 Maxwell D. Taylor – former chairman of the Joint Chiefs of Staff
 Orlando Thomas- NFL safety for the Minnesota Vikings
 Kevin Turner – NFL fullback for the New England Patriots and Philadelphia Eagles
 Roy Walford – gerontologist and life extensionist
 Henry A. Wallace – 33rd Vice President of the United States to Franklin D. Roosevelt
 Charlie Wedemeyer – former athlete and coach; motivational speaker
 Doddie Weir – former Scottish rugby union player[6]
 Joost van der Westhuizen – former South African Rugby Union player;
former Supersport commentator[7]
 Michael Zaslow – soap actor
 Mao Zedong – Chairman of the Chinese Communist Party
 Per Villand – Norwegian Biologist, Doctor of Science
 Catherine G. Wolf – American psychologist and expert in human-computer interaction
 Aditya Sarvankar – Author[8]
___________________________________________________________________
10
Chapter 2
Overview of the Eye-Writer
Chapter2
Chapter 2
Overview of the Eye-Writer
2.1 What is Eye-Writer?
Eye-Writer project is the most simple and inexpensive eye-tracking system. Obviously, there are
numerous ways to make eye-tracking system. Many of these designs, especially those produced for
academic research projects (Open Eyes), have already been published openly on the internet. There are
also commercial products available, costing in the range of $20,000 US or more that are specifically
designed to enable people with ALS to communicate using their eyes. We are not in the business of re-
inventing these systems. This project is an attempt to address a gap in the development of low-end eye-
tracking systems, in other words to make a super-cheap, eye-tracker that could be made by almost
anyone, almost anywhere. Eye-Writer system has several specific design limitations that were meant to
emphasize low-cost and ease of construction over other aspects of performance, robustness and
appearance. The specific parts and tools you use to build your own “Eye-Writer” will depend on your
ability, location, financial resources and creativity.
2.2 Why Eye-Writer
Eye-Writer has advantages over other eye-tracking systems such as:
a. Being much cheaper than other complicated eye-tracking systems

b. Totally practical and easy to deal with
c. It’s available for development process
d. Availing user to communicate, internet explore, check e-mails
_____________________________________________________________________________________
11
Chapter 3
Hardware and interface of the Eye-writer
Chapter3
Chapter 3
Hardware and interface of the Eye-writer
3.1 Functional design specifications
They are as follows:
1. The Eye-Writer should be as inexpensive as possible
2. The fabrication and assembly of the system should require only common hand tools
3. Whenever possible components and parts should be available for purchase locally versus online
4. The camera should produce 640 x 480 NTSC video
5. The camera should be sensitive to near-field IR light
6. The camera should not auto-iris (or auto-iris should be disabled in the camera’s driver).
7. IR LEDs should be used to illuminate the pupil
Beyond that it’s up to you. This instruction set details a solderless variation of the Eye-Writer that uses a
hacked PS3 Eye and a pair of glasses and suggests other possible Eye-Writer configurations.
12
Chapter3
NTSC
3.2 Parts and tools
3.2.1 Parts:
The following part and materials list details the components and tools we used to make a solderless Eye-
Writer:
1x IR sensitive Camera (without auto-iris), We will use PS3 Eye. ($39.95 US)
(Using this camera system removes the need for an additional video capture card)
1x camera-lens mount, you can use the lens mount that comes with the PS3, but it is glue together and
difficult to separate
Lens holder, M12x0.5, 15.5, centered ($6.00)
(This is the cheap one, but it requires some modification to match PS3 through-hole footprint),
13
Chapter3
More expensive ($20.00)
1x cheap glasses ($5.00)
Aluminum wire, 9-gauge wire ($7.99)
3x alligator clips ($7.00)
A pack of wire-ties $2.49
2x IR LEDs ($1.99)
Tape
1x 8mm camera lens, Fixed IRIS Lens Set for Webcams and Security/CCTV Cameras (6-Lens Pack)
($14.91)
IR written, infrared filter gel ($26.99)
Cheaper DIY version of IR filter include cutting a piece of film out of a floppy disk or using unexposed and
developed photographic film
Battery holder, AAA holder ($1.99)
3.2.2 Tools
Small screw driver set, 7ps set ($14.95)
Scissors
3.2.3 Other optional parts and tools
Electrical junction connectors
Screws
Drill
Soldering iron, solder, flux,
_____________________________________________________________________________________
14
Chapter3
Shrink tube
Tap and tap handle
Perforated circuit board
Dremel
A video capture card (if not using a PS3 Eye), no need to it as we are using PS3 eye.
15
Chapter3
_____________________________________________________________________________________
16
Chapter3
_____________________________________________________________________________________
17
Chapter3
_____________________________________________________________________________________
18
Chapter3
_____________________________________________________________________________________
_____________________________________________________________________________________
19
Chapter3
_____________________________________________________________________________________
_____________________________________________________________________________________
20
Chapter 3
3.3 Steps of interfacing and hardware solid work
Step1: Getting glasses:
We got this model for $4 US, Other models include:
A couple of glasses
Grey hipster frames
Step2: Make the Camera Arm:
The camera arm needs to hold the camera rigidly in front of one eye, but also be flexible, movable and
easy to manufacture.
The best material we have found in terms of rigidity, flexibility, machinability, cost and weight is 9-gauge
aluminum wire. This type of wire is often used as a support structure inside clay and plaster sculptures
and can be found at art supply, hardware, and craft stores. It often has a plastic coating over the
aluminum, which is helpful in our case in order to electrically isolate the camera arm from the camera
circuit board. If the wire you use is not insulated, you can wrap it with a few strips of electrical tape.
To machine aluminum wire you can use a pair of tin snips or simply bend the wire back and forth
repeatedly until it breaks.
Break off a long piece of wire 20 to 30 cm in length
_____________________________________________________________________________________
21
Chapter3
Step3: Attach the camera arm to the glasses
The easiest way to attach the camera arm to the glasses frame is to simply use wire ties to secure the
wire along the arm of the glasses. We use around 6-8 small wire ties.
A more elaborate method involves using aluminum electrical connectors. These can be found at
hardware stores. This method requires a drill, a tap, a tap handle and takes about 20 minutes.
1) Drill two appropriately-sized holes for tapping into the aluminum connector. The size of the hole
will depend on the size of the screw you intend to use. We used a 4-40 screw. To create a hole
ready for a 4-40 tap you would use a #43 drill bit (3/32nd). You can use the tap and die chart
linked below as a reference is you intend to use a different-sized screw.
2) Use the holes in the aluminum connector as a reference to mark where the holes need to be
drilled in your glasses frame and then drill two holes that would accommodate a 4-40 screw (a
#38 drill bit).
3) Using the tap handle and a 4-40 tap, you should tap the aluminum connector. Remember to
move slowly, use machine oil (or olive oil if you don’t have professional machine lubricants) and
clean the hole before trying to insert the screw
______________________________________________________________________________
22
Chapter3
______________________________________________________________________________
4) You can now assemble the pieces. Use washers, lock washers and lock-tite in order to securely
attach the connector to the frame. The connector we used has a flathead set screw that allows
us to screw down the aluminum camera arm and secure it tightly to the glasses.
______________________________________________________________________________
23
Chapter3
______________________________________________________________________________
__________________________________________________________________________________
24
Chapter3
__________________________________________________________________________________
__________________________________________________________________________________
25
Chapter3
__________________________________________________________________________________
Step4: Hack the PS3 Eye
There are a number of videos onine that explain how to take apart the PS3 Eye and remove the IR
blocking filter, and how to install a visible light filter using a floppy disk.
These videos document the process of hacking the PS3 Eye pretty thoroughly. But, In our case we
need to use a lens with a shorter focal length than the one provided with the PS3, so some extra
hacking is in order. To recap and expand on how to mod the PS3 Eye for use with the Eye-Writer
software:
1. Unscrew the four screws on the back of the PS3 eye
2. Crack open the case using a small flat head screw driver
3. Unscrew the screws that mount the camera circuit board to the plastic housing
4. Unscrew the camera lens mount
5. Either
a) Throw away the PS3 Eye lens mount and lens and use one of the lens mounts linked to in
the parts list (and the 8mm lens from our lens pack) or,
b) If you want to repurpose the PS3 lens mount you need to dig the IR light filter (as shown in
the video).
6. If you want to use the original PS3-native lens mount, you will need to separate the PS3-native
lens from the mount, which is attached with some industrial glue. To do this you need to scratch
away the glue around the outside lip of the mount. This is hard to do and requires some patience
and some luck. You will need to scratch and try to turn the lens to unscrew it. Keep repeating this
process until the lens separates and can be unscrewed. If you destroy the lens (which happened to
us about half the time) you will be forced to use one of the lens mounts linked to in the parts list.
7. Cut your IR Wratten down to fit inside the lens mount
8. If you have successfully separated the PS3 lens from the PS3 lens mount then just screw the PS3
lens mount back onto the camera circuit board. If you have not succeeded in separating the lens
from the mount, then screw the new lens mount on the camera circuit board.
9. Screw in the 8mm lens into the lens mount
__________________________________________________________________________________
26
Chapter3
_________________________________________________________________________________
Watch the video and look at the included photos for more tricks and information on how to
successfully hack the PS3 Eye.
_____________________________________________________________________________________
27
Chapter3
____________________________________________________________________________
_____________________________________________________________________________________
28
Chapter3
_____________________________________________________________________________________
_____________________________________________________________________________________
29
Chapter3
_____________________________________________________________________________________
_____________________________________________________________________________________
30
Chapter3
_____________________________________________________________________________________
Step5: Attach the Camera
To attach your newly modified PS3 Eye camera to the wire armature you will need wire ties and
some type of small, insulated substrate that will provide more rigidity to the camera/armature
assembly. For our prototype we used a small (3 in x 1 in) piece of hard rubber. You can also use a
piece of wood, half of a pop cycle stick, or any other sturdy, insulated material.
1. Put the rigid substrate in between the camera and the wire armature. The camera should be
pointed toward the eye of the glasses.
2. Use 4 wire ties to firmly attach all three pieces together.
You can put a small length of double sided tape between the rigid substrate and the wire armature
to make putting the three pieces together and to ensure a more secure assembly. The camera
should still be adjustable in terms of pitch and may require occasional adjustments (or even re-
assembly) between uses.
__________________________________________________________________________________
31
Chapter3
_________________________________________________________________________________
__________________________________________________________________________________
32
Chapter3
__________________________________________________________________________________
__________________________________________________________________________________
33
Chapter3
__________________________________________________________________________________
__________________________________________________________________________________
34
Chapter3
__________________________________________________________________________________
__________________________________________________________________________________
35
Chapter3
__________________________________________________________________________________
__________________________________________________________________________________
36
Chapter3
__________________________________________________________________________________
__________________________________________________________________________________
37
Chapter3
__________________________________________________________________________________
Step6: Light it up!
When you illuminate the eye with IR light and observe it through an IR sensitive camera with a
visible light filter, the iris of the eye turns completely white and the pupil stands out as a high-
contrast black dot. This makes tracking the eye much easier. In order to provide some IR
illumination, we have made a quick and dirty IR LED circuit using alligator clips, IR LEDs and a 2x AAA
battery holder.
The circuit is a simple 3 volt series circuit with two IR LEDs and a power supply (See the napkin
circuit drawing below for more details). Connect an alligator clip, preferably a red one, to the power
lead from the battery holder. Connect the other end of the alligator clip to the positive leg of one of
the IR LEDs. Connect another alligator clip, preferably white or yellow, to the negative leg of the
same IR LED and the positive leg of the second IR LED. Finally, connect an alligator clip, preferably
black, to the negative leg of the second IR LED. The other end of the black alligator clip should be
connected to the negative lead from the battery holder.
You can test to see if the IR LEDs are working by looking at them using most typical point and shoot
cameras. If they are sensitive to IR light, you should see a soft glow coming from both LEDs.
Wrap up the excess cable, wire tie the alligator clips to the arm of the glasses and the camera
armature. You can use wire ties to attach the alligator clips to the front of the camera. Bend the IR
LEDs so they are pointing in the same direction as the camera, bent in toward the eye. Make sure
the LED legs are not touching each other or any part of the camera circuit board. You can use
electrical tape to help keep all metal components electrically isolated from one another. You will
likely have to adjust the IR LEDs once you are looking at the eye in the Eye-Writer software in order
to get a strong illumination that removes shadows created by the eyelid, lashes and camera frame.
3.4 Connect to the Eye-Writer Software
The Eye-Writer software is two parts: an eye-tracking software designed for use with our low-cost
glasses, and software designed for dealing with eye movements.
The software for both parts has been developed using openFrameworks, a cross platform C++ library
for creative development. In order to compile and develop the Eye-Writer source code, you will
need to download 47penFrameworks (pre release v0.06). Documentation, setup guides and more
information can be found at http://openframeworks.cc .
__________________________________________________________________________________
38
Chapter3
__________________________________________________________________________________
3.5 PS3 Eye drivers & QT components
In order to use the PS3 eye you will need to download a driver/component and install it.
For a mac you will need to download the quick-time component here and put it in your-
hardDrive//Library/QuickTime.
3.6 Video Capture
Alternatively, if you plan to use another type of NTSC camera, you will need a video capture card.
We have successfully used the Pinnaccle Dazzle USB DVD recorder .
To use this device you will need to install a PC driver or use VideoGlide. This software does require a
user-license which costs roughly $25.00 dollars
3.7 Eye-Tracking Software
The eye-tracking software detects and tracks the position of a pupil from an incoming camera or
video image, and uses a calibration sequence to map the tracked eye/pupil coordinates to positions
on a computer screen or projection. Note that we use the GSL (gnu scientific library) for calibration,
which is GPL, thus the eye tracking source code is GPL.
The pupil tracking relies upon a clear and dark image of the pupil. The glasses we designed use near-
infrared LEDs to illuminate the eye and create a dark pupil effect. This makes the pupil much more
distinguishable and, thus, easier to track. The camera setting part of the software is designed so the
image can be adjusted with brightness and contrast to get an optimal image of the eye.
The calibration part of the software displays a sequence of points on the screen and records the
position of the pupil at each point. It is designed so that a person wearing the glasses should focus
on each point as it is displayed. When the sequence is finished, the two sets of data are used to
interpolate where subsequent eye positions are located in relation to the screen.
3.8 Eye-Drawing Software
The eye-drawing software is designed to work with the Eye-Writer tracking software as well as
commercial eye-trackers such as the Tobii eye tracker. It is currently a separate application from the
eye-tracker.
__________________________________________________________________________________
39
Chapter3
__________________________________________________________________________________
Tobii Eye Tracker
The tool allows you to draw, manipulate and style a tag using a time-based interface so that
triggering buttons or creating points for drawing is achieved by focusing on the position for a given
amount of time. Tags and tag data can also be uploaded via FTP and HTTP Post.
3.9 Eye-Writer installation
The Eye-Writer interface can be used to create drawings on screen, or using a small projector, you
can create drawings on the wall in a hospital room. We have also used the Eye-Writer software in
conjunction with a special version of the Laser Tag software to project Eye-Tags at large scale in
public space.
3.10 On-Screen Drawing
In order to create on-screen drawing you will simply need to follow the steps featured in the
previous step. This will work with both: the Tobii system software as well as the Eye-Writer
hardware and software suite.
__________________________________________________________________________________
40
Chapter3
__________________________________________________________________________________
3.11 Co-located and Projected Drawing
In order to create drawing using the Eye-Writer hardware/software or the Eye-Writer software/Tobii
system, you will need a digital projector and a projection screen or surface. You will need to
calibrate the Eye-Writer user to the projection surface. We have experimented with using a regular
bed sheet as a projection screen successfully.
3.12 Remote Drawing and Projection
To do remote projection you will need two computers, both connected to the internet, a mobile
broadcast system connected to one computer and the Laser Tag OSC receive software (super-beta).
We’ll also use Sprint wireless broadband cards to create wireless remote connection between the
two computers.
We are no currently supporting the Laser Tag OSC receive software, but you can download it and
hack around with it. In order to project the GML (graffiti markup language) tags you will need to
drag the GML data into /bin/data and rename the file tempt.gml. It’s predicted that the full Eye-
Writer send/receive software will be released soon with instructions.
__________________________________________________________________________________
41
Chapter4
Theoretical background about C++
Capter4
___________________________________________________________________________
Chapter4
Theoretical background about C++
4.1 What is C++?
C++ is a general-purpose object-oriented programming (OOP) language and is an extension of

the C language. It is therefore possible to code C++ in a “C style” or “object-oriented style.” In
certain scenarios, it can be coded in either way and is thus an effective example of a hybrid
language.
C++ is considered to be an intermediate-level language, as it encapsulates both high- and low-
level language features. Initially, the language was called “C with classes” as it had all the
properties of the C language with an additional concept of “classes”, but then renamed to C++.
4.2 Download, install, launch Code::Blocks
Step 1: Head to CodeBlocks Website
Step 2: Select and Download the Binary Release.
____________________________________________________________________________________
42
Chapter4
____________________________________________________________________________________
Step 3: Select Codeblocks mingw-setup.exe
Step 4: Open the Downloaded file and Install it.
____________________________________________________________________________________
43
Chabpter4
Step 5: Open the CodeBlocks IDE and Select Create new project
Step 6: In New form Template window Select Console Application
____________________________________________________________________________________
44
Chapter4
____________________________________________________________________________________
Step 7: On the next screen select the language which you want to program C or C++
Step 8: Select the project location click next and select the compiler to use . In our case we are going to
use GNU GCC Compiler
____________________________________________________________________________________
45
Chapter4
____________________________________________________________________________________
Step9: Write the program and Build and Run.
4.3 Main structure of the program
Structure of a program is Probably the best way to start learning a programming language is by writing a
program. Therefore, here is our first program:
// my first program in C++
#include <iostream>
#include <string>
using namespace std;
int main () {
cout << “Hello World!” ;
_____________________________________________________________________________________
46
Chapter4
_____________________________________________________________________________________
return 0;
Executed program: Hello World!
// my first program in C++
This is a comment line. All lines beginning with two slash signs (//) are considered comments and do not
have any effect on the behavior of the program. The programmer can use them to include short
explanations or observations within the source code itself. In this case, the line is a brief description of
what our program is.
#include
Lines beginning with a hash sign (#) are directives for the preprocessor. They are not regular code lines
with expressions but indications for the compiler’s preprocessor. In this case the directive #include tells
the preprocessor to include the iostream standard file. This specific file (iostream) includes the
declarations of the basic standard input-output library in C++, and it is included because its functionality
is going to be used later in the program.
Using namespace std;
All the elements of the standard C++ library are declared within what is called a namespace, the
namespace with the name std. So in order to access its functionality we declare with this expression that
we will be using these entities. This line is very frequent in C++ programs that use the standard library,
and in fact it will be included in most of the source codes included in these tutorials.
Int main ()
This line corresponds to the beginning of the definition of the main function. The main function is the
point by where all C++ programs start their execution, independently of its location within the source
code. It does not matter whether there are other functions with other names defined before or after it –
the instructions contained within this function’s definition will always be the first ones to be executed in
any C++ program. For that same reason, it is essential that all C++ programs have a main function. The
word main is followed in the code by a pair of parentheses (()). That is because it is a function
declaration: In C++, what differentiates a function declaration from other types of expressions are these
parentheses that follow its name. Optionally, these parentheses may enclose a list of parameters within
them. Right after these parentheses we can find the body of the main function enclosed in braces ({}).
_____________________________________________________________________________________
47
Chapter4
_____________________________________________________________________________________
What is contained within these braces is what the function does when it is executed.
Cout << “Hello World!”;
This line is a C++ statement. A statement is a simple or compound expression that can actually produce
some effect. In fact, this statement performs the only action that generates a visible effect in our first
program. Cout represents the standard output stream in C++, and the meaning of the entire statement
is to insert a sequence of characters (in this case the Hello World sequence of characters) into the
standard output stream (which usually is the screen). Cout is declared in the iostream standard file
within the std namespace, so that’s why we needed to include that specific file and to declare that we
were going to use this specific namespace earlier in our code. Notice that the statement ends with a
semicolon character (;). This character is used to mark the end of the statement and in fact it must be
included at the end of all expression statements in all C++ programs (one of the most common syntax
errors is indeed to forget to include some semicolon after a statement).
Return 0;
The return statement causes the main function to finish. Return may be followed by a return code (in
our example is followed by the return code 0). A return code of 0 for the main function is generally
interpreted as the program worked as expected without any errors during its execution. This is the most
usual way to end a C++ console program.
_____________________________________________________________________________________
48
Chapter4
_____________________________________________________________________________________
4.4 Fundamental data types
4.5 Declaration of variables
In order to use a variable in C++, we must first declare it specifying which data type we want it to be. The
syntax to declare a new variable is to write the specifier of the desired data type (like int, bool, float…)
followed by a valid variable identifier. For example
int a;
float mynumber;
To see what variable declarations look like in action within a program, we are going to see the C++ code
of the example about your mental memory proposed at the beginning of this section:
// operating with variables
#include <iostream>
_____________________________________________________________________________________
49
Chapter4
_____________________________________________________________________________________
#include <string>
int main ()
// declaring variables:
int a, b;
int result;
// process:
a = 5;
b = 2;
a = a + 1;
result = a – b;
// print out the result:
cout << result;
// terminate the program:
return 0;
Executed program: 4
4.6 Initialization of variables
type identifier = initial_value ;
_____________________________________________________________________________________
50
Chapter4
_____________________________________________________________________________________
For example, if we want to declare an int variable called a initialized with a value of 0 at the moment in
which it is declared, we could write:
int a = 0;
// my first string
#include <iostream>
#include <string>
int main ()
string mystring = “This is a string”;
cout << mystring;
return 0;
Executed program: This is a string
4.7 Operators
4.7.1 Assignment (=)
The assignment operator assigns a value to a variable.
A = 5;
This statement assigns to variable a (the lvalue) the value contained in variable b (the rvalue). The value
that was stored until this moment in a is not considered at all in this operation, and in fact that value is
lost.
_____________________________________________________________________________________
51
Chapter4
_____________________________________________________________________________________
For example, let us have a look at the following code – I have included the evolution of the content
stored in the variables as comments:
// assignment operator
#include <iostream>
#include <string>
int main ()
int a, b;
// a:?, b:?
a = 10;
// a:10, b:?
b = 4;
// a:10, b:4
a = b;
// a:4, b:4
b = 7;
// a:4, b:7
cout << “a:”;
cout << a;
cout << “ b:”;
_____________________________________________________________________________________
52
Chapter4
_____________________________________________________________________________________
cout << b;
return 0;
Executed program: a:4 b:7
4.7.2 Arithmetic operators ( +, -, *, /, % )
The five arithmetical operations supported by the C++ language are:
+ addition
- subtraction
* multiplication
/ division
% modulus
4.7.3 Compound assignment (+=, -=, *=, /=, %=, >>=, <<=, &=, ^=, |=)
When we want to modify the value of a variable by performing an operation on the value currently
stored in that variable we can use compound assignment operators:
_____________________________________________________________________________________
53
Chapter4
_____________________________________________________________________________________
For example:
// compound assignment operators
#include <iostream>
#include <string>
int main ()
int a,
b=3;
a = b;
a+=2;
_____________________________________________________________________________________
54
Chapter4
_____________________________________________________________________________________
// equivalent to a=a+2
cout << a;
return 0;
Executed program: 5
Increase and decrease (++, --)
4.7.4 Increase and decrease (++, --)
Shortening even more some expressions, the increase operator (++) and the decrease operator (--)
increase or reduce by one the value stored in a variable.
They are equivalent to +=1 and to -=1, respectively. Thus:
c++;
c+=1;
c=c+1;
are all equivalent in their functionality: the three of them increase by one the value of c.
4.7.5 Relational and equality operators ( ==, !=, >, =, <= )
In order to evaluate a comparison between two expressions we can use the relational and equality
operators. The result of a relational operation is a Boolean value that can only be true or false, according
to its Boolean result.
== Equal to
!= Not equal to
> Greater than
< Less than
_____________________________________________________________________________________
55
Chapter4
_____________________________________________________________________________________
>= Greater than or equal to
<= Less than or equal to
4.7.5 Logical operators ( !, &&, || )
The logical operators && and || are used when evaluating two expressions to obtain a single relational
result. The operator && corresponds with Boolean logical operation AND. This operation results true if
both its two operands are true, and false otherwise. The following panel shows the result of operator
&& evaluating the expression a && b:
4.7.6 Conditional operator ( ? )
The conditional operator evaluates an expression returning a value if that expression is true and a
different one if the expression is evaluated as false. Its format is:
condition ? result1 : result2
If condition is true the expression will return result1, if it is not it will return result2.
7==5 ? 4 : 3
// returns 3, since 7 is not equal to 5.
7==5+2 ? 4 : 3
_____________________________________________________________________________________
56
Chapter4
_____________________________________________________________________________________
// returns 4, since 7 is equal to 5+2.
5>3 ? a : b
// returns the value of a, since 5 is greater than 3.
a>b ? a : b
// returns whichever is greater, a or b.
4.8 Conditional structure: if and else
The if keyword is used to execute a statement or block only if a condition is fulfilled. Its form is:
4.8.1 if (condition) statement
For example:
if (x == 100)
cout << “x is 100”;
4.8.2 if (condition) statement1 else statement2
For example:
if (x == 100)
cout << “x is 100”;
else
cout << “x is not 100”;
4.8.3 if (condition1) statement1 if else (condition2) statement2 else statement3
The if + else structures can be concatenated with the intention of verifying a range of values. The
following example shows its use telling if the value currently stored in x is positive, negative or none of
them (i.e. zero): if (x > 0) cout << “x is positive”; else if (x < 0) cout << “x is negative”; else cout << “x is
0”;
_____________________________________________________________________________________
57
Chapter4
_____________________________________________________________________________________
4.9 Iteration structures (loops)
Loops have as purpose to repeat a statement a certain number of times or while a condition is fulfilled.
4.9.1 The while loop
Its format is:
while (expression) statement
and its functionality is simply to repeat statement while the condition set in expression is true. For
example, we are going to make a program to countdown using a while-loop:
// custom countdown using while
#include <iostream>
#include <string>
int main ()
int n;
cout << “Enter the starting number > “;
cin >> n;
while (n>0) {
cout << n << “, “;
--n;
cout << “FIRE!\n”;
_____________________________________________________________________________________
58
Chapter4
_____________________________________________________________________________________
return 0;
Executed program: Enter the starting number > 8
8, 7, 6, 5, 4, 3, 2, 1, FIRE!
4.9.2 The do-while loop
Its format is:
do statement while (condition);
Its functionality is exactly the same as the while loop, except that condition in the do-while loop is
evaluated after the execution of statement instead of before, granting at least one execution of
statement even if condition is never fulfilled. For example, the following example program echoes any
number you enter until you enter 0.
// number echoer
#include <iostream>
int main ()
unsigned long n;
do {
cout << “Enter number (0 to end): “;
cin >> n;
cout << “You entered: “ << n << “\n”;
} while (n != 0);
_____________________________________________________________________________________
59
Chapter4
_____________________________________________________________________________________
return 0;
Executed program: Enter number (0 to end): 12345
You entered: 12345
Enter number (0 to end): 160277
You entered: 160277
Enter number (0 to end): 0
You entered: 0
4.9.3 The for loop
Its format is:
for (initialization; condition; increase) statement; The C++ Language Tutorial 37 © cplusplus.com 2008.
All rights reserved and its main function is to repeat statement while condition remains true, like the
while loop. But in addition, the for loop provides specific locations to contain an initialization statement
and an increase statement. So this loop is specially designed to perform a repetitive action with a
counter which is initialized and increased on each iteration.
Here is an example of countdown using a for loop:
// countdown using a for loop
#include <iostream>
int main ()
for (int n=10; n>0; n--) {
_____________________________________________________________________________________
60
Chapter4
_____________________________________________________________________________________
cout << n << “, “;
return 0;
Executed program: 10, 9, 8, 7, 6, 5, 4, 3, 2, 1, FIRE!
4.10 Jump statements
4.10.1 The break statement
Using break we can leave a loop even if the condition for its end is not fulfilled. It can be used to end an
infinite loop, or to force it to end before its natural end. For example, we are going to stop the count
down before its natural end (maybe because of an engine check failure?):
// break loop example
#include <iostream>
int main ()
int n;
for (n=10; n>0; n--) {
cout << n << “, “;
if (n==3) {
cout << “countdown aborted!”;
_____________________________________________________________________________________
61
Chapter4
_____________________________________________________________________________________
break;
return 0;
Executed program: 10, 9, 8, 7, 6, 5, 4, 3, countdown aborted!
4.10.2 The continue statement
The continue statement causes the program to skip the rest of the loop in the current iteration as if the
end of the statement block had been reached, causing it to jump to the start of the following iteration.
For example, we are going to skip the number 5 in our countdown:
// continue loop example
#include <iostream>
int main ()
for (int n=10; n>0; n--) {
if (n==5)
continue;
cout << n << “, “;
_____________________________________________________________________________________
62
Chapter4
_____________________________________________________________________________________
return 0;
Executed program: 10, 9, 8, 7, 6, 4, 3, 2, 1, FIRE!
4.10.3 The goto statement
goto allows to make an absolute jump to another point in the program. You should use this feature with
caution since its execution causes an unconditional jump ignoring any type of nesting limitations. The
destination point is identified by a label, which is then used as an argument for the goto statement. A
label is made of a valid identifier followed by a colon (.
// goto loop example
#include <iostream>
int main ()
int n=10;
loop:
cout << n << “, “;
n--;
if (n>0)
goto loop;
return 0;
_____________________________________________________________________________________
63
Chapter4
_____________________________________________________________________________________
Executed program: 10, 9, 8, 7, 6, 5, 4, 3, 2, 1, FIRE!
4.10.4 The exit function
exit is a function defined in the cstdlib library.
The purpose of exit is to terminate the current program with a specific exit code. Its prototype is:
void exit (int exitcode);
4.11 The selective structure: switch
switch (expression)
case constant1:
group of statements 1;
break;
case constant2:
group of statements 2;
break;
default:
default group of statements
4.12 Functions (I)
_____________________________________________________________________________________
64
Chapter4
_____________________________________________________________________________________
Using functions we can structure our programs in a more modular way, accessing all the potential that
structured programming can offer to us in C++.
A function is a group of statements that is executed when it is called from some point of the program.
The following is its format:
type name ( parameter1, parameter2, …) { statements }
where:
• type is the data type specifier of the data returned by the function.
• name is the identifier by which it will be possible to call the function.
• parameters (as many as needed): Each parameter consists of a data type specifier followed by an
identifier, like any regular variable declaration (for example: int x) and which acts within the function as
a regular local variable. They allow to pass arguments to the function when it is called. The different
parameters are separated by commas.
• statements is the function’s body. It is a block of statements surrounded by braces { }.
Here you have the first function example:
// function example
#include <iostream>
int addition (int a, int b)
int r;
r=a+b;
return (r);
_____________________________________________________________________________________
65
Chapter4
_____________________________________________________________________________________
int main ()
int z;
z = addition (5,3);
cout << “The result is “ << z;
return 0;
Executed program: The result is 8
4.13 Functions with no type: void functions
If you remember the syntax of a function declaration:
type name ( argument1, argument2 …) statement
you will see that the declaration begins with a type, that is the type of the function itself (i.e., the type of
the datum that will be returned by the function with the return statement). But what if we want to
return no value?
Imagine that we want to make a function just to show a message on the screen. We do not need it to
return any value. In this case we should use the void type specifier for the function. This is a special
specifier that indicates absence of type.
// void function example
#include <iostream>
void printmessage ()
_____________________________________________________________________________________
66
Chapter4
_____________________________________________________________________________________
cout << “I’m a function!”;
int main ()
printmessage ();
return 0;
Executed program: I’m a function!
4.14 Functions (II)
4.14.1 Arguments passed by value and by reference
// passing parameters by reference
#include <iostream>
void duplicate (int& a, int& b, int& c)
a*=2;
b*=2;
c*=2;
_____________________________________________________________________________________
67
Chapter4
_____________________________________________________________________________________
int main ()
int x=1, y=3, z=7;
duplicate (x, y, z);
cout << “x=” << x << “, y=” << y << “, z=” << z;
return 0;
Executed program: x=2, y=6, z=14
4.14.2 Default values in parameters
When declaring a function we can specify a default value for each of the last parameters. This value will
be used if the corresponding argument is left blank when calling to the function. To do that, we simply
have to use the assignment operator and a value for the arguments in the function declaration. If a
value for that parameter is not passed when the function is called, the default value is used, but if a
value is specified this default value is ignored and the passed value is used instead. For example:
// default values in functions
#include <iostream>
int divide (int a, int b=2)
int r;
r=a/b;
return (r);
_____________________________________________________________________________________
68
Chapter4
_____________________________________________________________________________________
int main ()
cout << divide (12);
cout << endl;
cout << divide (20,4);
return 0;
Executed program: 6
4.14.3 Overloaded functions
In C++ two different functions can have the same name if their parameter types or number are different.
That means that you can give the same name to more than one function if they have either a different
number of parameters or different types in their parameters. For example:
// overloaded function
#include <iostream>
int operate (int a, int b)
return (a*b);
float operate (float a, float b)
_____________________________________________________________________________________
69
Chapter4
_____________________________________________________________________________________
return (a/b);
int main ()
int x=5,y=2;
float n=5.0,m=2.0;
cout << operate (x,y);
cout << “\n”;
cout << operate (n,m);
cout << “\n”;
return 0;
Executed program: 10
2.5
4.15 Declaring functions
Until now, we have defined all of the functions before the first appearance of calls to them in the source
code. These calls were generally in function main which we have always left at the end of the source
code. If you try to repeat some of the examples of functions described so far, but placing the function
main before any of the other functions that were called from within it, you will most likely obtain
compiling errors. The reason is that to be able to call a function it must have been declared in some
earlier point of the code, like we have done in all our examples.
_____________________________________________________________________________________
70
Chapter4
_____________________________________________________________________________________
But there is an alternative way to avoid writing the whole code of a function before it can be used in
main or in some other function. This can be achieved by declaring just a prototype of the function
before it is used, instead of the entire definition. This declaration is shorter than the entire definition,
but significant enough for the compiler to determine its return type and the types of its parameters.
Its form is:
type name ( argument_type1, argument_type2, …);
It is identical to a function definition, except that it does not include the body of the function itself (i.e.,
the function statements that in normal definitions are enclosed in braces { }) and instead of that we end
the prototype declaration with a mandatory semicolon (;).
The parameter enumeration does not need to include the identifiers, but only the type specifiers. The
inclusion of a name for each parameter as in the function definition is optional in the prototype
declaration. For example, we can declare a function called protofunction with two int parameters with
any of the following declarations:
int protofunction (int first, int second);
int protofunction (int, int);
Anyway, including a name for each variable makes the prototype more legible.
// declaring functions prototypes
#include <iostream>
void odd (int a);
void even (int a);
int main ()
int i;
_____________________________________________________________________________________
71
Chapter4
_____________________________________________________________________________________
do {
cout << “Type a number (0 to exit): “;
cin >> i;
odd (i);
} while (i!=0);
return 0;
void odd (int a)
if ((a%2)!=0)
cout << “Number is odd.\n”;
else even (a);
void even (int a)
if ((a%2)==0)
cout << “Number is even.\n”;
else odd (a);
Executed program: Type a number (0 to exit): 9
_____________________________________________________________________________________
72
Chapter4
_____________________________________________________________________________________
Number is odd.
Type a number (0 to exit): 6
Number is even.
Number is even.
Number is even.
4.16 Arrays
4.16.1 Initializing arrays
When declaring a regular array of local scope (within a function, for example), if we do not specify
otherwise, its elements will not be initialized to any value by default, so their content will be
undetermined until we store some value in them. The elements of global and static arrays, on the other
hand, are automatically initialized with their default values, which for all fundamental types this means
they are filled with zeros.
In both cases, local and global, when we declare an array, we have the possibility to assign initial values
to each one of its elements by enclosing the values in braces { }. For example:
int billy [5] = { 16, 2, 77, 40, 12071 };
4.16.2 Accessing the values of an array
In any point of a program in which an array is visible, we can access the value of any of its elements
individually as if it was a normal variable, thus being able to both read and modify its value. The format
is as simple as:
name[index]
4.16.3 Multidimensional arrays
_____________________________________________________________________________________
73
Chapter4
_____________________________________________________________________________________
Multidimensional arrays can be described as “arrays of arrays”. For example, a bidimensional array can
be imagined as a bidimensional table made of elements, all of them of a same uniform data type.
Jimmy represents a bidimensional array of 3 per 5 elements of type int. The way to declare this array in
C++ would be:
int jimmy [3][5];
and, for example, the way to reference the second element vertically and fourth horizontally in an
expression would be:
jimmy[1][3]
(remember that array indices always begin by zero).
Multidimensional arrays are not limited to two indices (i.e., two dimensions). They can contain as many
indices as needed. But be careful! The amount of memory needed for an array rapidly increases with
each dimension. For example:
char century [100][365][24][60][60];
declares an array with a char element for each second in a century, that is more than 3 billion chars. So
this declaration would consume more than 3 gigabytes of memory!
Multidimensional arrays are just an abstraction for programmers, since we can obtain the same results
with a simple array just by putting a factor between its indices:
int jimmy [3][5]; // is equivalent to
int jimmy [15]; // (3 * 5 = 15)
4.16.4 Arrays as parameters
At some moment we may need to pass an array to a function as a parameter. In C++ it is not possible to
pass a complete block of memory by value as a parameter to a function, but we are allowed to pass its
address. In practice this has almost the same effect and it is a much faster and more efficient operation.
In order to accept arrays as parameters the only thing that we have to do when declaring the function is
to specify in its parameters the element type of the array, an identifier and a pair of void brackets []. For
_____________________________________________________________________________________
74
Chapter4
_____________________________________________________________________________________
example, the following function:
void procedure (int arg[])
accepts a parameter of type “array of int” called arg. In order to pass to this function an array declared
as:
int myarray [40];
it would be enough to write a call like this:
procedure (myarray);
Here you have a complete example:
// arrays as parameters
#include<iostream>
void printarray (int arg[], int length)
for (int n=0; n<length;n++)
cout<< arg[n] << “ “;
cout << “\n”;
int main ()
int firstarray[] = {5, 10, 15};
int secondarray[] = {2, 4, 6, 8, 10};
_____________________________________________________________________________________
75
Chapter4
_____________________________________________________________________________________
printarray (firstarray,3);
printarray (secondarray,5);
return 0;
Executed program: 5 10 15
2 4 6 8 10
4.17 Character Sequences
As you may already know, the C++ Standard Library implements a powerful string class, which is very
useful to handle and manipulate strings of characters. However, because strings are in fact sequences of
characters, we can represent them also as plain arrays of char elements.
For example, the following array:
char jenny [20];
4.17.1 Initialization of null-terminated character sequences
Because arrays of characters are ordinary arrays they follow all their same rules. For example, if we want
to initialize an array of characters with some predetermined sequence of characters we can do it just
like any other array:
char myword[] = { ‘H’, ‘e’, ‘l’, ‘l’, ‘o’, ‘\0’ };
In this case we would have declared an array of 6 elements of type char initialized with the characters
that form the word “Hello” plus a null character ‘\0’ at the end. But arrays of char elements have an
additional method to initialize their values: using string literals.
Char myword [] = { ‘H’, ‘e’, ‘l’, ‘l’, ‘o’, ‘\0’ };
char myword [] = “Hello”;
In both cases the array of characters myword is declared with a size of 6 elements of type char: the 5
characters that compose the word “Hello” plus a final null character (‘\0’) which specifies the end of the
_____________________________________________________________________________________
76
Chapter4
_____________________________________________________________________________________
sequence and that, in the second case, when using double quotes (“) it is appended automatically.
4.17.2 Using null-terminated sequences of characters
Null-terminated sequences of characters are the natural way of treating strings in C++, so they can be
used as such in many procedures. In fact, regular string literals have this type (char[]) and can also be
used in most cases.
For example, cin and cout support null-terminated sequences as valid containers for sequences of
characters, so they can be used directly to extract strings of characters from cin or to insert them into
cout. For example:
// null-terminated sequences of characters
#include <iostream>
int main ()
char question[] = “Please, enter your first name: “;
char greeting[] = “Hello, “;
char yourname [80];
cout << question;
cin >> yourname;
cout << greeting << yourname << “!”;
return 0;
_____________________________________________________________________________________
77
Chapter4
_____________________________________________________________________________________
Executed program: Please, enter your first name: John
Hello, John!
4.18 Pointers
4.18.1 Reference operator (&)
ted = &andy;
This would assign to ted the address of variable andy, since when preceding the name of the variable
andy with the reference operator (&) we are no longer talking about the content of the variable itself,
but about its reference (i.e., its address in memory).
4.18.2 Dereference operator (*)
We have just seen that a variable which stores a reference to another variable is called a pointer.
Pointers are said to “point to” the variable whose reference they store.
Using a pointer we can directly access the value stored in the variable which it points to. To do this, we
simply have to precede the pointer’s identifier with an asterisk (*), which acts as dereference operator
and that can be literally translated to “value pointed by”.
Therefore, following with the values of the previous example, if we write:
beth = *ted;
4.18.3 Declaring variables of pointer types
Due to the ability of a pointer to directly refer to the value that it points to, it becomes necessary to
specify in its declaration which data type a pointer is going to point to. It is not the same thing to point
to a char as to point to an int or a float.
The declaration of pointers follows this format:
type * name;
where type is the data type of the value that the pointer is intended to point to. This type is not the type
of the pointer itself! But the type of the data the pointer points to. For example:
_____________________________________________________________________________________
78
Chapter4
_____________________________________________________________________________________
int * number;
char * character;
float * greatnumber;
4.19 Classes
Classes are generally declared using the keyword class, with the following format:
class class_name {
access_specifier_1:
member1;
access_specifier_2:
member2;
} object_names;
Where class_name is a valid identifier for the class, object_names is an optional list of names for objects
of this class. The body of the declaration can contain members, that can be either data or function
declarations, and optionally access specifiers.
All is very similar to the declaration on data structures, except that we can now include also functions
and members, but also this new thing called access specifier. An access specifier is one of the following
three keywords: private, public or protected. These specifiers modify the access rights that the members
following them acquire:
• private members of a class are accessible only from within other members of the same class or from
their friends.
_____________________________________________________________________________________
79
Chapter4
_____________________________________________________________________________________
• protected members are accessible from members of their same class and from their friends, but also
from members of their derived classes.
• Finally, public members are accessible from anywhere where the object is visible.
4.19.1 Constructors and destructors
Objects generally need to initialize variables or assign dynamic memory during their process of creation
to become operative and to avoid returning unexpected values during their execution. For example,
what would happen if in the previous example we called the member function area() before having
called function set_values()? Probably we would have gotten an undetermined result since the members
x and y would have never been assigned a value.
In order to avoid that, a class can include a special function called constructor, which is automatically
called whenever a new object of this class is created. This constructor function must have the same
name as the class, and cannot have any return type; not even void.
We are going to implement Crectangle including a constructor
// example: class constructor
#include <iostream>
class Crectangle {
int width, height;
public:
Crectangle (int,int);
int area () {
return (width*height);}
};
_____________________________________________________________________________________
80
Chapter4
_____________________________________________________________________________________
Crectangle::Crectangle (int a, int b) {
width = a;
height = b;
} int main () {
Crectangle rect (3,4);
Crectangle rectb (5,6);
cout << “rect area: “ << rect.area() << endl;
cout << “rectb area: “ << rectb.area() << endl;
return 0;
Executed program: rect area: 12
rectb area: 30
4.20 Friendship and inheritance
4.20.1 Friend functions
In principle, private and protected members of a class cannot be accessed from outside the same class
in which they are declared. However, this rule does not affect friends.
Friends are functions or classes declared as such.
If we want to declare an external function as friend of a class, thus allowing this function to have access
to the private and protected members of this class, we do it by declaring a prototype of this external
function within the class, and preceding it with the keyword friend:
// friend functions
#include <iostream>
_____________________________________________________________________________________
81
Chapter4
_____________________________________________________________________________________
class Crectangle {
int width, height;
public:
void set_values (int, int);
int area ()
return (width * height);
friend Crectangle duplicate (Crectangle);
};
void Crectangle::set_values (int a, int b) {
width = a;
height = b;
Crectangle duplicate (Crectangle rectparam)
Crectangle rectres;
rectres.width = rectparam.width*2;
rectres.height = rectparam.height*2;
_____________________________________________________________________________________
82
Chapter4
_____________________________________________________________________________________
return (rectres);
int main ()
Crectangle rect, rectb;
rect.set_values (2,3);
rectb = duplicate (rect);
cout << rectb.area();
return 0;
4.20.2 Friend classes
Just as we have the possibility to define a friend function, we can also define a class as friend of another
one, granting that first class access to the protected and private members of the second one.
// friend class
#include <iostream>
class Csquare;
class Crectangle {
int width, height;
public:
_____________________________________________________________________________________
83
Chapter4
_____________________________________________________________________________________
int area () {
void convert (Csquare a);
};
class Csquare {
private:
int side;
public:
void set_side (int a) {
side=a;
friend class Crectangle;
};
void Crectangle::convert (Csquare a) {
width = a.side;
height = a.side;
int main () {
Csquare sqr;
_____________________________________________________________________________________
84
Chapter4
_____________________________________________________________________________________
Crectangle rect;
sqr.set_side(4);
rect.convert(sqr);
cout << rect.area();
return 0;
4.20.3 Inheritance between classes
Classes that are derived from others inherit all the accessible members of the base class. That means
that if a base class includes a member A and we derive it to another class with another member called B,
the derived class will contain both members A and B.
In order to derive a class from another, we use a colon ( in the declaration of the derived class using
the following format:
class derived_class_name: public base_class_name { /*…*/ };
// derived classes
#include <iostream>
class Cpolygon {
protected:
int width, height;
public:
void set_values (int a, int b) {
_____________________________________________________________________________________
85
Chapter4
_____________________________________________________________________________________
width=a;
height=b;
};
class Crectangle: public Cpolygon {
public:
int area () {
};
class Ctriangle: public Cpolygon {
public:
int area () {
return (width * height / 2);
};
int main () {
Crectangle rect;
Ctriangle trgl;
_____________________________________________________________________________________
86
Chapter4
_____________________________________________________________________________________
trgl.set_values (4,5);
cout << rect.area() << endl;
cout << trgl.area() << endl;
return 0;
10
4.20.4 What is inherited from the base class?
In principle, a derived class inherits every member of a base class except:
• its constructor and its destructor
• its operator=() members
• its friends
4.20.5 Multiple inheritance
In C++ it is perfectly possible that a class inherits members from more than one class. This is done by
simply separating the different base classes with commas in the derived class declaration. For example,
if we had a specific class to print on screen (Coutput) and we wanted our classes Crectangle and
Ctriangle to also inherit its members in addition to those of Cpolygon we could write:
class Crectangle: public Cpolygon, public Coutput;
class Ctriangle: public Cpolygon, public Coutput;
here is the complete example:
// multiple inheritance
#include <iostream>
_____________________________________________________________________________________
87
Chapter4
_____________________________________________________________________________________
class Cpolygon {
protected:
int width, height;
public:
void set_values (int a, int b) {
width=a;
height=b;
};
class Coutput {
public:
void output (int i);
};
void Coutput::output (int i) {
cout << i << endl;
class Crectangle: public Cpolygon, public Coutput {
public:
int area () {
_____________________________________________________________________________________
88
Chapter4
_____________________________________________________________________________________
};
class Ctriangle: public Cpolygon, public Coutput {
public:
int area () {
return (width * height / 2);
};
int main () {
Crectangle rect;
Ctriangle trgl;
trgl.set_values (4,5);
rect.output (rect.area());
trgl.output (trgl.area());
return 0;
10
4.21 Separating C++ program into 3 files
_____________________________________________________________________________________
89
Chapter4
_____________________________________________________________________________________
When dealing with classes we create a file called main.cpp and known as implementation file which
contains the called functions and the core main parts of the program, including variables inserted by
user or programmer, the other file is named after the class created and follows the coming formula:
Class-name.cpp, and that one contains the functions of the class, constructor and deconstructor of the
class. The third and last file follows that formula: Class-name.h, and it’s called the header file that simply
contains the addressed and the prototypes of used functions constructors and deconstructors.
__________________________________________
90
Chapter5
Theoretical background about OpenCV
Chapter5
_____________________________________________________________________________________
Chapter5
Theoretical background about OpenCV
5.1 What is OpenCV?
It’s an open source computer vision and machine learning software library.
OpenCV was built to provide a common infrastructure for computer vision
applications and to accelerate the use of machine perception in the commercial
products. Being a BSD-licensed product, OpenCV makes it easy for businesses to
utilize and modify the code.
OpenCV course shows you how to modify the nature of image, how to load
videos saved on your computer, or taken from a live capture device, or live from a
website, and stream it into your own program so that you can do live processing.
You’ll learn how to get DFT (Discrete Fourier Transform) of an image so that you
can process it more easily, how to separate a colored image into three gray-scale
images representing red, blue, and green components of the original image, how
to merge them again into one single image or merge two of them into one image
with three channels. OpenCV course includes the way you setup OpenCV into
your visual Studio program, how to add libraries and separated classes that have
special uses in image processing field. You’ll learn how to calibrate camera.
Camera calibration is very vital for any image processing application is real life,
also principles of how to detect specific objects and determine the 3 main
dimensions (x, y, z) for Aruco markers you insert to your program.
5.2 Building OpenCV from Source Using Cmake, Using the Command Line
______________________________________________________________________________
91
Chapter5
_____________________________________________________________________________________
1. Create a temporary directory, which we denote as , where you want to put the generated Makefiles,
project files as well the object files and output binaries.
2. Enter the and type
cmake [<some optional parameters>]<path to the OpenCV source directory>
For example
cd ~/103penFr
mkdir release
cd release
cmake –D CMAKE_BUILD_TYPE=RELEASE –D CMAKE_INSTALL_PREFIX=/usr/local ..
3. Enter the created temporary directory () and proceed with:
make
sudo make install
5.2.1 Using OpenCV with gcc and Cmake
• The easiest way of using OpenCV in your code is to use Cmake. A few advantages (taken from the
Wiki):
1. No need to change anything when porting between Linux and Windows
2. Can easily be combined with other tools by Cmake( i.e. Qt, ITK and VTK )
• If you are not familiar with Cmake, checkout the tutorial on its website
Steps:
1. Create a program using OpenCV
Let’s use a simple program such as DisplayImage.cpp shown below.
#include <stdio.h>
_____________________________________________________________________________________
92
Chapter5
_____________________________________________________________________________________
#include <opencv2/openCV.hpp>
using namespace cv;
int main (int argc, char** argv )
if ( argc != 2 ) {
printf(“usage: DisplayImage.out \n”);
return -1;
Mat image;
image = imread( argv[1], 1 );
if ( !image.data ) {
printf(“No image data \n”);
return -1;
namedWindow(“Display Image”, WINDOW_AUTOSIZE );
imshow(“Display Image”, image);
waitKey(0);
return 0;
2. Create a Cmake file:
Now you have to create your CmakeLists.txt file. It should look like this:
_____________________________________________________________________________________
93
Chapter5
_____________________________________________________________________________________
cmake_minimum_required(VERSION 2.8)
project( DisplayImage )
find_package( OpenCV REQUIRED )
add_executable( DisplayImage DisplayImage.cpp )
target_link_libraries( DisplayImage ${OpenCV_LIBS} )
3. Generate the executable:
This part is easy, just proceed as with any other project using Cmake:
cd <DisplayImage_directory>
cmake .
make
4. Result
By now you should have an executable (called DisplayImage in this case). You just have to run it giving
an image location as an argument, i.e.:
./DisplayImage lena.jpg
You should get a nice window
5.3 Installation in Windows
5.3.1 Installation by Using the Pre-built Libraries
1. Launch a web browser of choice and go to our page on Sourceforge.
2. Choose a build you want to use and download it.
3. Make sure you have admin rights. Unpack the self-extracting archive.
4. You can check the installation at the chosen path as you can see below.
_____________________________________________________________________________________
94
Chapter5
_____________________________________________________________________________________
5. To finalize the installation go to the Set the OpenCV environment variable and add it to the systems
path section.
5.3.2 Installation by Making Your Own Libraries from the Source Files (Building the library)
1. Make sure you have a working IDE with a valid compiler. In case of the Microsoft Visual Studio just
install it and make sure it starts up.
2. Install Cmake. Simply follow the wizard, no need to add it to the path. The default install options are
OK.
3. Download and install an up-to-date version of msysgit from its official site. There is also the portable
version, which you need only to unpack to get access to the console version of Git. Supposing that for
some of us it could be quite enough.
4. Install TortoiseGit. Choose the 32 or 64 bit version according to the type of OS you work in. While
installing, locate your msysgit (if it doesn’t do that automatically). Follow the wizard – the default
options are OK for the most part.
5. Choose a directory in your file system, where you will download the OpenCV libraries to. I
recommend creating a new one that has short path and no special charachters in it, for example
D:/OpenCV. For this tutorial I’ll suggest you do so. If you use your own path and know, what you’re
doing – it’s OK.
(a) Clone the repository to the selected directory. After clicking Clone button, a window will appear
where you can select from what repository you want to download source files
(https://github.com/opencv/opencv.git) and to what directory (D:/OpenCV).
(b) Push the OK button and be patient as the repository is quite a heavy download. It will take some
time depending on your Internet connection.
6. In this section I will cover installing the 3rd party libraries.
(a) Download the Python libraries and install it with the default options. You will need a couple other
python extensions. Luckily installing all these may be automated by a nice tool called Setuptools.
Download and install again.
_____________________________________________________________________________________
95
Chapter5
_____________________________________________________________________________________
(b) Installing Sphinx is easy once you have installed Setuptools. This contains a little application that will
automatically connect to the python databases and download the latest version of many python scripts.
Start up a command window (enter cmd into the windows start menu and press enter) and use the CD
command to navigate to your Python folders Script sub-folder. Here just pass to the easy_install.exe as
argument the name of the program you want to install. Add the sphinx argument.
(c) The easiest way to install Numpy is to just download its binaries from the sourceforga page. Make
sure your download and install exactly the binary for your python version (so for version 2.7).
(d) Download the Miktex and install it. Again just follow the wizard. At the fourth step make sure you
select for the “Install missing packages on-the-fly” the Yes option, as you can see on the image below.
Again this will take quite some time so be patient.
(e) For the Intel © Threading Building Blocks (TBB) download the source files and extract it inside a
directory on your system. For example let there be D:/OpenCV/dep. For installing the Intel © Integrated
Performance Primitives (IPP) the story is the same. For exctracting the archives I recommend using the
7-Zip application.
(f) In case of the Eigen library it is again a case of download and extract to the D:/OpenCV/dep directory.
(g) Same as above with OpenEXR.
(h) For the OpenNI prebuilt binaries you need to install both the development build and the
PrimeSensor Module.
(i) For the CUDA you need again two modules: the latest CUDA Toolkit and the CUDA Tools SDK.
Download and install both of them with a complete option by using the 32 or 64 bit setups according to
your OS.
(j) In case of the Qt framework you need to build yourself the binary files (unless you use the Microsoft
Visual Studio 2008 with 32 bit compiler). To do this go to the Qt Downloads page. Download the source
files (not the installers!!!):
7. Now start the Cmake (cmake-gui). You may again enter it in the start menu search or get it from the
All Programs → Cmake 2.8 → Cmake (cmake-gui). First, select the directory for the source files of the
OpenCV library (1). Then, specify a directory where you will build the binary files for OpenCV (2).
_____________________________________________________________________________________
96
Chapter5
_____________________________________________________________________________________
5.4 Image Watch: viewing in-memory images in the Visual Studio debugger
5.4.1 Prerequisites
You have the following available:
1. Visual Studio 2012 Professional (or better) with Update 1 installed.
2. An OpenCV installation on your Windows machine (Tutorial: Installation in Windows).
3. Ability to create and build OpenCV projects in Visual Studio (Tutorial: How to build applications with
OpenCV inside the Microsoft Visual Studio).
5.4.2 Installation
Download the Image Watch installer. The installer comes in a single file with extension .vsix (Visual
Studio Extension). To launch it, simply double-click on the .vsix file in Windows Explorer. When the
installer has finished, make sure to restart Visual Studio to complete the installation.
5.5 Load and Display an Image
#include <opencv2/core/core.hpp>
#include <opencv2/highgui/highgui.hpp>
#include <iostream>
using namespace cv;
int main( int argc, char** argv )
if( argc != 2)
cout <<” Usage: display_image ImageToLoadAndDisplay” << endl;
_____________________________________________________________________________________
97
Chapter5
_____________________________________________________________________________________
return -1;
Mat image;
image = imread(argv[1], CV_LOAD_IMAGE_COLOR); // Read the file
if(! Image.data )
cout << “Could not open or find the image” << std::endl ;
return -1;
namedWindow( “Display window”, WINDOW_AUTOSIZE ); // Create a window for display.
Imshow( “Display window”, image ); // Show our image inside it.
waitKey(0); // Wait for a keystroke in the window
return 0;
• CV_LOAD_IMAGE_UNCHANGED (<0) loads image as it is
• CV_LOAD_IMAGE_GRAYSCALE ( 0) loads the image as an intensity one
• CV_LOAD_IMAGE_COLOR (>0) loads the image in the BGR format
5.6 Load, Modify, and Save an Image
_____________________________________________________________________________________
98
Chapter5
_____________________________________________________________________________________
#include <iostream>
using namespace cv;
char* imageName = argv[1];
Mat image;
image = imread( imageName, 1 );
if( argc != 2 || !image.data )
printf( “ No image data \n “ );
return -1;
Mat gray_image;
cvtColor( image, gray_image, CV_BGR2GRAY );
imwrite( “../../images/Gray_Image.jpg”, gray_image );
namedWindow( imageName, CV_WINDOW_AUTOSIZE );
namedWindow( “Gray image”, CV_WINDOW_AUTOSIZE );
imshow( imageName, image );
imshow( “Gray image”, gray_image );
waitKey(0);
_____________________________________________________________________________________
99
Chapter5
_____________________________________________________________________________________
return 0;
When you run your program you should get something like this:
5.7 How the image matrix is stored in the memory
5.7.1 The efficient way
When it comes to performance you cannot beat the classic C style operator[] (pointer) access.
Therefore, the most efficient method we can recommend for making the assignment is:
Mat& ScanImageAndReduceC(Mat& I, const uchar* const table)
// accept only char type matrices
CV_Assert(I.depth() == CV_8U);
int channels = I.channels();
_____________________________________________________________________________________
100
Chapter5
_____________________________________________________________________________________
int nRows = I.rows;
int nCols = I.cols * channels;
if (I.isContinuous())
nCols *= nRows;
nRows = 1;
int i,j;
uchar* p;
for( i = 0; i < nRows; ++i)
p = I.ptr(i);
for ( j = 0; j < nCols; ++j)
p[j] = table[p[j]];
return I;
5.7.2 The iterator (safe) method
_____________________________________________________________________________________
101
Chapter5
_____________________________________________________________________________________
Mat& ScanImageAndReduceIterator(Mat& I, const uchar* const table)
// accept only char type matrices
CV_Assert(I.depth() == CV_8U);
const int channels = I.channels();
switch(channels)
case 1:
MatIterator_<uchar> it, end;
for( it = I.begin(), end = I.end(); it != end; ++it)
*it = table[*it];
break;
case 3:
MatIterator_ <vec3b> it, end;
for( it = I.begin(), end = I.end(); it != end; ++it)
(*it)[0] = table[(*it)[0]];
_____________________________________________________________________________________
102
Chapter5
_____________________________________________________________________________________
(*it)[1] = table[(*it)[1]];
(*it)[2] = table[(*it)[2]];
return I;
5.8 Discrete Fourier Transform
#include <iostream>
using namespace cv;
const char* filename = argc >=2 ? argv[1] : “lena.jpg”;
Mat I = imread(filename, CV_LOAD_IMAGE_GRAYSCALE);
if( I.empty())
return -1;
Mat padded; //expand input image to optimal size
_____________________________________________________________________________________
103
Chapter5
_____________________________________________________________________________________
int m = getOptimalDFTSize( I.rows );
int n = getOptimalDFTSize( I.cols ); // on the border add zero values
copyMakeBorder(I, padded, 0, m – I.rows, 0, n – I.cols, BORDER_CONSTANT, Scalar::all(0));
Mat planes[] = {Mat_(padded), Mat::zeros(padded.size(), CV_32F)};
Mat complexI;
merge(planes, 2, complexI); // Add to the expanded another plane with zeros
dft(complexI, complexI); // this way the result may fit in the source matrix
// compute the magnitude and switch to logarithmic scale
// => log(1 + sqrt(Re(DFT(I))^2 + Im(DFT(I))^2))
split(complexI, planes); // planes[0] = Re(DFT(I), planes[1] = Im(DFT(I))
magnitude(planes[0], planes[1], planes[0]); // planes[0] = magnitude
Mat magI = planes[0];
magI += Scalar::all(1); // switch to logarithmic scale
log(magI, magI);
// crop the spectrum, if it has an odd number of rows or columns
magI = magI(Rect(0, 0, magI.cols & -2, magI.rows & -2));
// rearrange the quadrants of Fourier image so that the origin is at the image center
int cx = magI.cols/2;
int cy = magI.rows/2;
Mat q0(magI, Rect(0, 0, cx, cy)); // Top-Left – Create a ROI per quadrant
_____________________________________________________________________________________
104
Chapter5
_____________________________________________________________________________________
Mat q1(magI, Rect(cx, 0, cx, cy)); // Top-Right
Mat q2(magI, Rect(0, cy, cx, cy)); // Bottom-Left
Mat q3(magI, Rect(cx, cy, cx, cy)); // Bottom-Right
Mat tmp; // swap quadrants (Top-Left with Bottom-Right)
q0.copyTo(tmp);
q3.copyTo(q0);
tmp.copyTo(q3);
q1.copyTo(tmp); // swap quadrant (Top-Right with Bottom-Left)
q2.copyTo(q1);
tmp.copyTo(q2);
normalize(magI, magI, 0, 1, CV_MINMAX); // Transform the matrix with float values into a
// viewable image form (float between values 0 and 1).
Imshow(“Input Image” , I ); // Show the result
imshow(“spectrum magnitude”, magI);
waitKey();
return 0;
_____________________________________________________________________________________
105
Chapter5
_____________________________________________________________________________________
5.9 Video Input with OpenCV and similarity measurement
#include <sstream>
#include <string>
#include <iomanip>
_____________________________________________________________________________________
106
Chapter5
_____________________________________________________________________________________
#include <iostream>
using namespace cv;
double getPSNR ( const Mat& I1, const Mat& I2);
Scalar getMSSIM( const Mat& I1, const Mat& I2);
help();
if (argc != 5)
cout << “Not enough parameters” << endl;
return -1;
stringstream conv;
const string sourceReference = argv[1], sourceCompareWith = argv[2];
int psnrTriggerValue, delay;
conv << argv[3] << endl << argv[4]; // put in the strings
conv >> psnrTriggerValue >> delay; // take out the numbers
char c;
int frameNum = -1; // Frame counter
_____________________________________________________________________________________
107
Chapter5
_____________________________________________________________________________________
VideoCapture captRefrnc(sourceReference), captUndTst(sourceCompareWith);
if (!captRefrnc.isOpened())
cout << “Could not open reference “ << sourceReference << endl;
return -1;
if (!captUndTst.isOpened())
cout << “Could not open case test “ << sourceCompareWith << endl;
return -1;
Size refS = Size((int) captRefrnc.get(CV_CAP_PROP_FRAME_WIDTH),
(int) captRefrnc.get(CV_CAP_PROP_FRAME_HEIGHT)),
uTSi = Size((int) captUndTst.get(CV_CAP_PROP_FRAME_WIDTH),
(int) captUndTst.get(CV_CAP_PROP_FRAME_HEIGHT));
if (refS != uTSi)
const char* WIN_UT = “Under Test”;
const char* WIN_RF = “Reference”;
// Windows
_____________________________________________________________________________________
108
namedWindow(WIN_RF, CV_WINDOW_AUTOSIZE);
namedWindow(WIN_UT, CV_WINDOW_AUTOSIZE);
cvMoveWindow(WIN_RF, 400 , 0); //750, 2 (bernat =0)
cvMoveWindow(WIN_UT, refS.width, 0); //1500, 2
cout << “Reference frame resolution: Width=” << refS.width << “ Height=” << refS.height
<< “ of nr#: “ << captRefrnc.get(CV_CAP_PROP_FRAME_COUNT) << endl;
cout << “PSNR trigger value “ << setiosflags(ios::fixed) << setprecision(3)
<< psnrTriggerValue << endl;
Mat frameReference, frameUnderTest;
double psnrV;
Scalar mssimV;
for(;;) //Show the image captured in the window and repeat
captRefrnc >> frameReference;
captUndTst >> frameUnderTest;
if (frameReference.empty() || frameUnderTest.empty())
cout << “ < < < Game over! > > > “;
break;
++frameNum;
cout << “Frame: “ << frameNum << “# “;
psnrV = getPSNR(frameReference,frameUnderTest);
cout << setiosflags(ios::fixed) << setprecision(3) << psnrV << “dB”;
_____________________________________________________________________________________
109
Chapter5
_____________________________________________________________________________________
if (psnrV < psnrTriggerValue && psnrV)
mssimV = getMSSIM(frameReference, frameUnderTest);
cout << “ MSSIM: “
<< “ R “ << setiosflags(ios::fixed) << setprecision(2) << mssimV.val[2] * 100 << “%”
<< “ G “ << setiosflags(ios::fixed) << setprecision(2) << mssimV.val[1] * 100 << “%”
<< “ B “ << setiosflags(ios::fixed) << setprecision(2) << mssimV.val[0] * 100 << “%”;
cout << endl;
////////////////////////////////// Show Image /////////////////////////////////////////////
imshow(WIN_RF, frameReference);
imshow(WIN_UT, frameUnderTest);
c = (char)cvWaitKey(delay);
if (c == 27) break;
return 0;
double getPSNR(const Mat& I1, const Mat& I2)
Mat s1;
_____________________________________________________________________________________
110
Chapter5
_____________________________________________________________________________________
absdiff(I1, I2, s1); // |I1 – I2|
s1.convertTo(s1, CV_32F); // cannot make a square on 8 bits
s1 = s1.mul(s1); // |I1 – I2|^2
Scalar s = sum(s1); // sum elements per channel
double sse = s.val[0] + s.val[1] + s.val[2]; // sum channels
if( sse <= 1e-10) // for small values return zero
return 0;
else
double mse = sse / (double)(I1.channels() * I1.total());
double psnr = 10.0 * log10((255 * 255) / mse);
return psnr;
Scalar getMSSIM( const Mat& i1, const Mat& i2)
const double C1 = 6.5025, C2 = 58.5225;
/***************************** INITS **********************************/
int d = CV_32F;
Mat I1, I2;
_____________________________________________________________________________________
111
Chapter5
_____________________________________________________________________________________
i1.convertTo(I1, d); // cannot calculate on one byte large values
i2.convertTo(I2, d);
Mat I2_2 = I2.mul(I2); // I2^2
Mat I1_2 = I1.mul(I1); // I1^2
Mat I1_I2 = I1.mul(I2); // I1 * I2
/*************************** END INITS **********************************/
Mat mu1, mu2; // PRELIMINARY COMPUTING
GaussianBlur(I1, mu1, Size(11, 11), 1.5);
GaussianBlur(I2, mu2, Size(11, 11), 1.5);
Mat mu1_2 = mu1.mul(mu1);
Mat mu2_2 = mu2.mul(mu2);
Mat mu1_mu2 = mu1.mul(mu2);
Mat sigma1_2, sigma2_2, sigma12;
GaussianBlur(I1_2, sigma1_2, Size(11, 11), 1.5);
sigma1_2 -= mu1_2;
GaussianBlur(I2_2, sigma2_2, Size(11, 11), 1.5);
sigma2_2 -= mu2_2;
GaussianBlur(I1_I2, sigma12, Size(11, 11), 1.5);
sigma12 -= mu1_mu2;
///////////////////////////////// FORMULA ////////////////////////////////
_____________________________________________________________________________________
112
Chapter5
_____________________________________________________________________________________
Mat t1, t2, t3;
t1 = 2 * mu1_mu2 + C1;
t2 = 2 * sigma12 + C2;
t3 = t1.mul(t2); // t3 = ((2*mu1_mu2 + C1).*(2*sigma12 + C2))
t1 = mu1_2 + mu2_2 + C1;
t2 = sigma1_2 + sigma2_2 + C2;
t1 = t1.mul(t2); // t1 =((mu1_2 + mu2_2 + C1).*(sigma1_2 + sigma2_2 + C2))
Mat ssim_map;
divide(t3, t1, ssim_map); // ssim_map = t3./t1;
Scalar mssim = mean(ssim_map); // mssim = average of ssim map
5.10 How to read a video stream (online-camera or offline-file)
const string sourceReference = argv[1],sourceCompareWith = argv[2];
VideoCapture captRefrnc(sourceReference);
// or
VideoCapture captUndTst;
captUndTst.open(sourceCompareWith);
if ( !captRefrnc.isOpened())
cout << “Could not open reference “ << sourceReference << endl;
return -1;
_____________________________________________________________________________________
113
Chapter5
_____________________________________________________________________________________
if( frameReference.empty() || frameUnderTest.empty())
// exit the program
Size refS = Size((int) captRefrnc.get(CV_CAP_PROP_FRAME_WIDTH),
(int) captRefrnc.get(CV_CAP_PROP_FRAME_HEIGHT)),
cout << “Reference frame resolution: Width=” << refS.width << “ Height=” << refS.height
<< “ of nr#: “ << captRefrnc.get(CV_CAP_PROP_FRAME_COUNT) << endl;
captRefrnc.set(CV_CAP_PROP_POS_MSEC, 1.2); // go to the 1.2 second in the video
captRefrnc.set(CV_CAP_PROP_POS_FRAMES, 10); // go to the 10th frame of the video
// now a read operation would read the frame at the set position
5.11 Camera calibration With OpenCV
5.11.1 Read the settings
Settings s;
const string inputSettingsFile = argc > 1 ? argv[1] : “default.xml”;
FileStorage fs(inputSettingsFile, FileStorage::READ); // Read the settings
if (!fs.isOpened())
cout << “Could not open the configuration file: \”” << inputSettingsFile << “\”” << endl;
_____________________________________________________________________________________
114
Chapter5
_____________________________________________________________________________________
return -1;
fs[“Settings”] >> s;
fs.release(); // close Settings file
if (!s.goodInput)
cout << “Invalid input detected. Application stopping. “ << endl;
return -1;
5.11.2 Get next input
for(int i = 0;;++i)
Mat view;
bool blinkOutput = false;
view = s.nextImage(); //----- If no more image, or got enough, then stop calibration and show result -------
if( mode == CAPTURING && imagePoints.size() >= (unsigned)s.nrFrames )
if( runCalibrationAndSave(s, imageSize, cameraMatrix, distCoeffs, imagePoints))
mode = CALIBRATED;
else
mode = DETECTION;
_____________________________________________________________________________________
115
Chapter5
_____________________________________________________________________________________
if(view.empty()) // If no more images then run calibration, save and stop loop.
if( imagePoints.size() > 0 )
runCalibrationAndSave(s, imageSize, cameraMatrix, distCoeffs, imagePoints);
break;
imageSize = view.size(); // Format input image.
If( s.flipVertical )
flip( view, view, 0 );
5.11.3 Find the pattern in the current input
vector pointBuf;
bool found;
switch( s.calibrationPattern ) // Find feature points on the input format
case Settings::CHESSBOARD:
found = findChessboardCorners( view, s.boardSize, pointBuf, CV_CALIB_CB_ADAPTIVE_THRESH |

CV_CALIB_CB_FAST_CHECK | CV_CALIB_CB_NORMALIZE_IMAGE);
break;
case Settings::CIRCLES_GRID:
found = findCirclesGrid( view, s.boardSize, pointBuf );
_____________________________________________________________________________________
116
Chapter5
_____________________________________________________________________________________
break;
case Settings::ASYMMETRIC_CIRCLES_GRID:
found = findCirclesGrid( view, s.boardSize, pointBuf, CALIB_CB_ASYMMETRIC_GRID );
break;
// we will draw the found points on the input image using findChessboardCorners function.
If ( found) // If done with success,
// improve the found corners’ coordinate accuracy for chessboard
if( s.calibrationPattern == Settings::CHESSBOARD)
Mat viewGray;
cvtColor(view, viewGray, CV_BGR2GRAY);
cornerSubPix( viewGray, pointBuf, Size(11,11),
Size(-1,-1), TermCriteria( CV_TERMCRIT_EPS+CV_TERMCRIT_ITER, 30, 0.1 ));
if( mode == CAPTURING && // For camera only take new samples after delay time
(!s.inputCapture.isOpened() || clock() – prevTimestamp > s.delay*1e-3*CLOCKS_PER_SEC) )
{ imagePoints.push_back(pointBuf);
prevTimestamp = clock();
_____________________________________________________________________________________
117
Chapter5
_____________________________________________________________________________________
blinkOutput = s.inputCapture.isOpened();
// Draw the corners.
drawChessboardCorners( view, s.boardSize, Mat(pointBuf), found );
5.11.4 Show state and result
//----------------------------- Output Text ------------------------------------------------
string msg = (mode == CAPTURING) ? “100/100” :
mode == CALIBRATED ? “Calibrated” : “Press ‘g’ to start”;
int baseLine = 0;
Size textSize = getTextSize(msg, 1, 1, 1, &baseLine);
Point textOrigin(view.cols – 2*textSize.width – 10, view.rows – 2*baseLine – 10);
if( mode == CAPTURING )
if(s.showUndistorsed)
msg = format( “%d/%d Undist”, (int)imagePoints.size(), s.nrFrames );
else
msg = format( “%d/%d”, (int)imagePoints.size(), s.nrFrames );
putText( view, msg, textOrigin, 1, 1, mode == CALIBRATED ? GREEN : RED);
_____________________________________________________________________________________
118
Chapter5
_____________________________________________________________________________________
if( blinkOutput )
bitwise_not(view, view);
//If we ran calibration and got camera’s matrix with the distortion coefficients we may want to correct
//the image using undistort function:
//------------------------- Video capture output undistorted ------------------------------
if( mode == CALIBRATED && s.showUndistorsed )
Mat temp = view.clone();
undistort(temp, view, cameraMatrix, distCoeffs);
//------------------------------ Show image and check for input commands -------------------
imshow(“Image View”, view);
char key = waitKey(s.inputCapture.isOpened() ? 50 : s.delay);
if( key == ESC_KEY )
break;
if( key == ‘u’ && mode == CALIBRATED )
s.showUndistorsed = !s.showUndistorsed;
if( s.inputCapture.isOpened() && key == ‘g’ )
mode = CAPTURING;
imagePoints.clear();
_____________________________________________________________________________________
119
Chapter5
_____________________________________________________________________________________
5.11.5 Show the distortion removal
if( s.inputType == Settings::IMAGE_LIST && s.showUndistorsed )
Mat view, rview, map1, map2;
initUndistortRectifyMap(cameraMatrix, distCoeffs, Mat(),
getOptimalNewCameraMatrix(cameraMatrix, distCoeffs, imageSize, 1, imageSize, 0),
imageSize, CV_16SC2, map1, map2);
for(int i = 0; i < (int)s.imageList.size(); i++ )
view = imread(s.imageList[i], 1);
if(view.empty())
continue;
remap(view, rview, map1, map2, INTER_LINEAR);
imshow(“Image View”, rview);
char c = waitKey();
if( c == ESC_KEY || c == ‘q’ || c == ‘Q’ )
break;
_____________________________________________________________________________________
120
Chapter5
_____________________________________________________________________________________
5.12 The calibration and save
5.12.1 Code
bool runCalibrationAndSave(Settings& s, Size imageSize, Mat& cameraMatrix, Mat&

distCoeffs,vector<vector > imagePoints )
vector rvecs, tvecs;
vector reprojErrs;
double totalAvgErr = 0;
bool ok = runCalibration(s,imageSize, cameraMatrix, distCoeffs, imagePoints, rvecs, tvecs, reprojErrs,

totalAvgErr);
cout << (ok ? “Calibration succeeded” : “Calibration failed”) << “. avg re projection error = “ <<
totalAvgErr ;
if( ok ) // save only if the calibration was done with success
saveCameraParams( s, imageSize, cameraMatrix, distCoeffs, rvecs ,tvecs, reprojErrs, imagePoints,

totalAvgErr);
return ok;
void calcBoardCornerPositions(Size boardSize, float squareSize, vector& corners,
Settings::Pattern patternType /*= Settings::CHESSBOARD*/) {
corners.clear();
switch(patternType)
_____________________________________________________________________________________
121
Chapter5
_____________________________________________________________________________________
case Settings::CHESSBOARD:
case Settings::CIRCLES_GRID:
for( int i = 0; i < boardSize.height; ++i )
for( int j = 0; j < boardSize.width; ++j )
corners.push_back(Point3f(float( j*squareSize ), float( i*squareSize ), 0));
break;
case Settings::ASYMMETRIC_CIRCLES_GRID:
for( int i = 0; i < boardSize.height; i++ )
for( int j = 0; j < boardSize.width; j++ )
corners.push_back(Point3f(float((2*j + i % 2)*squareSize), float(i*squareSize), 0));
break;
double rms = calibrateCamera(objectPoints, imagePoints, imageSize, cameraMatrix, distCoeffs, rvecs,

tvecs, s.flag|CV_CALIB_FIX_K4|CV_CALIB_FIX_K5);
double computeReprojectionErrors( const vector<vector >& objectPoints, const vector<vector >&

imagePoints, const vector& rvecs, const vector& tvecs, const Mat& cameraMatrix , const Mat&
distCoeffs, vector& perViewErrors)
vector imagePoints2;
int i, totalPoints = 0;
double totalErr = 0, err;
_____________________________________________________________________________________
122
Chapter5
_____________________________________________________________________________________
perViewErrors.resize(objectPoints.size());
for( i = 0; i < (int)objectPoints.size(); ++i ) {
projectPoints( Mat(objectPoints[i]), rvecs[i], tvecs[i], cameraMatrix, // project
distCoeffs, imagePoints2);
err = norm(Mat(imagePoints[i]), Mat(imagePoints2), CV_L2); // difference
int n = (int)objectPoints[i].size();
perViewErrors[i] = (float) std::sqrt(err*err/n); // save for this view
totalErr += err*err; // sum it up
totalPoints += n; }
return std::sqrt(totalErr/totalPoints); // calculate the arithmetical mean
5.12.2 Results
_____________________________________________________________________________________
123
Chapter 6
Theoretical background about OpenFrameworks
Chapter 6
______________________________________________________________________________
Chapter 6
Theoretical background about OpenFrameworks
6.1 What is openFrameworks?
It’s an open source C++ toolkit designed to assist the creative process by providing
a simple and intuitive framework for experimentation.
6.2 Functionality of openFrameworks
OpenFrameworks is designed to work as general purpose glue, and wraps

together several commonly used libraries, including:
 OpenGL, GLEW, GLUT, libtess2 and cairo for graphics

 rtAudio, PortAudio, OpenAL and Kiss FFT or FMOD for audio input, output
and analysis
 FreeType for fonts
 FreeImage for image saving and loading
 Quicktime, Gstreamer and videoInput for video playback and grabbing
 Poco for a variety of utilities
 OpenCV for computer vision
 Assimp for 3D model loading
6.3 Operating systems supported by openFrameworks
The code is written to be massively cross-compatible as openFrameworks

supports five operating systems (Windows, OSX, Linux, iOS, Android) and four
IDEs (Xcode, Code::Blocks, and Visual Studio and Eclipse).
_____________________________________________________________________________________
124
Chapter 6
______________________________________________________________________________
6.4 Basics of openFrameworks
6.4.1 How to add an addon to a project
Where do addons come from?

Addons are located in the addons folder in your OF installation. When you download OF you
will already find several core addons there. Move your own addons or download addons
from other people and place them into this folder as well.
Using projectGenerator (OSX / Windows /

Linux)
You can choose an addon to add in the drop down of the gui:
_____________________________________________________________________________________
125
Chapter 6
______________________________________________________________________________
When you hit generate these addons will be added to the project
You can also import an existing project and change the addons it is using, such as adding
or removing addons it needs.
_____________________________________________________________________________________
126
Chapter 6
______________________________________________________________________________
Makefile based projects

If you compile your project using Makefile, adding an addon is as simple as writing the
name of the addon into a file called addons.make. It should be placed in the root folder of
your project, next to the Makefile. This is an exemplary addons.make:
ofxXmlSettings
ofxGui
6.4.2 Save/export files

openFrameworks will save to your bin/data folder unless you specify another filepath.
If you want to save many files, each file will need to have its own unique file name. A quick
way of doing this is to use the current timestamp because it is never the same. So instead
of naming it “myFile.xml”, which will write over itself everytime you save, you can
do “myFile_” + ofGetTimestampString() + “.xml” to give each file its own name.
_____________________________________________________________________________________
127
Chapter 6
______________________________________________________________________________
You can save a file anywhere in your application, but you may want to trigger it at a
specificmoment. You might want to your file to save everytime you press a specific key.
Void ofApp::keyPressed (int key){
if(key == ‘s’){
// save your file in here!!
Or you might want to call it everytime you exit your application.
Void ofApp::exit (){
// save your file in here!!
Note: that exit() will fire automatically when you close or esc your app, but not if you stop
the app from the IDE.
A Text File
in the header file (.h)
ofFile myTextFile;
in the implementation file (.cpp)
______________________________________________
128
Chapter 6
______________________________________________________________________________
To create the file in setup.
myTextFile.open(“text.txt”,ofFile::WriteOnly);
or if you want to append to an existing txt file.
myTextFile.open(“text.txt”,ofFile::Append);
To add text.
myTextFile << “some text”
This will automatically save, so there is no need to call a save function.
XML Settings
Include the XML addon at the top:
#include “ofxXmlSettings.h”
Initialize your variable:
ofxXmlSettings XML;

add something to your XML:
XML.setValue(“settings:number”, 11);
save it!
XML.saveFile(“mySettings.xml”);
For more info refer to the examples/utils/xmlExample.
_____________________________________________________________________________________
129
Chapter 6
______________________________________________________________________________
An Image
ofImage img;

Make an image. There are many ways of creating an image! Including grabbing from a
camera, creating it pixel by pixel, grabbing from an FBO. I am only showing one option,
which is to draw to the screen and then grab that as an image.
//in draw
ofSetColor(255,130,0);
ofFill();
ofDrawCircle(100,100,50);
// in keyPressed
img.grabScreen(0,0,300,300);
Then trigger a save in your location of choice. Perhaps in the keyPressed or exit functions.
You can save as either a .png or a .jpg.
img.save(“myPic.jpg”);
Optionally you can specify the quality at which you wish to save by adding an additional
parameter.
_________________________________________________________________________
130
Chapter 6
______________________________________________________________________________
img.save(“myPic.jpg”,OF_IMAGE_QUALITY_LOW);
the default is OF_IMAGE_QUALITY_BEST and all the options are: OF_IMAGE_QUALITY_BEST,

OF_IMAGE_QUALITY_HIGH,
OF_IMAGE_QUALITY_MEDIUM, OF_IMAGE_QUALITY_LOW, OF_IMAGE_QUALITY_WORST
Here is what the output should look like in this case.
For more info refer to the examples/graphics/imageSaverExample.
6.4.3 Set the dimensions of the window
In the main.cpp file:
_____________________________________________________________________________________
131
Chapter 6
______________________________________________________________________________
The ofSetupOpenGL method allows you to specify how you want your project displayed on
screen.
ofSetupOpenGL(width, height, DISPLAY_MODE);
The first two parameters specify the width and height of the window:
ofSetupOpenGL(1024, 768, DISPLAY_MODE);
With the third parameter, you can specify how you want the window to be displayed using
three possible modes:
OF_WINDOW mode will create a free floating window of the size specified by width and height.
ofSetupOpenGL(1024, 768, OF_WINDOW);
OF_FULLSCREEN mode will display your project in the top left corner of the screen at the size
specified by width and height, with the rest of the screen a solid grey.
ofSetupOpenGL(1024, 768, OF_FULLSCREEN);
Alternative, using ofApp::setup():
Alternatively, you can set the window size (and position) in the setup function of ofApp,
which gets called right at the start as the app launches:
void ofApp::setup(){
ofSetWindowShape(500, 500);
ofSetWindowPosition(10, 10);
_____________________________________________________________________________________
132
Chapter 6
______________________________________________________________________________
6.4.4 View the value of a variable
There are multiple ways on how to display values of a variable.
Console output
The following examples will show you how you can create output at the console.
Using std::cout
Probably the simplest way is to use std::cout. This command lets you combine different
types of values with strings. Appending endlcreates a line break.
Float value = 0.2;
std::cout << “value: “ << value << endl;
Output:
value: 0.2
Using std::printf
printf can be used to force all sorts of different output formats. %f is a placeholder for a float
variable you are appending. %.0f and %.3f set the decimal places of the printed
value. \n creates a line break. Have a look at the reference for details and examples.
Float value = 0.2;
std::printf(“value: %f\n”, value);
std::printf(“value: %.0f\n”, value);
std::printf(“value: %.3f\n”, value);
Output:
_________________________________________________________________________
133
Chapter 6
______________________________________________________________________________
value: 0.200000
value: 0
value: 0.200
Using ofLog()
The best way to integrate with the openFrameworks workflow is to use the implemented
logging functions. There are different log levels and 146penFram ways of usage – have a
look at the ofLog() documentation. Here is one example:
float value = 0.2;
ofLogNotice() << “value: “ << value;
Output:
[notice ] value: 0.2
You can also redirect the log messages to a file:
ofLogToFile(“myLogFile.txt”, true);
// all following logs will be appended to myLogFile.txt
Graphical output
Drawing Text
Drawing text to the screen is as simple as this:
_________________________________________________________________________
134
Chapter 6
______________________________________________________________________________
void draw() {
ofBackground(ofColor::black);
void draw() {
float value = 0.2;
ofSetColor(ofColor::white);
ofDrawBitmapString(“value: “ + ofToString(value), 10, 10);
Using ofxGui
Another nice way of viewing your variable that also gives you the ability to change it is
using ofParameter and the core addon ofxGui. Read how to add an existing addon for
details on how to add ofxGui to your project.
In the header file, wrap your variable with ofParameter. You can still work with this variable
as you were used to, but it makes you able to add listeners to the variable or to add the
variable to a GUI that will interact with it.
_________________________________________________________________________
135
Chapter 6
______________________________________________________________________________
//ofApp.h
#include “ofxGui.h”
#include “ofMain.h”
class ofApp : public ofBaseApp {
..
ofParameter<float> value;
ofxPanel gui;
In the source file, you can give the value a name, a default value and minimal / maximal
borders (in case of numerical values). You have to setup the GUI and add the value, draw
the GUI. You will then be able to interact with the value.
_________________________________________________________________________
136
Chapter 6
______________________________________________________________________________
//ofApp.cpp
void setup() {
value.set(“value”, 0.2, 0, 1); // name, default value, min, max
gui.setup();
gui.add(value);
void draw(){
gui.draw();
6.4.5 Create a new project
You can use the new project generator on most platforms (osx, windows, linux)
_____________________________________________________________________________________
137
Chapter 6
______________________________________________________________________________
It’s recommended that you make your project within the openFrameworks folder in a
subfolder of “apps”. For example, I could make a project called simpleSketch and it could go
in apps/myApps so the full path from the root of openFrameworks is
apps/myApps/simpleSketch. You can also create folders inside “apps” to organize, for
example if you have different projects you are working on.
_________________________________________________________________________
138
Chapter 6
______________________________________________________________________________
You may have to adjust the settings as to where openFrameworks is located on your hard
drive
One important note is that openFrameworks projects work relatively, meaning, the project
looks for the libs folder in openFrameworks in a relative manner, ie ../../../libs. If you create a
project, it should always live that level deep from the root of the openFrameworks folder.
Alternatively, you can update it using the project generator if you want to change its folder
height.
_____________________________________________________________________________________
139
Chapter 6
______________________________________________________________________________
6.4.6 How to make a GUI slider to control a variable
Creating a GUI slider is very simple. You simply generate a project with the GUI add on,
initialize an ofxFloatSlider and gui, draw the gui, and link the slider to a specific variable.
When you generate your project, include the ofxGui addon.
_________________________________________________________________________
140
Chapter 6
______________________________________________________________________________
When you open your app in xCode, you should see the gui add on source files here:

Include the “ofxGui.h” file.
#include “ofxGui.h”
Initialize a slider and a panel. Here we will use ofxFloatSlider radius to control the size of
a circle. If you wish to work with intergers, use ofxIntSlider.
ofxFloatSlider radius;
ofxPanel gui;

Setup your panel named ‘gui’ and add the radius slider using gui.add(). Here we are
labeling the slider with the string “radius”, starting the inital value at 140, and giving the
slider a range of 10 to 300.
_________________________________________________________________________
141
Chapter 6
______________________________________________________________________________
gui.setup();
gui.add(radius.setup(“radius”, 140, 10, 300));
For the sake of example, draw a circle in the draw() function and pass the variable ‘radius’
as the third parameter.
Void ofApp::draw(){
ofDrawCircle(ofGetWidth()/2, ofGetHeight()/2, radius);
gui.draw();
When you run the app, move the radius slider back and forth to change the size of the
circle.
_________________________________________________________________________
142
Chapter 6
______________________________________________________________________________
6.5 Graphics
6.5.1 Load and display an image
Select images to load and display. Images can be of .gif, .jpg, or .png file format.
Create a new folder in the bin/data folder of your OF project, name it “images” and drop
your images in it.
In the ofApp.h file:
Add an instance variable of type ofImage for each image you wish to load.
ofImage bikers;
ofImage bikeIcon;
In the ofApp.cpp file:
Load the images by calling the load() method of ofImage, with the relative path to the
image:
imageName.load(“INSERT FILE PATH HERE”)
example:
bikers.load(“images/bikers.jpg”);
bikeIcon.load(“images/bike_icon.png”);
Display the images by calling the draw() method of ofImage, 155penFramewo them on the
stage by specifying their horizontal and vertical coordinate positions. The coordinate
positions reference by default the top left corner of the image.
_________________________________________________________________________
143
Chapter 6
______________________________________________________________________________
imageName.draw(xPosition, yPosition)
example:
void ofApp::draw(){
bikers.draw(0, 0);
bikeIcon.draw(190, 490);
Additionally, you can resize images by specifying the new width and height of the displayed
image.
imageName.draw(xPosition, yPosition, width, height)
example:
void ofApp::draw(){
bikeIcon.draw(190, 490, 20, 20);
For more information:

take a look at: examples/graphics/imageLoaderExample
6.5.2 Mask an image with a shape
Since 0.9.0 openFrameworks has had the ability to set the alpha mask for a texture with
another texture. In this example, we draw a path into an FBO (offscreen image) and then
pass the result to the image we want to mask.
_________________________________________________________________________
144
Chapter 6
______________________________________________________________________________
ofPath path;
ofImage img;
ofFbo fbo;
void setup(){
path.lineTo(…);
// draw on the path, level of gray means alpha
fbo.allocate(w,h,GL_LUMINANCE); //or GL_RED if you are using the programmable

renderer
fbo.begin();
path.draw();
fbo.end();
img.getTexture().setAlphaMask(fbo.getTexture());
void draw(){
img.draw();
_________________________________________________________________________
145
Chapter 6
______________________________________________________________________________
6.5.3 Take a screenshot
Creating a screenshot of your work is very simple. You simply initialize an ofImage, draw
something, and then use img.grabScreen(); to capture what you drew.
In the header file (.h)

Initialize an ofImage labeled “img”.
ofImage img;

Draw something! For the sake of example, draw a circle in the draw() function.
Void ofApp:draw(){
ofDrawCircle(ofGetWidth()/2, ofGetHeight()/2, 200);
Next, trigger grabbing and saving the screen. Here, when “x” is pressed, a rectangle starting
at point (0,0) with a width and height of ofGetWidth() and ofGetHeight() is grabbed and
saved.
Void ofApp:keyPressed(int key){
if(key == ‘x’){
img.grabScreen(0, 0 , ofGetWidth(), ofGetHeight());
img.save(“screenshot.png”);
_________________________________________________________________________
146
Chapter 6
______________________________________________________________________________
After adding this to any of your apps, press “x” and a screenshot of your work will save to
the bin >> data folder within your specific app folder.
_________________________________________________________________________
147
Chapter 6
______________________________________________________________________________
6.5.4 The new GLM syntax
Note!: This page is useful only if you are working with the current master branch. You can
skip this information if you are working with the version 0.9.3 that you have downloaded
from this website.
The next version of openFrameworks will replace the internal math library with GLM. GLM is
a solid C++ library used for all the math operations needed when doing vectors and
matrices operations. The use of this library implies some change in the syntax used to
declare vectors and to execute vector’s operation. The legacy mode is still supported, but
the new mode, enabled by default, uses the new glm syntax.
If you are not interested using this library and you want to continue using the syntax you
were used to, or if you want to run an old project using the last openFrameworks master
branch, you can define the OF_USE_LEGACY_MESH constant in ofConstants.h. Doing
this, glm will be disabled for ofPolyline and ofMesh.
Instead, if you want to use GLM and prepare yourself for what will be the future syntax
adopted by openFrameworks, these are the things that are changed:
Declaring the vectors

When declaring vectors in the header file, you have to change
ofVec3f myVector;
ofVec2f my2dVector;
To:
glm::vec3 myVector
glm::vec2 my2dVector;
When the vectors come, for example, from a mesh, you don’t have to worry about the type
declaration, since c++ 11 you have the wonderful auto keywords, that tells to the compiler
to figure out the type for you. For example, if you have code like:
_________________________________________________________________________
148
Chapter 6
______________________________________________________________________________
vector<ofVec3f> 161penFr = mesh.getNormals();
And you want to migrate it to the new mode, just use auto
auto 161penFr = mesh.getNormals();
Methods names
Methods are now plain c functions. Most methods have the exact same name but without
camel case and others are slightly different:
v.length()
becomes
glm::length(v)
And
v.squaredLength()
becomes
glm::length2(v)
And
a.getInterpolated(b, 0.5);
becomes
glm::mix(a, b, 0.5);
And
_________________________________________________________________________
149
Chapter 6
______________________________________________________________________________
v.getMiddle(v1)
becomes
glm::mix(v, v1, 0.5)
Containers
When using the pure glm mode (enabled by default right now) you’ll usually need to change
any container of ofVec to a container of glm::vec.
Vector<ofVec3f> myContainer
becomes
vector<glm::vec3> myContainer
As said before, if it’s in a function using auto is the best solution since the compiler can
automatically detect the type, and it will work for both modes, the glm one and the legacy
one.
Transitional typedefs
There’s also some typedefs that can be used to make containers compatible with both the
legacy and glm mode:
std::vector<ofDefaultVec3>
will be std::vector<ofVec3f> in legacy mode or std::vector<glm::vec3> in glm mode
Caveats when using ofVec

All the old ofVec (and matrices) classes autoconvert and can be created from glm so
although everything in the core now returns or accepts as parameters glm you can still use
ofVec even in pure glm mode:
_________________________________________________________________________
150
Chapter 6
______________________________________________________________________________
ofVec3f pos = camera.getGlobalPosition();
will still work, although getGlobalPosition is now returning glm::vec3. But:
camera.getGlobalPosition().distance(node.getGlobalPosition());
won’t work anymore you have to change it to:
glm::distance(camera.getGlobalPosition(), node.getGlobalPosition());
or
ofVec3f cameraPos = camera.getGlobalPosition();
cameraPos.distance(node.getGlobalPosition());
6.6 Video (load and play)
Select a video to load and play.
Create a new folder in the bin/data folder of your OF project, name it “movies” and drop
your video in it.
In the ofApp.h file:
Add an instance variable of type ofVideoPlayer for the video you wish to load.
ofVideoPlayer fingerMovie;
In the ofApp.cpp file:
In the setup function:
Load the video by calling the load() method of ofVideoPlayer, with the relative path to the
video:
videoName.load(“INSERT FILE PATH HERE”);
_________________________________________________________________________
151
Chapter 6
______________________________________________________________________________
Start playing the video:
videoName.play();
Example:
void ofApp::setup()
{fingerMovie.load(“movies/fingers.mov”);
fingerMovie.play();}
In your update() function:
videoPlayer.update();
Example:
void ofApp::update(){
fingerMovie.update();
In your draw() function:
videoPlayer.draw(xPosition, yPosition, width, height);
Example:
void ofApp::draw(){fingerMovie.draw(0, 0, 400, 300);}
For more information:

take a look at: examples/video/videoPlayerExample
_________________________________________________________________________
152
Chapter 6
______________________________________________________________________________
6.7 Sound (load and play)
Loading and playing sound is very simple. You simply initialize an ofSoundPlayer, load the
sound file, and play the sound file.
Before starting save your sound file in your bin/data folder.

Initialize an ofSoundPlayer.
ofSoundPlayer mySound;

Load your sound file in the setup method. You can load a variety of different sound files --
.mpr, .wav, etc.
_________________________________________________________________________
153
Chapter 6
______________________________________________________________________________
void ofApp:setup(){
mySound.load(“166penFram.mp3”);
Next, play the sound file. If you add this to the setup method, the sound will play once right
when you start your app. You can also set your sound to loop if you want it to play
continuously.
Void ofApp:setup(){
mySound.load(“166penFram.mp3”);
mySound.play();
You can also trigger the play function for mousepress, keys, mousedrag, etc. For example,
void testApp::keyPressed (int key){
if(key == “p”){
mySound.play();
Additional Resources
For more information on how to manipulate sound files using OF in the sound chapter of the
OF book.
_________________________________________________________________________
154
Chapter 6
______________________________________________________________________________
6.8 Loading and 3-D modeling
To load a 3d model in your openFrameworks app you have to use

the ofxAssimpModelLoader addon, that already comes with your openFrameworks
installation.
First, you have to include and define ofxAssimpModelLoader in your ofApp.h:
#include “ofxAssimpModelLoader.h”
ofxAssimpModelLoader yourModel;
Then, in your ofApp.cpp file you load the model and draw it like this:
yourModel.loadModel(“squirrel/NewSquirrel.3ds”, 20);
void ofApp::draw(){
yourModel.drawFaces();
In the folder addons/3DmodelLoaderExample/ of your openFrameworks installation you can

find the complete working example.
6.9 Listening to events
Introduction
In order to be able to listen to an event in your application you need three things: a listener,
an event and an handler. The listener tells to your application to listen for a certain event,
the event is the action that you want to catch, and the handler tells to your application what
_________________________________________________________________________
155
Chapter 6
______________________________________________________________________________
to do when that event is triggered. To add a listener to your app, you have to define it in the
setup method of your App.cpp file, using the method ofAddListener.
ofAddListener(theEventThatImListening , this, &myHandler);
In the App.cpp file, you also have to define what myHandler is doing
void myHandler(arguments & event){ //do something
Default events
Any openFrameworks app, comes at the beginning with handlers for a lot of events:
void keyPressed(int key);
void keyReleased(int key);
void mouseMoved(int x, int y );
void mouseDragged(int x, int y, int button);
void mousePressed(int x, int y, int button);
void mouseReleased(int x, int y, int button);
void mouseEntered(int x, int y);
void mouseExited(int x, int y);
void windowResized(int w, int h);
void dragEvent(ofDragInfo dragInfo);
void gotMessage(ofMessage msg);
_________________________________________________________________________
156
Chapter 6
______________________________________________________________________________
You don’t need to add the listeners for this handlers, because they are already there for
you, ready to use.
Resources
 If you want to create listener for your class (not in ofApp.ccp), have a look at the
example examples/events/SimpleEventsExample. In general, in the
folder examples/events there are a lot of useful resources about events, also about
how to create custom events.
 The 169penFrameworks documentation has a section for events
_________________________________________________________________________
157
Chapter 7
Software and codes
Chapter7
______________________________________________________________________________
Chapter 7
Software and codes
7.1 Main File
7.2 Other implementation files
7.2.1 ransac_ellipse
______________________________________________________________________________
158
Chapter7
______________________________________________________________________________
______________________________________________________________________________
159
Chapter7
______________________________________________________________________________
______________________________________________________________________________
160
Chapter7
______________________________________________________________________________
______________________________________________________________________________
161
Chapter7
______________________________________________________________________________
______________________________________________________________________________
162
Chapter7
______________________________________________________________________________
______________________________________________________________________________
163
Chapter7
______________________________________________________________________________
______________________________________________________________________________
164
Chapter7
______________________________________________________________________________
7.2.2 testApp
______________________________________________________________________________
165
Chapter7
______________________________________________________________________________
______________________________________________________________________________
166
Chapter7
______________________________________________________________________________
______________________________________________________________________________
167
Chapter7
______________________________________________________________________________
7.3 Class files
We’ll build these class files where we’ll create classes needed for our program equipping them
with functions and defining variables.
We’ll use 3 classes: cvEyeTracker, svd, and timing:
7.3.1 timing file
______________________________________________________________________________
168
Chapter7
______________________________________________________________________________
7.3.2 svd file
1. #include <math.h> 25. p[m][n], diag d[n] and 47. for (k = i; k < m; k++)
2. #include <stdlib.h> q[n][n]. 48. scale += fabs(p[k][i]);
3. #include "svd.h" 26. */ 49. if (scale)
27. void svd(int m, int n, double 50. {
**a, double **p, double *d, 51. for (k = i; k < m; k++)
double **q) 52. {
28. { 53. p[k][i] /= scale;
4. //svd function 29. int flag, i, its, j, jj, k, l, 54. s += p[k][i] * p[k][i];
5. #define SIGN(u, v) ( nm, nm1 = n - 1, mm1 = m - 55. }
(v)>=0.0 ? fabs(u) : -fabs(u) ) 1; 56. f = p[i][i];
6. #define MAX(x, y) ( (x) 30. double c, f, h, s, x, y, z; 57. g = -SIGN(sqrt(s), f);
>= (y) ? (x) : (y) ) 31. double anorm = 0, g = 58. h = f * g - s;
0, scale = 0; 59. p[i][i] = f - g;
32. //double *r = 60. if (i != nm1)
tvector_alloc(0, n, double); 61. {
33. double 62. for (j = l; j < n; j++)
7. static double radius(double *r = 63. {
u, double v) (double*)malloc(sizeof(doubl 64. for (s = 0.0, k = i; k < m;
8. { e)*n); k++)
9. double w; 65. s += p[k][i] * p[k][j];
10. u = fabs(u); 66. f = s / h;
11. v = fabs(v); 67. for (k = i; k < m; k++)
12. if (u > v) { 68. p[k][j] += f * p[k][i];
13. w = v / u; 34. for (i = 0; i < m; i++) 69. }
14. return (u * sqrt(1. + w * w)); 35. for (j = 0; j < n; j++) 70. }
15. } else { 36. p[i][j] = a[i][j]; 71. for (k = i; k < m; k++)
16. if (v) { 37. //for (i = m; i < n; i++) 72. p[k][i] *= scale;
17. w = u / v; 38. // p[i][j] = 0; 73. }
18. return (v * sqrt(1. + w * w)); 74. }
19. } else 75. d[i] = scale * g;
20. return 0.0; 76. g = s = scale = 0.0;
21. } 77. if (i < m && i != nm1)
22. } 39. /* Householder reduction to 78. {
bidigonal form */ 79. for (k = l; k < n; k++)
40. for (i = 0; i < n; i++) 80. scale += fabs(p[i][k]);
41. { 81. if (scale)
42. l = i + 1; 82. {
23. /* 43. r[i] = scale * g; 83. for (k = l; k < n; k++)
24. Given matrix a[m][n], m>=n, 44. g = s = scale = 0.0; 84. {
using svd decomposition a = 45. if (i < m) 85. p[i][k] /= scale;
p d q' to get 46. { 86. s += p[i][k] * p[i][k];
______________________________________________________________________________
169
Chapter7
______________________________________________________________________________
87. 140. if (i < nm1) 192. g = d[i];

88. } 141. for (j = l; j < n; j++) 193. h = radius(f, g);
89. f = p[i][l]; 142. p[i][j] = 0.0; 194. d[i] = h;
90. g = -SIGN(sqrt(s), f); 143. if (g) 195. h = 1.0 / h;
91. h = f * g - s; 144. { 196. c = g * h;
92. p[i][l] = f - g; 145. g = 1.0 / g; 197. s = (-f * h);
93. for (k = l; k < n; k++) 146. if (i != nm1) 198. for (j = 0; j < m; j++)
94. r[k] = p[i][k] / h; 147. { 199. {
95. if (i != mm1) 148. for (j = l; j < n; j++) 200. y = p[j][nm];
96. { 149. { 201. z = p[j][i];
97. for (j = l; j < m; j++) 150. for (s = 0.0, k = l; k < m; 202. p[j][nm] = y * c + z * s;
98. { k++) 203. p[j][i] = z * c - y * s;
99. for (s = 0.0, k = l; k < n; k++) 151. s += p[k][i] * p[k][j]; 204. }
100. s += p[j][k] * p[i][k]; 152. f = (s / p[i][i]) * g; 205. }
101. for (k = l; k < n; k++) 153. for (k = i; k < m; k++) 206. }
102. p[j][k] += s * r[k]; 154. p[k][j] += f * p[k][i]; 207. }
103. } 155. } 208. z = d[k];
104. } 156. } 209. if (l == k)
105. for (k = l; k < n; k++) 157. for (j = i; j < m; j++) 210. { /* convergence */
106. p[i][k] *= scale; 158. p[j][i] *= g; 211. if (z < 0.0)
107. } 159. } else 212. {
108. } 160. for (j = i; j < m; j++) 213. d[k] = -z;
109. anorm = MAX(anorm, 161. p[j][i] = 0.0; 214. for (j = 0; j < n; j++)
fabs(d[i]) + fabs(r[i])); 162. ++p[i][i]; 215. q[j][k] = (-q[j][k]);
110. } 163. } 216. }
164. /* diagonalization of the 217. break;
bidigonal form */ 218. }
165. for (k = n - 1; k >= 0; k--) 219. if (its == 30)
166. { /* loop over 220. {
111. /* Accumulation of right- singlar values */ 221. //error("svd: No convergence
hand transformations */ 167. for (its = 0; its < 30; its++) in 30 svd iterations",
112. for (i = n - 1; i >= 0; i--) 168. { /* loop over non_fatal);
113. { allowed iterations */ 222. return;
114. if (i < nm1) 169. flag = 1; 223. }
115. { 170. for (l = k; l >= 0; l--) 224. x = d[l]; /* shift from
116. if (g) 171. { /* test for splitting */ bottom 2-by-2 minor */
117. { 172. nm = l - 1; /* note that r[l] 225. nm = k - 1;
118. for (j = l; j < n; j++) is always 226. y = d[nm];
119. q[j][i] = (p[i][j] / p[i][l]) / g; 173. zero */ 227. g = r[nm];
120. for (j = l; j < n; j++) 174. if (fabs(r[l]) + anorm == 228. h = r[k];
121. { anorm) 229. f = ((y - z) * (y + z) + (g - h)
122. for (s = 0.0, k = l; k < n; k++) 175. { * (g + h)) / (2.0 * h * y);
123. s += p[i][k] * q[k][j]; 176. flag = 0; 230. g = radius(f, 1.0);
124. for (k = l; k < n; k++) 177. break; 231. /* next QR transformation */
125. q[k][j] += s * q[k][i]; 178. } 232. f = ((x - z) * (x + z) + h * ((y
126. } 179. if (fabs(d[nm]) + anorm == / (f + SIGN(g, f))) - h)) / x;
127. } anorm) 233. c = s = 1.0;
128. for (j = l; j < n; j++) 180. break; 234. for (j = l; j <= nm; j++)
129. q[i][j] = q[j][i] = 0.0; 181. } 235. {
130. } 182. if (flag) 236. i = j + 1;
131. q[i][i] = 1.0; 183. { 237. g = r[i];
132. g = r[i]; 184. c = 0.0; /* cancellation of 238. y = d[i];
133. l = i; r[l], if 239. h = s * g;
134. } 185. l>1 */ 240. g = c * g;
135. /* Accumulation of left-hand 186. s = 1.0; 241. z = radius(f, h);
transformations */ 187. for (i = l; i <= k; i++) 242. r[j] = z;
136. for (i = n - 1; i >= 0; i--) 188. { 243. c = f / z;
137. { 189. f = s * r[i]; 244. s = h / z;
138. l = i + 1; 190. if (fabs(f) + anorm != anorm) 245. f = x * c + g * s;
139. g = d[i]; 191. {
________________________________________________________________________
170
Chapter7
______________________________________________________________________________
246. g = g * c - x * s; 263. s = h * z;
247. h = y * s; 264. }
248. y = y * c; 265. f = (c * g) + (s * y);
281. // dhli add: the original code
249. for (jj = 0; jj < n; jj++) 266. x = (c * y) - (s * g);
does not sort the eigen value
250. { 267. for (jj = 0; jj < m; jj++)
282. // should do that and change
251. x = q[jj][j]; 268. {
the eigen vector accordingly
252. z = q[jj][i]; 269. y = p[jj][j];
253. q[jj][j] = x * c + z * s; 270. z = p[jj][i];
254. q[jj][i] = z * c - x * s; 271. p[jj][j] = y * c + z * s;
255. } 272. p[jj][i] = z * c - y * s;
256. z = radius(f, h); 273. }
283. }
257. d[j] = z; /* rotation can be 274. }
arbitrary 275. r[l] = 0.0;
258. id z=0 */ 276. r[k] = f;
259. if (z) 277. d[k] = x;
260. { 278. }
261. z = 1.0 / z; 279. }
262. c = f * z; 280. free(r);
7.3.3 cvEyeTracker
1. #include <stdio.h> 36. #define ISIN(x,l,u)

2. #include <stdlib.h> ((x)<(l)?((0)):((x)>(u)?(0):(1)))
3. #include <signal.h> 25. #ifndef _EiC
4. #include <sys/types.h> 26. #include "cv.h"
5. #include <sys/stat.h> 27. #include "highgui.h"
6. #include <fcntl.h> 28. #endif 37. #define CALIBRATIONPOINTS
7. #include <errno.h> 9
8. #include <unistd.h>
9. #include <linux/videodev.h>
10. #include <sys/ioctl.h>
38. FILE *logfile;
11. #include <string.h>
29. #define UINT8 unsigned char 39. #define Log(args...)
12. #include <sys/mman.h>
fprintf(logfile,args);
13. #include <time.h>
14. #include <math.h>
15. #include <sys/time.h> 30. #ifndef PI
16. #include 31. #define PI 3.141592653589 40. FILE *ellipse_log;
<libraw1394/raw1394.h> 32. #endif
17. #include
<libdc1394/dc1394_control.
h> 41. #define
18. #include 33. #define DEBUG 1 MIN_PUPIL_CONTOUR_POIN
"remove_corneal_reflection. TS 500
h" 42. #define
19. #include "ransac_ellipse.h" MAX_PUPIL_CONTOUR_POI
20. #include "timing.h" 34. #define INFO(args...) if NTS 10000
21. #include "svd.h" (DEBUG) printf(args) 43. #define
PUPIL_SIZE_TOLERANCE
1000
//range of
22. #ifdef _CH_ 35. #define CLIP(x,l,u)
allowed pupil diameters
23. #pragma package <opencv> ((x)<(l)?((l)):((x)>(u)?(u):(x)))
24. #endif
______________________________________________________________________________
171
Chapter7
______________________________________________________________________________
44. #define
MAX_CONTOUR_COUNT
20 67. char Feature_Names[9][30] 92. int
={ number_calibration_points_s
68. "BRIGHTNESS", et = 0;
69. "EXPOSURE", 93. int ok_calibrate = 0;
45. // Firewire Capture Variables 70. "SHARPNESS",
46. int dev; 71. "WHITE BALANCE",
47. int 72. "HUE",
width=640,height=480,frame 73. "SATURATION", 94. CvPoint
rate=30; 74. "GAMMA", calipoints[CALIBRATIONPOIN
48. FILE* imagefile; 75. "SHUTTER", TS]; //conversion from
49. dc1394_cameracapture 76. "GAIN"}; eye to scene calibration
cameras[2]; points
50. int numNodes; 95. CvPoint
51. int numCameras; scenecalipoints[CALIBRATIO
52. raw1394handle_t handle; 77. typedef struct { NPOINTS]; //captured (with
53. nodeid_t * camera_nodes; 78. int offset_value; mouse) calibration points
54. dc1394_feature_set 79. int value; 96. CvPoint
features; 80. int min; pucalipoints[CALIBRATIONPO
81. int max; INTS]; //captured eye points
82. int available; while looking at the
83. void (*callback)(int); calibration points in the
55. // Load the source image. 84. } camera_features; scene
56. IplImage *eye_image=NULL; 97. CvPoint
57. IplImage crcalipoints[CALIBRATIONPOI
*original_eye_image=NULL; NTS]; //captured corneal
58. IplImage 85. camera_features reflection points while
*threshold_image=NULL; eye_camera_features[9]; looking at the calibration
59. IplImage points in the scene
*ellipse_image=NULL; 98. CvPoint
60. IplImage vectors[CALIBRATIONPOINTS
*scene_image=NULL; 86. CvPoint pupil = {0,0};
]; //differences between
//coordinates of pupil in
the corneal reflection and
tracker coordinate system
pupil center
87. CvPoint corneal_reflection =
61. // Window handles {0,0}; //coordinates of
62. const char* eye_window = corneal reflection in tracker
"Eye Image Window"; coordinate system 99. //scene coordinate
63. const char* 88. CvPoint diff_vector = {0,0}; interpolation variables
original_eye_window = //vector between the 100. float a, b, c, d, e;
"Original Eye Image"; corneal reflection and pupil //temporary storage of
64. const char* ellipse_window = 89. int corneal_reflection_r = 0; coefficients
"Fitted Ellipse Window"; //the radius of corneal 101. float aa, bb, cc, dd, ee;
65. const char* scene_window = reflection //pupil X coefficients
"Scene Image Window";
66. const char* control_window
= "Parameter Control
Window"; 90. int view_cal_points = 1;
91. int do_map2scene = 0;
__________________________________________________________________________________
172
Chapter7
______________________________________________________________________________
102. float ff, gg, hh, ii, jj; (double*)malloc(FRAMEH*siz 132. b = y + ((u*1814) >> 10);\
//pupil eof(double)); //horizontal 133. r = r < 0 ? 0 : r;\
Y coefficients intensity factor for noise 134. g = g < 0 ? 0 : g;\
reduction 135. b = b < 0 ? 0 : b;\
115. double *avg_intensity_hori = 136. r = r > 255 ? 255 : r;\
(double*)malloc(FRAMEH*siz 137. g = g > 255 ? 255 : g;\
103. float centx, centy; eof(double)); //horizontal 138. b = b > 255 ? 255 : b
// translation to center pupil average intensity
data after biquadratics
104. float cmx[4], cmy[4];
// corner 139. #define FIX_UINT8(x) ( (x)<0
correctioncoefficients 116. //parameters for the ? 0 : ((x)>255 ? 255:(x)) )
105. int inx, iny; algorithm
// translation to center pupil 117. int edge_threshold = 20;
data before biquadratics
//threshold of 140. //----------------------- Firewire
pupil edge points detection Image Capture Code -----------
118. int rays = 18; ------------//
106. int
White,Red,Green,Blue,Yello //number of rays
w; to use to detect feature
107. int frame_number=0; 141. void Open_IEEE1394()
points
142. {
119. int min_feature_candidates =
143. int i;
10; //minimum
number of pupil feature
108. #define FRAMEW 640
candidates
109. #define FRAMEH 480
120. int cr_window_size = 301; 144. handle =
dc1394_create_handle(0);
//corneal 145. if (handle==NULL) {
110. int refelction search window 146. fprintf( stderr, "Unable to
monobytesperimage=FRAME size aquire a raw1394
W*FRAMEH; handle\n\n"
111. int a. "Please check \n"
yuv411bytesperimage=FRAM b. " - if the kernel
121. double map_matrix[3][3];
EW*FRAMEH*12/8; modules
122. int save_image = 0;
`ieee1394',`raw13
123. int image_no = 0;
94' and `ohci1394'
124. int save_ellipse = 0;
are loaded \n"
112. int 125. int ellipse_no = 0;
c. " - if you have
cameramode[2]={MODE_640 126. char eye_file[30];
read/write access
x480_MONO,MODE_640x48 127. char scene_file[30];
to
0_YUV411}; 128. char ellipse_file[40];
/dev/raw1394\n\
n");
147. exit(1);
129. #define YUV2RGB(y, u, v, r, g, 148. }
113. const double beta = 0.2;
//hysteresis factor b)\
for noise reduction 130. r = y + ((v*1436) >> 10);\
114. double 131. g = y - ((u*352 + v*731) >>
*intensity_factor_hori = 10);\
__________________________________________________________________________________
173
Chapter7
______________________________________________________________________________
149. numNodes = e. FRAMERATE_30,4 182. int i;

raw1394_get_nodecount(ha 0,1,"/dev/video13
ndle); 94",
150. camera_nodes = f. &cameras[i])!=DC
dc1394_get_camera_nodes( 1394_SUCCESS) { 183. for (i=0; i<numCameras; i++)
handle,&numCameras,1); 160. fprintf( stderr,"unable to {
151. fflush(stdout); setup camera\n"); 184. dc1394_dma_done_with_bu
152. if (numCameras<1) { 161. dc1394_release_camera(han ffer(&cameras[i]);
153. fprintf( stderr, "no cameras dle,&cameras[i]); 185. }
found :(\n"); 162. dc1394_destroy_handle(han 186. }
154. dc1394_destroy_handle(han dle);
dle); 163. exit(1);
155. exit(1); 164. }
187. void Close_IEEE1394()
156. } 165. if
188. {
(dc1394_start_iso_transmissi
189. int i;
on(handle,cameras[i].node)
!=DC1394_SUCCESS) {
157. for (i = 0; i < numCameras; 166. fprintf( stderr, "unable to
i++) { start camera iso 190. for (i=0; i<numCameras; i++)
158. dc1394_camera_on(handle, transmission\n"); {
camera_nodes[i]); 167. dc1394_release_camera(han 191. if
dle,&cameras[i]); (dc1394_stop_iso_transmissi
168. dc1394_destroy_handle(han on(handle,cameras[i].node)!
dle); =DC1394_SUCCESS) {
159. if
169. exit(1); 192. printf("couldn't stop the
(dc1394_dma_setup_capture
170. } camera?\n");
(handle,camera_nodes[i],
171. printf("Camera %d 193. }
1. i
Open\n",i); 194. dc1394_camera_off(handle,
,
172. } cameras[i].node);
173. } 195. dc1394_dma_release_camer
/
* a(handle,&cameras[i]);
196. }
c 197. dc1394_destroy_handle(han
174. void Grab_IEEE1394()
h dle);
175. {
a 198. }
176. if
n (dc1394_dma_multi_capture
n (cameras,
e numCameras)!=DC1394_SUC 199. //------------ map pupil
l CESS) { coordinates to screen
177. fprintf( stderr, "unable to coordinates ---------/
* capture a frame\n"); 200. CvPoint
/ 178. } homography_map_point(CvP
179. } oint p)
b. FORMAT_VGA_N
201. {
ONCOMPRESSED,
202. CvPoint p2;
c. cameramode[i],
d. SPEED_400, 180. void Release_IEEE1394()
181. {
__________________________________________________________________________________
174
Chapter7
______________________________________________________________________________
203. double z = 204. p2.x =

map_matrix[2][0]*p.x + (int)((map_matrix[0][0]*p.x +
map_matrix[2][1]*p.y + map_matrix[0][1]*p.y +
map_matrix[2][2]; map_matrix[0][2])/z);
205. p2.y = 224. int i, j; 253. scene_nor =
(int)((map_matrix[1][0]*p.x + 225. double result[9]; normalize_point_set(cal_sce
map_matrix[1][1]*p.y + 226. double v = 0; ne, dis_scale_scene,
map_matrix[1][2])/z); 227. for (j = 0; j < 3; j++) { scene_center,
206. return p2; 228. for (i = 0; i < 3; i++) { CALIBRATIONPOINTS);
207. } 229. v = a[j][0]*b[0][i]; 254. eye_nor =
230. v += a[j][1]*b[1][i]; normalize_point_set(cal_eye
231. v += a[j][2]*b[2][i]; , dis_scale_eye, eye_center,
232. result[j*3+i] = v; CALIBRATIONPOINTS);
208. // r is result matrix 233. }
209. void 234. }
affine_matrix_inverse(doubl 235. for (i = 0; i < 3; i++) {
e a[][3], double r[][3]) 236. r[i][0] = result[i*3]; 255. printf("normalize_point_set
210. { 237. r[i][1] = result[i*3+1]; end\n");
211. double det22 = 238. r[i][2] = result[i*3+2]; 256. printf("scene scale:%lf
a[0][0]*a[1][1] - 239. } center (%lf, %lf)\n",
a[0][1]*a[1][0]; 240. } dis_scale_scene,
212. r[0][0] = a[1][1]/det22; scene_center.x,
213. r[0][1] = -a[0][1]/det22; scene_center.y);
214. r[1][0] = -a[1][0]/det22; 257. printf("eye scale:%lf center
215. r[1][1] = a[0][0]/det22; 241. int (%lf, %lf)\n", dis_scale_eye,
cal_calibration_homography( eye_center.x, eye_center.y);
void)
242. {
216. r[2][0] = r[2][1] = 0; 243. int i, j;
217. r[2][2] = 1/a[2][2]; 244. stuDPoint cal_scene[9], 258. const int homo_row=18,
cal_eye[9]; homo_col=9;
245. stuDPoint scene_center, 259. double
eye_center, *eye_nor, A[homo_row][homo_col];
218. r[0][2] = -r[2][2] * 260. int M = homo_row, N =
*scene_nor;
(r[0][0]*a[0][2] + homo_col; //M is row; N is
246. double dis_scale_scene,
r[0][1]*a[1][2]); column
dis_scale_eye;
219. r[1][2] = -r[2][2] * 261. double **ppa =
(r[1][0]*a[0][2] + (double**)malloc(sizeof(dou
r[1][1]*a[1][2]); ble*)*M);
220. } 247. for (i = 0; i < 9; i++) { 262. double **ppu =
248. cal_scene[i].x = (double**)malloc(sizeof(dou
scenecalipoints[i].x; ble*)*M);
249. cal_scene[i].y = 263. double **ppv =
221. // r is result matrix
scenecalipoints[i].y; (double**)malloc(sizeof(dou
222. void
250. cal_eye[i].x = vectors[i].x; ble*)*N);
matrix_multiply33(double
251. cal_eye[i].y = vectors[i].y; 264. double pd[homo_col];
a[][3], double b[][3], double
252. } 265. for (i = 0; i < M; i++) {
r[][3])
266. ppa[i] = A[i];
223. {
______________________________________________________________________________
175
Chapter7
______________________________________________________________________________
267. ppu[i] = 296. min_d_index = i; 316. printf("%8lf ",

(double*)malloc(sizeof(doubl 297. } map_matrix[j][i]);
e)*N); 317. }
268. } 318. printf("\n");
269. for (i = 0; i < N; i++) { 319. }
270. ppv[i] = 298. for (i = 0; i < N; i++) { 320. printf("\nT: \n");
(double*)malloc(sizeof(doubl 299. map_matrix[i/3][i%3] = 321. for (j = 0; j < 3; j++) {
e)*N); ppv[i][min_d_index]; //the 322. for (i = 0; i < 3; i++) {
271. } column of v that corresponds 323. printf("%8lf ", T[j][i]);
to the smallest singular 324. }
value, 325. printf("\n");
i. //whic 326. }
272. for (j = 0; j< M; j++) { h is the
273. if (j%2 == 0) { solutio
274. A[j][0] = A[j][1] = A[j][2] = 0; n of
275. A[j][3] = -eye_nor[j/2].x; the 327. matrix_multiply33(map_mat
276. A[j][4] = -eye_nor[j/2].y; equati rix, T, map_matrix);
277. A[j][5] = -1; ons
278. A[j][6] = scene_nor[j/2].y * 300. }
eye_nor[j/2].x;
279. A[j][7] = scene_nor[j/2].y * 328. T[0][0] = T[1][1] =
eye_nor[j/2].y; dis_scale_scene;
280. A[j][8] = scene_nor[j/2].y; 301. double T[3][3] = {0}, T1[3][3] 329. T[0][2] = -
281. } else { = {0}; dis_scale_scene*scene_cent
282. A[j][0] = eye_nor[j/2].x; 302. printf("\nT1: \n"); er.x;
283. A[j][1] = eye_nor[j/2].y; 303. for (j = 0; j < 3; j++) { 330. T[1][2] = -
284. A[j][2] = 1; 304. for (i = 0; i < 3; i++) { dis_scale_scene*scene_cent
285. A[j][3] = A[j][4] = A[j][5] = 0; 305. printf("%8lf ", T1[j][i]); er.y;
286. A[j][6] = -scene_nor[j/2].x * 306. } 331. T[2][2] = 1;
eye_nor[j/2].x; 307. printf("\n");
287. A[j][7] = -scene_nor[j/2].x * 308. }
eye_nor[j/2].y;
332. printf("\nmap_matrix: \n");
288. A[j][8] = -scene_nor[j/2].x;
333. for (j = 0; j < 3; j++) {
289. }
309. T[0][0] = T[1][1] = 334. for (i = 0; i < 3; i++) {
290. }
dis_scale_eye; 335. printf("%8lf ",
310. T[0][2] = - map_matrix[j][i]);
dis_scale_eye*eye_center.x; 336. }
291. printf("normalize_point_set 311. T[1][2] = - 337. printf("\n");
end\n"); dis_scale_eye*eye_center.y; 338. }
312. T[2][2] = 1; 339. printf("\nT: \n");
340. for (j = 0; j < 3; j++) {
341. for (i = 0; i < 3; i++) {
292. svd(M, N, ppa, ppu, pd, ppv); 342. printf("%8lf ", T[j][i]);
293. int min_d_index = 0; 313. printf("\nmap_matrix: \n"); 343. }
294. for (i = 1; i < N; i++) { 314. for (j = 0; j < 3; j++) { 344. printf("\n");
295. if (pd[i] < pd[min_d_index]) 315. for (i = 0; i < 3; i++) { 345. }
______________________________________________________________________________
176
Chapter7
______________________________________________________________________________
373. // correct eye position by 393. return p2;

recentering offset: 394. }
374. x1 = (float) p.x;
375. y1 = (float) p.y;
346. affine_matrix_inverse(T, T1);
347. matrix_multiply33(T1,
map_matrix, map_matrix);
376. // translate before
biquadratic:
377. x1 -= inx; 395. //---------- calibration
348. printf("\nmap_matrix: \n"); 378. y1 -= iny; coefficient calculation ---------
349. for (j = 0; j < 3; j++) { ------//
350. for (i = 0; i < 3; i++) { 396. // biquadratic equation fitter
351. printf("%8lf ", 397. // x, y are coordinates of eye
map_matrix[j][i]); 379. // biquadratic mapping: tracker point
352. } 380. xx = 398. // X is x or y coordinate of
353. printf("\n"); aa+bb*x1+cc*y1+dd*x1*x1+ screen point
354. } ee*y1*y1; 399. // computes a, b, c, d, and e
381. yy = in the biquadratic
ff+gg*x1+hh*y1+ii*x1*x1+jj* 400. // X = a + b*(x-inx) + c*(y-iny)
y1*y1; + d*(x-inx)*(x-inx) + e*(y-
355. for (i = 0; i < M; i++) { iny)*(y-iny)
356. free(ppu[i]); 401. // where inx = x1, y1 = y1 to
357. } reduce the solution to a 4x4
358. for (i = 0; i < N; i++) { 382. // translate after biquadratic: matrix
359. free(ppv[i]); 383. x1 = xx - centx;
360. } 384. y1 = yy - centy;
361. free(ppu);
362. free(ppv); 402. void dqfit( float x1, float y1,
363. free(ppa); a. float x2, float y2,
385. // determine quadrant of b. float x3, float y3,
point: c. float x4, float y4,
386. if (( x1<0 )&&( y1<0 )) d. float x5, float y5,
364. free(eye_nor); quad = 0; e. float X1, float X2,
365. free(scene_nor); 387. else if (( x1>0 )&&( y1<0 )) float X3, float X4,
366. printf("\nfinish calculate quad = 1; float X5 )
calibration\n"); 388. else if (( x1<0 )&&( y1>0 )) 403. {
367. } quad = 2; 404. float den;
389. else if (( x1>0 )&&( y1>0 )) 405. float x22,x32,x42,x52; //
quad = 3; squared terms
406. float y22,y32,y42,y52;
368. CvPoint map_point(CvPoint
p)
369. { 390. // fix up by quadrant:
370. CvPoint p2; 391. p2.x = (int)(xx + 407. inx = (int)x1; // record
371. int quad=0; x1*y1*cmx[quad]); eye tracker centering
372. float x1,y1,xx,yy; 392. p2.y = (int)(yy + constants
x1*y1*cmy[quad]); 408. iny = (int)y1;
409. a = X1; // first
coefficient
______________________________________________________________________________
177
Chapter7
______________________________________________________________________________
410. X2 -= X1; X3 -= X1; // 430. X2*x52*y4*y32- 441. x3*y2*X4*y52-

center screen points x32*y4*X2*y52- x3*y2*y42*X5+x3*y42*y5*X
411. X4 -= X1; X5 -= X1; x32*y2*y42*X5+x32*y2*X4* 2-x3*y4*X2*y52-
412. x2 -= x1; x3 -= x1; // y52+ 442. y22*y4*x5*X3+y22*X4*y3*x
center eye tracker points 431. X4*x22*y5*y32- 5-
413. x4 -= x1; x5 -= x1; y42*x22*y5*X3- y2*X4*x5*y32+y2*y42*x5*X
414. y2 -= y1; y3 -= y1; x22*y4*y32*X5+x22*y4*X3* 3+
415. y4 -= y1; y5 -= y1; y52+ 443. x2*y3*y42*X5-
416. x22 = x2*x2; x32 = x3*x3; // 432. y22*x42*y5*X3+x32*y22*y4 y42*y3*x5*X2+X4*x2*y5*y3
squared terms of biquadratic *X5-y22*x52*y4*X3- 2+y4*X2*x5*y32-
417. x42 = x4*x4; x52 = x5*x5; x32*y22*y5*X4)/den; 444. y42*x2*y5*X3-
418. y22 = y2*y2; y32 = y3*y3; x2*y4*y32*X5+x2*y4*X3*y5
419. y42 = y4*y4; y52 = y5*y5; 2-x2*y3*X4*y52)/den;
433. c = (-
x32*x4*y22*X5+x32*x5*y22
420. //Cramer's rule solution of *X4- 445. e = -(-
4x4 matrix */ x32*y42*x5*X2+x32*X2*x4* x3*y2*x52*X4+x22*y3*x4*X
421. den = -x2*y3*x52*y42- y52+ 5+x22*y4*x5*X3-
x22*y3*x4*y52+x22*y5*x4* 434. x32*x2*y42*X5- x3*x42*y5*X2-
y32-y22*x42*y3*x5- x32*x2*X4*y52- 446. x42*x2*y3*X5+x42*x2*y5*X
422. x32*y22*x4*y5- x3*y22*x52*X4+x3*y22*x42 3+x42*y3*x5*X2-
x42*x2*y5*y32+x32*x2*y5* *X5+ y2*x42*x5*X3+
y42-y2*x52*x4*y32+ 435. x3*x22*X4*y52- 447. x32*x2*y4*X5-
423. x52*x2*y4*y32+y22*x52*y3 x3*X2*x42*y52+x3*X2*x52* x22*y3*x5*X4+x32*y2*x5*X
*x4+y2*x42*x5*y32+x22*y3 y42-x3*x22*y42*X5- 4-x22*y5*x4*X3+
*x5*y42- 436. y22*x42*x5*X3+y22*x52*x4 448. x2*y3*x52*X4-
424. x32*x2*y4*y52- *X3+x22*y42*x5*X3- x52*x2*y4*X3-
x3*y22*x52*y4+x32*y22*x5 x22*x4*X3*y52- x52*y3*x4*X2-
*y4-x32*y2*x5*y42+ 437. x2*y32*x42*X5+X2*x42*x5* x32*y2*x4*X5+
425. x3*y22*x42*y5+x3*y2*x52* y32+x2*X3*x42*y52+x2*y32 449. x3*x22*y5*X4+x3*y2*x42*X
y42+x32*y2*x4*y52+x42*x2 *x52*X4+ 5+y2*x52*x4*X3-
*y3*y52- 438. x22*x4*y32*X5- x32*x5*y4*X2-
426. x3*y2*x42*y52+x3*x22*y4* x22*X4*x5*y32- 450. x32*x2*y5*X4+x3*x52*y4*X
y52-x22*y4*x5*y32- X2*x52*x4*y32- 2+x32*x4*y5*X2-
x3*x22*y5*y42; x2*X3*x52*y42)/den; x3*x22*y4*X5)/den;
451. }
427. b = (-y32*y2*x52*X4- 439. d = -(-

X2*y3*x52*y42- x4*y22*y3*X5+x4*y22*y5*X
x22*y3*X4*y52+x22*y3*y42 3-
*X5+ x4*y2*X3*y52+x4*y2*y32*X 452. int CalculateCalibration(void)
428. y32*y2*x42*X5- 5- 453. {
y22*x42*y3*X5+y22*y3*x52 440. x4*y32*y5*X2+x4*y3*X2*y5 454. int i, j;
*X4+X2*x42*y3*y52+ 2- 455. float x, y, wx[9], wy[9];
429. X3*y2*x52*y42- x3*y22*y5*X4+x3*y22*y4*X //work data
X3*y2*x42*y52- 5+ points
X2*x42*y5*y32+x32*y42*y5
*X2+
______________________________________________________________________________
178
Chapter7
______________________________________________________________________________
456. int calx[10], caly[10]; 474. // Solve Y biquadratic 497. cmx[i] = (calx[j]-wx[j]-
//scene 475. dqfit((float)eye_x[0],(float)ey centx)/(wx[j]*wy[j]);
coordinate interpolation e_y[0],(float)eye_x[1],(float) 498. cmy[i] = (caly[j]-wy[j]-
variables eye_y[1],(float)eye_x[2], centy)/(wx[j]*wy[j]);
457. int eye_x[10], eye_y[10]; 476. (float)eye_y[2],(float)eye_x[3 499. }
//scene ],(float)eye_y[3],(float)eye_x[
coordinate interpolation 4],(float)eye_y[4],
variables 477. (float)caly[0],(float)caly[1],(fl
oat)caly[2],(float)caly[3],(floa 500. return 0;
t)caly[4]); 501. }
478. ff = a; gg = b; hh = c; ii = d; jj =
458. // Place scene coordinates e;
into calx and caly
459. for(i = 0; i<9;i++) {
460. calx[i] = scenecalipoints[i].x;
caly[i] = scenecalipoints[i].y; 479. // Biquadratic mapping of 502. void Draw_Cross(IplImage
461. } points *image, int centerx, int
480. for(i = 0; i < 9; i++) { centery, int x_cross_length,
481. x = (float)(eye_x[i] - inx); int y_cross_length, double
482. y = (float)(eye_y[i] - iny); color)
462. // Set the last "tenth" point 483. wx[i] = 503. {
463. calx[9] = scenecalipoints[0].x; aa+bb*x+cc*y+dd*x*x+ee*y 504. CvPoint pt1,pt2,pt3,pt4;
caly[9] = scenecalipoints[0].y; *y;
484. wy[i] =
ff+gg*x+hh*y+ii*x*x+jj*y*y;
505. pt1.x = centerx -
485. }
464. // Store pupil into eye_x and x_cross_length;
eye_y 506. pt1.y = centery;
465. for(i = 0; i < 9; i++) { 507. pt2.x = centerx +
466. eye_x[i] = vectors[i].x; 486. // Shift screen points to x_cross_length;
467. eye_y[i] = vectors[i].y; center for quadrant compute 508. pt2.y = centery;
468. } 487. centx = wx[0];
488. centy = wy[0];
509. pt3.x = centerx;

469. // Solve X biquadratic 510. pt3.y = centery -
470. dqfit((float)eye_x[0],(float)ey 489. // Normalize to center: y_cross_length;
e_y[0],(float)eye_x[1],(float) 490. for(i = 0; i < 9; i++) { 511. pt4.x = centerx;
eye_y[1],(float)eye_x[2], 491. wx[i] -= centx; 512. pt4.y = centery +
471. (float)eye_y[2],(float)eye_x[3 492. wy[i] -= centy; y_cross_length;
],(float)eye_y[3],(float)eye_x[ 493. }
4],(float)eye_y[4],
472. (float)calx[0],(float)calx[1],(fl
oat)calx[2],(float)calx[3],(floa 513. cvLine(image,pt1,pt2,color,1,
t)calx[4]); 494. // Compute coefficents for 8);
473. aa = a; bb = b; cc = c; dd = d; each quadrant 514. cvLine(image,pt3,pt4,color,1,
ee = e; 495. for(i = 0; i < 4; i++) { 8);
496. j = i + 5; 515. }
_____________________________________________________________________________________
179
Chapter7
________________________________________________________________________
516. void 554. } else {

Show_Calibration_Points()
517. { 539. if
518. int i; (number_calibration_points_
519. for set<CALIBRATIONPOINTS) { 555. Zero_Calibration();
(i=0;i<CALIBRATIONPOINTS;i
++)
520. Draw_Cross(scene_image,
540. //store xy mouse "scene" 556. }
scenecalipoints[i].x,
coordinates into calibration 557. }
scenecalipoints[i].y, 25, 25,
CV_RGB(255,255,255)); array
521. } 541. scenecalipoints[number_cali
bration_points_set].x = x;
542. scenecalipoints[number_cali 558. void
bration_points_set].y = y; Set_Calibration_Point1(int x,
int y)
522. void Zero_Calibration()
559. {
523. {
524. int i;
543. //grab the "pupil" position
544. pucalipoints[number_calibra
tion_points_set].x = pupil.x; 560. if
545. pucalipoints[number_calibra (number_calibration_points_
525. for
tion_points_set].y = pupil.y; set<CALIBRATIONPOINTS) {
(i=0;i<CALIBRATIONPOINTS;i
++) {
526. scenecalipoints[i].x = 0;
527. scenecalipoints[i].y = 0; 561. //store xy mouse "scene"
546. //grab the "corneal
reflection" points coordinates into calibration
547. crcalipoints[number_calibrati array
on_points_set].x = 562. scenecalipoints[number_cali
528. pucalipoints[i].x = 0;
corneal_reflection.x; bration_points_set].x = x;
529. pucalipoints[i].y = 0;
548. crcalipoints[number_calibrati 563. scenecalipoints[number_cali
on_points_set].y = bration_points_set].y = y;
corneal_reflection.y;
530. crcalipoints[i].x = 0;
531. crcalipoints[i].y = 0;
564. //grab the "pupil" position
549. //grab the "delta pupil cr" 565. pucalipoints[number_calibra
position tion_points_set].x = pupil.x;
532. vectors[i].x = 0; 550. vectors[number_calibration_ 566. pucalipoints[number_calibra
533. vectors[i].y = 0; points_set].x = diff_vector.x; tion_points_set].y = pupil.y;
534. } 551. vectors[number_calibration_
535. number_calibration_points_s points_set].y = diff_vector.y;
et=0;
567. //grab the "corneal
536. }
reflection" points
552. number_calibration_points_s 568. //crcalipoints[number_calibr
et++; ation_points_set].x =
537. void 553. printf("calibration points corneal_reflection.x;
Set_Calibration_Point(int x, number: %d (total 9)\n", 569. //crcalipoints[number_calibr
int y) number_calibration_points_s ation_points_set].y =
538. { et); corneal_reflection.y;
______________________________________________________________________________
180
Chapter7
______________________________________________________________________________
570. 607. }
587. INFO("Calibration result =

%d\n", calibration_result);
571. //grab the "delta pupil cr" 608. }
position
572. vectors[number_calibration_
points_set].x = pupil.x; 588. do_map2scene =
573. vectors[number_calibration_ !do_map2scene;
points_set].y = pupil.y; 589. view_cal_points =
!view_cal_points; 609. void on_mouse_scene( int
event, int x, int y, int flags )
610. {
574. number_calibration_points_s 611. int i;
et++; 590. INFO("Scene
coordinates:\n");
591. for(i=0;i<
CALIBRATIONPOINTS;i++) { 612. switch (event) {
575. } else { 592. INFO("pt %d x = %d , y = %d 613. //This is really the left mouse
\n", i, scenecalipoints[i].x, button
scenecalipoints[i].y); 614. case
593. } CV_EVENT_LBUTTONDOWN:
576. Zero_Calibration(); 594. INFO("\n"); 615. Set_Calibration_Point(x,y);
616. break;
577. } 595. INFO("Eye coordinates\n");

578. } 596. for(i=0;i< 617. //This is really the right
CALIBRATIONPOINTS;i++) { mouse button
597. INFO("pt %d x = %d , y = %d 618. case
\n", i, pucalipoints[i].x, CV_EVENT_MBUTTONDOWN
579. void Activate_Calibration() :
pucalipoints[i].y);
580. { 619. Activate_Calibration();
598. }
581. int i; 620. break;
599. INFO("\n");
582. int calibration_result;
600. INFO("Corneal reflection 621. //This is really the scroll

583. INFO("Map eye to scene button
coordinates\n");
image\n"); 622. case
601. for(i=0;i<
CALIBRATIONPOINTS;i++) { CV_EVENT_RBUTTONDOWN:
602. INFO("pt %d x = %d , y = %d 623. break;
\n", i, crcalipoints[i].x, 624. }
584. if
crcalipoints[i].y); 625. }
(number_calibration_points_
set==CALIBRATIONPOINTS) { 603. }
585. //calibration_result = 604. INFO("\n");
CalculateCalibration(); 605. } else {
626. void on_mouse_eye( int
586. calibration_result = 606. INFO("Attempt to activate
event, int x, int y, int flags )
cal_calibration_homography( calibration without a full set
627. {
); of points.\n");
__________________________________________________________________________________
181
Chapter7
______________________________________________________________________________
628. int i; *now_image, UINT8 684. }

629. static bool start = 0; *next_image) 685. }
652. { 686. */
653. int npixels = FRAMEW * 687. for (y=0;y<height;y++) {
FRAMEH; 688. linesum=1;
630. switch (event) { 654. int i; 689. for
631. //This is really the left mouse 655. for (i = 0; i < npixels; i++) { (x=0;x<width;x+=subsample)
button 656. *result_image = {
632. case (*prev_image + *now_image 690. linesum+=*s;
CV_EVENT_LBUTTONDOWN: + *next_image) / 3; 691. s+=subsample;
633. printf("left mouse eye 657. result_image++; 692. }
window (%d,%d)\n", x, y); 658. prev_image++; 693. s-=width;
634. pupil.x = x; 659. now_image++; 694. factor=hwidth/((double)lines
635. pupil.y = y; 660. next_image++; um);
636. //if (!start) { 661. } 695. for (x=0;x<width;x++) {
637. printf("start point: %d, 662. } 696. *s=(unsigned
%d\n", x, y); char)(((double)*s)*factor);
638. start_point.x = x; 697. s++;
639. start_point.y = y; 698. }
640. start = 1; 663. void 699. }
641. //} Normalize_Line_Histogram(I 700. }
642. break; plImage *in_image)
664. {
665. unsigned char *s=(unsigned
char *)in_image->imageData;
643. //This is really the right 666. int x,y;
mouse button 667. int linesum; 701. void
644. case 668. double factor=0; Calculate_Avg_Intensity_Hori
CV_EVENT_MBUTTONDOWN 669. int subsample=10; (IplImage* in_image)
: 670. double 702. {
645. break; hwidth=(100.0f*(double)widt 703. UINT8 *pixel =
h/(double)subsample); (UINT8*)in_image-
671. /* >imageData;
672. char adjustment; 704. int sum;
646. //This is really the scroll 705. int i, j;
673. for (y=0;y<height;y++) {
button 706. for (j = 0; j < in_image-
674. linesum=0;
647. case >height; j++) {
675. for
CV_EVENT_RBUTTONDOWN: 707. sum = 0;
(x=0;x<width;x+=subsample)
648. break; 708. for (i = 0; i < in_image-
{
649. } >width; i++) {
676. linesum+=*s;
650. } 709. sum += *pixel;
677. s+=subsample;
678. } 710. pixel++;
679. s-=width; 711. }
680. adjustment=(char)(128- 712. avg_intensity_hori[j] =
(double)(linesum)/(double)( (double)sum/in_image-
651. void Average_Frames(UINT8 width/subsample)); >width;
*result_image, UINT8 681. for (x=0;x<width;x++) { 713. }
*prev_image, UINT8 682. *s=MIN(*s+adjustment,255); 714. }
683. s++;
__________________________________________________________________________________
182
Chapter7
______________________________________________________________________________
715. 736. register int j = NumPixels + ( 768. void Grab_Camera_Frames()

NumPixels << 1 )-1; 769. {
737. register int y0, y1, y2, y3, u, 770. Grab_IEEE1394();
v;
716. void 738. register int r, g, b;
Reduce_Line_Noise(IplImage
* in_image) 771. memcpy(eye_image-
717. { >imageData,(char
718. UINT8 *pixel = 739. while (i > 0) { *)cameras[0].capture_buffer,
(UINT8*)in_image- 740. y3 = (unsigned char) src[i--]; monobytesperimage);
>imageData; 741. y2 = (unsigned char) src[i--]; 772. //memcpy(scene_image-
719. int i, j; 742. v = (unsigned char) src[i--] - >imageData,(char
720. double beta2 = 1 - beta; 128; *)cameras[1].capture_buffer,
721. int adjustment; 743. y1 = (unsigned char) src[i--]; monobytesperimage);
744. y0 = (unsigned char) src[i--]; 773. FirewireFrame_to_RGBIplIm
745. u = (unsigned char) src[i--] - age((unsigned char
128; *)cameras[1].capture_buffer,
722. Calculate_Avg_Intensity_Hori 746. YUV2RGB (y3, u, v, r, g, b); scene_image);
(in_image); 747. dest[j--] = r;
723. for (j = 0; j < in_image- 748. dest[j--] = g;
>height; j++) { 749. dest[j--] = b;
724. intensity_factor_hori[j] = 750. YUV2RGB (y2, u, v, r, g, b); 774. original_eye_image =
avg_intensity_hori[j]*beta + 751. dest[j--] = r; cvCloneImage(eye_image);
intensity_factor_hori[j]*beta 752. dest[j--] = g;
2; 753. dest[j--] = b;
725. adjustment = 754. YUV2RGB (y1, u, v, r, g, b);
(int)(intensity_factor_hori[j] - 775. if (frame_number == 0) {
755. dest[j--] = r;
avg_intensity_hori[j]); 776. Calculate_Avg_Intensity_Hori
756. dest[j--] = g;
726. for (i = 0; i < in_image- (eye_image);
757. dest[j--] = b;
>width; i++) { 777. memcpy(intensity_factor_ho
758. YUV2RGB (y0, u, v, r, g, b);
727. *pixel = ri, avg_intensity_hori,
759. dest[j--] = r;
FIX_UINT8(*pixel+adjustmen eye_image-
760. dest[j--] = g;
t); >height*sizeof(double));
761. dest[j--] = b;
728. pixel++; 778. }
762. }
729. } 763. }
730. }
731. }
779. Release_IEEE1394();
780. frame_number++;
764. void
781. }
FirewireFrame_to_RGBIplIm
732. //----------------------- uyyvyy age(void *FirewireFrame,
(i.e. YUV411) to rgb24 --------- IplImage *OpenCV_image)
--------------// 765. { 782. void process_image()
733. void uyyvyy2rgb (unsigned 766. uyyvyy2rgb((unsigned char 783. {
char *src, unsigned char *)FirewireFrame, (unsigned 784. int i, j;
*dest, unsigned long long int char *)OpenCV_image- 785. int *inliers_index;
NumPixels) >imageData, 640*480);
734. { 767. }
735. register int i = NumPixels + (
NumPixels >> 1 )-1;
__________________________________________________________________________________
183
Chapter7
______________________________________________________________________________
786. CvSize ellipse_axis; pupil_param[3],

787. CvPoint gaze_point; pupil_param[4],
788. static int lost_frame_num = 804. //starburst pupil contour inliers_num);
0; detection
789. Grab_Camera_Frames(); 805. starburst_pupil_contour_det
790. cvZero(ellipse_image); ection((UINT8*)eye_image-
>imageData, eye_image- 817. bool is_inliers = 0;
>width, eye_image->height, 818. for (int i = 0; i <
i. edge_t edge_point.size(); i++) {
791. cvSmooth(eye_image, hresho 819. is_inliers = 0;
eye_image, CV_GAUSSIAN, 5, ld, 820. for (int j = 0; j < inliers_num;
5); rays, j++) {
792. Reduce_Line_Noise(eye_ima min_fe 821. if (i == inliers_index[j])
ge); ature_ 822. is_inliers = 1;
candid 823. }
ates); 824. stuDPoint *edge =
edge_point.at(i);
793. if (save_image == 1) { 825. if (is_inliers)
794. printf("save image %d\n", 826. Draw_Cross(ellipse_image,
image_no); 806. inliers_num = 0; (int)edge->x,(int)edge->y, 5,
795. sprintf(eye_file, 807. inliers_index = 5, Green);
"./Eye/Eye_%05d.jpg", pupil_fitting_inliers((UINT8*) 827. else
image_no); eye_image->imageData, 828. Draw_Cross(ellipse_image,
796. image_no++; eye_image->width, (int)edge->x,(int)edge->y, 3,
797. cvSaveImage(eye_file, eye_image->height, 3, Yellow);
eye_image); inliers_num); 829. }
798. } 808. ellipse_axis.width = 830. free(inliers_index);
(int)pupil_param[0];
809. ellipse_axis.height =
(int)pupil_param[1];
799. //corneal reflection 810. pupil.x = (int)pupil_param[2]; 831. if (ellipse_axis.width > 0 &&
800. remove_corneal_reflection(e 811. pupil.y = (int)pupil_param[3]; ellipse_axis.height > 0) {
ye_image, threshold_image, 812. Draw_Cross(ellipse_image, 832. start_point.x = pupil.x;
(int)start_point.x, pupil.x, pupil.y, 15, 15, Red); 833. start_point.y = pupil.y;
(int)start_point.y, 813. cvLine(eye_image, pupil, 834. //printf("start_point:
cr_window_size, corneal_reflection, Red, 4, %d,%d\n", start_point.x,
801. (int)eye_image->height/10, 8); start_point.y);
corneal_reflection.x, 814. cvLine(ellipse_image, pupil, 835. Draw_Cross(eye_image,
corneal_reflection.y, corneal_reflection, Red, 4, pupil.x, pupil.y, 10, 10,
corneal_reflection_r); 8); Green);
802. printf("corneal reflection: 836. cvEllipse(eye_image, pupil,
(%d, %d)\n", ellipse_axis, -
corneal_reflection.x, pupil_param[4]*180/PI, 0,
corneal_reflection.y); 815. printf("ellipse a:%lf; b:%lf, 360, Red, 2);
803. Draw_Cross(ellipse_image, cx:%lf, cy:%lf, theta:%lf; 837. cvEllipse(ellipse_image,
corneal_reflection.x, inliers_num:%d\n\n", pupil, ellipse_axis, -
corneal_reflection.y, 15, 15, 816. pupil_param[0], pupil_param[4]*180/PI, 0,
Yellow); pupil_param[1], 360, Green, 2);
pupil_param[2],
__________________________________________________________________________________
184
Chapter7
______________________________________________________________________________
838. pupil_param[2], 873. cvReleaseImage(&original_ey

839. diff_vector.x = pupil.x - pupil_param[3], e_image);
corneal_reflection.x; pupil_param[4]); 874. cvShowImage(scene_window
840. diff_vector.y = pupil.y - 863. } , scene_image);
corneal_reflection.y; 875. cvShowImage(ellipse_windo
841. if (do_map2scene) { w, ellipse_image);
842. gaze_point = 876. cvResizeWindow(eye_windo
homography_map_point(diff 864. printf("Time elapsed: w,320,240);
_vector); %.3f\n", Time_Elapsed()); 877. cvResizeWindow(original_ey
843. printf("gaze_point: 865. fprintf(logfile,"%.3f\t%d\t%d e_window,320,240);
(%d,%d)\n", gaze_point.x, \t%d\t%d\t%d\t%d\t%d\t%d 878. //cvResizeWindow(scene_wi
gaze_point.y); \n", ndow,320,240);
844. Draw_Cross(scene_image, i. Time_E 879. cvResizeWindow(ellipse_win
gaze_point.x, gaze_point.y, lapsed( dow,320,240);
60, 60, Red); ), 880. // only OpenCV 0.9.6 has the
845. } ii. pupil.x, function of
846. lost_frame_num = 0; iii. pupil.y, cvMoveWindow(), now we
847. } else { b. corneal_reflection are using version 0.9.5
848. lost_frame_num++; .x, 881. /*if (first) {
849. } c. corneal_reflection 882. cvMoveWindow(eye_windo
850. if (lost_frame_num > 5) { .y, w, 200, 0);
851. start_point.x = FRAMEW/2; i. diff_ve 883. cvMoveWindow(scene_wind
852. start_point.y = FRAMEH/2; ctor.x, ow, 200+320, 0);
853. } ii. diff_ve 884. cvMoveWindow(ellipse_win
854. Draw_Cross(ellipse_image, ctor.y, dow, 200, 240);
(int)start_point.x, iii. gaze_p 885. first = 0;
(int)start_point.y, 7, 7, Blue); oint.x, 886. }*/
855. Draw_Cross(eye_image, iv. gaze_p
(int)start_point.x, oint.y);
(int)start_point.y, 7, 7, Blue);
887. cvSetTrackbarPos("Edge
Threshold", control_window,
866. if (view_cal_points) pupil_edge_thres);
856. if (save_ellipse == 1) { Show_Calibration_Points(); 888. }
857. printf("save ellipse %d\n", 867. }
ellipse_no);
858. sprintf(ellipse_file,
"./Ellipse/Ellipse_%05d.jpg", 889. void Open_GUI()
868. void Update_Gui_Windows() 890. {
ellipse_no);
869. { 891. int i;
859. ellipse_no++;
870. //static int first = 1;
860. cvSaveImage(ellipse_file,
ellipse_image);
861. fprintf(ellipse_log, "%.3f\t
892. //Make the eye image (in
%8.2lf %8.2lf %8.2lf %8.2lf 871. cvShowImage(eye_window, monochrome):
%8.2lf\n", eye_image); 893. eye_image=cvCreateImageH
862. Time_Elapsed(), 872. cvShowImage(original_eye_ eader(cvSize(640,480), 8, 1 );
pupil_param[0], window,
pupil_param[1], original_eye_image);
__________________________________________________________________________________
185
Chapter7
______________________________________________________________________________
894. eye_image- 910. cvSetMouseCallback(scene_ 929. cvDestroyWindow(control_w

>imageData=(char window, on_mouse_scene); indow);
*)malloc(640*480); 911. cvSetMouseCallback(eye_wi
ndow, on_mouse_eye);
930. cvReleaseImageHeader(&eye
895. //Make the eye image (in _image );
monochrome): 912. cvCreateTrackbar("Edge 931. cvReleaseImageHeader(&thr
896. threshold_image = Threshold", control_window, eshold_image );
cvCloneImage(eye_image); &pupil_edge_thres, 255, 932. cvReleaseImageHeader(&ori
NULL ); ginal_eye_image );
913. cvCreateTrackbar("Rays 933. cvReleaseImageHeader(&elli
Number", control_window, pse_image );
897. //Make the ellipse image (in &rays, 180, NULL ); 934. cvReleaseImageHeader(&sce
RGB) : 914. cvCreateTrackbar("Min ne_image );
898. ellipse_image=cvCreateImag Feature Candidates",
eHeader(cvSize(640,480), 8, control_window,
3 ); &min_feature_candidates,
899. ellipse_image- 30, NULL ); 935. cvReleaseImage(&eye_image
>imageData=(char 915. cvCreateTrackbar("Corneal );
*)malloc(640*480*3); Window 936. cvReleaseImage(&threshold_
Size",control_window, image);
&cr_window_size, FRAMEH, 937. cvReleaseImage(&original_ey
NULL ); e_image);
900. //Make the scene image: 938. cvReleaseImage(&ellipse_im
901. scene_image=cvCreateImage age);
Header(cvSize(640,480), 8, 3 939. cvReleaseImage(&scene_ima
); 916. //Init colors ge);
902. scene_image- 917. White = 940. }
>imageData=(char CV_RGB(255,255,255);
*)malloc(640*480*3); 918. Red = CV_RGB(255,0,0);
919. Green = CV_RGB(0,255,0);
920. Blue = CV_RGB(0,0,255); 941. void Open_Logfile(int argc,
921. Yellow = CV_RGB(255,255,0); char** argv)
903. //Create the windows 942. {
922. }
904. cvNamedWindow(control_wi 943. char
ndow, 1); defaultlogfilename[]="logfile.
905. cvNamedWindow(ellipse_wi txt";
ndow, 0); 923. void Close_GUI() 944. char *logfilename;
906. cvNamedWindow(scene_win 924. {
dow, 0); 925. cvDestroyWindow(eye_wind
907. cvNamedWindow(eye_wind ow);
ow, 0); 926. cvDestroyWindow(original_e 945. if (argc>1) {
908. cvNamedWindow(original_e ye_window); 946. logfilename=argv[1];
ye_window, 0); 927. cvDestroyWindow(ellipse_wi 947. } else {
ndow); 948. logfilename=defaultlogfilena
928. cvDestroyWindow(scene_wi me;
ndow);
909. //setup the mouse call back
funtion here for calibration
_____________________________________________________________________________________
186
Chapter7
______________________________________________________________________________
949. }
950. logfile=fopen(logfilename,"w
+"); 973. int main( int argc, char**
argv )
974. {
975. char c;
951. if (logfile!=NULL) { 1003. while
952. fprintf(logfile,"Timestamp ((c=cvWaitKey(50))!='q') {
(seconds)\t pupil X\t pupil 1004. if (c == 's') {
Y\t Scene X\t Scene Y\n"); 976. Open_IEEE1394(); 1005. sprintf(eye_file,
953. } else { "eye%05d.bmp", image_no);
954. fprintf(stderr,"Error opening 1006. sprintf(scene_file,
logfile %s.",logfilename); "scene%05d.bmp",
955. exit(-1); 977. Open_GUI(); image_no);
956. } 1007. image_no++;
957. } 1008. cvSaveImage(eye
_file, eye_image);
978. Open_Logfile(argc,argv);
1009. cvSaveImage(scen
e_file, scene_image);
958. void Close_Logfile() 1010. printf("thres:
959. { %d\n", pupil_edge_thres);
979. Start_Timer();
960. fclose(logfile); 1011. } else if (c == 'c') {
961. } 1012. save_image = 1 -
save_image;
980. int i, j; 1013. printf("save_imag
981. double T[3][3], T1[3][3]; e = %d\n", save_image);
962. void Open_Ellipse_Log() 1014. } else if (c == 'e') {
982. for (j = 0; j < 3; j++) {
963. { 1015. save_ellipse = 1 -
983. for (i = 0; i < 3; i++) {
964. static char save_ellipse;
984. T[j][i] = j*3+i+1;
*ellipse_log_name = 1016. printf("save_ellips
985. }
"./Ellipse/ellipse_log.txt"; e = %d\n", save_ellipse);
986. }
965. ellipse_log = 1017. if (save_ellipse ==
987. T[2][0] = T[2][1] = 0;
fopen(ellipse_log_name,"w+ 1) {
988. printf("\nT: \n");
"); 1018. Open_Ellipse_Log
989. for (j = 0; j < 3; j++) {
990. for (i = 0; i < 3; i++) { ();
991. printf("%6.2lf ", T[j][i]); 1019. } else {
992. } 1020. fclose(ellipse_log)
966. if (logfile!=NULL) {
993. printf("\n"); ;
967. fprintf(logfile,"Timestamp
994. } 1021. }
(seconds)\t a\t pupil b\t
995. affine_matrix_inverse(T, T1); 1022. }
centerx\t centery\t
996. printf("\nT1: \n"); 1023. if (start_point.x
theta\n");
997. for (j = 0; j < 3; j++) { == -1 && start_point.y == -1)
968. } else {
998. for (i = 0; i < 3; i++) { 1024. Grab_Camera_Fra
969. fprintf(stderr,"Error opening
999. printf("%6.2lf ", T1[j][i]); mes();
logfile %s.",
1000. } 1025. else
ellipse_log_name);
1001. printf("\n"); 1026. process_image();
970. exit(-1);
971. } 1002. }
972. }
__________________________________________________________________________________
187
Chapter7
______________________________________________________________________________
1027. if
(frame_number%1==0)
Update_Gui_Windows();
1028. }
1029. Close_Logfile();
1030. Close_GUI();
1031. Close_IEEE1394();
1032. return 0;
1033. }
1034. #ifdef _EiC

1035. main(1,"cvEyeTra
cker.c");
1036. #endif
7.3.4 remove_corneal_reflection
1. #include <stdio.h> reflection approximate 17. angle_array[i] =

2. #include <stdlib.h> radius i*angle_delta;
3. #include <math.h> 9. crx = cry = crar = -1; 18. sin_array[i] =
4. #include sin(angle_array[i]);
"remove_corneal_reflection. 19. cos_array[i] =
h" cos(angle_array[i]);
20. }
10. float angle_delta = 1*PI/180;
11. int angle_num =
(int)(2*PI/angle_delta);
12. printf("(corneal reflection)
sx:%d; sy:%d\n", sx, sy); 21. locate_corneal_reflection(im
13. double *angle_array = age, threshold_image, sx, sy,
5. void
(double*)malloc(angle_num* window_size,
remove_corneal_reflection(I
sizeof(double)); (int)(biggest_crr/2.5), crx,
plImage *image, IplImage
14. double *sin_array = cry, crar);
*threshold_image, int sx, int
(double*)malloc(angle_num* 22. crr =
sy, int window_size, int
sizeof(double)); fit_circle_radius_to_corneal_
6. biggest_crr, int& crx, int&
15. double *cos_array = reflection(image, crx, cry,
cry, int& crr)
(double*)malloc(angle_num* crar, (int)(biggest_crr/2.5),
7. {
sizeof(double)); sin_array, cos_array,
8. int crar = -1;
16. for (int i = 0; i < angle_num; angle_num);
//corneal
i++) { 23. crr = (int)(2.5*crr);
__________________________________________________________________________________
188
Chapter7
______________________________________________________________________________
24. interpolate_corneal_reflectio 44. cvMinMaxLoc(image, 69. }

n(image, crx, cry, crr, &min_value, &max_value, 70. else
sin_array, cos_array, &min_loc, &max_loc); 71. continue;
angle_num);
45. int threshold, i; 72. if (scores[threshold-1] -

25. free(angle_array); 46. CvSeq* contour=NULL; scores[threshold] < 0) {
26. free(sin_array); 47. CvMemStorage* storage = 73. //found the corneal
27. free(cos_array); cvCreateMemStorage(0); reflection
28. } 48. double *scores = 74. crar = (int)sqrt(max_area /
(double*)malloc(sizeof(doubl PI);
e)*((int)max_value+1)); 75. int sum_x = 0;
49. memset(scores, 0, 76. int sum_y = 0;
sizeof(double)*((int)max_val 77. CvPoint *point;
29. void ue+1)); 78. for (i = 0; i < max_contour-
locate_corneal_reflection(IplI 50. int area, max_area, >total; i++) {
mage *image, IplImage sum_area; 79. point =
*threshold_image, int sx, int 51. for (threshold = CV_GET_SEQ_ELEM(CvPoint,
sy, int window_size, int (int)max_value; threshold >= max_contour, i);
30. biggest_crar, int &crx, int 1; threshold--) { 80. sum_x += point->x;
&cry, int &crar) 52. cvThreshold(image, 81. sum_y += point->y;
31. { threshold_image, threshold, 82. }
32. if (window_size%2 == 0) { 1, CV_THRESH_BINARY); 83. crx = sum_x/max_contour-
33. printf("Error! window_size 53. cvFindContours(threshold_i >total;
should be odd!\n"); mage, storage, &contour, 84. cry = sum_y/max_contour-
34. } sizeof(CvContour), >total;
CV_RETR_LIST, 85. break;
CV_CHAIN_APPROX_NONE); 86. }
54. max_area = 0; 87. }
55. sum_area = 0; 88. /*/ printf("(corneal
35. int r = (window_size-1)/2; 56. CvSeq *max_contour = reflection) max_value = %lf;
36. int startx = MAX(sx-r, 0); contour; threshold = %d\n",
37. int endx = MIN(sx+r, image- 57. for( ; contour != 0; contour = max_value, threshold);
>width-1); contour->h_next) { 89. printf("(corneal reflection)
38. int starty = MAX(sy-r, 0); 58. area = contour->total + Scores:\n");
39. int endy = MIN(sy+r, image- (int)(fabs(cvContourArea(con 90. for (int i = (int)max_value; i
>height-1); tour, CV_WHOLE_SEQ))); >= threshold-1; i--) {
40. cvSetImageROI(image, 59. sum_area += area; 91. printf("%6.2lf", scores[i]);
cvRect(startx, starty, endx- 60. if (area > max_area) { 92. }
startx+1, endy-starty+1)); 61. max_area = area; 93. printf("\n");*/
41. cvSetImageROI(threshold_im 62. max_contour = contour;
age, cvRect(startx, starty, 63. }
endx-startx+1, endy- 64. }
starty+1)); 65. if (sum_area-max_area > 0) {
66. scores[threshold-1] = 94. free(scores);
max_area / (sum_area- 95. cvReleaseMemStorage(&stor
max_area); age);
67. //printf("max_area: %d, 96. cvResetImageROI(image);
42. double min_value, max_contour: %d, sum_area: 97. cvResetImageROI(threshold_
max_value; %d; scores[%d]: %lf\n", image);
43. CvPoint min_loc, max_loc; 68. // max_area,
//location max_contour->total,
sum_area, threshold-1,
scores[threshold-1]);
__________________________________________________________________________________
189
Chapter7
______________________________________________________________________________
98. if (crar > biggest_crar) { 121. x = (int)(crx + 146. if (crx == -1 || cry == -1 || crr
99. printf("(corneal) size wrong! (r+r_delta)*cos_array[i]); == -1)
crx:%d, cry:%d, crar:%d 122. y = (int)(cry + 147. return;
(should be less than %d)\n", (r+r_delta)*sin_array[i]);
crx, cry, crar, biggest_crar); 123. x2 = (int)(crx + (r-
100. cry = crx = -1; r_delta)*cos_array[i]);
101. crar = -1; 124. y2 = (int)(cry +
102. } (r+r_delta)*sin_array[i]); 148. if (crx-crr < 0 || crx+crr >=
125. if ((x >= 0 && y >=0 && x < image->width || cry-crr < 0
image->width && y < image- || cry+crr >= image->height)
>height) && {
126. (x2 >= 0 && y2 >=0 && x2 < 149. printf("Error! Corneal
103. if (crx != -1 && cry != -1) { image->width && y2 < reflection is too near the
104. printf("(corneal) startx:%d, image->height)) { image border\n");
starty:%d, crx:%d, cry:%d, 127. sum += *(image- 150. return;
crar:%d\n", startx, starty, crx, >imageData+y*image- 151. }
cry, crar); >width+x);
105. crx += startx; 128. sum2 += *(image-
106. cry += starty; >imageData+y2*image-
107. } >width+x2);
129. } 152. int i, r, r2, x, y;
130. } 153. UINT8 *perimeter_pixel =
131. ratio[r-crar] = sum / sum2; (UINT8*)malloc(array_len*siz
132. if (r - crar >= 2) { eof(int));
108. } 133. if (ratio[r-crar-2] < ratio[r- 154. int sum=0, pixel_value;
crar-1] && ratio[r-crar] < 155. double avg;
ratio[r-crar-1]) { 156. for (i = 0; i < array_len; i++) {
134. free(ratio); 157. x = (int)(crx +
135. return r-1; crr*cos_array[i]);
109. int 136. } 158. y = (int)(cry +
fit_circle_radius_to_corneal_ 137. } crr*sin_array[i]);
reflection(IplImage *image, 138. } 159. perimeter_pixel[i] =
int crx, int cry, int crar, int (UINT8)(*(image-
biggest_crar, double >imageData+y*image-
*sin_array, double >width+x));
*cos_array, int array_len) 160. sum += perimeter_pixel[i];
110. { 139. free(ratio); 161. }
111. if (crx == -1 || cry == -1 || 140. printf("ATTN! 162. avg = sum*1.0/array_len;
crar == -1) fit_circle_radius_to_corneal_
112. return -1; reflection() do not change
the radius\n");
141. return crar;
142. } 163. for (r = 1; r < crr; r++) {
164. r2 = crr-r;
113. double *ratio = 165. for (i = 0; i < array_len; i++) {
(double*)malloc((biggest_cra 166. x = (int)(crx + r*cos_array[i]);
r-crar+1)*sizeof(double)); 167. y = (int)(cry + r*sin_array[i]);
114. int i, r, r_delta=1; 143. void 168. *(image-
115. int x, y, x2, y2; interpolate_corneal_reflectio >imageData+y*image-
116. double sum, sum2; n(IplImage *image, int crx, >width+x) =
117. for (r = crar; r <= int cry, int crr, double (UINT8)((r2*1.0/crr)*avg +
biggest_crar; r++) { *sin_array, double (r*1.0/crr)*perimeter_pixel[i
118. sum = 0; *cos_array, ]);
119. sum2 = 0; 144. int array_len) 169. }
120. for (i = 0; i < array_len; i++) { 145. {
_____________________________________________________________________________________
190
Chapter7
______________________________________________________________________________
170. (r*1.0/crr)*perimeter_pixel[i 173. }

171. //printf("r=%d: %d (avg:%lf, -1]), 174. free(perimeter_pixel);
end:%d)\n", r, 172. // avg, perimeter_pixel[i-
(UINT8)((r2*1.0/crr)*avg + 1]);
7.4 Header files
For all class files there must be header files containing prototypes of functions, so we’ll go back
to the C++ part especially to the function part so that we know how to type the prototype of a
function.
7.4.1 ransac_ellipse
______________________________________________________________________________
191
Chapter7
______________________________________________________________________________
7.4.2 remove_corneal_reflection
7.4.3 svd
7.4.4 testApp
______________________________________________________________________________
192
Chapter7
______________________________________________________________________________
7.4.5 timing
7.5 User’s interface
When you illuminate the eye with IR light and observe it through an IR sensitive camera with a
visible light filter, the iris of the eye turns completely white and the pupil stands out as a high-
contrast black dot. This makes tracking the eye much easier. In order to provide some IR
illumination, we have made a quick and dirty IR LED circuit using connecting wires, IR LEDs
and a 2x AAA battery holder
______________________________________________________________________________
193
Chapter7
______________________________________________________________________________
7.5.1 IR filter removal
without the IR filter removal VS with the IR filter removal
7.5.2 Software
The EyeWriter software is two parts or actually two separate software; an eye-tracking software
designed for use with our low-cost glasses used for calibrating and testing that everything works
fine and the other is theoretically the same but with some changes and a keyboard to actually
type with it. The software for both parts has been developed using openframeworks, OpenCv for
images processing and a c++ library for creative development.
______________________________________________________________________________
194
Chapter7
______________________________________________________________________________
7.5.3 Image processing steps
1. Filtering: from RGB color image to greyscale image to make it easier to process on.
2. Another panel is to make a threshold function in which you can set certain pixels which are
above the threshold value (adjustable) to white and below it to black to distinguish the pupil from
the eye.
______________________________________________________________________________
195
Chapter7
______________________________________________________________________________
3. Finding the contour of the pupil from the binary image.
input: the binary image.
output: the outline of all the objects found in the binary image.
7.6 Final formula
7.6.1 Result and conclusion
1. Simple and inexpensive Prototype costs about 25$.
2. Option for the ALS patients to communicate easily and with a high accuracy based on the rig
(position and calibration)
7.6.2 Futurristic Modifications
1.Wireless rig other than the wired one which will be more reliable and easier for the patient to
use.
2. Smaller camera and rig so that it can be lighter and doesn't feel the burden to wear it or use it.
3. Software compatibility with different operating systems.
______________________________________________________________________________
196
Sources
1. http://www.instructables.com/id/The-EyeWriter/
2. https://www.youtube.com/playlist?list=PLAE85DE8440AA6B83
3. https://www.youtube.com/playlist
4. https://www.edx.org/cour…/introduction-c-microsoft-dev210x-5
5. https://www.youtube.com/watch?v=Rub-JsjMhWY
6. http://eyewriter.org/videos/
7. http://fffff.at/eyewriter/The-EyeWriter.pdf
8. https://opencv.org/
9. http://www.instructables.com/id/The-EyeWriter-20
10. http://www.instructables.com/id/Eye-Writer-30
11. https://www.mediafire.com/folder/0zxtaeqtquziu/Main_Basic
12. http://www.cplusplus.com/files/tutorial.pdf
13. https://docs.opencv.org/2.4/opencv_tutorials.pdf
14. https://openframeworks.cc/
______________________________________________________________________________
197

The Eye Writer Project

Uploaded by

Document Information

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

The Eye Writer Project

Uploaded by

Copyright:

Available Formats

Mansoura University

The Eye-Writer Project

Under supervision of Dr. Ahmed Mohammed Abu-Taleb

Introduced to you by:

Hussein Zaghloul Mohammed

Sameh Nabil Elsharawy

Abdel Rahman Mohammed Abdel Halim

Ahmed Abdel Rahman Mutwally

0.1 Requirements to get through that course ……………………………………………………………………002

1.1 what is ALS ……………………………………………………………………………………………………………….….005

1.3 people who got ALS ……………………………………………………………………………………………………..007

Overview of Eye-Writer ………………………………………………………………………………………………………..011

2.1 What is Eye-Writer ………………………………………………………………………………………………………………………011

2.2 Why Eye-Writer …………………………………………………………………………………………………………………………..011

Hardware and interface of the Eye-writer ………………………………………………………………………….012

3.1 Functional design specifications …………………………………………………………………………………..012

3.2 Parts and tools …………………………………………………………………………………………………….……….013

3.3 Steps of interfacing and hardware solid work …………………………………………………………………………….021

3.4 Connect to the Eye-Writer Software …………………………………………………………………………………………..038

3.5 PS3 Eye drivers & QT components ……………………………………………………………………………………………..039

3.6 Video Capture …………………………………………………………………………………………………………………………….039

3.7 Eye-Tracking Software ………………………………………………………………………………………………………………..039

3.8 Eye-Drawing Software ………………………………………………………………………………………………………………..039

3.9 Eye-Writer installation ………………………………………………………………………………………………………………..040

3.10 On-Screen Drawing …………………………………………………………………………………………………………………..040

3.12 Remote Drawing and Projection ………………………………………………………………………………………………..041

Theoretical background about C++ ……………………………………………………………………………………..042

4.1 What is C++ …………………………………………………………………………………………………………………..042

4.2 Download, install, launch Code::Blocks …………………………………………………………………………042

4.3 Main structure of the program ……………………………………………………………………………………..046

4.4 Fundamental data types ……………………………………………………………………………………………………………..049

4.5 Declaration of variables ……………………………………………………………………………………………………………….049

4.6 Initialization of variables ……………………………………………………………………………………………………………..050

4.7 Operators ……………………………………………………………………………………………………………………………………051

4.8 Conditional structure: if and else ………………………………………………………………………………………………..057

4.9 Iteration structures (loops) …………………………………………………………………………………………………………058

4.10 Jump statements ………………………………………………………………………………………………………………………061

4.11 The selective structure: switch ………………………………………………………………………………………………….064

4.12 Functions (I) ………………………………………………………………………………………………………………………………064

4.13 Functions with no type: void functions ……………………………………………………………………………………..066

4.14 Functions (II) ……………………………………………………………………………………………………………………………..067

4.15 Declaring functions …………………………………………………………………………………………………………………….070

4.16 Arrays …………………………………………………………………………………………………………………………………………073

4.17 Character Sequences ………………………………………………………………………………………………………………….076

4.18 Pointers ………………………………………………………………………………………………………………………………………078

4.19 Classes ………………………………………………………………………………………………………………………………………..079

4.20 Friendship and inheritance …………………………………………………………………………………………………………081

4.21 Separating C++ program into 3 files …………………………………………………………………………………………….089

Theoretical background about OpenCV …………………………………………………………………………….091

5.1 What is OpenCV …………………………………………………………………………………………………………..091

5.3 Installation in Windows …………………………………………………………………………………………………………094

5.5 Load and Display an Image ………………………………………………………………………………………………………..097

5.6 Load, Modify, and Save an Image ………………………………………………………………………………………………098

5.7 How the image matrix is stored in the memory …………………………………………………………………………100

5.8 Discrete Fourier Transform ………………………………………………………………………………………………..………..103

5.9 Video Input with OpenCV and similarity measurement ………………………………………………………...……..106

5.10 How to read a video stream (online-camera or offline-file) ……………………………………………..…….…..113

5.11 Camera calibration With OpenCV ……………………………………………………………………………………………….114

5.12 The calibration and save ……………………………………………………………………………………………………………..121

Theoretical background about OpenFrameworks …………..………………………………………………….. 124

6.1 What is openFrameworks ………………………………………..…………………………………………………….124

6.2 Functionality of openFrameworks ……………………………………………..…………………………………..124

6.3 Operating systems supported by openFrameworks …………………………………………………………124

6.4 Basics of openFrameworks ………………………………………………………….………………………………………………..125