UNIVERSIDADE FEDERAL DO RIO GRANDE DO SUL

INSTITUTO DE INFORMTICA

JAIME ANDRS RIASCOS SALAS

Evaluation mental workload of cognitive

tasks in virtual reality using EOG eye
tracker and EEG signals

Plano de Estudos e Pesquisa

Advisor: Prof. Dr. Dante Barone

Coadvisor: Prof. Dr. Luciana Nedel

Porto Alegre
November 2016
 ABSTRACT

Each day, human beings search for alternatives to communicate themselves with different
machines or systems. Motivated for overcoming the difficulties experienced by impaired
people or, simply, to develop comfortable interfaces, the Brain Computer Interface (BCI)
area was created. It employs brain signals (EEG) to control or communicate with many
devices. Indeed, BCI can be also used to measure, study and understand the brain behavior
in different conditions. Likewise, it is possible to evaluate mental health and cognition.
Techniques like the Electrooculography (EOG) have been used to quantify eye move-
ments, which can be used in visual perception and attention studies. Scenarios in virtual
reality (VR) are an important tool to study the mental states in many situations. VR is
employed to create brain stimulus in the user, simulating a realistic environment. These
responses are measured using EEG. This master research uses BCI to evaluate mental
workload in attention tasks and object recognition in virtual reality. The evaluation will
be done using the EOG as an eye-tracking system (attention) and the EEG signals to es-
timate the cognitive workload. This document presents the introduction and motivation
of the dissertation; its main objectives and the methodology with a revision of previous
works. Finally, a summary of tasks and the schedule is presented.

Keywords: Brain Computer Interface, EOG, EEG, Eye-Tracking, Cognition.

4

1 INTRODUCTION AND MOTIVATION

Brain Computer Interface (BCI) is a system that uses brain signals to communicate
with machines (Computers, Robots and so on). In recent years, these systems have been
focused on research due to the large amount of applications that could be developed,
manly, for disabled people. Indeed, it is a hot research topic due to the fact that the
BCI implies several research fields, such as Signal Processing, Pattern Recognition and
Machine Learning, Neuroscience, Real-Time Systems and others. The importance of this
emergent area can be visualized through the creation of the project BNCI Horizon 2020:
The Future of Brain/Neural Computer Interaction by the European Commission (EC).
This project comprises 12 European partners with a deep background in BCI. Figure 1.1
shows the partners. The main goal of the project is to make a road-map and sketch up the
future of BCI for the next 10 years (or more) (GRAZ, 2015).
Figure 1.1: BCNI Horizon 2020 project partners.

Brazil is not left behind. In 2009, it was created the National Institute for Brain-
Machine Interfaces (INCEMAQ) supported by the National Institutes Program of Science
and Technology of the National Council for Scientific and Technological Development
CNPq/MCT. The main goal was to establish and increase the scientific research in BCI,
involving around of 16 Universities from 9 states of the country. Figure 1.2 shows the
partners and its location in Brazil (IINN-ELS, 2011).
Currently, according to CNPqs Directory of Research Groups in Brazil, there are
10 research groups working in BCI (CNPQ, 2016). The research in BCI in Rio Grande
do Sul state is still beginning and there are not established research groups that work with
focus in this area. For this reason, is essential that Federal University of Rio Grande do
Sul (UFRGS) and its Informatics Institute begins formally with BCI research.

BCI systems have four main steps: signal acquisition; feature extraction; classi-
 5

Figure 1.2: INCEMAQ project partners.

fication and the application interface. Figure 1.3 describeds a BCI system to estimate
imagined hand movements (LOTTE, 2014). A little description of each stage will be
presented below.

Figure 1.3: BCI system (LOTTE, 2014).

1.1 Collected data

The hardware to collect signals from brain depends of the technique that will be
used for that. The main non-invasive techniques are the Electroencephalography (EEG),
Magnetoencephalography (MEG), Functional magnetic resonance imaging (FMRI) and
Functional Near-Infrared Spectroscopy (fNIRS). The characteristics of acquisition meth-
6

ods are summarized in Figure 1.4 (adapted of (ABDULKADER; ATIA; MOSTAFA,

2015)). Also, there is a method based on EEG to acquire data about the eye movements,
called Electrooculography (EOG). It will be explained in the methodology chapter. The
following steps of BCI system will be focused in EEG technique.

Figure 1.4: Characteristics of acquisition methods (ABDULKADER; ATIA; MOSTAFA,

2015).

1.2 Feature extraction

From the data collected, it becomes necessary to extract the relevant information
to identify the mental states of the user and rejected noise. For feature extraction, it is
necessary to make a pre-processing (filtering) from raw-EEG. With the data filtered, three
possible information can be extracted (LOTTE, 2014): a) Spatial data, that describes the
place of the headset where the signal activity occurred. b) Spectral data, which describes
the power of some specific frequency bands, and c) Temporal data, which consist of the
variation of signal values by time. The most relevant methods for feature extraction are
Common Spatial Filter (CSP), Fast Fourier Transform (FFT), Principal Component Anal-
ysis (PCA), Wavelet Decomposition (WD) and so on (VAID; PREETI; KAUR, 2015).

1.3 Classification

The features vector (obtained in the last step) is the input to the classifiers to be
used. The idea is to assign a class to a set of features that allows to identify the mental
state of user. The Machine Learning (ML) approach is used to recognise patterns in
the data. The main techniques are the Linear Discriminant Analysis (LDA), Support
Vector Machine (SVM), Neural Network (NN), Multilayer Perceptron (MLP), K-Nearest
Neighbors (KNN) and so on (LOTTE et al., 2007). The successfulness of BCI relies
on the accuracy of the classifiers used; The cited classifiers achieve more than 70% of
accuracy (ABDULKADER; ATIA; MOSTAFA, 2015). It is clear that the classification
 7

depends of the type of features extracted.

1.4 Application Interface

In the literature, there are many contributions about the use of BCI systems in dif-
ferent applications. Mainly, to provide the movement of people with disabilities(LOPES
et al., 2011; VIDAURRE et al., 2016; RAMLI et al., 2015), the robotics control (BI; FAN;
LIU, 2013; CHAE; JEONG; JO, 2012) and feedback in virtual reality (VR) (GERT et al.,
2011; VOURVOPOULOS; FERREIRA; BADIA, 2016). Indeed, neuroscience applica-
tions have been well done with BCI, since experiments to recognize mental states(JATUPAIBOON;
SETHA; PASIN, 2013), learning and attention (NANDAGOPAL et al., 2013; NUNEZ;
VANDEKERCKHOVE; SRINIVASAN, 2016), until sleep levels (MENSEN; RIEDNER;
TONONI, 2016).

The focus of dissertation will be in the field of computational neuroscience, mainly

the evaluation of cognitive tasks, such as both attention or seeking in virtual reality. The
evaluation will be measured using EEG and EOG. In effect, EOG is used as an Eye Track-
ing (ET) system. The ET using EOG will be explained in the methodology chapter.

On one side, the use of EOG and virtual reality have been very well considered
in physiological research. Courgeon et. al. (COURGEON et al., 2014) developed a joint
attention simulation for research and treatment of autism, with the premise that the inter-
action of the user with virtual humans can enhance the interaction in social behavior. The
authors used an eye-tracking approach so that a virtual human performs a joint attention
task in real-time. Thus, the user trains the gaze control before a social interaction.

Another interesting approach was presented by Boukhalfi et. al. (BOUKHALFI

et al., 2015), where VR and EEG data were used to understand several mental health
disorders, mainly in the forensic psychiatry. Moreover, the authors showed different chal-
lenges in the creation of a therapy using VR. The EEG and ET were used to perform an
adequately neurofeedback stimuli. Finally, the authors highlighted the potential of EEG
and ET to evaluate mental states.

On other side, an important concept in physiological and ergonomics research is

8

the Mental Workload (MWL), which is defined as the relation between the costs to per-
form a task and the mental resources that the human operator employs for that (YOUNG
et al., 2015). Different secondary stimulus can increase MWL. In this sense, Scheer et. al.
(SCHEER; BLTHOFF; CHUANG, 2016) studied the amount of MWL that steering task
takes. For that, a steering task was performed while an irrelevant distraction is executed.
The authors used EEG and EOG to evaluate the overall workload and the event-related
potential (ERP), which is a typical metric in BCI systems.

Finally, in this way, the dissertation is aiming to evaluate the attention using EOG
as an eye-tracking system and the cognitive states with EEG signals. These evaluations
will be performed in a VR scenario that is not established yet. Initially, the possible sce-
nario will be a outdoor environment with natural navigation using a joystick, where the
user performs attention tasks and objects recognition. However, It is still being consid-
ered.
 9

2 OBJECTIVES

The main objective of this dissertation is to evaluate and measure MWL in atten-
tion and object recognition tasks in VR scenarios. To achieve this goal, it is proposes
to use EOG as an eye-tracking system (attention) and EEG to measure and identify the
mental states (cognition). We believe that the successfulness of this objective will be
completed performing the next specific tasks:

a) Review the state-of-art aiming to evaluate the best technique to perform an eye-tracking
with EOG and measure the cognitive workload with EEG;

b) Simulation and implementation of a BCI system with EEG;

c) Implementation of an eye-tracking system with EOG;

d) Evaluation of MWL with EEG on a Graphic User Interface (GUI) and in virtual reality;

e) Data fusion of eye-tracking system and EEG to evaluate the cognitive workload in
virtual reality.

The results can be summarized in four contributions:

1. BCI system using EEG;

2. A real-time eye-tracker system with EOG for the Graphics, Visualization and Interac-
tion Laboratory of Institute of Informatics at UFRGS;

3. Evaluation of MWL on GUI;

4. Evaluation of MWL using EOG-ET and EEG in virtual reality.

The last objectives are strongly dependent of equipment to acquire the data and
its successfulness will be done if it arrives on time. In this case, a EEG system called
Open-BCI that will be described in the next chapter.
10

3 METHODOLOGY

The eyes are our main receptors of information about the world, with their move-
ment we can process relevant information of a visual scene in real-time (perception). This
is used to perform different tasks, such as reading, writing, detection of objects, learning
and so on (AL-RAHAYFEH; FAEZIPOUR, 2013). In recent years, the increase of assis-
tive technology allowed the measurent of eye movements. Thus, Eye Tracker (ET) sys-
tems were created. Initially, preliminary techniques used a video-based system. After, the
Electrooculography (EOG) was used. A revision of methods to detect the eye-movements
is presented in (AL-RAHAYFEH; FAEZIPOUR, 2013; MAJARANT; BULLING, 2014).
The EOG works recording the potential difference of polarization in the cornea and retina
area. This can be taken as an electrical dipole. Figure 3.1 shows the response (electrical
signal) for an eye movement (BAREA et al., 2012).

Figure 3.1: Electrical response of an eye movement. a) Ocular dipole. b) EOG signal

To collect the EOG signal, an EEG electrode can be used and placed near the
ocular zone. The system that will be used is a 16-channel Biosensing 32bit & Daisy
Open BCI Board. It is an Open Source Arduino-compatible component that can be used
to measure the brain activity (EEG), heart activity (EKG) and muscle activity (EMG). For
EOG, the gold cup electrodes could be adapted. Figure 3.2 shows the features of Open
BCI are described (BCI, 2016).
Vidal et. al. (VIDAL et al., 2012) performed a review about the ET equipments
and their importance in experimental psychology research, focused in the link between
eye movements and cognition. The review yields that ET is a powerful tool for men-
tal health monitoring in daily life settings. Also, Flad et. al. (FLAD; BLTHOFF;
CHUANG, 2015) presented a study of EEG data combined with ET to understand visual
information processing. In that work, different methods and approaches were proposed
 11

Figure 3.2: Open BCI 32-channel (BCI, 2016).

to perform the analysis of EEG and EOG data. Finally, Flad et. al (FLAD et al., 2016)
performed a comparison between two methods of EOG-based ET and optical-based ET.
The results showed that the performance of EOG methods is better, because to the light
conditions and display dynamics are a limitation for optical ETs. Also, they established
that their studied methods could be used to study simultaneous eye movements and men-
tal activity with EEG. With this initial revision, it was demonstrated that it is feasible to
perform the ET system to develop the study proposed in the dissertation. All methods
presented here will be evaluated to establish the best technique for the application.

For virtual reality, the possible scenarios will be developed on Unity. As base to
begin the experiments we can use the work of Chun. et. al. (CHUN; BAE; JO, 2016)
where a BCI approach was used with an optical ET to performed a 3D object control
in virtual reality. Following this work, it is possible to develop the same application with
EOG-based ET. Another approach is the work of Karacan et al (KARACAN; CAGILTAY;
TEKMAN, 2010) which may be used to start the studies of cognition in virtual reality.
The authors performed a study of attention and its relationship with learning using a vir-
tual model of a natural scene. For the evaluation, it was measured the gaze time from
video analyzes which were used to evaluate the attention. The study reveals that the users
have a familiarity with the environment and the change of objects. Also it was demon-
strated how the gaze direction is an important factor for cognitive actions. However, the
approach does not work in real-time and the gaze direction could be inaccurate to measure
the attention because it does not yield information about the eye movements.
12

Finally, a recent work performed by (DAN; REINER, 2016) shows a similar study
like the presented master research. They examined the cognitive load of simple and com-
plex tasks of learning in a visual system (2D and stereo 3D displays). They calculated the
cognitive load index (CLI) which is made whit a EEG-system. The results show measure-
ments of average power on alpha and betha bands for each subject. It was demonstrated
that CLI is higher for 2D than 3D tasks. Also, the power is different for some electrodes
and bands. These results are well and recent to start with the setup up the experiments of
this dissertation.

Part of this master research will be developed in an international internship to be

held at the Laboratory of Cognition Control in Human-Machine Systems, which is part
of Max Planck Institute at University of Tbingen, in Germany (MPI, 2016), under the
supervision of Prof. Dr. Lewis Chuang. The main objective is to develop the project
called "Mobile gazetracking and EEG in head-mounted displays for user-interface proto-
typing." In this project, the main activities will be to implement and perform gazetracking
and EEG measurements in augmented and virtual reality scenarios. The laboratory has a
great background in immersive virtual environments, especially in vehicle handling sim-
ulators as flight simulator and EEG recordings. Precisely, one of its main research area is
very related with the subject of this proposal, which is to estimate the perceptual-motor
workload using EEG (MPI, 2016).

In summary, Figure 3.3 shows the path to achieve the masters thesis (including
the tasks to make) and finally Table 3.1 presents the schedule.
 13

Figure 3.3: Path to masters thesis

 15

REFERENCES

ABDULKADER, S. A.; ATIA, A.; MOSTAFA, M.-S. M. Brain computer interfacing:

Applications and challenges. Egyptian Informatics Journal, v. 16, p. 213230, 2015.

AL-RAHAYFEH, A.; FAEZIPOUR, M. Eye tracking and head movement detection:

A state-of-art survey. IEEE Journal of Translational Engineering in Health and
Medicine, v. 1, p. 21002122100212, 2013.

BAREA, R. et al. Eog-based eye movements codification for human computer

interaction. Expert Systems with Applications, v. 39, n. 3, p. 2677 2683, 2012.

BCI, O. 16-channel Biosensing Board 32bit Daisy. 2016. Avail-

able from Internet: <http://shop.openbci.com/collections/frontpage/products/
copy-of-r-d-kit-16-channel-32bit-daisy-accessories?variant=28159622851>.

BI, L.; FAN, X. A.; LIU, Y. Eeg-based brain-controlled mobile robots: A survey. IEEE
Transactions on Human-Machine Systems, v. 43, n. 2, p. 161176, 2013.

BOUKHALFI, T. et al. Tools and techniques for real-time data acquisition and analysis
in brain computer interface studies using qeeg and eye tracking in virtual reality
environment. IFAC-PapersOnLine, v. 48, n. 3, p. 46 51, 2015.

CHAE, Y.; JEONG, J.; JO, S. Toward brain-actuated humanoid robots: Asynchronous
direct control using an eeg-based bci. IEEE Transactions on Robotics, v. 28, n. 5, p.
11311144, 2012.

CHUN, J.; BAE, B.; JO, S. Bci based hybrid interface for 3d object control in virtual
reality. In: 2016 4th International Winter Conference on Brain-Computer Interface
(BCI). [S.l.: s.n.], 2016. p. 14.

CNPQ. Diretorio Grupos de Pesquisa no Brasil. 2016. Available from Internet:

<http://dgp.cnpq.br/buscagrupo/>.

COURGEON, M. et al. Joint attention simulation using eye-tracking and virtual humans.
IEEE Transactions on Affective Computing, v. 5, n. 3, p. 238250, 2014.

DAN, A.; REINER, M. Eeg-based cognitive load of processing events in 3d virtual

worlds is lower than processing events in 2d displays. International Journal
of Psychophysiology, p. , 2016. ISSN 0167-8760. Available from Internet:
<http://www.sciencedirect.com/science/article/pii/S0167876016306882>.

FLAD, N.; BLTHOFF, H.; CHUANG, L. Combined use of eye-tracking and eeg to
understand visual information processing. In: SCHULZ H.-J., B. U. U. v. L. (Ed.).
Stuttgart, Germany: Fraunhofer Verlag, 2015. p. 115124. ISBN 978-3-8396-0960-6.

FLAD, N. et al. Unsupervised clustering of eog as a viable substitute for optical

eye-tracking. In: First Workshop on Eye Tracking and Visualization (ETVIS 2015).
[S.l.]: Springer, 2016.

GERT, P. et al. Brain-computer interface systems used for virtual reality control. In:
KIM, J.-J. (Ed.). Virtual Reality. [S.l.]: InTech, 2011. chp. 7, p. 119.
16

GRAZ, U. T. BNCI Horizon 2020. 2015. Available from Internet: <http:

//bnci-horizon-2020.eu/>.

IINN-ELS. INCT Brain-Machine Interface (INCEMAQ). 2011. Available from

Internet: <http://www.natalneuro.org.br/_eng/incemaq/index.asp>.

JATUPAIBOON, N.; SETHA, P. ngum; PASIN, I. Real-time eeg-based happiness

detection system. The Scientific World Journal, v. 2013, p. 12, 2013.

KARACAN, H. U.; CAGILTAY, K.; TEKMAN, H. G. Change detection in desktop

virtual environments: An eye-tracking study. Computers in Human Behavior, v. 26,
n. 6, p. 1305 1313, 2010.

LOPES, A. C. et al. Wheelchair navigation assisted by human-machine shared-control

and a p300-based brain computer interface. In: IEEE/RSJ INTERNATIONAL
CONFERENCE ON INTELLIGENT ROBOTS AND SYSTEMS. San Francisco, CA -
USA: IEEE, 2011. p. 24382444.

LOTTE, F. A tutorial on eeg signal processing techniques for mental state recognition
in brain-computer interfaces. In: RECK, M. E.; JULIEN, C. (Ed.). Guide to
Brain-Computer Music Interfacing. [S.l.]: Springer, 2014. chp. 7, p. 132.

LOTTE, F. et al. A review of classification algorithms for eeg-based braincomputer

interfaces. Journal of Neural Engineering, v. 4, p. 24, 2007.

MAJARANT, P.; BULLING, A. Eye tracking and eye-based humancomputer

interaction. In: . Advances in Physiological Computing. London: Springer
London, 2014. p. 3965.

MENSEN, A.; RIEDNER, B.; TONONI, G. Optimizing detection and analysis of slow
waves in sleep {EEG}. Journal of Neuroscience Methods, v. 274, p. 1 12, 2016.

MPI. Max Planck Institute - Cognition Control in Human-Machine Systems.

2016. Available from Internet: <http://www.kyb.tuebingen.mpg.de/research/dep/bu/
cognition-control-in-human-machine-systems.html>.

NANDAGOPAL, N. et al. Computational techniques for characterizing cognition using

eeg data new approaches. Procedia Computer Science, v. 22, p. 699 708, 2013.

NUNEZ, M. D.; VANDEKERCKHOVE, J.; SRINIVASAN, R. How attention influences

perceptual decision making: Single-trial {EEG} correlates of drift-diffusion model
parameters. Journal of Mathematical Psychology, p. , 2016.

RAMLI, R. et al. Using finite state machine and a hybrid of {EEG} signal and {EOG}
artifacts for an asynchronous wheelchair navigation. Expert Systems with Applications,
v. 42, n. 5, p. 2451 2463, 2015.

SCHEER, M.; BLTHOFF, H. H.; CHUANG, L. L. Steering demands diminish the

early-p3, late-p3 and ron components of the event-related potential of task-irrelevant
environmental sounds. Frontiers in Human Neuroscience, v. 10, p. 73, 2016. Available
from Internet: <http://journal.frontiersin.org/article/10.3389/fnhum.2016.00073>.
 17

SFB-TRR. Quantitative Methods for Visual Computing Project C03 | Immersive

Virtual Environments. 2016. Available from Internet: <http://www.trr161.de/interfak/
forschergruppen/sfbtrr161/research/project_c03/index.html>.

VAID, S.; PREETI, S.; KAUR, C. Eeg signal analysis for bci interface: A review.
In: 5TH INTERNATIONAL CONFERENCE ON ADVANCED COMPUTING
COMMUNICATION TECHNOLOGIES (5ACCT). Rohtak - India: CPS, 2015. p.
143147.

VIDAL, M. et al. Wearable eye tracking for mental health monitoring. Computer
Communications, v. 35, n. 11, p. 1306 1311, 2012.

VIDAURRE, C. et al. Eeg-based bci for the linear control of an upper-limb

neuroprosthesis. Medical Engineering & Physics, v. 38, n. 11, p. 11951204, 2016.

VOURVOPOULOS, A.; FERREIRA, A.; BADIA, S. B. i. Neurow: An immersive

vr environment for motor-imagery training with the use of brain-computer interfaces
and vibrotactile feedback. In: Proceedings of the 3rd International Conference on
Physiological Computing Systems. [S.l.: s.n.], 2016. p. 4353.

YOUNG, M. S. et al. State of science: mental workload in ergonomics. Ergonomics,

v. 58, n. 1, p. 117, 2015.