CHI 2006 · Work-in-Progress April 22-27, 2006 · Montréal, Québec, Canada

Do We Need Eye Trackers to Tell

Where People Look?

Sune Alstrup Johansen Abstract

IT University of Copenhagen We investigated the validity of two low-cost alternatives
Rued Langgaards Vej 7, to state-of-the-art eye tracking technology: 1)
DK-2300 Copenhagen S, Denmark prompting users to report from memory on their own
sune@itu.dk eye movements, and 2) asking experienced web
designers to predict the eye movements of a typical
John Paulin Hansen user. Users could reliably remember 70 % of the web
IT University of Copenhagen elements they had actually seen. Web designers could
Rued Langgaards Vej 7, only predict 46 % of the elements typically seen. Users
DK-2300 Copenhagen S, Denmark were not particularly good at remembering the order of
paulin@itu.dk their fixations. We discuss how to further improve the
validity of self-reported gaze patterns and suggest new
areas that it may be used in.

Keywords
Eye Tracking, Usability Evaluation, Visual Attention,
Web Design, Cognition, User Centered Design.

ACM Classification Keywords

H.5.2 User Interfaces: Evaluation/methodology, H.1.2
User/Machine Systems: Human information processing
Copyright is held by the author/owner(s).
CHI 2006, April 22–27, 2006, Montréal, Québec, Canada.
ACM 1-59593-298-4/06/0004.
Introduction
Visual perception is an essential part of users’
interaction with interfaces. Modern eye tracking

923
 CHI 2006 · Work-in-Progress
Figure 1: A Tobii eye tracker
recording visual attention.
Afterwards, the user reproduced
gaze patterns from memory.
equipment makes it possible to record and analyze
parts of this process. Which elements are actually
seen? Where do users look first? What do users look at
the most? Did modifications of the graphic design lead
to a wanted change in user gaze patterns?
Eye tracking has been criticized for being costly and
tedious [1,9]. Older generations of eye tracking
equipment did not deliver sufficient value for usability
professionals. Difficulties calibrating the equipment to
users with glasses, contact lenses, heavy make-up, or
even dark/brown eyes were common. Precision was
low, and tiny head movements could jeopardize the
validity of the recorded eye tracking data.
State-of-the-art eye tracking equipment has solved
most of these problems, and accurate recordings of eye
movements can be made without obtrusive head-
mountings or unnatural fixations of the head. However,
some barriers for deploying eye tracking in usability
studies still remains: Data analysis can be very time
consuming, even with the use of advanced software
tools that come with modern eye trackers.
Furthermore, state-of-the-art eye trackers are
relatively expensive, (especially when purchasing both
the hardware and analysis software) with market prices
above 20,000 $ (as of January 2006). Last but not
least, eye tracking equipment may reduce the test
validity, for instance by notably slowing the system
response or by requiring users to re-calibrate between
tasks.
These barriers may well disappear with improvements
of technology. Until this happens, there will be a need
for alternatives to eye tracking by costly equipment.
Even when tracking technology becomes mature, there
April 22-27, 2006 · Montréal, Québec, Canada
could still be room in the usability toolbox for a simple
and fast method that everybody could use. In a similar
way, paper and pen are often the preferred tools by
usability professionals, although they have lots of
advanced systems available for data collection and
notation. For instance, stick notes and paper cards have
survived over sophisticated digital tools for e.g. “card-
sort” analysis of information architectures.
People’s patterns of fixation are known to be highly
predictive of what they can remember afterwards [3],
and people tend to have a consistent viewing pattern
when they re-visit something that they have seen
previously, cf. the scan path theory [10]. It is an open
issue to what degree people can reliably remember
where they have looked. Obviously, they can tell from
moment to moment what they look at, and users can
explain their own eye movements in details during
retrospections of gaze recordings [4], but can they
keep the attended elements in their memory – even for
a short while - before reporting on them?
Procedure
In the first experiment, state-of-the-art eye tracking
equipment (Tobii 1750 with Clearview 2.5.1 analysis
software, cf. Figure 1) was used to track the actual eye
movements of 10 users, while they searched for an
answer to simple questions on 8 different web pages.
For instance, they were asked: “Where would you click
on this web page to find the shop nearest to your
residence?”
The user first pressed a button to get the question, and
then again to view the webpage and search for an
answer. Immediately after answering, the users would
report their remembered eye movements by repeating
924
 CHI 2006 · Work-in-Progress April 22-27, 2006 · Montréal, Québec, Canada

them as accurately as possible by looking at the screen
with the original webpage shown again. This made it
possible to track remembered eye movements, and
later to analyze how well they compared to the original
ones (cf. Figure 2).
To analyze the data, each webpage was divided into a
number of Areas of Interest (AOI´s) on basis of the
gestalt laws (e.g. “law of good continuation”, “law of
proximity”, “law of similarity”, “low of closure”, and
“law of symmetry”). Each AOI was furthermore given a
unique identifier and classified from a list of common
web page elements, including banners, contact
information, drawings, email address, input fields,
logos, mixed content, navigation elements, pictures,
search fields, text blocks, and URL´s.

Green boxes indicate the defined

Areas of Interest (AOI’s).
Black text indicate the unique
identifier given to each AOI for
analysis purposes, and a text
code according to the nature
of the element.

Red circles indicate fixations. The

sizes of the circles illustrate time
spent looking at a particular point,
e.g. bigger circles means longer
time spent. Numbers in circles
indicate the order of fixations.

Figure 2: Plot of a user repeating from memory his gaze pattern associated with the question
“Where would you click to find out if this hotel has a pool?”

925
 CHI 2006 · Work-in-Progress
In the second experiment, 17 web designers (all of
them with more than 18 months of experience in web
design) were asked to predict the eye movements by
marking a “typical user scan path” on print-outs of the
8 web pages that had been used in the first
Seen Remembered experiment. The designers’ predictions were compared
with typical user scan paths constructed by n-gram
analysis of the total eye tracking data from the 10
0 5 10 15 20
subjects. N-gram’s are sequences of characters from
the ASCII character set, where a bi-gram (N=2)
7
consists of 2 characters, a tri-gram (N=3) consists of 3
14 characters, etc. Based on the unique identifiers of
AOI’s, each user scan path could be expressed as a
17 sequence of ASCII characters. “Typical user scan paths”
of each web page was constructed from the bi-grams
20 with the highest appearance, matched together in a
continuing order to create a complete sequence. The
22 first and last AOI that had been seen by most people
would define the beginning and the end of the
24 sequence.
28 Results
The users could remember 70 % (SD 17.7%) of the
28 AOI´s they had fixated on for more than 125 ms. The
average number of AOI´s users remembered they had
Figure 3: For each of the 8 web seen did vary across different web pages (from 3.2 to
pages, white bars illustrate the 11.4), as some of the web pages had more elements
number of AOI’s seen, and black than others (cf. Figure 3). Also some of the questions
bars show AOI’s remembered.
were harder to answer than others, thereby initiating a
Numbers at the left side of the
bars indicate the total number of longer visual search of the web page. The amount of
AOI’s that each web page was AOI´s that the user could remember having seen
divided into. depended on how many AOI’s they had actually seen,
but there were no differences in the relative memory
between simple and complex web pages (i.e. web
pages with many elements).
April 22-27, 2006 · Montréal, Québec, Canada
On average, users had 1.9 false memories per web
page (SD=1.4), with a non-significant tendency that
web pages with many AOI´s would introduce more
false memories than simple ones. For instance, on the
simplest web page with only 7 pre-defined AOI´s, users
had only 0.5 false memories, while on the most
complex one with 28 pre-defined AOI´s, they had an
average of almost 3 false memories.
The users were not particularly good at remembering
the actual order by which they had looked at AOI´s. We
measured this by calculating the Levensthein Distance
(LD) between the sequence of AOI´s seen [5,8] and
the sequence remembered. On average, the LD was 16
(SD = 12.8) and this was highly dependent on the
length of the sequence to remember (R2 = 0.91, p
<0.05). A LD of 0 would indicate perfect memory, while
each basic operation (insertion, deletion, or
substitution), needed to get from the correct string of
elements to the one actually remembered, would
increase the LD by one.
We also analyzed if there were any differences in users’
memory of their eye movements for different types of
web page elements seen. It turned out that people
could only remember having seen a logo in 34 % of the
cases they had actually seen it, while they could
remember having seen 77 % of the photos they had
looked at, 74 % of the navigation elements, and 75 %
of the text elements. This difference in memory
percentages between logos and the other elements
were significant (photos: (t(9) = -3.26, p < 0.01),
(navigation elements: (t(9) = -4.74, p < 0.01), (text
elements: (t(9) = -4.97, p < 0.001).
926
CHI 2006 · Work-in-Progress
In the second experiment, the group of 17 web
designers could predict 46 % (SD = 22.1 %) of a
typical user scan path, when given the question that
the user had to answer. This is significantly worse than
the 70 % that users could remember themselves (t(25)
= -3.04, p < 0.01). Still using the same questions for
each task, we also asked the web designers to try to
predict the first 3 AOI´s that a user would be most
likely look at, using the “squint test” recommended by
[7] for predictions of scan paths. With this method,
designers could only predict 36 % (SD = 27 %) of the
first 3 AOI´s in the typical user scan paths.
Discussion
Even a “quick-and-dirty” method should produce
reliable indications, though not necessarily 100 %
complete data sets. It should definitely not lead to
wrong conclusions based on skewed data, and the
possible limitations of the method should be clear to all
using it. For instance, our experiment indicates that
there are severe limitations to people’s memory of
having seen a logo. So we would not recommend
testing logo design and/or placement by the self-
reporting method.
Earlier research by [2] totally rejected the
trustworthiness of users’ self-reported eye movements.
On the basis of the present findings, we find enough
reasons to keep improving the self-reporting methods
and to identify the best conditions for using them.
Reporting of eye movements should be done for one
web page at a time, and as soon as the user had found
the answer, thus using the principles of recency [6]
while retaining the context with the original stimuli still
shown in front of the user. Also, we recommend that
instructions to the users should be highly task-oriented,
April 22-27, 2006 · Montréal, Québec, Canada
which has proven to increase their incentives for the
presented stimuli [11], and we recommend short and
simple search tasks, since memory of long sequences
tend to be more erroneous.
The results suggest that the predictions of web
designers are generally less reliable than asking the
users themselves. When web designers are to make
important decisions on e.g. how to organize a layout,
they should not just trust their “professional intuition”
but ask users how they actually looked at the interface.
We believe that the self-reporting method is efficient
enough to be part of an iterative design routine, and
the “squint test” does not seem to be an alternative in
tests where users are to search for particular
information.
In future research, we intend to analyze if users can
remember their scan paths for a longer time than just
the few seconds we required them to, and if they can
hold the memory of them when conducting several
tasks or visiting several web pages before they deliver
the report. As an alternative, the mouse can be used to
indicate continuously where one is looking – indeed a
cheap and ubiquitous attention tracker. What are the
differences then between a state-of-the-art eye
tracking and real-time reports given by mouse?
We would also like to investigate if it is possible to
increase the validity of self-reported gaze patterns by
reducing the complexity of the stimuli. In practice, this
could be done by de-selecting “display of graphic
elements” in the browser settings. Also, for tests
conducted with simple prototypes, it will be interesting
to know how well they will predict gaze patterns on the
final interface.
927
CHI 2006 · Work-in-Progress
The self-reporting method could prove especially
relevant in large-scale user studies conducted over the
Internet, since it would then be practically possible to
make users report on their scan paths remotely. Can
we give precise instructions – in written or multimedia
formats – that would teach users to report their gaze
patterns from a distance?
Finally, we would like to investigate, if the immediate
memory of eye movements on a web page is related to
other dimensions of usability. For instance, cluttered
web pages are probably more difficult to report eye
movements on than well-designed web pages. The
immediate memory of own eye movements could also
turn out to be a good predictor of the long-term
memory of the web page, and thus to the feeling of
familiarity with it.
Conclusion
The validity of users’ memory of their scan paths on
web pages is limited when compared to actual
recordings, especially for logo elements. The validity is
high enough to justify further research, and several
improvements are possible. This may lead to a range of
new usability tools that everybody, including
unsupervised users, can apply. Even now, the validity is
higher than relying on predictions from web designers.
Acknowledgements
We thank the Royal School of Library and Information
Science in Copenhagen for supporting this research.
References
[1] Aaltonen, Annti ‘Eye Tracking in Usability Testing:
Is It Worthwhile?’, ‘Usability & eye tracking’ workshop
at CHI'99, ACM Press (1999).
April 22-27, 2006 · Montréal, Québec, Canada
[2] Babcock, J.S., Pelz, J.B., & Fairchild, M.D. ‘Eye
tracking observers during rank order, paired
comparison, and graphical rating tasks’, Proc. PICS
Digital Photography Conference 2003 (2003).
[3] Chapman, Peter ‘Remembering what we´ve seen:
Predicting recollective experiences from eye
movements when viewing everyday scenes’ Cognitive
Processes in Eye Guidance, G. Underwood (Ed.), Oxford
University Press, UK, 2005, pp. 237 – 258.
[4] Hansen, J. Paulin ‘The use of eye mark recordings
to support verbal retrospection in software testing’,
Acta Pcychologica 76 (1991), pp. 31 – 49.
[5] Josephson, S. and Holmes, M. E. ‘Visual attention
to repeated internet images: testing the scanpath
theory on the world wide web’, Proc. ETRA '02, ACM
Press (2002), pp. 43 - 49.
[6] Matlin, Margaret W. Cognition, Wadsworth
Thomson Learning, NY, USA, 2002.
[7] Mullet, Kevin & Sano, Darrell Designing Visual
Interfaces – Communication Oriented Techniques, Sun
Soft Press, Mountain View, California, USA, 1995.
[8] Pan, B., Hembrooke, H. A., Gay, G. K., Granka, L.
A., Feusner, M. K., and Newman, J. K. ‘The
determinants of web page viewing behavior: an eye-
tracking study’, Proc. ETRA ’04, ACM Press (2004).
[9] Schnipke, S. K. and Todd, M. W. ‘Trials and
Tribulations of Using an Eye-tracking System’, Proc.
CHI 2000, ACM Press (2000), pp. 273-274.
[10] Stark, L.W. & Ellis, S.R. ‘Scanpaths revisited:
cognitive models direct active looking’, Eye
movements: cognition and visual perception, D. F.
Fisher et al. (Eds.), Hillsdale, NJ: Lawrence Erlbaum
Associates (1981), pp. 193-226.
[11] Van Waes, Luuk ‘Thinking Aloud as a Method for
Testing the Usability of Websites: The Influence of Task
Variation on the Evaluation of Hypertext’, IEEE
Transactions on Professional Communication, vol. 43,
no. 3, 2002.
928