chapter 8

evaluation techniques
Evaluation Techniques

• Evaluation
– tests usability and functionality of system
– occurs in laboratory, field and/or in collaboration
with users
– evaluates both design and implementation
– should be considered at all stages in the design life
cycle
Goals of Evaluation

• assess extent of system functionality

• assess effect of interface on user

• identify specific problems

 Evaluating Designs

Cognitive Walkthrough
Heuristic Evaluation
Review-based evaluation
Cognitive

• Cognitive psychology is the study

of mental processes such as
"attention, language use, memory,
perception, problem solving,
creativity, and thinking".
Cognitive Walkthrough

• The cognitive walkthrough is a usability

evaluation method in which one or
more evaluators work through a series
of tasks and ask a set of questions from
the perspective of the user.
• The focus of the cognitive walkthrough
is on understanding the
system's learnability for new or
infrequent users.
Cognitive Walkthrough
Procedure
• Define the inputs to the walkthrough
• Convene / organize the analysis
• Walk through the action sequences for
each task.
• Record critical information
• Revise the interface to fix the problems
Cognitive Walkthrough

Proposed by Polson et al.

– evaluates design on how well it supports user
in learning task
– usually performed by expert in cognitive
psychology
– expert ‘walks though’ design to identify
potential problems using psychological
principles
– forms used to guide analysis
Cognitive Walkthrough (ctd)

• For each task walkthrough considers

– what impact will interaction have on user?
– what cognitive processes are required?
– what learning problems may occur?

• Analysis focuses on goals and

knowledge: does the design lead the
user to generate the correct goals?
Heuristic

• A heuristic technique mean

"find" or "discover"
• Heuristics can be mental
shortcuts that ease the
cognitive load of making a
decision.
What is Heuristic Evaluation?

• Heuristic evaluation is a usability

engineering method for finding usability
problems in a user interface design.
• Heuristic Evaluation (originally proposed
by Nielsen and Molich, 1990) is a
discount method for quick, cheap, and
easy evaluation of the user interface.
What is Heuristic Evaluation?

• The process requires that a small set of

testers (or “evaluators”) examine.
• The interface, and judge its
compliance with recognized
usability principles (the “heuristics”).
• The goal is the identification of any
usability issues so that they can be
addressed as part of an iterative design
process.
What is Heuristic Evaluation?

• Heuristic Evaluation is requires few

resources in terms of money, time or
expertise.
• So any can enjoy the benefits of
usability testing – not just those with
thousands to spend on a professional
assessment.
Good Reasons to Use
Heuristic Evaluation
• It’s fast, quick and cheap to conduct a
heuristic evaluation.
• This is especially true if you are only
going to use a single evaluator.
• It provides good insight into
possible usability problems that might
damage the user experience.
Heuristic Evaluation

• A heuristic is a simple principle or “rule

of thumb”.
• A heuristic evaluation is one where
an evaluator or group of evaluators
walks through the design of an interface
and decides whether or not it complies
with these “rules of thumb”.
How to Conduct a Heuristic
Evaluation for Usability in HCI
Visibility of the system’s
status
Users should be given feedback
on what is happening within a
reasonable timescale.
Match between the system
and real world
Information should be presented in a
means familiar to the user including
language and conventions rather than
terms developed for the system.
Information should be presented in a
logical order.
User control and freedom

Users make mistakes. There should be an

“emergency exit” which is easy to find
and exit the current system state without
having to jump through hoops. Undo
and redo functions are essential.
Consistency and standards

Words, actions, situations, etc. should

always mean the same thing and users
should be able to understand that.
Error prevention

Preventing error is better than clear error

messages.
Eliminate error conditions or make users
aware that they may be about to occur
and ask them if they want to proceed.
Recognition rather than recall

Reduce the load on a user’s memory.

Make actions, options, and objects visible.
Users shouldn’t have to remember things from
one screen to the next.
Instructions should be easy to access when
needed.
Flexibility and efficiency of
use
The use of accelerators, where
appropriate, may be invisible to new
users but improve the efficiency of use
for experienced users. Actions could be
customized by users.
Aesthetics and minimalist
design
Dialogue should not give irrelevant or
rarely useful information.
The more data in dialogue the more it
diminishes the overall visibility of
individual points for the user.
Help users recognize, diagnose
and recover from error
This means error messages should be in
clear language and avoid the use of
codes. They should explain the problem
and offer a solution to that problem.
Help and documentation

The best approach is to construct a system

which needs no help or documentation but if it is
necessary – it should be easy to search, be
based on the tasks the user wishes to execute,
offer concrete steps to follow and be kept to a
minimum.
Why Multiple Evaluators?

• Every evaluator
doesn’t find every
problem
• Good evaluators
find both easy &
hard ones

34
Interface Hall of Shame or Fame?
• Standard MS calculator
– on all
Win95/98/NT/2000/XP

35
Interface Hall of Shame!
• What is the empty
button above MC for?
• Can’t resize
• Blue for numbers!
– goes against all we know
• hard to focus on
• combined with red
eye strain

36
Heuristics (revised set)

searching database for matches

• H2-1: Visibility of system status

– keep users informed about what is going on
– example: pay attention to response time
• 0.1 sec: no special indicators needed, why?
• 1.0 sec: user tends to lose track of data
• 10 sec: max. duration if user to stay focused
on action
• for longer delays, use percent-done progress
37
bars
Heuristics (cont.)

• Bad example: Mac desktop

– Dragging disk to trash
• should delete it, not eject it
• H2-2: Match between system &
real world
– speak the users’ language
– follow real world conventions

38
 Heuristics (cont.)
• Wizards
• H2-3: User control &
freedom
– “exits” for mistaken choices,
undo, redo
– don’t force down fixed paths
• like that BART machine…
– must respond to Q
before going to next
– for infrequent tasks
• (e.g., modem
config.)
– not for common tasks
– good for beginners
• have 2 versions
(WinZip)
39
Heuristics (cont.)

• H2-4: Consistency & standards

40
 Heuristics (cont.)

• MS Web Pub. Wiz. • H2-5: Error prevention

• Before dialing • H2-6: Recognition rather
– asks for id & password
than recall
– make objects, actions,
• When connecting options, & directions visible
– asks again for id & pw or easily retrievable
41
Heuristics (cont.)

Edit
Cut
Copy
Paste

• H2-7: Flexibility and efficiency of use

– accelerators for experts (e.g., gestures, kb
shortcuts)
– allow users to tailor frequent actions (e.g., macros)

42
Heuristics (cont.)

• H2-8: Aesthetic and minimalist design

– no irrelevant information in dialogues

43
Heuristics (cont.)

• H2-9: Help users recognize, diagnose,

and recover from errors
– error messages in plain language
– precisely indicate the problem
– constructively suggest a solution
44
Heuristics (cont.)

• H2-10: Help and

documentation
– easy to search
– focused on the user’s
task
– list concrete steps to
carry out
– not too large

45
Real system
world
Query Techniques

Interviews
Questionnaires
Interviews

• analyst questions user on one-to -one basis

usually based on prepared questions
• informal, subjective and relatively cheap

• Advantages
– can be varied to suit context
– issues can be explored more fully
– can elicit user views and identify unanticipated
problems
• Disadvantages
– very subjective
– time consuming
Questionnaires

• Set of fixed questions given to users

• Advantages
– quick and reaches large user group
– can be analyzed more rigorously
• Disadvantages
– less flexible
– less probing
Questionnaires (ctd)

• Need careful design

– what information is required?
– how are answers to be analyzed?

• Styles of question
– general
– open-ended
– scalar
– multi-choice
– ranked