USABILITY EVALUATION

REPORT



Mobile Trip Assistant Project for













Page 2

USABILITY EVALUATION REPORT

Table of Contents

1. Usability Analysis
1.1 Introduction3


2. Usability Strategy..3
2.1 Paper Prototype...5
2.2 1
st
Phase Model...5
2.3 Final Recommendation...5

3. Users
3.1 User Profiles...6
3.2 Process Description8
3.3 Schedule of Tests..11

4. Feedback & Critical Incidence
4.1 Introduction...12
4.2 Pre and Post Test Questions..12
4.3 Analysis and Incorporated Changes..13
5. Screen Shot.21

6. References...28









Page 3

USABILITY EVALUATION REPORT
1. USABILITY ANALYSIS

1.1 INTRODUCTION
Mobile devices have and are becoming extensively important in our daily lives.
Explosion in IT and the development of newer software tools has set in a revolution in
Mobile Technology. Trained professionals are required who can ideate, innovate and put
New Mobile Systems in place and manage them effectively for the benefit of the
stakeholders in particular and the society in general. It is important for such professional
to evaluate the usability of the system they are implementing.


2. STRATEGY

Our usability report will be focusing on the following development methods:
a. Generic Description: Presenting a general idea of our usability approach.
b. Purpose: Will highlight the rationale for our analysis in designing the mobile software
for PAAC.

This analysis involves a deep and extensive study of users performing real tasks in their
environment. The methods that are focused are:

a. User Context Logging:
Selected users providing the feedback on the fly. The factors taken into account are
activity, time, location and social situation. We planned our user tests in an environment
which would be used by the user himself when interacting with the application.

b. Patterns of Activity:
Monitoring the technology interface and the user feedback/satisfaction through
interviews and questionnaires. This step enables us to evaluate how the mobile
technology is integrated in the users activities and how effectively the user accepts such
integration. The data collected in this stage will be used by the developing team to
incorporate the required changes in to the software. This data would also provide an
insight for carrying out the usability analysis for other users.




Page 4

USABILITY EVALUATION REPORT

Mobile Tapping:
For any mobile application, the first and foremost necessity is to enable an effective user
interaction. This pattern of study enables us to perform and analyze how the user will
interact with our mobile application while moving or sitting. Since the Mobile Trip
Assistant will be used by individual on the move as well as when stationary, we have to
ensure that the application does not affect the daily activities of the individual. This study
will however be performed through user context logging. We also test our application on
individual who are left handed and right handed.

We performed our usability study in 3 steps:

ITERATIVE DESIGN PROCESS [2]

PAPER PROTOTYPE



1
st
PHASE DESIGN



FINAL PHASE DESIGN





Design
Goals
Design
Guidelines
Prototype
Interface
Formative
Evaluation
Operational
Interface
Summative
Evaluation

Page 5

USABILITY EVALUATION REPORT
2.1 Paper Prototype:
A user centered design process which involved creation of a throwaway rough sketch of our
mobile application interface used for development. This provided a useful feedback for the entire
creative team as everyone brainstormed on the design together. The paper prototypes used in our
user test are displayed here.


2.2. 1
st
Phase Model
This phase led to the development of our initial mobile application. The feedback we got from
the 1
st
user tests were incorporated in this phase.


2.3 Final Model
This was our last phase of development. The feedback received from 2
nd
round of user tests were
integrated in our final version of mobile application.
















Page 6

USABILITY EVALUATION REPORT
3. USERS

3.1 User Profiles:

# Name Gender/Age Category Profession
1. USER A M/22 Expert Mobile
User
Student at Carnegie Mellon University
2. USER B M/32 Mid-Level
User
Working Lady
3. USER C F/48 Basic User House wife

Expert Mobile User: This type of user is termed Expert on the basis of his/her usage of
mobile applications in daily life. This user interacts heavily, and much of his daily activities
involve using of mobile applications. Being a student, the expert is a frequent user of the bus
service. He commutes to all the places within Pittsburgh using the bus transport. He relies on
the bus schedules to a major extent, and plans his trips by using the transport authority bus
schedules via the mobile app or websites.

Mid Level Mobile User: This user has moderate or average usage of mobile apps in the
daily life. This user interacts moderately, and much of his daily activities dont involve using
of mobile applications. Being a working man, the user is a moderate user of the bus service.
He commutes to certain places within Pittsburgh using own transport. He relies on the bus
schedules to some extent, and plans his trips by using the transport authority bus schedules
via the mobile apps or websites, but the frequency of using them is moderate.

Basic User: This user has very less or no usage of mobile apps. This user hardly uses the
mobile apps in his daily activities. Being a house wife, the user is not a major or moderate
user of the bus service. She commutes to most places within Pittsburgh using own transport.
She relies on the bus schedules to some extent, and plans her trips by using the transport
authority bus schedules via the mobile apps or websites, but the frequency of using them is
very low.






Page 7

USABILITY EVALUATION REPORT
The user age group was selected on the basis of the following demographic:


We see that the majority of our users are spread over android, iPhones and other Smartphones.
Hence we chose our users on this basis.









Page 8

USABILITY EVALUATION REPORT
3.2 Process Description

User evaluation was conducted in a simple classroom with the phone and the web based
application given to the user. One of the team members took the responsibility of conducting the
interview and another member took the notes. We decide to have a 15min session with each user.
We divided our tests with 3 users in each of our stages of development.

Our usability testing was divided into the following tasks:
Test # Test Case Test Case Description Paper Prototype
1. Locate Routes a.1Current Location as keyword (already
entered in the Source field)
a.2 Enter the destination
a.3 Enter date (default will be already entered-
System Date)
a.4 Enter time (default will be already entered-
System Time)



2. Trip Planner a.1 Enter some location other than the current
location as the keyword
a.2 Enter the destination
a.3 Enter date (will be already entered- System
Date)
a.4 Enter time (will be already entered- System
Time)


3. Favorite Feature We tell the user to select the current route and add
it to the Favorite Feature in our application. This
will allow the user to add/save his favorite, and
most used route.


Page 9

USABILITY EVALUATION REPORT
4. Search Routes
by Bus Number

We tell the user to search a particular route by
entering the bus number the user wishes to search
for. We provide the user with a list of buses to
select from.

5. Nearby
Application
Feature
Here we tell the user to interact with out feature
Nearby. This feature allows the user to know
about the nearest hotels and restaurants around the
users current position.

6. News Feed We tell the user to search for latest news and
updates from the PAAC Website news feed we
have in our News Feed application.


Page 10

USABILITY EVALUATION REPORT
7. Crowd Shout Its a social media interaction feature which allows
users to post comment and photos about their
experience with a particular bus. This feature will
be addressed as a recommendation in our design.

8. Weather We tell the user to check for weather updates using
the weather application

9. Explore In this feature we tell the user to explore various
restaurants, hotels and coffee shops, etc around his
current location. This feature is different from our
nearby feature.







Page 11

USABILITY EVALUATION REPORT





SCHEDULE OF TESTS:

Following information provides the date on the various user tests we performed.

TEST #1: Paper Prototype Test (With 2 Users)
Date of Test Name of User
24
th
April- Sunday USER A
24
th
April- Sunday USER B

TEST #2: 1
st
Phase Development Test (With 3 users)
Date of Test Name of User
26
th
April- Tuesday USER A
26
th
April- Tuesday USER B
26
th
April- Tuesday USER C

TEST #3: 2
nd
Phase (Final) Development Test (With 3 users)
Date of Test Name of User
29
th
April- Friday USER A
29
th
April- Friday USER B
29
th
April- Friday USER C






Page 12

USABILITY EVALUATION REPORT





4. FEEDBACK &CRITICAL INCIDENCE

4.1 Introduction
We performed a series of tests to check the functionality and features of our mobile application.
Each of the users was required to answer a set of different questions (tasks) briefly explained below
in table 2.

4.2 Pre and Post Usability Question Feedback.

a. Pre- Usability
Question User A User B User C
1. What phone do you use? iPhone Android Motorola Flip Flop
2. How many times a day do you use
the apps on your phone (1-10)?
8 5 2
3. What is your data plan?
4. Do you travel by bus? If yes, then
how often (on a scale of 1-5)
5 3 2
5. Do you use Google Maps to get the
information of the bus schedules?
5 4 2


b. Post- Usability
Question User A User B User C
1. How much score would you give to
our application (1-5)?
4.5 4 4.5
2. How much score would you give to
our applications design (1-5)?
4 4.5 4
3. How much score would you give to
our applications functionality (1-5)?
5 4.5 5
4. Is our application difficult to use/ is 4.5 4.5 3.5

Page 13

USABILITY EVALUATION REPORT
our application complex?
5. Do you like our application? 5 5 5
6. Would you recommend the
application to your friends?
5 4 5





4.2 Analysis and Incorporated Changes:
The following tables provide a detailed feedback of our user tests and our the changes we
incorporated by analyzing the user tests.

Feedback Table (Table 1):
a. Paper Prototype Feedback:
# Screen Users Feedback Reason for Feedback Scope Severity
1. Tripplan1-
Basic
The radio buttons were
uncomfortable to be used on
touch screen. The use also
questioned as to how they will
enter the date and time.
The users were not able select the
radio buttons effectively.
Tripplan1
screen
High
2. Tripplan2-
enter dest
The user was not sure of his
current position, and also the
map we provided was not a
real map.
The users wanted that their
current position would be entered
by default and the user only had
to enter the destination.
Tripplan2
screen
High
3. Weather The weather option should
also provide the user to select
the time of the day he wants
to known the weather. The
use also questioned as to how
they will enter the date and
time.
The users wanted not only the
date but also the option to select
the time in order to know the
temperature on that day
Weather
Screen
Medium
The C/F option is good The users liked the C and F
option for displaying the
temperature
Weather
Screen
Medium
4. Favorite There should be option to
search for buses which are not
mentioned.
The users could not find the bus
he wanted to add as favorite. So
he wanted to a search option that
will help the user add a bus which
is not mentioned in the given list.
Favorite
Screen
High
5. Nearby There should be option to The users wanted to find more Nearby Medium

Page 14

USABILITY EVALUATION REPORT
search for more places
(specific) which are not
mentioned.
options like libraries, etc. So he
wanted to a search option that
will help the user with the search.
Screen
6. All Screen Logo of the application must
be on each and every screen.
The users pointed this out
because while he was using the
app, he forgot the name of the
app.
All screen High
7. Explore There should be option to
search for more places
(specific) which are not
mentioned.
The users wanted to find more
options like libraries, etc. So he
wanted to a search option that
will help the user with his search
Explore
Screen
Medium
The names of the places
should also have a comment
mentioned by one of the
customers
The users was comparing the
Explore Feature of our app with
that of FourSquare. The user
wanted a comment from a
customer either telling about the
place, or some specialty of that
place
Explore
Screen
Medium
8. All Screen The user was not able to go
back from a particular page
without selecting the tabs we
had provided below for
navigation.
The users wanted a Back button
to be on all the sub-pages
All Screens High



b. 1
st
Phase Development Feedback:
# Screen Users Feedback Reason for Feedback Scope Severity
1. All Screens All the features must be
enlisted on each page of the
application.
The users were told about the
Favorite, weather, and explore,
but was not able to find the
tabs/link on the main screens.
The users wanted all the tabs to
be present on all the screens, so
that the navigation to other
features becomes easy.
All
concerned
screens
High
The heading Port Authority
of Allegheny County should
be larger and clearer.
The users were not able to clearly
read the heading of our
application.
All
concerned
screens
High
Icon image on Map and
Feeds tab were not correct.
The users were confused with
image on the tab map and feeds.
They wanted a map image on the
Map tab and news image on
All
concerned
screens
High

Page 15

USABILITY EVALUATION REPORT
the Feeds tab.
The options button on the
top right corner was not
working correctly.
The options button was not
displaying the required content.
All
concerned
screens
High
2. Tripplan1-
Basic
The font of the header Find
Route By was small.
The users were not able to read
the font of the heading on the
main screen easily. Moreover, the
other headings had font greater
than the heading.
Tripplan1 High

3. Tripplan1
Select Date
and Time
The font of the header
Choose a Route was small.
The users were not able to read
the font of the heading on the
main screen easily. Moreover, the
other headings had font greater
than the heading.
Tripplan1
Select Date
and Time
High
Icon image on Select a date
and time was not correct.
The users were confused with
image on the Select a date and
time tab. He wanted a time
image on the tab.
Tripplan1
Select Date
and Time
Medium

4. Tripplan1-
date time
entered
The date and time format was
not readable.
The users were not able to clearly
understand the date and time
format as the they were not
clearly spaced (no commas)
Tripplan1
Select Date
and Time
High
Icon image on Select a date
and time was not correct.
The user was confused with
image on the Select a date and
time tab. He wanted a time
image on the tab.
Tripplan1-
date time
entered
Medium
The font of the header
Choose a Route was small.
The users were not able to read
the font of the heading on the
main screen easily. Moreover, the
other headings were clear (had
font greater than the heading).
Tripplan1-
date time
entered
High

5. Tripplan1-
enter curr
and dest
The date and time format was
not readable.
The users were not able to clearly
understand the date and time
format as the they were not
clearly spaced (no commas)
Tripplan1-
enter curr
and dest
High
Current and Destination
entering feature is good.
The users liked our interface of
entering the current and
destination location.
Tripplan1-
enter curr
and dest
High

6. Tripplan1-
enter date
and time
Date and time entering
method is good
The users liked our interface of
selecting date and time
Tripplan1-
enter date
and time
High

Page 16

USABILITY EVALUATION REPORT
7. Tripplan1-
Results
Page
The page is dull. The users were not impressed
with the look and feel of the page
Tripplan1-
Results
Page
High

8. Tripplan2-
enter dest
The users wanted to know
their current location.
The map that only wanted the
users to enter their destinations
did not show where the user was
currently.
Tripplan2-
enter dest
High
The destination feature is
good.
The users liked the idea of
displaying the current location
and only allowing the users to
enter their destination details
only.
Tripplan2-
enter dest
High

9. Favorite The font of the header
Choose a Bus Number was
small.
The users were not able to read
the font of the heading (Choose
a Bus Number) on the main
screen easily.
Favorite High
Filter items is a confusing
phrase.
The users were confused on the
phrase Filter Items. The users
though new that they had to write
on the tab, but the phrase were
unclear to them.
Favorite High

10. Favorite-
with filter
The font of the header
Choose a Bus Number was
small.
The users were not able to read
the font of the heading (Choose
a Bus Number) on the main
screen easily.
Favorite High
11. New Feed Just the twitter feeds are not
helpful.
The user wanted more
options/features for the news
feed.
News-Feed High









Page 17

USABILITY EVALUATION REPORT





c. Final Phase Development Feedback:
# Screen Users Feedback Reason for Feedback Scope Severity
1. Tripplan1-
Basic
Users liked the interface for
the main screen.
The users were easily able to
understand what the functionality
of the main page is. The users
also felt that the phrase used:
how can we help, is a good way
to interact.
Tripplan1-
Basic
High
Good way to represent
favorite routes
The users liked the representation
of our favorite feature tab- which
had a different icon.
Tripplan1-
Basic
High
2. Tripplan1-
Results
Page
Good look and feel. The users liked our design of
representing the results page and
also providing options to select a
particular bus stop.
Tripplan1-
Results
Page
High
3. Tripplan1-
Route
Results
Page
The selected stop result was
nicely displayed.
The users liked our route result
map page which displayed the
nearest stop address and location
(marked by red pin) selected by
the user on the Trip Planner Stop
Results Page.
Tripplan1-
Route
Results
Page
High
4. Nearby
Stops
Good way of providing list of
all nearby stops.
The users liked our idea and
interface of displaying all the
stops near to the users current
location.
Nearby
Stops
High
5. User
Feedback
Feedback Interface was
intuitive and simple.
The users liked our feature of
providing Feedback for a given
route/trip. The slider bar which
provided them a way to rate their
trip was effective and easy to use.
The users also liked the idea of
expressing their comments on the
feedback column.
User
Feedback
High

6. Tripplan1-
enter dest
An effective and simple
design to help locate nearby
The users liked our map interface
which tells the routes to user from
Tripplan1-
enter dest
High

Page 18

USABILITY EVALUATION REPORT
or required stops. his current location.
7. Select Bus-
1,2
Interactive Bus Selection
System
The users liked our Bus Select
Feature in our application.
Select Bus-
1,2
High
8. Bus
Schedule -
1,2
Interactive way of getting
information about the bus
stops.
The users liked our feature of
selecting the bus stops and then
having a drop down menu for
getting the information about the
bus stop timings.
Bus
Schedule -
1,2
High











Implemented Changes (Table 2):
a. Paper Prototype Changes:
Screen Name Way(s) to Rectify
Tripplan1 1. We provided the user with a way (used in iPhone) to select the date and time. A
submit button was also provided to the user to submit their date and time requests.
The selected date and time appear on the screen of the user.
2. After the date and time is entered, the same way the user can enter their locations
(source and destinations) as mentioned above.
3. Once the user was given with the list of the available buses, we provided a Filter
Option to them. For Ex: Suppose a user doesnt want to see all the buses, so in the
Filter Item tab, the user can type the number of the bus, and all the buses with the
similar number like 61A,61B,61C,61D will show up.
Favorite 1. We added the list of all the buses available in Pittsburgh. This way we provided the
user the option to select whichever bus they want to.
Tripplan2 1. Here we provided the user with the option of entering the destination via the method

Page 19

USABILITY EVALUATION REPORT
mentioned in point 3. Our GPS system locates the current position of the user and
shows that position on the map. From there the user can select the destination.
2. We also provided the user with a real time map, with the pin pointing to the users
current position.
Nearby 1. This feature will be implemented once the basic operation of our mobile application
is completed. This is an added feature, which we intend to implement before our
final development phase.
Explore 1. This feature will be implemented once the basic operation of our mobile application
is completed. This is an added feature, which we intend to implement before our
final development phase.
All Screens 1. We added a Back Button on each of the sub pages.
2. We introduced the application Logo on each of our main category pages.










b. 1
st
Phase Development Changes:
Screen Name Way(s) to Rectify
All Screens 1. All the features Route, Nearby Stops and News Feed tabs were added on all the
screens to enable easy navigation. Also their icon images were changed keeping the
feature name in consideration.
Ex: For Routes- we have a map tab.
2. The heading Port Authority of Allegheny County was resized to make it clear.
Tripplan1-Basic 1. The font for the heading: header Find Route By was changed to How can we
help? and also the following titles were rephrased.
Ex:
a. Address Trip Planner
b. Bus Number Bus Schedules
c. Favorite Favorite Routes
d. Maps Nearby Stops

2. We also added an advertisement bar on the same page.

Page 20

USABILITY EVALUATION REPORT
Tripplan1-
Select Date and
time
1. The font for the heading: header Choose a Route was resized to make it clear.
2. Icon image for Select Date and Time was changed and relevant one was put.
Tripplan1- date
and time
entered
1. The date and time format was corrected with commas and proper spacing added
where required.
2. Icon image for Select Date and Time was changed and relevant one was put.
3. The font for the heading: header Choose a Route was resized to make it clear.
Tripplan1-enter
curr and dest
1. The date and time format was corrected with commas and proper spacing added
where required.
Tripplan1-
Results Page
1. The design for the results page was improved by adding the stop information in bold
and color box.
Tripplan1-
enter dest
1. We added a red pin indicating users of their current location.
Favorite 1. The font for the heading: header Choose a Bus Number was resized to make it
clear.
2. We feel that the phrase Filter Items was apt for the given screen. Too many words
were not providing a good interface.
Favorite-with
filter
1. The font for the heading: header Choose a Bus Number was resized to make it
clear.
News Feed 1. New Options were added into the news feed feature. They are
a. PAAC Announcements
b. Crowd Shout
c. Trip Review






c. Final Phase Development Changes
Instead of the design changes for screens in the 1
st
phase development, in the below table we
describe the extra features that were added into our mobile trip application.
Screen Name Way(s) to Rectify
Routes Result
Page
1. We added another page to display the address of the nearest bus stop.
Nearby Stops 1. We improvised the design and functionality for the nearby stops feature in our
application by providing a list of all the stops near the users current location.
User Feedback 1. We added another feature of User Feedback into our application.
Select Bus 1. We provided the users an effective way of selecting a particular bus from the Bus
Schedules feature on the main screen and then get the route for the selected bus.
Tripplan1-Basic 1. The home screen was revised keeping in mind the functionality and features
required.

Page 21

USABILITY EVALUATION REPORT
Tripplan1-
Current and
Destination
Secletion
1. We provided a link called Current Location on this screen helping the user enter
his current location by selecting the link.
Bus Schedule -
1,2
1. We added a drop down menu on the list of all the bus stops and the timings of the
stops in the respective bus routes.

5. Screen Shots:
Below are the screen shots of our mobile application from the first phase development to
the last and final version.

The following table represents the first phase development and the second side the final
mobile application. There are certain pages in which there are no 1
st
phase development
screen shots. This is because we included certain features in our application in the final
phase development, which were not present in the 1
st
phase.
Screen Name 1
st
Phase Development Final Phase Development

Tripplan1-
Basic



Page 22

USABILITY EVALUATION REPORT
Tripplan2-
Departing and
Destination
Selection


Tripplan2
Time and Date
selection



Page 23

USABILITY EVALUATION REPORT
Bus Schedule -
1


Bus Schedule -
2 (Expanded)



Page 24

USABILITY EVALUATION REPORT
Tripplan-1:
Results Page



Tripplan1-
Route Result
Screen




Page 25

USABILITY EVALUATION REPORT
Favorites

Favorites-
with Filter
Option


Page 26

USABILITY EVALUATION REPORT
News Feed

Map and
Destination
Screen



Page 27

USABILITY EVALUATION REPORT
Nearby Stops
1


Nearby Stops
2



Page 28

USABILITY EVALUATION REPORT
User Feedback

Select Bus-1


Page 29

USABILITY EVALUATION REPORT
Select Bus -2
















Page 30

USABILITY EVALUATION REPORT

6. REFERENCES:

1. Mobile Computing:
http://en.wikipedia.org/wiki/Mobile_computing

2. Usability Checklist for the Usability Evaluation of Mobile Phone User Interface