Ten Tips of Web App Testing

How to test and launch a world-class application

White Paper
June 2011
 White Paper: Web App Testing

White Paper
Web App Testing
How to test and launch a world-class application

Table of Contents
Introduction.. 3
- New Challenges, Old Solutions.. 3
- The Crowdsourcing Advantage.. 4
- Testing Types.... 4

10 Tips of Web App Testing. 6 Why is there never time

1. Dont Be a Slave to Your Metrics.. 6
2. Know Thy Third Party Apps... 7 to do it right, but always
3. The Testing Managers Dilemma.. 7 time to do it over?
4. Invest in Testers, Then Tools.. 8
5. Protect Users, Save Yourself.. 8
6. Multimedia, Multi-Problems. 9 - Colonel Mike Mullane
7. Forfeit the Blame Game 9 Retired NASA Astronaut
8. Be Everywhere.. 10
9. Plugins: Proceed With Caution.. 10
10. Cross-Browser Testing. 10

About uTest.. 12

2
 White Paper: Web App Testing

Introduction
New Challenges, Old Solutions for Testing Your Web Application
Establishing procedure amidst a constant state of change may seem like an absurdity
(and a losing battle), but this is what todays software companies face when it comes to
testing their web applications. Aside from an expanding matrix of new browsers, plugins,
third party apps, programming languages and more, there are now thousands, if not
millions, of new users from all around the globe. Of course, none of this was in the
manual!
Complexity kills. It sucks the life out
Thus, the purpose of this whitepaper is of developers, it makes products
twofold. While our 10 Tips section will
difficult to plan, build and test, it
help you stay on top on of the latest
trends in web app testing, our
introduces security challenges, and it
introduction will outline the basics of causes end-user and administrator
functional, usability and load testing for frustration.
web-based applications. Along the - Ray Ozzie
way, well cover topics such as: Chief Software Architect, Microsoft
Third Party Apps: Your
product is only part of the equation - what about the satellite apps that are
orbiting your web application?
Cloud Computing: With opportunity comes risk. Theres a ton of upside to cloud
computing, but what happens to your web app when it fails to functional
properly?
Multimedia: How will your web app perform for users with different versions of
Flash, Acrobat, Java, etc.? Have you thought about HTML5 and the challenges
of managing H.264 vs. Theora?
Plugins: While the use of plugins may be slowing, the use of extensions are not.
Learn why plugins and their extensions should be a testing focal point.
Localization: Obviously, your application needs to work where your users
reside. Discover why most software companies underestimate the importance of
localization in their web testing.
Deadlines: Your web application will never be perfect, so how will you know
when its safe to launch? Its called the Testing Managers Dilemma and well
show you how to deal with it.
Metrics: As one testing expert put it, When your car is about to go off a cliff, its
a weird time to be thinking about gas mileage and drag coefficients. Find out
how metrics can mislead.

3
 White Paper: Web App Testing

The Crowdsourcing Advantage

While the following material is suited for companies that rely on traditional staffing
models for testing (i.e. internal QA teams or offshoring firms), this whitepaper will be
especially useful for those who leverage some aspect of crowdsourced (or community-
based) testing.

To learn more about the growing trend of crowdsourcing - and how companies of all
sizes can use it to complement their in-house efforts check out the 8 Essentials of
Crowdsourcing whitepaper.

Web Applications: Testing Types

For a successful technology,
Functional Testing reality must take precedence
No surprises here. The most frequent and critical over public relations, for nature
task that QA takes on before launch is to perform
comprehensive functional testing. This can
cannot be fooled.
include - but is not limited to - testing of: - Richard Feynman
Renowned Scientist
Web Forms: This includes sign-up, sign-
in, contact and purchase forms; database
queries, checking for cross-site scripting/AJAX vulnerabilities and other issues
related to incorrect inputs.
Links: Ensuring that all outgoing and internal links are functional and accurate.
GUI: Despite back-end test automation, front end problems with web applications
are almost always discovered by human testers, including issues related to
browser compatibility or CSS flaws.
Language: Does your web app properly support Unicode? Can it accept and
handle foreign characters? This is a growing concern for developers and should
be considered as an essential part of your testing efforts.
Security: Security testing including checks for open redirects, cookie
tampering, SQL injections and other vulnerabilities that could harm your users
(and reputation) should also be performed prior to launch.

Bottom line: Does your web application work when, where and how your users need it
to work?

Usability Testing
Beta testing may be suitable for some large companies (mainly those with name
recognition) who want to solicit user feedback, but it is NOT an effective substitute for
usability testing, which should include professional analysis of:

Graphics: Have you identified and addressed the major GUI bugs in your web
app? Are the color schemes easy to understand (in every country of your user

4
 White Paper: Web App Testing

base)? Many of these flaws will not be reported by beta users, but they should be
known about and fixed before launch.
Feature Set: This can include the intuitiveness of shopping lists, online
checkouts and other e-commerce actions; the accuracy of search results, data
entry, sign-up forms and other features.
Accessibility: Is your application accessible for all users, including the visually
impaired? Does your app involve parental or administrator restrictions? Be aware
of your legal obligations.
Benchmarking: How does your web application stand up to that of your closest
competitors? What about your own previous versions? What areas does it
compare favorably or unfavorably?
Overall Usability: Beta testers - if they report bugs at all - are likely to respond
with this-site-sucks or this-site-is-great type of analysis. When launching a
web application, professional, non-biased analysis is essential.

Bottom line: Is your web application intuitive and highly usable for your target
audience?

Load Testing
How will your web application hold up when it
The real value of tests is not
needs to most? Synthetic load tools are helpful in
that they detect bugs in the code,
simulating these types of scenarios, but with the but that they detect inadequacies
assistance of real testers, you can obtain a in the methods, concentration
complete picture of how your application and skill of those who design and
performs under stress. produce the code.
Load testing should, at the very least, involve one
of the following methods: - Tony Hoare
Testing Expert
Live Load Testing: A team of live testers
(preferably from where your users reside whether it be North America or a
global audience) will test your application simultaneously, enabling you to see
how your app performs under real-world, non-simulated load. This process, as its
name would indicate, requires NO automated tools. Ideal for web apps that
contain Flash, streaming video or other types of multimedia that are difficult to
evaluate with simulated load.

Simulated Load Testing: Using the simulated load testing tool of your choice
(more on this below) you can obtain a realistic snapshot of your web application's
performance under peak synthetic usage.

Hybrid Load Testing: A combination of the aforementioned methods, this

process involves having live testers perform functional testing on your web
application while under automated load. This lets you see bugs or performance
issues that only show themselves under peak loads.

5
 White Paper: Web App Testing

Bottom line: Will your web application perform as expected under maximum pressure?
Will it crash? Will the performance degrade?

Web Testing Checklist: A Summary

Regardless of whether you are performing functional, usability or load testing, it is
imperative that each of these practices achieve maximum testing coverage that matches
your user base across:

Location: If you have a global

user base, why would you only When your car is about to go off a cliff,
test your web application in your its a weird time to be thinking about
office? Its therefore critical to gas mileage and drag coefficients;
extend your testing coverage better to take the right control action
beyond your borders if needed. look out the window and steer or use
Language: If your application is
available in multiple languages, the brake until youre back on course.
you must verify that nothing is
lost in translation (like the - Michael Bolton
intuitiveness of your content, CEO, DevelopSense
error messages and core
features).
Operating Systems: Your app
needs to work seamlessly across
all the different flavors and versions of Windows, Mac and Linux.
Browsers: The days of Internet Explorer hegemony are over. Todays web
applications must be tested thoroughly against multiple versions of IE, Firefox,
Chrome, Safari and others.

And now, the Ten Tips of Web Testing.

The 10 Tips of Web Testing

1. Dont Be a Slave to Your Metrics
When testing a web application, its easy to become overwhelmed by the data thats
available to you. It can drag you in all sorts of directions and distract you from more
important matters if you allow it to. Dont. Said testing expert Michael Bolton of
DevelopSense:

Some people enslave numbers. They make numbers work too hard, and too
often. Ive seen organizations collect piles of data about defect escape ratios and
defect detection percentages. They hire market research firms and calculate the
ratio of happy customers to unhappy customers. But the aggregated data doesnt

6
 White Paper: Web App Testing

tell you anything specific on how to make things better for the unhappy
customers.

In other words, while comprehensive testing will present you with a ton of raw data,
the most important information will still come from your testers, managers and
customers.

2. Know Thy Third-Party Apps

When launching a web application, remember that it wont operate in a vacuum.
There are many third-party applications that could interfere with its performance,
apps like:

Live chat
Checkout processes
Search plug-ins
RSS Feeds
Embedded videos or audio players
Ad servers or embedded ads
Web analytics packages
Blogs, forums and message boards
Social networking modules or toolbars (like Tweet streams or Digg)

Do any of these apply to your web application? To your users, these are part of your
app.

3. Understand the Testing Managers Dilemma

As any honest test manager will tell you, it is NOT his or her responsibility to make
the final decision about when the product is completed that is clearly the role of the
product owner. Testing managers have enough to worry about. Between time limits
and budget constraints, theres no shortage of obstacles to must deal with so dont
make them the final arbiter of product readiness.

Besides, to a good testing manager, their job is never complete. In theory, software
testing could go on forever! Despite knowing better, software executives regularly
pressure their testing managers into providing an answer, and end up being told
what exactly they want to hear.

Also, it is okay to fall off the agile wagon every once and awhile. With the short sprint
cycles and constant deadlines, its amazing that more companies dont wind up there
more often. But just because you cant maintain a truly agile schedule 24/7, doesnt
mean that certain aspects of your development (like testing consistently) should ever
be abandoned.

7
 White Paper: Web App Testing

4. Invest In Testers, Then Tools

While bug-tracking systems and automated test tools are an essential part of any
test team, theres a price to be paid for relying on them too heavily literally and
figuratively. Consider automated tools, for instance. Automation will never find bugs
for use cases that havent been conceived and documented. It simply helps to
ensure that old bugs dont resurface and that the main test cases are clear.

Conversely, live testers will find

bugs that are new or unique to
your app - which is probably
most of them. In short, ignoring
real-world testing at the expense
of tools is a recipe for disaster.
Speaking of expense, if you dole
out big money for elaborate test
tools (and expect an even bigger
ROI), you will feel compelled to
keep using it, regardless of its
effectiveness. In this regard,
software companies would be wise to follow the advice James Bach gave to
individual testers (see sidebar).
I urge you not to use expensive tools,
even if they work. Never let your
manager buy them. Because expensive
tools become something you MUST use,
even if they dont work. A free tool may
be freely abandoned. This gives you
flexibility.
- James Bach
Testing Expert

5. Protect Your Users, Save Yourself

To have a safe, secure web application is obviously a no-brainer nobody launches
with anything less in mind. Whats not so obvious is the way to achieve this type
safety for your web app. To point you in the right direction, here are a few questions
to ask yourself before your next launch:

Is your web app behaving the way your privacy policy claims it does?
Can a tester easily get an app to cough up the private data of another user?
For example, if they see user_id=232 in the URL, what happens if they
change it to user_id=231? Do they get to see someone elses personal
data?
Are you sharing personally identifiable information about your users with
third-parties like Salesforce.com or Google Analytics? What about the
company thats hosting your app?
What happens if your web app is cached when it shouldnt be? Does it share
the wrong data with people? What happens if you actually want caching? Is it
sending out the right things to be cached? Are end users actually seeing a
benefit?

8
 White Paper: Web App Testing

Is your site vulnerable to common security exploits like XSS, injection flaws,
broken authentication, flawed session management, unvalidated redirects
and forwards?

Since these areas are sometimes overlooked by in-house teams, it can be helpful to
leverage testers who are skilled in testing, but new to your application.

6. Multimedia, Multi-Problems
How much does your web application rely on Flash, HTML 5, Java and other rich-
media tools? Even if you answered a little bit, multimedia testing - including load
testing - should become a critical component of your testing from this day forward.

Of course, many in-house teams

will find they have neither the Its important to have a tester that can
time nor the headcount to tell the difference between an
achieve testing across all enhancement and a true bug. I know, it
versions of multimedia players. If
sounds so ridiculously trivial but finding
this is the case, you should
strongly consider leveraging a quality testers who can also understand
community of professional the nuances of business is key.
testers to quickly and easily find
the versions you need.
- Jack Margo
SVP of Web Operations, DevShed
7. Forfeit the Blame Game
Reproducing defects, filing bug reports and verifying fixes are tasks commonly
believed to be bottlenecks in the testing process. They are not. While they can
comprise close to 20% of a test teams duties - and while testing teams are often
blamed for delays in these processes - it is often actually a bottleneck on the
development side (where the engineers are competing with their own challenges,
obstacles and deadlines).

If test teams can work with development (rather than blame them) to improve the
quality of the software prior to code completion, then it will improve the speed of the
whole system. Buggy software can seem like the test teams fault, but it is important
realize that other factors are contributing the pace of each development cycle - so
dont take it out on your testing team.

Making developers better, helping them understand failures and the factors that
cause them will mean fewer bugs to find in the future. Testers are quality gurus
and that means teaching those responsible for anti-quality what they are doing wrong
and where they could improve.

9
 White Paper: Web App Testing

8. Be Everywhere

Does your web application work in New Zealand? What about Singapore? If your
web app is localized that is, if your users are dispersed throughout the globe then
youll need to test language translation, currency conversion and other location-
specific features. Aside from functionality, it is equally important to run usability
testing with users from around the world.

Neither in-house QA teams nor outsourcing firms can efficiently complete such a
task. It would be (and indeed, has been) time-consuming, costly and impractical. And
thus, it has been ignored. However, crowdsourcing enables you to easily leverage a
global community of professional testers, letting you handpick the testers you need
to fill out your testing coverage matrix.

9. Plugins: Proceed With Caution

Plugins remain a significant problem for web application developers. In fact, most of
the security issues on the web today are a direct result of Flash and Acrobat bugs. In
many cases, people end up with ancient versions of Acrobat reader that can run
random exploits simply because they forgot to make the needed updates. Does this
sound familiar?

Also, while the sheer number of browser plugins has declined in recent years,
extensions have gained in popularity. While less common (and usually only found on
more advanced users browsers), extensions can dramatically impact how an app
will work if they are buggy. As such, they too will require testing prior to launch.

10. Respect The Dead Make Way For The New

Its no secret that IE6 is on its way out, but large organizations cant seem to kill it off
just yet. As frustrating as it may
be, support for older browsers can The internet? Is that thing still around?
be critical, especially for B2B
applications.
- Homer Simpson
Conversely, the next wave of
HTML5 is going to introduce a flurry of competing multimedia standards. HTML5 vs.
Flash is going to be a big debate in the years ahead, and companies will have to
make a very important choice: Either they deal with Flashs security issues and lack
of support on mobile, or they opt for HTML5s confusing vendor specific standards
(H.264 vs. Theora).

10
 White Paper: Web App Testing

Also, CSS3 is slowly making its presence felt. In fact, many web apps already use it
for additional layout effects. If you go down this road, how will non-CSS3 browsers
deal with this?

Cloud computing is another major trend to keep in mind when testing your web app.
How will your application behave when different parts of the cloud are misbehaving?
What happens to your lead generation efforts, for example, if your CRM system
malfunctions? Or if Omniture goes down for an hour? What effect will this have on
your users? Consider these cloud issues when testing.

While some trends will die before their time, others will persist far beyond whats
anticipated. For web application companies, it is critical to be mindful of these trends
and remain on the offensive.

Then again, you dont have much of a choice.

11
 White Paper: Web App Testing

About uTest
uTest provides in-the-wild testing services that span the entire software development lifecycle
including functional, security, load, localization and usability testing. The companys community
of 50,000+ professional testers from 180 countries put web, mobile and desktop applications
through their paces by testing on real devices under real-world conditions.

Thousands of companies -- from startups to industry-leading brands rely on uTest as a critical

component of their testing processes for fast, reliable, and cost-effective testing results.

More info is available at www.utest.com or blog.utest.com, or you can watch a brief online
demo at www.utest.com/demo.

uTest, Inc.
153 Cordaville Road
Southborough, MA 01772
p: 1.800.445.3914
e: info@utest.com
w: www.utest.com

Case Study
Case Study

12