2012 Bryan Eisenberg & Jeffrey Eisenberg & Associates

Table of

4 2
CONTENTS

7 6
About Bryan Eisenberg

Introduction

About This Guide

36 18 8
Truth in Testing & Targeting

5 Best Practices

Stakeholder Questions

70
Vendor Profiles

68 How Can Eisenberg Brothers & Assoc. Help You?

Appendix: Sample RFP Questions

 About
Bryan Eisenberg is the co-author of the Wall Street Journal, Business Week, USA
Today and New York Times bestselling books Call to Action, Waiting For Your

Bryan Eisenberg
Cat to Bark? and Always Be Testing: The Complete Guide to Google Website
Optimizer. Bryan is a professional marketing speaker who has been the keynote
speaker for corporate events and conferences such as Search Engine Strategies,
Shop.org, Direct Marketing Association, MarketingSherpa, E-consultancy,
Webcom, SEM Konferansen Norway, and the Canadian Marketing Association.

He is also the co-founder and Chairman Emeritus of the Web Analytics Association.
Bryan serves as an advisory board member of Search Engine Strategies, the
eMetrics Marketing Optimization Summit and several venture capital backed
startup companies (e.g., Bazaarvoice, iPerceptions, UserTesting.com, Monetate,
Runa.com, TagMan, BoostCTR, etc.) and is on the board of trustees of the Direct
Marketing Educational Foundation.

Bryan Eisenberg is the recognized authority and pioneer

in online marketing, improving online conversion rates,
Persuasion Architecture, and persona marketing.

Bryan was recently recognized by eConsultancy members as one of the top 10

User Experience Gurus, and was selected as one of the inaugural iMedia Top 25
Marketers. He also was honored with a DMEF Rising Star Award in 2010.

2
Bryans expertise has been featured in The Wall Street Journal and The New York
Times, and he has been quoted in Business 2.0, CXO Europe, Advertising Age,
CNN, Forrester Research, Jupiter Research, Miami Herald, Publish, Internet
Advertising Report (IAR), Chicago Business Tribune, Inc Magazine, Entrepreneur,
Target Marketing, DM News, Microsofts bCentral, MarketingSherpa, the Toronto
Star, Smart Money, and Internet Retailer for his thought leadership in the critical
area of internet marketing and improving online conversion rates.

In 1998, Bryan co-founded FutureNow, Inc. (Bryan & Jeffrey Eisenberg, his brother
and business partner, are no longer associated in any way with FutureNow),
the company that has been helping businesses generate more engagements,
leads, subscriptions, and sales with its unique framework he helped develop,
Persuasion Architecture. Bryans proudest professional accomplishments are
the thousands of companies, students and clients (including HP, NBC Universal,
GE, WebEx, Overstock, and Dell) that have consistently enjoyed dramatic
improvement in sales.

For more on Bryan,

please visit www.bryaneisenberg.com, and check out the full website testing tools directory at
www.websitetestingtools.com.

3
 Introduction
In the past several years,
a new generation of testing and targeting solutions has
sprung onto the optimization scene, ushering in a new
normal for the industry. These next-generation tools are
more intuitive, powerful and cost efficient than those
that came beforeand their purview extends far
beyond website testing as the world tends to
understand it.

More than merely running A/B or multivariate

website tests, these tools optimize the
entire consumer experience, including
across channels. The most innovative tools
among them variously offer sophisticated
targeting features, IT-free campaigns,
consistency of messaging across channels,
and truly intuitive content creation-including
WYSIWYG interfaces. Having raised the bar
for the industry overall, these technologies
have spurred the legacy tools to roll out new,

4
more powerful features to stay in lockstep
with the times.
As the industry has matured, a new
set of best practices has also taken
rootpractices that simply werent
possible in the earliest days of testing,
when solutions were far more code-
and time-intensive. Thats good
news for any marketing executive
in charge of site optimization. The
downside is that the marketplace
is more crowded than ever, and it
can be hard to winnow down your
consideration set to the one best
solution for your company.
This Guide will help you do that,
reviewing eight of the best-known
testing tools available today. For
ease of reference, theyre divided into
categories, including the industrys
only free tool, landing page tools,
and tools for the enterprise.
However, this Buyers Guide marks a
departure from those that have come
before it. Rather than regurgitating
what each company says about itself,
creating feature matrices, or basing
recommendations around which
tools have the most boxes checked,
this guide approaches tool selection
from your perspectivethe buyers,
influencers, and end usersand the
Whether youre the CMO
or marketing manager (or,
questions you should ask yourself as
well as prospective vendors.
perhaps, not a marketer at
all), its your job to stay a
Ill explain why each of these
questions is so critical to a
successful buying decision, in light
step ahead of leading-edge
innovations. Welcome to
of the best practices that a cutting-
edge solution must have in order
to deliver the highest ROI possible.
These questions and best practices
your edge.
concepts on the vanguard of
optimizationwill not only help you
identify the tool best suited for your
company, theyll give you what you
need to sell your recommendation
both up and down the ladder.
5
 About This Guide
This Buyers Guide is opinionated, and rightfully so.

Most Buyers Guides weve encountered try to provide a general

overview of the space, and of all the vendors contained therein. They
dont assess vendors based on whether they make testing easy for the
enterprise, or on how well they allow marketers in your organization to
act quickly, follow best practices, and drive incremental revenue.

Make no mistake about itthere are right ways and wrong ways to test.
Among the vendors profiled in this Guide are tools that I have used,
companies I have advised, and whose customers and users I have
interviewed extensively.

Because of this, Ive included (and excluded) vendors from this Guide
based on how well they allow you to follow the best practices contained
within this guide. Ive then gone a step further and made explicit
recommendations based on how innovative and effective each of these
solutions is, and which ones are a better fit for your organization.

Remember, not all testing tools are created equal. Some are better at
one thing than another. And selecting the wrong tool can carry steep
consequences: missed opportunities, money left on the table, and

6 wasted time for you and your team. So, let these words be your guide
... and choose wisely.
Truth in Testing
The website testing industry is, unfortunately, horribly opaque.
Remember, website testing as a practice is only, at most,
15 years old. So, true experts in the space are scarce. Most
marketers dont fully understand website testing, how it works,
or what the best practices are. And, all too often, vendors take
advantage of this fact by over-promising and under-delivering.

Some things to watch out for:

Especially in the enterprise market: Vendors that emphasize reaching some
vague, opaque, and otherwise nirvana of fully optimized perfection. In
jargon-y websites that show very little reality, only the companies that make
of the product or discuss the full extent testing part of their culture reap rewards.
of product features. To see significant lifts in key business
metrics like lead generation and
conversion, testing must be a continual
practice, since the marketplace is ever-
changingas are consumer habits. If
youre aiming to come to a point where
Solutions that are easy. Some tools
are easy to use, others arent. The
problem is that almost all testing
solutions promise that theyre easy.
So, talk to customers. Try your hand
at using the tool if you can. And make
sure you understand everything it takes
to get started and to actually launch a
campaign. Remember, you dont want
a tool that becomes shelf-ware.

7
youre finished with testing, youre
missing the point.
 5 Best
Practices 1
Message consistency
across channels
When consumers interact with the web,
they follow scent trails, which help them get to a specific destination:
namely, the product or information they need and want. When they click
on your PPC ad, its not for the sake of clicking. Its because that ad
promises to take them one step closer to the very specific destination
theyre sniffing toward. When at last they arrive on your landing page,
the creative and messaging had better deliver the same creative and
messaging as the referring ad. If it doesnt, youre likely to lose that
visitor to another search, where a more promising scent trail could lead
them into your competitions arms.

Its a basic tenet of both on- and offline marketing: When you have
a consumer in a buying state, dont break that state. Yet too few
solutions are built to keep consumers in a buying state across their
online experience. At its heart, this is a targeting issue. To ensure that
visitors dont feel theyve been taken off trail, targeted content must

8
be delivered to visitors from specific referring URLslike a Facebook
URL, PPC ad, or an email link.
The most advanced solutions then do one better: The same scent, or
information, that attracted a visitor to your sitefor instance, a PPC
advertisement that addresses outdoor enthusiastscontinues to be
part of their experience on your site, following them from page to page
to page.

As mentioned earlier, a number of solutions have

the ability to target. But when you get down to it,
too many built-in barriers make it impractical.

A similar thing happens here. Tools that truly foster consistency across
consumer touchpointsand from email to homepage to catalog page,
for exampleare few and far between.

9
 5 Best
Practices 2
There is no Finish
Line in Optimization

Imagine a perfect world

in which one all-out cardio-training blitz at the gym would keep you fit
for the rest of your life. Well, a perfect world this is not. To stay in shape,
you soldier off to the gym three times a week. Optimization? Same
thing. To wring out significant returns on your investment, optimization
must be an ongoing part of your daily marketing activities. It has to be
a habit.

So , if youre aiming to come to a

point where youre finished with optimization,
youre missing the point.

10
And yet, its not uncommon for
companies to test as if they were
headed toward a finish line: They
run an experiment to yield the one
best version, slotting in that winning
element or page before moving on to
the next element or page. Their goal
is to achieve some ideal state of fully
optimized (but ultimately nonexistent)
perfection. One reason this approach
dominates is the resource-intensive
nature of many testing solutions.
To run a simple A/B test, a
companys design and IT
departmentsboth of which
are typically overextended
must develop multiple
creatives and code each
element for implementation.
Given their already long to-do list, theyre
likely to put your campaign toward the
bottom of their priorities. In that context,
an uninterrupted stream of concurrent
tests is simply unrealistic.
The other major barrier to this best
practice emerges in the pricing
department. Some solutions charge per
server call, according to the number
of tests launched in a month and/
or the duration of your campaigns.
With the meter running, a marketers
natural impulse is to test judiciously, not
continually. Ostensibly, thats a cost-
cutting measure. In reality, it means
youre leaving money on the table.
To reap the rewards of ongoing,
consistent testing and optimization,
look to the next generation of testing
solutions in this guide. How do you
{ With the meter running, a
know if a tool is built to support testing
as a habit? It must have true one-tag marketers natural impulse is to
integration, a flat or cost-efficient pricing test judiciously, not continually. }
model, easy content creation, and top-
flight customer support. Ask vendors
about each of these elements: How long
does implementation really take? Do
you charge per server call? And how
do you price packages for support from
your team of experts?
11
 5 Best
Practices 3
Targeting for the
real world
Until you deliver targeted content
to meet the specific needs of each visitor segmentnew versus returning
visitors, for instance, or U.S. versus international visitorsyou havent fully
tapped the promise of optimization to move the needle on your bottom line.
Youll notice that many tools have the capacity to serve up optimized content
for different visitor segments, engaging consumers based on their traffic
source or demographic profile. However,

having the capacity to target differs dramatically from

making it easy and affordable to target.

Before you invest in any solution, look for hidden barriers to targeted
campaigns. Again, it comes down to implementation, content creation and
pricing. Complicated coding and content-creation requirements put targeting
in the control of IT experts and leaves many marketers feeling shackled.

12
Similarly, solutions that charge by server
call or according to campaign duration
make targeting a cost-prohibitive affair.
In the end, the investment required to
implement targeted campaigns undermines
the lifts it might otherwise yield.

When youve identified a few

tools that might meet your needs,
look through their case studies
to gauge how much emphasis
Remember:
they place on targeting. To make a meaningful difference,

your tool must be made for the real

Then ask the vendors in your consideration
set: How many concurrent, targeted site
versions are your clients running right now,
world, not a perfect world.
today, on their websites? (Understand, too,
And in the real world, most IT and design
the distinction between running multiple,
departments will get to it when they get to it.
targeted site versionsconcurrently and
Thats just not fast enough.
continually according to visitor segments

13
and testing multiple page variations to
determine the one best version.)
 5 Best
Practices 4
Increase revenue
across channels
In many online businesses,
a marketer or marketing team owns the process of testing. In the case of
code-heavy solutions, an IT specialist might lead the charge.

That kind of centralized ownership model neglects the

reality that revenue is driven by performance across multiple
channels. Centralized ownership is a problem particularly for
the enterprise-level company, whose customer experience
extends across email, search, affiliate, advertising,
merchandising, display, brick-and-mortar, and more.

No one group or person can understand what it would take

to lift conversions in each of those varied channels. But
when multiple users from those business functions work
with a testing tool, each of those channel experts can drive
margins by optimizing their respective area of expertise.
That promotes growth across your entire business.

14
How do you know when a tool supports cross-channel growth? The
first sign is that many users across your organization can log in
with a username and work with the tool. In itself, of course, a
username isnt worth the keystrokes it takes to type it when
the tool is too complicated to understand. Its imperative
that the technologyincluding implementation,
content creation, and data analysisalso be easy
to master for different users, many of whose
backgrounds will not be IT-related.

15
 5 Best
Practices 5
A culture of testing
sparks innovation
When you work with a tool
that offers easy implementation and content creation, it does more than fuel
conversions for your online business: It drives innovation and shapes the
strategic direction of your entire marketing plan. Consider a scenario as it might
have played out years ago: A senior executive, concerned about shopping cart
abandonment rates, asks you what would happen if you clarified the labels on
your forms required fields. Would you see better numbers?

Getting an experiment into play is time-consuming and expensive, so you

and your team first meet to discuss the merits of several new forms, debating
countless iterations of the form until you eventually come to a decision. You
ask your design department to create the form, then you wait for IT to find time
to code the experiment. Did it decrease cart abandonment? Maybe. But not
necessarily enough to compensate for the effort you poured into it.

16
Now consider whats possible with
a fast, affordable, IT-light solution.
Your executive wonders what would
happen if you tweaked a form. This
time, internal debate is unnecessary.
Instead, your team can implement an
experiment in a fraction of the time
without calling on your developers and
designers. Think minutes as opposed
to weeks or months. In that case, you
can do what every marketer is trained
to do: Use the dataand not debate or
opinionto guide your decisions.
And now weve arrived at the heart of
the matter, the game-changing benefit
of testing as a habit: When testing is IT-
free and easy, it changes your culture.
It creates a culture of innovation. And
it drives strategy for your marketing
across channels, including offline
channels. Rather than discussing
every suggestionand bracing for a
protracted and tedious processyour
marketing team embraces new ideas.
You test and optimize them liberally.
And what are new ideas if not critical
assets for innovation?
To understand how a culture of
testing can guide the strategic plan
for marketing across channels, this
hypothetical example will shed light:
Marketers at a fashion website, noting
the popularity of local offers, want
to try that tactic on for size. Without
sophisticated testing tools, they would
have no recourse but to run a local offer
and hope that something, somewhere,
sticks. It might work. It might not. Either
way, theyre flying blind.
By contrast, if testing is woven into
their culture, theyd do a deep dive into
So if your idea of testing is
their analytics, identifying a market evaluating a red versus blue
with enough traffic to yield quick button, think again.
results. Maybe its New Yorkers during
Think bigger than buttons.
New Yorks Fashion Week. If that test
offer performs well, theyll understand
that theyve identified a demographic
Way Bigger.
sweet spot. With that insight in
their pockets, they can then jump
channelsfor instance, running print
advertising during Fashion Week
with the certainty that theyre in the
17
right place at the right time.
 Stakeholder
Questions
When it comes to testing
organizations have a range of goals, skill levels, and resources at their
disposal. To make sure you buy the right solution for your organization,
each stakeholder needs a seat at the table. But does each person know
what to look for in a testing solution, what questions to ask, and why?

In my experience as a marketing optimization educator and consultant,

there are typically five stakeholders in the buying process, each with a
different role, set of responsibilities, and metrics for measuring success:

18 Chief Marketing Officer Ecommerce Director Channel Managers IT Directors Web Analyst
In this section, I list the questions that are most important for each of
these stakeholders to ask. They consist of two types:
Introspective questions that the stakeholder should
ask himself; Vendor questions regarding ease of use,
pricing, and capabilities
Whats most important, though, is that you understand why these
questions are the most important for you to ask, and how each
ties into the best practices in testing. With this understanding, the
process of evaluatingand eventually selectinga solution will
become easier and faster, and pose less of a risk that you end up
with a mismatched purchase
Chief Marketing Officer
Introspective Questions
(1) Am I looking for a solution that facilitates simple, small-scale
testing, or one that encourages continuous use and helps me keep
pace with competitors and my customers rate of change?
Why It Matters: You neither want to pay for features you wont use
nor pay for those you need but dont get. But in addition to price, if
your goal is to run as many tests as possible (and it should be), you
need to consider both a tools:
Pricing Model: Some tools charge by the server callmeaning the
more you test, the more you pay.
Ease of Use: The speed with which campaigns can be created
These two factors drive a tools total cost of ownership. Remember,
theres no finish line in optimization, so to develop a culture of
continuous testing and optimization, you need a tool that lays the
groundwork for your success. The easier a tool is to use, the more
likely youll see high rates of adoption and, as a result, a culture
where testing truly takes root.
That said, extensive testing and optimization may be premature if
your organization has little-to-no experience in these tasks. In such
a case, a less expensive (even free) tool can help you get your feet
wet with simpleyet insightfulcampaigns.
But Buyer Beware: The least expensive tools
often carry hidden costs. Namely, tools may be
cheap, but your people are not..
19
 Chief Marketing Officer
Introspective Questions

(2) Who in my organization is responsible for conversion rate

optimization (CRO)?
Why It Matters: When CRO is everyones job, its no ones job. And
having a dedicated team reflects a companys belief that it does, in fact,
have influence over its conversion rate in the first place. According to
an Econsultancy study, many companies dont believe they do, which
is why, on average, they spend $92 to acquire new traffic for every $1
spent trying to convert it. 1

At its core, though, this question is about whether or not youre ready
to test in the first place. If conversion rate optimization is no ones
responsibility, then no testing solutionexpensive or free, robust or
simpleis likely to produce a strong ROI. So, if youre worried about
the CEO asking, How will you address the conversion rate problem?,
start by discussing the need to change hearts and minds.

Changing mindsets, though, can be a chicken-and-egg problem. Do

you buy a robust tool firstto prove value and then change minds? Or

20
do you get a free tool to help you make small steps first, with the goal of
switching later to a more capable solution?
Regardless of how you answer this last question, the key issue here
is a tools ease of use. The easier it is to use a tool, the more likely
you are to see high rates of user adoption, and therefore, a change
in the belief that employees can influence conversion rates. Change
the mindset, and it will show in your ROI.
you name it. In other words, theres a direct correlation between a
tools ease of use and just about every key performance indicator
(KPI) that your company tracks.

Vendor Questions
Four other ways to
gauge ease of use:
(1) On average, how many simultaneous campaigns do your
customers run?

1
Look for testimonials on the vendors
Why It Matters: For obvious reasons, the question, How easy site and across the industry
is your tool to use?, wont get you very far. The answer you receive
will have less to do with ease of use and more to do with how quickly
2
Ask for references
the Sales Manager wants to close the deal or whether he thinks he
can justify full-service management as an add-on.
3
Check to see how many screenshots
of the tool are on the website
Asking about simultaneous campaigns, therefore, is your best proxy
for a tools ease of use. Tools that are difficult to use manifest this
4
Consider the demo: Are you seeing
quality by having averages in the 1-to-5 range.
all the details? Does the workflow
make sense?
Tools that are the easiest to use average close to
50 simultaneous campaigns.
And, of course, more simultaneous campaigns means greater
opportunities to increase AOV, revenue, conversion rates, and
21
 (2) I have millions (or billions) of page views a month.
Can you handle our traffic?

Why It Matters: All marketers like to boast about the amount of

traffic their website gets, and in my experience, the higher you are in
the organization, the louder that conversation gets. But when it comes
to testing, the question is about more than just ego-stroking. Lets start
with the basics: testing and targeting solutions are all architected one
of three ways:

Client-side: Cloud-based tools leveraging JavaScript

to dynamically change the website experience.

Server-side: The original form that testing solutions took.

Each test element requires a separate server
call to serve the content.

Proxy: Proxy solutions alter web content as it passes

22
through the networkresiding between the
host and end user.
Only client-side solutions are equipped to efficiently
handle high-traffic websites, and heres why:

Server-side: Generally, server-side solutions charge

usage-based fees. This lends unpredictability
to marketing budgets and causes most users
to throttle their usage of the tool itself.

Proxy: Proxies are limited in the amount of traffic

they can process at any given time.
High traffic sites will typically have to throttle
the percentage of total site traffic that
passes through the proxy.

Client-side tools offer near-infinite scalability by virtue of their

cloud-based architectures-no need for throttling. But keep
your eyes peeled for a vendor that offers fixed,

23
flat-fee pricing.
 Ecommerce Director
Introspective Questions

(1) How extensive are my content and creative needs? Do I have a

means to create this content?

Why It Matters: Scalability is one of the key drivers of how high your
return on investment will be, and it comes in four flavors:

1
No scalability whatsoever

2 Ability to create landing pages

3 Ability to render dynamic text in banners

and other creative assets

4 Ability to repeat campaign actions

across multiple pages and creative assets

The tools that offer no scalability are those that require additional,
throwaway code around each test element. As a result, the very idea of a

24
multivariate test becomes cumbersome. Prepare your army of developers,
as thats what it will take to get many tests up and running.
The ability to render dynamic
text and repeat actions are what
quickly turn simple tests into a
site-wide experience.
Tools that offer these capabilities define
tests not according to the page or element,
but rather the rule. The ability to use a simple
rule that says, Render this badge on top of
this type of product, or this type of page, is
incredibly powerful to the marketer.
(2) Which channels give you the most
trouble converting visitors into customers
(e.g., paid search, display, social media,
affiliates, email, etc.)?
Why It Matters: Companies have a
tendency to compartmentalize optimization
efforts. Channel Managers are spread
across a range of teams and departments,
not to mention Design, User Experience,
Product, etc. But conversion optimization
doesnt take place in a silo, and a key based on a fixed, flat monthly fee. (Tools
ingredient in your success is the ability to with usage-based fee structures quickly
optimize holisticallyacross channels and make multi-page campaigns prohibitively
internal departments. expensive.)
However, this involves more than just Lastly, look for customer testimonials that
identifying, for example, visitors from paid speak to a tools decentralized ownership.
search and adjusting the landing page When Channel Managers are among the
experience. You need to maintain the scent tools experts, theyll be among the tools
trail that attracted the visitor to your site in users. By contrast, with tools requiring a
the first place by preserving the messaging, heavy reliance on IT, channel managers
offer, look, and feel of acquisition lacking programming experience are
campaigns throughout the entire website. unable to use them to their fullest potential.
The result? Customers enjoy a consistent
experience from before the click on through
to checkout. By keeping visitors in a ready-
to-buy state, the likelihood of converting
those visitors increase.
Can the vendor youre talking to do this?
Listen for the term multi-page campaigns,
25
which is the keyword you want to hear. In
addition, make sure the pricing model is
 (3) How much experience do we have in website testing and
conversion rate optimization?

Why It Matters: Ecommerce Directors worry about expensive

software becoming shelf-ware, and rightly so. But thats ultimately
because many teams lack the resources and knowledge to use these
purchases to their fullest potential. Look for a vendor that provides:

Dedicated Team: If your company lacks a dedicated team whose

sole responsibility is conversion rate optimization, a vendors full-
service managers can provide the next best alternative. And make
no mistake, this job is hugely importanta culture of continuous
testing requires resources to develop a strong testing strategy,
execute against it, and generate meaningful reports and analysis
that drive constant improvement.

Extensive Experience: Full-service teams manage optimization

campaigns for a living, and so they bring a depth and breadth of
experience about whats worked (and equally important, what hasnt
worked) for companies just like yours. This means they can hit the

26
ground running with successful, high ROI campaigns, and ensure
you have a great story to tell when the CEO asks.
Familiarity With the Solution: The best race
car drivers are the ones with a hand in the design and
development process. A full service teams familiarity
with the ins-and-outs of the tool means that the time
it takes to execute campaigns is minimized. Can You:
Vendor Questions

(1) Does your tool provide me with real-world, 1 Define segments based on proprietary data (such as Customer ID

business-relevant targeting options? numbers for a loyalty program) or demographic data (such as average
income by ZIP code)?
Why It Matters: Cookie-cutter approaches to
targeting can look great in product demos, but they 2 Geotarget based on proximity to your stores or those of competitors?
reveal their weaknesses when clients need to put
the pedal to the pavement. The way you understand 3 Segment visitors based on device, such as the iPad and Kindle Fire?
your business can differ significantly from the way
that vendors design their tools. You want a tool that
can adapt to you, not one that forces you to adapt
to it. Even more importantcan you act on these segments
in relevant ways, such as by making the website
finger friendly for tablet users?

27
 Channel Managers
Introspective Questions

(1) Do we have sufficient resources not just to run campaigns, but

also to conduct actionable analysis and drive continuous improvement?

Why It Matters: At a tactical level, testing and targeting involve

much more than running the campaigns themselves. The full spectrum
of responsibilities includes iterating three steps:

Campaign 1 strategy

2 Campaign creation and execution

3 Performance measurement and analysis

And, of course, performing each of these steps requires a certain degree

of testing expertise. Its a full-time job, and if you lack the resources
or the expertise to do it, your experience with testing may be slower,
more painful, and less impressive in terms of the impact on revenue,
conversions, AOV, and conversion rate.

28
Therefore, as you evaluate potential
vendors, pay closest attention to:
Ease of use: Do you have the resources
to perform these tasks and to manage
them internally?
Expert, full-service management:
Do you lack the resources and need others
to provide them?
(2) Can you echo my off-site creative
messaging throughout my website or only
on my landing pages?
Why It Matters: Stated earlier,
conversion optimization doesnt take place
in a vacuum. Your customers follow scent
trails that begin prior to the click (i.e.,
in emails, search engine results pages,
etc.) and rely on you to preserve those
scent trails throughout the entire shopping
experience. The goal is curation. Many
testing tools can only deliver single-page
experiences limited to the visitors landing
page. As soon as the visitor leaves the
landing page, the buying state is broken
because the messaging, offers, look,
and feel of the acquisition campaign that
attracted the initial click are gone.
Look for vendors that deliver the ability to
run multi-page campaigns that can span the
entire website. By delivering a consistent
shopping experience, from before the
click through to checkout, visitors stay in
a ready-to-buy state and are more likely to
become your newest customers.
In addition, multi-page campaigns indicate
the ability to repeat actions based on
simple, yet powerful, business rules
another important feature discussed in this
guide.
Vendor Questions
(1) Is the tool easy enough to handle
that users spread across multiple business
functions can use it, or is it designed for
centralized ownership and use?
Why It Matters: In the case of code-
heavy solutions, an IT specialist might
lead the charge. That kind of centralized
ownership model neglects the reality
that revenue is driven by performance
across multiple channels. Centralized
ownership is a problem particularly for the
enterprise-level company, whose customer
experience extends across email, search,
affiliate, advertising, merchandising,
and more.
No one group or person can
understand what it would take
to lift conversions in each of
those varied channels.
But when multiple users from those
business functions work with a testing tool,
each of those channel experts can drive
margins by optimizing their respective area
of expertise. That promotes growth across
your entire business.
29
 As a result, youll want to be confidentespecially if you opt for self-
service managementthat you can use the tool efficiently. Pay close
attention to:

1 The extent to which the tools UI is shown during the
demo process

Screenshots provided in RFP responses (if you opt

2
to go this route)

Testimonials that speak to the tools ease of use

3


4 The entire workflow, from campaign creation
to deployment

To maximize the likelihood that each visitor converts,

you need the ability to act on behaviors as they occur
in real time.

30
(2) Does the tool update visitor segments
in real time?
Why It Matters: The ability to define
segments based on multiple attributes,
such as:
Visitors from paid search
Located in California
Using an iPad
Who, yesterday,viewed products
in the Lawnmowers category
is important and powerful. But this
represents just a fraction of the information
you need in order to deliver the best
shopping experience possible. For many
tools, a visitors segment is fixed at the
start of the session (or worse, at the start two things more than any other
of the day). But good information has an
represent your silver bullet:
expiration date.
If your tool cant act on in- 1
Multi-page campaigns
session behaviors, then by the
time you can act, it may be too 2
Real-time intelligence
late. Those visitors are long
gone, and theyre not coming
back!
Acting on in-session behavior gives you the
ability to say, When someone views this
page, show this offer or When a visitor
places $100 of items in the shopping cart,
automatically apply this coupon code.
The bottom line is, if channel managers are
struggling to improve conversion rates and
feel powerless to influence the shopping
experience post-click,
31
 IT Director
Introspective Questions

(1) Whats the infrastructure of my companys website? Is it made up

of many integrated systems, or is it mostly flat HTML?

Why It Matters: Look for two attributes:

Catalog Intelligence: For retailers, the product catalog is the

nerve-center of the website. Without an understanding of brands,
categories, and SKUs, merchandising campaigns lack scalability.
The value of catalog integration is the scalability it lends to the
campaign creation process. Simple business rules allow the
marketer to say, Perform this action on all pages that meet these
user-defined attributes. That means less of a headache for you.

Knowledge of the site map: In addition to the product

catalog, knowledge of the sites page types and page structure
should be every bit as high on the IT checklist. When a tool
knows what a Category Page is, and the blocks of code that

32
comprise the sections of that page, campaign creation becomes
incredibly simple.
Most importantly, it means that the Marketing
department no longer bothers IT. Writing
custom throwaway code is a thing of the
past. Regular expressionsyou dont need
those either. Theyre replaced by simple,
yet powerful, business rules set up by the
marketer directly in the tools UI.
(2) Will we be replatforming within the next
one to two years?
Why It Matters (or Doesnt
Matter): In the past, replatforming projects
spelled the end of web optimization for
extended periods of time, sometimes years.
And historically, this occurred for necessary
reasons: Engineers cant hit a moving target,
so companies reached a point during the
replatforming process where they must lock
the website down, stopping all optimization in
its tracks.
Times have changed, though. The latest
generation of testing and targeting solutions
utilizes a cloud-based, client-side architecture.
These tools serve campaign content directly
from the cloud, where it renders in the users
browser in real time.
This architecture makes client-side solutions
platform agnostic. Regardless of whether
youre in the middle of a replatforming
projector about to embark on oneyou
can continue optimizing the online shopping
experience with this type of tool.
Vendor Questions
(1) How easy is the tool to implement?
Why It Matters: Ease of use and ease
of implementation are two different things,
and the latter has two dimensions:
1 Ease of initial setup (also known as
integration)
2 Extent of IT involvement in campaign
creation and deployment
Both dimensions revolve around the same
topic, though: tagging. Some vendors make
the one tag only promise and fail to make
clear that this refers to initial setup only.
Individual tests still require campaign-
specific code, which drastically slows down
the campaign creation process and places
an unnecessary (and often unpredictable)
burden on IT.
True one-tag implementation
refers to both initial setup and ongoing
campaign creation, so individual campaigns
never require additional code for any purpose.
This one tag:
1 Looks for or sets a first-party cookie
2 Collects data about the visit and visitor
(no PII or PCI is tracked)
3 Evaluates data against active
campaign rules
4 Modifies the browser content, and
records the visitor interactions
Lastly, look beyond a vendors IT-free
message in favor of IT-friendliness. Good
characteristics include:
1 Use of an asynchronous JavaScript
tag for faster, risk-free loading
2 Serving only your test content, as
opposed to both test and control
3 Strict monitoring controls, with the
ability to automatically kill tags that do
not finish processing within a defined
33
time frame
 Web Analyst
Introspective Questions

(1) How easily can we demonstrate the impact of our companys

online marketing efforts?

Why It Matters: You cant improve upon what you dont know or
cant easily understand. Testing is only as good as the insights you take
away, and the three building blocks to great insights are:

Extent of Data: All things being equal, the more data you
have, the better. Pay close attention to the metrics and KPIs that
are tracked out-of-the-box for all clients.

Relevance of Data: Out-of-the-box metrics are important, but

only get you so far. Look for tools that let you customize metrics,
as well as those that let you define segments around proprietary
data. In addition, you want to be able to view campaign-specific
data, as well as create reports whose data spans multiple
campaigns based on the attributes you choose.

Ease of Understanding: Look for tools that include

visualizations of campaign performance in addition to just raw

34
data. Performance visualizations make data easily digestible and
also help the marketer understand what they should care about
in the first place.

Vendor Questions
(1) Can the tool incorporate my complete
product catalog and put full product data at
my fingertips?
Why It Matters: The web analysts
job is to identify trends in data, and then
translate those trends into strategies that
improve performance. But, once again, you
cant improve what you cant act on. The
more granular your tests, the more relevant
the experiences that you can deliver. As
with other stakeholders, knowledge of
the product catalog and site map should
be extremely important to youbut not
because they make campaign creation
easy.
Rather, catalog and site map intelligence
mean that marketers can produce
business-relevant campaigns that match
the hierarchies, structure, and logic of the
website. And that means performance
reports that are hyper-relevant and can
you should also
be concerned with your ability to glean
new insights about customers that your
analytics system cant provide. Key points
to focus on are:
inform tactics for improving existing
Does the tool provide
campaigns and strategies for producing
visualizations of test
new ones.
performance?
(2) Does the tools reporting and analytics
Is statistical significance
capabilities add meaningful insights to what
provided at the test level, or for
I already know?
each KPI?
Why It Matters: When it comes to
If significance hasnt been
reporting, many tools emphasize their
achieved, does it estimate how
integration with third-party systems such
long the test must need to run?
as SiteCatalyst, CoreMetrics, or Google
Analytics. To be clear, these integrations
Can I define custom KPIs to
are important because they facilitate one-
report on what matters to my
stop shopping for reports, which can save
business?
web analysts significant amounts
of time.
Can I easily create reports not
But wait
just on individual tests, but on
segments, actions, and
attributes that span multiple tests?
35
 Vendor Profiles

This Guide is meant for a specific audience:

the enterprise. As a result, some vendors have been excluded due to
specialization in the needs of small- and medium-sized business.

Still, not every profiled vendor is recommended. Just as Ive repeatedly

criticized companies that put design decisions in the hands of engineers,
so too do I criticize testing tools that are built for audiences other than the
ultimate end user: the marketer.

And just as tests have winners and losers, there are clear standouts among
the competition in terms of capabilities, scalability, and ease of use.

Introductory Tools

The vendors profiled are of three types, and include: Landing Page Tools

36
Tools for the Enterprise
 1 2 3
Introductory Tools: Landing Page Tools Tools for the Enterprise

For each vendor, Ive also highlighted a sample A/B test

implemented using the tool (sources: WhichTestWon.com
and Vendor Websites). Read on to see who you should
consider, and not consider.
37
 Introductory Tools
Google Content Experiments

Googles June 2012 announcement that Website Optimizer will no

longer be available as a standalone product caught the website
optimization landscape by storm.

However, the new Content Experiments is not so much a departure

from the core capabilities of Website Optimizer, as it is an integration
with Google Analytics. And to be clear, a more seamless integration
between a tool that delivers experiments, and a tool that measures the
results of them is an important watershed for the tens of thousands who
will undoubtedly use both of these systems.

A Strong Introduction to
Benefits of the integration include:
Testing for Novice Users.

1 Use of existing Google Analytics tags to measure

campaign outcomes;

38 2 Easier optimization of existing goals; and

3 Better segmentation of performance data.

Look under the hood, however, and youll still find some of the more frustrating
aspects of the old Google Website Optimizer:
A
Separate Pages for Separate Tests
Structural Page Changes are Required
No Preconfigured Targeting

Still, Google Content Experiments is a great way to make your first forays into
testing. The real value, however, of Google Content Experiments is that it further
elevates the importance and stature of website testing.

In other words, it helps shift the perception that testing is expensiveand

something that only deep-pocketed companies can afford. Rather, website testing
is investmentand one that Google has fortunately made accessible to everyone.
B
While you cant argue with free, remember that your employees come with
significant expense, and that the opportunity cost of a resource-intensive tool such
as Google Content Experiments could potentially outweigh its benefitsand put
you off of testing altogether.

If youre new to the world of website testing, or arent sure where to start, the new
Google Content Experiments definitely is a low-risk place to start. And if you find
yourself graduating to more advanced needs later on, this Guide will help you
determine the vendor that is right for you.

Clients (Note: Google Website Optimizer) Sample Test (Note: Google Website Optimizer)
Cabot Heritage Corporation Site: RegOnline.com

Test: Employee Pictures

39
RJMetrics
Winner: Version A (36% lift in free account sign-ups)

New York Public Library
 Optimizely is among the newest class of testing tools designed to unclog
the IT bottleneck that typically interrupts innovation in ecommerce
environments. With a point-and-click visual editor, Optimizely makes it
possible to set up and analyze multi-page A/B tests without knowing any
HTML.

Websites appear within Optimizelys interface just as they would while

surfing the web. To run an A/B test, for instance, just hover over the page
until a red box appears around the element youd like to test. Select it
and create your variations. It really is quite easy.

Optimizelys implementation is simple as well: Drop a line of JavaScript

code into the header of the page and youre ready to go. As soon as the
experiment begins, Optimizely reports real-time data on a dashboard,
tracking core metrics like conversion and engagement.

Optimizely can optimize the entire website experience, and has earned
extra points from a functional standpoint, with targeting capabilities that
aim tailor-made experiments at specific visitor segments.

Look closely, however, and youll quickly find several important

40
weaknesses. For one,advanced campaigns require extensive knowledge
of jQuery. The problem with this requirement has several dimensions:
 A
1. Because Optimizely is a self-service solution only, the requirement of jQuery
development falls mainly on you. (Note that Optimizelys highest subscription
tier does include some custom coding.)

2. If your company has chosen to use a JavaScript framework other than jQuery,
your developers may find it difficult to use Optimizely.

Among Optimizelys stated goals is to make a serious dent in Adobes dominance

of the testing space, but for reasons related to service and overall delivery,
Optimizely is a bandage on Test&Target rather than an outright replacement.
B
For example, in instances where you find both Test&Target and Optimizely
deployed on the same website, Optimizely is often a departmental (rather than
organizational) purchase designed to alleviate Test&Target pain points (e.g.,
production site changes). However, Optimizely lacks Adobes enterprise delivery
model, which wraps campaign development, strategy, and reporting into full
service management.

However, the companys recently closed Series B round of venture financing

(of an undisclosed sum) may help them narrow the gap between the needs of
Test&Targets enterprise customers, and the capabilities Optimizely can currently
offer.

Clients Sample Optimizely Test

Starbucks Site: DNC Facebook Page
Test: Calls-to-Action

41
JibJab
Winner: Option B lifted invitations to Facebook
friends to use the app between 4.5%
Change.org
and 19.2% (depending on the element)
 Landing Page Tools
Visitors convert at higher rates when theyre led to
a page that aligns with the referring source, whether
thats an ad or email link.

The hallmark of landing page tools, though, is their ability not just to
conduct tests, but to allow the marketer to create and deploy landing
pages independently of their website or ecommerce platform. Its
this capability that lends agility to the enterprise retail marketer.

A cloud-based landing page tool for the enterprise.

Web-based LiveBall focuses on landing pages arrived at via email

marketing, banners, mobile and paid search. But this A/B and multivariate
testing tool, created by online marketing and software development firm

42
Ion Interactive, adds targeted marketing to the post-click optimization
mix, giving clients the chance to provide content according to visitor
behavior, geography and other demographics.
With a drag-and-drop, WYSIWYG
editor, you can design and test landing
pages, plus microsites and conversion
paths, without ever knocking on ITs
door. And its framework is a boon
for brand managers in particular,
who can decide which elements, like
messaging, can be edited and which
must be locked in for the sake of
brand consistency, like logos, fonts
and brand colors.
LiveBalls made for
marketers by marketers
reporting includes a global dashboard
that displays both basic and more
granular performance metrics. Thirty-
day snapshots deliver real-time data
on core statistics like traffic and
bounce and conversion rates.
Deeper-dive information is a click
away: A performance view, with
user-configurable gauges, allow
personalized glimpses into the
effectiveness of your media buys,
forms, segmentation, messaging and
user behaviors.
With the ability to easily create and
launch large numbers of landing
pages, and to target these pages A
to various key audience segments
and offsite campaigns, LiveBall
delivers on ease of setup, usability,
scalability, and message consistency.
Ion Interactive also offers several
supports tiers - from self-serve to full
service - making it a viable choice for
companies large and small, with a
range of support needs.
B
Example Clients:
American Greetings
DHL
Citrix
Sample LiveBall Test
Site: TimeTrade.com
Test: Image Descriptors
(Enterprise vs. Corporate)
Winner: Version A (42% increase in leads)
43
 With WYSIWYG simplicity, Unbounce eliminates the HTML
headache.

To squeeze every drop of conversion potential out of your landing

pages, Unbounce is a clear winner. Maybe youve been lured before by
solutions that promise to be dead simplethen fail to deliver. But with
its point-and-click, design and editing tools, Unbounce is as headache-
free as its cracked up to be.

Among the newer online marketing tools to have sprung onto the
scene, the self-serve, fully hosted solution, which requires no software
downloads, provides templates on which technically challenged
marketers can build landing pages without HTML. Users can then run
A/B split tests to optimize pages on which visitors arrive via search
results, Google AdWords or email links.

Across the board, Unbounce gets high marks for

customer service, and its team is committed to
updating the tool regularly.

44
 A

Unbounce stays laser focused on landing pages. In

fact, on its site, the company liberally cites a statistic
which claims that promotion-specific, standalone
landing pages (as opposed to sending paid search
visitors to a homepage) can improve conversion
rates by 25 percent. In other words, Unbounce B
wont optimize your entire website, nor does it offer
targeted testing. But its user-friendly interface and
traffic-based pricing makes it an excellent choice for
companies that want to kick their landing-page results
up a notch without the complications of coding.

Example Clients:

Rackfish

ReTargeter Sample Unbounce Test

MailChimp Site: WorldClassDriving.com
Test: Hero Captions (Tagline vs. Literal

45
Description of Service)
Winner: Version A (34% increase in leads)
 Tools for the
Enterprise
Marketers at enterprise-level companies face unique
obstacles to effective testing and optimization,
including layered corporate structures, overextended
IT departments, and divided opinion.

Working on large-scale projects as part of an equally large groupand

charged with maintaining strongly enforced brand guidelinesthey need
a tool whose power, ease and speed surmount those challenges. The
tools in this section are built specifically for the enterprise.

Adobe Test&Target

46
Part of the Adobe Online Marketing Suite, this legacy tool offers a robust
palette of services, but requires heavy IT involvement to both implement
and use on an ongoing basis.
Test&Targetone of the longest-running
in the marketis a popular option for
enterprise-level companies with complex
conversion processes, high volumes of traffic
from varied sources and a budget with dollars
earmarked for testing. It offers a vast palette
of functions, including the ability to execute
A/B and multivariate tests, measure the
effectiveness of content across varied online Problem 1
channels, generate recommendations, and
target and personalize campaigns according
to visitor behavior, geography, traffic source

With Test&Target, however,

and more.

problems are twofold:

As part of the Adobe Online Marketing
Suite of applications for online business
optimization, Test&Target features a visual

1
interface through which you can create and
implement tests and targeted campaigns
directly on your websitejust as visitors will
see themwithout any special configuration.
You can also create comprehensive,
anonymous views of visitor interactions,
and then use these views to deliver relevant
content to targeted visitor segments.
Despite often being marketed as a one-tag solution, Test&Target
actually requires marketers to place code around each and every
element theyd like to test (what Test&Target calls an Mbox) as
well as implement separate conversion tags. In many ways, this
is similar to the type of integration Google Website Optimizer
requires and is, in effect, the products Achilles heel.

47
 To avoid excessive server calls,
and the impact this has on both page load time as well as the customers
cost, Test&Target recently made available what it calls a global Mbox,
which can load multiple page elements via a single server call.

But while the global Mbox can substitute JavaScript and

CSS, the client must still:

Create 1 the alternate content themselves

2 Write the script that loads the alternate content

3 Put that code in the global Mbox

In addition, this is throwaway code

usable for no other purpose anywhere else on the website. And because
you can only have one global Mbox per page, clients lose the ability to
target that page to different audiences.

48
In general, though, the level of
coding expertise required to use
Test&Target renders the product
unusable for most enterprise
teams.
Its just too complex and requires too much
support from IT, meaning tests often cannot
2
be deployed with the speed and agility
marketers need. Although Test&Target has
both self-serve and full-service support
offerings, most companies will find they
need a large degree of support (either from
Adobe itself or through one of Test&Targets
Certified Experts) to use the tool to its full
potential.
Problem 2
Test&Targets fee structure discourages marketers from ongoing,
habitual experimentation and optimization. Its sticker price is
merely the price of entry, since Test&Target charges per server
call: When visitors arrive on your website, the system calls Adobe
Test&Targets server to retrieve a designated campaign. With each
server call, the proverbial cash register rings. Naturally, marketers
want to keep that register as quiet as possible. Which means theyre
far less likely to test continuallyand, ultimately, effectively.
Whats more is that once Test&Target helps you
identify winning variations, youll need additional
technical resources to code them permanently so
that you avoid the cost of repeated server calls.
49
 All things considered, Test&Target can be a highly effective tool for
companies with large budgets and extensive resources to design and
implement tests. Most companies, though, may determine that the
high cost of ownershipcombined with IT-intensive implementation
requirementsmake other tools more suitable.

Example Clients:

Dell

Readers Digest

CNET

50
A B

Sample Test&Target Test

Site: Dell.com
Test: Navigation vs. Promos at Page Top
Winner: Version A (19% lift in Conversion Rate)
51
 Testing solution from an infrastructure software provider.

Autonomy Optimosts features hit the requisites of a solid performer.

Users work within an easy-to-navigate user interface, building tests free
of HTML or CSS. The tool can test unlimited content combinations
including Flash, JavaScript, AJAX, video, and audio.

With its drag-and-drop interface, clients quickly create,

launch, and optimize landing pages to maximize email,
search, and advertising campaign ROI.

To deploy the tool, users place a single snippet of code on the test page
and access reporting on customizable dashboards.

52
Its adaptive targeting function creates
customer segments according to context
(visitors arriving via search versus a
referring website, for instance), geography,
time of day, and consumer profiles. Based
on patterns and trends, like keyword clusters
or referring websites, Optimost tests and
delivers targeted content variations. If a
retailer has a large number of visitors from
websites associated with new moms, for
instance, Optimost launches offers targeted
to that group specifically.
Forrester gave Autonomy Optimost high
marks in its August 2010 review of Online
Testing solutions, designating it a leader
in the testing sphere, along with Adobe
Test&Target. In particular, the report called
out Optimost for quality administration,
deployment options and breadth of testing
techniques.
Optimost is particularly good for teams for
whom theres no such thing as too much
data. The products reporting is best in
class and offers unparalleled amounts of
data. That said, Optimist is better suited
for highly technical, statistically oriented
teams (in other words, where the budget
and responsibility for testing belongs to
an organizations engineering or customer
intelligence teams). For the average
marketer, though, Optimosts interface
and reports may be difficult to use and
understandespecially without the the
tools full-service consulting.
In the end, whether or not to consider
Optimist as your testing solution begins
with one (1) fundamental question:
Who will need to use it? Example Clients:
If your users are technically savvy and have Sky Italia
a competence for statistics, Optimist may be
a near-perfect addition to your teams suite Freshfields Bruckhaus
of products. But if your marketers are in the Deringer
drivers seat (or at least, you want them to
be and they should be), look elsewhere at John Lewis
one of the other enterprise-level solutions
featured.
Note: WhichTestWon.com has no
sample test from Autonomy Optimost.
53
 A leader in the European enterprise market.

According to online magazine VentureBeat,

Maxymiser is one of the top three website
optimization providers in Europe,

having been among the first to introduce optimization to the U.K. in

2006. With an enterprise-level client roster, the London-based company
established its first of two U.S. offices in 2009.

Maxymiser, which promotes its multichannel marketing services

alongside its testing application, offers A/B and multivariate testing,
as well as segment targeting and personalization solutions, with the
capacity to handle dynamic content. Its per-test technical time is minimal,
as implementation requires a single line of JavaScript code inserted into
your website template once, and performance data is available via real-

54
time, self-serve reporting dashboards.
 A

Marketers manage and deploy Maxymisers one-tag setup and

tests from Maxymisers OneTouch workflow approach combine to deliver B
Platform, which includes three ease of implementation, scalability,
components: First is the web-based and a degree of managed controls that
on-page console, from which enterprise-level clients will like. The
marketers manage JavaScript companys recent $12 million round
functions libraries and execute tests in venture financing may help address
on any page. Clients manage the its other issues, but Maxymiser may
workflow and version control of still be best suited for clients that feel
changes in the Publish Center, which confident enough with testing to use
is similar to a web CMS system, while its self-serve support model.
a Global Content Delivery Network
enables the high-speed delivery of
test and campaign content.
Example Clients:

However, there are whispers of
Teleflora
discontent from some Maxymiser
customers weve talked to. Customers
speak of the lengthy process involved
Hertz Sample Maxymiser Test
in getting campaigns up and running.
Lufthansa Site: ViewLondon.co.uk
In addition, weve seen evidence
Test: Checkout Page Template
of web users complaining about
Winner: Version B (23% lift in Conversion Rate)
Maxymisers synchronous JavaScript

55
tag failing to load properly, thereby
preventing pages from loading.
 Designed for the age of agile commerce, Monetate raises
the bar.

Since its launch in 2008, Monetate has become the

gold standard for enterprise-level optimization.

Its five-product Agility Suite raises the bar for test performance,
streamlining the optimization process dramaticallyeven as it arms
enterprise marketers with the most sophisticated testing and targeting
capacities available. For full-service engagements, Monetate couples
its breakthrough technology with a strategic service team that works in
close partnership with clients.

Through a single asynchronous JavaScript tag (known as ExpressTag,

and inserted only once), Monetate presents built-in analytics, compiles a
complete catalog of products, serves up custom, targeted content in real-

56
time, and feeds offers and promotions to back-end systems.
Monetate also recognizes
that the key to winning the
conversion game lies in the
agility with which a business
can create consistency and
relevance across consumer
touchpoints.
Forrester Research calls this concept,
agile commerce, and Monetate leads
that charge in a number of ways.
Its echo capacities, for instance,
ensure that consumers who click
on an email or PPC ad are led to
pages that present complementary
messages and visuals throughout
the websitea key best practice
in website optimization. Agility, of
course, is also a matter of speed
and ease. While other solutions in
this guide perform targeted testing,
Monetate takes it one step further,
with rules-based product badging by
dynamically rendering the badge on
top of the retailers existing image
thumbnails. That means the retailer
never has to modify anything in the
image catalog.
Interested in more than just testing
and targeting? Enterprises will find
Monetates future proof solution
especially attractive. ExpressTag is
the single point-of-integration for the
entire Agility Suite, making it as easy Even the companys price structure
for companies to use all five of the supports the position that agility is
companys products as it is just one. critical.
Monetate clients pay a fixed, flat
monthly fee and can run an
unlimited number of campaigns.
57
 Combining ease of implementation, a high degree of usability, the ability
to create and deploy large numbers of tests quickly, and the functionality
to echo offsite messaging throughout the website, Monetate is a powerful
solution for organizations both large and small. In addition, Monetate
recently closed a $15 million Series B financing round, setting up the
company for continued expansion of its already-impressive offerings and
feature set.

Example Clients:

BestBuy

QVC

LOccitane

58 Initial Email:
 Landing Page Checkout Page
A

Landing Page

Sample Monetate Test

Site: CasualMale.com
Test: Email Echo (all site pages)
Winner:

Version B (14% increase in
revenue per session) 59
 An external solution that swaps content on the fly.

When SiteSpect refers to itself as a non-intrusive solution, it separates

its offering from both JavaScript-based testing tools and server-side,
software plug-in applications.

As an external solution, SiteSpect operates as a go-

betweena kind of filter between the website youre
testing and the visitors who see it.

Thats where all the testing activity happens, including content swapping,
user behavior tracking, targeting, and reporting. Within its web-based,
WYSIWYG editor, the SiteSpect Control Panel features a wizard that
guides your A/B, multivariate, and targeted test design, helping you select
test-worthy site elements, behavioral metrics, and campaign options.
It can optimize any type of contentincluding dynamic content, AJAX,
Flash, CSS, and mobile.

60
Instead of inserting JavaScript or tagging
pages to activate a campaign, SiteSpect
steps in when you activate a test,
creating content variations on the fly and
serving them up to the appropriate test
visitors without changing your website. It
creates those variations after the content
leaves your websitespecifically, as it
passes through SiteSpectand before
its delivered to viewers browsers.
SiteSpect implementation comes in
two flavors. First, the Enterprise, or
hardware-based, solution, which your IT
team will need to install within your data
center. SiteSpect recommends this first
option to larger websites with specific
security and privacy requirements or
site administrators with multiple and
complex sites.
The unavoidable reality of working
with a non-intrusive external solution,
however, is that it creates a single point
of potential failure, a consideration
that youll want to weigh carefully. To
eliminate the hardware hassle, you can
opt for a hosted, ASP solution. In that
case, youll need to update your DNS,
ensuring that test users are directed to
the test page and the rest of your traffic
to your website.
A
SiteSpect tends to be favored by
companies whose IT departments
control the website optimization process.
In part, thats because SiteSpect
whose Enterprise hardware needs
to be installed and maintainedis not,
in the end, as IT-free as the company
purports, since technical experts will
need to install and maintain the device.
In all, integrating SiteSpect with your
website requires more effort than the B
company leads prospects to believe
and, likely, more effort than the one-
tag solution offered by many of the
JavaScript-based providers. But despite
its not so IT-free implementation, the
tools ease of use, intuitive workflow,
and robustness of testing options make
it a solution worth considering.
Sample SiteSpect Test
Example Clients:
Site: SkinnerInc.com
Test: Image Size vs. Above/Below-
Walmart the-Fold Description
Winner: Version B (63% increase
TigerDirect.com visitors entering bids)
61
Vehix.com
 Recommended
Tools

As I mentioned earlier, in testing, there are winners

and losers. In the section above, a total of eight out of
perhaps 40+ tools made the cut of being included in
this Buyers Guide.

But this Buyers Guide doesnt end with vendor profiles. It isnt about
endless facts, unfounded praise, or checking off boxes in feature matrices.
Its about the questions that enterprise retailers should ask during the
buying process. Its about the best practices in testing and targeting, and
how vendors deliver on the needs of the modern enterprise retailer.

62
Out of the eight vendors profiled, only three earn my recommendation:

Introductory Tools:

Landing Page Tools

Tools for the Enterprise

63
 Introductory Tools

Google Content Experiments

Its not difficult to recommend Google Content

Experiments. After all, free is free.

However, while the tool may be free, your people are not. If you have
extensive IT resources, but little experience testing, you cant go wrong
by testing the waters with GCE, and companies can learn a great deal
about simple, yet fundamental, changes they can make to the web
experience in order to maximize conversion rates.

Those who need help can generally find excellent assistance (ranging
from basic consultation to full-blown implementation) from a Google
Authorized Consultant. But Google Content Experiments isnt for
everyone. If you dont have the IT resources to use GCE to its fullest
potential, look elsewhere.

64
For most companies, use of Google Content Experiments
will be (as it was with Website Optimizer) temporary, and
is intended to be so from the very beginning. Core features
that are missing include:

Multivariate testing
The ability to push winning tests to 100% of eligible traffic
Preconfigured targeting (e.g., New vs. Returning)
Maximum of 12 active tests per account profile

But for the enterprise retailer, this get your feet wet

Overall:
testing solution can serve you well as you learn the basics
of testing. And for the right company with the right set
of resources -- combined with its free price tag, its an
incredibly powerful tool to have at ones disposal. Indeed,
the likelihood of a positive return on investment is higher
than for most other tools.
No cost
Authors Sidenote: But the question thats likely on your
mind is, Why not Optimizely? Ive often recommended to Likes
get good at free, and then pay. And a key component of
this process is to learn what your organization really wants
to accomplish, and what it will need to get there. For some, Out-of-the-box targeting options,
that decision will lead to the easier-to-use Optimizely. Heavy coding/IT involvement
But for many, a comprehensive survey of goals will lead
Dislikes
companies to one of the enterprise vendors listed in this
Guide. In other words, while its true that few drivers can
afford to jump from entry model cars to luxury lines, the Companies new to testing, or
jump from Google Content Experiments to an enterprise those with extensive IT resources

65
solution is one they cant afford not to make.
Good For
 Landing Page Tools

Companies deciding between LiveBall and

Unbounce will likely see LiveBalls edge in terms
of features and functionalities (for example, MVT
and segmentation). The tradeoff, though, is price
with Unbounce coming in as the significantly less
expensive tool to use.

I recommend Unbounce over LiveBall primarily for the formers ease

of use and considerably lower cost. The simplicity and ease of use of
Unbounces UI cannot be overstated. We spoke to users whose email
and lead generation tools include the ability to produce landing pages,
but who use Unbounce solely on account of its ease of use. Combine
that with the ability to run A/B tests without involving IT, and youll be

66
wishing you knew about Unbounce a long time ago.
Still, Unbounce isnt for everyone. Because
its limited to single-page tests, Unbounce is
only appropriate where the conversion event
takes place on the landing page. Irrelevant
to the retailer, you say? Not quitefor use in
email campaigns (such as for a replenishment
offer) and with non-retail conversion events
(such as a newsletter sign-up), Unbounce is
a valuable tool for any enterprise retailer to
have. Overall:
Additional Note: Among landing page
tools, it may be safe to discount the value
of multivariate testing and segmentation
that LiveBall brings to the table. For the Price, User interface, Landing page creation
majority of landing page tests Ive seen, the combined with A/B testing
amount of traffic simply isnt high enough
Likes
for a multivariate test to achieve statistical
significance in a timely manner (two weeks
or less). If MVT is a business requirement
Single-page tests only
(along with segmentation), you may be better
off considering an enterprise tool that also
delivers the ability to run multi-page tests. Dislikes
However, if targeted, multivariate landing
page tests are appropriate for your business,
Companies with testing needs for single,
you can accomplish a lot with LiveBall.
purpose-built landing pages

Good For 67
 Tools for the
Enterprise

Among testing and targeting solutions for the

enterprise, Monetate stands first in line among the
preferred solutions of retailers.

Indeed, Monetate has set new bars for capabilities, ease of implementation,
ease of use, and practical value to the retailer. Among its best attributes
are catalog integration, the ability to run multipage campaigns, dynamic
product badging and text insertion, and real-time weather targeting.

As this Guide has tried to illustrate, though, selecting the right tool involves
a lot more than checking boxes in a feature war. Here again, though,
Monetate outperforms the competition. Its full-service management team
averages more than 10 years of experience each, so whether you lack

68
expertise in testing or the bodies to test, you wont lack the ability to use
this tool.
And most importantly, Monetate scores big on
ease of use. Its current client base averages
more than 52 simultaneous campaigns
each, with a high of more than 200. But you
dont have to take their word for it. The most
noticeable way that Monetate sets itself apart
from its competitors is the lengths it goes to
actually show prospects inside the tool. The
website is replete with screenshots of the UI,

Overall:
videos that show campaign creation in action,
and real customer examples.

Monetates pricing model is a fixed, flat

monthly fee, so you can test as much as you
Scalability, ease of use, relevance of
want. It also means that you can push winning
features to online retailers, service expertise,
variants to 100% of site traffic for no additional
multi-page campaigns turn simple tests into
or incremental cost. Smaller retailers may find
full-blown experiences
Monetate to be on the expensive side, but Likes
remember that this solution, more than any
other, offers the likelihood of a strong return
on investment. And ROInot priceshould
High price of entry
be your benchmark for selecting a testing and
targeting tool.
Dislikes

Im not biased because Im a Monetate

Enterprise companies with extensive product
advisor. Im a Monetate advisor because Im
or asset catalogs; companies thinking about,
biased. Watch them, and youll quickly agree

69
or engaged in, replatforming
that Monetates built for marketers claim is
Good For
packed with real substance.
 How Can
Eisenberg Brothers
& Associates
Help You?
70
Bryan Eisenberg is a partner in Eisenberg
Brothers & Associates; they
collaborated with, coached, mentored and
trained companies and their executives
in digital marketing since 1996.
Eisenberg Brothers & Associates
have successfully worked with companies
like Overstock, Dell, Orvis, LowerMyBills,
WebEx, PriceWaterHouseCoopers, HP, GE
Healthcare, Travelocity, 1-800-FLOWERS,
Acceler, CafePress, ICE, Forbes, HH Brown,
QVC and dozens of other multi-channel
retailers, lead generation businesses and
pure-play Internet companies. While most
Eisenberg Brothers & Associates clients
are large companies they also work with
promising start-ups like Bazaarvoice, Runa,
BoostCTR & Monetate.
have
Here is a sample of the challenges Eisenberg
Brothers & Associates clients have resolved:
The
team
Creating innovative strategies and optimization frameworks to drive
continuous improvement both quantitatively and qualitatively Creating
company-wide initiatives to improve revenue generating customer-centricity
Developing initiatives to glean social media data to be used in optimization
and creation of creative campaigns and content development
Reconciling user experience, testing, persona and web analytics-based
approaches to understanding and optimizing customer behavior

Eisenberg Brothers & Associates offer

exceptional depth on topics like web analytics,
conversion optimization, persuasive
strategies for online lead generation, persona
development, persona marketing, social Eisenberg Brothers & Associates custom tailor
commerce, online publishing and advertising,
and design new approaches to meet your needs.
persuasive strategies for e-commerce and

71
persuasive online copywriting. They can be contacted via their blog: http://www.BryanEisenberg.com
 Appendix:
Sample RFP Questions

In my experience, most vendor RFPs get it all wrong.

The worst ask questions that make little to no sense
whatsoever. The best still fail to understand which
questions about, and attributes of, a tool are most
important.

If youre undecided in the buying process, or your corporate bylaws

simply require you to do RFPs, the questions below are a great place
to start. They speak directly to the best practices in the Guide and will
reveal a lot about a vendors strengths, weaknesses, capabilities, and
competencies.

I welcome and invite you to use any or all of these

72
questions word-for-word, or as you see fit.
Infrastructure and Integration
1. On average, what is the length of time
between when a contract is signed and
when clients launch their first campaign?
Attach a sample project plan and timeline.
2. Describe your integrations with email
service providers (ESPs), paid search
APIs, and third-party analytics.
Capabilities and Benefits
8. What distinguishes your tool from its
competitors?
9. Describe the steps for creating a
multipage campaign (a) spanning a
portion of the site, such as a single brand
or category, and (b) spanning the entire
website.
16. Describe your 12 month roadmap.
What features and capabilities do you
intend to build into the tool?
17. Describe the process of previewing
campaign experience outside of
production. Can campaign previews be
shared with users who do not have formal
access (username and password) to your
system?

Ease of Use
3. On average, how many simultaneous
campaigns do your clients run? On
average, how many do the top five
customers run?
10. Does the tool update segments in
real time, based on in-session visitor
behavior or activity?
11.Describe the steps needed to badge a
product or video image thumbnail.
18. Describe your tools day parting
capabilities. Does the ability to day part
revolve around the visitors location,
company location, or both? Does the
ability to day part extend to times of day
as well as campaign start/end dates?

12. Does the tool offer business-relevant

4. Illustrate the steps for creating a
geo-targeting options, such as shipping
targeted testing campaign. Provide
zones, proximity to location, etc.?
screenshots.

13. Describe the tools ability to detect

5. Do individual campaigns require
whether a visitor is new to the website
campaign-specific or variable-specific
versus new to a campaign.
code to implement?

14. Does the tool support targeting by

6. Describe the typical user of the tool
custom and proprietary segmentation
(marketers, IT, creative, etc.).

73
data?
15. Can the tool deploy fully functional
7. Describe the process of pushing
site features, apps, and widgets?
winning test variations into production.
 Scalability
19. Does the tool offer true catalog
integration (either feed-based or via site
crawl)?
20. Does the tool support the scalable
use of creative assets via dynamic text
insertion?
21. Does the tool meter visitor sessions,
either because of price or infrastructure
capacities?
22. How is the tool architected (client-
side, server-side, or proxy)?
Security, Safety, and Risk
Management
23. If applicable, does your JavaScript
load synchronously or asynchronously?
24. Does your tool load only test/
campaign content, or both test/campaign
and control content?
25. Do you offer multiple access levels?
Describe
26. these levels and their capabilities.
74
27. Describe your tools collection of, or
requirements for access to, PII or PCI.
28. Describe your tools requirements
for access to any internal systems or
networks.
29. Does your tool log all user interactions
including, but not limited to, campaign
creation/changes/launches, user login
times, failed logins, report creation, etc?
30. How does the tool manage errors and
failed campaigns, such as when content
is available?
Pricing and Returns
31. Describe your pricing model. Is it
fixed/flat fee or variable by usage or
traffic?
31. How do you ensure our success with
the tool?
31. How do you interact with our people
and processes to get the most out of the
tool?
31. What is the typical amount of time it
takes one of your clients to see a return
on the cost of the contract of the tool?
used in the creation of reports?
Service Management
32. Do you offer self- and full-service
management? List the services and/or
tiers provided in each option, and provide
pricing details.
33. Provide a detailed list of the resources
you expect (client) to dedicate to this
project. Include roles, estimated time of
commitments, and whether commitments
are ongoing or one-time.
34. Describe your training process. Is
training provided on-demand? Unlimited?
What topics are covered?
Reporting and Analytics
35. What default metrics and KPIs does
your tool track and report on?
36. Can custom KPIs be tracked and
reported on?
37. Describe the tools reporting
capabilities. Does it provide visualizations
of campaign performance? The ability to
manually build ad-hoc reports? Support
with screenshots.
38. Can campaign targets/segments be
 2012 Bryan Eisenberg & Jeffrey Eisenberg & Associates