Three open source Web service testing tools get high marks http://www.infoworld.

com/print/28995

Published on InfoWorld (http://www.infoworld.com)

Home > Test Center > Architecture > Three open source Web service testing tools get... > Three
open source Web service testing tools get high marks

Three open source Web service testing tools

get high marks
By Rick Grehan
Created 2007-05-11 03:00AM

Thanks to the IT world's ongoing love affair with

Web services and the appearance of more and
more Web-service construction tools, Web
services are becoming easy to create -- and oh
so easy to botch.

A Web service is, after all, a collection of

procedures exposed to the unforgiving
thoroughfare of the Net (be it inter- or intra-).
And a failed Web service invites wrath not only
from the managers and administrators who
monitor and maintain the server's well-being,
but from the clients who have called your Web
service. Get your Web service right, or get two
earfuls of complaints.

In this roundup, I examined three tools that purport to verify that your Web services do what
they are supposed to do, that they resist graceless failure, and (in some cases) that they
conduct themselves with efficiency. The tools are soapUI, TestMaker, and WebInject. All are
open source, and are available for free download and incorporation into your next Web services
project.

I will note one thing: you need to have a good understanding of SOAP and HTTP protocols to
use these tools. One might say that one should have that regardless of what tool you're using,
but some commercial products offer a helping hand with a "pseudocode" translation of SOAP
code. It helps both neophyte and experienced SOAP users understand what's going on in a
particular SOAP request or respond block by translating to difficult-to-read XML into easier-
to-read pseudocode. These three open-source Web service testing tools require a little more
work, and I would recommend them for moderate-to-expert developers, where the learning curve
would be only modestly longer than for a commercial product.

soapUI1.6
I reviewed Version 1.6 of soapUI, a Java-based tool from Eviware [1]. This version executes within
its own stand-alone UI; the new 1.7 release includes plug-ins for the NetBeans [2], IntelliJ, and
Eclipse [3] IDEs.

The user interface conforms to the architecture of the typical IDE: a

1 of 7 9/20/2010 3:49 PM
Three open source Web service testing tools get high marks http://www.infoworld.com/print/28995

navigation pane on the left, a content pane on the right, and Click for larger view.
additional properties panes tucked near the bottom. If you've used [4]
an IDE like Visual Studio lately, you'll find your way around soapUI
instantly.

soapUI arranges work into projects. Each project is primarily identified by the interfaces that the
project is built to test. Here, an interface is the “other end” of a URI (uniform resource identifier)
pointing to a site that is exposing Web service methods. You can quickly generate a
skeletal project by aiming an empty project at a Web service's WSDL code; soapUI will
accept WSDL from either a file or a Web service end point that transmits the WSDL for its
services.

Projects are arranged hierarchically and contain one or more TestSuites, which contain one or
more TestCases, which in turn contain one or more test steps. The actual work – sending
requests, receiving responses, analyzing results, and altering test execution flow – happens at
the test step level. TestCases gather and organize the steps need to perform a specific
operation on the target. TestSuites gather TestCases into larger aggregates that exercise a
particular area of a Web service (such as the operations necessary to order a book). You can
create new TestSuites, TestCases, and test steps by right-clicking on the parent node in the
project's tree and selecting New from the pop-up context menu.

Then soapUI determines a given test's success or failure by assertions that you attach to test
responses. There are a half dozen assertions to choose from, ranging from a "simple contains"
test -- which succeeds if a provided string is matched -- to "XPath matching," which succeeds if
an arbitrarily complex XPath expression matches the response.

A test step is analogous to a line of code in a program. Currently,

soapUI defines six test step types, the most common being a
Request, which sends an HTTP request to a destination and
accepts a response. Control flow through test steps can be
modified by inserting a Conditional GoTo test step, essentially a
collection of one or more Xpath expressions that examines the
most recent response. The first expression in the collection that
succeeds causes branching to an associated target test step.

The Groovy test step (I am not making that name up) is soapUI's
most powerful. Here, Groovy refers to the lightweight Javalike Click for larger view.
scripting language [6]. A Groovy test step executes whatever [5]

Groovy code you want to place in it, which means that it can do
pretty much anything that can be done in Groovy. The Groovy code in such a test step has
access to the soapUI framework. For example, a Groovy test step might read information from a
database (via JDBC), compare that information with a previous test step's response, and alter
the execution flow of the TestCase accordingly – perhaps even executing a different TestCase.

Aside from the functional testing, SoapUI will also unleash load tests on a Web service. Each
load test consists of the execution of one or more TestCases, and can be tuned to simulate
various scenarios. You can, for example, control whether the test executes for a specific amount
of time or a specific number of iterations, whether the activity executes in “bursts” or varies
linearly over time, and so on.

When the load test is complete, a Load Test Editor provides a raft of statistics for each
TestCase: number of executions; minimum, maximum, and average execution time; and others.

2 of 7 9/20/2010 3:49 PM
Three open source Web service testing tools get high marks http://www.infoworld.com/print/28995

You can even examine the results graphically on the Statistics Diagram page.

It is easy to get soapUI running; it will take you almost no time to build a basic project and
construct rudimentary tests. My one gripe with the tool is that there is no contextual help in the
system, which makes figuring out what your options are in particular areas of the application
difficult. Nevertheless, the documentation provided is quite good, so any initial confusion
evaporates with continued use.

TestMaker
Click for larger view.
TestMaker is a Web service testing application from PushToTest. It
[7]
requires Java 1.4 (or later) to execute. Although I tested the other
tools on Windows, I installed TestMaker 4.4 on Ubuntu Linux 6.10 to see what Web service
testing on Linux was like. Installation was simple, and once I had specified a JAVA_HOME
environment variable, TestMaker launched and ran with no problems.

TestMaker's tests are embodied in scripts called “test agents.” The product lives up to its name
by providing an Agent Wizard that will read a WSDL definition and automatically create a
skeletal test agent.

I should point out that TestMaker is not limited to testing Web services; it can also be used to
test Web applications. Bundled with TestMaker is a network monitoring tool that can watch
HTTP traffic between your browser and a target Web application, and generate test cases from
the interaction. However, I did not experiment with this capability, since it has limited value in
association with Web services, which are usually driven by a client application.

TestMaker test agents are written in Jython [8] (Python written in Java). This forges a
double-edged sword. On the one hand, TestMaker's scripts can be as powerful as your
programming abilities allow. Jython can access all the Java libraries (and unleash all their
attendant capabilities), as well as classes and methods provided with TestMaker. The largest of
TestMaker's libraries is TOOL (Test Object Oriented Library), and it includes classes for
handling all sorts of communication protocols: HTTP, HTTPS, SOAP, POP3, JDBC, and more.
You can, therefore, create magnificently elaborate test cases that approach or surpass any
client application the Web service is likely to be called by.

On the other hand, you're going to need to know Jython to get the
most out of TestMaker, or in other words, you'll need to know
Python and Java. That is not necessarily a bad thing, but it does
mean that the learning curve for TestMaker is steeper than the
other tools.

The skeletal test agent created by the Agent Wizard is extremely

barren: It knows about the Web methods of the target service, and
it will execute without error, but it doesn't actually perform any
requests, responses, or tests of results. I found that I had to
examine the source code of one of the example test agents to fill in
the missing pieces.
Click for larger view.
Once you've passed the learning hump, it's easy to create new [9]
tests by copying, pasting, and tweaking existing code. In addition,
the user interface is a joy to work with. When you first start TestMaker, it opens a “QuickStart”
window, from which you can launch the Agent Wizard, jump directly to a menu of the example

3 of 7 9/20/2010 3:49 PM
Three open source Web service testing tools get high marks http://www.infoworld.com/print/28995

test agents provided, or dip into the extensive documentation. Also, TestMaker's UI is a standard
multiwindowed IDE, with navigation in the left pane, editing in the right pane, results in the lower
right, and a class navigation view in the lower left.

TestMaker can be executed from the command-line, so that your test agents can be executed
by an automation system. In addition, TestMaker bundles the Apache Axis TCPMonitor tool,
which allows you to monitor HTTP exchanges on a specified port. This is useful when you need
to examine the internals of request/response pairs to determine how to craft your Jython test
code.

A commercial version of TestMaker adds XSTest, which provides performance and scalability
testing, a monitoring dashboard that runs tests cases automatically and provides live results to a
viewing console, reporting capabilities, and TestNetwork – which can execute test agents
remotely, thereby allowing you to build “farms” of test agent servers that can exercise a target
Web application en masse.

TestMaker's documentation is exceptional, and the tool has the look and feel of a professional
application. However, it is difficult to master. Set aside plenty of time to go through the tutorials
and examine the invaluable source code examples.

WebInject

WebInject is a super-lightweight testing tool that can automate the testing of both Web services
and Web applications. In fact, WebInject's ability to test XML/SOAP Web services appears to be
a recent addition to the tool, as earlier versions could not readily handle the SOAP protocol.

Written in Perl, WebInject is primarily a command-line tool, though its author provides a thin
Perl/Tk user interface that at least simplifies the execution of tests for those unwilling to spend
too much time at the command prompt. If you're not familiar with Perl, don't panic. WebInject is
built so that you can construct your tests without having to touch so much as a byte of Perl
code.

WebInject is really an execution and reporting engine. Unlike the

other tools, it has no IDE-style user interface, so tests must be
written in an editor outside of the WebInject UI. This gives
WebInject a less professional feel, but doesn't hamper the tool. I
envision users of WebInject having directories filled with text files
of various test “templates.” To add a new test case, the user just
pops open his or her favorite editor, does some cutting, some
pasting, and a bit of tweaking to alter the template to fit the
specific circumstance, and ba-ding!, you've got a new test case.

When you launch WebInject, it reads an XML file containing a

description of the test cases that the engine is to perform. Each
test case is described by a set of attributes within an XML <case ...
/> element. So, a simple test case that verifies that a specific Web Click for larger view.
method returns a list of book titles that includes My Antonia would [10]
look like this:

<case

id=”5”

4 of 7 9/20/2010 3:49 PM
Three open source Web service testing tools get high marks http://www.infoworld.com/print/28995

description1 = “Verify My Antonia in list”

method=”post”

url=”Http://localhost:8080/axis/bkService.jsw”

postbody=”file=>soapListTest.xml”

posttype=”text/xml”

addheader=”SOAPaction: urn:getBookList”

verifypositive=”My Antonia”

/>

The “id” attribute not only provides a unique name for the test case, it also identifies the
execution order of test cases. The rest of the attributes are reasonably straightforward (once
you've perused the documentation, at least). The value passed to the “postbody” attribute tells
WebInject to go grab an XML file named “soapListTest.xml” in the current directory and use that
to craft the SOAP content for the request.

If the result includes the string “My Antonia,” this test will succeed.
WebInject provides three additional “verifypositive” attributes, and
the value of each is treated as a regular expression. This means
you can create quite complex verification specifications --
specifically, you can construct a test step that passes only after it
has made it through the filters of four regular expressions, one for
each “verifypositive” attribute. A test case element could also
include four “verifynegative” attributes that work just the opposite
of “verifypositive”: The testcase fails if any of the regular
expressions match.

In essence, a WebInject “project” is nothing more than an XML file

filled with a set of <case ... /> elements strung one after the other.
Click for larger view. WebInject's simple structure lets you build tests with amazing
[11] rapidity. You must, however, have a moderately good
understanding of the mechanics of SOAP protocols as well as a
tool that lets you generate and capture HTTP/SOAP requests and responses. You'll need the
requests to build the POST body and the responses so that you can create proper
“verifypositive” and “verifynegative” regular expressions to check for success or failure. I used
the Web Service Toolkit add-on for Eclipse to grab requests and responses for WebInject; once
I had gotten the hang of it, I fell easily into the groove of building test cases.

As each test case executes, WebInject's UI displays the status (pass or fail). You can configure
WebInject to provide a complete dump of all HTTP requests and responses, an exceedingly
useful feature that you will certainly call upon when you need to debug a failing test case.

In addition, the UI will optionally produce a real-time graph of the response (round-trip) time for
each request/response pair, so you can use WebInject to build and monitor performance tests.
And, WebInject provides a plug-in for MRTG (Multi Routing Traffic Grapher), a network
monitoring and data collection tool, allowing you to execute and capture the results of test cases
over a period of time as well as examine the data for patterns and trends.

5 of 7 9/20/2010 3:49 PM
Three open source Web service testing tools get high marks http://www.infoworld.com/print/28995

WebInject's single greatest characteristic is its simplicity. Once you've gotten the hang of
WebInject's XML commands, you can build, modify, and extend test cases quickly. The entire
documentation consists of a one long Web page that can be read in a single sitting. However,
that one page of documentation sometimes leaves you wondering how to proceed. Furthermore,
you'll need a moderately good understanding of the SOAP protocol, as well as a tool to extract a
Web service response's POST body, which you'll need to build test cases.

At your service

These three tools place themselves along the spectrum from quick and easy to complex and
powerful. Do you need to assemble some code quickly for hurling tests at your Web service?
WebInject is the logical choice; you'll be buffeting your Web service code in an afternoon. Do
you need a high-end tool that lets you create powerful tests that can be extended to draw upon
other system resources – the filesystem, databases, e-mail, and such? Then roll up your
sleeves and plow into TestMaker. Grab a Jython manual first, though, and prepare yourself for
some heavy lifting.

I prefer the middle balance struck by soapUI. The skeletal tests created by soapUI's wizard were
easier to flesh out than those built by TestMaker. And, if I needed to do something elaborate
and off the wall, I could always call upon soapUI's Groovy capabilities – funny name aside, they
do their job well.

In terms of how these products compare to commercial Web service testing tools, I'd say it's a
mixed bag. They are, of course, inexpensive (free), and work well for easy to moderately-difficult
jobs; on the other hand, they're somewhat less user-friendly than commercial tools and if you
need to do something complex, you have to build it yourself. TestMaker comes closest to
looking like a commercial product, but having to learn Jython means that it takes longer to set
up some tests than it would with, say, Mindreef's SoapScope. soapUI is a tad less professional
looking, but makes up for it by allowing you to construct useable tests without having to
program. WebInject, however, is definitely a developer's tool. You need to know SOAP to use it
well, and it isn't going to be as capable as soapUI or TestMaker because its test cases are very
much driven by templates.

Developer World Application Development Applications Architecture Languages and

Standards SOA (service-oriented architecture)

Source URL (retrieved on 2010-09-20 03:48PM): http://www.infoworld.com/d/architecture/three-open-source-

web-service-testing-tools-get-high-marks-995

Links:

6 of 7 9/20/2010 3:49 PM
Three open source Web service testing tools get high marks http://www.infoworld.com/print/28995

[1] http://www.eviware.com/
[2] http://www.infoworld.com/article/07/03/26/13FEjavaides_4.html
[3] http://www.infoworld.com/article/07/03/26/13FEjavaides-sb_1.html
[4] http://www.infoworld.com/img/19TCwebservicetest1.jpg
[5] http://www.infoworld.com/img/19TCwebservicetest2.jpg
[6] http://groovy.codehaus.org/
[7] http://www.infoworld.com/img/19TCwebservicetest3.jpg
[8] http://www.jython.org/Project/index.html
[9] http://www.infoworld.com/img/19TCwebservicetest4.jpg
[10] http://www.infoworld.com/img/19TCwebservicetest5.jpg
[11] http://www.infoworld.com/img/19TCwebservicetest6.jpg

7 of 7 9/20/2010 3:49 PM