The Hrmes Framework

A data-driven test framework for Selenium RC

The Universal Manual for Hrmes

Document Title

Hrmes User Manual

Document Abstract

Hrmes is the Human Readable Machine Executable Specification(pronounced Hermes) Test Framework designed and
developed by Andrew Thompson and W3QA Ltd.
This document describes the procedures for installing and configuring Hrmes. Additionally it outlines how to run,
develop and understand its reporting features.
This document will be in-line with the code release of the Hrmes project.

Document History

Date Hrmes Purpose of Revision Author

Version

14/02/2009 Hrmes_0_1_0 This revision supports the initial release Hrmes. Andrew Thompson

Page 2 of 66
The Universal Manual for Hrmes

Contents

1 Introduction............................................................................................5

2 The Hrmes License ................................................................................6

3 The Hrmes Framework Pre-requisites ........................................................7

3.1 Step 2 – Install Eclipse...............................................................................7
3.2 Step 3 – Install Firefox 2 with appropriate plug-in..............................................8

4 Installation and Configuration of the Hrmes Framework.....................................9

4.1 Installing through Eclipse...........................................................................9
4.2 Installing Additional Eclipse Plug-ins - Optional...............................................12
4.3 Remaining Configuration..........................................................................13

5 The Hrmes Format and the Selenium IDE......................................................14

5.1 The Hrmes Selenium IDE Format.................................................................14
5.2 The Hrmes Format File Conversion..............................................................21
5.3 The Hrmes Format Files...........................................................................22

6 Running Hrmes.......................................................................................25
6.1 Test Execution via Eclipse.........................................................................25
6.2 Test Suites...........................................................................................26
6.3 The Hrmes Test Methodology.....................................................................27

7 Demo - An End to End Example .................................................................29

7.1 Record ...............................................................................................29
7.2 Export and Convert.................................................................................30
7.3 Run and Report......................................................................................31

8 Using CVS through Eclipse.........................................................................33

9 Understanding the Hrmes Framework Reports................................................36

10 Developing the Hrmes Framework.............................................................39

10.1 The Application mappings.......................................................................39
10.2 Java Custom Methods............................................................................42

11 Selenium GRID and the Hrmes Framework...................................................45

11.1 Selenium GRIDs integration with the Hrmes Framework...................................45
11.2 Selenium GRID and its Hrmes Framework Ant scripts......................................47
11.3 Selenium GRID integration with the Hrmes Framework....................................48

12 Release Management and the Hrmes..........................................................50

12.1 Upgrading To Newer Versions...................................................................50
12.2 The Impact On Shared Data From Upgrades..................................................51

13 The Hrmes Frameworks Database Schema...................................................53

13.1 Building the Hrmes Schema......................................................................53
13.2 Loading Test Case Data into the Hrmes Schema............................................54

Page 3 of 66
The Universal Manual for Hrmes

13.3 Loading Test Results into the Hrmes Schema................................................54

13.4 Using the SQL Explorer Eclipse Plug-in........................................................54
13.5 Database Run Mode...............................................................................55
13.6 Graphical Analysis................................................................................55

14 Debugging tests with the Hrmes Framework................................................57

14.1 Debugging...........................................................................................57
14.2 Debugging through Eclipse.......................................................................57
14.3 Debugging through the IDE.......................................................................59
14.4 Debugging using Hrmes...........................................................................59
14.5 Debugging Manually...............................................................................60

15 Troubleshooting the Hrmes Framework.......................................................61

15.1 Ants CLASSPATH settings.........................................................................61
15.2 Hrmes Frameworks Java CLASSPATH settings................................................61
15.3 Hrmes Frameworks Source......................................................................62
15.4 The S-IDE Export Process........................................................................63
15.5 The Hrmes Framework Source within the Selenium-IDE ..................................63
15.6 S-IDE Export and Extraction Failure ...........................................................64

16 Support for the Hrmes Framework.............................................................66

16.1 Use SourceForge to Raise Support Requests.................................................66

Page 4 of 66
1 Introduction
This is the user manual for the Human Readable Machine Executable Specification
(Hrmes – pronounced Hermes) Selenium RC Test Framework. It will take you through
its installation, configuration, development and execution.

The Hrmes framework is a test automation framework built on the OSS Selenium RC
(SRC) platform. It is designed for use on all web based applications.

Hrmes enables a unified process for test specification, execution and reporting. It has
a clear domain specific language for test specification which supports manual and
automated testing through the same singular Hrmes Format asset.

The structure and narrative of Hrmes is based closely on Seleniums’ Fitnesse. All test
assets produced via the Selenium IDE can be easily converted into Hrmes tests
assets.

Hrmes is the first Selenium framework to achieve a data driven approach on a fully
enabled Selenium RC platform.

Hrmes is fully integrated with Selenium GRID. Out of the box you can run distributed
tests on the platform configurations of your choice - in the same way you could
using Selenium GRID directly.

Hrmes provides a powerful and usable set of tools. It facilitates a standard test
process and a singular environment for all test assets across all projects.

The Hrmes Framework gives the general test community a scalable framework for
developing and rolling out structures, processes and software for itself and its clients.

Automation frameworks cannot mitigate the need for technical resources on a test
automation project. All such projects are technical by their nature. The Hrmes
project goes a long way to enable the full and continuous involvement of non
technicians in a Selenium RC automation project. However, Hrmes does require
some technical ability and knowledge of Selenium to fully exploit. Hrmes will work
best with teams that have access to test engineering resources with some
programming experience.

The Hrmes Framework provides increased clarity, usability and accessibility to a

Selenium RC automation project, without diluting any of its power.

Page 5 of 66
2 The Hrmes License
The Hrmes Framework is available on an Apache 2 license. It is assumed that you
have read and agreed to these licensing terms.

FURTHERMORE: THE Hrmes FRAMEWORK SOFTWARE IS PROVIDED "AS-IS" WITHOUT ANY

WARRANTY WHATSOEVER. YOU ASSUME ALL RISKS AND RESPONSIBILITIES FOR SELECTION
OF THE SOFTWARE TO ACHIEVE YOUR INTENDED RESULTS, AND FOR THE INSTALLATION OF,
USE OF AND RESULTS OBTAINED FROM THE SOFTWARE. THE POINT CLARK NETWORKS
MAKES NO WARRANTY THAT THE SOFTWARE WILL BE ERROR FREE OR FREE FROM
INTERRUPTION OR FAILURE. TO THE MAXIMUM EXTENT PERMITTED BY APPLICABLE LAW,
W3QA LTD DISCLAIMS ALL WARRANTIES, EITHER EXPRESS OR IMPLIED, INCLUDING BUT NOT
LIMITED TO IMPLIED WARRANTIES AND CONDITIONS OF MERCHANTABILITY, FITNESS FOR A
PARTICULAR PURPOSE, AND NONINFRINGEMENT WITH RESPECT TO THE SOFTWARE AND

Page 6 of 66
3 The Hrmes Framework Pre-requisites
The Hrmes Framework can be installed on any operating system that supports Java,
The Hrmes test data resides in a spreadsheet format. This manual assumes that you
are using MS Excel. However, Open Office or any other spreadsheet package will
work equally well.

You need to install the following before running the Hrmes Framework:

Hrmes Framework Pre-requisites

Software Version Location Size

Eclipse Eclipse IDE for Java developers - http://www.eclipse.org/downloads/ 80MB

3.2.1 (aka - Europa)

Mozilla Firefox 2.0 – Not 3.0 http://www.mozilla.com/en-US/

Browser Install Selenium IDE plugin (v1.0)for
from Firefox:
https://addons.mozilla.org/en-
US/firefox/addon/2079
and Firebug:
https://addons.mozilla.org/en-
US/firefox/addon/1843

If versions listed here are not available choose the closest match.

The Hrmes video need QuickTimePlayer. The video are stored under “Hrmes
Framework\Framework\QuickTimeMovies”.

Most of Hrmes’ dependencies are bundled. You will automatically get them when
you install Hrmes through Eclipse. This includes the Selenium client and server jars,
Java SDK, the Java API for Excel, Java JDBC drivers and other miscellaneous
dependencies that you can reference through Eclipse.

The Hrmes project requires about 120MB of disk space. The core Hrmes code itself is
less than 100K, but the project does come heavily packaged with almost everything
you will ever need for a Selenium based test framework.

Time to download will obviously depend on your bandwidth, but allow about 30
minutes to 1 hour to complete the install and configuration.

3.1 Step 2 – Install Eclipse

1. Using Windows Explorer, create a folder off your C drive and call it Hrmes.
2. Download Eclipse and extract it into Hrmes folder.
3. Launch Eclipse (from C:\Hrmes\Eclipse\ folder) and select the C:\Hrmes as
your workspace.
4. Create a shortcut and drop this short-cut into your Windows Start menu.

Page 7 of 66
3.2 Step 3 – Install Firefox 2 with appropriate plug-in

You probably have these, but just in case:

1. Go to the Mozilla site and download Firefox version 2.

2. Search for Selenium IDE, Web Developer and Firebug under plug-ins and
install.

Page 8 of 66
4 Installation and Configuration of the Hrmes Framework

4.1 Installing through Eclipse

The Hrmes Framework code, the vast majority of its dependencies and all related
project assets are entirely under source control in the SourceForge CVS repository.

Getting hold of the entire Hrmes project is simple using Eclipse. Within Eclipse, go to
Window->Open Perspective -> Other, then select CVS repository.

In the CVS repository tab right-click and select New -> Repository Location.

Page 9 of 66
Add a new repository with the settings described above.

Those settings again:

Host: hrmes.cvs.sourceforge.net
Repository path: /cvsroot/hrmes
User: anoymous

There is no need to enter a password for the anonymous user, so leave it blank. The
anonymous user gives you read access to the repository.

Clicking finish will show you the Hrmes Framework as it is in CVS. Expand the
repository so that you can see the “HEAD”, “Branches” and “Versions” entries. It is
recommended that you take the latest release. Tagged versions are stable releases,
whereas “HEAD” reflects the latest check-in.

Whether you choose “HEAD” or a tagged release from “Versions”, the steps are the
same.

Page 10 of 66
Important note on Eclipses’ default settings
Before checking out the project ensure that Eclipse will check out the entire project
including all currently empty folders. In Eclipse->Window->Preferences->Team->CVS-
>Files and Folders, uncheck “Prune empty directories”.

To check out the entire framework right click on “Hrmes” and “WebProjects” then
select “Check out”.

This action checks out Hrmes onto your hard drive. This will take several minutes.

After check out, Eclipse will alert you to any problems it has via the “Problems” tab.
There should be no errors or warnings.

If there are any errors – you will have to resolve them before continuing. Have a look
at the “Trouble Shooting” section at the end of this document if you run into
problems.

You can now view the code base of the Hrmes Framework. You do this via the Java
perspective (Window-> Open Perspective -> Java).

Page 11 of 66
4.2 Installing Additional Eclipse Plug-ins - Optional

You can skip this section if you do not wish to use CubicTest right now. It is not
required by Hrmes Framework and can be installed anytime.

In Eclipse go to Help->Software Updates -> Find and Install.

Change the default to search for new features. After clicking new remote site you
will be presented with the following:

Page 12 of 66
CubicTest settings for the update manager should be set accordingly (URL:
http://www.cubictest.org/update):

One of many useful things about Eclipse is that you can manage plugin upgrades
easily through it.

4.3 Remaining Configuration

The Demo project has been created to provide a basis for exploring what the
framework can do. It is also template for what a Hrmes project should look like
initially.

Build a project for your application using the Demo project as a template. Using
copy and paste in Eclipse, is probably the easier way.

The remaining configuration is based on the run-time properties of Hrmes. They are
best explained in that context.

Page 13 of 66
5 The Hrmes Format and the Selenium IDE

You can build your test scripts directly in the Hrmes format. You may also convert
Selenese tests created through the Selenium IDE into this format.

If you build your scripts directly in the Hrmes format, you will not have to wait for your
AUT to be available to start writing your test scripts.

The ability to abstract test definitions with Hrmes gives you more flexibility on how you
compose your test scripts.

For instance you may write dozens of test cases with a reference to a target called
“search box”, before the AUT is delivered. Subsequently, all you need do is create a
mapping rather than rewrite all your test cases.

In this way Hrmes, supports an important Agile principle of “test early”.

Hrmes may also be used as a common format for manual and automation tests. This
will minimise the reworking of manual test assets when they need to get automated.

When you compose your tests directly in the Hrmes format, use the Selenium IDE as
much as the state of your application will allow. The Selenium IDE will be a help
when you need to identify your applications various locators and to remind you of
the pretty extensive Selenium command set. All of which can be used directly in
Hrmes.

You can skip the next 2 sections if you are not intending to convert tests from the
Selenium IDE into the Hrmes format.

5.1 The Hrmes Selenium IDE Format

The Selenium IDE supports the export of its native Selenese HTML into several formats.
We are now going to add an extra one – the Hrmes Format.

In Eclipse go to Hrmes->Framework->SeleniumIDE->Formats. Open up the file called

“Hrmes – Hrmes Format”.

Start Firefox and open the Selenium IDE. Within the IDE open Tools. Select Options ->
Options -> Formats.

Page 14 of 66
Next select “Add” and give the name of the new format “Hrmes - Hrmes Format”.

Keep your Firefox session open and return to your Eclipse session which should have
the Hrmes source you viewed earlier. Copy this into the Selenium IDE format source
and click “OK”.
You should now have something resembling the following:

Page 15 of 66
The Hrmes format is based on Selenese HTML and facilitates simple conversion from
it. The format options allow the user to specify how test data within the Selenium IDE
gets parsed into steps by the Hrmes extraction process.

Click “OK” and return to the main Selenium IDE screen. Now you can export any
Selenese scripts in the new format. In the Selenium IDE go to File->Export Test Case
as-> Hrmes. The “Save as” dialogue will appear and you can now save your export
to your project location C:\Hrmes\<Project>\SeleniumIDE\Exported.

You have now exported a HTML file that contains all the information required to
convert it into the Hrmes Format. The exported file itself is still almost identical to the
original Selenese HTML version.
The next step is to convert it into the Hrmes spreadsheet format.

Upload the user-extensions.js file

Page 16 of 66
This step is optional. Whether you wish to perform it will depend on how match
backwards compatibility you wish to preserve between the Hrmes and the Selenium-
IDE. This issue is covered later on. You may skip this section if you do not want
backwards compatibility.

With the Selenium IDE options window still open click on the “General” tab.

Under the Selenium Core extensions, click the browse button and upload the user-
extensions.js file.

Location: “C:\Hrmes\WebProjects\Demo\SeleniumIDE\user-extensions.js”.

Page 17 of 66
Now click OK. This gives the Selenium-IDE the ability to use custom Javascript
methods defined through this file. There is more on what use you can do with this
functionality later. Refer to the Selenium website ( www.openqa.org ) for the full
explanation on what user extensions are.

If you already make use of user extensions, just merge the contents of the two files. If
you do not, I would advise skipping the rest of this section.

Note on S-IDE user extensions

Some user extensions that have been written for the S-IDE will not translate to any
Selenium Java API method – or indeed any Selenium RC (SRC) client language
method. These are ones that make use of purely S-IDE or Selenium Core objects.

Page 18 of 66
There are few of these. Indeed, I can only recall two - the “goto” and “include”
methods.

Whereas many user extensions get incorporated into the core S-IDE language set,
these will never make it. They would break the S-IDE code translations, i.e. from
Selenese HTML to Java.

The vast majority of pure Javascript user extensions will have some meaning to SRC
and can be made available in the Hrmes in one of three ways.

Firstly and probably the simplest way is via custom methods calling getEval. See the
development section later on for an explanation. If you have a lot of complex
Javascript this can get a bit messy and unreadable.

The other approaches leave the Javascript in the user extension file. They are then
available when you start the Selenium server with the –userExtension parameter. This
is the default setting for invoking the Selenium server with the Hrmes.

For the user extension method to have any meaning a Java method must be
created to handle its use.

One way of doing this is by writing a method in the Hrmes class. There is one already
created for the user extension “typeRepeated”.

public void typeRepeated(String locator, String pattern)

throws ScreenShotException {

/*
* This method relies on the doTypeRepeated method being defined as a
* user extension.
*/

try {

proc.doCommand("typeRepeated", new String[] { locator,

pattern });

} catch (RuntimeException e) {
// TODO Auto-generated catch block

screenShot(e);

This method acts as a proxy to the Javascript method defined in the user-extension.js
file. This is a clean way of making any extension available. However, as the Hrmes
class is NOT a web project data file you will have to merge this change every time
you do an upgrade of the Hrmes project.

Page 19 of 66
Lastly, you may directly incorporate your user extensions into the Selenium Java
client. In this case the Selenium Java client will need rebuilding.

Although there is some convenience in having a user extension method available

directly through the Selenium Java client, it does require the greatest effort of the
three approaches.

Firstly you will need to install Mavern and get hold the source code for at least the
Selenium client: http://svn.seleniumhq.org/svn/selenium-rc/trunk/ . The Hrmes project
has mvn pre-installed and a snapshot of the selenium-rc, circa November 2008. This
is provided as no more than an example of what you should end up when you go
down this route. See “Hrmes->Mavern”.

You now need to add your new method in iedoc2java.xml and rebuild with Mavern:
mvn install.

If you are not comfortable with using Mavern to build the Selenium project, check
the Selenium developers’ guide:
http://wiki.seleniumhq.org/display/SRC/Developer's+Guide#Developer%27sGuide-
DevelopingtheJavaRCClientDriver

If after reading this you find that your comfort level has not been elevated: I would
go for either of the first two approaches: getEval or write a method into the Hrmes
class.

Incorporating user extensions written for the Selenium IDE or Core, into a Selenium RC
platform is a fairly common Selenium challenge.

All three approaches of incorporating pure Javascript user extensions into a

Selenium RC framework are discussed on the seleniumhq website in the following
thread: http://clearspace.seleniumhq.org/thread/14467 .

Forcing user extensions to live in the world of Selenium RC can be needlessly hard
work. Review all the dependencies your test scripts have on user extensions. Look to
replace them where you can with custom methods written in Java. Those that
cannot be replaced “like for like“, are the ones that will not work on a Selenium RC
platform. These will need to be re-worked in other ways – probably by altering the
test specifications. This should be rare.

One of the strengths of the Selenium project is that it offers so many entry points for
test automation, from the Selenium IDE to Selenium GRID. However, moving from
one flavour can be painful. When you move onto the Selenium RC platform from the
comfort of the Selenium IDE you do need to “mind the gap”. If your automation
project has a lot of dependencies on user extensions this point is especially pertinent.

Of course, if your project does use extensions, then the issue is less relevant and you
are probably reading for academic reasons.

Page 20 of 66
5.2 The Hrmes Format File Conversion

Selenium IDE tests scripts once exported can be converted into the Hrmes format.
The conversion process runs via an Ant script called “Start Hrmes Extraction Process”.

If you have not done so already, drop this build file (located under
Hrmes/Framework-/AntScripts/Hrmes Extraction) into the Ant Window within Eclipse.

Like all Ant scripts in Eclipse – just double click it when you want it to run. It will
process all the Selenium IDE exported files found in your Web Project folder and
move them to the neighbouring Extracted folder.

The conversion process will preserve your original file. It will also create a new version
of it with an XML file extension and an entirely new Hrmes Format spreadsheet file.

Batch Mode

The conversion process can run against multiple S-IDE Selenese data files at once
without the data having been previously exported into the Hrmes format. The batch
option is available should you wish to convert a large amount of Selenese data in
one run. This is ideal for Selenese test suites.

You need to let the Hrmes know how your test scripts should be parsed. You should
use these as you would when you configure the Hrmes export format in the S-IDE.
These settings will apply to all test scripts processed in any one batch run. The Hrmes
itself will create a S-IDE /Hrmes export version of the raw Selenese file. It then
converts it as it would do if you had exported it manually.

From a user’s point of view, the process is almost exactly the same. You need to run
the same Ant script and the Selenese data files are expected to be in the S-IDE
export location.

You need to let the extraction process know that it’s working in batch mode. So
change the side.conversion.mode property to “batch”. By default it is not.

###############################################################
##

side.conversion.mode=batch
side.parse.token=clickAndWait
side.when.to.parse=after

#########################################################
#########

Page 21 of 66
In batch mode you do not have to configure anything in the S-IDE. You do not even
have to add the Hrmes export format to the S-IDE. In batch mode things are driven
by the “hrmes.properties” file.

5.3 The Hrmes Format Files

The Hrmes format is entirely derived from the Selenese structure of Command-Target-
Value and Selenium‘s version of FIT.

This similarity will minimise any pain adjusting from the Selenium IDE. It also means
that testers can avoid the standard route into Selenium RC via a programming
language.

The Hrmes format is an enhanced form of Selenese. It can be enhanced further

through application mappings (aka GUI mappings) and custom methods.

The Hrmes format facilitates the development of a single test asset, which can be
used for manual and automated testing. It is a document that can facilitate
auditing and review of test specifications. It provides a clear and easily read
document. The flow of the raw Selenium IDE script is transformed into test steps and
formatted according to the different type of actions performed.

You may develop your test scripts straight into this format without using the recording
features of the Selenium IDE.

Using the Hrmes format directly requires a high level of familiarity with Seleniums
command set. If you are new to Selenium, the Selenium IDE is probably a better
place to start.

Although spreadsheet packages such as MS Excel have many limitations, they can
be easily used to produce sophisticated documentation. They are also a simple and
familiar interface to most testers.
It is often overlooked how important good documentation is, in keeping managers
and to people in the more “business” focused roles in IT involved in an automation
project. These are usually the very people who often need to know the detail of
what is actually getting tested.

You may use the Hrmes format to abstract all the implementation specific detail you
get through the Selenium IDE. This will make your tests easier to read and far easier to
maintain. This is dealt with later – under the development section.

Note on Eclipse and Excel

Eclipse will make little effort keeping itself in sync with Excel files on your file-system.

When you open an Excel file it is instantly marked as having been changed (in
Eclipse denoted by an asterix to the left of the file name) – despite the fact it has

Page 22 of 66
not. You can certainly read and write Excel data directly through Eclipse, but do not
expect to do much more.

When you will need to do more than this – for instance opening screenshots and
stack traces from the Hrmes reports – do so outside of Eclipse, e.g. in Windows
explorer.

Preserving backwards compatibility with the Selenium IDE

Using the S-IDE, you may export tests directly into the Hrmes format. You may also
import Hrmes test data back into the S-IDE.

During test execution a Selenese data file will be created dynamically. This will
correspond directly to the Hrmes test data being run.

The file created is an XML file called seleniumIDE.xml and stored under
“WebProjects\SeleniumIDE-\Imported”.

This file can be imported into the Selenium IDE and used like any other test script
recorded through the S-IDE.

This may require an explanation on why anyone would wish to do this. Test analysts
will need to build up a level of comfort using the Hrmes – especially those that are
not technical.

The Hrmes project goes further than any other to ease the transition from the world
of S-IDE into Selenium RC. However, the learning curve has not been removed
completely. Some testers will feel more comfortable knowing their Hrmes tests are
fully compatible with the Selenium IDE. It is mainly included as a fall back option.

The task of maintaining compatibility is relatively simple, even when we start

extending Hrmes and writing custom methods.
Keeping test scripts that have custom methods compatible with the Selenium IDE,
means making use of Selenium’s user extension functionality.

Page 23 of 66
The default Hrmes user-extension.js file that we uploaded earlier, contains the
following:

Selenium.prototype.doStartUp = function() {

};

Selenium.prototype.doTearDown = function() {

};

Selenium.prototype.doComment = function() {

};

These entries correspond to the basic custom methods of the Hrmes Framework such
as “startUp”, “teardown” and “comment”. They are dummy methods that allow any
Hrmes test to run within the Selenium-IDE.

To preserve backwards compatibility, this user extension file needs to be kept in line
with any custom methods you write.
You will need to write a dummy method for each new custom method you write.
Hardly onerous, but remember to do so.

As mentioned earlier, preserving backwards compatibility is not a requirement of the

Hrmes, but you may find it desirable.

Page 24 of 66
6 Running Hrmes

6.1 Test Execution via Eclipse

Using CVS through Eclipse

Hrmes can be run directly through the Eclipse JUNIT test environment and is setup
ready to do so.
You may also run your Hrmes Framework scripts through Ant. This is probably the
simpler user interface.

The Hrmes framework is built to support system testing of all web projects.

You control the entire Hrmes configuration through property files and each project
has its own version appropriate to it under “PropertyFiles”.

Refer to the property files themselves for all documentation on them and how to
configure them. After you make all the appropriate changes to your projects
property files, remember to save them.

They are straightforward and intuitive. Configuring them is not onerous and if you are
“happy” with the default settings, the task should be complete in minutes if not
seconds.

Hrmes needs to know which project it is currently running against. This is configured in
project.properties file. This file is the only one that needs to live under the core Hrmes
project. Copy and paste it into “Hrmes/Framework/conf” folder.

There are few Ant scripts that control the running of the entire framework. All are
located under the “AntScripts” directory. Open the Ant view in Eclipse - Window->
Show View -> Ant. Drop all the Hrmes ant scripts (build.xml files) into the Ant Window.

Page 25 of 66
One configuration file you will almost certainly need to change is the
“myBuild.properties” file. Have at look at the default settings and ensure they match
the settings on your machines.

Ant scripts are simple to run – just double click them and they will run.

The “Selenium Remote Control Server” script will start the Selenium Server on your
machine. After starting the Selenium Server you can now run the Hrmes framework –
through the “Start Hrmes tests” Ant script.

Hrmes will run all the tests found under the “Run” folder of your Web Project area.

You now have Selenium IDE tests running on Selenium RC environment, without
writing a single line of code. You can not do this with any other framework.

6.2 Test Suites

Creating Hrmes Framework Test Suites

Page 26 of 66
When you wish to create a series of test scripts designed to be run together simply
create a single Hrmes spreadsheet, with a series of ordered worksheets within it.

The framework will run each test script in turn – from left to right. You will also get an
individual report generated per worksheet.

6.3 The Hrmes Test Methodology

The Hrmes Framework is designed to be non proscriptive. This flexibility should

maximise its usability.
Apart from adopting the structure and definition of Selenium’s Fitnesse almost
without question – Hrmes will insist on little.

Test Case Design

However, structure and methodology is good for goodness sake. There is an area
where Hrmes is prescriptive that might not be immediately obvious.

Page 27 of 66
Within the Hrmes framework, test scripts should be developed to run independently
of all others. There should be no interdependency in regards to either test flow or test
data.

This condition may seem an inefficient way to develop test scripts, but it delivers the
far bigger prize of scalability. Selenium based tests have to run independently and in
parallel in order to scale. If they do not your tests may still be running when you go
out of business.

Test authoring in this way will enable greater robustness in the test run. It will also
make any test script more readable. Test scripts that jump around fulfilling
dependencies, are hard to follow. It is fundamentally poor practice and should be
avoided. Make sure all test cases are standalone.

If there is a test flow that is repeated throughout test scripts in a test suite, testers
should compose a single test method with all the steps “rolled up”. In this way the
overhead is minimised and tests preserve their independence.

Test Case Names

The worksheet names of the Hrmes spreadsheets are synonymous with a test case.
Ensure that you give these worksheet names a meaningful and unique description.

If the same test case features in other Hrmes test suites maintain its name. Although,
this is not essential you will find that the Hrmes test reports and analysis of results is far
clearer on data that has been normalised in this way. It is also far more effective.

Page 28 of 66
7 Demo - An End to End Example

This section demos the Hrmes Framework by using a real example. It will take you
through the recording, exporting, conversion, running and reporting within the
framework. This will be a good test of your install and configuration.

We will look at creating the Hrmes test script via the Selenium IDE. Let’s assume you
have no test script recorded yet.

7.1 Record

Using the Firefox browser go to the http://www.w3schools.com/ and open the

Selenium IDE.

Now record some user activity. Additionally, perform some verification and write a
comment for your test after you have finished. You script may look something like
the following:

Under the S-IDE options, ensure you have the Hrmes Framework format set, then save
you test.

Page 29 of 66
7.2 Export and Convert
Next export your test in the Hrmes format to the relevant area within your
WebProjects directory, e.g. “C:\Hrmes\WebProjects\Demo\SeleniumIDE\Exported”.

Page 30 of 66
Within Eclipse run the Ant script “Start Hrmes Extraction Process”. You will now have 3
data files with the “Extracted” folder. Remember to refresh the Eclipse view - right
click on directory-> refresh. The file we are going to run is the “Demo.xls” file. This is in
the Hrmes format.

7.3 Run and Report

Drop the Hrmes file into the Run folder. Ensure that your starting.url is configured to
point at your base URL - http://www.w3schools.com/. Now run the “Start Hrmes
tests” Ant script.

Page 31 of 66
After the test has completed you should now have a Hrmes report looking
something like the screenshot below. If you do: congratulations as your install and
configuration are probably sound. More importantly, your understanding of the
Hrmes’s key processes are probably sound too.

If you did not get a report - go back and ensure that you followed all the steps. Also
have a look at the troubleshooting guide at the end of this manual.

Page 32 of 66
8 Using CVS through Eclipse
This section assumes you have access to a CVS repository. It is recommended that
you implement code control over your project specific Hrmes test assets.

Remember to share your assets

Always consider sharing the assets you have created. There are some that you
should not, but amongst those you should are:

• The Hrmes Framework test scripts

• The Selenium IDE exported files and original Selenese ones.
• The applicationMappings.properties file

If you are familiar with using CVS through Eclipse you may skip the rest of this section.

Sharing your assets …

Page 33 of 66
Right click on the asset and select Team->Commit. You will get prompted with the
screen above.

Enter a comment that will be meaningful to yourself or a third party who may come
to use it later. Don't be too concise: "fixed typo" is too short; "fixed typo on test case
10 step 5" or "fixed typo in home page assertions" is OK.

The aim is to make it easier to find the desired change easily from just the commit
messages.

Note on Excel files

This point is particularly pertinent when committing spreadsheet data. As Excel files
are stored in a binary format, CVS will be unable to provide any other tracking of the
changes over time, like it can with ASCII data.

Page 34 of 66
The converse of this is including too much information. CVS automatically maintains
information like the date and time of the commit, the person who made the
commit, etc. You don't need to include it in the commit message yourself.

Also remember that Excel can get big – so commit only what you really need to
share – do not use CVS as an archive for test reports.

Clicking “Finish” commits the file to the repository and will make it available to the
rest of the team and anyone else who comes to work on this project in future.

Page 35 of 66
9 Understanding the Hrmes Framework Reports

As well as the run-time logs, each time you run the Hrmes Framework it will produce
a single Excel spreadsheet report for each worksheet of every Hrmes that is being
run. The default location for this is in the “Reports” directory of your WebProjects
directory in Eclipse.

This report will match any Hrmes Format file you have produced. Apart from a few
extra columns, it will also look much like it. This report will detail the results of your test
run and provide the status of each test and related screenshot and stack trace
should that test have failed. This is the main report the Hrmes generates.

A test summary report is created for each test run of the Hrmes – including those run
through GRID and will look something like the one below.

If your machine is set up to send email, then you will also receive an email
notification with this summary report attached. There is a README file under Hrmes-
>Jars->Ant-libs for further information on getting this setup.

The summary report provides a clear and precise overview of the test run – with
number of tests executed, number of passes and fails.

Page 36 of 66
However, always refer to the aforementioned spreadsheet for detail. It will give you
the stack traces and the occasionally priceless: screenshot.

There is also a consolidated report on all theses summaries which provides links back
to the data specification being run and the detailed test reports. This can be found
under the reporting area, for the demo project this will be:

C:\Hrmes\WebProjects\Demo\Reports\ReportSummary\junit-noframes.html

As well as a general summary on the test runs it contains data on the run-time
environment (for both GRID and SRC runs).

Note on why JUNIT reporting via Ant was abandoned

Other JUNIT test frameworks will have dozens of unique test methods per test run and
thousands in a test suite. They will also require a team of Java developers to
maintain.

The Hrmes Framework is a code super lightweight and has only 1 test method in the
entire framework. A report on its test methods will not be informative.

Page 37 of 66
The only reporting functionality that does remain in this area is the use of the JUNIT
report XSL format. This is why the consolidated report is still prefixed with junit.

The Hrmes is still based on the JUNIT test framework.

Page 38 of 66
10 Developing the Hrmes Framework

It may seem odd having the section on developing the Hrmes Framework after the
section on running the Hrmes.

As the Hrmes is an execution-ready framework, you do not have to write any code in
order to run it. All the code has already been written.

However, to make full use of all the features of the Hrmes, in particular its ability to
perform application mappings and fully extend Selenium it is likely that you will want
to develop it.

10.1 The Application mappings

The Hrmes allows users to abstract out implementation level aspects of their test
specifications.

This mapping feature is useful is maintaining a distinction between what is test

requirement and what is test implementation. This is often blurred in automation with
a detrimental impact on maintenance and clarity.

Developing the application mapping layer of the Hrmes is simple. You just create a
set of (key value pair) mappings in a property file.

The “applicationMapping.property” file is the place for all these mappings. Please
refer to it for all documentation on how to use it.

Mapping application data to test specification data

We are going to go through a simple example to see how application mappings

work. Go to: “C:\Hrmes\WebProjects\Demo\test suites\Examples”. Here you will see
a set of Hrmes files at various stages of abstraction.

The first Hrmes file we need to look at is the “raw-google” one. This contains the test
script as it looks straight after being exported from the S-IDE and converted into the
Hrmes format.

Page 39 of 66
Let’s assume that we wish to use our test data as a test document for our managers
and business stake holders. Furthermore, let us assume we are testing a web
application that is still under-going development and which has locators that keep
changing every release. Lastly let’s assume that we would like to abstract our test
data. Using the Hrmes we decide to abstract all the target and value data.

We need to define what these targets will get mapped to. This is done through the
applicationMapping.properties file. We already have these mapping defined there
as default. The relevant section is extracted below:

###############################################################
###########
#
# Target/Locator Mappings
#
query.location=q
search.button=btnG
starting.url=www.google.co.uk
###############################################################
###########

###############################################################
###########
#
# Value/Data Mappings
#
query = bbc
###############################################################

Page 40 of 66
Mapping test specification data to application data

We have created a set of “key value pairs” that define the map between the
application layer and our test requirement. We can now make use of these
mappings throughout all our Hrmes test data files.

The end result is a far clearer document. We also have a document which can
remain unchanged even when there are changes to the application. We simply
change the mapping value rather than all the test specification documents. These
documents will change only when the test requirement changes.

In the “Examples” folder we have a template Hrmes file. This can be used to author
and structure our tests without having to record through the S-IDE. This is much like
the way manual test scripts are initiated.

When to automate

Deciding when to automate at the system test level is more of an art than a science.
The main driver tends to be the readiness of the application under test (AUT).

At the system test level, test execution is usually performed manually at the initial
stages of a project. The AUT is either not available or too brittle for a machine to test
against.

Page 41 of 66
Hence, tests generally get “migrated” into an automation suite during the life of a
project rather than at its inception.

Under circumstances where manual and automated test specifications are distinct,
this can be complex and require significant expertise not only to translate the
manual test asset, but to get the timing of the migration right. Too early and the
automation scripts will need a lot of reworking and significant manual intervention.
However, if the tester delays too much, the return on investment from automation
gets diminished.

Tests specified at a projects inception, when there is no working web application, will
usually require significant reworking as the full functionality of the application gets
delivered. However, with a common format, we do not need a new set of test
artefacts.

Switching between the two types of testing (manual and automated) is relatively
painless. With a common format we do not need to maintain two independent
streams of work, i.e. one for manual and one for automation. We have a common
mutable document for all types of testing.

10.2 Java Custom Methods

The Hrmes is a code-free solution to automation. Out of the box, it does not require
additional code to run and can be entirely data driven.

However, one of the advanced features of the Hrmes is that its functionality can be
extended through Java.

Writing code in certain circumstances will certainly be the most efficient, productive
and easiest way to perform certain actions and validations.

Where you have an AUT that has volatile and dynamic test results, requires test flow
management, or database validation – then you will want a programmatic solution.
So it is vital we have the means to create one.

Writing code for the Hrmes is not particularly difficult as the development framework
has already been written and is in place. The most you ever need to write are your
custom Java methods.

Under the project specific area you will find the src directory and the
ProjectSpeciciMethods Java class file.

Here are examples of project specific Java methods contained in this class that may
be of use to you in constructing your own. There are further instructions in the class
file itself on what to do with it once you have finished writing your custom methods.

Once in place, all methods will be available for you (and the rest of your team) to
invoke via your test specifications in the Hrmes Format.

Page 42 of 66
For example, let’s say that you wished to combine the Selenium “waitForElement”
command with the “verifyText” command. You decide to call your custom method
“waitThenVerifyText”.

If you convert the original Selenese format into Java using the S-IDE you end with the
following source:

package com.example.tests;

import com.thoughtworks.selenium.*;
import java.util.regex.Pattern;

public class NewTest extends SeleneseTestCase {

public void setUp() throws Exception {
setUp("http://change-this-to-the-site-you-are-testing/", "*chrome");
}
public void testNew() throws Exception {
for (int second = 0;; second++) {
if (second >= 60) fail("timeout");
try { if (selenium.isElementPresent("locator")) break; } catch (Exception
e) {}
Thread.sleep(1000);
}

verifyEquals("pattern", selenium.getText("locator"));
}
}

The magic of the Selenium IDE has instantly created a complete Java test class for
our 2 commands.

However, using the Hrmes we already have our class and indeed the complete
framework to run it in. All we need is the code for our custom method – highlighted
in bold.

All custom methods reside in the ProjectSpeciciMethods.java class file. I need to

create a custom method called “waitThenVerifyText” within this file. In this case just
copy the appropriate code from both Selenium methods above. Then drag and
drop the file from “C:\Hrmes\WebProjects\[Project]\src” into the code source of
the core Hrmes project – “C:\Hrmes\Framework\src\hrmesFramework”, overwriting
the existing one.

Page 43 of 66
 public void waitThenVerifyText(String Locator, String Pattern) {

/*
* Write code here
*/

for (int second = 0;; second++) {

if (second >= 60) fail("timeout");
try { if
(selenium.isElementPresent("locator")) break; }
catch (Exception e) {}
Thread.sleep(1000);
}

verifyEquals("pattern", selenium.getText("locator"));

As Eclipse, by default automatically compiles projects before they get run there
should be no need to explicitly compile.

This method is now available to specify within your Hrmes test specification. With the
new custom method available we can now have Hrmes tests that reference it. Like
the following:

Unlike many proprietary automation tools, you are not limited to writing methods
solely based on the tools API. You can extend the Hrmes by writing any custom code
that can be implemented in Java.

So, for example you can write a custom method that interacts with a database to
verify the state of backend data at a particular stage in your test. This functionality is
invaluable if you need to perform more “whitebox” type of tests. It is one type of test
that would be impossible within the Selenium IDE.

Page 44 of 66
11 Selenium GRID and the Hrmes Framework

11.1 Selenium GRIDs integration with the Hrmes Framework

Since version Hrmes_0_1_0_0, the Hrmes Framework is fully integrated with Selenium
GRID.

This means that you can evoke any number of instances of the Hrmes to run against
the environments you have configured and made available on your Grids’ hub.

The Hrmes provides an Ant script that sets up 6 environments on your default hub
location - localhost.

selenium.grid.server.host.one=localhost
selenium.grid.server.environment.one=*chrome
selenium.grid.server.port.one=5555
selenium.grid.server.args.one=-multiwindow

selenium.grid.server.host.two=localhost
selenium.grid.server.environment.two=*chrome
selenium.grid.server.port.two=5556
selenium.grid.server.args.two=-multiwindow

selenium.grid.server.host.three=localhost
selenium.grid.server.environment.three=*chrome
selenium.grid.server.port.three=5557
selenium.grid.server.args.three=-multiwindow

selenium.grid.server.host.four=localhost
selenium.grid.server.environment.four=*iehta
selenium.grid.server.port.four=5558
selenium.grid.server.args.four=-multiwindow

selenium.grid.server.host.five=localhost
selenium.grid.server.environment.five=*iehta
selenium.grid.server.port.five=5559
selenium.grid.server.args.five=-multiwindow

selenium.grid.server.host.six=localhost
selenium.grid.server.environment.six=*iehta
selenium.grid.server.port.six=5560
selenium.grid.server.args.six=-multiwindow

The hub configuration you get with the Hrmes is just to give you a basis to run your
Hrmes tests through the Grid. It can all be changed to match your particular
Page 45 of 66
circumstances. You can have as many environments as you like, configured to
support whichever combinations of browser type, version and platform you wish to
test for. Assuming, of course you have access to the physical resources to run them
on.

If at this point you are wondering what on earth the GRID is, then before diving into
the Hrmess implementation of the GRID, refer to the Selenium GRIDs home page on
openqa:
http://selenium-grid.openqa.org/.

You start the Selenium GRIDs hub and run the Hrmes tests with 2 Ant scripts. One
“Start Selenium GRIDs Hub and SRC instances”, will provide the GRID configuration
just mentioned.

They can be viewed via the hubs console.

By default this is on: http://localhost:4444/console

The second Ant script controls how your Hrmes Framework/SRC instances access
these environments. When running instances of SRC (through Hrmes or directly) via
the GRID you need to let them know where your hub lives and which environment
you want this instance to run against.

Page 46 of 66
 selenium.grid.src.environment.one=*chrome
selenium.grid.src.starting.url.one=http://www.bbc.co.uk/
selenium.grid.src.Hrmes.file.location.one=C:/Hrmes/WebProjects/Demo/Run/Demo-
001.xls

selenium.grid.src.environment.two=*chrome
selenium.grid.src.starting.url.two=http:// www.bbc.co.uk /
selenium.grid.src.Hrmes.file.location.two=C:/Hrmes/WebProjects/Demo/Run/Demo-
002.xls

selenium.grid.src.environment.three=*iehta
selenium.grid.src.starting.url.three=http:// www.bbc.co.uk /
selenium.grid.src.Hrmes.file.location.three=C:/Hrmes/WebProjects/Demo/Run/Demo-
003.xls

selenium.grid.src.environment.four=*iehta
selenium.grid.src.starting.url.four=http:// www.bbc.co.uk /

This configuration is stored in the seleniumGRID.properties file and by default has 4

Hrmes Framework/SRC instances defined – something like the abstract above.

You can have as many instances as the hardware your hub references can handle.
Each of the 4 instances will run in parallel. Should your hub have available
environments you can run any number of configurations. However, you will probably
hit hardware issues if you rely too much on your localhost. If your access to hardware
is limited, it will start to crawl after about 5 or 6 active Selenium RC instances on the
hub.

The key concepts of Selenium GRID are distribution and configuration, rather than
just parallelism.

Indeed, running parallel instances of the SRC can be achieved without the GRID.
Being able to easily access distributed instances of the SRC running on varying
configurations, is the Selenium GRIDs great achievement.

11.2 Selenium GRID and its Hrmes Framework Ant scripts

The 2 Ant scripts written for the Selenium GRID need to know about a few of the
GRIDs’ jar files.

You will need to do this via Eclipse in the similar way you made Ant aware of tools.jar
earlier: Eclipse-> Window-> Ant-> RunTime -> CLASSPATH-> Global Entries-> Add Jars.

There are 3 jars you need to add – highlighted in the screenshot below and found
under Hrmes/Hrmes Framework/ Selenium GRID.

Page 47 of 66
11.3 Selenium GRID integration with the Hrmes Framework

When the Hrmes Framework runs outside of the GRID it expects to use the standard
Selenium RC server. It also expects only one.

It order to make the Hrmes work with the GRID it simply needed to be run with
parameters that would tell it which SRC instance to run against. The Hrmes now runs
in two modes, one where it will expect multiple instances of SRC and one where it
does not.

These modes are controlled by toggling a property in the hrmes.properties file.

If you want to use the SRC standard set up leave the selenium.grid.mode.set as false:
selenium.grid.mode.set=false

If you want to use the GRID set the selenium.grid.mode.set to true:

selenium.grid.mode.set=true
In terms of integration that is pretty much it. You start the Hub as you would without
the Hrmes and you start the Hrmes instances as you would without the GRID. The

Page 48 of 66
only difference is that you now have multiple instances of the Hrmes on SRC rather
than one.

This is important to bear in mind when you are troubleshooting the Hrmes operating
with the Selenium GRID. If you have managed to get the GRID working with SRC, i.e.
the standard way and if you have also managed to get the Hrmes working without
the GRID, you will also get the Hrmes working with the GRID.

The integration of Hrmes Framework with the Selenium GRID is not complex. You do
not end up running different “versions” of either the Hrmes or the GRID. Essentially,
you are running the GRIDs Hub as you would normally and evoking the Hrmes in a
“GRID aware” way.

Page 49 of 66
12 Release Management and the Hrmes

12.1 Upgrading To Newer Versions

The Hrmes Framework has been designed so that its core data is entirely isolated
from web project specific data it interacts with.

This makes upgrading from one version to the next straightforward.

When to upgrade

Simply put: if you do not need to upgrade them do not. If you are working happily
on a stable release and have no need for new functionality or potential bug fixes
stay where you are. All versions of Hrmes are available in perpetuity and you will
never be asked to upgrade.
However, if you want to make use of new functionality then at some stage you will
need to upgrade.

Upgrade process

Ensure you backup the file system under Hrmes before you start, just in case you
make any mistakes during the upgrade – which does involve deleting files.

Start by deleting the “Framework” directory and all files under it. Make sure that
Eclipse has been closed before you begin.

Leave the rest of the Hrmes data, including the “WebProjects” area alone. After the
initial install of the Hrmes you never need to check-out that part of the project again.
If this area needs changing in a future release – the release notes will tell you how to
modify what you have. This part of Hrmes should be under your own code control.

It is a good idea to restart your machine to prevent any locking issues you may
encounter.

Before you upgrade, ensure that the hidden .project file for “Framework” is deleted.
This is best done in Eclipse. This file is often displayed as a casserole pot in Eclipse.

Once you have deleted the entire “Hrmes” structure, you are ready to get the
newer version. You will need to connect to the sourceForge repository to do this. See
the notes on the install process should you need reminding on how to do this.

Expand the “Versions” entry in the CVS Repositories tab. Have a look at the versions
available by expanding the “Hrmes” entry.

Page 50 of 66
Right-click on the one that you want and check it out. After the check out process
has finished you will have the new version in the same place as the old one was.

12.2 The Impact On Shared Data From Upgrades

There are some property files that reside in the web project area that are shared.
The core project needs values for these properties, but those values are project
related. The starting.url for instance, needs to be defined for the Hrmes to run, but its
value is dependent on the particular web project you are testing.

The hrmes.properties, myBuild.properties and seleniumGRID.properties are the three

properties files that contain shared data.

In rare cases new properties may be added in a new release of the Hrmes. These
new properties will be flagged in the release notes of the release that they were
introduced in. If your upgrade is not contiguous, remember to check for any new
properties from within those intermittent.

Notes on the restructuring the WebProjects area

A new release may bring with it new directories and a new layout to the Webproject
area under Hrmes. This is rare, but can happen.

Page 51 of 66
Again changes will be mentioned in the release notes. You can also review any
changes in the CVS entries, to assess what this may mean for your project. At the
most it may mean creating or renaming directories and moving test assets like
properties files and Hrmes test scripts around.

The Hrmes release label number convention

Only major and minor releases, could impact the WebProject area. By convention
any changes to the WebProject will mean an increment in the minor release
number.

So, if the first 2 digits of the release tag have not changed, the webproject area
needs no changes to support a new release. If you do upgrade to a new major or
minor release then read the release note and assess the possible impact on your
web projects under test.

Page 52 of 66
13 The Hrmes Frameworks Database Schema
The Hrmes framework has a MySQL database that may be used for uploading test
data – both test cases and test results.

It works with MySQL version 6 and above - as it makes use of a XML interface only
available with that version. As MySQL releases are platform specific – this is one
application used by the Hrmes than does not come bundled with it. MySQL jars
(which are platform independent) do come pre-bundled and are found under the
“Jars” directory.

The MySQL download and configuration is simple – see the link below:
http://dev.mysql.com/downloads/mysql/6.0.html

As you will want to share and compile data across your web project, set your MySQL
instance up on a server, rather than on your local machine.

The Hrmes schema is simple and has been designed solely to make the task of
loading XML data into a relational database easy. This works through the MySQL
load xml function (on MySQL version 6 or above), which is a robust way of getting
XML into a database.

With each run of Hrmes, a set of XML data files are created that correspond to the
test cases being run and the results of the test run. These two distinct sets of files
reside (by default) under the WebProjects area. Test case XML data gets created
under “Data/TestCaseXML” and test run XML data under “Reports”. There are also
SQL scripts that reference these XML data files. It is these that are run through the Ant
scripts and are used to upload the XML data into the appropriate database tables.

If Hrmes had a roadmap one objective would be to improve the reporting and
analysis features of Hrmes. The schema and its functionality will be enhanced in
future versions of Hrmes project. Of course, as Hrmes is OSS you do not have to wait
until then.

There are Ant scripts to assist data loads and schema build: one for loading Hrmes
test data: "Start DB Load of Hrmes Test Case Data", one for test result data: "Start DB
Load Of Test Run Data" and one to build the Hrmes schema. All scripts are found
under the “AntScripts” directory.

13.1 Building the Hrmes Schema

To make use of this functionality you firstly need to build the Hrmes database
schema. Running the Ant script: "Start Hrmes Schema Creation" creates the Hrmes
schema objects. This execution the MySQL-schema-creation.sql script under “Hrmes/
Framework/ReportDatabase/Schema/”. You need to do this as the root database
user. You will also need to create a Hrmes database for this script to succeed. Do this
via the MySQL client.

Page 53 of 66
13.2 Loading Test Case Data into the Hrmes Schema
After each test run, the test case XML data can be loaded into the Hrmes database
by clicking the Ant script: "Start DB Load of Hrmes Test Case Data", in the Eclipse Ant
window.

All SQL files found in the “Data/TestCaseXML” will be loaded. Delete or archive any
you do not want loaded. There are no uniqueness constraints in either of the Hrmes
tables, so all data will be inserted regardless.

13.3 Loading Test Results into the Hrmes Schema

In a similar way the test result XML data – which is also created at run time, can be
loaded into the Hrmes database by clicking the Ant script: "Start DB Load Of Test Run
Data".

Again, all SQL files found in the “Reports” will be loaded.

13.4 Using the SQL Explorer Eclipse Plug-in

This section will take you through the install and configuration of the SQL Explorer
plug-in found here:
http://eclipsesql.sourceforge.net/

Installing SQL Explorer is entirely optional, as you can of course interact with the
Hrmes database using any database tool that you prefer – including those that
come with MySQL itself.

However, as Hrmes is Eclipse based you should find the SQL Explorer option the most
convenient. See the section earlier on installing Eclipse plug-in, if you need reminding
on how this is done. The update site reference is the same link as above.

Once you have installed the plug-in, configure a connection to the Hrmes
database. Use the MySQL documentation to help you if this is unfamiliar to you.

Once you have a working connection you can then view data that has been in
loaded into the Hrmes database.

You can of course do much more: including running SQL operations against it,
creating your own view of the raw data and pretty much anything you can do with
similar standalone database applications.

As mentioned SQL Explorer is one of many ways to interact with the Hrmes database,
including using the command line.

Page 54 of 66
13.5 Database Run Mode

The advent of a database schema has introduced another way of running Hrmes.

Now you can use test case specifications that reside in the database rather than
Hrmes spreadsheets. This can provide the user with much more flexibility over the test
data. When run in this way, Hrmes will reference the “Hrmes.raw_test_case_data”
table rather than the “Run” directory. All tests will be executed in “load_id” order.
This is left NULL initially for the user to configure should they wish to use this mode.
They run in the same way as the standard Hrmes tests, but are sourced from the
database.

As they run in the same way, you can also run your database test specifications
through Selenium GRID.

The GRID configuration is identical to the standard mode. Hrmes simply uses the
spreadsheet name of the selenium.grid.src.Hrmes.file.location parameter (within the
hrmes.properties file) to match the spreadsheet name in the database – in particular
the “raw_test_case_data.spreadsheetname” column .

13.6 Graphical Analysis

The Hrmes project includes the ability to perform graphical analysis on run time and
test case data. Although this is fairly limited, it is one area that will be expanded on.

At this stage the inclusion of Entrance with this version is experimental. It is what
Entrance calls the “community” or free version and has a few constraints for the
Hrmes project. Currently, the ability to run it on the command line is limited with this
version – making it hard to run via Ant. It would also be ideal if it was available as an
Eclipse plug-in.

That aside, Entrance (http://dbentrance.com/) is on a MySQL platform and can be

used to produce some impressive graphics. It is the only application bundled with
Hrmes that has to run outside of Eclipse.

Under “Hrmes/Framework/Entrance/charts/benchmark” is the Entrance-SQL code

need to produce graphs like the one below.

Page 55 of 66
This graph compares the failure rate of a regression suite with an earlier benchmark
run. If there has been no change in the test results we would expect the circles to
align with the squares.

The Hrmes scheme provides the raw data of all test runs and test case specifications.
It may be used as a basis for a varied graphical representation of your projects test
data – both results and specifications.

Page 56 of 66
14 Debugging tests with the Hrmes Framework

14.1 Debugging
Up to now we have generally assumed that your tests run, get reported on and then you
move on to the next test cycle.
When tests fail you will want to isolate what bits you are most concerned about and
reproduce the test within a much more controlled environment than the run-time
environment.

Hrmes offers many ways to debug your tests.

14.2 Debugging through Eclipse

As Hrmes is Eclipse based you can bring all the power of its debugging capabilities to bear
to trouble shoot your test.

You can of course debug the any of the code within the Hrmes project, but we will go
through the standard case of debugging a test which has failed.

As Hrmes is fundamentally a JUNIT test framework the class we launch the debugger from
is the “HrmesTestSuite.java” class file, found under the “Hrmes/Framework->src”. If we
right-click on the file within Eclipse we see the option to “Debug as” in the panel second
from bottom. Select Java Application will launch the debugger.

As mentioned we can debug any part of the project, but if we are concerned with the
workflow a test, we need to locate the part of Hrmes which handles this. This is handled
by the “InvocationData.java” class. If we open that file within Eclipse and search for
invokeSeleniumMethod, we are in the part of the code that handles the calls to the
Selenium API. If we breakpoint here as the commentary suggests we can step through the
test workflow step by step.

Page 57 of 66
Additionally we can put conditions on the breakpoint. We will often want a test to run up
to a particular sequence, rather than stop at every command.
If we right-click on any breakpoint we set, we select its properties and control under what
conditions it breaks.

Page 58 of 66
In the screenshot above we will stop when we have reached sequence 10.

Controlling Hrmes in this way does take you into territory normally preserved by
developers and not all users of Hrmes will feel comfortable with this. There are
alternatives.

14.3 Debugging through the IDE

As Selenium-IDE versions of every Hrmes tests being run are created at run-time, we can
use these as a fallback to the Selenium IDE.

These tests can be found under the WebProjects area, under “SeleniumIDE\Imported”.

14.4 Debugging using Hrmes

We can add “pause” commands in the Hrmes spreadsheet data to slow down the running
of a test at a particular point.

Page 59 of 66
14.5 Debugging Manually
There is always a strong element of manual verification of automation test failure. There
are many reasons why you will wish to make sure that a failure reported from automation
run is actually reproducible by a real user.

One of Hrmes key features is to provide a common format for manual and automated
tests. Making use of abstraction you can construct clear test specifications that can be
handed over to a test analyst to investigate.

Page 60 of 66
15 Troubleshooting the Hrmes Framework

15.1 Ants CLASSPATH settings

If you run into any problems running your Ant scripts check your CLASSPATH settings.
In Eclipse under Window->Preferences->Runtime->Classpath.

They should look like as they are in the screen shot.

If you are missing these, add them with “Add External JARS”.

15.2 Hrmes Frameworks Java CLASSPATH settings

Your CLASSPATH settings should be as the screenshot below.
Under Eclipse, in the package explorer right click on the “Framework” project, then
go to properties->Java Build Path.
If you are missing Jars then add them through “Add External JARs”.

Page 61 of 66
Also, if you are using Selenium GRID review that section to ensure you have made
Ant aware of the Jar files particular to the GRID.

15.3 Hrmes Frameworks Source

Your “Source” tab should also look something like the one below:

Page 62 of 66
15.4 The S-IDE Export Process

The Hrmes extraction process is based around exporting native Selenese (HTML) from
the Selenium IDE and transforming the data into a spreadsheet format. The Hrmes
extraction process works by loading XML data files. After you export from the S-IDE in
the Hrmes format, you will have XML data essentially. The file may not have a XML
file extension – do not give it one the extraction process renames all files to have this
extension before transforming them.

The Hrmes extraction process will enforce format compliance and is fairly strict. This
has the advantage of finding any data issues early. However, the exported data
needs to be well formed and correctly defined and structured.

The extraction process which builds the Hrmes Excel documents will enforce the
Hrmes data structure as defined by the Hrmes XSD - found under
”Hrmes\Framework\ SeleniumIDE \XSD”. If any document exported from S-IDE does
not conform they will get rejected. Generally, this will not be an issue as the Hrmes
structure is built by simply selecting the Hrmes format option under options in the S-
IDE.

However, sometimes data within the S-IDE can get corrupted and the source that
gets exported will contain formatting errors. This can happen for many reasons
including the switching between S-IDE formats, for instance: HTML to Hrmes.

If your XML files do get rejected then you will need to correct them. Firstly, ensure
that you currently have the Hrmes format set – in the Firefox plug-in: S-IDE -> Options -
> Format -> Hrmes. Then check the source tab in the IDE. You should see something
like the following at the top of your source.

15.5 The Hrmes Framework Source within the Selenium-IDE

Page 63 of 66
Within the S-IDE click on the source tab to see how your HTML looks in the Hrmes
format.

<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=Hrmes-8" />
<link rel="selenium.base" href="" />
<title>New Test</title>
</head>
<body>
<table cellpadding="1" cellspacing="1" border="1">
<thead>
<tr><td rowspan="1" colspan="3">New Test</td></tr>
<tr><td rowspan="1" colspan="3">parseToken: clickAndWait</td></tr>
<tr><td rowspan="1" colspan="3">whenToParse: after</td></tr>
</thead><tbody>
<tr>
<td>open</td>
<td>www.google.com</td>
<td></td>
</tr>

</tbody></table>
</body>
</html>

An example of the Hrmes data structure is contained in Hrmes->Framework ->

SeleniumIDE ->Formats-> Hrmes-SIDE-Source-Format.txt.

It is critical that the structure in your source matches this. The piece highlighted in
bold above will correspond to your Selenium commands and comments that you
have created. Ensure that they are situated as above – make any necessary
changes in the source itself, save and re-export the data into Hrmes.

15.6 S-IDE Export and Extraction Failure

If after exporting your test scripts from the Selenium IDE, you get something like the
following “undefined” message in the exported file, the then export from S-IDE has
failed:

undefined<tr>
<td>open</td>
<td>/</td>
<td></td>
</tr>
<tr>
<td>type</td>
<td>q</td>

Page 64 of 66
This can be resolved by forcing a reload of the Hrmes format in the Selenium IDE. To
do this check the source tab (within the Selenium IDE) as described in the previous
section. You will need to re-export your test script.

You may first encounter this problem when you try to extract Selenese data into the
Hrmes format.
If you do then you will get a null pointer exception and a message in the Eclipse
console window like:

“ org.exolab.castor.xml.MarshalException: Content is not allowed in prolog”

Page 65 of 66
16 Support for the Hrmes Framework

16.1 Use SourceForge to Raise Support Requests

The Hrmes Framework project is an OSS project that hosted on SourceForge.

You can review all issues raised against the project there, via
https://sourceforge.net/projects/hrmes/. You will need a sourceforge account.

Once there follow the “Tracker” link under the services drop down.

Raise all requests for support there, including any bugs or indeed any requests for
additional features.

It is easy to access and offers standard features for handling these issues.

Page 66 of 66