Performance Engineering Platform for DevOps

Performance Testing
Element of Solution Contribution to Project Open Source Commercial
JMeter Taurus CA BlazeMeter
JMeter UI Construct tests manually in the JMeter UI X X X
JMeter Plugins Extend JMeter functionality to cover nearly any use case. X X X
Taurus Domain Specific Generate JMeter Tests from brief, human readable blueprints,
Language empowering dev's to do earlier testing. X X

Custom JMeter Plugins Develop custom plugins in rare cases where open source community
does not yet have a needed feature. Many are then contributed to X
open source, but that is optional.
Parallel test suite execution Compress time window for test runs down to the length of the single
longest unit test, shortening test cycles X

Infrastructure on demand Eliminate CapEx and maintenance. Simply turn it on when needed,
turn it off when done. X

Secure Mobile App Build accurate mobile app usage scenarios directly on real devices
Recorder X

Test orchestration, Achieve real-time spin-up, control and results graphing for tests of any
aggregation, reporting scale, whether on-prem, private cloud or public cloud X

APM Integration Rich APM integrations to speed identification of performance

bottlenecks and tracking of performance baselines all the way to the X
method level.
© 2017 CA. Confidential. All rights reserved.
 Limitations of Open Source (1/2)
CA
Feature Criteria JMeter Taurus BlazeMeter Notes
Language What's the level of education and High Medium Low With JMeter you need a high level of proficiency. It can take
Proficiency expertise required? (affects how many months to master JMeter. Taurus makes it easier to execute
people will actually be part of the tests and BlazeMeter makes it a breeze, enabling any
practice) member of the engineering group to participate in the test
operation.
Scale Ability to support large scale, multi Low Low High JMeter and Taurus enable you to run low scale tests where
GEOs, cloud tests BlazeMeter enables you to run large scale tests from
multiple geographies
Productivity
How many tests can you run in general Small Medium High Using JMeter, every test requires an investment in time and
(Number of and in parallel? The more tests you can resources. It's tough to scale your test operation with JMeter
Tests) run the better you coverage is. The only. Taurus helps you increase your productivity.
more tests you can run in parallel, the BlazeMeter enables you to run x100 more tests at the same
less delay you have to release. time period compared to the open source tools
Test and The ability to manage test Low Medium High Your testers run hundreds tests per day and thousands per
Report configurations and versioning across month. Its very easy to get lost in the high number of test
Management projects configurations and reports. It's literally impossible to manage
your test configurations using the open source tools. Taurus
helps and BlazeMeter provides a complete test management
platform allowing you to run thousands of tests and manage
the various configurations and reports across different teams
and projects
Reporting & Ability to visualize the results in a self Medium Low High Using JMeter provides a basic reporting level. Taurus
Analytics explanatory fashion provides a run-time dashboard, while BlazeMeter provides
superior reporting capabilities with easy to understand
reports
© 2017 CA. Confidential. All rights reserved.
 Limitations of Open Source (2/2)
CA
Feature Criteria JMeter Taurus BlazeMeter Notes
Real time The ability to monitor a test in real None Low High A critical requirement: With JMeter you are unable to
test time, see the results and influence the view result in real time and monitor the test progress.
dashboard test during the run Taurus provides basic capabilities to monitor the test in
real time. BlazeMeter provides a dashboard to monitor
& influence the test in real time.
CI ready Ability to set and enforce thresholds None High High Fast-moving projects need not only "war room" all-
(trends, (SLA) and to observe and respond to hands on deck tests, but also automated thresholding
thresholds) trends enforced on tests of all sizes. Detect drift and call
attention quickly to performance regressions.
APM Bi-directional correlation (see APM None None High No time and mental energy wasted shifting context or
Correlation metrics in load testing graphs, see correlating events manually.
load testing metrics in APM
diagnostics)
Time To Time to prepare and launch. Time to High Medium Low Taurus and BlazeMeter make test set-up and re-testing
Test retest second nature, whether running on desktop (Taurus) or
full-scale production load (BlazeMeter)
Real time Ability to pause ramp-up, increase or None None High No need to cancel a test part way through if ramp is too
Test Control decrease load while test is in flight fast or to too low a level. Preserve time during big test
windows and keep war-room tests on track.
Data Ability to secure data behind fire-wall None None High BlazeMeter Private Cloud offering allows organizations
Security and anonymizing test results outside to secure data behind firewall, while delivering the
firewall benefits of SaaS and anonymizing test results outside
the firewall
© 2017 CA. Confidential. All rights reserved.