Zeenyx Academy

A Thousand G2 Reviews Later: Software Testing Tools Are Too Expensive

Print Friendly, PDF & Email

Table of Contents

Apparently, Commercial Software Testing Tools are Too Expensive

What a Thousand G2 “Dislikes” Taught Us

Over the last few weeks, we analyzed over a thousand reviews entered into the “Automated Software Testing” category on the G2 website. While we prefer relying on Gartner Reviews (they tend to be a little more comprehensive), there simply aren’t enough of them available to draw meaningful insights. Gartner only lists about 73 vendors, and only a dozen or so of the reviewed tools have more than 10 reviews. G2, on the other hand, covers about 231 vendors and contains about 13k reviews. So, while there may be a lot of debate about the value and veracity of the G2 reviews, it is great place to start.

We categorized the “dislikes” in the following buckets:

  • Buying it and getting started (cost, onboarding etc.)
  • Learning Curve
  • Poor UI
  • Lacking Features
  • Test Creation (complex object recognition etc.)
  • Test Execution
  • Test Management & Data Management
  • Test Reporting
  • Buggy Releases
  • Support/Learning Resources/Community
  • Slow or Unstable Execution
  • Resource Allocation Issues

Leading Dislike #1 – “Great software, but The license cost is just too high”

While many of the expressed dislikes were nuanced, idiosyncratic and somewhat difficult to categorize, several stood out as 1) frequent and 2) easy to categorize. They were – cost, support & learning resources, test creation issues and test object recognition/synchronization issues.

Here, we will address the cost issue, and hopefully provide the solo testing community with a full-featured testing tool at a reasonable cost.

Many of the dislikes related to pricing also mention the ready availability of Selenium, a free, open source solution for testing, albeit with severe limitations relative to commercial tools. And as best as we can tell, we have not seen another enterprise-level tool that attempts to provide anything remotely price competitive with free.

The Selenium Conundrum – When Free Isn’t Free

Why isn’t it free? Simple: the cost of labor swamps the cost of your testing software. Consider that in any test automation project, the fee for the tool is only a small part of the overall budget when compared with the cost of building the tests. So if you pay $10,000.00 for a testing tool and $100,000.00 in labor costs, have you saved money for your organization by selecting an open source tool, when it costs more than twice as much to build and maintain the tests?

Selenium Increases Labor Costs Relative to Good Testing Software

Nowhere will you read that Selenium is an awesome testing tool. You won’t read that it is easy to use or that the tests are easy to maintain. If there were a charge for Selenium, it is unlikely that it would be selected at all when compared to tools like AscentialTest that have been designed to promote productivity and provide a substantial return on investment in the first year of usage.

How Do We Know?

The AscentialTest team has been creating software testing solutions since the early days of the software testing industry (Segue, where we created Silk Test, was founded in 1988). Brian, our co-founder, was the head of R&D at Segue, and Dave, our co-Founder, was the Chief Architect at Segue. They also formed our own consulting firm (Star Quality) in 1998, specializing in automated software testing.  We have seen it all.

Don’t Believe Us? Ask Google:

Query – What are the downsides of using Selenium for automated software testing?


  • Not an All-in-One Solution
  • Steeper Learning Curve
  • Limited Desktop and Mobile Support
  • Maintenance Challenges
  • Browser Dependency
  • Limited Image Testing
  • Lack of Built-in Reporting
  • No Technical Support

An Example: Browser Navigation

When a user navigates your target application, do they close and reopen the browser after each transaction? Probably not. But when you run a test with Selenium, each test opens its own private browser. That limitation leaves out a lot of usage cases. It also makes it very difficult to debug a test. If a test fails several minutes into the execution, there is no way to run just a part of that test, because Selenium needs to start a new browser with each test run. The productivity loss that results from just this one limitation is astounding. If AscentialTest worked that way, we’d be out of business.

The image on the left below depicts a typical test step in AscentialTest. The one on the right
displays a function that is part of a Selenium test. What do you want your resources spending time on: testing or writing more code? Do you really want to spend money on programming to create automated tests. You decide.

AscentialTest vs. Selenium: Feature, Function and Productivity Comparison:

While it’s true that Selenium is a free set of open source tools, using them is not really free after all. The costs and benefits of test automation are significant. We’ve worked hard to provide a testing solution that makes testers more productive, saving your organization time and money. If you still think Selenium is free after reading this document, we have some lake- front property in {name your location} that we would just love to sell to you.

Features AscentialTest SeleniumTime Cost
Tool setup and configuration No setup or configuration Requires knowledge of required development environments and programming conceptsSeveral Days
Test frameworkAutomatically generated. Users begin creating tests right awayMust be custom-built and then tested. Frameworks can take weeks or months to complete.Several Weeks
Browser supportEdge, Firefox, Chrome. Handles all web application components including dynamic tablesIE, Firefox , Chrome, Opera. Does not handle all web components, e.g. popup windows, dynamic tablesVaries based on complexity
Non-browser based
application support
Yes, including windows, dotNet, PowerBuilder, Terminals, Java and many othersNoDays to build (or search for and implement a thirdparty solution)
Coding experience None required Yes. Requires experienced staff with skills to integrate the framework and code testsSkillset is costly
Product support Zeenyx support providedOpen source community along with user forumSupport is not curatated so it can take hours or days to find solutions
Software cost License and annual Free maintenance fee
Script generation time Drag and drop acceleratesScripting is time-consuming test developmentOngoing costs are high, easily take 4 or 5 times longer to develop tests
Test maintenance Abstraction and encapsulation results in high reuse and low maintenanceHigh, tests are difficult and costly to maintainOngoing costs are high, Easily take 4 or 5 times longer to maintain tests
Test Execution Time
30 to 40% faster than SeleniumExecution time is slowTests run 5x faster on AscentialTest
Object recognition
Powerful object recognition with a multitude of attributes used to identify objects. Builtin Repository. Dynamic objects recognizedUses xPath, id, name and others, but not very effective. Repository not available. Dynamic objects are difficult to recognize reliably Twice as long to define objects
Image-based object recognitionYesRequires third-party tools and coding to implementDays to build or search for and implement a thirdparty solution
Test Data tablesProvides built-in data tables that automatically handles data types of any level of complexity Requires use of external data tables Days to search for and implement a thirdparty solution
Startup and recovery Built-in and customizable. No. Must be called for each test.No solution in Selenium, each test starts a new browser
Drag & Drop UIYes, making test development quick and easy No, uses a code editor Scripting takes 2 or 3 times longer than drag & drop
Distributed testing Yes, single or multiple remote targets can be controlledPossible, but requires skill-set and coding to configureDays to build (or search for and implement a thirdparty solution)
Scheduling tests Built-in A lot of code must be written to implement schedulingDays to build (or search for and implement a thirdparty solution)
Batch executionYes from Plan, Suite or Test Set Yes, but knowledge required to run through Pom.xml Days to build (or search for and implement a thirdparty solution)
Image capture/compare Yes, includes masking and toleranceRequires third-party tools and a lot of code to implementDays to build (or search for and implement a thirdparty solution)
Integration with SVNBuilt-in, including external non-AscentialTest files A lot of custom code must be written for integration and management of filesDays to build (or search for and implement a thirdparty solution)
Integration with JIRABuilt-in A lot of custom code must be written for integrationDays to build (or search for and implement a thirdparty solution)
Integration with JenkinsYesYes
Test reportsStandard and custom reports available for test results and metrics NoDays to build (or search for and implement a thirdparty solution)
Test managementBuilt-in NoDays to build (or search for and implement a thirdparty solution)
Dependence on access to target applicationTests can be developed using snapshots when target application is unavailable Application must be available to develop tests
Software upgradesZeenyx provided installer User must manually track upgrades for their IDE, webdrivers and any plug-ins being used

Try The Selenium Killer – AscentialTest Solo

Software testing isn’t a puzzle to be solved, it is a job to be done. Now that you have seen the true cost of using open source tools, get started with Solo Tester, AscentialTest’s Selenium killer package.

Share the Post:

Related Content

Instructional Content, Thought Leadership

Build Tests, Not Frameworks

Instructional Content

AscentialTest Error Tracking and Debugging

Instructional Content

Self-Guided PowerBuilder Tutorial

Instructional Content

Returning the Target Application to the Base State

Get Started For Free: 30 Day Trial

Click here to download the Host ID generator. The download package includes full instructions. Please send us your details below to speed the process along.

Our Distribution and Consulting Partners



Loop Software


Matryxsoft Tech


OCS Consulting


What We Are Up To Next

TFS/Azure Integration

Check our new Azure extension for executing AscentialTest command lines in the Azure Marketplace.

We look forward to offering additional integration with Microsoft’s Team Foundation Server and Azure through defect tracking and version control soon.

Omnis Studio

Automated testing support for OMNIS Studio is coming soon. All OMNIS elements will be supported including complex grids, list boxes, tables and tree views.

Custom Objects

Add automated testing support easily for your custom application elements by responding to simple test API messages with json strings. This feature will open up the possibility of testing any GUI element and  will be offered free of charge.

Test Debugger

Set breakpoints in your automated tests, view global and local variables and execute statements to make it easier to debug tests running on any of our supported platforms.

Get a Competitive Quote

Thank you for submitting your inquiry!

We will get back to you ASAP.  We aim to respond to all inquiries within 24 hours.