Table of Contents
Apparently, Commercial Software Testing Tools are Too Expensive
What a Thousand G2 “Dislikes” Taught Us
Over the last few weeks, we analyzed over a thousand reviews entered into the “Automated Software Testing” category on the G2 website. While we prefer relying on Gartner Reviews (they tend to be a little more comprehensive), there simply aren’t enough of them available to draw meaningful insights. Gartner only lists about 73 vendors, and only a dozen or so of the reviewed tools have more than 10 reviews. G2, on the other hand, covers about 231 vendors and contains about 13k reviews. So, while there may be a lot of debate about the value and veracity of the G2 reviews, it is great place to start.
We categorized the “dislikes” in the following buckets:
- Buying it and getting started (cost, onboarding etc.)
- Learning Curve
- Poor UI
- Lacking Features
- Test Creation (complex object recognition etc.)
- Test Execution
- Test Management & Data Management
- Test Reporting
- Buggy Releases
- Support/Learning Resources/Community
- Slow or Unstable Execution
- Resource Allocation Issues
Leading Dislike #1 – “Great software, but The license cost is just too high”
While many of the expressed dislikes were nuanced, idiosyncratic and somewhat difficult to categorize, several stood out as 1) frequent and 2) easy to categorize. They were – cost, support & learning resources, test creation issues and test object recognition/synchronization issues.
Here, we will address the cost issue, and hopefully provide the solo testing community with a full-featured testing tool at a reasonable cost.
Many of the dislikes related to pricing also mention the ready availability of Selenium, a free, open source solution for testing, albeit with severe limitations relative to commercial tools. And as best as we can tell, we have not seen another enterprise-level tool that attempts to provide anything remotely price competitive with free.
The Selenium Conundrum – When Free Isn’t Free
Why isn’t it free? Simple: the cost of labor swamps the cost of your testing software. Consider that in any test automation project, the fee for the tool is only a small part of the overall budget when compared with the cost of building the tests. So if you pay $10,000.00 for a testing tool and $100,000.00 in labor costs, have you saved money for your organization by selecting an open source tool, when it costs more than twice as much to build and maintain the tests?
Selenium Increases Labor Costs Relative to Good Testing Software
Nowhere will you read that Selenium is an awesome testing tool. You won’t read that it is easy to use or that the tests are easy to maintain. If there were a charge for Selenium, it is unlikely that it would be selected at all when compared to tools like AscentialTest that have been designed to promote productivity and provide a substantial return on investment in the first year of usage.
How Do We Know?
The AscentialTest team has been creating software testing solutions since the early days of the software testing industry (Segue, where we created Silk Test, was founded in 1988). Brian, our co-founder, was the head of R&D at Segue, and Dave, our co-Founder, was the Chief Architect at Segue. They also formed our own consulting firm (Star Quality) in 1998, specializing in automated software testing. We have seen it all.
Don’t Believe Us? Ask Google:
Query – What are the downsides of using Selenium for automated software testing?
Answer:
- Not an All-in-One Solution
- Steeper Learning Curve
- Limited Desktop and Mobile Support
- Maintenance Challenges
- Browser Dependency
- Limited Image Testing
- Lack of Built-in Reporting
- No Technical Support
An Example: Browser Navigation
When a user navigates your target application, do they close and reopen the browser after each transaction? Probably not. But when you run a test with Selenium, each test opens its own private browser. That limitation leaves out a lot of usage cases. It also makes it very difficult to debug a test. If a test fails several minutes into the execution, there is no way to run just a part of that test, because Selenium needs to start a new browser with each test run. The productivity loss that results from just this one limitation is astounding. If AscentialTest worked that way, we’d be out of business.
The image on the left below depicts a typical test step in AscentialTest. The one on the right
displays a function that is part of a Selenium test. What do you want your resources spending time on: testing or writing more code? Do you really want to spend money on programming to create automated tests. You decide.
AscentialTest vs. Selenium: Feature, Function and Productivity Comparison:
While it’s true that Selenium is a free set of open source tools, using them is not really free after all. The costs and benefits of test automation are significant. We’ve worked hard to provide a testing solution that makes testers more productive, saving your organization time and money. If you still think Selenium is free after reading this document, we have some lake- front property in {name your location} that we would just love to sell to you.
| Features | AscentialTest | Selenium | Time Cost |
|---|---|---|---|
| Tool setup and configuration | No setup or configuration | Requires knowledge of required development environments and programming concepts | Several Days |
| Test framework | Automatically generated. Users begin creating tests right away | Must be custom-built and then tested. Frameworks can take weeks or months to complete. | Several Weeks |
| Browser support | Edge, Firefox, Chrome. Handles all web application components including dynamic tables | IE, Firefox , Chrome, Opera. Does not handle all web components, e.g. popup windows, dynamic tables | Varies based on complexity |
| Non-browser based application support | Yes, including windows, dotNet, PowerBuilder, Terminals, Java and many others | No | Days to build (or search for and implement a thirdparty solution) |
| Coding experience | None required | Yes. Requires experienced staff with skills to integrate the framework and code tests | Skillset is costly |
| Product support | Zeenyx support provided | Open source community along with user forum | Support is not curatated so it can take hours or days to find solutions |
| Software cost | License and annual | Free maintenance fee | |
| Script generation time | Drag and drop accelerates | Scripting is time-consuming test development | Ongoing costs are high, easily take 4 or 5 times longer to develop tests |
| Test maintenance | Abstraction and encapsulation results in high reuse and low maintenance | High, tests are difficult and costly to maintain | Ongoing costs are high, Easily take 4 or 5 times longer to maintain tests |
| Test Execution Time | 30 to 40% faster than Selenium | Execution time is slow | Tests run 5x faster on AscentialTest |
| Object recognition /repository | Powerful object recognition with a multitude of attributes used to identify objects. Builtin Repository. Dynamic objects recognized | Uses xPath, id, name and others, but not very effective. Repository not available. Dynamic objects are difficult to recognize reliably | Twice as long to define objects |
| Image-based object recognition | Yes | Requires third-party tools and coding to implement | Days to build or search for and implement a thirdparty solution |
| Test Data tables | Provides built-in data tables that automatically handles data types of any level of complexity | Requires use of external data tables | Days to search for and implement a thirdparty solution |
| Startup and recovery | Built-in and customizable. | No. Must be called for each test. | No solution in Selenium, each test starts a new browser |
| Drag & Drop UI | Yes, making test development quick and easy | No, uses a code editor | Scripting takes 2 or 3 times longer than drag & drop |
| Distributed testing | Yes, single or multiple remote targets can be controlled | Possible, but requires skill-set and coding to configure | Days to build (or search for and implement a thirdparty solution) |
| Scheduling tests | Built-in | A lot of code must be written to implement scheduling | Days to build (or search for and implement a thirdparty solution) |
| Batch execution | Yes from Plan, Suite or Test Set | Yes, but knowledge required to run through Pom.xml | Days to build (or search for and implement a thirdparty solution) |
| Image capture/compare | Yes, includes masking and tolerance | Requires third-party tools and a lot of code to implement | Days to build (or search for and implement a thirdparty solution) |
| Integration with SVN | Built-in, including external non-AscentialTest files | A lot of custom code must be written for integration and management of files | Days to build (or search for and implement a thirdparty solution) |
| Integration with JIRA | Built-in | A lot of custom code must be written for integration | Days to build (or search for and implement a thirdparty solution) |
| Integration with Jenkins | Yes | Yes | |
| Test reports | Standard and custom reports available for test results and metrics | No | Days to build (or search for and implement a thirdparty solution) |
| Test management | Built-in | No | Days to build (or search for and implement a thirdparty solution) |
| Dependence on access to target application | Tests can be developed using snapshots when target application is unavailable | Application must be available to develop tests | |
| Software upgrades | Zeenyx provided installer | User must manually track upgrades for their IDE, webdrivers and any plug-ins being used |
Try The Selenium Killer – AscentialTest Solo
Software testing isn’t a puzzle to be solved, it is a job to be done. Now that you have seen the true cost of using open source tools, get started with Solo Tester, AscentialTest’s Selenium killer package.
