Case Studies

Testing the Back End Through the Front End in an Agile Development Environment
Autodesk, Inc. makes 3D modeling and graphics design software for the architecture, engineering, construction, manufacturing, media, and entertainment industries. Over 100 million people use Autodesk software like AutoCAD, Revit, Maya, 3ds Max, Fusion 360, SketchBook, and more to unlock their creativity and solve important design, business, and environmental challenges. One of their products, Dynamo Studio, is a stand-alone programming environment that enables designers to create visual logic to explore parametric conceptual designs and automate tasks.
Client Contact: Neal Burnham

Executive Summary:

Autodesk selected AscentialTest for test automation of Dynamo Studio and developed a set of cross- platform tests running on both the Web and WPF clients targeted at identifying resource allocation issues in a complex multi-tiered application stack. They successfully found and fixed serious back-end issues by driving thousands of workflows through application engines and analyzing allocation of memory and other system resources to ensure that the system could handle the expected load. With AscentialTest, Autodesk was able to:

  • Normalize thick and thin clients using object recognition capability so that one set of tests can be used across platforms.
  • Validate complex data conversions by parsing XML, console logs and operating system metrics.
  • Verify proper graphical rendering using image capture and compare features.

Client Objective:

When Neal Burnham of Autodesk was tasked with developing a testing solution for Dynamo Studio, he faced both organizational and technical challenges. There was a mindset in place to use open-source tools and to test at the unit level using harnesses embedded in the code itself, but the harness that he inherited did not run reliably. His intuition told him that the most efficient way to identify issues with the backend was to test through the application’s GUI using a commercial testing tool.

Before selecting AscentialTest, Neal looked at open-source alternatives including AutoIt, Selenium and Winium, but he knew from experience that given the complexity of what had to be validated, something much more powerful was required. Any of these solutions could have been used to drive use cases through the GUI, but the target was a validation of the underlying engine that converts graphical programs into other formats. This is where the rubber hits the road. To validate a multi-tiered application, Neal needed to be able to analyze XML, data accessed from the Chrome dashboard and resource allocation metrics from the OS. He also needed to verify that all file contents were correct, and that the graphics were correctly rendered.

The Dynamo GUI is available in two versions. Users can access Dynamo through either a WPF desktop application or through the Chrome browser. To reduce the level of effort to build and maintain tests, Neal wanted to create a meta layer where he could drive real world workflows through both versions of the application.

Our Approach:

Neal selected AscentialTest because he recognized the potential for designing a class of solutions to solve a class of problems. He understood the power of building small, reusable components that could be glued together and scaled up to form a broad solution.

Using AscentialTest’s snapshots, he built a matching set of object definitions to normalize the thick and thin client user interfaces. This layer allowed all of the test components to be shared across platforms. Instead of having to build separate tests for the WPF and Web versions of the Dynamo GUI, he cut his work in half by building one set that could reference either object repository.

Neal also built a validation mechanism to ensure the integrity of numerous complex file conversions. His approach involved parsing data from the Chrome developer console and XML files stored in the file system and comparing sets of data to expected results. He also used AscentialTest image capture and compare features to ensure that graphics were rendered properly.

The ability to have deployed agents on various target machines was also an important component of the solution. Neal needed to ensure that he could drive the GUI hard and fast from multiple client machines in an industrial fashion, sending thousands of workflows through the engines to confirm proper allocation of memory and other system resources necessary for the system to hold up over time.

Because Neal’s solution is not embedded in the application code, he can continue to run the regression tests without being impacted by change as the back-end technologies continue to evolve or be replaced. He believes that one of the strengths of a productized offering like AscentialTest is that he is assured that his test assets are permanent. He cites a recent announcement that the Selenium ‘Keys’ command will be deprecated soon, making it necessary to modify tests to keep them running. Because AscentialTest is a complete solution, there is no need to search for plug-ins and apply workarounds. The Zee testing language is fully formed, and aspects of the product will not be deprecated in the future.

Results Delivered:

Over the course of four months, the set of tests that Neal built with AscentialTest were used to improve the stability and performance of Dynamo Studio. Initially the system failed after only an hour of execution. The tests needed to be bullet-proofed and retry loops were added because it was important to be able to keep the tests running even as the system began to fail. At first small incremental improvements were made. Eventually an incorrect memory setting was found in AWS. It had manifested in a way that the regression caught, but it was difficult to find and fix. Resolving that problem was a big win not only for Dynamo Studio. It also highlighted how process improvement can be made by breaking out of a culture of open source that had almost become religion. Using an alternative approach proved that testing from the front end with a commercial solution can yield benefits that were not obtainable using the strategy that had been become common practice. The fix in this case was not small potatoes. Neal estimates that it was worth hundreds of thousands of dollars to the bottom line.

The Dynamo team also found that AscentialTest brought immediate value in an unexpected way when a Mongo database was hacked, and all of the test data lost. Fortunately, Neal had built ‘tests’ that were capable of restoring data. He said that capability “really got people’s attention” and even naysayers were stunned when it came to the rescue a second time several months later.

Being able to start building tests immediately without having to spend a lot of time building a test harness is something Neal appreciates about AscentialTest. He found that he can whip something up that can be used to identify a memory leak in a half hour. He considers it a huge win for development to be able to build tests that have a limited shelf life because the level of effort is so small. That works well in an Agile, CI environment where the feedback cycle is short.

When compared to an open-source solution, Neal found that AscentialTest is not limited. When the inevitable stumbling blocks occur, there is a dedicated technical support staff to call on along with a user forum in the Zeenyx eco system that is very targeted. While there is a huge base of Selenium users, the available help is un-curated. Neal says that in that environment “you have to do all the work, whereas with AscentialTest, there’s one place to go for an answer”.

Tracking Performance, Verification of Uploads, and maintaining dynamic object IDs
Starchive is a cloud-based media management application designed to create organization in the midst of the media chaos and challenges. Started from their collaboration with the Bob Dylan Music Company, the web-based software application allows individuals, companies, and creative brands to solve media challenges by making it easy to find, manage, or share media within thousands of files. Starchive is constantly adapting to bring the best user experience forward so that the everyday user can utilize the application and maneuver the intricacies of media handling.

Executive Summary:

Starchive, a client of Loop Software Services and Testing, recently selected AscentialTest for its web-based application testing. Starchive made the switch for a variety of reasons and through AscentialTest, Loop was able to provide Starchive with the ability to:

  • Track performance of upload, download, page load times to gain insight when new features or backend changes impact performance.
  • Easily maintain the automated suite while Starchive’s object IDs dynamically change
  • Verify large upload sets of 10,0000+ which was not feasible when done manually
  • Shorten overall test result, test building, and maintenance time.

Client Objective:

When Starchive first made the switch from manual to automated testing, it was not without challenges, both general and technical. When it came to the performance data of their application, there was no historical data. At that point in time it was not cost-effective to gather data on Starchive’s application performance, therefore it was not always known how a new feature or change affected the application performance. When creating new features and changes in the backend, regressions were run to ensure the changes did not create any breakage, but other than apparent positive or negative performance changes, no performance metrics could be easily tracked.

Another challenge that arose was related to Starchive’s React Framework architecture which creates code generated classes for delivering asynchronous experiences. This caused the application’s object ID’s to be dynamic, making open-source automation testing solutions extremely difficult to maintain. Automated tests consistently broke, leaving Starchive concerned about their automated product testing and the costs to maintain it.

Creation and maintenance of the regression suites presented a unique challenge, as Starchive’s highly effective development team was able to rapidly create new features and backend updates. The application’s mission, storing massive file sets and handling extremely large uploads/downloads has led to issues with verification of uploads/downloads during testing, with the manual approach unable to verify such massive file sets in a cost effective manner.

Our Approach:

Starchive chose to implement AscentialTest based on the recommendation of Matt Osiecki, Head of Automation at LSTS as he saw the large benefits that Starchive would see by choosing this solution. Because of his recommendation, Starchive began utilizing AscentialTest. The initial automation build was focused on performance testing.

AscentialTest is now used to track performance times when uploading, downloading, as well as general page load times within Starchive’s application. The tests are run daily which document performance times in a client facing dashboard and the results are compared to daily, weekly, and monthly results. Any test results that are abnormal are automatically reported to the client within an “Automation” Slack channel. The performance tests are now able to run after any change in the application on both frontend and backend which allows Starchive to see the immediate performance impact of changes.

Another position in which AscentialTest is bolstering Starchive is in its ability to identify objects based on innumerable attributes such as color, position, webtext, parent objects, and others. Dynamic IDs which would normally break open sourced framework automation can be defensively defined within AscentialTest, giving it the definite advantage.

The ability for rapid development was another significant advantage of using AscentialTest. The test case creation and maintenance within AscentialTest proved to be much more efficient than when compared to manual or traditional automated frameworks.

Regarding maintenance, the tests built in AscentialTest are composed of steps and even sub-steps. These building blocks are then used across a variety of different tests. When a change occurs on the frontend, the step that is impacted can be updated, which updates all references and tests containing the step. Because Starchive consistently rolls out new features, a feature change that affects 1000+ test cases would take days to update within a manual suite, but can be done in minutes with AscentialTest’s prowess.

With reference to the creation side of things, Step, the building blocks of tests, can be reused within a test or across multiple tests. Once the steps are created within AscentialTest, it is a simple process to drag and drop them into a new test, instead of building an entire new test from scratch. This means that reusing previously created steps can  significantly reduce the hours spent on test creation.

Starchive also shows huge improvements in the verification process via AscentialTest. Before, verifying upload/download/management of thousands of files would be cost-prohibitive and nearly impossible, with the likelihood of  human error  if done manually. Now, a verification step can be created to verify a file’s image, title, created by date, along with 30+ other properties. This step can be rerun on every file uploaded/download/edited, verifying each file in seconds, while not consuming person hours.

Results Delivered:

The effects of AscentialTest were seen straight away. There was immediate validation of new features and backend change impact on performance which was directly reported to the Starchive team within Slack. Improved user experience and adoption also rose and performance pain points are now able to be identified (slow uploading time, long page load time) via automation and remedied quickly by the development team.

When compared to open-source solutions, AscentialTest’s automation suite proved that it is resilient to dynamic ID changes, saving time and money on maintenance. Less maintenance also means that Starchive is able to obtain the test results more quickly.

The Starchive team and Loop found that the regression suite can be built and maintained much faster when compared to manual testing, saving Starchive considerable monetary investment. The automated suite is faster than manual regressions, completes more test cases with each of them covering more scenarios.

The file set verification system ensures Starchive that any files set, file type, file set size, can be uploaded, edited, and downloaded within Starchive while retaining all original data. Any issues found during file set verification can be remedied before the push to production. AscentialTest has saved Starchive time and money while opening up new testing possibilities.

Philips Significantly Reduces the Cost of Test Automation
Philips is a leading health technology company focused on improving people’s health and enabling better outcomes across the health continuum from healthy living and prevention, to diagnosis, treatment and home care. Philips leverages advanced technology and deep clinical and consumer insights to deliver integrated solutions. Headquartered in the Netherlands, the company is a leader in diagnostic imaging, image-guided therapy, patient monitoring and health informatics, as well as in consumer health and home care.
Client Contact: Vinay Krishnamurthy – Analytics Manager
Increase in Localized Languages
0
Reduction in Test Execution Time
0 %
Increase in test automated
0 %
Framework development time eliminated
0 mos

Executive Summary:

Philips selected AscentialTest for test automation of PACS, a software platform for accessing a patient’s radiological history, including diagnostic reports, exam notes, clinical history and images. The test team developed a set of cross-platform tests running on both thick and thin clients, saving a significant amount of time and effort as compared with previous testing tool solutions. With AscentialTest, Philips was able to:

  • Normalize thick and thin PACS clients using object recognition capability so that one set of tests can be executed across three platforms, localized in 6 different languages.
  • Eliminate the need for a testing framework, which in the past had taken 6 to 8 months to build and stabilize.
  • Increase the number of tests that could be automated by 15% because of image and text recognition features.
  • Reduce the average time to automate a test scenario from 6 days to 2 1/2 to 3 1/2 days. Reduce test execution time by 50%.
  • Achieve maintenance goal of 5% of test creation time.

Client Objective:

When Vinay Krishnamurthy of Philips was tasked with developing a testing solution for PACS, he faced a number of technical challenges. The target application is deployed in three different versions to meet the needs of various types of end users. The GUI is a mixture of MFC and web controls, deployed as both thick and thin clients. These versions are then translated into 6 different languages for localized markets. Test verification requires a lot of comparison of images, along with text that is drawn upon those images to report measurements and other attributes of the scans.

Developing a separate set of test for each of the deployed versions of PACS was cost prohibitive and the organization wanted to avoid building a testing framework, which in past projects had taken 6 to 8 months to build and stabilize with existing commercial and open source tools. They were also concerned about the huge maintenance costs that they had experienced.

Furthermore, with more than 8000 tests to automate Philips wanted a scriptless solution, but had learned from experience that a record-playback tool would not provide the robustness that was required.

Our Approach:

Philips selected AscentialTest because they recognized the potential for designing a solution that could be used across platforms. Using AscentialTest’s drag and drop interface, the team uses snapshots to create object definitions that are recognized across all three platforms. They benefit from an array of options for defining objects that were difficult to recognize with other testing tools that they evaluated. Philips heavily relies on the ‘RenderText’ and ‘RenderImage’ capabilities of AscentialTest. ‘RenderText’ allows them to validate text that is drawn on scanned images. In the past, tests that require verification of that text had to be completed by manual testers. Vinay estimates that the team can automate 15% more tests because of this text recognition feature. Likewise, ‘RenderImage’ has allowed for the recognition of images for both navigation and verification of tests.

Because of the image-centric domain of PACS, Philips requires very sophisticated image ‘capture and compare’ capability. Zeenyx worked closely with Philips to enhance the existing image verification capabilities of AscentialTest. Several features were added to provide the ability to more specifically focus on the ‘area of interest’, apply masks, and tolerate for pixel and color differences.

Philips also cites AscentialTest data options as a critical component to allow the tests to be shared across platforms. By making use of ‘Project Data’, ‘Plan Data’, and ‘Attributes’, Philips is able to easily change key pieces of test configuration data in a single location to launch tests against different versions of PACS in the various test environments.

Vinay explains that the team creates two files for each page in the target application, one with the object definitions and a second with a meta-layer of steps for driving and verifying the application. Test developers use those components to build the automated tests via drag and drop.

Vinay says that the pace of automation has increased because of the reusable nature of the test components. As the more objects and steps are created, the time required to build new tests continues to decrease.

The bulk of the tests that the team has created to date are used for their ‘Smoke’ and ‘Regression’ tests runs. Recently they created a ‘Sanity’ test comprised of a handful of tests that were ‘ported’ to test six localized versions of PACS, created for different language markets. The team was surprised that it took only two days to port a test to all six languages using AscentialTest’s ‘Translate’ feature.

Results Delivered:

Philips has been able to exceed their goal of reducing the time and cost of developing automated tests by 50%. Using AscentialTest, they have reduced the average time to build an automated test scenario from 6 to 2 ½ to 3 ½ days. They have also been able to cut execution time in half. The automation team has integrated AscentialTest with TFS to launch tests and have taken advantage of a recent AscentialTest feature where ‘failed’ tests can be easily selected and rerun without user intervention.

Philips credits the support that they received from both Zeenyx and MatryxSoft, their local distributor, for their success with AscentialTest. We are happy to have exceeded their expectations for response time for fixes and enhancement requests.

Transitioning from manual to automated testing in a PowerBuilder Environment
NCC develops and supports the Nutrition Data System for Research (NDSR), a dietary analysis software application widely used for the collection and analyses of 24-hour dietary recalls and the analysis of food records, menus, and recipes. Hundreds of researchers at institutions across the United States rely on NDSR for an array of nutrition-related studies.
Client Contact: Gordon Weil

Executive Summary:

The Nutrition Coordinating Center at the University of Minnesota selected AscentialTest for test automation of its Nutrition Data System for Research application and developed over a thousand automated tests which:

  • Reduce the testing cycle from two to three weeks to one to two days.
  • Validate hundreds of calculations that would require trained staff to complete.
  • Raise the level of confidence in new releases of the software.

Client Objective:

Gordon Weil of the Nutrition Coordinating Center found that testing NDSR manually is time-consuming. It takes 2 to 3 person weeks to run a regression test on one version of windows, but since they support multiple versions at any given time, it just isn’t feasible, especially given that the organization doesn’t have a dedicated testing staff. Gordon found that there weren’t people available to execute the tests when he needed them.

He began investigating automated testing solutions. He tried a few open source and commercial tools, but none of them worked well with PowerBuilder, upon which NDSR is built. He also needed to find a solution that was easy to learn since he planned to use students to build the automated tests.

Our Approach:

Gordon selected AscentialTest both because it provides support for PowerBuilder and because it’s easy to learn and use. Students were hired to convert their well-documented manual tests into automated tests. Gordon said that he spent only an hour or two training the students to use AscentialTest. Over the course of a summer, two students automated 1075 tests. The regression suite comprises a comprehensive set of tests that cover all aspects of the NDSR application. When the project started, NDSR was built with PowerBuilder 12.5, but was recently converted to PB 2017. Tests are separated into different plans by functional area. One plan, which runs for a couple of hours, generates a thousand reports which are compared to baselines that have been previously captured. Reports contain a large number of calculations that need to be validated. AscentialTest is used to find any differences that are later examined.

Results Delivered:

The test cycle for the original set of tests was reduced from 2 to 3 person weeks to 1 to 2 days. Additional tests have since been added to increase coverage. The regression is primarily used to ensure that when changes are made, nothing is broken. It occasionally uncovers a problem, but even when it doesn’t identify an issue, it provides confidence to the organization that the application is going to perform well for its customers. Prior to using AscentialTest, adding a new report to the system was costly because validation requires a huge amount of labor and time. Some of the tests would need to be executed by a trained nutritionist, whose time is very valuable to the organization. Gordon estimates that $10,000.00 is saved every time a regression needs to be executed and the feedback loop is significantly shorter. He no longer has to wait the 2 to 3 weeks that it would take to run the same tests manually.

Get Started For Free: 30 Day Trial

Click here to download the Host ID generator. The download package includes full instructions. Please send us your details below to speed the process along.

Our Distribution and Consulting Partners

Appeon

Cigniti

Loop Software

Marlabs

Matryxsoft Tech

Novalys

OCS Consulting

Loading...

What We Are Up To Next

TFS/Azure Integration

Check our new Azure extension for executing AscentialTest command lines in the Azure Marketplace.

We look forward to offering additional integration with Microsoft’s Team Foundation Server and Azure through defect tracking and version control soon.

Omnis Studio

Automated testing support for OMNIS Studio is coming soon. All OMNIS elements will be supported including complex grids, list boxes, tables and tree views.

Custom Objects

Add automated testing support easily for your custom application elements by responding to simple test API messages with json strings. This feature will open up the possibility of testing any GUI element and  will be offered free of charge.

Test Debugger

Set breakpoints in your automated tests, view global and local variables and execute statements to make it easier to debug tests running on any of our supported platforms.

Get a Competitive Quote

Thank you for submitting your inquiry!

We will get back to you ASAP.  We aim to respond to all inquiries within 24 hours.