Here is the latest article in the eWEEK feature series called IT Science, in which we look at what actually happens at the intersection of new-gen IT and legacy systems.
Unless it’s brand new and right off various assembly lines, servers, storage and networking inside every IT system can be considered “legacy.” This is because the iteration of both hardware and software products is speeding up all the time. It’s not unusual for an app-maker, for example, to update and/or patch for security purposes an application a few times a month, or even a week. Some apps are updated daily! Hardware moves a little slower, but manufacturing cycles are also speeding up.
These articles describe new-gen industry solutions. The idea is to look at real-world examples of how new-gen IT products and services are making a difference in production each day. Most of them are success stories, but there will also be others about projects that blew up. We’ll have IT integrators, system consultants, analysts and other experts helping us with these as needed.
Today’s Topic: Walmart Embraces Test Automation and Open Source
Name the problem to be solved: Walmart has long been a disruptor in the retail industry by revolutionizing the supply chain to offer lower prices. They also continue to make big investments in their digital strategy to help customers save time.
Throughout this transformation, Walmart Labs (Walmart’s technology arm) was finding it difficult to seamlessly transition away from legacy practices. There was a misalignment on goals across the organization, and teams were using their own tools and technologies.
They found themselves relying on time-consuming manual testing, which only allowed updates to deploy twice per month on a three-week development cycle, leaving them unable to keep up with the pace of app improvement they wanted.
Describe the strategy that went into finding the solution: They knew they needed to increase application quality and increase deployment velocity with zero friction for the dev teams, which led to the adoption of automated testing.
In their move to automated testing, Walmart realized there were no test management tools that worked at the scale that they required. So in 2015, they started developing Test Armada -- an open source quality automation platform developed to support native, web and backend testing services. This now is a central tool that supports functional testing, performance testing, mockups and data insights across all Walmart.com, grocery and backed store systems.
Walmart also understood that in order to be successful they needed to change the entire culture of how their teams developed and tested their apps. This led to the creation of their Center of Excellence: the DXT (datawallet) team. Their focus was to choose the best testing frameworks and processes to complement Test Armada. Once selected, they standardized and rolled these out to various teams by having everyone working similarly.
Furthermore, because their tests were completely manual, teams weren’t able to scale testing to ensure coverage across the multitude of browsers, operating systems and mobile devices that their customers would be accessing their apps from.
Walmart realized that open source software gave them the latest tools, flexibility and support they needed. So they chose the open source frameworks Selenium, Appium, Espresso and XCUITest for automated testing.
List the key components in the solution: These open source frameworks allowed them to develop automated tests with widely adopted frameworks used by hundreds of thousands of organizations worldwide. This was important because they could build and scale an automated testing organization knowing that they could hire engineers that had experience in these technologies who could be productive immediately.
Walmart also leveled up its testing by implementing the Sauce Labs Continuous Testing platform, the world’s largest cloud-based solution for web and mobile app testing. Sauce Labs gave them access to hundreds of browser and operating system combinations, increasing test coverage and saving time.
Test Armada imports data from their own runners and Sauce Labs test details pages to display a quality dashboard so QA and engineers can quickly view the quality status of their builds.
Describe how the deployment went, perhaps how long it took, and if it came off as planned: This change in culture and process left not only the technical teams happy and more productive, but it has also made the executive team excited. Within a year’s time, the efforts of the DXT team, along with the implementation of Sauce Labs, saved 750,000-plus man hours that would have been spent on manual testing/updating support for new browsers and operating systems.
Describe the result, new efficiencies gained, and what was learned from the project: As Walmart standardized on testing frameworks and began managing them with Test Armada, Sauce Labs provided the tools that allowed it to scale testing, enhance productivity, improve code quality and speed delivery. And with these systems in place, each team at Walmart knows that they are using the best tools, and they don’t have to constantly reinvent the wheel when they add a new app to their suite.
With Continuous Testing practices in place, teams at Walmart are deploying updates to their apps daily on every front end build. This velocity is only possible with the confidence knowing their code is well-tested and covered across every browser/OS/device combination.
Describe ROI, carbon footprint savings, and staff time savings, if any: Today, Walmart has more than 40 different projects around the world using Test Armada to manage all of their tests. Used in conjunction with the Sauce Labs Continuous Testing Cloud, these teams have run more than 14 million tests in the past seven years--or about 50,000 tests per day currently. These tools have given teams the time to build out more sophisticated automation, with an increase from 200 test cases in 2016 to 500 test cases with every build to production, in the case of one team.
Thanks to Sauce Labs, Walmart is also able to ensure that their apps work on over 700 browser/OS combinations, 172 device emulators and over 300 unique real devices, and all without having to manage any infrastructure internally.
However, this increase in quality and coverage has not come at the expense of speed. With Continuous Testing practices in place, teams at Walmart are deploying updates to their apps daily on every front end build. This velocity is only possible with the confidence knowing their code is well-tested and covered across every browser/OS/device combination.
If you have a suggestion for an eWEEK IT Science article, email [email protected].