GCP and the Future of Software Testing

In this video, Dan Belcher who is one of the founders of Mabl (test automation platform) along with Joseph Lust, one of the engineers at Mabl, explain the Artificial Intelligence to improve QA effectiveness and how it has helped the QA industry to leave behind the antique tooling and exhausting manual processing.

Software Testing Needs to Evolve

At 1:09 Belcher explains how the need to evolve software testing became the need of hour with the industry shifting to the rapid pace of continuous integration, delivery and the fast-growing use of DevOps. Due to such circumstances, the team set out to build Mabl. He explains it further by quoting a QA leader, “Development cycles are getting shorter and new features are coming faster than ever… There simply isn’t enough time to test”.

The old ways that we had thought about QA are simply not holding up anymore due to the continuous development and delivery. More and more releases are coming our way and the challenge to us as an industry is how can we achieve that velocity while still be able to deliver high-quality products and build our user’s trust.

The Era of Intelligent Testing

From 6:07 Belcher starts to give the idea of introducing the Era of Intelligent Testing. He further states 4 principles for intelligent testing which are explained one by one as follows:

Tests adapt seamlessly to change

At 7:37 Belcher states the first principle that says the tests must adapt to change. In legacy testing frameworks, the tests used to crash because of some changes to the front end. So, the problem that tests don’t break when the user experience isn’t affected because false tests can create all kinds of dysfunction. That is one of the reasons Mabl was created to be intent-driven.

With the Mabl chrome extension, a QA analyst or anyone else uses the application as they would expect their end users to. By this, Mabl captures the intent of their actions and creates tests based on these actions.

From 9:15 the actual working of the Mabl is explained to adapt to certain changes. With Mabl, we have implemented heuristics that use the intent to click a button (say it “Dismiss”). But unlike the originally used locators, Mabl will find the button using heuristics, even if the button was not where it used to be, making sure the test continues to pass. And then if it does, we will update the test. By this, we can create tests that can survive the frequent changes.

At 10:38 Joseph begins to explain how that test will work on GCP (Google Cloud Platform) by a continuous and seamless workflow. Referring to the workflow diagram, Joe indicates how the tests stored in Cloud Data storage get loaded in testing cluster in Kubernetes which runs loads of parallel instances of the used browser. These instances now produce tons of observations which go through Cloud Pub/Sub and into the Cloud Dataflow.

In the Cloud Dataflow, we can determine if something odd happened? Could we not find the button? Say, in case if we really don’t know where the button is. In that case, we would go out further to the Cloud Functions and send user a message saying that something went wrong. But in most cases, we can start closing the loop from Cloud Dataflow again to Cloud Data Storage.

We can figure out that “Dismiss” is the correct button and we can change the script and update the test that this is the button you need to click. We then feed this information to our data store, update the script and the next time we run the test, we know exactly where it is and we won’t have to adapt again. By doing this, we seamlessly adapt and are able to achieve continuous testing. That is how we create tests that survive changes to the structure of page, to the IDs of elements and to the CSS classes.

Tests run in the cloud

From 13:07 Belcher explains how 3 years from now, test codes will be running on cloud instead of manual environments. He outlines the efficiency of GCP and GKE (Google-Kubernetes-Engine) by comparing the 50 test scripts executed in a day using a manual method that will take approximately 1260 compute hours per day, with that of Mabl in the presence of many Kubernetes Clusters that would take minutes to do that.

Using these tools, we are easily able to take the historical Selenium Infrastructure and just seamlessly scale it in the cloud.

From 19:12 Joseph shows the real time example of how it would work on Mabl. Using the Chrome and Firefox browser, Mabl tries to find ‘shirts’ and ‘pants’ in a provided web app frontend. Taking the screenshots, all of this is happening in parallel within the seconds of time.

Test results are insightful

At 21:04 Belcher now drives a conclusion from previous two principles that as we are dealing with tons of test over a given release cycle, we cannot possibly focus on each test running as a separate. As to see if it is running effectively. We only care to know if it broke the product. Otherwise, we are going to be on the next feature. That is what Mabl actually does. It looks at a variety of aspects of quality of the application instead of only checking its HTML or working on screenshots. Then it decides whether or not it is working based on those insights.

At 22:12 he explains about the pipeline structure of Mabl, that captures all kinds of data, the data is then fed to the analytics pipeline and then it generates insights based on either of those heuristics we have implemented.

From 22:34 Belcher gives the two examples of the insights that this pipeline gives us. The first being heuristic-driven insight, which includes the detection of broken links and the client-side JS errors. The second is model-driven insight, which comes with time anomalies and visual anomalies. Mabl can detect the time difference e.g., the page loading time, test execution time and step time. It then predicts how long it will take to run test the next time. And if the actual observation is different than the prediction, Mabl will surface an insight based on that.

At 24:40 Belcher talks about the visual anomalies. For example, if we run a test to see if there are any visual anomalies associated with the page, the Mabl will only see the visual changes that will occur in the non-dynamic region, because this is where the change appears unusually. And only then, it will surface that as an insight.

From 26:14 Joseph explains the technicality of serverless scaling insights by a workflow model. It starts with the Kubernetes Engine which is currently churning through thousands of web pages making continuous observations coming out of it. Then these observations are fed into different pipelines which include continuous prediction, continuous training of data and the bulk training. In first one, the continuous prediction pipeline, Cloud Dataflow works on the data, transforms and feeds it to Cloud ML. There, if it detects any visual anomaly or time anomaly, it pushes through Cloud Functions which notifies the user by a message to slack.

That feedback also goes as an insight to the next pipeline, the continuous training pipeline, where it does a little bit of cleanup in Cloud Dataflow and then again by Cloud ML, updates the model, makes it adaptable and tunes the insights. These insights are then put in Cloud Data Store. So, these insights are continuously trained and predicted again and again.

We also have the Kubernetes Engine, where the bulk training happens if a new model comes in or we have to again churn through 100 thousand pictures. We can do it through a ML Script. By updating everybody’s models, we now have a something very useful and challenging. We can now send it to the continuous prediction and training pipeline. All of this being perfectly happening serverless. That concludes the third principle that test results need to be more insightful.

Testing is a part of continuous pipeline

From 30:00 Belcher concludes the discussion by stating that testing should be a part of continuous delivery pipeline. In this era QA cannot be off to the side. It needs to be the fully automated part of the pipeline.

From 30:47 Joseph then scales the process of continuous delivery. In which he shows how we are going to create continuous testing and continuous deployment tied together to get us a testing feedback loop.

For example, we commit a code to GitHub. That can trigger a deployment trigger in Cloud-Build, which gets our code automatically from Git. It then builds it, deploys it and tells Mabl there’s a new deployment available via its API. That will then comeback and test our app. And if it is successful, it can call back to Cloud Build and keep propagating through our deploy, test, deploy, test, deploy, test cycle.

To show what that works like, at 31:30 Joseph shows a demo for the above-mentioned example using a simple calculator app for deploying. Going through the same steps as mentioned above, Cloud Build gets our code, calls Mabl’s deployment event API and starts building our web app. It then run some tests across multiple browsers like Chrome, Firefox. We have some 100 instances pumping in Google Cloud that just turned on for these two browsers.

They are running all our tests in parallel and it literally just took under 2 minutes. While in a classic environment, that would have taken three hours and all this was driven by somebody making a single commit.

Learned Milestones

At the end. Belcher concludes the presentation by saying that there is hope for QA industry in the near future, to keep up with the pace of development. As an industry we are going to find a way to deliver tests that:

• Adapt seamlessly to Applications that change rapidly.
• Will run in the cloud with exceptional scale and speed.
• Produce insights based on output that are actionable.
• Integrate with automated CI/CD pipeline.

And it is all powered by and possible because of Cloud Computing and Google Cloud