Cloud & DevOps

DevOps Toolchain

Kubernetes

As we continue into the era of hyperscale Cloud computing and high-frequency DevOps we can examine the increased role and importance of QA and software testing.

A recent research report from Canonical highlights how we are still in the very early days of the shift to ‘Cloud Native’ computing. Pioneered by Netflix, this is an approach to development based on the use of containers, Continuous Delivery and Microservices software architecture, and represents the most significant evolution of enterprise IT that will impact software testing.

Software Testing and Kubernetes

For starters, we can look at Kubernetes and its relationship to testing. Kubernetes is an open-source system for automating deployment, scaling, and management of containerised applications that has become the centrepiece of the Cloud Native movement.

Sauce Labs explores this in What Does Kubernetes Mean for Software Testing.

First, they highlight that the main functionality of a containerised application isn’t different from a non-containerised one, so the functional testing will remain the same. The main evolution is the ability to leverage the environment itself to support the activities of testing.

Packaging an application doesn’t change how these scripts run, but if you have a set of automated tests built, they can be incorporated into the continuous build pipeline used to produce your containers so tests can be performed and reported on with more consistency and much more timely results.

Writing for TechBeacon Glenn Buckholz highlights how Kubernetes enables complete application deployment models to be defined in YAML, enabling the environment to be easily replicated for testing purposes, bringing critical benefits to testers including a decreased dependence on shared environments, direct access to logs, the ability to get to a complicated failure state quickly and cheap disposable environments.

In short, it empowers testers with the same dynamic Cloud-powered capabilities the developers enjoy and enables both to unify their work around the same consistent environment.

Software Testing and Cloud Services

As organisations move to adopt the Cloud, software testing continues to be a component part essential to its success. Indeed as they embrace a DevOps high-frequency release culture it becomes paramount to ensuring this faster working does not come at the expense of degraded quality and increased error rates.

As we wrote in previous blogs, like the role of testing in ‘Infrastructure as Code’ and a Cloud-Native microservices architecture, testing is a practice that can be applied at multiple levels in the new Cloud world, across the whole environment as well as a functional step to managing the release of new software.

This becomes evident as we take a quick, high-level look at testing within the domain of AWS Cloud services. They offer a wealth of blogs on the field, exploring a multitude of sub-topics, ranging from testing mainframe code through how to integrate the practices into your AWS environment and CI workflows through their own tools like CodeBuild and CodePipeline.

In our previous blog on microservices software testing, we highlighted the use of tools like Postman, and in this blog “Automating your API testing with AWS CodeBuild, AWS CodePipeline, and Postman”, they explain how it integrates with these services:

“In the install phase, we install the required Newman library. Recall this is the library that uses Postman collection and environment files to execute tests from the CLI. We also install the jq library that allows you to query JSON. In the pre_build phase, we execute commands that set up our test environment. In this case, we need to grab the Postman collection and environment files from Amazon S3. Then we use a shell script, update-postman-env-file.sh to update the Postman environment file with the API Gateway URL for the API created in the build stage.”

A brief round-up of other experts starts to define the full scope of testing considerations for AWS adoption:

• Test environments – The TestProject team walk through the basics of setting up an AWS test lab.
• Serverless – The Dev Coach explains how to do testing on Lambda, the AWS Serverless product, and Joyce Lin goes further to focus specifically on using it for HTTP API performance testing.
• Mobile – The SoftwareTestingTimes and ASTAQC both look at a use case of testing a mobile application, working with AWS Device Farm, where you can interact with real Android and iOS devices from your browser or run automated tests written in popular frameworks like Appium, Espresso, and XCTest.
• Performance – Distributed Load Testing on AWS helps you easily simulate thousands of users connecting to your application so that you can better understand your application performance under load.
• Security – Rhino Security explains what you need to know about penetration testing of AWS services, covering what can and can’t be tested.

Cloud Scale Testing

Then we can consider a broader context of ‘Cloud Scale Testing’, which can be defined to have two main dimensions: The testing of whole infrastructure operating at a global Cloud scale, and the use of Cloud to implement testing practices at an equivalent scale.

Netflix offers a great tutorial on the first part, in their 2016 presentation Microservices at Netflix Scale.

What this demonstrates is that testing practices can be applied to the whole IT environment, not just isolated application code, and the key to this is primarily the assumption that failures will occur and therefore the system is engineered with a capacity to rapidly adapt to them. Netflix pioneered and operates an approach of ‘circuit breakers‘, implemented through their Hystrix component.

Again automation of these adaptations is critical as is the ongoing destructive testing of the environment to ensure the capability. They implement these tests at a very large scale, simulating the failure of entire AWS regions, demonstrated in visual form at 35m:35s.

Speaking at Cloud Next the founders of Mabl provide an in-depth review of the second aspect – Harnessing the Cloud for enhanced testing capabilities. They talk through deployment across the Google Cloud and how this enables large scale ‘intelligent testing’ features.

Primarily this intelligence refers to adaptability and automation of high volumes of testing functions. For example, testing can adapt to changed forms and code so it doesn’t break when these changes are made, and data analysis is performed to surface insights for testers and developers that eliminates much repetitive grunt work for them. All of this can be integrated into the CI/CD workflow so that it happens automatically as part of the build process.

At the start of the talk Dan Belcher sets the scene for why these new capabilities are needed:

“Development cycles are getting shorter and new features are coming faster than ever… There simply isn’t enough time to test”. The old ways that we had thought about QA are simply not holding up anymore due to the continuous development and delivery. More and more releases are coming our way and the challenge to us as an industry is how can we achieve that velocity while still being able to deliver high-quality products and build our user’s trust.

His presentation states the key improvements needed for software testing in the Cloud era, being able to deliver tests that:

• Adapt seamlessly to applications that change rapidly.
• Will run in the cloud with exceptional scale and speed.
• Produce insights based on the output that is actionable.
• Integrate with automated CI/CD pipeline.

In conclusion, we can see that rather than decreasing the importance of testing, the shift to Cloud Native computing magnifies its importance even more so. Development organisations are creating much larger, much more complex and dynamic code estates than ever before, and being able to assure its performance and reliability is therefore equally important as ever before.

Chaos Engineering

In our previous blog, we begin to set the scene for ‘Software Testing in the Cloud Era’, the pinnacle of which is defined as ‘Chaos Engineering’. We also describe it as ‘Cloud Scale Testing’, as it represents the fusion of hyperscale Cloud engineering and software testing practices.

In their blog for DevOps.com Casey Rosenthal provides a brief introduction to Chaos Engineering and explores the key dynamics of the practice, in particular sharing the key insight, the difference between testing and experimentation: Testing is usually based on a very specific understanding of what is being tested and the anticipated outcomes, whereas as the name suggests Chaos Engineering is about experimenting with unknowns.

As we highlight in the previous blog, pioneers like Netflix routinely instigate major outage situations to understand the cascade effect and recycle that learning back into their architecture.

Testing Maturity

Chaos Engineering represents the maturity pinnacle of Cloud engineering practices, and ultimately software testing too. It would be unwise for any organisation to leap into it without already having a very well developed Cloud capability, and we can see the preceding steps as the maturity journey they can undertake to evolve their testing function to its fullest potential.

For example, in the Cloud Era blog and other previous articles, we examine the fundamental intersection of testing and Cloud engineering, the domain of ‘infrastructure as code’ and the use of the main building blocks like Kubernetes. With the core principle of managing infrastructure provisioning as software, then so software testing practices can be applied.

This sets the foundation for Chaos Engineering, namely expanding the scope of what testing addresses, evolving from only testing application code to regulating the whole IT environment. In our blogs ‘Infrastructure as Code’ and Testing Infrastructure Code on AWS, we introduce the principle concepts and their application on platforms like AWS.

This includes tools like Terratest, a Go library that makes it easier to write automated tests for your infrastructure code, which Yevgeniy Brikman explains in this talk, providing a thorough walkthrough of how to write automated tests for infrastructure code, including the code written for use with tools such as Terraform, Docker, Packer, and Kubernetes. Topics covered include unit tests, integration tests, end-to-end tests, dependency injection, test parallelism, retries and error handling, static analysis, property testing and CI/CD for infrastructure code.

Matt Young describes the fundamental challenge of developing this capability in this blog, most notably cultivating an integrated toolchain that can handle all of these different layers and activities:

“Combining related software development tools in one package is a trend which results in a testing framework like Cypress for example, which is an extension of Selenium, JMeter, Jenkins, and Cucumber, sewn together like Frankenstein’s monster. Until a truly intelligent solution is adopted, we can claim that this is a necessary evil.”