Writing for DevOps.com Alex Husar offers a great starting point for this. He starts with the key insight, that testing can be guilty of focusing on how tests are performed, not what result they produce, meaning situations can arise where you are scoring 100% test pass rates but still they are not really strong enough.
Alex proposes five categories of metrics. The first two could really be considered part of the broader DevOps framework – User satisfaction and process performance, such as how long it takes to translate tasks into deployed code.
In the other three he then zooms in on testing specifically, describing:
- Coverage metrics – The amount of code that is tested.
- Code quality – Define an effective method for assessing quality, such as identifying legacy debt.
- Bug or incident metrics – Develop an accurate system for handling and reporting incidents.
What is important about this approach is that while they deal with testing they do so in a manner that contributes to the overall DevOps performance, acting to yield insights that can help improve the quality and throughput of software development.
Tricentis also provides a guide to developing testing metrics, offering a detailed system that expands to a larger, granular level of reporting ideal for a large enterprise dealing with the complexity that arises from their size.
They break them down into two main types: Result Metrics – an absolute measure of an activity/process completed, and Predictive Metrics – metrics that are derivatives and act as early warning signs of an unfavourable result, and then populate these with 64 specific measures, such as the number of defects found and the number of bugs found after shipping.
They also suggest metrics such as coverage, quality and testing effectiveness, and as they explain through various formulae all of these data points can then be utilised to report on desired outcomes including financial and business factors as well as software.
This is particularly useful as large enterprise CIO’s need to be able to quantify important business factors such as the total budget for testing, the cost per test, and so forth so that they can best manage the ROI and report to the board in a meaningful manner.
Furthermore, their approach addresses other complexity challenges, such as managing the changes made to large scale systems, where they define metrics such as the ‘Defect Injection Rate’ – the number of problems attributable to new changes made.
Knowing this number will help predict the number of defects that could be expected per new change, enabling test teams to strategically use retrospective meetings to understand their capacity to help identify and fix defects coming from new changes.
