Why should we use QA Metrics?
As an indispensable part of the IT world Quality Assurance (QA) has become an important institution in developers’ and testers’ lives. As modern websites and apps have become more dynamic in the last few years, the QA process has become equally extended. Wider websites and apps usually demand more extensive testing and must be free of thousands of defects/bugs before they become eligible for public release.
In fact, the QA process needs to be carefully planned out and observed so that it can be sufficiently successful. The most effective way to follow the efficacy of QA activities is to use the proper metrics. Select the markers of success in the planning stage, and compare them with how each metric stands after the actual process.
When discussing the pertinence of a QA metric, it’s also important to keep in mind that QA metrics aren’t referred to a single area or category in the company. Instead, there are different levels in which a shared metric can exist and be appropriate. For example, you might have metrics at the project stage. There are metrics connected to the number of defects for a given period. Such metrics are most suitable for their specific products, though you can also estimate them at the department level.
Before determining what Quality Assurance metrics to use, ask what questions these metrics are pointed to answer.
So the questions to be asked before:
- How long will the test implementation take?
- How much money is counted for the testing process?
- What kind of bugs do we have? (state/statuses of bugs)
- What percentage of the software existing tests are covering?
- Are the estimated timelines sufficiently for tests to be completed?
Main QA Metrics
Now let’s move on to the metrics used to measure the effectiveness of the testing process. There are absolute values that are always presented with numbers:
- Number of test cases
- Number of passed test cases
- Number of failed test cases
- Number of blocked test cases
- Number of identified bugs
- Number of accepted bugs
- Number of rejected bugs
- Number of deferred bugs
- Number of critical bugs
- Number of determining test hours
- Number of actual test hours
- Number of bugs detected after release
But eventually, all these metrics above are not enough to measure the success and effectiveness of the testing process. To see and evaluate the overall image we need more specific and calculated metrics classified under different units:
Test effort metrics:
Basic facts about your test effort can help establish the basis for future test planning. Metrics include the Number of Tests Run, Defects per Test Hour, and Average Time to Test a Bug Fix.
- Number of tests in a specific period = Number of tests run/Total time
- Test design efficiency = Number of tests designed/Total time
- Test review efficiency = Number of tests reviewed/Total time
- Number of bugs per test = Total number of defects/Total number of tests
Test Coverage metrics:
Helps understand which areas of the application need to be tested. Having tests of good quality, this metric can reveal which parts of the software have a known level of defects vs. unknown.
- Test Coverage Percentage = (Number of tests runs/Number of tests to be run) X 100
- Requirements Coverage = (Number of requirements coverage/Total number of requirements) X 100
Team effectiveness metrics:
These measures work distribution and test results, for the team or team members.
- The number of defects returned per team member
- The number of open bugs to be retested by each team member
- The number of test cases allocated to each team member
- The number of test cases executed by each team member
Bug Distribution metrics:
Helps to understand which part of the software is most sensitive to defects, and thus where to focus testing effort. Metrics include the number, percentage, or severity of defects spread by categories like severity, priority, module, platform, test type, testing team, etc.
- Bug distribution by cause
- Bug distribution by feature/functional area
- Bug distribution by Severity
- Bug distribution by Priority
- Bug distribution by type
- Bug distribution by tester (or tester type) — Dev, QA, UAT or End-user
Test Effectiveness metrics:
Use this metric to answer the questions — “How successful are the tests?”, “Are testers handling high-value test cases?” So, the higher the percentage, the better the test effectiveness. Therefore, the lower the test case maintenance effort needed in the long-term.
- (Bugs found in 1 test / Total number of bugs found in tests + after release) X 100
Our practice
We as a QA company can’t get by without measuring our effectiveness using the metrics mentioned above.
Typically, the metrics used by our team depend on the current project the team/pair of engineers work on.
If it is a compact project handled by a pair of QAs (as already told on one of our previous blogs, most of our projects are handled by pairs or individual engineers) there is no need to measure for example the “Team effectiveness” metrics. But generally, there are metrics that should always be taken into consideration and well-measured no matter which kind of project we have deal with, such as “Bug Distribution” metrics as well as “Test Coverage” metrics which have primary importance in this regard.
So, use these metrics and be effective in everything you do!
