“What cannot be measured cannot be improved.”
This saying forms the basic reason why we have quality metrics defined and measured. There are two major purposes which these metrics serve. The first purpose is to know the current condition of the software and identify if things need to be improved. The second purpose is that it forms the basis to showcase the improvements done over a period of time. In this respect, software quality metrics are an essential part of any Quality Management System and allow projects to quantitatively measure the project’s quality.
But like any other metrics or statistics, the quality metrics also tend not to give the full picture in certain scenarios. Let’s take an example of Code Review Yield. A high Code Review Yield is supposedly an indicator of a good review mechanism. But this could also be an indicator of poor coding standards within a team. On the other hand, a low code review yield which is supposedly an indicator of a poor review mechanism can also mean that the coding standards within the team are very high. Most Quality Management Systems tend to have statically defined goals for metrics which will tend to hide these underlying weaknesses of the metrics to capture the correct picture. What’s worse is that individuals might tend to show figures in the defined range to show their project in good light and tend to dilute the whole reason of having these metrics in the first place.
So it is paramount that metrics are clearly understood and the goals are defined and assessed in the proper context of the project.
Another problem usually related with Quality Metrics is the problem of generalization. If a particular metric has been useful in a certain kind of project, it is assumed to be useful in all kinds of project. Nothing can be farther from the truth. For e.g. Bugs Turnaround Time could be a critical metric for an UI based project which has a lot of bugs raised and it is paramount that the issue needs to be fixed as soon as possible. But, on the other hand, this same metric might not make sense of backend projects, where there are hardly any bugs raised and even those raised have to be fixed only by next version. So it is essential for the Project Quality Manager and the Project Managers to sit together and identify clearly what metrics make sense to a specific project and what should be the duration at which it should be collected and how it should be assessed.
Ultimately, the goal of a quality metric is to measure and improve the state of a project. That can happen only if the information provided by that metric is relevant to the project.
Another key criterion to Quality Metrics is that it also depends on the style of the Project Manager. Of course, it is often said that one of the objectives of a QMS is to ensure that it is independent of an individual which is quite an honorable goal. But, the fact of matter is that finally it is being implemented by an individual and until and unless it tends to be of added value to the person, the metric is useless. For e.g. suppose there are two individuals, one who feels comfortable with an implementation freeze followed by a bulk code review, while the other feels more comfortable with incremental code reviews as and when a component is completed. Now it is easier to collect metrics in a bulk review as a file undergoes only one set of review, while in an incremental approach a single file might undergo multiple reviews leading to multiple collection points of metrics. The added overhead in collecting of metrics should not be a reason for the second individual to change to the bulk review mode. This aspect is very critical as the success of any project depends on people playing to their strengths.
So the project manager and the relevant stakeholders should be comfortable with the metrics and the mode of collection and assessment for the metrics to really add value.
At the end of the day it is important to remember that Quality is after all a Qualitative goal and cannot be made quantitative. But, Quality Metrics definitely add value, provided they are defined, collected and assessed in the proper context of the project, project team and the project manager. If this is not done properly, then they will add unnecessary overhead of effort and time and will result in very little or no value addition to the projects and the organization.
Remember Quality Metrics on their own do not reveal the true picture and are only one of the indicators of the Quality of a project.
– Ram Narayanan Sastry