Applying Classification Modeling to software development — can machine learning help us assign priority and severity to defects or identify blocker issues?
We have talked about solving two types of issues using machine learning models — regression issues / models and classification issues /models. We also covered applying regression model to generate new metrics to track software release progress. In this article, I am going to talk about applying classification models to assign priority and servility to defect when reporting a software malfunction issue or defect.
Every engineering organization has its own way to prioritize defect backlog. Using priority and severity is one way of communications to capture the sense of emergency and importance of the issue from reporters’ perspectives to begin with. The purpose of these two fields is to make sure critical issues are addressed in a timely manner and also to help the development team communicate timeline and follow-up actions within and outside the engineering organization.
However, assigning priority and severity properly can be a challenge. First, is there a properly defined standard? Second, are defect reporters trained to assign priority and severity by the defined standard? Third, the world is changing and issues encountered are challenging too, how to we capture this change and make it part of the process with less human intervention possible?
Often time, priority and severity lost its value to the team and become a routine as part of the defect reporting process. We might see most of the defects carrying the default value as a result of no predefined standard or failure to transfer and communicate the standard across the organization.
The human intervention required to prioritize defect backlog can also be a hideous and tiring process. Why? First, depending on how many defects are in the backlog and how often the backlog is reviewed, the process can be long. If the process is not managed carefully, we can easily be falling behind, lost sights to critical issues that are not urgent, and create unnecessary stress to the team as a whole.
The idea here is to apply classification models to address challenges mentioned above and this is how it can be done. We can vectorize defect title or description by applying various Natural Language Processing methods and then combine it with other data point, such as component, area, platform, application, version, with CFD or not, reporter, and so on to train the model to classify defects.
We will train the model using the defect that’s already been reported and reviewed to make sure priority and severity are properly captured. This is one way to transfer and reuse the wisdom of the crowd. The newly reported defects will be channeled back to train the model so that we can capture and incorporate changes into the process.
Before putting the model in production, we will need to carefully evaluate the validity of the model to make sure the model works as expected and channel the evaluation back to the model training process to improve the model. A miss classified defect can cause delay of action and resulted in unnecessary stressful situation if the issue is reported from the field. A miss classified defect can also accidentally escalate an issue and create busy work for the development team.
Once the model is online, we need to monitor the model to address any deviation observed. Otherwise, people will lose confidence and trust to the system if there are too many miss classified classes getting into the system.
This model can also be used as a training and recommendation system to guide defect reporters during the defect reporting process how to fine tune the title or description to reflect the importance and sense of urgency of the issue.
Text or email me if you are interested in this topic.
