By: Ipsita Mohanty, Walmart Global Tech

In large e-commerce organizations, many defects are generated periodically with a massive pool of software teams and developers spread across geographies to pick from, each with a unique domain specialization. Most organizations have a large pool of human triaging agents responsible for routing these product defects across various teams. However, large-scale software releases are time-sensitive, and effective defect assignments are critical in the process prone to bottlenecks. Inefficiencies arising from human and process dependencies in this domain have motivated research in automated approaches using machine learning to assign defects to qualified teams accurately.

Prior industry research on automated defect triage has primarily focused on using traditional machine-learning approaches. However, with the recent surge of state-of-the-art pre-trained language models, one underexplored field of application is operations in agile software development. In defect triage, handling scenarios require Natural Language Understanding to utilize the context of the defects logged by human testers to predict all the teams associated with resolution. Most defect triage processes are primarily human- agents drove. Our work integrates an automated defect triage framework, DEFTri using product defect’s contextual features to achieve operational excellence within the software development lifecycle.

We propose a novel framework, DEFTri, a model architecture for fine-tuning pre-trained BERT for our multi-label classification task. The architecture has two proposed variations – Label Fused Model with [SEP] & Label Fused Model without [SEP]. We denote the defect corpus(title and description) tokens as Di and their corresponding token embeddings as EDi, where K is the total number of words in the input defect, and DK represents the last token. Similarly, let Lj be the team label text of the jth team of the overall number of teams corresponding to the defect corpus. Finally, we derive the positional embedding using BERT and apply the classification layer with activation to the last layer of the hidden state at the [CLS] token. Additionally, our dataset uses domain-specific lexicons to generate labeled training data using weak supervision in a few-shot setting. We propose using adversarial learning to increase our training sample size while increasing the robustness of our models.

Based on our experiments, we observed that label-fused contextual learning-based fine-tuned BERT models significantly outperformed the base model using only the context of the defect text. The performance boost over the base BERT pre-trained fine-tuned model is because of the context in the label embeddings used in addition to the defect text in the label-fused models, which optimizes the alignment of features, improvising the classification task. We also observed interesting results from using adversarial learning used in our data.

For details, check out my research publication at ACL 2022.
DEFTri: A Few-Shot Label Fused Contextual Representation Learning For Product Defect Triage in e-Commerce