Sid and Ally provide AI assistance in bug resolution

Together with TNO’s ESI, Philips is eyeing the use of artificial intelligence to improve its software development efficiency. The advanced bug search interface they developed in the Accelerando project is already spreading like wildfire within Philips, while their AI-enabled bug triage tool is saving the company’s software experts hours of administrative work.

“Azurion is our advanced platform for image-guided minimally-invasive therapy, with over 4 million treatments already performed using the system. It’s the result of many years of development and for many years to come, it will be enabling the innovation of procedures in hospitals across the world. And throughout the platform’s lifetime, our developers will be keeping systems in the field continuously up-to-date,” says Patrick Bronneberg, principal cloud architect at Philips Image Guided Therapy (IGT). “The challenge is to strike a balance between this ongoing system maintenance and our ability to create innovations, while at the same time ensuring that the knowledge gained during product support is fed through into our development efforts.”

To address these challenges, Philips IGT partnered with TNO’s ESI in the Accelerando project. “Thanks to methods such as Agile and Lean, our engineering and manufacturing processes have continually improved over the years. Together with ESI in the Accelerando project, we’re now looking at how we can use AI to improve our development processes,” explains Bronneberg, who specializes in developing cloud and AI-based services.

According to Ronald Begeer, Accelerando project lead at ESI, the challenge is industry-wide. “All across the Dutch high-tech ecosystem, we see a growing need for the more efficient deployment of people and resources. In Accelerando, which started in 2020, we’re working on optimizing R&D processes using natural language processing and other forms of machine learning. For the AI expertise, we’re collaborating closely with TNO’s Data Science department.”

The Accelerando project focuses specifically on issue tracking – the process of reporting, assessing and solving software defects. “When a software bug is found, a description of the problem is entered into the bug database,” details Bronneberg. “Every few days, an accumulation of recent issues is sent to the project’s change control board – a panel of software experts who determine the severity of each problem, try to identify the component that caused it and assign the issue to the development team responsible for the component, who then attempt to fix it.”

Bronneberg points out that there are several places where this process can hit a snag. “The problem may already have been reported, for example, resulting in a duplicate entry in the bug database, or it may be difficult for the change control board to pinpoint the root cause, potentially resulting in the issue being handed to the wrong development team. And even if the defect does land in the right team, an inadequate bug description may make solving it harder than necessary.”

Today, software teams in the high-tech industry are spending about 60 percent of their time not on bug fixing but on the administration around it. “But the chances are a lot of information about similar bugs is already available in historical bug reports,” Bronneberg reckons. “Through AI-enabled automation, we want to connect our developers to that information and make it come to life, giving them a better insight and allowing them to focus their attention where they can add value.”

The Accelerando project is addressing three stages in the issue tracking process: duplicate detection, bug triage and bug localization. “Resolving duplicates is a time-consuming process,” finds Dennis Dams, a senior research fellow at ESI. “Taking off-the-shelf search technology that uses NLP, natural language processing, we’ve developed a tool to help speed up duplicate bug detection. In a web-based application, developers can enter a short description of the issue and they get back a list of possible duplicates as well as a couple of suggestions for related keywords to improve their search.”

The Similar Issue Detector (Sid), as the application has been dubbed, is trained on a collection of bug reports that Philips has built up over the years. Employing an established NLP technique called Term Frequency-Inverse Document Frequency (TF-IDF), words and phrases are grouped together that often appear together in these bug descriptions. The user input is then correlated in real-time to the TF-IDF output, after which the top results are displayed. “Every once in a while, usually at the start of a new project, we retrain Sid on the latest version of the report database,” notes Dams.

Sid was very well received by the software developers at Philips – much better even than Bronneberg had expected. “We started with a rudimentary search interface, the idea being to streamline it later on but generate enough clout in the meantime. But it spread like wildfire. People who witnessed colleagues working with Sid wanted to get in on it as well. It not only helps them detect duplicates earlier, it also stimulates them to make a general problem description more specific, and it gives them an all-round better search interface to our bug database. More and more of our developers are using it on a daily basis, so we’re even more intent on integrating it into our existing toolchain.”

Once an issue has passed duplicate detection, it’s assigned to a software team for resolution in a process called bug triage. “Triaging is a very difficult task, with first-time-right traditionally being a near impossibility,” explains Dams. “To support the process, we developed a second tool using similar NLP technology and integrated it into the change control board workflow. New issues are delivered to the board in an Excel spreadsheet, which we’ve enhanced with functionality to automatically identify the software teams that are the best candidates for assigning a defect to. With the push of a button, a top-three is generated for each issue.”

Ally the Algorithm, as the Accelerando partners have dubbed this tool, is again trained (and occasionally retrained) on Philips’ collection of bug reports. In a supervised-learning approach, the algorithm is fed with problem descriptions from the past, each labeled with the team that ultimately fixed the issue. Using the TF-IDF technique, Ally builds up a model that connects terms appearing in a report to likely candidates for bug resolution. A new problem description is then correlated in real-time to this model, yielding the three best matches together with scores that indicate how confident the algorithm is about each match. It’s then up to the change control board to pick a team and assign the issue.

When comparing Ally’s performance to that of the human panel, the algorithm achieves virtually the same accuracy but in a fraction of the time. “Supported by its AI capabilities, Ally produces a result in a matter of seconds and, as a result, one of our change control board members told us they’re now able to shorten their meetings by 20 minutes. With seven or eight experts convening three times a week, that’s a massive time saving – valuable time they can spend on expert tasks rather than administrative work,” says Bronneberg. “It also gives us a much better insight into common triaging pitfalls. When one of the change control boards was presented with a matrix of historical data showing where they initially assigned an issue and where it ultimately got fixed, they realized how powerful Ally was in moving towards first-time-right decision-making. Insights like this really make the data come to life and help us increase our efficiency.”

A software team that gets an issue assigned to it for resolution needs to localize the bug first, and research is now ongoing to see how AI can assist in performing such an analysis efficiently. “We’ve run some experiments with different techniques to see how AI can help. We’ve found that it’s not enough to only look at the text of a bug report,” observes Dams. “We also need to examine the log files that come with the report.”

Another avenue of research slated for this year is improving the accuracy of Sid and Ally by improving the quality of reporting. “Seeing their performance plateau despite the application of more advanced AI techniques like deep learning is a sign that perhaps the input data needs some work,” Dams reckons. “That’s why we’re now also looking at raising the quality of the bug reports – the more precise the problem descriptions, the easier it becomes to detect duplicates and do bug triage.”

Bronneberg’s vision is to have a chatbot-like tool. “Improving the quality of reporting has always been a challenge, with the risk of getting bogged down in detail. No developer wants to have to fill in fifty fields. Instead, we’d like to gather the required information in a structured yet user-friendly way. Think of a chatbot, guiding you through the process and only asking you for information that can’t be deduced automatically or extracted from your earlier responses.”

Bronneberg heralds the power of data science as his main takeaway. “By leveraging that power, you’ll gain so much more insight into your business processes.”

ESI project lead Begeer agrees. Applying standard data science techniques provides a wealth of process information. All of high tech can benefit from it.