ESI helps Thermo Fisher and Philips

When new feature development for its electron microscopes started to get hampered by older software, Thermo Fisher Scientific called on TNO joint innovation center ESI to clean up its code and make it future-proof. The approach developed in their public-private partnership was also successfully adopted by Philips to tackle the legacy challenge in its medical scanners.

Highly advanced systems such as an electron microscope or a medical scanner contain a boatload of software to make them work – millions to tens of millions of lines of program code. This software is the product of years, sometimes even decades, of development by a team that constantly changes shape. New engineers come in, while others move on, leaving their work behind as a legacy to be cared for by the rest.

As this legacy codebase grows, it’s getting increasingly difficult to maintain an overview of the inner workings and keep the software well-structured.

“Most of the code was written a long time ago and over the years, many people with advancing insights have contributed. The result is less than optimal, to put it mildly. Layer upon layer, reflecting the software engineering progress the company has made through the years,”

explains Dennis Dams, a senior research fellow specializing in legacy software within ESI, the high-tech embedded systems joint innovation center of the Netherlands Organization for Applied Scientific Research (TNO).

The lack of overview and structure makes it increasingly hard to add new functionality – a challenge recognized all too well at Thermo Fisher Scientific in Eindhoven, one of the companies with which ESI works.

“The software of the electron microscopes we develop here uses old interface technology to enable subsystems to communicate with each other,”

illustrates senior R&D manager Arjen Klomp. “We used it all over the place, the code was practically polluted with it. This hampered our new feature development and brought down the efficiency of our software engineers to such an extent that we needed to do something about it.”

Thermo Fisher called on ESI for help.

Together, they embarked on two consecutive public-private research projects to develop a novel approach to tackle the legacy challenge, Renaissance (2016-2018) to clean up the existing code and Phoenix (2018-2020) to prepare the software architecture for future extension. This resulted in initial tooling. The approach was effectively turned into practice at Thermo Fisher. At the halfway mark of the second project, Klomp is already very pleased with the results.

“We’ve managed to significantly reduce the time it takes us to build a new version of the software. We’re now releasing four times as often compared to three or four years ago.”

The results haven’t gone unnoticed. Also assisted by ESI, Philips, in Best, has adopted the tools developed within Renaissance to take on a similar legacy challenge in its medical scanners – with similar success.

“The quality of our code has increased significantly,”

says Hans van Wezep, a software architect within Philips’ Image Guided Therapy business. “By giving us more insight, the tooling really opens new doors in innovation for us. We’ve integrated Renaissance in our own collaboration project with ESI, Vivace, to build on these promising results.”

One way to clean up a mess is to throw everything away and start over. Thermo Fisher’s Klomp explains why this is not an option:

“We’ve added a huge number of features over the years. When you start from scratch, it will take an enormous amount of time and money to get to the same level of functionality – actually, more than it will cost to clean up your existing codebase. When your dishes are dirty, you could throw them away and just get some new ones. But if you do that every time, in the end, buying a dishwasher is much less expensive. So that’s what we did: we got ourselves a dishwasher for our software.”

In essence, the approach developed by Thermo Fisher and ESI in the Renaissance project cleans up the code by uncovering structural bottlenecks and offering pointers to resolve them. It does this automatically.

“Instead of using a dishwasher, we could also clean the code manually, but that’s very labor-intensive, so again very expensive, and very tiresome for the engineers, so highly error-prone,”

clarifies Klomp.

“Automatically cleaning the code not only reduces the effort but also lowers the risks as it ensures that the software remains working.”

Cleanup consists of two phases: analysis and transformation, in multiple iterations. First, the ESI team analyzed the software architecture by creating pictures of it and discussing them to pinpoint suboptimalities. During this process, they were supported by a lead engineer from Thermo Fisher, who provided in-depth knowledge about the code. Klomp:

“He helped them understand the architecture pictures. What does it all mean? Why is it like that? What should it look like? For this domain knowledge, we sometimes had to go shopping within our organization, ask around, do some digging.”

Following the findings, the code was restructured to get rid of the suboptimalities. “This transformation wasn’t actually fully automated from the beginning,” Dams from ESI notes. “We started with dividing the cleanup into about 70 parts. The first parts we optimized mostly manually. This provided insight into what needed to be done and brought to light recurring patterns, enabling us to write some tool support to do those things automatically. As we proceeded, piece by piece, we automated more and more patterns and by the time we got to the last parts, most of the work was automated. Not everything, because there will always be some work that requires manual intervention.”

All the while, the cleanup crew worked in parallel to the Thermo Fisher team developing new features. Or, as Dams puts it:

“We were repairing the plane while it was flying.”

For Klomp, this is one of the project’s big achievements. “At times, it was challenging to merge the cleanup crew’s work with new feature delivery, but in the end, we managed to break nothing. Had we disturbed the ongoing software engineering process, that would have generated a lot of complaints and resistance and the project wouldn’t have been anywhere as successful.”

The tools are now in broad, day-to-day use at Thermo Fisher. Klomp: “Our architects and senior designers use them to monitor the code structure to see whether it’s deteriorating again or, even better, improving.”

Early in 2018, the positive sounds from Thermo Fisher reached Philips. “We have an older product line of interventional x-ray scanners, for which we don’t do much new development anymore but mostly maintenance. A lack of insight into its dependencies made it quite challenging for us to coach new engineers to work with it or do a redesign for it,” recollects Hans van Wezep.

“Through ESI, we learned about the Renaissance project at Thermo Fisher and we quickly realized that what happened there could be beneficial for us as well. We adopted the analysis tooling and, with only some minor changes, had it up and running very quickly.”

Together with ESI, Philips successfully applied the Renaissance approach not only to the scanner software but also to the build environment in which that software is being developed.

Van Wezep: “We use a standard Microsoft setup and we saw people not using it to its full potential. Dropping all the code they’d written into the same folder and compiling it from there, for example. However, when you create separate packages and layers in your development environment like you do for the software that you’re building with it, as well as clean up the dependencies, it’s much easier to see how everything’s connected and it makes that environment much more efficient.”

“One of the great features of the tooling is the extensive way in which it presents its findings,”

states Van Wezep. “It’s very easy to see whether you’re dealing with a recurring pattern, which you then can have resolved automatically, or an exception, which you have to fix by hand. Most issues, about 80 percent, can be handled by the tooling; the rest requires manual effort. We had someone do everything by hand and it was very nice to see how effortless the tooling worked in comparison.”

“I had a similar experience at Thermo Fisher,” says Dams. “After having used the tooling, one of the developers came up to me and said,

‘Your tooling does in 3 seconds what I’ve been working on for three weeks and it even does it better!’ Because the work is so tedious, it’s very easy for a human to make a mistake.”

The visualizations play an important role here, Dams thinks.

“There are several systems available for browsing your code. In most of them, you click on links to go to another level – up or down. It’s very easy to lose sight of the bigger picture. Our tooling gives a bird’s eye view. You see all the important items and their dependencies in one glance, making it very easy to spot the structural suboptimalities. I heard someone call it a code browser on steroids.”

Source: Bits & Chips