Streamlining troubleshooting in the field

Canon Production Printing has teamed up with TNO’s joint innovation center ESI to identify potentially failing parts and predict potential issues in production printers. In the Carefree project, they’re developing hybrid AI technology to support service engineers.

“At Canon Production Printing, predictive maintenance has been a topic of interest for quite some time,” says Peter Kruizinga, a lead technologist at the Venlo-based company. “The aim is to get an accurate picture of when a printer or one of its components is going to break down by analyzing the machine’s data, and then dispatch a support engineer to preempt the issue. We’ve already successfully implemented this for the transport belts in our systems: a significant rise in power usage of a belt drive is a direct sign of impending failure, so when we see that happening, we can send someone over to replace the motor before it breaks down.”

Not all cases are so clear-cut, though. “A printer has thousands of parts,” notes Kruizinga. “For a lot of them, unfortunately, there are no such direct indicators, so in order to ascertain the source of a problem, we need a lot of indirect information, which often is unavailable. For a specific part, the failure mode can also vary from one situation to another. We realized that doing predictive maintenance on a structural basis requires us to solve a large number of puzzles, and to get all the pieces, we have to make strides in problem diagnosis.”

With predictive maintenance being the ultimate goal, the initial focus is on corrective maintenance. Kruizinga: “Although our printers come with an extensive manual and have some diagnostic tests built-in, we see our service engineers mostly relying on their experience when they’re called to fix a problem in the field. We want to guide them in finding the root cause faster. The first step is to improve our diagnostic capabilities after the fact, so after an issue has occurred. The next step is to also be able to see problems coming in advance.”

In the Carefree project, a two-year effort that started in January of 2020, Canon Production Printing (CPP) has teamed up with TNO’s joint innovation center ESI to identify potentially failing parts and predict potential issues. Combining models derived from the development phase with current machine data, the partners are building a self-learning diagnostic system that can pinpoint the cause of a future problem or advise where and how more information needs to be gathered in order to do so. As a result, timely action can be taken to prevent long and expensive downtimes.

The higher-level diagnostic models created in the Carefree project consist of abstract representations of printer components, their capabilities and observable erroneous behavior. “Each printer has a paper input module, with a paper tray that moves up a little every time a sheet is fed into the machine,” takes Kruizinga as an example. “For this module, we’ve constructed a model with components such as the lift motor and the tray sensors. The lift motor has as its capabilities that it can move up and down, while one of the sensors is capable of detecting the up position. Both the upward movement and the up position detection can trigger the observation that the lift table has arrived at the up sensor too late.”

The model includes for every component the chances of it being in a specific state, for every capability the chance of it being enabled and for every error the chance of it occurring. These probabilities are based on factory specifications or they can be derived from historical machine data. “The same paper input module has been used extensively in a wide range of printers in the field,” Kruizinga goes on to illustrate. “This gives us a detailed insight into the frequency with which components and capabilities break down and errors occur in practice, allowing us to more accurately determine the a priori probabilities.”

By adjusting the probabilities based on actual evidence gathered in the field, it’s possible to hone in on the root cause of a failure. Kruizinga: “When the printer displays an error message indicating the lift table has arrived at the up sensor too late, the probability of that error occurring can be set to 100 percent. This change then propagates across the model, affecting the other probabilities. It raises the chance of the lift motor being faulty, for example. Additional evidence can be obtained by running diagnostic tests on the printer. If one such test shows that the up position can be detected successfully, the probability of that capability can be set to 100 percent. Eliminating the sensor as the source of the problem increases the likelihood of a faulty motor to almost 100 percent, giving us the probable root cause.”

With the help of artificial intelligence, the project partners believe the diagnostic performance can be given a further boost. “We’re looking to make the probabilities even more accurate by applying AI techniques to actual sensor data from the printer. We’re also considering using AI to get useful information from user profiles – it might very well be that customized print jobs result in different wear and tear than mass printing,” argues Hegge. “This hybrid AI approach, refining human knowledge with computer-generated insights, has been well received within Canon.”

The Carefree approach is based on so-called Bayesian networks. “Given a problem, such a network can lead you to the most likely root cause,” explains Jos Hegge, senior project leader at ESI. “It allows you to enter ‘evidence,’ like error messages, and it then calculates the effects of this evidence, updating the probabilities and returning new conclusions on the most likely causes. It can also streamline your quest by suggesting which tests to run to get to a result more quickly.”

The main drawback of Bayesian networks is that they tend to get really big really fast. “For a professional printer, consisting of thousands of components, creating such a network by hand is a rather hopeless enterprise, even more so considering you have to change the network each time you tweak your system design,” observes Hegge. “Which is why we’ve added an abstraction layer in the form of a high-level language. We can use this language to describe a diagnostic model and then automatically convert it into a Bayesian network.”

At the moment, the higher-level models are still manually created. “The ultimate goal is to be able to generate the Bayesian network from a printer’s design documentation,” Hegge points out. “As that places very high demands on the source documents, we’ve decided to settle, for now, for this intermediate abstraction layer. Later, at a moment of its choosing, CPP can pick up where the project left off.”

For CPP’s Kruizinga, the higher-level description language is actually one of Carefree’s notable achievements. “Creating a Bayesian network requires really specific craftmanship. Thanks to the abstraction layer, we can now put the specification work into the hands of software designers who don’t have that very specialist background but who can do an excellent job just using their knowledge of the application. Also, we’re not faced with the daunting task of having to manually update the Bayesian network each time we change the system design.”

As the main deliverable, Carefree will lay down the technology foundation for CPP to incorporate in a maintenance tool for easy troubleshooting in the field. “Whenever there’s an issue at one of our customers, we dispatch a service engineer, who, after feeding the tool with the error messages and other ‘evidence’ found at the scene and running some additional tests as suggested by the tool, swiftly ascertains the cause of the problem by combining the results from the tool with his own expertise,” envisions Kruizinga. “This combination of computer smarts and human skills is key. The tool isn’t meant to replace our service engineers but to support them in doing their work even better.”

“At ESI, we’ve built up a lot of experience with this kind of diagnostic challenges – in the collaboration with ASML but also with Thales, for example. In the Carefree project, we’re bringing this knowledge to Canon,” concludes Hegge. “Our goal is to generate a hybrid AI approach that can be used to manage the maintenance of professional printers, but that’s also beneficial to the high-tech industry as a whole.”