Artificial Intelligence is hot right now and I’d like to share with you my experience in trying to use one of the most popular AI approaches to solve a real world problem in the medical device industry. I’d like to illustrate what works, what doesn’t, and where the pitfalls are. For the past four years I’ve been going deeper and deeper into the realm of machine learning and I’ve ended up focusing in the area of Reinforcement Learning (RL). RL is the approach that led to several breakthroughs over the past 3-4 years from the likes of Google’s DeepMind, OpenAI, and several others. You’ve probably seen the headlines of AI agents beating humans at Atari games, Go, Dota and other games.
But less well known are how the same techniques have led to improvements in robotic control. RL has allowed robots to learn to walk, manipulate objects, and is the technical backbone of self-driving cars. So, being a curious human being I asked myself how this methodology could be applied to the devices that I’ve helped design over the past 9 years in the medical device industry.
Before diving into the technical details, I’d like to describe a couple of medical device applications and how to frame them such that RL can be applied. The first application is vessel sealing and the second application is tissue ablation. When surgeons need to perform a laparoscopic procedure, they often turn to instruments that allow them to dissect tissue quickly and seal vessels confidently. These instruments typically deliver energy to the tissue in the form of RF or ultrasonic energy. I designed hardware and algorithms that power these instruments and they have transformed surgery over the past two decades by enabling minimally invasive procedures, lowering blood loss, and significantly speeding up procedural time. Before these instruments came along surgeons had to use manual energy application as well as sutures to achieve the same effects. Instruments such as Medtronic’s LigaSure and Ethicon’s Enseal lead the pack in the industry.
Tissue ablation is another area that uses energy to kill tumors in areas like the lung, liver, or other organs. The energy source could be in the form of RF or microwave energy and is typically used by oncologists to ablate a volume of tissue. The energy is typically deposited in tissue via a probe that acts as an antenna in the case of microwave or as an electrode in the case of RF. These procedures are less invasive and can have improved patient outcomes compared to open surgery or chemotherapy. Medtronic again offers several products such as RF Cool-Tip and Emprint which use RF and microwave energy sources respectively.
So what’s wrong with these products and how can RL be used to solve these problems? Well, quite frankly all of these products work very well and they save countless lives every day. I can tell you from experience that the products that I helped design went through rigorous and scientific evaluation and were designed with patient and operator safety foremost. That being said, there’s always room for improvement and for applications across a broader horizon than is being currently used. For instance, thoracic procedures may require sealing vessels in the lung and these vessels have very different properties compared to their counterparts elsewhere in the body. As such, it is more difficult to seal these vessels given the same instruments that were optimized for other anatomy.
In addition to expanding the horizons of these applications there is another argument to using RL that I believe outweighs the others. Exponential learning. Sebastian Thrun, the founder of Udacity and the “grandfather” of self-driving cars was interviewed a couple years ago and gave a profound statement to the power of AI. Paraphrasing: “When a human driver makes a mistake, he or she learns from that mistake but the rest of the population does not. When a self-driving car makes a mistake that car learns from that mistake and so does every other car in the fleet.” This type of learning is very different from humans and allows connected machines to learn far faster than our “meat-ware”. When medical devices can start to learn from the collective experience of all other devices in the field improvements will accelerate drastically.
Now for the sober reality. Applying AI in these applications is hard and the regulations governing medical devices mean that verification and validation of the designs will increase scrutiny and slow down the pace of development. That’s not all bad since people’s lives are on the line but the industry as a whole will be slow to take up these powerful technical advances and will most likely be the last one’s to the game due to a low tolerance of risk. For now, I will ignore this headwind and push forward with my agenda.
Now that we have an idea of medical device applications and how AI can be used to improve them how do we frame the problem such that we can use RL to improve their performance? At the highest level, we wish to replace the current hand-tuned algorithms that deliver the precise amount of energy with an AI agent that can do a better job. The AI agent can be thought of as a brain that tells the lower level control system what to do at any point in time while the device is delivering energy to the tissue. The assumption here is that the agent sits on top of a PID (Proportional Integral Derivative) control system. This reduces the complexity and amount of training that would be required for the agent to learn the open loop dynamics of the underlying hardware. Additionally, the PID can itself be separately optimized for stability, bandwidth, anti-windup, gain scheduling, SW-defined limits, etc. These objectives are all difficult enough to design and verify. We don’t want to add them to the agent’s to-do list.
What we really want to do is to give the agent the job of finding the best trajectory, or set of actions, that optimize some higher level objective. We could have several objectives such as speed - seal or ablate tissue as fast as possible, and reliability - achieve a minimum reliability metric or volume metric with low standard deviation across tissue types. In other words, we are handing off the job of hand-tuning algorithm knobs to the agent to figure out. This abstraction allows the engineer to focus on the types of rewards and behaviors that they want at a high level and not focus on sorting through both the high level objectives and low level implementation simultaneously. In the next post, we will dive into the details of how we frame this problem and some of the lessons that I learned along the way.
Comments