The FDA is Taking a Big Risk on AI

The FDA is charging ahead with generative AI, claiming it can cut scientific review times from days to minutes. But is this innovation or recklessness? 

In this episode of CareTalk, David Williams and John Driscoll debate whether the FDA’s rapid deployment of AI is a smart move or a potentially dangerous shortcut.

Stream our episode with FDA leader, Dr. Marty Makary, here. 

🎙️⚕️ABOUT CARETALK
CareTalk is a weekly podcast that provides an incisive, no B.S. view of the US healthcare industry. Join co-hosts John Driscoll (President U.S. Healthcare and EVP, Walgreens Boots Alliance) and David Williams (President, Health Business Group) as they debate the latest in US healthcare news, business and policy. 

GET IN TOUCH
Follow CareTalk on LinkedIn
Become a CareTalk sponsor
Guest appearance requests
Visit us on the web
Subscribe to the CareTalk Newsletter

Transcript

David: 0:00

FDA says, it's embracing generative AI to streamline scientific reviews, cutting days of work, down to minutes. So when an agency that's known for its deliberate pace and thorough evaluation adopts a powerful new technology in weeks, should we be pleased or alarmed? Welcome to Care Talk, America's home for incisive debate about healthcare, business and policy. I'm David Williams, president of Health Business Group.

John: 0:33

And I'm John Driscoll, the chairman of Yukon Health. So David, what's all this to

David: 0:37

worry about? Technology is our friend John. It is our friend John. It's our, it's our buddy. But you know, this came up last week at FDA and they, they issued a press release. Oh,

John: 0:48

Dave, I mean, what does the FDA actually do anyways? What, what are they responsible for? Food safety drugs. How kind of risky healthcare is in medical devices. I mean, why shouldn't we? Random. I mean like what could go wrong?

David: 1:09

Well, it makes a lot of sense that the FDA is evaluating, I. The use of ai, but what's actually happened is they suddenly issued a press release that they're gonna, said they're gonna scale the use of AI internally across all FDA centers by June 30th of this year. That's, that's a quote. And they're also claim they're gonna be operating on a common secure AI system that's integrated with the FDA's internal data platforms By that time.

John: 1:32

Beautiful AI system. So let's just unpack this a little bit. Artificial intelligence, as currently referred to is really our, uh, our, our large software models that, uh, to that, that can be run. Either autonomously where people are actually not involved and the code is running on itself based on a lot of data to develop insights, or it can have sort of a human in the loop sort of supervised learning. In each case, we're talking about large computer models, trillions of bits of data, and either human supervising the models or non supervising the models. The models are the engines that create the intelligence. Allegedly. And what's artificial is it's created by the models. The models are currently based on what are called the large language models, which scour the internet in any other. Data po. They can find, filter it through human interpretation and then feed it into models that are aimed at over time, automating tasks, creating insights, and potentially creating new insights that, and have shown that they have created new insights that we otherwise couldn't have figured out. I know one researcher who's got a very toxic form of cancer, who's an MD PhD in Compsci. Um, who's got a rare form of cancer? Who's using AI to try to identify new modes of treatment? I mean, David, these are big, powerful models. We like computers. We're using a computer. What, tell, tell me, tell me what could go wrong with this ai, this leveraging of the technical talent of the United States to do a better job at government. At the Food Drug Administration.

David: 3:24

So John, the FDA commissioners, uh, Dr. Marty McRay, Macari, you tell me he loved Marty McElroy, right? So he was a, a guest here. And, uh, if you wanna listen to that episode, which is arguably, uh, more sane that we, we were talking about, uh, medical blind spots. Um, you can look in the, in the show notes and you'll see the link there. So I, I commend that to you. I actually want to read. Verbatim a quote from him in the press release, and then let's talk about what they're planning to say and what could go wrong. I, I actually think it's self-evident almost when you hear what he said. So this is a quote. Quote, I was blown away by the success of our first AI assisted scientific review pilot. So our first pilot, we need to value our scientists time and reduce the amount of non-productive busy work that has historically consumed much of the review process. The agency-wide deployment of these capabilities holds tremendous promise in accelerating the review time for new therapies. Said Dr. McRay. So, you know, and then the next quote is from the person that's in charge of AI there, uh, Jin Liu. And he says, this is a game changer technology that has enabled me to perform scientific review tasks in minutes that used to take three days. So the risk, John, is that we're, we're going, we're taking something. When you say the first, he's blown away by the success of the first pilot. Therefore you go from the first pilot to, to agency-wide implementation right away. Well, where is this pilot? They didn't even release any information about it, and who cares if this one guy was impressed by it? It's ridiculous. It's the opposite of taking the real scientific evidence. John, if I came to you with how many drugs do people find in the lab? It says, oh, this thing looks great. It has promise to cure cancer, ulcerative colitis, diabetes. You know, if you just said. You know, we gotta put this on the market by the end of next month. It's absolutely insane in the opposite of what FDA does.

John: 5:19

So let's, let's think about this contextually. That's a great example. Um, we've been thinking, uh, since the cancer, uh. The, the commitment to end cancer as we know it under President Nixon, the cancer moonshot under Biden, and all the cancer attempts in the middle, including the notion that we would be able to take genomics and cure cancer. Um, we've consistently been able to. Look at what cures cancer in the lab with regard to mice and, and it's failed in humans. There's a complex biology, and then of course there's the medical device area, and then there's the food safety area, and you're, you're raising a really important point. One of the reasons why we rely on the FDA is to slow down the deployment of novel technologies and approaches to make sure that they're safe, the ultimate. Baseline re uh, uh, uh, uh, aim of the Food and Drug Act, which I think was part of the progressive movement. 1911, was to start to develop the leading safety regime in the world around deploying medical devices, drugs, and also to make sure that our food was safe. The notion that we would kind of, um, delegate that. Um, the other thing is it's coming at a really weird time. I mean, we've just gone through, um, the, the forced retirement, uh, or firing of thousands of probationary employees. Those employees are often the and, and almost. Almost exclusively, but not all are the youngest er uh, employees in the agency. Those employees are digital natives. You know, I had an interesting conversation with the, the head of the, the, uh, counter-terrorism center under Bush and Obama, and I said, what's the biggest thing you worried about? He said, well, under the last Trump administration, 70% of all the employees under the age of 40 had left the federal service. And those are our digital natives. We're gonna rely on them to. Govern, innovate and, uh, uh, invest in and deploy new technologies. If, if we lose the smartest young talent, we will not be able to retain our technical edge. So we've just fired a bunch of people who have that, um, while potentially deploying to your point technology that we don't really know. Whether it works. So John, all John of these large lang. Yeah.

David: 7:42

Go. So John, what we're left with here is, so you've got the people that really know how to use new technologies. So they, they're gone 'cause they're a probationary. And then you also have the people that were most experienced who know, hey, a bright new thing comes along, how do you evaluate, how do you use it? So now you have left, uh, a small stressed. Set of employees who are neither the voice of the most experienced, or the ones the most technical savvy, and they said, Hey, let this thing loose and boom. So you're gonna have something you're, you're within. This would be bad to deploy in an agency that was stable, but it's terrible to deploy right now given that the top, the most senior people and the most technologically savvy people are out and others are in a bad spot.

John: 8:26

And I think there's a really important point about what intelligence is. I mean, novelty and insight and um, uh, identifying something doesn't necessarily, from the data doesn't necessarily translate into reasoning. I. Discernment or judgment. You know, one of the things that, um, professor Ziad Obermeyer used to be at Harvard, now he's at, uh, at Berkeley, and he talks about this in some of his speeches. He's an a, he's a professor on ai, I was gonna say he's an AI professor, but that would suggest that he is provided by technology. And Dr. Obermeyer said, look, um, uh, there are forms of intel, uh, of, of intelligence that, that aren't human reasoning. I mean, a, uh, a squirrel. Um, we'll, we'll, we'll hide nuts in the fall and after the winter, basically be able to find 70% of the nuts it puts out. A human couldn't do that, but the human can determine whether someone has a. Is suffering from social determinants of health, and maybe that's feeding into their unstable diabetes or cancer. They can make a judgment call on whether, uh, the different, uh, different populations respond to drugs differently, don't he? His point is, don't confuse. Uh, uh, a brain that a, uh, uh, um, uh, an analysis that could be done by a squirrel to, and translate that into something that could use by a human. I mean, then you've got this whole issue of, um, AI hallucinations. Do you wanna just describe what

David: 9:57

those are? David? John, I would say I will do that. And I would say that, um, since the FDA hasn't been systematic in their assessment of this issue, let's do it. So the first one, hallucinations falls under accuracy. I'll come back to that in a second. There's also issues of data privacy, transparency over reliance on the technology and speed over safety. So specifically with hallucinations, you could say, wow, you know, this thing can summarize, this huge document could read thousands of pages in seconds and it can gimme the summary. So the thing is. Uh, the summary may be flawed. It may have something that's got nothing to do with actually what's in the paper for whatever reason, that's not actually even understood by the people that created the model, nevermind those that are using it. So you're gonna see a hallucination that's just kind of made up from something that's not there. And then also these drugs that are being, uh, submitted for approval, they're novel. So they're also talking about things that have never been talked about before. So the hallucinations come about. Let's unpack

John: 10:52

that. So what a hallucination is, is when you ask a model a very specific question, like, give me an example of a marathon on the West Coast of the United States, and it, and it gives you the example of a marathon in Philadelphia, which that was really one, one of the large language models created that answer. A hallucination is something that we know. To be wrong based on the facts. But if you didn't know the us, if you were running that search from Lagos, Nigeria, or Mumbai in India, you might not realize that it is factually inaccurate. And then to your point, what the FDA is dealing with is true novelty. And, and just to be clear. The hallucinations can run hallucinations. Failure to actually be consistent with the facts. As we know, it can range between five to 25% depending on how, how hard the novel the model is working on what kinds of issues. And just to be clear, unlike other forms of technology, when we talk about a model, the way that AI models work. They're constantly running different versions of logic. Uh, we don't, which we really can't backtrack or back propagate to actually determine, to your point, how it got to the wrong answer, which in AI is called the hallucination. I think you can, that's, that's to your point. Let's break this into pieces and then you've got, well, the FDA is primarily dealing with new stuff where we don't necessarily know the answer and judgements. Required. How could that possibly go wrong, David?

David: 12:22

I don't know. So, John, the, I would put this, this piece of, I, I would, I would mention you, you're bringing up a point about transparency. I would say. So normally, you know, you, you put the submission and it's thousands of pages not 'cause it's fun, but 'cause it's required and there's a lot of detail that's needed before you're gonna give a drug to somebody that could potentially cure them and potentially kill them. And the FD needs to look at that. So if we have an approval, how do we know? What they based it on. There was a summary here. You have the same model. Summarize it again. It may be, it may be actually different. What do we do when we go to court and someone has a lawsuit about this? You know, it doesn't make any sense. And let's talk about data privacy. John, we careful when you're using chat GT or something like that, not to put in your personal information. We know that, uh, you know, Elon Musk with his Doge cruise when he came in, take all his government data and put it somewhere that AI's gonna analyze. So now all this submission that you spend hundreds of millions of dollars typically to create as a, as a pharma company, is now gonna go into some sort of model. How do we know that The competitors won't see that, or it won't be incorporated, you know, into the, into the training model.

John: 13:28

But be, before we get there, I think we wanna put in kind of a, um, a one cheer for AI and technology. The one, the, there, there's a, there's a little good news here, first of all, and I think we should probably dig into AI in a more meaningful way because it's gonna be relevant for healthcare. And maybe we do some sections on that and talk about what works, what's possible and what's applied. Because we are gonna have be a million. Clinician short, roughly over the next five years in healthcare. And we absolutely need to leverage technology in novel ways, and we're gonna have to use novel technologies in novel ways. And the the power, these models are really fast at. At, at answering a lot of questions quickly, but to your point, the, the, the, the power of that models and its persuasiveness can sometimes mask inaccuracy. Early last year there was a, uh, lawyer who was using it to file legal briefs, and the model is designed to generate an answer. Even if the answer sometimes isn't there. And so what you found is it was, it was citing all kinds of legal cases, to your point about being in the law that never existed in very persuasive, detailed ways. And so I think that the, the, you can acknowledge the power and the promise of these models while also being a little bit skeptical about our ability today to deploy the models around areas where true life and death is at risk.

David: 15:01

So John, what's, you know, so we see this press release, which I will just go out and say, if I didn't say it before, it's ridiculous what they're planning to do. And I'd say it's not gonna happen. This isn't

John: 15:10

just what you said in the prep session, but I'm glad you've cleaned up your Exactly. Cleaned up your con your, your conversation.

David: 15:16

Exactly. So, um, but you know what's actually gonna happen, you know, let's take a look at, let's look, compare what's happened. I, we're, we're, dude, we're in Trump world. It's

John: 15:27

the, the world. I, what's interesting is how. We know that the White House is a reality TV show. I just, and, and we, and, and increasingly, the Pentagon is a daytime, uh, telenovela. Uh, what we do not need is that the Food and Drug Administration, and we, we both have a, a high regard for Marty as a surgeon. Uh, a as a researcher, as a voice of healthcare reform, but historically, he's been a skeptic of a lot of trends. And this was just a, a, a sort of a, almost like a reaction to the, the, the, the, the, the, the, the, the power of the deep model. Uh, like I, I, I just, I don't know, I don't know how we deal with

David: 16:13

this, David. You know, it. It's just what's, what's one of the ridiculous things about it is that if we, we remember the, the show where he was here and talking about. You know, these blind spots and it's like, oh, you know, people thought peanut allergy was a big thing and that opioids were good. And you know, these were people that actually looked at some evidence. They didn't say, Hey, the first pilot of this peanut allergy thing told me this, and therefore I'm gonna go and do this everywhere. It's nuts. A a,

John: 16:38

a AC Act. Act actually. Well, we will, we'll deal with the, the fact that you're now speaking ironically. Yeah. But it actually was one study. Yeah. And, and, and it was provocative because people use peanut butter and peanuts and it did become this sort of single point. Failure. Uh, and then, and then sort of, uh, um, um. Hysteria around, around, um, you know, uh, uh, peanuts and Right. Marty. What's interesting is, Marty, of all people, his book remember, is called, yeah, just remind our, our, our, our dutiful listeners is called Blind Spots, which is where Madison gets it wrong. And I think what you're saying is, Hey, Marty. You might have either, if it's not a blind spot, you, you, you might wanna take off the dark glasses around, uh, around ai.

David: 17:21

Yeah. Alright. So maybe when his boss, Bobby Kennedy, isn't swimming in sewage, she'll set him straight and put'em on a more rational path. John, what are your, I'm gonna, I'm gonna go talk about what's gonna happen, but let's, what do you think about that one first? Well, got, we've

John: 17:31

got, we've got, we've got, uh, uh, uh, reports and, and pictures of, of, uh, RFK Junior swimming in the Rock Creek Park, which the National Park Service has warned is a, is, is, is. It's collective Collect has a high collection of, of bacteria and is a, is a d depositing ground for raw sewage that's overflow from the DC metro area. So let's, let's as a, as a, as a, as, as a public service message kids. Pets, parents do not swim where Bobby swims. Um, but, but, but get getting back to it. I, I think, I think the balancing act here, David, is we should be leaning in and have to find ways to lean into technology, but they should meet the same standards, the very high standards that the FDA has established, you know, since, since, since it was established. Over a hundred years ago. And that's really the standard. And, and, and we know enough that, um, these models, uh, almost have an urge to complete answers where the, the data isn't there or to create some answers where it isn't. But that doesn't mean that we can't use them. Wisely with, but, but not just de deploy them, uh, randomly. I mean, that, that's really what we are. What's, what's scary about the press release and, and honestly jarring given, given the source of it.

David: 18:54

So John, I know it was Yogi Ber that said, you know, predictions are are dangerous, especially when you're talking about the future. But, uh, here I'm gonna go and say that, you know, what they announced in the press release last week is not gonna happen. So they say that it's gonna be rolled out everywhere by June 30th, and it's gonna be integrated with their systems. And the reason I say this. Is that, first of all, it takes longer to integrate it with their systems than, uh, next month, even if they had everybody working at full speed. And if we look at what's happened in other places in healthcare where it's tried to deploy generative ai, I'm talking about health systems for example. Uh, it takes a lot longer. And the type of people that work at FDA, even those that are remaining are more cautious than that. And there's no way that they're just gonna go and deploy this everywhere next month. It's just not gonna happen.

John: 19:42

Yeah, I, I, I, I worry about that. I, 'cause I do think that this, there's a lot of fear in the federal bureaucracy. There's a lot of fear in DC Metro. People don't wanna lose their jobs. And I, and there's an enormous desire to please, particularly in this administration, the boss. And so I think we, I, I just hope we get the best of Marty's. Um, uh, data and scientific informed skepticism, the best of Jay batter's, uh, scientific pedigree and, and, and somehow apply. I hope that the press release does not actually reflect actual policy because it shouldn't, and, and, you know, in my view, based on my knowledge of the models. But David, I'm not sure we got more on this topic. We may want, we may, we may wanna wrap on, on this, on this note that, uh, hopefully what they're saying they're doing, they aren't doing.

David: 20:33

All right, John. Well, we'll say that's it for another episode of Cure Talk. We've been talking about artificial intelligence and artificial reasoning as it relates to FDA. I'm David. False reasoning. False reasoning. I'm David Williams, president of Health Business Group, and I'm John Driscoll, the chair of the Yukon Health System,

John: 20:51

if you like that you heard you didn't, we'd love you to describe.