OpenAI, ChatGPT’s sycophant problem, Absci design

You’re reading the web edition of STAT’s AI Prognosis newsletter, our subscriber-exclusive guide to artificial intelligence in health care and medicine. Sign up to get it delivered in your inbox every Wednesday. 

We have a lot of new readers this week! Welcome — it was so great to meet many of you at our San Francisco event.

I have some unfortunate news for you regarding some ice cream products and your gut microbiome. As long as Jeni’s doesn’t use those ingredients, I think I’ll be OK, though.

If you want to forward this email to your friends, remind them that they can subscribe to STAT+ and get this email in their inbox every week for 50% off with our Memorial Day sale. And as always, if you have questions or comments: [email protected]

OpenAI’s history and future

If you’re anything like me, what finally made you stop rolling your eyes at the term “artificial intelligence” was OpenAI’s ChatGPT.

Karen Hao, formerly a journalist at MIT Technology Review and the Wall Street Journal, has a new book about OpenAI’s pursuit of an AI that matches human intelligence — artificial general intelligence, also known as “AGI.” Her play-by-play of how OpenAI brought us the LLM technology we know today (which isn’t AGI, just the project that it thought would be best to show to Microsoft to get an investment) is chilling. 

One example: GPT-2 was trained on a high-quality subset of articles and web pages. But GPT-3 needed more data, so the developers decided to include a wider subset of articles, English-language Wikipedia, and a mysterious repository of likely-pirated books and scholarly articles. But that was still not enough data, so developers turned to the Common Crawl, a scrape of the entire internet, with a filter to get rid of the lowest-quality content. For GPT-4, OpenAI needed even more data and was running out of places to get it, Hao writes in the book. So the company dropped the filter on the low-quality content and also began scraping transcripts of YouTube videos.

And now? When I asked Hao what she wanted to say to the readers of this newsletter, she pleaded with people not to upload their health data to ChatGPT, connecting it to the recent OpenAI sycophancy scandal: The company rolled out an update a few weeks ago to make ChatGPT more friendly, then rolled it back immediately when ChatGPT began encouraging and affirming everyone, even people who expressed violent thoughts like those of self-harm. Hao says that AI companies are trying to maximize users’ engagement with AI chatbots, just like social media did, because they’ve run out of other data to train on. 

“They’re basically trying to just create a tap straight into the source. What’s better than scraping the internet? It’s literally just pulling it directly from you,” she said. (ChatGPT will train its models on your conversations unless you opt out.)

This is why ChatGPT has a free tier, she said. Getting user data directly gives OpenAI a competitive advantage. “Any company can scrape from the internet. But if you’re talking directly to ChatGPT, then other companies are not getting that data.”

Hao’s book, released yesterday, is called “Empire of AI: Dreams and Nightmares in Sam Altman’s OpenAI.” It made me think differently about the world we want to build with artificial intelligence. Our conversation about OpenAI, AI, and health care below has been lightly edited for length and clarity.

After ChatGPT, we have a certain conception about what AI is. But reading your details about how it came about, it feels like we didn’t have to end up with language models as the thing that would launch AI into the public consciousness. Do you feel like there are other realities where our understanding of AI would be different? 

I’m really glad that you picked up on that in the book. One of the things that I really tried to highlight was key moments in OpenAI’s history in which they decided to take the next step on a particular path — they decided that they were going to try and scale these models aggressively and therefore they decided that they needed to raise tens of billions of dollars to do that and that they would scrape the entire internet to do that and that they would exploit a lot of contract workers in the global south to clean up those loads of data sets.

I think there could have absolutely been other realities, and part of my hope with the book is to also really underscore to people that there are still multiple futures and we have all the collective agency to decide which future we want and we shouldn’t just accept the fact that Silicon Valley has fired their first shot — we shouldn’t just accept what they want as our future reality.

How has OpenAI promised to change or leverage AI in health care?

One of the biggest promises that they have said since the very beginning [is], “We want to build AGI in part because it will bring greater access to health care, to everyone.” And this has been something that not just OpenAI does, but a lot of AGI developers have turned into dogma within this world, that somehow when we build this technology that no one can really define, it’s going magically make it better for everyone, it’s going to cure cancer, it is going to make psychotherapy super cheap and super effective. 

The story you sent me just last week is an interesting way that they’re now trying to more explicitly signal the gains they’re making when it comes to health care. [They say], “Oh, here’s a benchmark that shows we’re actually making advancements here,” or like, “Here’s a new collaboration with a partner to focus on this particular issue.” But the fact of the matter is, [Open AI has] been around for almost a decade now, and they haven’t actually made any substantive steps in giving more accessible health care to people. Their chatbots hallucinate like crazy and spew a lot of medical misinformation. There have been many other AI applications in health care that have demonstrated really great and effective uses within the health care industry, but OpenAI’s technologies are not among those.

How do you go about separating facts from fiction here? Because to some extent, eventually, yeah — if we had this perfect AI, it would help. But how do we separate the promise from what is actually possible?

I think there’s two ways to do that. One is just looking at the track record. OpenAI has done a lot of rhetorical maneuvering to suggest that the benefits of what they are building or are on their way to building will arrive only once it’s done, like, “AGI will solve health care once AGI is built,” or “AGI will solve climate change once AGI is built,” but that’s not really how the technology works. If that is something that they actually cared about making progress on, they would have already done that, because there are already plenty of AI technologies that have actually made an impact in the health care space. And so if OpenAI hasn’t made great strides on the path to what they say will be AGI, that is already a really, really strong signal that that’s not actually what they’re focused on. 

The other thing is, I have been covering AI for a long time, and one of the privileges I had was to read a lot of AI research before AI hype made everything really, really noisy.

[Recently], there was this really good piece in the New York Times by Cade Metz. The headline is “Why We’re Unlikely To Get AGI Anytime Soon” and there’s this paragraph that says, “In a recent survey of the Association for the Advancement of Artificial Intelligence, more than three-quarters of respondents said the methods used to build today’s technology were unlikely to lead to AGI.” I think that’s a really, really important dimension of AI discourse today, is it’s so polluted by people who are financially motivated. They have every financial incentive to pretend AGI is well-defined, that AGI is right around the corner, and that AGI can magically wave a wand and make everything perfect. But when you actually talk to scientists who’ve been doing this for a long time and are not being paid off by these companies, there’s enormous amounts of debate on all of those things. 

I think people want to be optimistic, and people want to believe, and people answer based on that hope. It’s kind of hard, sometimes, to have a real conversation with people to get past that optimism to the mechanism for benefits.

There’s a reason why these companies continue to use these promises over and over again. The idea that a system one day will cure cancer — who hasn’t known a loved one who has been affected by and potentially died from cancer? That’s why these companies wheel that promise out again and again because they know that is the thing that people want so badly that they are willing to suspend their disbelief a little bit, to just hold out on that hope. 

But the problem is, there is no actual evidence that we’re actually getting there with the type of technologies that these companies are developing. There is plenty of evidence that we can have some impact on just generally earlier cancer detection, also better drug discovery, with totally different types of AI models than generative AI models and language models. There has been plenty of work done on creating task-specific, small, deep learning models that make it much easier to detect different types of cancer, different types [of] neurodegenerative diseases far earlier, and in a way that is assistive to doctors and medical professionals, such that people have a much higher chance of being able to beat the disease. And there is also plenty of work that was done far before ChatGPT came into the picture […], but all of that kind of work is being overlooked now and under-invested in because Silicon Valley has captured everyone’s imagination and captured so much funding to pour into a technology that has not shown any kind of track record that it is actually helping with health.

From the STAT archives:

• On the topic of OpenAI promoting its partnerships to prove it’s improving health care, check out Mohana Ravindranath’s story from last year on this topic: OpenAI isn’t built for health care. So why is its tech already in hospitals, pharma, and cancer care? 

• As Hao mentioned, last week OpenAI came out with a new health benchmark data set. While experts hailed the amount of effort and resources OpenAI put into the openly shared resource, OpenAI’s models also performed the best on the test. Is that because they’re really that good, or because the models or the test are particularly attuned to OpenAI’s strengths? ICYMI last week, read more from me here.

What’s an “AI-designed” drug, and how good is AI compared to traditional experiments?

Last week, AI drug development startup Absci announced that its first candidate, ABS-101 for inflammatory bowel disease, has entered the clinic. Though the company doesn’t say so in its press release, media coverage still claims that Absci designed the antibody drug “from scratch” with AI. (The company simply says it is the “first AI-designed biologic for IBD.”)

But what’s the definition of an “AI-designed” drug, anyway? Multiple other companies have laid claim to the title of “first AI-designed drug.” 

In Absci’s case, the recently published patent application for its drug shows what went into ABS-101: Out of the six binding areas (called “complementarity-determining regions,” or “CDRs”) on an antibody, Absci “AI-optimized” one of them and “de novo designed” two of them, as seen in the image on the cover sheet of the patent application (below). 

But the areas Absci designed with AI may not really change the drug, said Sarel Fleishman, an associate professor at the Weizmann Institute of Science and founder of computational protein design startup Scala Biodesign. The L1 and L2 areas that Absci “de novo designed” are not very often involved in antigen binding, he told me via email, and all antibodies are structurally very similar in those spots and can’t change much.

Earlier this year, I reported that neither Absci nor Generate:Biomedicines, despite their marketing claims, have documented that they can develop antibodies from scratch (or “de novo”) with AI. How’s the field at large doing with this task?

At the PEGS Boston conference last week, Specifica chief scientific officer Andrew Bradbury presented preliminary, anonymized results from the “AIntibody” competition he hosted to test this very question. 33 teams from large and small pharmas, AI biotechs, and nonprofits/academia all participated. The overall verdict? Even the best AI-generated antibodies weren’t as good as the ones created through experimental techniques. “AI performed better than I was expecting, but it wasn’t performing as well as it was hyped up to be,” said Bradbury. 

Last week, health care leaders in 43 of 67 health systems in the Scottsdale Institute, a health system membership organization, said they believed “immature tools” to be the number one biggest barrier to AI adoption. 

The study was published in the Journal of the American Medical Informatics Association and has lots of interesting conclusions, including a chart showing the AI areas in which health systems have deployed tools, and the stages they’ve reached. Tools for imaging/radiology and early detection of sepsis were even more popular than ambient note AI tools and have been deployed for a longer time, but only 19% of respondents noted a high degree of success with tools for diagnosis. This indicates a need for developers to go back to doctors and figure out how to make their tools better, the authors said.

Last week at the STAT Breakthrough Summit West, I led a panel on legal liability, transparency, and other issues keeping health systems from implementing AI. STAT’s Tara Bannow recapped that panel here. My other favorite panels were Casey Ross’ interview with Megan Bent, whose father died after UnitedHealth used an algorithm to send her father home from rehab too early, and Allison DeAngelis’ one-on-one with Reed Jobs, Steve Jobs’ son who now has his own oncology-focused investment fund, Yosemite.

Takeaways from SAIL — AI ROI is important, and missing

Earlier this month, the annual Symposium on Artificial Intelligence in Learning Health Systems (SAIL) conference convened in Puerto Rico to check in on how the translation of AI to health care is going. The conference operates under the Chatham House Rule — you can repeat anything that was said at the conference, but can’t say who said it — so UC Berkeley computational precision health professor Irene Chen compiled her takeaways into a list of de-identified quotes, which you can find here. I called Chen to ask for more context on some of them.

One topic we talked about was the tangible return on investment in ambient scribes. It’s clear that there’s no hard ROI right now, she told me, and that might not actually exist at all. “If everyone had come in being like, ‘Yeah, yeah, the papers are coming, but anecdotally it’s crushing it,’” it would be a different story, she said. “The amount of hype there is out [there] right now, you would think it would have [a] clear impact on everything right now. And I don’t think that has been the case.”

One speaker on a panel also seemed to indicate that this lack of ROI for AI tools might be a problem for AI adoption when hospital budgets are getting squeezed by Medicaid changes, NIH grant funding being lost, and other changes. In such an environment, hospitals may lean toward keeping head count over investing in a technology that hasn’t demonstrated hundreds of millions of dollars in value. “AI needs to demonstrate that it’s worth 10 nurses,” said Chen, summarizing the speaker’s argument.

Song of the Week: “MYSTERY” by Turnstile

When Turnstile’s 2021 album GLOW ON leapt to the top of album-of-the-year lists, I first almost dismissed it as angsty “hide in my teenage bedroom and turn up the music” songs. But I couldn’t stop listening to it. If you don’t like hardcore but wish you did, this record is for you.

And there’s always a health care connection: Turnstile has a new album, NEVER ENOUGH, coming out June 6. In anticipation, the band recently held a free preview concert in Baltimore, which it used to raise $35,000 for Baltimore’s Health Care for the Homeless.

Do you have questions about what’s in this week’s newsletter, or just questions about AI in health in general? Suggestions? Story tips? Ideas for song of the week? Simply reply to this email or contact me at [email protected].