The FDA Needs to Adjust to the Reality of AI Software
Christopher Gardner and Kevin T. Frazier
AI and Health Care: A Policy Framework for Innovation, Liability, and Patient Autonomy—Part 6
Every regulatory process imposes opportunity costs on innovators. Time, money, and attention spent on checking boxes can distract from developing new, better, and safer tools. Regulatory hurdles imposed by the Food and Drug Administration (FDA) are no exception. The FDA’s regulatory review process drives artificial intelligence (AI) developers working on healthcare-related tools away from their most innovative and societally beneficial tasks. Companies attempting to navigate the FDA’s compliance maze may spend millions of dollars and wait years to get a product from development to market. Understandably, many companies don’t even try to find their way through. Rather than explore tools with a significant impact on an important healthcare issue, they may instead develop products for sectors of healthcare statutorily exempt from the FDA’s jurisdiction.
If we want companies to build transformative tools in this domain, FDA regulations must be reformed to better advance both patient protection and American innovation. Our current system does not recognize the clear need and potential for AI in healthcare. Nor does it recognize the widespread adoption of AI tools by Americans for every aspect of their lives—including health. A strong step forward in addressing these issues would be to reform the FDA’s guidance around Software as a Medical Device (SaMD) to allow conditional enforcement discretion in exchange for greater transparency with the FDA.
The FDA’s Regulatory Review Process
The FDA’s regulatory review process for medical devices emerged in 1976 following the codification of the Medical Device Amendments Act (MDAA). The MDAA laid out three risk classes for novel devices. Class I devices, such as a toothbrush or scalpel, are considered to present minimal risk. Nearly all these devices are exempt from the premarket notification requirement. They are instead subject to “general controls” that require the manufacturer to register their facility and device with the FDA. It also requires compliance with more general requirements surrounding labeling, manufacturing, and event reporting.
Class II devices, such as a powered wheelchair or scalpel with a retractable blade, present a moderate risk. These are devices for which the FDA determines general controls are insufficient to ensure the safety of the device, and for which sufficient information exists to establish special controls. These special controls can include performance standards, postmarket surveillance, and patient registries. In order for Class II devices to reach the market, there are two common regulatory pathways. Class II devices that can demonstrate substantial equivalence to a device already on the market are required to submit a 510 (k) and receive FDA clearance. If there is no equivalent device, they must submit a De Novo Classification Request. The De Novo pathway typically requires clinical data, bench performance testing, and a comprehensive treatment on why general or special controls “provide reasonable assurance” of the safety of the device.
Class III devices present the highest risk. Such devices include pacemakers, heart valves, or implantable infusion pumps, and all require Premarket Approval (PA). Unlike class I or II devices, these devices require FDA approval to reach the market, not just clearance. This process requires extensive human clinical trial data, pre-approval manufacturing inspections, post-approval annual reports, and potentially restricted distribution.
Each class of device has its own regulatory pathways and level of scrutiny, but the tremendous cost associated with bringing a device to market is ubiquitous. To bring a Class I device to market, a business must spend $11,423 annually to register each establishment and device with the FDA. For class II and III medical devices, compliance filings to bring a product to market grow exponentially. The combined costs of regulatory filings, bench testing, and FDA compliance can consist of a few hundred thousand dollars for the most basic software 510(k)s. For a complex class III device requiring clinical trials, this cost jumps to tens of millions of dollars.
Instead of spending money on hiring new engineers or developing life-saving technologies, companies must spend millions on lawyers to get their products to market. Additionally, FDA review periods can stretch up to a year. With new AI models being released every few months, this delay means that the submitted product will likely be out of date by the time FDA approval is granted.
Regulatory Uncertainty and Its Impact on AI Software
By placing a price tag of millions of dollars for market entry, the FDA drives American health AI innovators to develop software for use in the narrow areas of healthcare carved out from FDA jurisdiction by Congress. These statutory exemptions direct AI innovators to develop tools for basic administrative work, data collection, and limited data analysis. The opportunity cost of this is tremendous. While AI tools can and should be used in the aforementioned domains, those use cases are far short of what this new technology is capable of. AI can and should be acting as a force multiplier for doctors. It can help tie together the silos of the American Healthcare system, reduce medical errors by specialists, and organize treatment plans.
One of the core regulations holding back the widespread implementation of AI is the FDA’s Software as a Medical Device (SaMD) guidance. This guidance defines any software “intended to be used for one or more medical purposes…without being part of a hardware medical device” as a medical device. In other words, software running on a laptop or smartphone is required to get FDA clearance or approval if it is intended for the diagnosis, prevention, monitoring, treatment, or alleviation of disease.
A quick case study of how this framework applies to a leading AI health tool exemplifies its shortcomings. ChatGPT Health is a version of ChatGPT that allows users to connect their health records and wellness apps. This provides the chatbot additional context for users seeking answers for their health questions. ChatGPT Health could have been a tool to provide the knowledge, perspective, and expertise that medical providers would have by tapping into the medical professional research and literature, empowering providers and patients alike. Providers would benefit from a rapid, reliable check on their work. Patients would be better equipped to ask their providers questions and to address some minor issues on their own.
But OpenAI chose to forgo this clear potential. Instead, the model is narrowly tailored to fit under the statutory general wellness carveout codified in the 21st Century Cures Act. This Act exempts software from FDA jurisdiction if it is intended to encourage a healthy lifestyle and it is unrelated to the diagnosis, cure, mitigation, prevention, or treatment of a disease or condition. Such an exemption means that companies seeking to innovate in healthcare can save millions of dollars by limiting the range and purpose of their tools.
The same phenomenon occurs in areas that the FDA itself has exempted from its regulatory review process. “Symptom checkers” are a prime example of this regulatory practice. “Symptom checkers” are online interfaces where a user can input their symptoms, typically in natural language, and is then asked a series of clarifying questions before receiving a ranked list of possible conditions. These differ significantly from traditional database searches like Google or WebMD, which are only able to give you access to information, not route you through it. AI “symptom checkers” like Ada, or American equivalents Buoy and K Health, expressly fit under the FDA’s SaMD definition by providing consumers “likely” or “potential” diagnoses. However, instead of requiring these tools to go through their regulatory process, the FDA has released guidance that it will exercise enforcement discretion for symptom checkers.
One of the most promising solutions would be to significantly reform the FDA’s Software as a Medical Device (SaMD) regulatory review process.
Fortunately, the FDA is already taking steps to evolve its regulatory regime for AI without imposing undue costs. By releasing guidance allowing “predetermined change control plans” for AI-enabled software devices, the FDA has assisted AI developers looking to improve and update their software in a pre-approved manner. The FDA has also taken action to increase breathing room for AI innovators. It has expanded enforcement discretion for Clinical Decision Support (CDS) systems to allow non-device software to consider limited medical information and provide clinicians with options or recommendations. The FDA has also broadened guidance on general wellness products to include non-invasive sensing intended for wellness use. This means that products such as an optical sensor used to measure blood pressure or oxygen saturation for wellness purposes would no longer be enforced as a medical device by the FDA. Lastly, the FDA rescinded its “Software as a Medical Device: Clinical Evaluation” guidance. This rescission represents a shift at the FDA away from the internationally harmonized framework of the International Medical Device Regulators Forum (IMDRF) and toward a more flexible, risk-based approach.
These moves are a strong first step towards accelerating AI development and innovation, but a few key issues remain. The first is that these shifts in guidance and enforcement discretion do not provide the long-term policy certainty necessary for innovators and investors. Absent formal regulatory changes, the FDA only creates an effective policy environment for the duration of this administration. The next administration may rescind and reverse many of these alterations, potentially costing those who have entered or invested in AI for healthcare billions.
Another important issue is that the FDA’s reliance upon intent doesn’t address the pressing issue of general-purpose AI chatbots. For a device to be under FDA jurisdiction, it must be intended for use in healthcare. Frontier LLMs like ChatGPT, Claude, or Gemini are explicitly consumer-facing and general-purpose. Yet, their training has unintentionally given them extraordinary potential in diagnostic reasoning. It has also given them the ability to distill complicated medical concepts into simple language. This skill set attracts millions of Americans to these tools for their health needs every day. But because these chatbots are general-purpose, they are a tremendously powerful (but incomplete) tool for patients. Moreover, AI developers are hesitant to optimize their chatbots for use in healthcare because this could be interpreted as intent, bringing their chatbots under the jurisdiction of the FDA regulatory review process. As previously discussed, this would cost AI companies millions. Instead of working to maintain a status quo that is both failing and bankrupting Americans, we should encourage American AI companies to improve their healthcare models.
The promise of frontier LLMs in advancing patient autonomy should not be left to languish in regulatory purgatory. One solution would be to establish a framework for enforcement discretion conditioned upon compliance with transparency requirements and adverse event reporting to the FDA. This would extend the FDA’s logic for symptom checkers with added accountability mechanisms. These accountability checks could be modeled after the existing Medical Device Reporting (MDR) system. The MDR system mandates the reporting of adverse events and product issues to the FDA. When combined with regular performance updates to the FDA and disclosure requirements, the FDA is given a clear view of how LLMs are being used and the risks associated. If adopted, this regulatory system would enable the FDA to make informed decisions to address harms while allowing frontier LLMs companies to optimize their products for how Americans are already using them.
Conclusion
AI is not the first general-purpose technology with tremendous potential for healthcare. In the late 2000s and early 2010s, rapid advances in 3D printing technology gave Americans access to a highly adaptable and precise manufacturing process. Similar to AI, this raised concerns over how both doctors and patients alike would use the tools. The FDA’s response routed 3D-printed products primarily through its traditional review process. But the slow and unwieldy nature of the review process resulted in the delay of a key advantage of the technology: point-of-care manufacturing. 3D printing allows for the manufacturing of custom medical devices at scale. This could be a boon for patients, but adoption over the 2010s was extraordinarily slow. In 2010, three hospitals had adopted in-house 3D printing capabilities. This only grew to 99 in 2016, representing less than 2 percent of American hospitals. Even today, adoption levels remain low in part due to regulatory uncertainty. The FDA, now a decade after publishing its first draft guidance on 3D-printed medical devices, still has not issued final guidance or rules on point-of-care manufacturing.
We should learn from past mistakes in 3D printing and embrace the potential of AI. The FDA’s regulatory review process is tremendously costly and poorly suited to new technologies. Such costs drive frontier AI developers to limit their tools to avoid FDA clearance or approval requirements. This harms Americans. It means that AI companies aren’t attacking the hard problems in healthcare, and frontier LLM developers aren’t optimizing their tools to provide Americans the best health advice possible. The FDA is taking clear steps to address these issues, but it should use the APA process to ensure long-term policy stability. It should also acknowledge that millions of Americans are using LLMs for their health concerns and provide a pathway for AI companies to improve their products without spending millions on regulatory compliance.
To read other parts of this blog series, go here.