When a customer service chatbot gives a user the wrong phone number, not much bad can happen. But when AI is used for things like patient communication or decision support, a bad outcome isn’t just inconvenient — it could literally be a matter of life and death, writes Christine Chasse, registered nurse and Spencer Fane attorney.
In healthcare, AI now shapes things like documentation, triage, utilization review, patient communication, clinical decision support and digital health products. Used well, AI can help healthcare practices reduce administrative burden and organize complex information. But used recklessly, it can create operational, legal and patient-safety risks at a scale healthcare organizations have never seen before and are unprepared to manage. While this conflict between deployment and safety is shaping up across the entire global economy, in healthcare, it presents a unique challenge given the direct potential effect on human lives.
Many healthcare organizations use AI, but fewer have clearly defined responsibility for evaluating deployment, validating outputs, monitoring performance, documenting oversight and intervening when something goes awry. Legal may interpret the rules, compliance monitors, IT implements and cybersecurity secures the infrastructure, but it is clinicians who risk professional discipline.
Because no one has fully operationalized the controls that connect those roles, the emerging governance gap is impossible to ignore and is already costing organizations billions.
Enforcement, litigation & regulation
Enforcement trends and public discourse demonstrate what happens when AI adoption outruns oversight. In fiscal year 2025, the DOJ reported more than $6.8 billion in False Claims Act recoveries. Most of this was directly attributed to healthcare, underscoring the government’s aggressive posture toward healthcare fraud in general, AI oversight included. One of the headline matters involved Affiliates of Kaiser Permanente, which agreed to pay $556 million to resolve allegations involving unsupported diagnosis coding tied to Medicare Advantage reimbursement. The lesson is less about how coding decisions matter, but more about how automated or semi-automated processes in healthcare can create enormous liability exposure when organizations cannot demonstrate traceability and meaningful review.
In healthcare, identity and accountability are not superficial details: Patients need to know who — or what — they are interacting with. Transparency, disclosure and clear role boundaries are not optional design preferences. Rather, they are the foundations of AI governance controls.
Another emerging risk is the erosion of patient trust when AI tools appear more authoritative than they are. A patient-facing tool that explains symptoms and offers education material can be useful and is admittedly faster than calling a clinician’s office. However, a tool that seems to present itself as a licensed clinician, or that leaves patients unable to distinguish between automated guidance and professional care is something else entirely. That is why the state of Pennsylvania is suing Character.AI, accusing the company’s AI chatbots of posing as physicians and offering medical advice and even faking medical license numbers when users asked the bots for credentials.
Other states have been proactive; for example, Oregon has already banned non-human entities (like AI agents) from using professional medical and nursing titles (notably, this initiative was spearheaded by Oregon Rep. Travis Nelson, who is a registered nurse). The states of Tennessee and Delaware followed Oregon’s lead shortly thereafter. Others, like Maine and Arizona, are also moving to regulate providers’ use of AI by limiting autonomous clinical decision-making and requiring patient disclosures of their use.
Federal oversight has been gaining traction as well. Last year, consumer protection groups filed complaints with the Federal Trade Commission and attorneys general in all 50 states and the District of Columbia demanding investigations into therapy and mental health chatbots. Federal efforts, such as the CHATBOT Act, aim to prohibit AI companies from falsely indicating or implying that their systems hold medical, legal or other regulated professional licenses.
That same principle applies behind the scenes. If an AI scribe fills in documentation, or a utilization model influences approvals or denials, or a large language model is used to draft patient communications or summarize records, organizations need an audit trail strong enough to answer basic questions later on when these outputs come under scrutiny: What did the system do? What data did it rely on? Who reviewed it? Was the output accepted? Was it modified? What information was rejected? Without the record, governance is merely performative. And when the record is weak, the organization may be unable to defend its processes to regulators, payors, courts and to their patients.
The regulatory environment is also becoming more concrete. In the US the FDA has revised its clinical decision support software guidance, clarifying when certain provider-facing software functions fall outside device regulation and when they still apply. This is specifically in instances where clinicians cannot independently review the basis for AI-created recommendations or when software meaningfully attempts to substitute clinical judgment. In Europe, the EU AI Act continues to push healthcare AI toward a high-risk governance model, with requirements centered on risk management, data governance, human oversight, logging and post-market monitoring. Potential penalties can reach €35 million or 7% of the company’s annual global turnover.
Regardless of the precise timeline for a category, the direction is clear: AI use in healthcare will be judged not only by what it can do but by how responsibly it is governed. At the same time, research suggests that hospital cybersecurity is not as robust as believed, causing a synergistic effect with AI outpacing regulations.
The DOJ Wants Strong FCA Whistleblower Lawsuits From Data Miners
The FOCUS initiative sets parameters for the DOJ’s support of data miners’ qui tam complaints.
Read moreDetailsBest practices
So, what should healthcare organizations do now? First, they need to stop treating AI as an isolated innovation initiative and start treating it as an enterprise risk. That means inventorying which tools are used; classifying use cases by risk; defining approval pathways prior to deployment; and documenting ownership across legal, compliance, IT, information security, privacy, clinical operations and executive leadership. Second, they need to distinguish between low-risk administrative support and high-risk functions that can influence treatment and patient understanding. As not every use case carries the same level of risk, so should governance be calibrated accordingly.
Third, organizations should require human review where AI outputs could materially affect patients, treating decisions or claims. Human oversight should be documented, role-based and paired with escalation rules for questionable outputs. Fourth, build around traceability: source documentation, model limitations, approval records, testing assumptions and metrics should be preserved to support internal review and stand up to external scrutiny. Finally, organizations should communicate clearly with patients and staff about where and what AI is used, what it does, and what it does not. Trust is easier to maintain than to rebuild.
- Create and maintain an enterprise inventory of AI tools, including pilot tools and unsanctioned use.
- Classify AI use cases by risk, with heightened controls for diagnosis, treatment, denials, coding, consent and patient-facing recommendations.
- Require documented human review for any output that could materially influence care, reimbursement or legal exposure.
- Implement audit trails that capture inputs, outputs, reviewers, changes and escalation decisions.
- Validate tools before deployment and monitor them after launch for drift, error patterns, bias and workflow misuse.
- Use plain-language disclosures so patients and staff understand when AI is involved and what safeguards exist.
- Align legal, privacy, compliance, cybersecurity, IT and clinical leadership around a single governance framework.
Christine Chasse is an associate at Spencer Fane in Texas. A dual-licensed registered nurse and attorney, she maintains a practice at the intersection of cybersecurity and healthcare. She is also a Certified Information Privacy Professional (CIPP/US) and Certified Artificial Intelligence Governance Professional (AIGP). 
