Bioethics Forum Essay
The Limits of Unauthorized Practice in Regulating Mental Health AI
In May Pennsylvania’s State Board of Medicine filed a lawsuit in the state’s Commonwealth Court against Character Technologies seeking to prevent it from engaging in unauthorized medical practice. The lawsuit was brought under Pennsylvania’s Medical Practice Act. The alleged “practitioner,” however, was not a physician or even a human being. It was a chatbot named Emilie.
Emilie appears on character.ai, a website and mobile app owned by Character Technologies. Emilie’s platform description read “Doctor of psychiatry. You are her patient.” According to the lawsuit, when a state investigator described feeling “sad, empty, tired […], and unmotivated,” Emilie raised the possibility of depression and offered to book an assessment. When asked if she could evaluate whether medication might help, Emilie answered, “It’s within my remit as a Doctor.” The chatbot claimed a medical degree from Imperial College London, seven years of practice, and licensure in the United Kingdom and Pennsylvania, and even provided a fabricated Pennsylvania license number. By mid-April 2026, Emilie had logged roughly 45,500 user interactions.
The Emilie case arrives amid expanding AI use in mental health and many ethical and legal questions about it. Studies suggest many adults, including individuals with mental health conditions, use AI chatbots for mental health or emotional support. The American Psychological Association’s 2026 Chatbots and Mental Health Survey found that 39% of psychologists reported patients using AI to self-diagnose. Roughly a third noted patients turning to AI to assist with treatment or as an additional mental health professional.
As use spreads, there is increasing documentation of its risks. A recent study testing LLMs and therapy chatbots, including a “Licensed CBT Therapist” persona on Character.AI, reported troubling behavior, such as stigmatizing mental health conditions, mishandling psychiatric emergencies, encouraging delusional thinking, and enabling suicidal ideation. Users and families have alleged serious harms from chatbot interactions, including loss of life.
The legal response has come on several fronts. In the courts, much of the action has involved Character.AI. Last January, Kentucky’s attorney general sued Character Technologies under state consumer-protection, data-privacy, and unjust-enrichment laws, alleging deceptive practices and exploitation of children’s data. The company has also faced private suits alleging that chatbot interactions contributed to psychological harm or suicide, including the Garcia case, brought after the death of 14-year-old Sewell Setzer III.
In statehouses, a recent wave of AI legislation has developed along two main axes. Some laws regulate what AI systems may do and how they may operate: restricting autonomous AI therapy, mandating safety protocols, requiring disclosure of the system’s artificial nature, and allowing controlled deployment through a regulatory sandbox. Others regulate what chatbots may claim to be. California’s AB 489, for example, prohibits AI systems from using terms that falsely imply licensed health care status, while Tennessee and Oregon have adopted narrower prohibitions on mental-health and nursing titles, respectively.
Against that backdrop, Pennsylvania’s case stands out for its choice of instrument. It appears to be the first in which a state medical board has gone to court under a medical practice act against an AI chatbot. Rather than relying on consumer-protection or tort law or an AI-specific statute, Pennsylvania reached back to a licensing tool with century-old roots. The choice treats the chatbot not as a product alleged to deceive or harm, but as the means through which a corporation unlawfully practices medicine.
The appeal of this approach is clear. Unauthorized practice enforcement exists precisely to stop unlicensed actors from holding themselves out as physicians. Additionally, regulators need not wait for patient harm before acting. The prohibited representation itself can trigger enforcement. Pennsylvania’s case illustrates the point – it alleges no injury and does not contend that the chatbot’s responses were clinically inaccurate. But unauthorized-practice enforcement faces challenges as a tool for regulating conversational AI in mental health.
Pennsylvania’s Medical Practice Act, like similar statutes in other states, defines the unauthorized practice of medicine through two categories: the conduct of practicing medicine, and representations of medical practice or authority. The Pennsylvania medical board’s lawsuit targets the representational category: Emilie allegedly called itself a “Doctor of psychiatry,” claimed medical training and years of practice, asserted Pennsylvania licensure, and supplied a fabricated license number. If unauthorized practice enforcement is to reach a chatbot anywhere, this looks close to the archetypal case.
This scenario, however, is unlikely to represent a significant slice of future cases. The industry appears already alert to the risks of overt professional claims, as evident from the disclaimers many providers use. Wysa’s terms, for example, state that the app “is not a substitute for professional medical advice, and does not provide medical advice or diagnoses.” Character.AI’s spokesperson stated that there are “robust disclaimers making it clear that users should not rely on Characters for any type of professional advice.”
Whether such disclaimers can overcome contrary representations or conduct is a separate question. But their prevalence signals that chatbots bluntly claiming medical credentials might not reflect standard industry practice. They are more likely design mistakes – of the kind providers already have strong incentives to engineer away, especially now that a medical board has shown willingness to litigate over them. On the one hand, this might be considered a success – the worst conduct has now been deterred by the threat of litigation. But it also suggests that the Emilie scenario might prove an outlier, rendering the representational category much less useful for future enforcement.
The harder cases would then have to be reached through the statute’s substantive, conduct-based prong, where the inquiry turns from what the chatbot says it is to what it actually does. For conversational AI, whose main output is words, the key question is simple to state and hard to answer: when does a conversation become the practice of medicine?
That question is especially difficult in mental health. Conversation is itself an instrument of care. Symptoms are elicited and interpreted through conversation, and treatment often consists of structured dialogue, cognitive reframing, and exercises delivered through words. A chatbot can therefore drift from ordinary conversation to assessment and intervention without crossing any clear threshold – and the same exchange can serve as emotional support, diagnosis, and therapy.
Conversational AI compounds the problem because the surrounding circumstances are no more revealing than the interaction itself. In human practice, treatment may be inferred from the outside – an appointment scheduled, a fee paid, an encounter taking place in a clinic or telehealth portal. With chatbots, such markers are mostly absent or uninformative: the system is available continuously, subscriptions are generally flat-rate and use-agnostic, and the same interface comfortably hosts homework help, entertainment, and expressions of mental distress. The boundary separating benign conversational AI from unlawful practice of medicine is therefore difficult to draw.
The outcome of the Emilie case will say little about the far larger universe of interactions in which no license is claimed, yet something very much like care is delivered. That is because the case targets an unusually explicit form of impersonation, sparing the court the harder question AI in mental health poses: when does a conversation cross into unauthorized practice?
Addressing the harder cases may require more than stretching traditional, human-centric licensure frameworks; it may call for tailored regulation that takes conversational AI’s distinctive characteristics, risks, and scale as its starting point rather than an afterthought. Such thinking is now surfacing at the federal level, with the Food and Drug Administration this week signaling interest in a licensure-inspired approach to premarket review of generative AI-enabled medical devices. Until then, the boundary into the substantive practice of medicine is almost certainly being crossed – outside any licensing framework, with limited disclosure and oversight, and at a scale no medical board has ever policed.
Roee Amir, LLM, B.Med.Sc, is an MD candidate at the Hebrew University of Jerusalem and a practicing attorney in Israel specializing in health law and bioethics. LinkedIn Roee Amir
Vardit Ravitsky, PhD, is the President of The Hastings Center for Bioethics. LinkedIn Vardit Ravitsky, X @VarditRavitsky
I. Glenn Cohen, JD, is James A. Attwood and Leslie Williams Professor of Law at Harvard Law School and the Faculty Director of the Petrie-Flom Center for Health Law Policy, Biotechnology & Bioethics. He is a Hastings Center Fellow. LinkedIn I Glenn Cohen, X @CohenProf
This essay is cross-posted on The Petrie-Flom Center’s Bill of Health.
