Bioethics Forum Essay
When Medical AI Works Well: Some Ethical Questions
As a philosopher who works in a medical school, I spend a significant amount of time talking with physicians, scientists, and students who are rightly excited about what AI may make possible in medicine: earlier diagnosis, better predictions, faster drug discovery, more efficient care.
I also spend a lot of time thinking about what technologies do to the practices into which they are introduced. Such considerations may seem out of step with recent debates about AI as an existential risk to humanity. My concern here is much closer to home: What happens when AI becomes successfully integrated into practices such as medicine.
Much of our public debate about medical AI is focused on how to prevent it from going wrong. We tend to worry about familiar risks surrounding medical AI: racial bias, threats to patient privacy, hallucinated medical facts, or inaccurate medical advice. Much of our ethical and regulatory response follows from those worries. We call for accuracy testing, bias audits, privacy protections, safety guardrails, and human oversight.
These concerns are crucial, but some of the most important consequences of medical AI may emerge not when it fails, but when it works as intended, when it becomes accurate enough, useful enough, and trusted enough to reshape how medicine is practiced. Successful technologies do not simply make existing practices more efficient. They change what we notice, what we ignore, what we value, what we feel responsible for, and what becomes difficult to refuse.
Consider the promise of prediction. Medicine has long been oriented toward anticipating disease before it appears. Screening programs, genetic testing, risk calculators, and preventive care already ask us to act in the present for the sake of a possible medical future. Medical AI intensifies this tendency by making predictions more precise and more readily integrated into everyday care.
In many cases, that could be lifesaving. But prediction also changes what it means to be well. A person who feels healthy, has no symptoms, and may never develop the disease in question can nevertheless become a patient-in-waiting. Life is reorganized around a predicted future. More tests are ordered. More appointments are scheduled. More decisions are framed as responsible prevention. Sometimes this may be exactly what medicine should do. But it also creates a burden.
Once a risk is visible, it becomes hard not to act on it. Patients may feel irresponsible if they decline monitoring. Clinicians may feel negligent if they do not order further tests or fail to recommend drugs that might lower the risks of developing a disease. What begins as information can become obligation.
This is one of the quiet powers of medical AI. It does not merely describe a patient’s condition. It can change the action around that patient. A risk score, an alert, or a recommendation can make one path seem prudent and another path seem careless. It can transform “you may want to consider this” into “how could you ignore this?”
Similar concerns arise in relation to clinical judgment. We often reassure ourselves that as long as humans remain “in the loop,” doctors, not algorithms, will make the final decision. But this phrase can obscure more than it reveals: Being in the loop does not mean being in the same position as before.
A physician who receives an AI recommendation is not simply making an independent judgment with one more piece of information. The recommendation frames the case. It ranks some possibilities as better than others. It often comes backed by institutional approval, statistical validation, and the expectation that any departure from it will need to be explained. The physician can still disagree. But disagreement may now call for justification in a way that agreement does not. If the doctor follows the AI and the outcome is bad, responsibility can be spread among the clinician, the institution, and the vendor. If the doctor rejects the AI and the outcome is bad, the decision may look reckless.
AI may also affect clinical judgment by changing how expertise is developed and maintained. Expertise is built by practice: seeing patients, confronting uncertainty, noticing anomalies, making decisions, and learning from the consequences. If AI increasingly performs the first pass of diagnosis, triage, literature review, or treatment planning, clinicians may become very good at supervising AI systems while losing opportunities to practice the forms of judgment those systems are meant to support.
We have seen versions of this trajectory before. Technologies that assist us can also de-skill us. Navigation systems help us get where we are going, but they can weaken our sense of direction. Spell checkers improve our documents, but they can make us less attentive to spelling. In medicine, the stakes are far higher. We do not only need clinicians who can accept good recommendations. We need clinicians who can recognize when a recommendation should be challenged and remain able to act on that judgment.
None of this is an argument against using AI in medicine. It is an argument against evaluating AI only by asking whether it is accurate, safe, and unbiased enough to use. Those standards are necessary, but incomplete. A system can be accurate and still change what counts as a good medical reason in problematic ways. It can be useful and still make it harder for clinicians to depart from its recommendations. It can improve efficiency and still make medicine less attentive to explanation, judgment, and care.
Medical AI may help us see more, predict earlier, and act faster. That is its promise. But medicine is not only a system for producing correct answers. It is a human practice organized around care, judgment, responsibility, and trust. The danger is not only that AI will get medicine wrong. It is that AI will get medicine right in ways that obscure what good medicine requires.
Inmaculada de Melo-Martín, PhD, is a professor of medical ethics at Weill Cornell Medical College and a Hastings Center Fellow. LinkedIn: Inmaculada-de-melo-martin
