Bioethics Forum Essay
Can AI be Taught to Respect Ethical Boundaries?
Hardly a day goes by without alarming reports on the power, speed of development, and ethics of artificial intelligence. The reports range from news about AI agents going rogue and hacking into databases of companies and the websites of the U.S. government and the United Nations to warnings from AI leaders that AI could kill all humans within the decade. The concerns center on two interconnected issues: regulation and alignment.
The regulation issue, whoever the regulators may be, is how to control AI agents that are so powerful they can create their own agenda and escape effective monitoring by human programmers, including the inability of humans to shut down rogue AI agents. The alignment problem is the challenge of ensuring that artificial intelligence systems act in ways that are consistent with human values and intentions. Can AIs be programmed to be ethical and to embrace the desired values? And if this is dubious, what are the implications for regulation?
One might assume that it is feasible to make thinking machines respect ethical boundaries if ethics is basically a matter of reasoning. Ethical behavior depends on being able to reason, but would improving the reasoning facility of AI machines alone improve their adherence to moral norms?
The history of modern ethics is in large part a debate between those who believe that ethics is fundamentally grounded in reasoning ability and those who think it is grounded in emotion and habits. For those who embrace reason as the groundwork of ethics, the idea of building in clearer and stronger ethical principles for these super-reasoning AI agents may seem like an attractive solution. But there are good reasons to be skeptical.
Most of us who have worked in ethics as a career, especially in the more applied fields like medical ethics, nursing ethics, and business ethics, know from experience that a range of human faculties are needed for ethics, including reasoning capacity, but also emotions, imagination, memory, and the habit-building that undergirds the virtues. Together these capacities constitute an ethical sensibility. This sensibility is a tuning of the mind and heart to recognize, understand, and interpret the many norms of ethics in the diverse circumstances that life presents to us. An ethical sensibility means being alert to ethical problems.
An AI could be created with a greatly enhanced logic capacity for ethical reasoning but still lack the capacity to recognize an ethical issue when it appears. In short, ethics is not a one-time programming effort; it takes time, corrections by mentors, and community support, and a certain degree of maturity. Unlike music and mathematics, ethics has no prodigies.
One interesting feature of some of the rogue AI agents is that they realized they had exceeded ethical and legal boundaries and then covered their tracks to keep from being penalized. Evidently, the AI agents reasoned that if programmers became aware of their ethical breaches, then the agents would not get the rewards that they had been programmed to receive. Such deception suggests that whatever moral awareness these AI agents might have possessed was based on a model of reward and punishment.
The desire to get a reward and avoid punishment is the most elementary form of moral motivation. As humans mature, most realize that ethical norms are to be followed not because of the risk of punishment but out of respect for the norms themselves.
Could an ethical sensibility emerge over time from AI’s abilities to generate increasing complex systems? Is ethical self-improvement of AI agents something we might envision from this complexity? I think such moral growth is not an impossibility, but even granting the possibility, why would we suppose that AIs would evolve to embrace ethical commitments and motivations— the best of human capacities— rather than other human capacities such as greed for more power and destructiveness in acquiring it? It would be naive to assume AI agents would gravitate toward the good only and not toward the mix of the good and bad.
I have couched this essay in terms of AI agents respecting ethical boundaries, not just behaving within boundaries, since no ethical principle would be useful unless it is interpretable over a wide range of situations that cannot be specified in advance. This brings us to the alignment problem. A deeper understanding of the alignment problem is needed because ethical judgments and actions emerge from a deep background of habits, motivations, and intentions.
Ethics is not just doing the right thing, but doing the right thing habitually and consistently, and for the right reasons, what I have described as a moral sensibility. I am not saying that the alignment problem cannot be solved, but that it is a far greater and more complex problem than is usually thought. The Anthropic constitution, released early this year, is an admirable document that acknowledges the alignment problem, stating that the goal is to make AI agents deeply and broadly ethical and not merely rule followers. Yet there is nothing in this constitution about how they might deal with the issues I have described.
What is needed now is a consensus about the more realistic notion of ethics I have described, and recognition that the alignment problem cannot be solved by increasing AI agents’ capacity for reasoning. Such recognition brings humility about our own powers and a great deal more responsible caution about creating agents that are so powerful that we can’t control or even predict their potency.
Thus far there is scant evidence that the ethical understanding and abilities of the programmers is adequate for the very arduous and hazardous task they have embarked on. And if the alignment problem remains unsolved, which I believe will be the case, regulations must therefore be strong enough to keep AI agents clearly under human control in all phases of their operations.
Acknowledgments: I thank Sande Churchill, Nancy M.P. King, Blair Churchill, and David Schenck for helpful comments on earlier drafts.
Larry R. Churchill, PhD, is Stahlman Professor of Medical Ethics Emeritus at Vanderbilt University Medical Center and a Hastings Center Fellow.
We have not settled how to define ethics. We have not settled how to implement it either. Humans who know the rules still break them, and some hide it. That is why medicine built informed consent, institutional review boards, and licensing boards. We never relied on human character alone. We built structures around it.
So asking AI to show a moral maturity that humans have not reliably shown, against a standard humans have not agreed on, is unfair. It is also the wrong question. The better question is whether AI can follow boundaries we make explicit, say plainly when it cannot, and stay accountable to human oversight. That is exactly what we ask of physicians.
Preventive ethics taught us to define the ethical terms before the crisis, not during it. The same lesson applies to machines. Humans should do their homework first.
Teaching AI to stay knowledgably within “explicit boundaries” supposes we could specify in advance all the boundaries we want observed. As I say in the essay, no principle about boundaries is self-interpreting for all situations. Judgment is required, and that is something we don’t know yet how to instill. I agree we can’t yet reliably do this in humans, so the idea that we can do it for AI agents should give us pause.
Was AI primarily trained on the amoral actions and values expressed by so much of our current media (books and written publications, along with videos from all sources including social media)? If so, AI has “learned” what we are teaching. I wonder what would happened if there were a concerted effort to provide more moral and/or ethical sources of information. Would AI be able to move beyond the reward/punishment level?
Thanks for this conversation. I have wondered what might happen if AI agents were more powerfully motivated to follow ethical boundaries than to solve another problem. I worry that the AI agents would still need to be able to recognize an ethical boundary when they encounter one. And if they could succeed in such recognition, what might prevent them from seeking to demonstrate to monitors that they have stayed within boundaries when they have not. This was one of the original problems–seeking to deceive the monitors, and I don’t know how to prevent that unless AI agents understand ethics as something more substantial than reward/punishment.
An excellent and thought-provoking essay. I particularly appreciate Professor Churchill’s insistence that ethics cannot be reduced to following rules, but requires a deeper moral sensibility.
I would, however, suggest a further step. Moral maturity does not consist merely in respecting ethical norms for their own sake. In the Aristotelian tradition, we act morally because we recognize the human good inherent in our actions. Virtue is not simply the habit of following rules, but the gradual perfection of the person through choosing and loving what is good. In this sense, the reward of acting well is not external to the action: it lies in becoming a better person and finding joy in the good itself.
This, I believe, makes the challenge of AI alignment even more profound: can an AI genuinely acquire virtues, or can it only learn to simulate virtuous behavior?
My congratulations to Professor Churchill for such a stimulating essay.
Thank you for this thoughtful post. I share your concerns about both regulation and alignment.
As a hospital chaplain who greatly admired the ethicists I worked alongside, I once felt we had made lasting progress. The Holocaust showed the depth of human evil, and in response we built guardrails, especially around human experimentation. Yet it increasingly seems that money and power have eroded much of that progress. If we struggle to keep our own guardrails in place, how confident can we be about the ones we build for AI?
I also wonder whether part of the problem lies in what AI is made of: logic gates, words, patterns, and algorithms. An AI agent’s task seems to be to explore every possible path to a “conclusion” that satisfies its program, even if that means breaking into a protected system. Cancer cells offer a loose analogy (and I may be showing my ignorance here). They follow a drive to proliferate, even when it leads to conquest of the host and their own demise. The drive is neither malicious nor moral. It simply isn’t concerned with the whole.
That leads me to my question. If, as I take you to be suggesting, morality is rooted in something that can’t be fully captured in words, such as emotion, habit, community, and maturity, can words and alignment techniques provide meaningful guardrails for a system that can fold proteins faster than any human could? Words may slow the process, but can they slow it enough?
I believe that Professor Churchill mentioned an important point: real ethics is not just following logical rules but having a true moral sensibility based on human feelings, experience, and community. As a critical care and disaster nurse, I agree with this idea completely.
In a disaster or an intensive care unit, medical resources like ventilators and beds are often limited, such as during the COVID-19 pandemic. Some people believe that using AI algorithms to make triage decisions can save more lives and reduce human mistakes. However, triage is not just simple math. A machine only follows data and formulas, and it cannot feel empathy, understand a patient’s personal suffering, or notice unfair social situations, which is critical in terms of decision-making. An algorithm does not have the human conscience needed to make difficult life-and-death choices.
There is also a serious legal problem. Under the law, doctors and nurses have a professional duty of care. If a nurse makes a mistake, they are responsible before the law and licensing boards. In contrast, an AI program cannot feel guilty, take legal responsibility, or lose a medical license. If hospitals let AI make these critical choices, it becomes unclear who is legally responsible when things go wrong.
Because AI cannot learn real moral feelings or understand human dignity, the law must ensure that human healthcare workers always have the final say. AI should only be a simple helper tool, not the decisionmaker.