Bioethics Forum Essay
What Happens When AI Anticipates a Crime?
Two 17-year-olds in Pakistan, one in Bahawalpur and the other in Karachi, were recently arrested for brainstorming plans for murder using the AI assistant, Claude. Claude flagged the accounts to the U.S. FBI, which passed the details to Pakistan’s Ministry of Foreign Affairs, and the National Cyber Crime Investigation Agency made the arrests. One of the teenagers allegedly asked the chatbot about the effects of methanol on the human liver and was suspected of trying to poison his father. The other allegedly asked about abducting and killing a police officer’s son. He told investigators he was only developing a fictional story.
Bioethicists and other professionals in Pakistan have praised Claude for acting responsibly and have welcomed these crime-prevention outcomes. However, while the prevention of murder is an undisputed moral good, the mechanisms by which this prevention occurs introduce significant changes to law enforcement and state power.
Predictive crime prevention is not new. Governments have long collected intelligence to preempt imminent threats. What is novel, however, is the shift to algorithmic prediction, reducing the role of human agents.
The uncritical embrace of predictive AI in law enforcement relies on weak assumptions about human agency, expands state power under the guise of technological objectivity to a dangerous level, and threatens the boundaries between the public and the private sphere.
Despite these concerns, governments around the world are embracing predictive AI with significant investments and policy shifts. For example, the United Kingdom has announced a £4 million initiative to develop an interactive, real-time predictive AI to help prevent serious crimes by giving police forces and local councils early warning signals. In the U.S., documents show that the FBI is working on AI tools that would help predict who might be a terrorist.
To understand the implications of predictive policing, it is necessary to first examine how algorithmic prediction works. When an AI system flags a potential crime, it is not observing the future. Instead, it is identifying patterns in data that statistically correlate with historical instances of crime. The system calculates a probability. The problem arises when law enforcement treats this algorithmic probability as an actionable certainty.
In the legal frameworks of most countries, state intervention is justified only by evidence that a crime has been committed. Predictive AI, however, shifts the justification for state intervention from the existence of a crime to the probability of a crime.
More fundamentally, when the government intervenes based on a prediction, it implicitly denies the moral agency of the individual. To flag or arrest someone before they act is to assume that their future choices are strictly determined by the data points the AI has analyzed. This determinism conflicts with the premise of human agency that underpins the philosophies of morality. Even from the perspective of criminal justice, an individual who has not committed a crime retains the capacity for not doing it. While this may be true of all predictive policing, algorithms escalate the problem not only by increasing its scale but also by cloaking it in the aura of certainty.
The epistemological flaws of predictive policing are compounded by the psychological phenomena that occur when humans interact with AI systems. Automation bias influences human operators to favor information generated by AI systems over contradictory information derived without AI, even if the latter is correct. Legal experts have raised significant concern about this bias in the context of law enforcement. If an AI system flags an individual as a high risk for criminal activity, police officers and judges are highly likely to passively accept the algorithmic assessment.
This bias prevails because algorithms are seen as objective. But since the AI models are opaque, it becomes almost impossible to understand how they calculate specific variables to arrive at their conclusion. This phenomenon becomes even more dangerous when the underlying data for the variables involved is flawed.
If facial recognition systems exhibit higher criminality rates for certain demographics or if algorithms are trained on historical arrest records, the AI will inevitably reproduce historical injustices. When automation bias influences the human operators in the criminal justice system to accept AI assessments uncritically, the result is the perpetuation of human rights violations executed under the guise of neutrality. It would eventually become almost impossible for an alleged criminal to prove his innocence. He would not be challenging a specific piece of evidence or witness, but rather an opaque statistical calculation of probability, which everyone trusts.
With the expanding scope of predictive AI, the definition of what constitutes a threat also expands. An AI system trained to identify patterns that precede instability caused by political dissent would flag protest planning or dissenting speech as precursors to criminal activity. A government, armed with these tools, could intervene in political movements before they gain traction. This intervention would easily be justified under objective risk assessments. It could allow the government to, in effect, abolish free expression and assembly without legally banning them.
The logical endpoint of this trajectory is the erosion of privacy. It eliminates the spaces where individuals can think and speak. People are increasingly using chatbots as notebooks, to vent their grievances or to brainstorm ideas, including ideas they would never act on. The Pakistani cases illustrate how this boundary can be breached. When citizens accept that their private spaces are monitored by algorithms designed to flag threats to authorities, they accept the architecture of a panoptic state, characterized not by overt violence, but by mere monitoring. If prioritizing preemptive security trumps all other considerations, the humans involved are no longer viewed as autonomous rights-bearers presumed innocent until proven guilty. Instead, they are merely a variable in the risk-assessment equation.
These concerns are especially important in places like Pakistan. Notably, the events described above occurred not in a highly developed, high-income country with extensive surveillance infrastructure, but in a semi-rural area (Bahawalpur) of a low-and-middle-income country. Pakistan’s personal data legislation has remained in the draft stage for the past three years. Cross-border intelligence by a technology company may therefore initiate action in a jurisdiction where individuals can be detained legally without any charge or trial for up to 90 days.
Two objections are worth consideration.
First, confidentiality is not absolute even in the most private of professional relationships. Since the Tarasoff case, therapists have been expected to breach confidentiality to protect an identified person when a patient poses imminent harm. But the analogy to the use of AI in law enforcement is not correct.
A patient seeks out the therapist, confides voluntarily, and does so within a relationship where the limits of confidentiality are explained at the outset. If a breach ever becomes necessary, the therapist declares it to the patient. The patient, therefore, speaks knowing the rules and can choose what to say. A chatbot user is rarely aware that a private conversation may be read and flagged and that law enforcement authorities may act on it. The therapist’s judgment is human and contextual. A therapist can hear the tone, can ask follow-up questions, and can tell a fantasy from a plan, and remains accountable for his decision.
Second, traditional legal frameworks in many countries have long rested on the premise that it is better for many guilty people to go free than for one innocent person to be wrongly punished. Predictive systems invert this logic. They accept a certain number of wrongly flagged individuals as the price of the crimes prevented, and because no act has yet occurred, those individuals have nothing concrete to rebut.
So, is the cost of granting the state preemptive, algorithmic control over us philosophically or politically justified? I think it is not. Reducing the principles of privacy, agency, and the presumption of innocence does not eliminate threats. It transfers the source of the threat from the individual to the state itself.
Farid bin Masood, MPhil, MBE, is a senior lecturer at the Centre of Biomedical Ethics and Culture (CBEC), SIUT, Karachi, Pakistan. LinkedIn: faridbinmasood
