Artificial intelligence can calculate probabilities in milliseconds. It can analyse thousands of previous cases, draft contract clauses, recommend settlement figures, and even estimate litigation outcomes. Yet an essential question remains: can it understand fairness, responsibility, or human dignity?
AI is increasingly entering spaces traditionally reserved for human judgment. Negotiation platforms propose settlements, risk-assessment algorithms inform criminal justice decisions, and automated systems assist lawyers, insurers, and prosecutors in determining offers. These tools promise efficiency and consistency. But the law has never been only about efficiency. It is about judgment. And judgment remains profoundly human. For that reason, even when AI participates in negotiation processes, humans must ultimately decide.
When AI Enters Negotiation
When we say that AI “negotiates”, we do not mean that machines replace human negotiators entirely. Rather, AI systems increasingly take part in or support negotiation processes by assisting the people involved.
Today, AI tools can:
- Analyse large datasets to predict likely negotiation outcomes
- Recommend settlement amounts or negotiation strategies
- Draft contractual clauses or legal documents
- Conduct automated online dispute resolution
- Assist insurers, prosecutors, or lawyers in determining offers
Examples already exist across legal systems. Insurance companies use algorithmic tools to estimate settlement values for personal injury claims. Online dispute resolution platforms help resolve consumer conflicts through automated negotiation processes. In some jurisdictions, algorithmic systems analyse sentencing data to suggest plea agreements.
In short, AI participates in negotiation when it influences the terms, strategy, or potential outcomes of bargaining processes traditionally conducted by humans.
What Is Human Oversight?
Human oversight refers to the requirement that a qualified human decision-maker remains actively involved in the use of AI systems within a specific decision-making process. This means that a human must review the recommendations produced by the AI, understand the reasoning or methodology behind them to a reasonable degree, and retain the authority to modify, reject, or override the system’s suggestions.
Most importantly, the final decision must remain attributable to a human who bears legal responsibility for the outcome. This principle is reflected in the European Union’s Artificial Intelligence Act, which requires high-risk AI systems to be designed in a way that allows meaningful human intervention. Systems used in legal contexts, particularly those affecting fundamental rights, fall squarely into this category. In essence, human oversight ensures that artificial intelligence functions as a decision-support tool rather than an autonomous authority.
Why Human Oversight Matters
There are several legal reasons why meaningful human oversight remains indispensable when AI systems participate in negotiation processes. These reasons are grounded in fundamental principles of accountability, fairness, and equality.
1. Accountability Cannot Be Delegated to Algorithms
Legal systems are built on responsibility. Every decision that affects someone’s rights must ultimately be attributable to a person or institution that can be held accountable. Artificial intelligence cannot fulfil that role. Algorithms cannot be sued, sanctioned, or morally blamed. When an AI system produces a flawed recommendation, especially one that leads to unfair treatment - the question immediately arises: who is responsible?
This dilemma became visible in the U.S. case State v. Loomis, where courts used the COMPAS risk-assessment algorithm during criminal sentencing. The defendant argued that the algorithm’s opacity violated his due-process rights because neither he nor the court could examine how the risk score had been calculated.
Although the court ultimately allowed the tool’s use, it acknowledged the serious concerns surrounding transparency and fairness. The case illustrates a central problem: when algorithmic tools influence legal outcomes, accountability becomes blurred. If a judge relies heavily on an AI-generated recommendation that later proves biased or flawed, responsibility cannot simply be shifted to software. Human oversight preserves this chain of responsibility and ensures that constitutional guarantees are not outsourced to mathematical models.
2. The Right to a Fair Process Can Be Undermined
Legal systems do not only aim to reach outcomes - they also guarantee fair procedures. Article 6 of the European Convention on Human Rights protects the right to a fair trial. This includes transparency, equality of arms, and reasoned decision-making. Individuals must be able to understand and challenge the decisions that affect them. Automated decision-making can threaten these guarantees when systems operate without meaningful explanation or review.
A striking illustration comes from the Dutch childcare benefits scandal. In an attempt to detect welfare fraud, Dutch tax authorities used automated risk-profiling systems to flag applicants considered “high risk.” Many of those flagged were automatically required to repay large sums of money, often without an individualised assessment or clear explanation. As a result, thousands of families were wrongly accused of fraud. Many suffered severe financial hardship before the errors were eventually acknowledged.
The problem was not simply the existence of an algorithm. It was the absence of meaningful human oversight. Decisions were applied mechanically, and those affected struggled to challenge outcomes they could neither understand nor contest. The lesson is clear: fairness requires more than a formal appeal process. It requires that decisions affecting rights remain understandable, reviewable, and ultimately controlled by accountable human authorities.
3. Non-Discrimination and Equality Are Vulnerable
AI systems learn from historical data. But historical data often reflects historical inequalities. If past decisions contain patterns of discrimination, whether related to race, gender, or socioeconomic status, an AI trained on that data may reproduce those patterns. Worse still, it can scale them across thousands of decisions.
For example, an algorithm recommending lower settlement offers for certain demographic groups may simply mirror biases present in historical claims data. The danger is rarely malicious intent. Instead, it is structural bias embedded within data and amplified through automation.
Human oversight introduces a layer of critical judgment that algorithms lack. Lawyers, judges, and regulators can question whether recommendations are fair, contextual, and consistent with legal principles. Without that oversight, discrimination risks becoming invisible, automated, and difficult to challenge.
4. Efficiency Cannot Replace Judgement
Artificial intelligence will undoubtedly continue to shape legal negotiation. Its ability to analyse vast datasets and identify patterns offers real benefits for legal practice. Used responsibly, AI can improve efficiency, reduce costs, and help professionals make better-informed decisions.
But efficiency alone cannot define justice. Legal systems are not purely analytical systems. They are normative institutions concerned with fairness, rights, and human dignity. Determining what is just often requires interpretation, empathy, and moral reasoning - qualities that cannot be reduced to statistical optimisation. For that reason, the future of AI in law should not be framed as a choice between human judgment and machine intelligence.
The principle remains simple: AI may negotiate or suggest. Humans must decide.
