Ask how a company governs AI and the answer is often the same across fields — a human makes the final decision, writes Adnan Masood, professor and AI executive. But oversight alone may be just as meaningless as no oversight at all.
In 2023, University of Cologne researchers gave a group of radiologists 50 mammograms each to review with an AI assistant. At first, the system was accurate and readers learned to trust it. When it later began giving incorrect answers, accuracy among the least experienced professionals fell from about 80% to just under 20%. Even among senior radiologists, each with more than a decade of experience, accuracy fell from 82% to about 46% percent.
This experiment offers one of the clearest portraits of the limits to human-in-the-loop-based approaches to AI governance. After many governance reviews, I have found that automation bias — the tendency to accept an automated suggestion without verification — is the field’s single most repeated control assertion and its least tested.
A person being present in the process does not prove that a person was in control of it.
Why approval becomes automatic
Early accuracy from models builds trust, which reduces the instinct to verify output and so eventually underlying skills erode, a phenomenon that is reinforced by research. Compounding matters, reviewers are measured on throughput, meaning the amount of work they complete, which can make verification register as underperformance.
Furthermore, acceptance of an AI decision often requires just one click while disagreement must be justified through a form that can require a supervisor’s signature. The blame is also lopsided. Agree with the machine and any error is shared; override it and the error is yours alone.
The reality is that people catch what models miss. Australia’s Robodebt program automated welfare debt collection and stripped away the human checks that had previously caught errors. A commission condemned the scheme, courts found the debts unlawful, and in June 2026, the court approved a $549 million settlement on top of an earlier $112 million award.
Courts are catching up to limitations of automation with weak human oversight. In 2023, a European court held that when lenders predictably follow a credit score, the score itself is the automated decision under GDPR Article 22.
A 99% agreement rate may reflect a rubber stamp or a strong model being properly confirmed, but the rate alone cannot tell us which of those possibilities is true. Sometimes review is the weaker link and should be reduced but not always. Reviewer value is measurable, yet almost nobody measures it.
AI Is a Stickler for the Rules, but Rules Don’t See Everything
Human inconsistencies and man-made exceptions are often overlooked in AI deployments
Read moreDetailsAudit the human layer
Since 2011, banks have operated under SR 11-7, the Federal Reserve’s model risk guidance, which requires critical review by qualified people with the authority and incentive to change an outcome. The same standard should apply to the review layer itself.
In audit terms, most AI oversight programs document design effectiveness. The control exists on paper without testing operating effectiveness, whether or not it works in practice. No auditor would accept that gap for a reconciliation and none should accept it for the control standing between an algorithm and a customer.
The test program should start with a three-way comparison of reviewers alone, the model alone and both together on matched cases. The combined approach should be maintained only where it performs best.
One way to test this is to seed known wrong outputs into live queues, as security teams do with phishing simulations, and make the catch rate the primary oversight metric. Track override precision (how often reviewer disagreement proves correct) and override recall (the share of true model errors that reviewers catch). Set minimum dwell times below which real verification is impossible and log whether reviewers open the evidence. Make agreement and disagreement require the same effort, and where possible, capture the reviewer’s judgment before revealing the model’s answer. Finally, divide responsibility across the three lines of defense: the business runs the review, risk instruments it and internal audit tests it with seeded cases.
Where per-case review cannot be genuine, at machine speed, high volume or after an irreversible action, retire the ceremony and move the safety budget upstream to pre-deployment evaluation, continuous monitoring and appeals that function. Regulators in Europe are converging on this type of test.
Risk officers who test the human layer now will spend the next decade defending their evidence. Those who wait will spend it defending their assumptions.
Adnan Masood, PhD, is chief AI architect at UST, an IT services and consulting firm. He is also an adjunct professor and visiting scholar on AI and previously was a regional director at Microsoft. 
