Automation makes a model faster, cheaper, and more consistent. It does nothing at all to make the model right. Speed and trustworthiness are different properties — and we have started to mistake the first for the second. Independent validation is the discipline that keeps them apart.

Automation’s promise is real, and it should be said plainly before anything is said against it. A model that runs itself is faster than a committee, cheaper than a department, more consistent than a tired human at four in the afternoon, and it does not take leave. Across banking, insurance, retail credit and half a dozen other industries, decisions that once moved at the speed of a meeting now move at the speed of an API call. Efficiency, on its own terms, is a genuine good. Nobody serious wants to go back.

But efficiency and trustworthiness are not the same property, and the age of automation has encouraged us to confuse them. The speed with which a model produces an answer tells you precisely nothing about whether the answer deserves to be believed. Efficiency is a statement about the cost of producing an output. Trust is a statement about whether that output should be acted upon. Automation optimises the first with astonishing success and is entirely silent on the second. And a model that is wrong efficiently is not less wrong. It is simply wrong faster, more cheaply, and at greater scale.

This is the uncomfortable centre of the matter. Automation does not make error rarer. It makes error cheaper to commit and harder to notice. A mistaken judgement made by a person happens once, in a room, and can be argued with. The same mistaken judgement encoded in an automated model happens ten thousand times a day, silently, each instance wearing the calm authority of a number.

Which points to the deeper problem: automation launders judgement. When a human declines your loan, you can ask why, and the answer — however unwelcome — is a chain of reasons you can interrogate, contest, and appeal. When a model declines your loan, the judgement has not disappeared. It has been baked into the choice of features, the setting of thresholds, the composition of the training data, and the weights the model settled on, and then wrapped in a output that looks like an observation rather than a decision. “The model said so” closes the conversation that “I decided so” would have opened. The judgement is still there. It has merely been compressed, scaled, and hidden — and a score that looks objective is, for exactly that reason, the most dangerous kind of judgement, because its objectivity is a costume.

The standard reassurance is that there is a human in the loop. Someone reviews the output; someone can override. But the human in the loop is subject to a well-documented failure of her own, and it has a name: automation bias. It runs in two directions. There is overreliance — the error of omission, in which the reviewer simply misses the machine’s mistake because she has stopped genuinely looking. And there is overcompliance — the error of commission, in which she defers to the system even when the evidence in front of her contradicts it, because the machine “seems smart” and the deadline is real. Left to run, this produces what the literature calls being out of the loop: the reviewer’s own judgement, unused, quietly atrophies, until the safeguard that was meant to catch the model’s errors is no longer capable of catching anything. A human who rubber-stamps is not a control. She is a formality with a pulse.

So the question the title poses — on what basis, in an automated system, do we actually trust — cannot be answered by speed, and cannot be answered by a nominal human reviewer. It has to be answered from outside the model, by someone with the independence, the competence, and the standing to challenge it. That someone is the discipline of independent model validation, and it exists precisely because trust in a model is not something the model can generate for itself.

Model validation, at its core, is the independent assessment of whether a model is conceptually sound, whether it performs as intended in production, and whether it stays reliable as the world around it moves. The canonical framing — the United States Federal Reserve’s model risk guidance, known in the trade as SR 11-7, now echoed by the equivalent standards in the United Kingdom, Europe and here at home — organises the work across three pillars: evaluating conceptual soundness, monitoring ongoing performance, and analysing outcomes against what actually happened. But the beating heart of the discipline is a single idea it calls effective challenge: critical, objective analysis of a model, by informed and technically competent people who have the authority, the access, and the freedom from organisational pressure to genuinely push back — and, when necessary, to say no.

The word that carries the weight is independent, and it is a structural requirement, not a matter of good intentions. A validator who reports to the model’s developer, whose budget and timeline are set by the business line that wants the model live, is independent only on an organisation chart, and examiners have long since learned to see through the diagram. The builder cannot mark his own homework. And the challenge has to be real: a validation that agrees with everything has not validated anything. Its whole value lies in its capacity to disagree, to document push-back, to record the limitation the builder would rather leave unmentioned. Validation is not a compliance ritual performed at year-end. It is institutionalised skepticism, given standing.

Here is the part that matters most for the age of automation: the more automated and opaque a model becomes, the more validation it needs, not less. SR 11-7 was written before machine learning swept through financial services, but its principles apply with full force, and regulators have confirmed as much — the same rigour is expected of an AI system as of a regression, regardless of its complexity or whether it came from a vendor. Yet modern machine-learning models break several of the assumptions that traditional validation quietly relied upon. They often have no interpretable functional form to inspect. Their behaviour changes as they retrain on new data, so a model validated in January may be a different model by June. Their complexity makes “is this conceptually sound” a genuinely harder question than reviewing an equation ever was. And they can encode bias at scale, learning a discriminatory pattern from history and applying it, tirelessly, to the future. Treating a proprietary model as an unexaminable black box because the vendor will not disclose its workings is not a defence; it is the failure itself. You cannot validate what you refuse to understand — and the effort required to validate a model rises exactly as the model’s opacity rises, which is precisely why, under efficiency pressure, it is the step most tempting to skip.

Naturally, the response has been to automate validation too. And some of it genuinely can be: monitoring, backtesting, benchmarking and drift detection scale beautifully, and should. But the core cannot be fully automated, because effective challenge is not a measurement. It is a judgement about whether a model is appropriate for its purpose, made by someone with the independence to reach an inconvenient conclusion and the authority to act on it. You can automate the reading of the instruments. You cannot automate the standing to say no, the incentive to want to, or the accountability for having said yes. An automated validator wired into the same pressure to ship is not a safeguard. It is faster rubber-stamping.

The stakes are not hypothetical, and the instructive failures are never failures of arithmetic. When Zillow shuttered its home-buying business, it was not because anyone had added up wrong; it was because a proprietary, non-transparent pricing model was trusted to do something a volatile market would not let it do, and the trust outran the evidence — costing hundreds of millions of dollars and some two thousand jobs. When a JPMorgan trading desk changed a risk model mid-stream in 2012 in a way that roughly halved its reported risk, the change went through without adequate independent review, and the challenge function that should have caught it simply was not brought to bear. And when the Dutch government’s automated system for policing childcare benefits produced systematically discriminatory outcomes, officials deferred to it — and the harm, automated and scaled, fell hardest on families least able to contest a machine. In each case the model was efficient. In each case what was missing was the independent challenge that would have asked whether it was right.

That last example carries the point that matters most to us. Automated models increasingly stand at the gate of things people genuinely need — credit, cover, a price, a benefit, an approval. When those models are wrong and unchallenged, the harm is silent, it is scaled, and it lands with particular weight on the people least equipped to argue back: the thin-file borrower, the informal trader, the household on the margin, the applicant whose life does not resemble the data the model was trained on. An unvalidated model that quietly rations a whole segment out of the formal economy does so with the serene, unappealable authority of a number, and the person on the wrong side of it never learns why. Independent validation is, in a real and unsentimental sense, the mechanism by which that person has any recourse at all. It is the conscience the automated system does not possess.

This is why the institutional scaffolding around it is not bureaucratic ornament. In South Africa, the Prudential Authority’s validation expectations across those same three pillars, the peer-review requirements built into the Solvency Assessment and Management regime for insurers, and the 2025 joint work by the Reserve Bank’s Prudential Authority and the Financial Sector Conduct Authority on artificial intelligence in finance are all, at bottom, one insistence: that speed must not be mistaken for soundness, and that someone independent must remain answerable for the machine’s verdicts. The regulators have been candid that a capability gap exists — that many validation functions have not yet built the skills to genuinely challenge a modern machine-learning model. That gap is not a reason to relax the standard. It is exactly the space that rigorous, independent validation is meant to fill.

Efficiency asks a good and necessary question: can we do this faster, cheaper, at greater scale? Trust asks an older and harder one: has this earned the right to be believed and acted upon? Automation answers the first magnificently and cannot, by itself, answer the second at all. Trust in a model is never produced by the model. It is conferred from outside — by someone independent, competent, and willing to withhold it — and it has to be re-earned as the model and the world drift quietly apart.

That is what independent validation is. Not a tax on efficiency, and not a brake fitted to insult the machine. It is the discipline that slows the machine down just long enough to ask whether it is telling the truth. It does not flatter the model. It illuminates what the model would prefer to leave unexamined.