Why an unexplainable model is becoming a regulatory liability in South African finance.

In April, the South African government published a National Artificial Intelligence Policy. It had cleared Cabinet. It had passed a socio-economic impact assessment. It had secured concurrence across every director-general cluster in the relevant departments. By the standards any serious institution applies to a document of national consequence, it had been reviewed.

Three weeks later, the Minister of Communications and Digital Technologies withdrew it. A journalist had checked the reference list and found that several of the cited academic sources did not exist. The journals were real. The articles were not. The authors credited with foundational work had never written it. The most plausible explanation, the Minister conceded, was that a generative model had produced the citations and nobody had verified a single one.

Read that as a governance story rather than a technology story, because that is what it is. A document passed every checkpoint the state possesses and was still indefensible the moment one outsider asked the simplest possible question its authors could not answer: where did this come from? The Minister drew the right lesson himself, calling the episode proof that “vigilant human oversight over the use of artificial intelligence is critical.” Every control fired. None of them caught it. The output was confident, well-formatted, and impossible to stand behind.

This is the position a growing number of South African boards are quietly in. Most of them do not yet know it.

The exposure you have not booked

A financial model that no one in your organisation can explain is a liability you already own. It sits on the balance sheet the way a contingent claim does — unbooked, unquantified, and entirely yours. Audit does not transfer it. Validation does not extinguish it. A clean sign-off changes the paperwork, not the exposure.

By “explain,” I do not mean “document.” Documentation is a description of what the model is meant to do. Explanation is the ability of a competent person to state, in plain language, why a specific input produced a specific output — and to defend that reasoning to someone who disagrees with it and has standing to act. A model can be fully documented, exhaustively tested, and signed off, and still fail that test completely. The two are not the same thing, and the gap between them is precisely where the liability lives.

You have lived through IFRS 17. You know exactly what it costs to defend a number whose derivation you cannot see to the bottom of. Now apply that discomfort to the machine-learning models your institution is already running in credit scoring, insurance pricing and fraud detection — and notice that in most cases the people who could once trace a number by hand are no longer in the room when it is produced.

“It’s just the algorithm“

The textbook case is not hypothetical, and it happened to one of the most sophisticated institutions in the world. When Goldman Sachs issued the Apple Card, a prominent customer publicly complained that the card had granted him a credit limit twenty times higher than his wife’s, despite her stronger credit profile and their joint finances. When she contested the decision, the bank’s representatives could not tell her why the model had ruled as it had. Their explanation, repeated and now infamous, was that it was “just the algorithm.” Six representatives, across two of the largest technology and banking franchises on earth, had no visibility into the decision.

The New York regulator opened an investigation and asked the bank, in essence, to explain its own model. That investigation eventually cleared the bank of unlawful discrimination. That was never the point, and the finding arrived far too late to matter. The damage was done in the weeks when a regulated lender, facing a customer and a supervisor, could not account for a decision it had made. The liability was not bias. The liability was silence.

A model you cannot explain is one you cannot defend — to a regulator, to a board, or to a customer contesting a “no.” When any of those three asks how the decision was reached, “the model is proprietary,” “the model is complex,” and “the model was validated” are not defences. “We don’t fully know” is not a defence either. It is a confession, and increasingly it is one the law will hear.

The law is already here

It is tempting to treat all of this as a future problem awaiting a future regulator. It is not. The most relevant instrument has been in force for years.

Section 71 of the Protection of Personal Information Act already restricts decisions made solely by automated processing where they have legal consequences for a person or affect them substantially. The Act names the territory explicitly — it covers profiling of a person’s creditworthiness, conduct and reliability, which is to say it covers the core of credit scoring and a great deal of insurance pricing. Where an institution relies on the exceptions that permit such decisions, the Act imposes a condition that should stop any board cold: the responsible party must give the affected person “sufficient information about the underlying logic” of the processing, so that they can make representations against it.

Underlying logic. That phrase is in the statute. A model your own team cannot explain cannot satisfy it, by definition. You cannot disclose logic you do not possess.

Around that statutory floor, the scaffolding is going up quickly. King V took effect for financial years beginning on or after 1 January 2026, and its entire regime is “apply and explain” — it strengthens the governing body’s accountability for information and technology, and it states that sound governance is measured not by how many boxes are ticked but by whether the intended outcome was actually achieved. A model nobody can explain fails the second half of “apply and explain” before the conversation even starts. The prudential standards, meanwhile, have always placed model risk squarely with the board.

And the National AI Policy that opened this piece will return. It was withdrawn, not abandoned; a redraft is coming, and the direction of travel is not in doubt. Institutions are already being told to inventory their high-impact systems and assess model explainability against POPIA, the prudential standards and King V. The only question is whether you do that mapping on your own timetable, or on a supervisor’s.

Who’s name is on it

Here is the part that does not delegate. The liability does not rest with the modelling team, the vendor, or the auditor. King V puts accountability for governance outcomes on the governing body. POPIA puts it on the responsible party. Neither of those is the analyst who built the model or the consultancy that reviewed it. When the customer contests the “no,” when the regulator asks how the decision was reached, the institution answers — and the institution is the board.

That is the uncomfortable arithmetic. The work is delegated. The accountability is not. A board can outsource the building of a model and the auditing of a model. It cannot outsource the obligation to be able to explain one.

So the practical task is narrow and answerable. Put these questions to your teams, and treat hesitation as the finding:

• For each model that decides who we lend to, what we charge, or whom we flag — can someone state, in plain language, why a given customer received the outcome they did?
• If that person left tomorrow, who else could do it? If the answer is “the vendor,” we do not own the explanation.
• Where the model makes a decision largely on its own, have we satisfied Section 71 — including the duty to disclose its underlying logic to the person affected?
• When this model declines a customer or prices them up, what exactly do we tell them, and would it survive being read back to us by a regulator?
• Has anyone in this organisation tested whether “passed validation” and “can be defended” are the same answer for this model — or have we been assuming they are?

None of this requires you to become a modeller. It requires you to stop accepting a sign-off as a substitute for an explanation, and to notice that the two have quietly come apart.

The South African AI policy failed because a confident output went unchallenged until an outsider asked where it came from. Your models are confident outputs. The outsiders — a regulator, a board member, a declined customer with a lawyer — are already forming the question.

So the only question that matters this quarter is the one your audit committee has probably not yet written onto the agenda: of all the models currently deciding who we lend to, what we charge them, and whom we treat as a risk — which ones can we actually explain, and who in this building is prepared to put their name to the answer?