For fifteen years, a bank could answer a simple question with a number. How many models do you run? The model inventory gave the figure, and next to each line sat an owner, a developer, a validator and the date of the last review.
Ask the same question about AI these days, and the answer is no longer a number.
None of this means the risk is new. Banks governed models long before anyone had heard of a large language model. The US supervisory guidance that shaped the discipline, SR 11-7, asked in 2011 for every model to be reviewed “at least annually”, with validation carried out by people independent of those who build and use it. Inventory, independent validation, periodic review: the method exists, and it is well understood.
It was also under strain before AI arrived. Deloitte’s 2025 survey of 87 banks across Europe, the Middle East and South Africa found that only 39% of banks’ models are validated every year, and the rest follows multi-year cycles. Late last year, banks and insurers told the Bank of England that their traditional approach to validation “wouldn’t be sustainable in its current form as generative AI and agentic systems proliferated”.
To see why it breaks, it helps to look first at the discipline at its best.
Trading desks have lived with automated decisions for a long time: algorithms, often bought from vendors, acting faster than any person can follow, where a single fault cascades. In August 2012, Knight Capital lost more than $460 million in 45 minutes after new code was deployed incorrectly and woke up a function in its order router that had never been meant to be used. Europe’s answer, in force since January 2018, is a regulation most institutions in Luxembourg have never had to apply: RTS 6, the technical standard on algorithmic trading under MiFID II.
Read today, it is the most complete assurance regime finance has written for decisions taken by machines.
It refuses to let a firm hide behind its vendor. A firm that buys trading software “shall remain fully responsible” for its obligations, and must have a full understanding of its algorithms “irrespective of whether those systems and algorithms were developed by the investment firm itself, purchased from a third party” or built with a partner.
It ties testing to change. Before deployment or any substantial update, the firm must show the algorithm “does not behave in an unintended manner”, including under stressed market conditions.
It requires a stop that works. A kill function that cancels orders immediately, and, since ESMA’s supervisory briefing of February 2026, the capacity to suspend or terminate trading “without dependence on the third-party provider’s consent or intervention”.
It assigns the three lines of defence in a single article. Every year, the risk function drafts a self-assessment and validation report, internal audit audits it, and senior management approves it.
Responsibility for what you buy, testing triggered by change, a stop you control, validation shared across the lines. And it works for a reason that is easy to miss: a trading desk is centralised. A limited number of algorithms, an inventory, a named owner, a known moment when something changes.
Large language models and agents remove exactly that condition. Model use has moved from centralised to decentralised, and the unit of validation has moved with it, from the model to the use.
In classic model risk management, one model serves one purpose. A quantitative team builds a credit model for credit decisions, a validator validates it, and the model, the use and the owner sit in the same place. A large language model inverts the relationship. One model, thousands of uses, each configured by a prompt, a document base or a connected tool, often by someone in the business who has never met a validator. The same model with two different prompts is two different systems. Validating the model tells you very little about either of them.
Agents add a further dimension. They act, call tools, hand work to other agents and multiply as they run. The use stops being an object you can list and becomes a population that changes by the hour. A survey of 2,930 cybersecurity decision-makers in 20 countries, commissioned by Palo Alto Networks in spring 2026, puts machine identities, AI agents included, at an average of 109 for every human identity.
Underneath every one of those uses, the provider updates the model on its own schedule, which is a change management problem I have written about separately. Here it means something more basic: a single update moves thousands of uses at once, and none of their owners initiated it (see https://axon.lu/the-change-nobody-initiated/).
Luxembourg’s own numbers show where this already stands. The CSSF and BCL’s 2025 review counted 402 AI use cases across 461 respondents. Sixty-one percent involved generative AI, half of those were developed outside the institution, and model performance was actively monitored for only 56% of all use cases. An inventory that counts models will keep looking under control, because the growth is happening in the uses, and the uses sit outside it.
The more uncomfortable lesson comes from the trading desks themselves. In 2025, more than seven years into a binding regime, the UK’s FCA reviewed algorithmic trading controls at ten principal trading firms. It found that firms using third-party algorithms “relied on the simulation testing of their vendor”, and that some lacked “a good technical understanding” of how those algorithms had been built. If a centralised desk under explicit rules still hands its assurance to the vendor, a decentralised AI estate with no equivalent rules will do so by default.
Supervisors have not settled what to do about it. On 17 April 2026, US regulators replaced SR 11-7 with new guidance, SR 26-2, which places generative and agentic AI explicitly outside its scope. Six days later, the UK’s Prudential Regulation Authority republished its own principles, SS1/23, which, for the banks it covers, apply to all models, built or bought, whatever the technology. The same heritage, two opposite answers, one week apart.
The EU AI Act, for its part, regulates the model, through obligations on the providers of general-purpose models, and a small number of high-risk uses. Most of what a financial institution actually does with a large language model sits between the two.
The principles from the trading desk still hold: responsibility for what you buy, testing triggered by change, a stop that works without the vendor, validation shared across the three lines. What does not transfer is the unit they were applied to. Where to apply them, when you can no longer validate every use, is the subject of my next article.
In the meantime, one question. Your model inventory tells you which models you validated. Does anything tell you how many ways they were used this morning?
Alexandre Castaing is Managing Director of Axon Advisory & Consulting, working with supervised financial entities in Luxembourg on digital resilience, AI governance and regulatory compliance. Articles are AI-assisted, human-directed, and source-verified.
#AIGovernance #ModelRisk #AIAct #RiskManagement #MiFIDII #Luxembourg