AI Assurance Explained: Trust, Safety, and Operational Impact

The UK-USA Technology Prosperity Deal sees overseas organisations pledging £31 billion of investment into UK AI infrastructure. As AI investment continues to accelerate, the challenge of assuring new technology is safe to use is now more crucial than ever.

In September, the UK Government announced the Trusted Third-Party AI Assurance Roadmap. Its publication signals a broader recognition of the importance of AI Assurance as an industry, further evidenced by its forecasted growth to $276 billion in 2030 from a humble beginning of $1.62 billion in 2023. In reality, though, using AI systems in the most challenging use cases is inconceivable unless they have been assured to be safe. 

Refreshingly, however, the roadmap stays relatively high-level and is scant on specifics. This accurately reflects the nascent and highly dynamic nature of the technology and market; committing to specifics would be foolhardy. Instead, we should take the time to delve into what AI assurance means, what it entails, and how organisations can implement it successfully.

What is AI Assurance?

AI assurance is the practice of ensuring that AI systems are safe, reliable, and fit for purpose. Fundamentally, its role is to ensure that users, both internal and external, are not exposed to unacceptable risk from AI systems. Assurance varies from fine-toothed comb analysis of system components to more general “sense checking” of overall design.

The Need for Assurance in AI

Cautionary tales around misplaced or poorly scoped use of AI are numerous. From sexist credit cards to LLM-induced psychosis, as AI capability advances, the trust and reliance we place on it grow, and so too do the risks of using it incorrectly or falling foul of its mistakes. This is where AI assurance comes into play. It provides a sense check that an AI-enabled system is not ill-conceived, identifies and understands the risks of malfunction or misuse, and weighs up those risks, their likelihood and impact, against the broader value of the proposed system. 

Assessing a system for “safe use”, though, is no easy task. As models and use cases become more complex, greater expertise is also needed to truly understand a system. This, in turn, requires domain knowledge, legal input, and the involvement of advanced model architects and software engineers. The temptation to use AI itself and software tools to solve the issue is clear, but at this juncture, technology might not actually be the answer.

Assurance in Reality

In our experience deploying AI across multiple sectors, we have identified several repeated truths:

AI technology and use cases are moving fast.
Some years ago, platforms offering feature importances, SHAP values, and graphical explanations were state-of-the-art in explainable, fair, and assured AI. Today, agentic systems can inspect code for flaws and logical inconsistencies with impressive results, albeit with limited formal performance guarantees, offering a categorically different type of assurance. Investing too much, too early, can result in wasted effort.

Deployment is fragmented.
There is no common model management or assurance stack, which means integration costs remain high. Most model developers still build and operate bespoke deployment infrastructure or configure open-source solutions in-house, which has significant integration implications for any software developed to integrate with models at the point of deployment. Navigating this rapidly evolving, somewhat chaotic landscape is a losing battle.

The economics are challenging.
If a model must deliver 10x ROI, and assurance must deliver 10x ROI on top of that, the implied value of assurance may be just 1% of total system uplift. That creates a challenging business case, especially in early-stage deployments.

Assurance can be aspirational.
Many organisations know they need some form of assurance for AI systems, but can’t articulate what good looks like in practice. Oftentimes, they don’t yet have a system they want to deploy that actually needs assuring, and so initial interest often fades when clear requirements or alignments to organisational KPIs don’t emerge. In other cases, interest pivots somewhat randomly, searching for concrete requirements and following multiple “red herrings”, resulting in wasted effort and investment. 

Ownership is unclear.
In all cases where we deploy models operationally, assurance is at the forefront of our minds as model developers, and as such, assurance and monitoring are integrated into our systems. On several occasions, however, we have witnessed organisations request these features but lack clarity on who should own them internally. This delays adoption and limits effectiveness since capability remains relatively superficial.

A rapidly evolving landscape that lacks clarity

In summary, while there is growing interest around assurance, much of this is undirected or sporadic and subject to change at will. Justifying meaningful investment in building repeatable and scalable production-ready software for unpredictable and disparate problems is hard. In many cases, demand for assurance is speculative or anticipatory at best, and is not yet the top priority in unblocking AI impact. Much like explainability has shifted from a technology buzzword to a meaningful requirement, we believe that AI assurance will, in time, follow a similar path.  

What is the future of AI assurance?

Three factors will play a significant role in shaping and solidifying the AI assurance landscape:

1. Clearer regulation 

As laws mature and precedents are established, organisations will have more confidence in what’s required. AI legislation is relatively immature and vague, and lacks example precedents to give headline statements concrete, actionable meaning. Organisations that overinvest in exceeding or anticipating regulatory requirements risk over-engineering or misplaced investment. 

2. Cautionary tales 

Real-world failures and legal disputes will make the issue urgent and specific, forcing it into boardrooms and enabling ‘learning by example’. This will solidify assurance as playing a vital role in preventing reputational damage, financial penalties, and tangible dangers of failing to consider and mitigate AI risks adequately.

3. Operational maturity  

As more AI systems move into critical business workflows, assurance will shift from “nice to have” to “non-negotiable”. Equally, more examples of deployment within individual firms and the wider market will drive the derivation and solidification of best practices and productised approaches with smoother adoption curves. Projects like the ARIA Safeguarded AI programme will also help to develop use cases and methodologies for demonstrating how AI can be deployed safely in sectors like critical national infrastructure.

AI Assurance: An Enabler, Not a Blocker

Assurance is not (yet) the main risk or blocker to achieving operational AI. The challenge currently lies mostly in effecting organisational change, driving understanding, trust, and adoption of new systems. This must be supported by continued focus on maximising operational impact rather than exploration of AI technologies purely for the sake of novelty or innovation. 

Furthermore, fully deployed AI use cases and the regulatory landscape are still both relatively immature. This means that for now, software-driven assurance is best integrated into applications themselves where required, and built by suitably qualified model builders who already understand the data, architecture, and limitations. Third-party assurance may be beneficial at a high level, which will eventually become more integrated as problems and the landscape become more standardised.

Today, organisations must start by defining clear policies and validation processes. This includes appointing appropriate experts, like those mentioned in the UK’s AI Assurance Roadmap, to assess the assurance capability delivered by AI providers. At least with suitably qualified people in the room, organisations are more likely to identify issues as AI systems mature.

Furthermore, application builders must be aware of and use the best tools available rather than waiting for technology updates and assuming these will retroactively solve issues baked in by poor design. Many algorithmic techniques and best practices can already provide significant assurances from within model development, which will help us future-proof AI systems: 

• Differentially private mechanisms guaranteeing individual privacy are widely available in standard Python libraries. 
• Probabilistic programming paradigms also provide guarantees of correct uncertainty propagation. 
• Working with established and well-curated platforms such as AWS can provide security and traceability assurances. 
• Single and multi-agent generative systems can be guided towards explainability and consistency with techniques such as constitutional AI and chain-of-thought reasoning. 
• Infrastructure for monitoring AI performance deterioration caused by concept drift can help flag potential development of unintended bias and unfair decision-making. 

The Pursuit of Value

For now, the priority for any organisation looking to demonstrate safe and trustworthy AI should be meaningful impact over performative compliance. Demonstrating real operational utility will help to surface the real requirements for assurance and governance. Although AI assurance is in its infancy as a discipline, sufficient expertise does exist to support meaningful use cases in safe deployments that do not incur unacceptable risk. Waiting for the landscape to settle and for top-down clarity of what good assurance looks like is misguided. Instead, we must learn by doing and maintain focus on real problems today to better understand those we will solve tomorrow.