Thought Leadership

Most organisations approaching GenAI deployment assume their biggest challenge is the model. It is not. The bottleneck is the test data, and the three most common approaches to solving it each carry a fundamental flaw that only becomes visible after deployment.

A GenAI agent that performs well in controlled evaluation and fails in production is not a model failure. It is a data failure. The gap between the structured, predictable datasets used in lab environments and the messy, ambiguous reality of real customer communication is where most GenAI initiatives quietly break down.
 
This piece sets out the architecture of that gap, why the standard approaches to closing it are insufficient, and the question every organisation should be asking before their GenAI system goes live in a high-accountability environment.

The highest-value GenAI solutions in regulated B2C enterprises are the ones that process unstructured PII data - customer communications, complaints, case notes, voice recordings. That is precisely where the AI testing problem is most acute.

Real-world unstructured customer data cannot legally or practically be used to build and test these solutions. Masking and anonymisation address only part of the problem. The data you need either does not exist at sufficient scale, cannot be found across fragmented systems, or cannot be used at all. This piece makes the case — through three specific contentions — for why regulated enterprises must be able to generate synthetic-but-realistic unstructured PII data on demand, and what is at stake if they cannot. 

A December 2025 paper on Measuring Agents in Production surveyed GenAI projects across industries and reached two conclusions that should concern any organisation deploying AI in high-accountability environments: determining the reliability of AI agents remains unsolved, and generating the gold-standard unstructured test data needed to measure that reliability is described as "nearly infeasible."
 
This video sets out a direct response to both findings. Agent reliability must be solved — the operational and regulatory exposure of deploying agents whose reliability cannot be demonstrated is not a sustainable position. And while generating gold-standard unstructured test data is genuinely difficult, it is not impossible. This piece explains what a rigorous approach looks like, and why the distinction between "nearly infeasible" and "not possible" matters significantly for organisations under compliance pressure.