Industries
Testing GenAI Agents
By Wayne Addison
January 19, 2026
A recent (Dec-2025) Paper on "Measuring Agents in Production" surveyed various GenAI projects and detailed the challenges in determining the reliability of AI Agents used for "highly customized client integrations". In this video I give my take on these challenges.
A couple of key takeaways of the Paper:
•Determining the reliability of AI Agents remains unsolved
•Generating gold-standard unstructured test data (to determine reliability) is "nearly infeasible"
My take on the key takeaways
•Sooner or later Agent reliability must be solved
•Yes, generating gold-standard unstructured test data (to determine reliability) is "nearly infeasible" BUT it is possible
My second contention is the most pivotal.
Back in the days of predictive models that use structured data, obtaining sufficient training/testing data was (relatively) easy. You could go to your warehouse schema, find the fields you need, then extract real-world data at scale. And if there wasn't enough data, it was fairly easy to generate additional records and tweak the content if you needed edge cases that didn't exist. And to make it even easier, the use-cases for these types of predictive models rarely used PII, so the need to anonymise was not that common. These use-cases were more like the old, "predict how much ice-cream an ice-cream shop needs to pre-order" kind of problems which was a function of the day of the week, the season, and the weather forecast. It had nothing to do with PII.
The "ice-cream" solutions are still valuable predictive models but if my first contention is valid, there is more gold in the unstructured PII data use cases - especially now that we have the tool to mine it (i.e. Gen-AI). So why not just use your real-world unstructured PII data and run a masking/anonymisation protocol over it (like you already do for your structured data)? What's the big deal? Why is getting test data for Gen-AI solutions a new problem?
The first is to remind ourselves why you can't (or shouldn't) use your real-world PII data in the first place (structured or unstructured) for the purpose of testing solutions. There are of course legal issues, including class actions by consumers against enterprises that use their PII data for purposes other than providing the product or service the customer is paying for. The regulators will also be looking at what you do with your customer data. Legalities aside, there is also reputational risk if you are even perceived as misusing customer data.
Now someone might say, “we don’t train Gen-AI foundation models, we don’t even fine-tune them. We only use prompting and RAG, so that makes it okay to use real-world customer PII data for build and test, right?” Maybe. But a good lawyer could argue that customer PII injected into a dynamic prompt or RAG, has the same effect as training/fine-tuning because it alters the accuracy metrics. I’ll leave that one for the lawyers; and it would only partially solve the problem anyway.
The other reasons are more pragmatic. Assuming you decide to use your real-world customer PII data and just anonymise it (or not). The first problem is how do you find all the customer data you need to test all the scenarios? Keep in mind this is the unstructured world of data so we are talking about documents, emails, CSV files, voice recordings - if it's in a database, it's a multiline text field that may or may not contain PII anywhere within the content. Have you ever tried searching for a digital trail of customer activity across a shared drive, a SharePoint library and someone's email account? Good luck. And that's just to find a single end-to-end test case that you then should anonymise for every mention of the customer's name, address, account number, ABN, etc. across the trail of artefacts and keep it all consistent. Then you must label the artefacts so you know what the Gen-AI solution should do with each input/output. Lots of human effort, error prone, not scalable or timely.
Assuming you do all that anonymisation work or even say "damn the torpedoes" let's just use our real-world customer PII data as is. You must still generate synthetic-but-realistic data to some degree - it is inescapable. The following Ven diagram illustrates my point.
Note: you might say "why do we need to consider datasets 1 and 4 if they don't have PII?" In the case of customer communications, you need to test the scenario where the communication should have identifying PII, but doesn't, and how did your Gen-AI solution react to expecting PII and not finding it. So, it's still a PII use-case but you need to test it with some non-PII test cases.
A McKinsey survey (link reference below) gives us some interesting insights into what most organisations are concerned with when implementing Gen-AI. And what is the top concern? Inaccuracy (followed by Explain-ability then Personal Privacy).
More Articles
.jpg)
The Complexity of Gen-AI Test Data
Most regulated B2C enterprises are about to discover that the hardest part of deploying Gen-AI solutions is the data they legally and practically need to build and test them.
Explore related content across our Insights section
