Industries
The Complexity of Gen-AI Test Data
.jpg)
By Wayne Addison
December 4, 2025
Most regulated B2C enterprises are about to discover that the hardest part of deploying Gen-AI solutions is the data they legally and practically need to build and test them.
This challenge has direct implications for compliance, operational risk, customer outcomes and time-to-value. My contentions in this article are:
The best "bang-for-your-buck" (or high value) Gen-AI solutions within regulated B2C enterprises, process Personally Identifiable Information (PII) within unstructured data.
Real-world unstructured PII customer data cannot be used to build and test these high value Gen-AI solutions.
These high value Gen-AI solutions cannot go live until they can be demonstrated to meet acceptable levels of performance.
Assuming my contentions are true (and I will make the case for each), therefore:
Regulated B2C enterprises must have the ability to generate, on demand, large amounts of synthetic-but-realistic unstructured PII data for testing these high value Gen-AI solutions.
And keep in mind, the format of all this synthetic, PII filled, unstructured data could be; single and multipage documents; emails with and without attachments; CSV text files; multiline database fields and even voice recordings. But I'll make the case for my contentions first.
Contention #1 High-Value Solutions
The processing of unstructured PII data is traditionally analysed and actioned by humans. A good example within a regulated B2C enterprise is customer communications which can include sales inquiries, complaints, general feedback, support issues etc. And these could be across multiple products of varying priority and contain varying amounts of customer PII data. This requires a high level of human effort to classify, confirm and respond to each aspect of even a single communication - which could be in a wide variety of formats, spanning multiple interactions with the same customer. All this time-consuming human effort is expensive; very expensive actually.
In the Australian financial services industry, call centre costs bounce around 7.5% of revenue. For a typical financial institution pulling ~$20B in revenue (~$8B in profit), that translates to spending ~$1.5B on dealing with customer communications.
Hence, there is a lot of potential Gen-AI "bang-for-your-buck" on the table when it comes to processing your PII ladened customer communications.
Contention #2 Using Real-World PII Data
My second contention is the most pivotal.
Back in the days of predictive models that use structured data, obtaining sufficient training/testing data was (relatively) easy. You could go to your warehouse schema, find the fields you need, then extract real-world data at scale. And if there wasn't enough data, it was fairly easy to generate additional records and tweak the content if you needed edge cases that didn't exist. And to make it even easier, the use-cases for these types of predictive models rarely used PII, so the need to anonymise was not that common. These use-cases were more like the old, "predict how much ice-cream an ice-cream shop needs to pre-order" kind of problems which was a function of the day of the week, the season, and the weather forecast. It had nothing to do with PII.
The "ice-cream" solutions are still valuable predictive models but if my first contention is valid, there is more gold in the unstructured PII data use cases - especially now that we have the tool to mine it (i.e. Gen-AI). So why not just use your real-world unstructured PII data and run a masking/anonymisation protocol over it (like you already do for your structured data)? What's the big deal? Why is getting test data for Gen-AI solutions a new problem?
The first is to remind ourselves why you can't (or shouldn't) use your real-world PII data in the first place (structured or unstructured) for the purpose of testing solutions. There are of course legal issues, including class actions by consumers against enterprises that use their PII data for purposes other than providing the product or service the customer is paying for. The regulators will also be looking at what you do with your customer data. Legalities aside, there is also reputational risk if you are even perceived as misusing customer data.
Now someone might say, “we don’t train Gen-AI foundation models, we don’t even fine-tune them. We only use prompting and RAG, so that makes it okay to use real-world customer PII data for build and test, right?” Maybe. But a good lawyer could argue that customer PII injected into a dynamic prompt or RAG, has the same effect as training/fine-tuning because it alters the accuracy metrics. I’ll leave that one for the lawyers; and it would only partially solve the problem anyway.
The other reasons are more pragmatic. Assuming you decide to use your real-world customer PII data and just anonymise it (or not). The first problem is how do you find all the customer data you need to test all the scenarios? Keep in mind this is the unstructured world of data so we are talking about documents, emails, CSV files, voice recordings - if it's in a database, it's a multiline text field that may or may not contain PII anywhere within the content. Have you ever tried searching for a digital trail of customer activity across a shared drive, a SharePoint library and someone's email account? Good luck. And that's just to find a single end-to-end test case that you then should anonymise for every mention of the customer's name, address, account number, ABN, etc. across the trail of artefacts and keep it all consistent. Then you must label the artefacts so you know what the Gen-AI solution should do with each input/output. Lots of human effort, error prone, not scalable or timely.
Assuming you do all that anonymisation work or even say "damn the torpedoes" let's just use our real-world customer PII data as is. You must still generate synthetic-but-realistic data to some degree - it is inescapable. The following Ven diagram illustrates my point.
Datasets 1 through 4 must have the unstructured data synthetically generated (datasets 2 and 3 being PII). You have no choice. You either don't have real-world examples or you think you do but you can't find them.
Dataset 5 can be used as is (as there is no PII) - but it still needs to be labelled for testing the accuracy of the Gen-AI solution.
Dataset 6 must be anonymised and labelled, BUT if you have a solution for datasets 1 through 4, why not just use that solution for this dataset too?
Note: you might say "why do we need to consider datasets 1 and 4 if they don't have PII?" In the case of customer communications, you need to test the scenario where the communication should have identifying PII, but doesn't, and how did your Gen-AI solution react to expecting PII and not finding it. So, it's still a PII use-case but you need to test it with some non-PII test cases.
Contention #3 Measuring Performance
Measuring the performance of a Gen-AI solution (more or less) boils down to the degree of false positives, false negatives, true positives and true negatives that are within the risk profile of the business function being "Gen-AI-ified". If there is still a human-in-the-loop, then the acceptable thresholds of these metrics can be lower (because the human will spot any Gen-AI "crazy-talk") but a Gen-AI solution that interacts directly with a customer would need to meet high thresholds of performance before being let loose.
For example: In the Australian financial services world, there are specific regulatory obligations around how customer complaints are treated (they can't just be flagged as "feedback" and ignored). There are also obligations if the customer shows signs of financial hardship. And there are big fines (issued by the Regulators) to enterprises that breach these obligations. Let alone the bad press.
A McKinsey survey (link reference below) gives us some interesting insights into what most organisations are concerned with when implementing Gen-AI. And what is the top concern? Inaccuracy (followed by Explain-ability then Personal Privacy).
So how do you ensure a Gen-AI solution meets the acceptable thresholds of accuracy, is explainable and doesn’t compromise individual privacy? You test it with data and, for these high value Gen-AI solutions, lots of unstructured PII data that not only tests the "run-of-the-mill" customer communications but all the possible edge cases too. And don’t forget you have to continuously monitor your Gen-AI solution after it goes live in case the accuracy starts to drift.
So, what to do?
Assuming I have made some in-roads in validating my contentions - at least enough to make you consider that this issue exists inside your own organisation. We’re left with a practical question: where do you get all the labelled synthetic-but-realistic unstructured PII data needed to build and test your best “bang-for-your-buck” Gen-AI solutions?
In the coming weeks I’ll break down each of these challenges in short, practical posts; and will share an approach I am developing to solve them.
More Articles
Testing AI Agents: The Reliability Problem That Cannot Be Deferred
A December 2025 paper on Measuring Agents in Production surveyed GenAI projects across industries and reached two conclusions that should concern any organisation deploying AI in high-accountability...
Explore related content across our Insights section
