Industries
Home / solutions / unstructured synthetic data Services / why your test data is holding your genai back
RESOURCES
Why Your Test Data is Holding Your GenAI Back
Every organisation building a GenAI system faces the same question: where do you get the test data (to test that your GenAI solution is accurate and working as expected)? The most common answers: redaction, masking, anonymisation, and a range of DIY workarounds, all fail in the same fundamental way: they leave organisations believing their testing is adequate when it is not. Here is why each approach fails, and what a defensible alternative looks like.
Industry agnostic. Relevant to any organisation evaluating GenAI systems with unstructured data.
The Test and Evaluation Data Problem
High-value GenAI use cases typically process unstructured data including customer complaints, case records, service transcripts, incident reports. This is the data that your AI solution processes, how real people communicate: the tone, the escalation signals, the informal phrasing, the edge cases.
Production data is also, almost always, data that cannot be used safely for AI testing. It contains real customer information, personal data, sensitive records. Using it directly introduces privacy exposure, consent obligations, and regulatory risk.
So, organisations look for alternatives. The six most common approaches are set out below. Each appears workable. Each fails and some introduce new risks that are harder to detect than the original problem.
80%+
of enterprise will have deployed Gen AI in production by 2026. Up from less than 5% in 2023
Source: Gartner, 2025
Data Infusion
Every one of them needed evaluation data their team could defend.
74%+
of companies are still not generating tangible value from AI
Source: BCG, 2024
Data Infusion
BCG identified the reason as data governance and accessibility; not the model. That is the problem unstructured synthetic test data solves.
APPROACHES THAT START WITH REAL DATA
These four approaches all begin with the same raw material; real customer or operational data; and attempt to make it safe enough to use. The privacy problem is managed to varying degrees. The evaluation and testing data problem is not.
Privacy risk remaining:
None (data removed)
Redaction
Starts with real data
The irreversible removal of sensitive information from documents. Names, identifiers, and sensitive elements are permanently deleted.
WHY IT FAILS
Redaction protects privacy absolutely by eliminating the data entirely. What it also eliminates is any analytical or testing value. A redacted document tells an AI system very little about how real people actually communicate. But the deeper failure is behavioural: an AI system evaluated against redacted data will behave differently when it encounters real unredacted data in production. The patterns it has learned to recognise are patterns from a document with structural gaps. In production, those gaps are filled with language, context, and signals the system was never trained to handle. You cannot validate production behaviour against testing data that does not reflect production conditions.
Privacy risk remaining:
None (data removed)
Masking
Starts with real data
Sensitive data is obscured including names and identifiers replaced with placeholders, while preserving the structural format of the underlying record.
WHY IT FAILS
Masking removes the informal phrasing, emotional register, and implied context that define how real customers communicate under pressure. But the more significant failure is behavioural: an AI system tested on masked data is not tested on the data it will actually process in production. When deployed against real, unmasked correspondence, the model may produce outputs it never produced in testing: including surfacing, repeating, or mishandling personal information it was never exposed to during evaluation. You cannot rely on the behaviour of a model trained and tested on masked data to predict how it will behave on real data. The re-identification risk is a secondary concern. The behavioural unpredictability is the primary one. Masked data may still be vulnerable to re-identification.
Anonymisation
Starts with real data
Real data is transformed, so individuals cannot be identified by any reasonably likely means, including linkage attacks and contextual inference.
WHY IT FAILS
Reliable anonymisation of complex unstructured data is extremely difficult to achieve and harder to validate. Context that appears benign in isolation can reintroduce identity when combined with other information. It is also difficult to maintain consistency across the digital paper trail, Anonymisation is risk-managed, not risk-free.
Risk remaining:
Does not eliminate privacy risks
Privacy risk remaining:
None (data removed)
Synthetic Substitution
Starts with real data
Real PII (Personally Identifiable Information) is replaced with plausible fictitious equivalents - a real name becomes a different real-sounding name, a real account number becomes a different account number; while the structure, tone, and contextual flow of the original document are preserved. Unlike redaction and masking, the document remains linguistically intact after substitution.
WHY IT FAILS
Masking removes the informal phrasing, emotional register, and implied context that define how real customers communicate under pressure. But the more significant failure is behavioural: an AI system tested on masked data is not tested on the data it will actually process in production. When deployed against real, unmasked correspondence, the model may produce outputs it never produced in testing: including surfacing, repeating, or mishandling personal information it was never exposed to during evaluation. You cannot rely on the behaviour of a model trained and tested on masked data to predict how it will behave on real data. The re-identification risk is a secondary concern. The behavioural unpredictability is the primary one. Masked data may still be vulnerable to re-identification.
Masking removes the informal phrasing, emotional register, and implied context that define how real customers communicate under pressure. But the more significant failure is behavioural: an AI system tested on masked data is not tested on the data it will actually process in production. When deployed against real, unmasked correspondence, the model may produce outputs it never produced in testing: including surfacing, repeating, or mishandling personal information it was never exposed to during evaluation. You cannot rely on the behaviour of a model trained and tested on masked data to predict how it will behave on real data. The re-identification risk is a secondary concern. The behavioural unpredictability is the primary one. Masked data may still be vulnerable to re-identification.
DIY ALTERNATIVES | AVOIDING REAL DATA
These three approaches avoid real data entirely. That solves the privacy problem. But they introduce a different failure: the data they produce does not behave like the real thing. AI systems evaluated against it are not evaluated against reality.
Risk remaining:
Delay in delivering GenAI systems; non-core capability.
AI-Generated Data
DIY alternative
An organisation may consider solving the challenges of generating synthetic test data by simply prompting a large language model to produce their own synthetic complaints, records, or transcripts. In short, “Why buy synthetic data when we can build our own capability to generate it?”
WHY IT FAILS
A large enterprise could build a synthetic test data capability, but they’ll spend months creating a non-core capability and still need to operationalise and maintain it. So, the decision is not about whether an enterprise can build it, it’s about whether it would be prudent for them to spend the time, capital, and risk.
It is the same argument for why an organisation would buy a CRM or an accounting system. They could build it internally, but it’s prudent just to buy it.
A third-party provider, on the other hand, has a ready-made, independent, and scalable testing framework. It’s more than a series of prompts, it’s a production-grade testing capability for your GenAI systems.
Risk remaining:
reflects ideals, not reality
Manual Staff Creation
DIY alternative
Internal staff - often customer service or operations teams, are tasked with manually writing synthetic records, complaints, or case notes from a brief.
WHY IT FAILS
Staff writing from a brief produce grammatically correct, structurally logical documents. They reflect how employees think customers write; not how customers actually do. Real customer communications under pressure, frustration, or distress carry an informal register, emotional variability, and implicit signals that briefed writing never produces. The effort is significant, the output is privacy-safe, and the gap only becomes visible after deployment; when it is too late to course-correct without remediating the system from the ground up.
Risk remaining:
legal exposure remains
Scraping Public Data
DIY alternative
Publicly available complaint data is scraped from forums, review sites, and social media platforms and used as proxy test data.
WHY IT FAILS
Scraped data is uncontrolled, biased toward extreme sentiment, and structurally different from how customers communicate in a direct B2C service context. A customer posting a complaint on a public forum writes differently from a customer calling a contact centre, submitting a web form, or emailing a complaints team. The language, register, and intent are not comparable. Scraping public data does not give you a proxy for your customer base; it gives you a self-selected sample of people who chose to post publicly, writing in a channel your AI system will never operate in. It also carries legal risk: under the Privacy Act 1988 and similar frameworks scraping and repurposing publicly available data may still constitute collection of personal information if individuals are identifiable, even indirectly.
All three DIY alternatives avoid real data but produce test data that does not reflect how real customers, staff, or field teams actually communicate. AI systems built on this foundation will fail when they encounter the real thing.
This is not an evaluation inconvenience. It is a production failure with real consequences.
AI systems that fail in evaluation for the reasons described above do not fail quietly. They fail in production, in front of real customers, under real regulatory scrutiny, and at the full cost of a deployed system that must be remediated or withdrawn. In high-accountability environments, that means board notification, regulatory assessment, and remediation cycles measured in months and significant resource.
The organisations that have experienced this describe the cost of fixing inadequate evaluation data after deployment as materially higher than the cost of getting it right before. The question is not whether adequate evaluation data matters. It is whether your current approach produces it.
The Shared Problem
Every common approach either starts with real data and attempts to make it safe or avoids real data and produces something that does not behave like it.
The result is the same in both cases: AI systems that perform well in testing and fail in production.
“When real data is too risky and imitations are too unrealistic, unstructured synthetic data constructed for your context becomes essential.”
What Data Infusion Does Instead
Every approach described above has one thing in common: the team building the AI system controls what goes into the test data. Whether they are masking their own records, prompting an LLM, briefing staff to write documents, or scraping public sources, the evaluation data reflects their assumptions about what the system should encounter - not an independent, validated assessment of what it will. That is the root of the problem. And it is the structural gap that Data Infusion is built to close.
Data Infusion constructs unstructured synthetic datasets that are specifically built for each organisation’s context; their customer profile, communication channels, regulatory obligations, and the specific failure modes their AI system is most likely to encounter. The construction process is deliberate, documented, and arms-length from the team building the system.
NUROSCEND™ governs the persona construction, behavioural logic, and linguistic parameters that determine how language, tone, and intent are distributed across the dataset, ensuring it reflects the full range of real-world communication, not a cluster of averaged outputs. The completed dataset is placed into CALTREN™, where it is validated against the organisation's specific domain context and testing objectives before delivery.
Adequate test data requires construction that is independent of the team building the system, validated against the specific domain and regulatory context, and documented to a standard that survives external scrutiny. That is not a prompt. That is a capability.
The question is not whether an organisation could attempt something similar internally. Some have tried. In one engagement, an organisation had spent the equivalent of more than four months of analyst resource producing a dataset that failed pre-deployment review. The cost of remediating the deployed system exceeded the cost of the original dataset by a significant margin. The consequences described on this page are not hypothetical. The scenario pages for financial services, telecommunications, health, retail energy, government, and retail each show how this has played out in practice and what resolution required.
No real customer or personal data is accessed, transferred, or stored at any stage. This means organisations can test GenAI systems safely under any jurisdiction - Australian privacy legislation, GDPR (General Data Protection Regulation), CCPA (California Consumer Privacy Act), UK Data Protection Act, or PDPA (Personal Data Protection Act); simultaneously, without cross-border privacy obligations or regulatory exposure.
The scenario pages for Financial Services, Retail, Health, Retail Energy, Government & Education, and Telecommunications each show how this has played out in practice and what resolution required.
Unstructured Synthetic Data
Eliminate Risk. Preserve Realism. Scale with Confidence.
What This Enables
Organisations that work with Data Infusion resolve the test data problem that every other approach leaves open:
Test and evaluate GenAI systems against data that reflects how real customers, staff, and field teams actually communicate - not how educated professionals writing from a brief think they do
Cover the full distribution of scenarios, edge cases, and escalation signals including the implicit, indirect, and emotionally variable communications that every other approach strips out
Present a documented, auditable, arms-length testing methodology that survives pre-deployment review, compliance scrutiny, and board-level governance assessment
Move from AI initiative to safe deployment without waiting for data governance approvals, legal sign-off on real data use, or remediation cycles caused by test data that did not reflect production conditions
Next Steps
Request
Industries