The Golden Wooden Spoon: Manufacturing Evidence Instead of Collecting It
First place among my worst or most dangerous uses of AI in healthcare market research goes to manufacturing evidence instead of collecting it.
Synthetic respondents can help explore hypotheses, rehearse questions or test scenarios. The danger begins when generated answers are treated as evidence of what real patients or healthcare professionals think.
A plausible response is not a lived experience.
Inventing the patient voice
Imagine a company developing support materials for people starting treatment.
Instead of interviewing patients, researchers ask AI to simulate their concerns. It produces convincing accounts of injection anxiety, side effects and remembering doses.
Those concerns might be relevant. But real patients could identify something the simulation misses: e.g. difficulty collecting prescriptions while working unpredictable shifts.
The company creates reassuring materials while overlooking the practical obstacle.
The output sounds empathetic. The resulting service fails to address what patients need.
Turning simulated enthusiasm into investment
Now imagine asking synthetic physicians whether they would prescribe a proposed treatment.
Their responses appear positive, and the company interprets this as evidence supporting launch investment.
But generated answers may reflect assumptions embedded in the model or prompt. They cannot establish how actual prescribers will respond to local reimbursement restrictions, competing options or clinical uncertainty.
A company could commit resources on the strength of enthusiasm nobody expressed.
Adding respondents without adding evidence
Suppose a study struggles to recruit enough people with a rare condition.
Generating additional responses might make the dataset look more substantial. However, those rows are not independent interviews.
Without appropriate validation and transparent methods, they can create an unjustified impression of certainty.
The company may believe a finding is better supported than it really is and make decisions that the evidence cannot justify.
What could this cost?
Misallocated budgets, unsuitable services, unrealistic forecasts and expensive repeat research are all possible consequences.
For agencies, presenting synthetic material as genuine testimony could also damage credibility and client relationships.
These are hypothetical examples, but the underlying distinction matters: simulation and observation answer different questions.
Keep synthetic work in its proper place
Label generated material clearly. Explain its purpose, assumptions, validation and limitations.
Never present an invented quotation as something a respondent said. Never quietly replace missing interviews with generated answers.
Where synthetic methods inform decisions, establish that they are suitable for that use and distinguish their contribution from real research.
AI can help us imagine possibilities.
The golden wooden spoon belongs to pretending those possibilities are evidence.