Using synthetic users with care

I treat synthetic users as tools for exploring questions. A simulated response might help a researcher notice an assumption or rehearse a discussion guide. Before using it to support a finding about people, I would need evidence from real participants or observed behavior.

The distinction matters because generated answers can sound familiar and convincing. A readable response gives us something to examine. We still need to establish whether anyone has the experience it describes and whether that experience is relevant to the decision.

Be clear about what produced the response

A synthetic response comes from a model working with its training and the context in a prompt. The team chose what to supply, which role to simulate, and how to ask the question. Those choices affect what comes back.

A participant brings their own circumstances and can challenge the question, describe a surprise, or explain something the researcher had not considered. A model-generated response does not establish that a person holds the view or would act that way. I would keep that limitation attached whenever the material is shared.

Before using a generated answer

Ask what observed evidence would support the claim.

Give simulation a specific job

I would use a synthetic response for a task where we can inspect what it contributes. For example, a team could:

• draft competing hypotheses before fieldwork
• rehearse an interview guide and look for leading questions
• identify an edge case to consider during recruitment
• examine the assumptions behind a concept
• compare possible interpretations before returning to the source material

Those are preparation and review tasks. If a generated answer suggests a useful question, the next step is to work out how to investigate it. Claims about prevalence or what a population needs require evidence that can support those claims.

Look for missing context

A smooth persona can leave out the conditions that make a task difficult. Access to a tool, informal workarounds, disability, or fear of speaking openly may change what someone can do. A generated account may describe those conditions plausibly without giving us a way to know whether they apply.

I would ask what the supplied context includes and what is absent. Which experiences might the prompt overlook? What assumptions did we build into the role? The answers help us decide which people and situations we still need to hear from.

Compare the output with real evidence

If the team has interviews, observations, or relevant prior studies, I would compare the synthetic output with them. Keep a record of where it agrees with observed evidence and where it differs. Also record claims that sound credible but have no supporting reference.

A difference can give us a useful question. Agreement can be worth investigating too, especially if the model was given the same research it now appears to confirm. The team should be able to explain how the comparison was made and what independent evidence was available.

Keep a record people can review

Label the material as synthetic and retain the prompt and sources used to produce it. Describe how the team checked it and what human validation is still needed. The record should let another researcher understand what happened without relying on the confidence of the generated language.

For a consequential decision, I would start by identifying the observed evidence the decision requires. That gives the team a clear way to decide whether simulation has a useful supporting role and where additional fieldwork is needed.