The Analytic Flexibility Problem

Social science spent the last fifteen years learning a painful lesson about flexibility in analysis.

The core idea, developed in the methods literature on questionable research practices and popularised as the garden of forking paths, is this: when a researcher faces many defensible choices, and makes them after seeing the data, the effective false-positive rate rises far above the nominal five percent. No fraud is required. Each choice is individually reasonable. The problem is that the set of choices was large and the reported path was selected.

The field's response was procedural: preregistration, multiverse analyses that report results across all defensible specifications, and robustness checks as a default expectation.

Silicon sampling reintroduces exactly this hazard, in a more severe form, and it arrived before those norms were established for it. Every synthetic study requires many choices, most are undocumented, and almost none are preregistered.

Consider what you must decide before a single synthetic response exists.

Which model. Provider, size, and version. Models are deprecated and replaced, so a study is not reproducible once its version retires.

Sampling parameters. Temperature above all. Low temperature concentrates on the modal response and destroys variance; high temperature restores spread and may introduce incoherence.

Prompt format. First person or third. Direct question or roleplay framing. Scale presented as numbers, labels, or both. Whether an explanation is requested before the answer.

Backstory content. Which demographic variables to include, in what order, at what level of detail.

Context. Whether respondents answer questions independently or in a running conversation, and in which order.

Post-processing. How to parse a response that does not fit the scale, and whether to discard or retry refusals.

Each has a defensible answer. There is no convention fixing any of them. And journal reporting standards for this work are still forming, so most published applications specify only a subset.

Jamie Cummins tested this directly in "The threat of analytic flexibility in using large language models to simulate human data", first posted September 2025 and revised through 2026 (arXiv:2509.13397). The design is simple and the result is uncomfortable.

Study 1 generated 252 silicon-sample configurations for a controlled case study using two social-psychological scales, then evaluated each configuration on three criteria: whether it recovered participant rankings, response distributions, and between-scale correlations.

Two findings. Configurations varied substantially across all three criteria. And, in the paper's own words, "configurations that performed well on one dimension often performed poorly on another."

That second point is the sharper one. There is no single best setup that a field could standardise on. Tuning for accurate rankings can cost you the distribution. The choice is a trade-off, and which trade-off you make determines what your study can support.

Study 1 used a controlled case. Study 2 asked whether the problem afflicts real published work, and the target chosen was the field's founding paper.

Cummins re-examined Argyle and colleagues' (2023) Study 3 using 66 alternative configurations, all defensible, and measured the correlation between human and silicon association structures in each.

The correlations ranged from r = .23 to r = .84.

Sit with the spread. At the top of that range you would report a strong correspondence and conclude silicon samples reproduce human structure well. At the bottom you would report a weak correspondence and conclude they largely do not. Same underlying method, same human comparison data, same published study. The only thing that changed was configuration choices any competent researcher might have made.

The paper's conclusion is stated plainly: "different defensible configuration choices can materially alter conclusions about the fidelity of silicon samples."

This is not a critique of Argyle and colleagues, whose work established the method. It is a demonstration that any single-configuration result in this literature, including favourable ones, is a point sampled from a wide distribution.

The mechanism requires no bad intent. A researcher tries a setup, finds the correspondence disappointing, adjusts something defensible, and tries again. Stopping when the result looks good is a selection rule, and it converts a wide distribution of possible answers into a single reported number that looks like a finding.

Analytic flexibility in conventional research is bad. Here it is worse, for four reasons worth separating.

The choice space is larger. Conventional flexibility concerns analysis after data collection. Here the choices affect data generation itself, so the underlying dataset differs across configurations rather than the treatment of a fixed one.

Iteration is nearly free. A human study is too expensive to re-field twenty ways. A silicon sample can be regenerated in minutes, so the practical cost of exploring the space until something works is close to zero.

Norms have not caught up. Preregistration and multiverse reporting are established expectations in psychology. For synthetic samples the equivalent conventions are still being written, so the same practices that would be flagged in a conventional paper pass unremarked.

The ground truth is often absent. Cummins could measure the spread because human comparison data existed. Most applied uses have none, which is the whole point of using synthetic respondents. Without it there is no way to know where in that .23 to .84 range your single configuration sits.

That last point is the practical crux.

The defensible answer is not to pick better, it is to stop pretending you picked once. Report the distribution of results across the configuration space instead of a single point from it.

import itertools, statistics

GRID = dict(
    temperature = [0.0, 0.7, 1.0],
    persona     = ["first_person", "third_person"],
    scale       = ["numeric", "labelled"],
    detail      = ["minimal", "full_backstory"],
)

results = []
for combo in itertools.product(*GRID.values()):
    cfg = dict(zip(GRID, combo))
    sample = generate_silicon_sample(cfg)
    results.append((cfg, estimate_effect(sample)))

effects = [e for _, e in results]
print(f"n configs      : {len(effects)}")
print(f"median effect  : {statistics.median(effects):.3f}")
print(f"range          : {min(effects):.3f} to {max(effects):.3f}")
print(f"sign agreement : {sum(e > 0 for e in effects) / len(effects):.0%}")

The last line is the one that carries weight. Sign agreement asks whether your qualitative conclusion survives the configuration space at all. If 95 percent of specifications agree on direction, you have something. If 60 percent do, you have a coin flip dressed as a finding, and no amount of narrative about the chosen setup changes that.

Whether you are commissioning research, reviewing it, or producing it, the same checklist applies.

The single most useful question to ask about any synthetic finding is: how many other configurations were run, and what did they show? If the answer is "one" the result is a draw from an unknown distribution. If the answer is "we do not track that", treat it as unreported flexibility rather than as evidence.

And there is a failure mode this checklist cannot catch, because it operates even when every setup choice is disclosed and preregistered. That is the subject of the final lesson.