Opens in a new tab

Home / Resources / AI in Research

p-Hacking With AI:
Forking Paths, Re-Prompting and Tools That Agree With You

p-hacking with AI refers to using the flexibility of AI-assisted analysis, often without full disclosure, to search for a preferred statistical result. The underlying problem predates AI: choices about data, tests and reporting can inflate false-positive findings when guided by the observed p-values (Simmons et al., 2011). AI can make exploring options faster, so transparent planning matters even more.

AI in Research
TERMSForking paths
NEEDSPrespecified plan
REPORTAll analyses tried
GUIDELINEPreregistration
COMMON ERRORRe-prompting to agree

Exploration itself is not misconduct. The concern arises when data-driven decisions are described as if they were specified in advance, or when unsuccessful analyses vanish from the record.

How might AI increase analytic flexibility?

A branching path of analysis choices narrowing to the one result that gets reported.
Scenario Legitimate use Research-integrity risk
Asking for alternative tests Learning methods or sensitivity checks Selecting only the result with the smallest p-value
Re-prompting until an answer is agreeable Clarifying assumptions and revising code Quietly replacing critical explanations with confirming ones
Trying multiple exclusions Investigating influential observations Omitting exclusions that do not support the desired finding
Testing many outcomes and subgroups Exploratory hypothesis development Presenting post hoc subgroup discoveries as planned confirmatory tests
Simulating research participants Methods development and pilot exploration Choosing prompt/model settings that reproduce a desired pattern

The statistical problem existed before language models

Simmons, Nelson and Simonsohn (2011) demonstrated how undisclosed researcher flexibility can increase false positives in conventional psychological research. The result does not mean that trying multiple models is always wrong. It means the inferential status of a result depends partly on which decisions were driven by the observed data.

Prospective analysis plans, clear distinction between confirmatory and exploratory analyses, and transparent presentation of alternatives let readers judge the evidence. Nelson et al. (2018) describe how disclosure and preregistration became important parts of improvements in psychological research practice.

What is AI sycophancy, and why does it matter?

In language-model research, sycophancy refers to responses that align excessively with the user’s stated beliefs or preferences rather than reliably tracking the evidence. Sharma et al. (2024) documented the behavior in specified model evaluations. That does not establish that every AI output is sycophantic. It suggests a reason to ask neutral questions rather than instructing a model to justify a desired conclusion.

Compare “Why does my significant finding prove the intervention works?” with “What does this estimate support under this design, and which interpretations would be unjustified?” The second prompt is more likely to expose uncertainty, but it still needs independent human verification.

New evidence from synthetic participants

Cummins (2026) tested many model and prompt configurations used to create AI-generated “silicon samples.” The study found that different settings materially changed how well those samples matched human observations. This illustrates an AI-specific version of analytic flexibility: a researcher may choose not only a statistical model but also a simulated participant configuration that affects the apparent result.

It does not imply that every synthetic-data experiment is invalid. The correct response is to prespecify key design choices where possible, report alternatives and validate against suitable human benchmarks.

A defensible workflow against p-hacking with AI

  1. Specify the primary research question, outcome, estimand and confirmatory analysis before inspecting the findings where feasible.
  2. Keep a dated record of changes to eligibility, transformations, missing-data rules, model specification and AI prompts used for method selection.
  3. Label new analyses suggested after seeing results as exploratory or sensitivity analyses.
  4. Report relevant robustness checks and avoid showing only the preferred specification.
  5. Review the statistical inference against the design rather than judging the workflow by the number of significant p-values.

This checklist is a suggested application of established open-science principles, not an official universal journal requirement.

What about re-running the same AI prompt?

Repeated answers can differ depending on models, settings and context. For numerical research, the final published estimate should trace to a verified data-analysis procedure. Preserve the executed code and analytical decisions, not just the most convincing chat response (Sandve et al., 2013).

Frequently asked questions

Not automatically. Exploratory analyses can be scientifically valuable. Problems arise when the selection process is hidden or confirmatory claims exceed what the design supports.

AI can draft checklists and compare specifications, but prevention depends on transparent research practices, defensible methods and disclosure of analytic choices.

It can identify unsupported model changes and selective reporting. Independent Statistical Review covers analyses prepared with AI assistance. Review is a quality-control step, not an admission that AI or the researcher failed.

Independent Statistical ReviewWant a second methodological reading before submission? A statistician reviews the analysis plan, the specifications tried and what you report.

Request an Assessment

Independent Statistical Review

Independent Statistical Review at OnlineSPSS gives your analysis a second methodological reading from an in-house PhD-level statistician. The review checks method, assumptions, reproducibility and reporting. Authorship remains the researcher’s.

References

Last reviewed November 2026 by the OnlineSPSS Statistical Team. Worked examples use hypothetical values and are not client data.

NEXT STEP

Need this done for
your study?

Share your research in about a minute and receive a complimentary assessment the same day: scope, timeline and one fixed fee.

No Commitment Required

Strictly Confidential

Fixed Project Fee

In-House PhD-Level Statisticians

Same-Day Response

Serving Researchers Since 2014