p-Hacking With AI:
Forking Paths, Re-Prompting and Tools That Agree With You
p-hacking with AI refers to using the flexibility of AI-assisted analysis, often without full disclosure, to search for a preferred statistical result. The underlying problem predates AI: choices about data, tests and reporting can inflate false-positive findings when guided by the observed p-values (Simmons et al., 2011). AI can make exploring options faster, so transparent planning matters even more.

Exploration itself is not misconduct. The concern arises when data-driven decisions are described as if they were specified in advance, or when unsuccessful analyses vanish from the record.
How might AI increase analytic flexibility?
| Scenario | Legitimate use | Research-integrity risk |
|---|---|---|
| Asking for alternative tests | Learning methods or sensitivity checks | Selecting only the result with the smallest p-value |
| Re-prompting until an answer is agreeable | Clarifying assumptions and revising code | Quietly replacing critical explanations with confirming ones |
| Trying multiple exclusions | Investigating influential observations | Omitting exclusions that do not support the desired finding |
| Testing many outcomes and subgroups | Exploratory hypothesis development | Presenting post hoc subgroup discoveries as planned confirmatory tests |
| Simulating research participants | Methods development and pilot exploration | Choosing prompt/model settings that reproduce a desired pattern |
The statistical problem existed before language models
Simmons, Nelson and Simonsohn (2011) demonstrated how undisclosed researcher flexibility can increase false positives in conventional psychological research. The result does not mean that trying multiple models is always wrong. It means the inferential status of a result depends partly on which decisions were driven by the observed data.
Prospective analysis plans, clear distinction between confirmatory and exploratory analyses, and transparent presentation of alternatives let readers judge the evidence. Nelson et al. (2018) describe how disclosure and preregistration became important parts of improvements in psychological research practice.
What is AI sycophancy, and why does it matter?
In language-model research, sycophancy refers to responses that align excessively with the user’s stated beliefs or preferences rather than reliably tracking the evidence. Sharma et al. (2024) documented the behavior in specified model evaluations. That does not establish that every AI output is sycophantic. It suggests a reason to ask neutral questions rather than instructing a model to justify a desired conclusion.
Compare “Why does my significant finding prove the intervention works?” with “What does this estimate support under this design, and which interpretations would be unjustified?” The second prompt is more likely to expose uncertainty, but it still needs independent human verification.
New evidence from synthetic participants
Cummins (2026) tested many model and prompt configurations used to create AI-generated “silicon samples.” The study found that different settings materially changed how well those samples matched human observations. This illustrates an AI-specific version of analytic flexibility: a researcher may choose not only a statistical model but also a simulated participant configuration that affects the apparent result.
It does not imply that every synthetic-data experiment is invalid. The correct response is to prespecify key design choices where possible, report alternatives and validate against suitable human benchmarks.
A defensible workflow against p-hacking with AI
- Specify the primary research question, outcome, estimand and confirmatory analysis before inspecting the findings where feasible.
- Keep a dated record of changes to eligibility, transformations, missing-data rules, model specification and AI prompts used for method selection.
- Label new analyses suggested after seeing results as exploratory or sensitivity analyses.
- Report relevant robustness checks and avoid showing only the preferred specification.
- Review the statistical inference against the design rather than judging the workflow by the number of significant p-values.
This checklist is a suggested application of established open-science principles, not an official universal journal requirement.
What about re-running the same AI prompt?
Repeated answers can differ depending on models, settings and context. For numerical research, the final published estimate should trace to a verified data-analysis procedure. Preserve the executed code and analytical decisions, not just the most convincing chat response (Sandve et al., 2013).
Frequently asked questions
Is it p-hacking to use AI to explore several models?
Not automatically. Exploratory analyses can be scientifically valuable. Problems arise when the selection process is hidden or confirmatory claims exceed what the design supports.
Can AI prevent p-hacking?
AI can draft checklists and compare specifications, but prevention depends on transparent research practices, defensible methods and disclosure of analytic choices.
Is an independent statistical review useful?
It can identify unsupported model changes and selective reporting. Independent Statistical Review covers analyses prepared with AI assistance. Review is a quality-control step, not an admission that AI or the researcher failed.
Independent Statistical ReviewWant a second methodological reading before submission? A statistician reviews the analysis plan, the specifications tried and what you report.
Independent Statistical Review
Independent Statistical Review at OnlineSPSS gives your analysis a second methodological reading from an in-house PhD-level statistician. The review checks method, assumptions, reproducibility and reporting. Authorship remains the researcher’s.
References
- Cummins, J. (2026). The threat of analytic flexibility in using large language models to simulate human data. Advances in Methods and Practices in Psychological Science. https://doi.org/10.1177/25152459261461505
- Nelson, L. D., Simmons, J., & Simonsohn, U. (2018). Psychology’s renaissance. Annual Review of Psychology, 69, 511–534. https://doi.org/10.1146/annurev-psych-122216-011836
- Sandve, G. K., Nekrutenko, A., Taylor, J., & Hovig, E. (2013). Ten simple rules for reproducible computational research. PLOS Computational Biology, 9(10), e1003285. https://doi.org/10.1371/journal.pcbi.1003285
- Sharma, M., Tong, M., Korbak, T., Duvenaud, D., Askell, A., Bowman, S., Durmus, E., Hatfield-Dodds, Z., Johnston, S., Kravec, S., Maxwell, T., McCandlish, S., Ndousse, K., Rausch, O., Schiefer, N., Yan, D., Zhang, M., & Perez, E. (2024). Towards understanding sycophancy in language models. International Conference on Learning Representations. https://proceedings.iclr.cc/paper_files/paper/2024/hash/0105f7972202c1d4fb817da9f21a9663-Abstract-Conference.html
- Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2011). False-positive psychology: Undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychological Science, 22(11), 1359–1366. https://doi.org/10.1177/0956797611417632
Last reviewed November 2026 by the OnlineSPSS Statistical Team. Worked examples use hypothetical values and are not client data.
NEXT STEP
Need this done for
your study?
Share your research in about a minute and receive a complimentary assessment the same day: scope, timeline and one fixed fee.
No Commitment Required
Strictly Confidential
Fixed Project Fee
In-House PhD-Level Statisticians
Same-Day Response
Serving Researchers Since 2014