Can AI Choose the Right Statistical Test? The Design Facts It Cannot See
Can AI choose the right statistical test? It often can when the research question and data structure are described clearly, and recent benchmark studies report strong performance on standardized scenarios. But a test that runs successfully is not necessarily the right method: the decisive information concerns design, dependence, missing data and the intended scientific claim (Shukla et al., 2025; Hernán & Robins, 2020).

The useful question is not “Is AI good at choosing tests?” but “What facts must the choice be based on, and who checks that those facts apply?”
Can AI choose the right statistical test? What the evidence says
In a 2025 study, Shukla and colleagues evaluated six large language models using 20 standardized hypothesis-testing scenarios. All six selected the expected tests in that study, although the quality of their explanations varied. This is encouraging evidence for structured methodological questions, not proof that every model will correctly handle incomplete descriptions or complicated real-world studies (Shukla et al., 2025).
A separate 2026 investigation found that ChatGPT could generate accurate code for several quantitative methods, especially in R and Python, while still needing correction and guidance for more complicated work (Winberg et al., 2026). These studies examined different capabilities: recommending a method and writing executable code are not the same as validating an entire research design.
Which design facts should AI know first?

| Study information | Why it changes the analysis |
|---|---|
| Primary outcome and measurement scale | Continuous, binary, count, time-to-event and ordinal outcomes support different models |
| One measurement or repeated measurements | Observations from the same participant are correlated |
| Independent patients or clustered sampling | People from the same clinic, school or site may not be statistically independent |
| Randomized, observational or nonrandomized exposure | Causal interpretation and confounding strategies differ |
| Time structure and censoring | Follow-up duration and censoring matter for survival outcomes |
| Missing outcomes and covariates | The analyzed sample may differ by model and missing-data approach |
| Primary estimand or research question | Without randomization, comparing group averages is not the same as estimating an intervention effect |
A prompt may omit these facts without the omission being obvious in a polished answer. Human expertise is useful in recognizing what is not yet known.
How can a mathematically correct output answer the wrong question?
Consider a hypothetical study measuring anxiety before and after a training program. A paired t-test might calculate the mean difference correctly. But if the study also has an untreated comparison group and aims to estimate the intervention’s effect, the paired result alone may not address that question. The design and research aim determine whether a between-group, longitudinal or causal approach is required.
Similarly, a standard regression can run correctly on patients recruited at several hospitals while ignoring hospital clustering. The software may return coefficients and p-values, yet the uncertainty estimates may not match the dependence structure. This illustrates the difference between computational correctness and methodological suitability; it does not mean that AI inevitably makes that mistake.
A responsible way to ask AI for methods advice
Use a fictional or approved non-sensitive description, then request a shortlist rather than a single unexplained verdict. For example: “I am comparing [outcome] across [groups]. Participants were selected using [sampling method], measured at [time points], with clustering by [unit]. The primary research question is [question]. List candidate analyses, their assumptions, what additional design facts you need, and the consequences of choosing each approach. Do not invent missing design information.”
Then check: Does the answer use the right outcome? Does it account for pairing or clustering? Does it distinguish an association from a causal effect? Does it specify a missing-data strategy? Does it match the protocol or statistical analysis plan? Has the proposed model been independently checked against the actual dataset and study objective?
What should the statistician confirm?
A statistical review should examine the underlying decision, not just whether the code ran without errors. Depending on the study, that means reviewing the estimand, assumptions, unit of analysis, covariates, multiplicity, model fit, missing data, effect estimates and interpretation. Human judgment is not a magical guarantee either: it must be transparent and supported by analysis documentation.
This article focuses specifically on what changes when AI participates in the selection workflow.
Frequently asked questions
Is ChatGPT reliable at choosing statistical tests?
It can be useful and may perform well on clearly specified questions, as shown in the limited scenarios examined by Shukla et al. (2025). Its recommendation still needs to match the study’s full design and objectives.
Can AI do the analysis after choosing the test?
Often it can generate code, but the code and model still need to be executed, checked and interpreted in the approved environment (Winberg et al., 2026).
Do I need a statistician for a simple t-test?
Not necessarily. The level of review should reflect study complexity, stakes, requirements and the researcher’s expertise. A high-stakes or journal-bound analysis may benefit from an independent check even when the arithmetic is simple.
Independent Statistical ReviewWant the test confirmed against your design? A statistician reviews the outcome, the design and the assumptions before you report the result.
Independent Statistical Review
OnlineSPSS provides Independent Statistical Review through its in-house PhD-level statisticians. The review checks method, assumptions, reproducibility and reporting. Independent Statistical Review covers analyses prepared with AI assistance.
References
- Hernán, M. A., & Robins, J. M. (2020). Causal inference: What if. Chapman & Hall/CRC. https://www.hsph.harvard.edu/miguel-hernan/causal-inference-book/
- Shukla, M., Pandey, D., Kaur, S., Agarwal, M., Goyal, A., & Sharma, H. (2025). Evaluating the accuracy and explanatory quality of large language models ChatGPT, Claude, DeepSeek, Gemini, Grok, and Le Chat in statistical test selection for hypothesis testing decisions. Cureus, 17(10), e94949. https://doi.org/10.7759/cureus.94949
- Winberg, D., Tsai, E., Tang, T., Xuan, D., Marchi, N., & Shi, L. (2026). Can AI write your code? A case study of ChatGPT’s statistical coding capabilities for quantitative research. Health Economics Review, 16, Article 17. https://doi.org/10.1186/s13561-025-00714-8
Last reviewed November 2026 by the OnlineSPSS Statistical Team. Worked examples use hypothetical values and are not client data.
NEXT STEP
Need this done for
your study?
Share your research in about a minute and receive a complimentary assessment the same day: scope, timeline and one fixed fee.
No Commitment Required
Strictly Confidential
Fixed Project Fee
In-House PhD-Level Statisticians
Same-Day Response
Serving Researchers Since 2014