RESEARCH GUIDE
Core Statistical Concepts for Researchers
Estimation, Uncertainty and Error Rates
Core statistical concepts for researchers are the ideas that let a reader judge whether an analysis answers its question: what is estimated, how uncertain the estimate is and how often the procedure would mislead. Each idea below is tied to the decision it informs in a real study.
In statistical inference, a few ideas are applied again and again: a sample is used to estimate a quantity in a population, the estimate carries sampling uncertainty, and a decision rule has known error rates. Statistical reasoning in research adds the design, because the same numbers mean different things depending on how participants were selected, how groups were formed and how many analyses were run. Together these are the core statistical concepts for researchers who need to read output critically and defend it to reviewers. OnlineSPSS, a specialist statistical consultancy for academic, clinical and professional research, explains the reasoning behind each result in the report that accompanies the analysis, while the research questions and conclusions remain the researcher’s.
CONCEPT FAMILIES
Types of Statistical Reasoning and the Question Each Settles
The concepts fall into families that answer different questions, and mixing them up is the source of many familiar errors in published results.
Estimation A sample statistic estimates a population quantity, and the first task is to name that quantity, such as a mean difference or a risk ratio, before any calculation.
Uncertainty Standard errors and confidence intervals describe how much an estimate would vary across repeated samples, which is the honest way to show precision.
Hypothesis Tests A test asks how unusual the data would be if a specified null model were true, which is a narrower question than whether an effect exists.
Error Rates and Power The chance of a false positive is set by the threshold and the number of tests, and the chance of detecting a real effect depends on its size and the sample.
Effect Size The size of a difference or association, in original or standardized units, shows whether a result matters in practice, which a p-value cannot.
Association and Causation A statistical association supports a causal claim only through design, such as random allocation, or through adjustment for confounders whose assumptions are stated and defended.
MEASURE COMPARISON
Confidence Intervals, p-Values and Effect Sizes Compared
Three summaries are reported together in most analyses, and each answers a different question, as the table sets out.
| Feature | Confidence interval | p-value | Effect size |
|---|---|---|---|
| Question answered | Which values of the quantity are compatible with the data? | How unusual are the data if the null model were true? | How large is the difference or association? |
| Units | Those of the estimate | None, a probability from 0 to 1 | Original or standardized units |
| Effect of a larger sample | The interval narrows | The p-value tends to fall if the true effect is not zero | The estimate stabilizes around the true value |
| Common misreading | The probability that this one interval contains the true value | The probability that the null hypothesis is true | Judged against fixed benchmarks in every field |
| How to report | With the estimate, at a stated level | Exact, beside the estimate and interval | With its own interval |
KEY PLANNING DECISIONS
Reading Any Result Before Trusting It
These checks apply to a result from the researcher's own study or from a paper being read, and each can be answered from the methods section.
What Is the Estimand? Identify the exact quantity the analysis targets and the population it refers to, since a precise answer to a vague question is of little use.
How Was the Sample Obtained? Look at selection and response, because intervals and tests describe sampling variability only and cannot repair a sample that was not representative.
What Error Rate Was Accepted? Check that the significance level was set before the analysis and that it reflects the cost of a false positive against that of a missed effect.
Is the Outcome Measured Well? Ask how reliably the outcome and the main predictors were measured, since measurement error can weaken associations and widen intervals, and no analysis restores precision the instrument lacked.
Was the Study Large Enough? Ask what effect the sample could reliably detect, since a small study can miss real effects and exaggerate the ones it does find.
Which Assumptions Matter Here? Judge the assumptions behind the method from the design and from diagnostics in context, and not by passing a single test of them.
COMMON ISSUES
Concept Errors That Reach Published Results
Peer review does not reliably catch these errors, so they are worth checking in a draft while the conclusions can still change.
Significance Treated as Importance A very small effect can be significant in a large sample, so the size of the effect and its interval are judged alongside the p-value.
Interval Read as a Range of Values An interval for a mean describes the uncertainty of the mean and not the range of individual observations, a distinction the standard deviation or a prediction interval addresses.
Analyses Chosen After the Results Choosing the outcome, subset or model after seeing the data inflates false positives, so exploratory findings are labeled as such.
Percent Change From Baseline Expressing change as a percentage of the baseline value ties the outcome to its starting point and often behaves poorly in analysis, so change is usually modeled on the original scale with baseline as a covariate.
Missing Cases Dropped Silently Analyses that drop every case with a missing value can change the sample a result describes, so the number analyzed is reported and the likely reason for the gaps is considered.
Many Tests, One Threshold Running many tests against the same threshold raises the chance of at least one false positive, so the family of tests is defined and a correction or primary outcome is used.
REPORTING
Stating Estimates, Uncertainty and Decisions in Writing
Good reporting lets a reader recompute the reasoning, and it uses a few conventions that are easy to keep.
Results are reported as estimates with confidence intervals and, where a test is used, the exact p-value and the test statistic with its degrees of freedom. Under APA 7, effect sizes and their intervals are expected alongside tests, and describing a result only as significant or not leaves out the information a reader needs. The report states which analyses were prespecified and which were exploratory, and says how multiple comparisons were handled.
STATISTICAL METHODS
The Same Ideas Inside Individual Methods
Each method applies the same few ideas to a particular kind of data, with estimates, intervals and tests taking a form specific to that method.
Paired Samples t-Test
A mean change with its interval
One-Way ANOVA
Differences among several group means
Simple Linear Regression
A slope with its uncertainty
Bootstrapping
Uncertainty without a formula
Missing Data Analysis
Uncertainty added by absent values
Comparative & Hypothesis Testing
Group comparisons and their error rates
ANALYSIS SERVICES
Where Statistical Reasoning Is Checked in a Project
The concepts apply at every stage, from setting the error rate and the sample size to interpreting the result, so a check can be placed wherever the project needs it.
Quantitative Data Analysis
Analyses chosen for the question and design, with estimates, confidence intervals, effect sizes and diagnostics explained in the report.
Research Planning & Study Design
The estimand, the error rate, the sample size and the primary outcome fixed in an analysis plan before data collection starts.
Results Reporting & Interpretation
Estimates, intervals and tests written up with wording that matches what each can support, and no more.
Independent Statistical Review
A completed analysis is checked for the reasoning behind each claim, including multiplicity, power and the match between design and wording.
FREQUENTLY ASKED QUESTIONS
Questions About Core Statistical Concepts for Researchers
The two kinds of error, the meaning of a confidence level, the limits of a larger sample and regression to the mean are explained in turn.
What is the difference between a Type I and a Type II error?
A Type I error is rejecting a true null hypothesis, and a Type II error is failing to reject a false one. The significance level sets the first rate, and power, which is one minus the second rate, depends on effect size, sample size and variability. Neither error is always worse, and the balance is set by the consequences of a false positive and of a missed effect in the field.
What does a 95% confidence level mean?
It means that the procedure produces intervals that contain the true value in 95% of repeated samples, if the model is right. For one interval the true value is either inside or not, so the statement concerns the method and not the particular interval. A narrower interval signals more precision, and a different sample would give a different interval.
Does a larger sample make a result more trustworthy?
It makes the estimate more precise, but not necessarily more valid. A large sample narrows intervals and can make tiny effects significant, yet it does not remove bias from selection, measurement error or confounding. Trust depends first on the design and then on the precision.
What is regression to the mean, and why does it matter?
Regression to the mean is the tendency of units selected for an extreme value to score closer to the average when measured again, because part of the first value was chance. Patients enrolled for very high blood pressure can appear to improve with no treatment at all, so a study that selects on an extreme baseline needs a comparison group measured the same way before any change is credited to the intervention.
How do you protect my research data and confidentiality?
Your data, drafts and project details are handled confidentially. Project information is not published, and researcher names and institutions are not used publicly without permission. An NDA is available where required and can be arranged before sensitive materials are shared. Read more about confidentiality and data security.
What happens when I request an assessment, and how long does it take?
You share your research question, design and relevant materials, such as a protocol, an analysis plan or a dataset, and a statistically trained Research Coordinator reviews the request and responds the same day. The analysis, deliverables, timeline and one fixed project fee are then confirmed before any work begins, so the timeline follows from the agreed scope.
Still have a question? Find more answers in our full FAQ.
NEXT STEP
Have an Analysis Read
for Its Reasoning
Reviewer comments on an interval, a p-value or the size of a study are easier to answer once the reasoning behind the result has been checked, and every request is read first by a statistically trained Research Coordinator.
No Commitment Required
Strictly Confidential
Fixed Project Fee
In-House PhD-Level Statisticians
Same-Day Response
Serving Researchers Since 2014