RESEARCH DISCIPLINE
Online Behavioral Experiment Analysis
Exclusion Rules and Trial-Level Models
For experiments run in a browser or on a crowdsourcing platform, online behavioral experiment analysis means exclusion rules fixed in advance, attention checks reported by condition, trial-level models with the right random effects, and a check that conclusions do not hinge on who was removed.
No Commitment Required
Strictly Confidential
In-House PhD-Level Statisticians
Fixed Project Fee
Same-Day Response
How Are Online Behavioral Experiments Analyzed?
Online behavioral experiment analysis deals with participants who complete tasks or surveys over the internet, often recruited through a crowdsourcing platform. Such samples differ from laboratory samples in attention, equipment and motivation, so attention check exclusions and other rules are set before collection and reported by condition. Crowdsourced sample data analysis often deals with tasks that record many trials per participant, which calls for models with random effects for participants and for items. For OnlineSPSS, a specialist statistical consultancy for academic, clinical and professional research, in-house PhD-level statisticians fit these models and keep a record of every exclusion. What the results show about behavior is for the researcher to interpret.
WHEN TO ENGAGE
Online Experiments Between Design and Report
Whether the study is still a draft or already under review, the open question is a rule about who remains in the sample.
Rules Before Launch The study is built but not yet launched, and the exclusion criteria, the attention checks and the planned models need to be specified so that they can be preregistered.
Raw Data, Many Exclusions The data are in, a share of participants failed a check or finished suspiciously early, and the exclusion decision has to be made and documented without looking at the condition effect.
Trials, Not Participants Each person contributed many trials, and the plan must say whether to average by participant or model trials directly, with random effects and a distribution that suits the response.
A Reviewer Doubts the Sample A reviewer questions bots, duplicate respondents or unequal dropout between conditions, and the analysis must show what was removed and how far the conclusions depend on it.
EXCLUSIONS AND MODELS
Exclusions, Trial Data and Condition Effects
Each decision constrains the next, from who remains in the sample to how the condition effect is estimated and reported.
Rules for Who Stays
Criteria based on attention checks, completion time, duplicate responses and task accuracy are stated in advance, applied blind to condition and reported with the number removed in each arm.
Checks After Treatment
A check placed after the manipulation can be affected by the condition, so excluding those who fail it may leave groups that are no longer comparable, and the full-sample result is reported beside the restricted one.
Differential Attrition
Dropout and exclusion rates are compared across conditions, since unequal rates signal selection, and the sensitivity of the effect to the excluded participants is shown.
Trial-Level Models
Accuracy and binary choices are modeled trial by trial with Generalized Linear Mixed Models (GLMM), with random effects for participants and items and a justified structure.
Repeated Factors
Designs crossing a between-participant factor with repeated conditions can use Mixed ANOVA on participant means when the data allow it, and trial-level models when they do not.
Reporting the Sample
The platform, the screening and the final sample are described following Experimental Psychology and Cognitive Psychology conventions.
ONLINE SAMPLE FLAWS
Weaknesses Specific to Online Samples
In a pooled table these weaknesses are invisible; broken down by condition, the exclusions reveal them.
Exclusions Chosen After Looking Rules are set after seeing which version yields the expected effect, so the final sample is shaped by the result and the interval is too optimistic.
Short Completion Read as Bots Completion time alone removes participants, though duration varies with device and task, so genuine brisk respondents are dropped and the rule is hard to defend.
Averaging Over Items Responses are averaged over the stimuli for each participant, so the particular items are treated as fixed, and a claim about faces or words in general rests on the few that happened to be chosen.
Maximal Models That Fail A model with every possible random slope does not converge and is simplified by trial and error, with no record of the sequence of models tried.
Repeat Participants The same person joins related studies or several sessions under different identifiers, which breaks independence and can contaminate a replication.
One Pool, General Claims Results from a single crowdsourced pool are described as findings about people in general, without noting how the pool differs from the population of interest.
DELIVERABLES
Exclusion Log, Models and Condition Effects
The files separate what was removed from what was estimated, so a reader can recompute the result with and without any exclusion rule.
Exclusion Rule Sheet Each criterion with its threshold, the stage at which it applies and the number of participants it removes in every condition, for you to check and incorporate in a preregistration or methods section.
Retention by Condition Recruited, started, completed and retained participants by condition, with the reason for each loss, in the layout the target journal expects.
Random-Effects Record The fixed and random effects of each trial-level model, the distribution used, notes on convergence and the models compared on the way to the final one.
Condition Effect Tables The effect of each condition on the outcome with its 95% confidence interval, from the main sample and from the restricted sample side by side.
Robustness Summary A short table showing how the key effect changes under alternative exclusion rules and model structures, so the conclusion's dependence on them is visible.
Switchable Exclusion Script Starting from the platform export, the cleaning script applies each exclusion rule as a named, switchable step, so any rule can be turned off and every model refitted without it.
ILLUSTRATIVE EXAMPLE
Exclusions in an Online Visual Search Task
An illustration of how exclusion rules and a trial-level model were fixed for an experiment run on a crowdsourcing platform.
ILLUSTRATIVE EXAMPLE · ONLINE VISUAL SEARCH EXPERIMENT RESEARCH · UNITED STATES
A cognitive psychology researcher ran a visual search experiment on a crowdsourcing platform, with response time and accuracy recorded on every trial, and wanted to test whether a distractor manipulation slowed responses.
Before the data were examined, the exclusion rules were fixed: an attention item placed before the manipulation, limits on completion time and a minimum accuracy on easy trials. The analysis modeled trials directly, with random effects for participants and for the search displays and a distribution suited to positive, right-skewed response times. More participants had been excluded in one condition than in the other, which suggested that a rule interacted with task difficulty. The exclusion counts were reported by condition, and the main effect was estimated both on all participants who completed the task and on the retained sample.
Outcome: The researcher received the exclusion sheet, the retention table, the condition effect with its 95% confidence interval for both samples, the robustness summary and the code.
This example does not describe a real client or project. We never share details of actual client work, in line with the confidentiality we promise every researcher. This illustrative example is included to show the type, scope, and standard of work involved in this kind of analysis.
SOFTWARE
Preparing Platform Exports and Fitting Models
Browser-based data arrive as raw exports, so the package must handle the cleaning as well as the random-effects models used for trials.
HOW WE ASSESS YOUR PROJECT
We Review Your Actual Research
Your assessment is based on a review of your actual research materials and analytical requirements, not a generic project category.
You Submit Your Project Details
Share whatever is available at the current stage, such as the research aim, protocol or proposal, dataset, variable list, manuscript or statistical requirements.
We Review Your Research and Define the Scope
Your statistically trained Research Coordinator reviews the materials to understand the study design, research questions, data structure, analytical requirements and expected deliverables. If clarification is needed, additional documents or information can be requested before the assessment is finalized.
Your Analysis Is Completed and Delivered
Once the scope and fixed project fee are agreed, the in-house statistician completes the work and the Research Coordinator remains the communication point through delivery, with the analysis, deliverables and timeline as agreed.
RESEARCH RESPONSIBILITY
The Experimenter Argues How Far the Effect Reaches
Generalizing from a platform sample to people at large is a scientific argument, not a statistical output, and the researcher who designed the task makes it.
Agreed Statistical Support OnlineSPSS provides the agreed methodological, analytical and statistical reporting support, defined in a scope that is confirmed before any work begins.
No Intellectual Contribution OnlineSPSS does not claim any intellectual contribution to your experiment. Authorship of the preregistration, the manuscript and any thesis built on it remains yours.
Your Research Decisions The experimental design, the choice of task and sample, the psychological interpretation of the effect, the discussion and the conclusions remain yours, with overall responsibility.
Confidential Handling Research data and project details are handled confidentially, project information is not published, and an NDA is available where required.
FREQUENTLY ASKED QUESTIONS
Attention Checks, Dropout and Item Effects
A failed check, an early exit and a sampled stimulus raise three different questions about who and what an effect describes.
What does the analysis of an online experiment cover?
Analysis of experiments run over the internet covers the exclusion rules, the reporting of attention checks by condition and the models for trial-level data. Between-participant, repeated and mixed designs are covered, whether the task ran in a browser or through a platform panel. The experimental design and the interpretation of the effects are decisions for the researcher.
Should participants who fail an attention check be excluded?
It depends on when the check occurs and on what was preregistered. A check placed before the manipulation can define the sample in advance without disturbing randomization, but one placed afterward may be affected by the condition, so excluding failures can unbalance the groups. The usual approach is to state the rule beforehand, report exclusions by condition and show results with and without the exclusion.
What if more participants drop out in one condition?
Then the finishers in each condition may no longer be comparable, because the condition itself may have shaped who stayed, and the estimate on finishers can be biased. Dropout is reported by condition, the point at which people left is examined, and a bounds analysis shows how far the effect could move under the most unfavorable assumptions about those who left.
Do I need random effects for items as well as participants?
When the stimuli are a sample from a larger set and the claim extends beyond them, yes, because ignoring item variation can make an effect look more reliable than it is. When the stimuli are the only ones of interest, treating items as fixed may be defensible. The justification is stated either way, and the model’s random-effects structure is reported.
How do you protect my research data and confidentiality?
Your data and project details are handled confidentially, and project information is not published. Researcher names and institutions are not used publicly without permission. An NDA is available where required and can be arranged before sensitive materials are shared. Read more about confidentiality and data security.
What happens when I request an assessment, and how long does it take?
You share your research question, design and relevant materials, such as the preregistration, the exclusion rules and the platform export, and a statistically trained Research Coordinator reviews the request and responds the same day. The analysis, deliverables, timeline and one fixed project fee are then confirmed before any work begins, so the timeline follows from the agreed scope.
Still have a question? Find more answers in our full FAQ.
NEXT STEP
Exclusion Rules Settled
Before Analysis
An effect that survives only one exclusion rule is a weaker finding than it looks, so the rules for an online sample are worth settling while they can still be set blind to condition.
No Commitment Required
Strictly Confidential
Fixed Project Fee
In-House PhD-Level Statisticians
Same-Day Response
Serving Researchers Since 2014