ILLUSTRATIVE EXAMPLE
Prediction Model Study for a
Customer Churn Question
A logistic regression and a boosted-tree model compared for 90-day churn in about 30,000 subscribers, with a customer-level temporal split, calibration and bootstrap intervals, reported in journal format.
This example does not describe a real client or project. Every engagement is confidential, during the project and after it ends.
RESEARCH QUESTION
What the Study Set Out to Ask
This prediction model study example follows a research team that studies subscription retention and holds de-identified account records from a subscription service. The team wants to know how well cancellation within 90 days can be predicted from usage and billing history, and whether a boosted-tree model improves on a logistic regression by enough to justify its added complexity.
The aim is prediction, not explanation, so no coefficient or importance score is read as the reason a customer leaves. Performance has to be judged on customers the model has not seen, and at a time after the training period, because that is how a model is used. The plan fixes the outcome definition, the prediction date, the split and the performance measures before any model is fitted.
STUDY DESIGN AND DATA
About 30,000 Customers and a 90-Day Window
A record-based cohort of about 30,000 subscribers, each entering at one prediction date, with usage and billing features summarized over the preceding six months and an indicator of cancellation within the next 90 days. Around 9% cancel, so events are plentiful for a modest set of features, but the outcome is imbalanced. Subscribers whose 90-day window had not finished when the data were extracted are excluded, and customers who entered in an earlier period form the training data while those who entered later form the test data.
- Design: record-based cohort with a prediction date, a 90-day outcome window and a later test period
- Outcome: cancellation within 90 days of the prediction date
- Software requested: Python, with scripts and a fixed environment file
METHOD AND ASSUMPTIONS
Why a Temporal Split, and What Must Hold
Leakage occurs when a model is built with information that would not be available at the moment of prediction, including anything from the test data, and it inflates measured performance. Here no customer appears in both sets, and the test period follows the training period, which shows how performance holds up as behavior and pricing drift. Feature construction, scaling and tuning happen inside the training folds only. Logistic regression is the baseline and a boosted-tree model is the challenger, tuned by cross-validation within the training data, and both are scored once on the held-out test customers.
Accuracy is a poor yardstick when 9% of customers cancel, so discrimination is measured by the area under the ROC curve and by the precision-recall curve, and calibration is examined directly, since a retention team acts on predicted probabilities. Balancing the classes, by resampling or by class weights, pulls predicted probabilities away from the observed cancellation rate, so the data keep their natural balance, and probabilities are recalibrated on validation data only if the calibration plot shows a need. Feature importance describes what the model uses, not what causes cancellation. The prediction date and the exclusion of incomplete windows are stated, so the estimate does not rest on an unstated choice.
OUTPUT AND INTERPRETATION
How the Results Are Read
The output has the area under the ROC curve and the precision-recall summary for each model with 95% confidence intervals from a bootstrap that resamples customers, the paired difference between the models with its interval, a calibration plot with the calibration slope and intercept, and the Brier score. Precision and recall are given at the share of customers the team could afford to contact, with the threshold chosen on validation data and not on the test set.
The advantage of the boosted model is judged by the interval for the paired difference and by calibration, not by the point estimate alone, and a small difference with a wide interval supports keeping the simpler model. Performance in the later period is the best available estimate of future use, and its limits are stated: one service, one period, and a model that can degrade as behavior shifts. The model is not described as explaining churn, and any decision to act on its predictions remains with the research team.
REPORTING
What the Researcher Received
A written report in journal format, with the cohort definition, the leakage safeguards, the tuning procedure and the performance measures written so that another group can reproduce the study from the scripts. The research question, the interpretation and the conclusions remain with the research team.