Practical framework / 5 MIN READ
Check whether predicted probabilities mean what they say
Compare probability forecasts with mature outcomes before using them to plan resources.
AI-assisted research and analysis for the SNS Creative Fieldbook
Sources checked 2026-11-06
A model can rank likely converters well while its probabilities are too optimistic. Summing those scores into a sales forecast or multiplying them by deal value requires a different check from deciding which lead comes first.
What you will leave with: Build a calibration review that separates ranking, probability accuracy and operational usefulness.
Use this when
- Predicted conversions consistently exceed observed conversions.
- Probability scores are being used for staffing or expected-value planning.
This lesson does not certify an individual prediction or replace qualified model validation.
Fix the prediction contract
Define the outcome, prediction time, eligible population and outcome window. A probability of purchasing within seven days cannot be evaluated against eventual purchase or against only customers who stayed subscribed. Preserve the score recorded when the decision was made rather than recalculating it with later information.
Set aside evaluation data that reflect the intended deployment setting. Keep model fitting, any calibration fitting and final evaluation appropriately separated. Ask the analyst to account for time ordering and repeated customers so that later knowledge does not leak into an apparently impressive assessment.
Compare predicted groups with observed frequency
The scikit-learn documentation describes calibration curves that compare average predicted probabilities with observed positive frequencies in prediction bins. It also cautions that Brier loss reflects more than calibration, so a lower score alone does not prove probabilities are better calibrated.
For the business review, show the number of eligible observations, mature outcomes and average prediction in each group. Include uncertainty and sparse groups. Choose grouping and evaluation methods with the analyst before looking for a flattering picture; changing bins repeatedly can hide an important weakness.
Investigate where expectations fail
Inspect relevant operating segments and time periods without declaring every small group a separate model. A change in acquisition channels, offer availability or follow-up practice can change the relationship between scores and outcomes. Distinguish an outdated probability mapping from an outcome-recording problem.
Check whether all leads had a reasonable opportunity for the target outcome. If the old team contacted only high-scoring leads, observed results incorporate that treatment policy. Recalibrating against those results may reproduce the feedback loop. Do not invent outcomes for ignored leads or assume a calibration curve solves the selection problem.
Approve a bounded use and a review date
Specify what the probabilities may support: aggregate capacity planning, a scenario forecast or a limited decision aid. Keep a range around aggregate expectations and separate uncertain deal value from uncertain conversion probability. A calibrated forecast for a group still cannot promise which particular person will act.
If recalibration is justified, evaluate the updated model on untouched evidence before deployment. Record version, covered population, limitations and monitoring triggers. Preserve a fallback when inputs or outcome collection fail. The owner should know when the probabilities are unsuitable, not merely where to find the latest dashboard.
Make the reasoning concrete.
A hypothetical evaluation group contains 100 mature leads with an average predicted purchase probability of 0.60. Only 40 purchase in the defined window. The 20-outcome shortfall flags a calibration question, but one group alone does not establish the cause or uncertainty.
Trust the ranking
Opening: Use a strong ranking metric to justify forecasting 60 purchases.
Hypothesis: Higher scores may still identify better prospects.
Risk: Ranking quality does not validate the numerical probabilities.
Review probability evidence
Opening: Inspect the calibration curve, sample uncertainty and outcome process.
Hypothesis: The team can separate forecast optimism from missing or selected outcomes.
Risk: A repair needs fresh evaluation rather than fitting to this one discrepancy.
Hold off on treating the scores as dependable aggregate forecasts until the analyst reviews calibration and the prediction contract. Preserve any justified ranking use separately.
Take the questions into your next brief.
Use these prompts in your own notes. Nothing is submitted from this page.
Contract
What outcome and horizon does each probability describe?
Evaluation
Were scores frozen before outcomes and tested on separate data?
Calibration
How do predicted groups compare with mature outcomes?
Selection
Did earlier follow-up policies affect observed labels?
Use
Which decisions and populations are approved?
Watch for the wrong conclusion.
- Equating good ranking with accurate probabilities.
- Treating Brier loss as a pure calibration measure.
- Evaluating scores against a different outcome horizon.
- Fitting and validating a repair on the same observations.
Your next action
Build a probability-versus-outcome review for one model and ask the responsible analyst to assess uncertainty and deployment suitability.
Sources & scope
Frameworks, worksheets and hypothetical examples are original editorial synthesis. Linked primary sources support only the stated concepts. Source checking is not an independent audit of effectiveness. No named human author or reviewer is claimed.
- scikit-learn: Probability calibration →Primary technical documentation · Supports: Defines calibration curves comparing mean predicted probability with observed positive frequency, and warns that Brier loss combines calibration and discrimination. The marketing acceptance worksheet is original analysis.