Marketing prediction review workshop / 6 MIN READ
Read the two kinds of prediction error before praising accuracy
A model can be correct often while missing the rare request the team actually needs to find.
AI-assisted research and analysis for the SNS Creative Fieldbook
Sources checked 2026-11-07
A classification score is meaningful only when its positive class, evaluation population and error counts are clear. Review the complete prediction-versus-outcome table at a fixed operating rule before accepting a single headline percentage.
What you will leave with: A labeled error-count sheet with separate missed-case and unnecessary-review interpretations.
Use this when
- A supplier reports strong accuracy for a marketing enquiry classifier.
- The team needs to understand what a fixed model actually misses and falsely flags.
This evaluates a fixed classification rule; it does not choose a threshold, establish probability calibration or authorize automated decisions about people. Use ordinary service-routing examples, qualified analytic review and protected customer information.
Define a positive case in customer terms
Write the exact label the model should identify, such as an incoming enquiry requesting a product demonstration. State the unit of analysis and the permitted evidence used to establish the reference label. A thread, message and customer are different units. Record how mixed requests and ambiguous cases are handled instead of forcing them into whichever class improves the report.
Separate the model’s prediction from the reference decision. Keep the original prediction and its operating-rule version, then compare it with independently reviewed evidence. If the same automated system generated both columns, agreement may show consistency rather than correctness. Ask the responsible reviewer to resolve labeling uncertainty without exposing unnecessary customer text in a broadly shared presentation.
Recover all four cells before interpreting a rate
Create four counts: correctly identified requests, incorrectly flagged nonrequests, missed requests and correctly left-unflagged nonrequests. Reconcile their sum with the eligible evaluation population and report unresolved reference labels separately. Keep duplicates, exclusions and incomplete follow-up visible. A chart that omits the unflagged population cannot establish how many genuine requests were missed.
Google’s classification guidance defines precision among predicted positives and recall among actual positives. It also warns that overall accuracy can be misleading with imbalanced classes. These are different denominators, not interchangeable synonyms for quality. Label each percentage in plain language beside its count so the business reader can distinguish an unreliable review queue from an incomplete one.
Connect the errors to the actual workflow
Examine a small permitted sample from each error cell. A missed demonstration request may follow a wording pattern the classifier fails to recognize; an unnecessary flag may come from a customer asking about an existing booking. Describe the observed confusion before inventing its cause. Preserve the ordinary support route for unflagged messages so a classification mistake does not make a customer disappear.
Review whether the evaluation resembles the intended operating population. A deliberately balanced test collection may help diagnose language coverage while differing from the inbox mix the team will actually receive. Ask the analyst to qualify any expected workload or success-rate claim under that mismatch. Do not multiply a favorable test percentage by a production volume without checking that the implied population and error definitions still fit.
Report a bounded finding and a specific repair question
State which error matters to the present decision and why. For a staff review queue, show both the burden of false flags and the requests left outside it. Keep counts and uncertainty beside the rates, especially for small language or channel groups. If a denominator is empty, report the metric as unavailable under its stated definition rather than assigning a flattering perfect value.
Choose a focused next step such as auditing reference labels, collecting permitted examples of an underrepresented phrasing pattern or asking the supplier to evaluate a candidate revision. Preserve the fixed-rule result for comparison. Do not change the threshold mid-report and call the difference a correction to arithmetic. Acceptance means the team understands the observed mistakes and their workflow consequences, not that one attractive percentage certifies the system for every audience.
Make the reasoning concrete.
A fictional equipment showroom receives many routine service messages and a smaller stream of demonstration requests. Its classifier is praised for high overall accuracy, but the review team has never examined demonstration requests left in the unflagged inbox.
Approve the headline
Opening: Accept the overall percentage and inspect only the flagged queue.
Hypothesis: The team can report a simple quality claim.
Risk: Missed requests remain outside the inspection and may dominate the relevant business problem.
Reconstruct the error table
Opening: Compare predictions with independently reviewed labels across the full permitted evaluation sample.
Hypothesis: The team can distinguish unnecessary review from missed demand.
Risk: Reference-label ambiguity and limited sample coverage must remain visible.
The showroom reviews both flagged and unflagged evidence and keeps ordinary service handling intact. It does not treat a high aggregate accuracy as proof that every demonstration request reaches the right team.
Take the questions into your next brief.
Use these prompts in your own notes. Nothing is submitted from this page.
Positive class
What specific customer request should the model identify?
Reference
Who or what establishes the comparison label independently?
Counts
Do all four error cells reconcile to the eligible sample?
Population
How does the evaluation mix differ from the intended inbox?
Consequence
Which missed-case and false-flag effects matter to the workflow?
Watch for the wrong conclusion.
- Inspecting only flagged cases and claiming full recall.
- Using accuracy to hide errors in an uncommon class.
- Treating an empty metric denominator as perfect performance.
Your next action
Ask for the four labeled counts behind one classification claim and inspect one permissible example from each nonempty error cell.
Sources & scope
Workflows, worksheets and hypothetical cases are original editorial analysis. Sources support explicitly attributed technical principles. Qualified owners select and validate methods; no live model changes, measured business results or named human review are claimed.
- Google Machine Learning Crash Course: Accuracy, recall, precision, and related metrics →Primary technical guidance · Supports: Supports precision/recall denominators, accuracy limits under imbalance and undefined metrics with empty denominators. No source examples, exercises or numeric results are reproduced.