Keep the ratio’s two moving parts inside the experiment analysis

State the exact ratio and the population it weights

Write the numerator, denominator and aggregation in plain language. Total revenue divided by total orders is different from calculating each customer’s revenue per order and averaging those personal ratios. The first gives more influence to people with more orders; the second requires an explicit rule for people with no orders. Neither should silently replace the other after the result is visible.

Use a tiny invented fixture to make the difference concrete. One customer with one order and another with nine orders do not receive equal weight in an overall per-order ratio. Have the analyst confirm which quantity matches the business question and how zero denominators are handled. Keep the planned assigned population visible even when some members generate no denominator events.

Preserve dependence at the randomization unit

Provide the analyst with the numerator and denominator contributions at the appropriate independent assignment level, rather than only two campaign totals. A user who generates many sessions creates related observations. Treating every session as if it were independently assigned can make uncertainty look smaller than the design supports.

Statsig documents using the delta method to approximate ratio variance while accounting for covariance between numerator and denominator from the same users. This is a specific platform method and a useful reminder that the two components are not independent. Ask the analyst which method is suitable here, including any clustering, skew or small-denominator concerns, rather than copying a formula into a spreadsheet without its assumptions.

Read the component movements beside the headline rate

Show numerator and denominator changes in compatible units and time windows, alongside the ratio contrast. If revenue per order rises because small orders disappear, the business may have lost customers or revenue despite the attractive rate. Conversely, a lower average order value may accompany useful additional purchases. The ratio alone cannot decide which experience is better.

Inspect the mechanism of any denominator change. Did the campaign alter behavior, eligibility, logging or the opportunity to act? A real behavioral change belongs in the interpretation; a measurement break needs correction. Do not neutralize a treatment-created denominator movement by arbitrarily holding it fixed and still call the answer the same estimand. If another quantity is needed, define and analyze it explicitly with the specialist.

Present uncertainty and decision guardrails without false precision

Ask for the method’s limitations when denominators are small or highly variable and when relative lift divides by an uncertain baseline. A symmetric narrow interval is not automatically reliable in those settings. Report undefined or unstable quantities honestly rather than replacing them with zero or a visually tidy percentage. Keep the absolute business components available for readers.

Connect the result with the planned outcome and guardrails, such as total contribution, successful customers and service quality. A ratio improvement should not override a clear deterioration in the customer outcome it was meant to represent. Preserve the metric definition, unit-level aggregation and analysis version for future tests. The goal is a decision about a changed experience, not a celebration of whichever fraction happens to move upward.

Make the reasoning concrete.

A fictional retailer tests an order minimum. Revenue per order rises, but the number of orders falls. The dashboard reports a strong result from a calculator that treats each order as independent even though customers, not orders, were randomized.

The retailer separates the ratio from total commercial outcomes and requests appropriate inference. It does not relabel an order-level calculation as a customer-randomized experiment result.

Take the questions into your next brief.

Use these prompts in your own notes. Nothing is submitted from this page.

1. Definition

   Is this a ratio of totals or an average of individual ratios?

2. Unit

   At what level were experiences independently assigned?

3. Components

   How did numerator and denominator each change?

4. Uncertainty

   Does the method account for their dependence and denominator behavior?

5. Decision

   Which customer and business guardrails can overturn the ratio-only verdict?

Watch for the wrong conclusion.

• Treating every repeated event as an independent randomization.
• Changing between ratio of totals and average of ratios silently.
• Calling a higher rate a success without examining the denominator.

Sources & scope

Workflows, worksheets and fictional examples are original editorial analysis. Named documentation supports identified method principles. Qualified analysts must assess the actual design; no measured business result or named human reviewer is claimed.

• Statsig: Delta Method →Primary experimentation-method documentation · Supports: Statsig explains ratio variance using numerator and denominator variance and covariance. Its implementation is not asserted to be the correct method for every experiment; the acceptance workflow is original analysis.