Test shared-capacity changes with a time-and-region design

Map the resource customers share

Identify how one customer’s treatment changes another’s experience. A promotion may consume scarce appointments, increase a dispatch queue or use a shared inventory pool. Draw the operational boundary rather than assuming a postcode or campaign label contains the interference.

Check whether effects cross proposed regions through staff movement, customer travel or shared fulfillment. Specify whose outcome matters and whether the decision concerns a typical customer, order or operating period. These choices affect weighting and interpretation; they cannot be settled merely by choosing whichever summary looks most favorable.

Design the assignment units with the analyst

DoorDash’s engineering research describes randomizing region-time units to address shared-capacity effects and warns that observations inside those units are dependent. Its example illustrates why thousands of transactions do not automatically provide thousands of independent experimental assignments.

Have a qualified analyst design the schedule, time blocks and balance across relevant operating conditions. A predictable every-other-hour pattern may align with recurring demand or operational habits. Preserve the actual assignment record and deviations. Do not let local teams switch conditions opportunistically when demand becomes difficult.

Specify transitions and carryover

Ask how long the previous condition can affect the next: outstanding orders, depleted stock, customer expectations or a backlog may persist after the switch. Consider whether the intervention is reversible within the proposed period at all. A lasting brand message may be unsuitable for a short switchback.

Define transition handling before results are available. Any washout or exclusion period needs a substantive rationale and an analysis plan, not an arbitrary duration that produces a cleaner chart. Record incomplete transitions and operational failures. If meaningful carryover cannot be contained or modeled credibly, choose another design.

Analyze the assigned process and its limitations

Require inference that respects assignment, dependence, time patterns and the target weighting. Do not run a generic customer-level comparison that ignores the design. Ask how many genuinely informative units and transitions exist and how uncertainty changes under plausible carryover assumptions.

Review harm controls for customers and staff throughout the test. Stop unsafe exposure under the agreed plan without disguising the interruption. Report the tested operating conditions and limits on transfer to other regions or demand levels. A result from one resource-constrained system may not predict the same effect after capacity expands.

Make the reasoning concrete.

A hypothetical service tests a promotional message in region-time blocks. One treatment block creates a queue that persists into the next control block. Comparing all customer outcomes as independent rows would mix assignment effects with carryover and understate the design’s dependence.

Proceed only if the analyst and operations owner can justify assignment and transition handling. Otherwise use a different evaluation approach rather than forcing a switchback.

Take the questions into your next brief.

Use these prompts in your own notes. Nothing is submitted from this page.

1. Interference

   Which resources connect customer outcomes?

2. Unit

   What region and time boundary contains the treatment?

3. Carryover

   What persists after a switch and for how long?

4. Inference

   How are dependence and weighting handled?

5. Safety

   Who can stop exposure and document deviations?

Watch for the wrong conclusion.

• Counting transactions as independent assignments.
• Using an alternating schedule without checking demand patterns.
• Choosing washout periods after inspecting results.
• Assuming a reversible interface means a reversible customer effect.

Sources & scope

Frameworks, worksheets and hypothetical examples are original editorial synthesis. Linked primary sources support only the stated concepts. Source checking is not an independent audit of effectiveness. No named human author or reviewer is claimed.

• DoorDash Engineering: Experiment Rigor for Switchback Experiment Analysis →Primary experimentation research · Supports: Explains region-time randomization for shared-capacity interference and the nested, dependent observations it creates. No reported DoorDash performance improvement or specific estimator is generalized to marketing.