Measure simulation precision without certifying the input assumptions

Define exactly what one replication reports

Law’s tutorial analyzes independent replication summaries rather than treating correlated observations inside a run as independent evidence. Hold the model and reporting definition fixed for the precision comparison.

The synthetic operation attempts 100 fulfillments per replication. Each attempt independently succeeds with stipulated probability 0.2; one summary is the number fulfilled, in kits per 100 attempts. These are invented inputs, not an estimate of a retailer’s reliability. The mathematical expectation is 20, which supplies a known answer for this teaching fixture.

Agree what the report will be used for before choosing a numerical tolerance. An owner comparing broad planning alternatives may need different resolution from an analyst checking a calculation. Neither purpose automatically justifies a customer promise. Record the proposed use so the precision number does not acquire a stronger meaning as it moves into a presentation.

Recompute the numerical standard error

For R replication summaries, calculate their mean, sample variance s² = sum of squared deviations / (R − 1), and Monte Carlo standard error s / √R. NIST’s mean-uncertainty formula uses this standard-error factor. In the fixed 40-run experiment, total fulfilled is 784, so the mean is 19.6. Squared deviations sum to 685.6; s = √(685.6 / 39) = 4.1928, giving standard error 0.6629 kits per 100 attempts.

The independently streamed, prespecified 160-run experiment has mean 19.98125, s = 4.3183 and standard error 0.3414. Its closer answer in this realization is not a guarantee that every larger experiment will be more accurate.

NumPy documents high-probability independence for spawned streams. With NumPy 2.3.5 and root 202611070081, use children 1, 2 and 3 of SeedSequence(root).spawn(4). Spawn 40, 160 and 160 children respectively; each default_rng(child).random(100) is compared with its experiment’s probability.

A second calculation uses standard-library arithmetic and checks the results against NumPy. Preserve the unrounded values in the accompanying record and round only for explanation. A reviewer should be able to trace the displayed precision back to the replication summaries without trusting the formatting of a dashboard.

Change an input without relabeling the uncertainty

Song, Nelson and Pegden separate simulation noise conditional on supplied inputs from uncertainty about those inputs. More replication does not establish that the supplied model describes the real system.

A separate fixed 160-run sensitivity case sets success probability to 0.3. Its observed mean is 30.19375 and numerical standard error 0.3597 kits per 100 attempts. Comparing these cases does not assign probabilities to the two input assumptions or quantify an input-uncertainty distribution.

Present that change beside the original assumption, not inside its precision label. Ask the commercial owner what evidence could support the proposed reliability figure. A precise answer under an unsupported assumption can still be a poor basis for action, while an honest sensitivity comparison can identify which evidence is worth seeking.

Do not select the case that makes the recommendation easiest. Preserve the owner’s unresolved question and explain whether better input evidence, a different analytical model or a narrower decision is needed. This exercise does not fit a distribution to production records or recover information that the organization has never collected.

Choose the next effort for the right reason

Write separate conclusions about the calculation and its business basis. If the numerical result is too imprecise for its stated use, discuss a justified computation plan with the analyst. If the business assumption is weak, additional computing alone is not the relevant remedy. The decision owner needs to know which problem remains.

Keep the initial run-count commitment and all results when requesting further analysis. Do not continue until a favored answer appears and then describe the last result as the original plan. A revised inquiry should explain why the additional work was commissioned and preserve the earlier version.

Close with a bounded recommendation about evidence, not an operational change. The fulfilled-kit figures are synthetic and do not establish actual service quality. Retain their labels when sharing the analysis so an illustrative numerical exercise cannot become an unsupported forecast or a supplier-performance claim.

Make the reasoning concrete.

A fictional operations reviewer is told that a large simulation is trustworthy because its output looks precise.

The reviewer preserves both records and requests evidence for the business assumption. No fulfillment policy or customer promise changes.

Take the questions into your next brief.

Use these prompts in your own notes. Nothing is submitted from this page.

1. Summary

   What does one run report and in which units?

2. Calculation

   Can the displayed result be reconstructed from saved records?

3. Commitment

   Which run counts were fixed before inspecting results?

4. Input

   What business evidence supports the assumed reliability?

5. Effort

   Does the next useful step require computation or different evidence?

Watch for the wrong conclusion.

• Treating an invented probability as measured reliability.
• Presenting the final favorable run as the original plan.
• Allowing a synthetic illustration to become a supplier claim.

Sources & scope

Original decision reasoning and synthetic examples. Source-derived methods and their applications remain attributed. Fixtures were executed locally with NumPy 2.3.5 and independently checked arithmetic. No actual operational data, staffing decision, customer promise or configuration change is involved.

• Statistical Analysis of Simulation Output Data: The Practical State of the Art →Primary methodological documentation · Supports: Supports replication-level analysis.
• Advanced Tutorial: Input Uncertainty Quantification →Primary methodological documentation · Supports: Supports conditional and input uncertainty distinctions.
• Confidence Limits for the Mean →Primary methodological documentation · Supports: Supports the standard-error factor.
• Parallel random number generation →Primary methodological documentation · Supports: Supports spawned stream construction.