AI Evaluation Sample Size & Confidence Lab
Calculate Wilson confidence intervals, conservative sample requirements and whether observed task success supports a release target.
← Explore the full tool libraryEnter your scenario
The form starts with hypothetical examples. Replace them with your own measurements or assumptions. Inputs and exports stay in this browser.
What decision does this help answer?
Does an observed 93% pass rate from 200 examples actually establish a 90% target?
Calculation and methodology
Calculate an observed proportion from accepted and evaluated tasks. The 95% Wilson score interval accounts for finite binomial samples better than a naive normal interval, particularly near 0% or 100%. A separate conservative sample-size estimate uses z² × 0.25 divided by squared target margin; this is a planning estimate, not a power analysis. Release criteria use the lower confidence bound, not the point estimate.
Limitations and responsible interpretation
Representative, independent observations and a stable scoring rubric are assumed. Correlated examples, multiple language slices and severity-specific failures require a richer design. A statistical interval cannot certify a safe launch; this is an educational decision aid.
Research and supporting guidance
This planning tool complements Ninth Post's related AI product analysis. Consult the underlying technical guidance or official framework where applicable, and see our editorial methods. The tool does not independently validate submitted assumptions, current vendor prices or legal obligations.