AI Prompt Regression Comparison Lab
Compare paired baseline and candidate outcomes, identify net improvements and regressions, and explain confidence limits.
← Explore the full tool libraryEnter your scenario
The form starts with hypothetical examples. Replace them with your own measurements or assumptions. Inputs and exports stay in this browser.
What decision does this help answer?
Is the new prompt genuinely better on the same set of tasks?
Calculation and methodology
Accept paired pass/fail classifications from a common evaluation set: both pass, both fail, baseline-only pass (regressions) and candidate-only pass (improvements). Calculate version pass rates, net gain, and a discordant-pair diagnostic. When enough discordant pairs exist, display a continuity-corrected McNemar chi-square approximation as an indication of paired evidence, not a universal pass/fail rule.
Limitations and responsible interpretation
The tool does not run prompts or inspect content. Changes to the test set or scoring rubric invalidate a fair comparison. Severity-weighted failures and subgroup regressions may matter more than a small net gain. The chi-square approximation is unreliable for few discordant pairs.
Research and supporting guidance
This planning tool complements Ninth Post's related AI product analysis. Consult the underlying technical guidance or official framework where applicable, and see our editorial methods. The tool does not independently validate submitted assumptions, current vendor prices or legal obligations.