LLM Judge Agreement Calculator
Calculate agreement rates and Cohen's kappa between human and automated evaluators using a paired binary confusion matrix.
← Explore the full tool libraryEnter your scenario
The form starts with hypothetical examples. Replace them with your own measurements or assumptions. Inputs and exports stay in this browser.
What decision does this help answer?
Does our model-as-judge make comparable decisions to a human reviewer?
Calculation and methodology
Enter four counts for paired human/AI pass/fail decisions. Calculate observed agreement, expected chance agreement from marginal distributions, Cohen's kappa where defined and the disagreement types. The output makes disagreement directions visible rather than reporting one correlation-like number.
Limitations and responsible interpretation
Kappa depends on prevalence and class balance; two reviewers may agree on easy cases while failing on high-severity edge cases. Human labels are not automatically ground truth. For ordinal or multi-label outcomes use a more appropriate agreement measure.
Research and supporting guidance
This planning tool complements Ninth Post's related AI product analysis. Consult the underlying technical guidance or official framework where applicable, and see our editorial methods. The tool does not independently validate submitted assumptions, current vendor prices or legal obligations.