Paul R. Yarnold
Optimal Data Analysis, LLC
Decision consistency is “…appropriate for describing the degree of consistency that is realized when educational and psychological measures are used to make pass/fail decisions about examinees”. In this method a test or parallel form is administered twice and the pass/fail decision at each testing is cross-classified. The fraction (or percent) of overall agreement, and kappa, are used to index decision consistency, but both of these statistics have limitations overcome by the chance- and maximum-corrected ESS statistic.
