How to Assess the Inter-Method (Parallel-Forms) Reliability of Ratings Made on Ordinal Scales: Emergency Severity Index (Version 3) and Canadian Triage Acuity Scale

Paul R. Yarnold

Optimal Data Analysis, LLC

An exact, optimal (“maximum-accuracy”) psychometric methodology for assessing inter-method reliability for measures involving ordinal ratings is used to evaluate and compare two emergency medicine triage algorithms—both of which classify patients into one of five ordinal categories. Ten raters independently evaluated the identical set of 200 patients, five with each algorithm. UniODA revealed moderate levels of inter-method agreement, which is theoretically consistent with recent findings of moderate inter-observer reliability for triage ratings derived using both algorithms.

View journal article