Paul R. Yarnold
Optimal Data Analysis, LLC
Kendall’s coefficient of concordance W is a non-parametric statistic used to assess agreement in rankings of multiple stimuli made by multiple raters. A normalization of the test statistic for Friedman’s non-parametric alternative to ANOVA with repeated measures, W ranges from 0 (no agreement) to 1 (complete agreement). Problems with W include complications arising from tied rankings, invalid Type I error estimates for small samples, the absence of an effect strength index normed versus chance, and the absence of an explicit model relating ranks to stimuli. The use of UniODA to assess inter-rater agreement is demonstrated as an alternative approach that overcomes these issues.
odajournal.com 