Paul R. Yarnold
Optimal Data Analysis, LLC
CTA models may be constructed using three different strategies with respect to consideration of “leave-one-out” (LOO) jackknife validity analysis: (1) ignore LOO validity analysis; (2) only include attributes yielding the same ESS in training and LOO analysis in the model (the “LOO stable” criterion); or (3) include attributes with highest ESS in LOO analysis in the model (“LOO p < 0.05” criterion). CTA software produces the confusion table for a CTA model for training analysis, but not for LOO analysis. This article shows how to use UniODA to determine the ESS of CTA models in LOO analysis. Exposition clearly demonstrates that failing to account for model cross-generalizability performance in classification analysis can produce models with good training performance and chance (or worse) reproducibility.
odajournal.com 