Determining Jackknife ESS for a CTA Model with Chaotic Instability

Paul R. Yarnold

Optimal Data Analysis, LLC

CTA models are developed using one of three different strategies as concerns “leave-one-out” (LOO) analysis: (a) ignore LOO analysis; (b) only include attributes having identical ESS in training and LOO analysis in the model (the “LOO stable” criterion); or (c) only include attributes having the highest ESS in LOO analysis in the model (the “LOO p < 0.05” criterion). Software for performing CTA reports ESS for training but not for LOO analysis, so a recent article demonstrated the use of UniODA to assess ESS in LOO for CTA models with well-organized instability propagation—for example, restricted to a pair of endpoints for a node, or invalidating the CTA model for statistically unreliable replication. The present article illustrates assessing LOO ESS under chaotic conditions in which instability propagates down and across the left- and the right-hand sides of the CTA model.

View journal article

Using UniODA to Determine the ESS of a CTA Model in LOO Analysis

Paul R. Yarnold

Optimal Data Analysis, LLC

CTA models may be constructed using three different strategies with respect to consideration of “leave-one-out” (LOO) jackknife validity analysis: (1) ignore LOO validity analysis; (2) only include attributes yielding the same ESS in training and LOO analysis in the model (the “LOO stable” criterion); or (3) include attributes with highest ESS in LOO analysis in the model (“LOO p < 0.05” criterion). CTA software produces the confusion table for a CTA model for training analysis, but not for LOO analysis. This article shows how to use UniODA to determine the ESS of CTA models in LOO analysis. Exposition clearly demonstrates that failing to account for model cross-generalizability performance in classification analysis can produce models with good training performance and chance (or worse) reproducibility.

View journal article