Novometrics vs. ODA vs. One-Way ANOVA: Evaluating Comparative Effectiveness of Sales Training Programs, and the Importance of Conducting LOO with Small Samples

Paul R. Yarnold

Optimal Data Analysis, LLC

Immediately after graduating from one of four alternative sales training programs, graduates were randomly assigned to sales areas putatively having comparable sales opportunities: number of sales made by each of N=23 graduates at the end of their first week was recorded. Analysis by one-way ANOVA yielded F(3,19)=3.13, p<0.0281. It was concluded: “…evidence is sufficient to indicate a difference in mean achievement for the four training programs” (p. 383). If the omnibus effect (comparing all of the groups simultaneously) effect has p<0.05, then all-possible pairwise comparisons (or a more efficient range test procedure) are used to disentangle the omnibus effect and identify the statistically significant inter-group differences. This was not reported, but the combination of a test of a non-directional hypothesis (the anticipated relative ordering of mean sales by group was not specified a priori), in conjunction with the small sample and associated weak statistical power, limit the detectable effects to those reflecting extremely strong inter-group differences. Non-directional ODA treating group as the class variable and sales as the ordered attribute was unable to identify a statistically reliable model for discriminating all four sales groups (ESS=42.46, D=5.42, p<0.32). A single novometric model emerged: if sales<87.5 then predict group<4; otherwise predict group=4. Model performance in total sample analysis was relatively strong and statistically reliable: ESS=69.74, D=0.87, p<0.042 (sensitivity for group 4=75.00%, for groups 1-3=94.74%). Jackknife analysis suggested the effect may not cross-generalize if the model is used to classify different samples of graduates: ESS=43.42, D=2.61, p<0.015 (sensitivity for group 4=75.00%, for groups 1-3=68.42%).

View journal article