Precision and Convergence of Monte Carlo Estimation of Two-Category UniODA Two-Tailed p
Paul R. Yarnold, Ph.D. and Robert C. Soltysik, M.S.
Optimal Data Analysis, LLC
Monte Carlo (MC) research was used to study precision and convergence properties of MC methodology used to assess Type I error in exploratory (post hoc, or two-tailed) UniODA involving two balanced (equal N) classes. Study 1 ran 106 experiments for each N, and estimated cumulative p’s were compared with corresponding exact p for all known p values. Study 2 ran 105 experiments for each N, and observed the convergence of the estimated p’s. UniODA cumulative probabilities estimated using 105 experiments are only modestly less accurate than probabilities estimated using 106 experiments, and the maximum observed error (±0.002) is small. Study 3 ran 105 experiments for Ns ranging as high as 8,000 observations in order to examine asymptotic properties of optimal values for balanced designs.