Statistical Evaluation of the Findings of Qualitative Comparative Analysis

Paul R. Yarnold

Optimal Data Analysis, LLC

Qualitative data analysis is a structured observational and clustering methodology which facilitates hypothesis development and variable generation for quantitative research, fruitfully employed in agriculture, anthropology, astronomy, biology, forensic investigation, education, history, marketing, medicine, political science, psychology, sociology, and zoology, to name a handful of diciplines. The method known as Qualitative Comparative Analysis (QCA) is adept in producing evidence in complex policy problems involving interdependencies among multiple causes. Recent research used QCA to study factors underlying high rates of teenage conceptions in high-risk areas in England. Nine binary attributes reflecting five different “variable constellations” were identified. Variable constellations are putatively associated with areas with teenage conception rates which are narrowing versus not narrowing (the outcome or class variable) with respect to the national average. This article discusses use of UniODA and CTA to ascertain which attributes are statistically reliable predictors of outcome.

View journal article

Exploratory Analysis for an Ordered Series of a Dichotomous Attribute: Airborne Radiation and Congenital Hypothyroidism of California Newborns

Paul R. Yarnold & Robert C. Soltysik

Optimal Data Analysis, LLC

Confirmatory hypothesis-testing methodology was recently demonstrated with an example assessing the effect of airborne beta nuclear radiation emanating from the Fukushima nuclear meltdown on the risk of confirmed congenital hypothyroidism (CH) for newborns in California in the years 2011-2012. Eyeball inspection of the data suggests that the a priori hypothesis which was evaluated is inconsistent with the actual data, so an exploratory analysis is conducted.

View journal article

Confirmatory Analysis for an Ordered Series of a Dichotomous Attribute: Airborne Radiation and Congenital Hypothyroidism of California Newborns

Paul R. Yarnold & Robert C. Soltysik

Optimal Data Analysis, LLC

Ordered series involving a dichotomous (binary) variable are widely used to describe changes in phenomena which occur across time. Examples of such series include the percentage of a sample or population each year (or other unit of time) that marries, dies, or is arrested. This article demonstrates how UniODA is used for such designs to test a confirmatory (a priori), omnibus (overall), optimal (maximum-accuracy) hypothesis, subsequently disentangled by a confirmatory (if hypotheses are composed) or otherwise by an exploratory (post hoc) optimal range test. This methodology is demonstrated with an example assessing the effect of airborne beta nuclear radiation emanating from the Fukushima nuclear meltdown on the risk of congenital hypothyroidism (CH) for newborns in California in the years 2011-2012.

View journal article

MegaODA Large Sample and BIG DATA Time Trials: Maximum Velocity Analysis

Paul R. Yarnold & Robert C. Soltysik

Optimal Data Analysis, LLC

This third time trial of newly-released MegaODA™ software studies the fastest-to-analyze application, known as a 2×2 cross-classification table. Designs involving unweighted binary data are arguably currently the most widely employed across quantitative scientific disciplines as well as engineering fields including communications, graphics, data compression, real-time processing and autonomous synthetic decision-making, among others. The present simulation research is run on a 3 GHz Intel Pentium D microcomputer and reveals MegaODA returns the exact one- or two-tailed Type I error rate, as well as all of the other classification-relevant statistics provided in UniODA analysis, in fractions of a CPU second for samples of a million observations.

View journal article

Determining When Annual Crude Mortality Rate Most Recently Began Increasing in North Dakota Counties, I: Backward-Stepping Little Jiffy

Paul R. Yarnold

Optimal Data Analysis, LLC

Recent research tested the hypothesis that the annual crude mortality rate (ACMR) was higher after versus before 1998 in counties of North Dakota, due to increased exposure of the population to environmental toxins and hazards beginning approximately at that time. This hypothesis was confirmed with experimentwise p<0.05 for 16 counties. This article investigates the ACMR time series for each of these counties using a backward-stepping little jiffy UniODA analysis to ascertain precisely when ACMR began to increase. As hypothesized the ACMR began increasing in Bowman and Kidder counties precisely in 1998. Consistent with the a priori hypothesis the initial (and presently sustained) increase in ACMR occurred in McLean county in 1997, and in Foster county in 1996. Significant sustained increases in ACMR initially began in Stark county in 1993, and in Burleigh county in 1988. The UniODA models identified hypothesized, recent, powerful, sustained, statistically significant increases in ACMR.

View journal article

Surfing the Index of Consumer Sentiment: Identifying Statistically Significant Monthly and Yearly Changes

Paul R. Yarnold

Optimal Data Analysis, LLC

Published monthly by the Survey Research Center of the University of Michigan, the Index of Consumer Sentiment (ICS) is widely followed, and one of its factors (the Index of Consumer Expectations) is used in the Leading Indicator Composite Index published by the US Department of Commerce, Bureau of Economic Analysis. Using household telephone interviews the ICS provides an empirical measure of near-term consumer attitudes on business climate, and personal finance and spending. Variation in ICS influences price and volume in currency, bond, and equity markets in the US and in markets globally. The practice of releasing monthly ICS values five minutes to two seconds earlier for elite customers via high-speed communication channels was recently suspended because it provided unfair trading advantages. This article investigates the trajectory of the ICS over the most recent three-years, evaluating the statistical significance of month-over-month and year-over-year changes. These analyses define a longitudinal series of class variables which may be modeled temporally using time-lagged single- (UniODA) and multiple- (CTA) attribute ODA methods.

View journal article

ODA Range Test vs. One-Way Analysis of Variance: Patient Race and Lab Results

Paul R. Yarnold

Optimal Data Analysis, LLC

Mean scores on a continuous dependent measure are compared across three or more groups using one-way analysis of variance (ANOVA). If a statistically significant overall or “omnibus” effect emerges, then a multiple comparisons procedure is used to ascertain the exact nature of any interclass differences. In contrast, the dependent measure may be compared between classes with UniODA to assess if thresholds on the dependent measure can discriminate the classes. If the resulting ESS accuracy statistic for the overall effect is statistically reliable then an optimal (maximum-accuracy) range test is employed to ascertain the exact nature of interclass differences. ANOVA and UniODA are used to investigate the differences between n=377 white, n=378 African American, and n=257 Hispanic patients with HIV-associated Pneumocystis carinii pneumonia (PCP) on two laboratory tests (albumin and alveolar-arterial oxygen difference) associated with PCP outcomes.

View journal article

MegaODA Large Sample and BIG DATA Time Trials: Harvesting the Wheat

Robert C. Soltysik & Paul R. Yarnold

Optimal Data Analysis, LLC

In research involving multiple tests of statistical hypotheses the efficiency of Monte Carlo (MC) simulation used to estimate the Type I error rate (p) is maximized using a two-step procedure. The first step is identifying the effects that are not statistically significant or ns. The second step of the procedure is verifying that remaining effects are statistically significant at the generalized or experimentwise criterion (p<0.05), necessary in order to reject the null hypothesis and accept the alternative hypothesis that a statistically significant effect occurred. This research uses experimental simulation to explore the ability of MegaODA to identify p values of 0.01 and 0.001, and sample sizes of n=100,000 and n=1,000,000. Solution speeds ranged from 5 to more than 83,000 CPU seconds running MegaODA software on a 3 GHz Intel Pentium D microcomputer. Using MegaODA it is straightforward to rapidly rule-in p<0.05 for weak and moderate effects by Monte Carlo simulation with large samples and BIG DATA in designs having ordinal attributes with or without weights applied to observations. Significantly greater time was required for problems involving continuous attributes but even the most computer-intensive analyses were completed in less than a day.

View journal article

ODA Range Test vs. One-Way Analysis of Variance: Comparing Strength of Alternative Line Connections

Paul R. Yarnold & Gordon C. Brofft

Optimal Data Analysis, LLC

Among the most popular conventional statistical methods, Student’s t-test is used to compare the means of two groups on a single dependent measure assessed on a continuous scale. When three or more groups are compared, t-test is generalized to one-way analysis of variance (ANOVA). If the F statistic associated with the overall or “omnibus” effect is statistically reliable, then a range test that is more efficient than performing all possible comparisons is used to ascertain the exact nature of interclass differences. In contrast, the dependent measure may be compared between classes via UniODA, to assess if thresholds on the dependent measure separate the classes. If the resulting ESS accuracy statistic for the omnibus effect is statistically reliable, then a recently-developed optimal range test is used to assess the exact nature of interclass differences. ANOVA and UniODA are used to compare three methods commonly used in competitive big-game sport fishing for connecting segments of fishing line. Similarities and differences of parametric GLM and non-parametric ODA methods are demonstrated.

View journal article

MegaODA Large Sample and BIG DATA Time Trials: Separating the Chaff

Robert C. Soltysik & Paul R. Yarnold

Optimal Data Analysis, LLC

Just-released MegaODA™ software is capable of conducting UniODA analysis for an unlimited number of attributes using samples as large as one million observations. To minimize the computational burden associated with Monte Carlo simulation used to estimate the Type I error rate (p), the first step in statistical analysis is identifying effects that are not statistically significant or ns. This article presents an experimental simulation exploring the ability of MegaODA to identify ns effects in a host of designs involving a binary class variable, under ultimately challenging discrimination conditions (all data are random) for sample sizes of n=100,000 and n=1,000,000. Most analyses were solved in CPU seconds running MegaODA on a 3 GHz Intel Pentium D microcomputer. Using MegaODA it is straightforward to rapidly rule-out ns effects using Monte Carlo simulation with BIG DATA for large numbers of attributes in simple or complex, single- or multiple-sample designs involving categorical or ordered attributes either with or without weights being applied to individual observations.

View journal article

Creating a Data Set with SAS™ and Maximizing ESS of a Multiple Regression Analysis Model for a Likert-Type Dependent Variable Using UniODA™ and MegaODA™ Software

Paul R. Yarnold

Optimal Data Analysis, LLC

This note presents SAS™ code for creating a requisite data set and UniODA™ and MegaODA™ code for maximizing the accuracy (ESS) of a multiple regression analysis-based model involving a Likert-type dependent measure with ten or fewer response options.

View journal article

Univariate and Multivariate Analysis of Categorical Attributes with Many Response Categories

Paul R. Yarnold

Optimal Data Analysis, LLC

A scant few weeks ago disentanglement of effects identified in purely categorical designs in which all variables are categorical, including notoriously-complex rectangular categorical designs (RCDs) in which variables have a different number of response categories, was poorly understood. However, univariate and multivariate optimal (“maximum-accuracy”) statistical methods, specifically UniODA and automated CTA, make the analyses of such designs straightforward. These methods are illustrated using an example involving n=1,568 randomly selected patients having either confirmed or presumed Pneumocystis carinii pneumonia (PCP). Four categorical variables used in analysis include patient status (two categories: alive, dead), gender (male, female), city of residence (seven categories), and type of health insurance (ten categories). Examination of the cross-tabulations of these variables makes it obvious why conventional statistical methods such as chi-square analysis, logistic regression analysis, and log-linear analysis are both inappropriate for, as well as easily overwhelmed by such designs. In contrast, UniODA and CTA identified maximum-accuracy solutions effortlessly in this application.

View journal article

Analyzing Categorical Attributes Having Many Response Options

Paul R. Yarnold

Optimal Data Analysis, LLC

Rectangular categorical designs (RCDs) are among the most prevalent designs in science. Categorical designs involve categorical measures. Examples of categorical measures include gender with two response categories (male, female), color with three response categories (red, blue, green), or political affiliation with many response categories. RCDs cross-tabulate two or more categorical variables that have a different number of response options: that is, there are more columns than rows in the cross-tabulation table, or the opposite is true. Analysis via chi-square doesn’t enable researchers to disentangle effects which are identified in such designs, especially when the number of response options is large—as are typically used for attributes such as ethnicity, marital status, type of health insurance, or state of residence, for example. This article demonstrates how to disentangle effects within rectangular (or square) cross-classification tables using UniODA, and demonstrates this first for a sample of n=1,568 patients with confirmed or presumed Pneumocystis carinni pneumonia (PCP) measured on city and gender, and a second time for n=144 ESS and ESP values arising in a comparison of analysis of qualitative strength categories achieved by models identified using raw or z-score data.

View journal article

Ascertaining an Individual Patient’s Symptom Dominance Hierarchy: Analysis of Raw Longitudinal Data Induces Simpson’s Paradox

Paul R. Yarnold

Optimal Data Analysis, LLC

Examples of ordered “N-of-1″ single-case series include the monthly closing price of a corporate stock, a dieter’s weight each Friday morning upon waking, or real-time heart rate of an ICU patient recorded minute-by-minute. The present study examined a single-case series for a patient with fibromyalgia, consisting of 297 sequential daily ratings of nine symptoms commonly associated with the disease. Symptoms were compared statistically by UniODA to identify their dominance hierarchy (most to least severe) for this patient. Analysis of raw data suggested that fatigue was the dominant symptom, but analysis of ipsatively standardized data indicated that finding was paradoxically confounded: the primary symptom, in fact, was stiffness. Of 36 comparisons of symptom pairs performed using raw data, 28 (78%) were found to be confounded, ten of which (28%) identified the opposite effect (what raw data analysis suggested occurred was, in fact, the opposite of what occurred). These findings reveal a crucial difference between the meaning of the numbers and labels constituting the response scale as perceived and interpreted by the investigator, and the meaning of the numbers and labels as perceived and applied by the individual to quantify internal cognitive and emotional experiences. The latter issue is the purpose of the scale (if subject perception and response scale did not interact, then valid measurements would be impossible to obtain) and motivation underlying the research. Present findings show raw score analysis addresses the investigator’s perspective, and ipsative z-score analysis addresses the patient’s.

View journal article

How to Create a Data Set with SAS™ and Compare Attributes with UniODA™ in Serial Single-Case Designs

Paul R. Yarnold

Optimal Data Analysis, LLC

The use of UniODA to compare response distributions for different attributes has been demonstrated for binary and ordered measures, using specifically-constructed data sets. This note presents SAS™ code for creating such data sets, and UniODA™ code for comparing attributes in single-case applications.

View journal article

Comparing Responses to Dichotomous Attributes in Single-Case Designs

Paul R. Yarnold

Optimal Data Analysis, LLC

Imagine a single case is assessed on two or more binary attributes over a series of measurements or testing sessions. For example, a chronic pain patient might rate their pain and fatigue each evening before sleep, as “worse than average” or “better than average”. Alternatively, imagine following two or more different corporate stocks over a given time period, keeping a record at the end of every trading day regarding whether the closing price was higher or lower than on the prior day. Both examples are single-case series, and in both examples an interesting question is whether the binary responses to the different attributes differ. Does the patient have more bad pain days than bad fatigue days? Does one stock close higher more often than another stock? This article discusses how to accomplish such analysis and address such questions using UniODA. The example selected to illustrate the method provides clear evidence that ipsative standardization is critical in obtaining meaningful results in the analysis of serial data.

View journal article

Comparing Attributes Measured with “Identical” Likert-Type Scales in Single-Case Designs with UniODA

Paul R. Yarnold

Optimal Data Analysis, LLC

Often at the advice of their physician, patients managing chronic disease such as fibromyalgia or arthritis, or those undergoing therapy in rehabilitation medicine or oncology, will record weekly or daily—sometimes even real-time ratings—of physical (e.g., pain, fatigue) and emotional (e.g., depression, anxiety) symptoms. Similarly, often at the advice of their coaches, athletes ranging from elite professionals to everyday people engrossed in an astonishing variety of group and customized personal fitness programs, record ratings of physical (e.g., vigor, focus) and emotional (e.g., anger, fear) states at the beginning and/or the end of workout sessions. In these types of applications, within a given study (also typically in most related studies in any given discipline, and across different disciplines), physical and emotional symptoms and states are all assessed using the same type of measuring scale, specifically an ordered Likert-type scale with 3-11 response categories. This paper shows how to employ UniODA to compare such symptom or state ratings to identify the strongest and weakest symptoms/states using data obtained across multiple measurements for an individual.

View journal article

The Most Recent, Earliest, and Kth Significant Changes in an Ordered Series: Traveling Backwards in Time to Assess When Annual Crude Mortality Rate Most Recently Began Increasing in McLean County, North Dakota

Paul R. Yarnold

Optimal Data Analysis, LLC

Serial or longitudinal measurement is widely practiced by people in all facets of real-world phenomena: an athlete wishes to know how many training sessions are required before significant increase in performance is realized; a musician wishes to assess how many on-stage performances are required before a significant increase in media attention is attracted; and a medical patient wishes to understand how many units of medication are needed before significant improvement in health is achieved. This paper discusses how to use UniODA to statistically evaluate a longitudinal series, either by starting at the series beginning and traveling forwards in time, or starting at the series end and travelling backwards in time. This is illustrated with an example involving backwards travel in a time-ordered series, to determine when recently observed statistically significant increases in annual crude mortality rate occurred in McLean County, North Dakota.

View journal article

Trajectory of Crude Mortality Rate in North Dakota Counties

Paul R. Yarnold, Ph.D.

Optimal Data Analysis, LLC

Recent research reported support for the a priori hypothesis that annual crude mortality rate (ACMR) was higher after widespread commercial usage of toxic chemicals and biocides began in the environment in North Dakota in 1998. UniODA was used to compare ACMR in 1934-1997 versus 1998-2005 (the most current data available). Different county types identified included those for which ACMR increased significantly (p<0.05) at the experimentwise criterion; increased significantly (p<0.05) at the generalized criterion; had a statistically marginal increase (p<0.10) at the generalized criterion; or did not have a statistically significant increase (p>0.10) at the generalized criterion. Prior research is extended by investigating the path of ACMR over time for these four county groupings, and for each county considered separately. The pattern of mean ipsative ACMR scores across time, observed for counties experiencing a statistically significant increase in ACMR after 1998, revealed the means fell into three almost perfectly discriminable levels: (a) low mean scores (mean z<0) seen early (1965 or earlier); (b) medium scores (0<z<0.5) in the middle of the series (1966-1985); and (c) high scores (mean z>0) late in the series (1986 or later). Series achieved mean z>1 in 1998, and mean scores observed since 1998 have been among the highest on record.

View journal article