Small N, high stakes: Simulating adverse impact with Bayesian logistic regression analysis and permutation tests

Organizations frequently evaluate adverse impact with rules of thumb and statistical tests that perform poorly when hiring cohorts are small, raising scientific concerns that have legal and ethical implications. To address these concerns, this study conducts Monte Carlo simulations to evaluate Bayesian logistic regression analysis and permutation tests as two new approaches to conducting adverse impact analysis under small-sample conditions. Simulations were varied across a realistic range of effect sizes, selection rates, and group sample sizes. The performance of each method was evaluated in terms of Type I error rates and statistical power. Results indicated that both Bayesian logistic regression analysis and permutation tests are viable alternatives to traditional methods under small-sample conditions. The Bayesian logistic regressions in particular demonstrated higher power than frequentist methods, but lower Type I error rates than the 4/5ths rule. Implications and advantages of each method are discussed in detail.

See the full article at Organizational Research Methods