Finding the Optimal Configuration for Computerized Adaptive Testing

Computerized adaptive testing (CAT) provides  a personalized test for each examinee by administering fewer questions while adapting to each examinee’s answers. Xinyu Liu, graduate student, and David Weiss, faculty, conducted a study, “Interactions Between Termination Criteria and Ability Estimators in Computerized Adaptive Testing,” to examine the effects of ability estimators and termination criteria used by CAT on the measurement bias, precision, and length of the tests.

The pair investigated four ability estimators: maximum likelihood (MLE), weighted likelihood (WLE), maximum a posteriori (MAP), and expected a posteriori (EAP). Also, four termination criteria: fixed-length, standard error of measurement (SEM), minimum information (MI), and change-in-estimate. They evaluated various combinations across low- and high-information item banks, with different information distribution shapes. Liu and Weiss found that the optimal CAT configuration relies on both item bank size and shape. Also, the optimal CAT design is not a one-size-fits-all solution because of the variation within the combinations. Liu and Weiss recommended pairing WLE with an SEM or fixed-length termination rule when using high-quality banks, and adopting a change-in-estimate rule or hybrid SEM approach when dealing with lower-quality banks.