The use of person-fit statistics in computerized adaptive testing

Several person-fit statistics have been proposed to detect item score patterns that do not fit an item response theory model. To classify response patterns as not fitting a model, a distribution of a person-fit statistic is needed. Recently, the null distributions of several fit statistics have been investigated using conventional administered tests. For computerized adaptive testing (CAT), however, less is known about the distribution of fit statistics. In this study, a three-part simulation study was conducted. First, the theoretical distribution of the often used lz-statistic across θ levels in a conventional testing CAT environment was investigated, where θ and �θ were used to calculate lz. Second, two procedures for simulating the distribution of lz were examined: (1) item scores were simulated with a fixed set of administered items, and (2) item scores were generated according to a stochastic design, where the choice of the administered item i +1 depended on item i. Finally, a study was performed to evaluate the usefulness of person-fit in the CAT environment to detect aberrant response patterns. Results indicated that the distribution of lz differed from the theoretical distribution, and that simulating the sampling distribution of lz was problematic when �θ was used to determine the values of lz. As a result, the detection rates for different types of aberrant behavior were relatively low. However, it was also shown that, for some types of aberrant behavior, the error in �θ was not dramatically different from that of normal-responding simulated test takers, and that, in practice, �θ can be reasonably well estimated, even for simulated test takers not fitting the model.