Quality control of on-line calibration in computerized assessment

In computerized adaptive testing, updating parameter estimates using adaptive testing data is often called online calibration. In this paper, how to evaluate whether the adaptive testing model used for online calibration fits the item response model used sufficiently is studied. Three approaches are investigated, based on a Lagrange multiplier (LM) statistic (J. Aitchison and S. Silvey, 1958), a Wald statistic, and a cumulative sum (CUMSUM) statistic (W. Veerkamp, 1996). The power of the tests was evaluated with a number of simulation studies. The theoretical advantage of the CUMSUM procedure was that it is based on a directional hypothesis and can be used iteratively. The power of the procedures ranged from rather moderate to good, depending on the change. It was also found that all three tests were equally sensitive to changes in item difficulty and the guessing parameter. All these statistics detected that something has happened to the parameters, but it is very difficult to attribute misfit to specific parameters with these methods.