Quality control of on-line calibration in computerized assessment

Research output: Book/ReportReportProfessional

8 Downloads (Pure)

Abstract

In computerized adaptive testing, updating parameter estimates using adaptive testing data is often called online calibration. In this paper, how to evaluate whether the adaptive testing model used for online calibration fits the item response model used sufficiently is studied. Three approaches are investigated, based on a Lagrange multiplier (LM) statistic (J. Aitchison and S. Silvey, 1958), a Wald statistic, and a cumulative sum (CUMSUM) statistic (W. Veerkamp, 1996). The power of the tests was evaluated with a number of simulation studies. The theoretical advantage of the CUMSUM procedure was that it is based on a directional hypothesis and can be used iteratively. The power of the procedures ranged from rather moderate to good, depending on the change. It was also found that all three tests were equally sensitive to changes in item difficulty and the guessing parameter. All these statistics detected that something has happened to the parameters, but it is very difficult to attribute misfit to specific parameters with these methods.
Original languageEnglish
Place of PublicationEnschede
PublisherUniversiteit Twente TO/OMD
Publication statusPublished - 1998

Publication series

NameOMD Research Report
PublisherUniversity of Twente, Faculty of Educational Science and Technology
No.98-03

Fingerprint

quality control
statistics
Lagrange multipliers
estimates
simulation

Keywords

  • Simulation
  • Item Response Theory
  • Quality Control
  • Test Items
  • Power (Statistics)
  • IR-103761
  • Computer Assisted Testing
  • Adaptive Testing
  • Models
  • Foreign Countries
  • METIS-136528
  • Online Systems

Cite this

Glas, C. A. W. (1998). Quality control of on-line calibration in computerized assessment. (OMD Research Report; No. 98-03). Enschede: Universiteit Twente TO/OMD.
Glas, Cornelis A.W. / Quality control of on-line calibration in computerized assessment. Enschede : Universiteit Twente TO/OMD, 1998. (OMD Research Report; 98-03).
@book{54ac0e04eb9949cba7c6b2fc774ac516,
title = "Quality control of on-line calibration in computerized assessment",
abstract = "In computerized adaptive testing, updating parameter estimates using adaptive testing data is often called online calibration. In this paper, how to evaluate whether the adaptive testing model used for online calibration fits the item response model used sufficiently is studied. Three approaches are investigated, based on a Lagrange multiplier (LM) statistic (J. Aitchison and S. Silvey, 1958), a Wald statistic, and a cumulative sum (CUMSUM) statistic (W. Veerkamp, 1996). The power of the tests was evaluated with a number of simulation studies. The theoretical advantage of the CUMSUM procedure was that it is based on a directional hypothesis and can be used iteratively. The power of the procedures ranged from rather moderate to good, depending on the change. It was also found that all three tests were equally sensitive to changes in item difficulty and the guessing parameter. All these statistics detected that something has happened to the parameters, but it is very difficult to attribute misfit to specific parameters with these methods.",
keywords = "Simulation, Item Response Theory, Quality Control, Test Items, Power (Statistics), IR-103761, Computer Assisted Testing, Adaptive Testing, Models, Foreign Countries, METIS-136528, Online Systems",
author = "Glas, {Cornelis A.W.}",
year = "1998",
language = "English",
series = "OMD Research Report",
publisher = "Universiteit Twente TO/OMD",
number = "98-03",

}

Glas, CAW 1998, Quality control of on-line calibration in computerized assessment. OMD Research Report, no. 98-03, Universiteit Twente TO/OMD, Enschede.

Quality control of on-line calibration in computerized assessment. / Glas, Cornelis A.W.

Enschede : Universiteit Twente TO/OMD, 1998. (OMD Research Report; No. 98-03).

Research output: Book/ReportReportProfessional

TY - BOOK

T1 - Quality control of on-line calibration in computerized assessment

AU - Glas, Cornelis A.W.

PY - 1998

Y1 - 1998

N2 - In computerized adaptive testing, updating parameter estimates using adaptive testing data is often called online calibration. In this paper, how to evaluate whether the adaptive testing model used for online calibration fits the item response model used sufficiently is studied. Three approaches are investigated, based on a Lagrange multiplier (LM) statistic (J. Aitchison and S. Silvey, 1958), a Wald statistic, and a cumulative sum (CUMSUM) statistic (W. Veerkamp, 1996). The power of the tests was evaluated with a number of simulation studies. The theoretical advantage of the CUMSUM procedure was that it is based on a directional hypothesis and can be used iteratively. The power of the procedures ranged from rather moderate to good, depending on the change. It was also found that all three tests were equally sensitive to changes in item difficulty and the guessing parameter. All these statistics detected that something has happened to the parameters, but it is very difficult to attribute misfit to specific parameters with these methods.

AB - In computerized adaptive testing, updating parameter estimates using adaptive testing data is often called online calibration. In this paper, how to evaluate whether the adaptive testing model used for online calibration fits the item response model used sufficiently is studied. Three approaches are investigated, based on a Lagrange multiplier (LM) statistic (J. Aitchison and S. Silvey, 1958), a Wald statistic, and a cumulative sum (CUMSUM) statistic (W. Veerkamp, 1996). The power of the tests was evaluated with a number of simulation studies. The theoretical advantage of the CUMSUM procedure was that it is based on a directional hypothesis and can be used iteratively. The power of the procedures ranged from rather moderate to good, depending on the change. It was also found that all three tests were equally sensitive to changes in item difficulty and the guessing parameter. All these statistics detected that something has happened to the parameters, but it is very difficult to attribute misfit to specific parameters with these methods.

KW - Simulation

KW - Item Response Theory

KW - Quality Control

KW - Test Items

KW - Power (Statistics)

KW - IR-103761

KW - Computer Assisted Testing

KW - Adaptive Testing

KW - Models

KW - Foreign Countries

KW - METIS-136528

KW - Online Systems

M3 - Report

T3 - OMD Research Report

BT - Quality control of on-line calibration in computerized assessment

PB - Universiteit Twente TO/OMD

CY - Enschede

ER -

Glas CAW. Quality control of on-line calibration in computerized assessment. Enschede: Universiteit Twente TO/OMD, 1998. (OMD Research Report; 98-03).