A comparison of two procedures for constrained adaptive test construction

Frédéric Robin, Willem J. van der Linden, Daniel R. Eignor, Manfred Steffen, Martha L. Stocking

Research output: Book/ReportReportProfessional

45 Downloads (Pure)

Abstract

The relatively new shadow test approach (STA) to computerized adaptive testing (CAT) proposed by Wim van der Linden is a potentially attractive alternative to the weighted deviation algorithm (WDA) implemented at ETS. However, it has not been evaluated under testing conditions representative of current ETS testing programs. Of interest was whether STA would, under typical high-stakes on-demand testing situations, produce tests of comparable or better psychometric quality as those produced by the current weighted deviation algorithm. Based on simulated data, we found that the STA performed as well or slightly better than the WDA on two of the three commonly accepted testing objectives: measurement and content. The WDA appeared to perform slightly better than the STA when the issue is security or item exposure control. The paper provides a review of the rationale that led to the specific testing objectives employed, an outline of the test construction steps common to the two procedures investigated, and a description of the specific models and algorithms employed by both procedures. Detailed description of the simulation study conducted and the results obtained from both of the procedures for one of the pools are also provided. Results are summarized and further research needs in particular areas discussed.
Original languageEnglish
Place of PublicationPrinceton, New Jersey
PublisherEducational Testing Service
Publication statusPublished - 2004

Publication series

NameETS research report
PublisherUniversity of Twente, Faculty of Educational Science and Technology
No.04-39

Fingerprint

Testing
Exposure controls

Keywords

  • METIS-219564
  • test security
  • IR-103559
  • Automated test assembly
  • Computerized Adaptive Testing
  • Item Response Theory
  • item exposure

Cite this

Robin, F., van der Linden, W. J., Eignor, D. R., Steffen, M., & Stocking, M. L. (2004). A comparison of two procedures for constrained adaptive test construction. (ETS research report; No. 04-39). Princeton, New Jersey: Educational Testing Service.
Robin, Frédéric ; van der Linden, Willem J. ; Eignor, Daniel R. ; Steffen, Manfred ; Stocking, Martha L. / A comparison of two procedures for constrained adaptive test construction. Princeton, New Jersey : Educational Testing Service, 2004. (ETS research report; 04-39).
@book{53b8bc7433464f04b044933c52b7248a,
title = "A comparison of two procedures for constrained adaptive test construction",
abstract = "The relatively new shadow test approach (STA) to computerized adaptive testing (CAT) proposed by Wim van der Linden is a potentially attractive alternative to the weighted deviation algorithm (WDA) implemented at ETS. However, it has not been evaluated under testing conditions representative of current ETS testing programs. Of interest was whether STA would, under typical high-stakes on-demand testing situations, produce tests of comparable or better psychometric quality as those produced by the current weighted deviation algorithm. Based on simulated data, we found that the STA performed as well or slightly better than the WDA on two of the three commonly accepted testing objectives: measurement and content. The WDA appeared to perform slightly better than the STA when the issue is security or item exposure control. The paper provides a review of the rationale that led to the specific testing objectives employed, an outline of the test construction steps common to the two procedures investigated, and a description of the specific models and algorithms employed by both procedures. Detailed description of the simulation study conducted and the results obtained from both of the procedures for one of the pools are also provided. Results are summarized and further research needs in particular areas discussed.",
keywords = "METIS-219564, test security, IR-103559, Automated test assembly, Computerized Adaptive Testing, Item Response Theory, item exposure",
author = "Fr{\'e}d{\'e}ric Robin and {van der Linden}, {Willem J.} and Eignor, {Daniel R.} and Manfred Steffen and Stocking, {Martha L.}",
year = "2004",
language = "English",
series = "ETS research report",
publisher = "Educational Testing Service",
number = "04-39",

}

Robin, F, van der Linden, WJ, Eignor, DR, Steffen, M & Stocking, ML 2004, A comparison of two procedures for constrained adaptive test construction. ETS research report, no. 04-39, Educational Testing Service, Princeton, New Jersey.

A comparison of two procedures for constrained adaptive test construction. / Robin, Frédéric; van der Linden, Willem J.; Eignor, Daniel R.; Steffen, Manfred; Stocking, Martha L.

Princeton, New Jersey : Educational Testing Service, 2004. (ETS research report; No. 04-39).

Research output: Book/ReportReportProfessional

TY - BOOK

T1 - A comparison of two procedures for constrained adaptive test construction

AU - Robin, Frédéric

AU - van der Linden, Willem J.

AU - Eignor, Daniel R.

AU - Steffen, Manfred

AU - Stocking, Martha L.

PY - 2004

Y1 - 2004

N2 - The relatively new shadow test approach (STA) to computerized adaptive testing (CAT) proposed by Wim van der Linden is a potentially attractive alternative to the weighted deviation algorithm (WDA) implemented at ETS. However, it has not been evaluated under testing conditions representative of current ETS testing programs. Of interest was whether STA would, under typical high-stakes on-demand testing situations, produce tests of comparable or better psychometric quality as those produced by the current weighted deviation algorithm. Based on simulated data, we found that the STA performed as well or slightly better than the WDA on two of the three commonly accepted testing objectives: measurement and content. The WDA appeared to perform slightly better than the STA when the issue is security or item exposure control. The paper provides a review of the rationale that led to the specific testing objectives employed, an outline of the test construction steps common to the two procedures investigated, and a description of the specific models and algorithms employed by both procedures. Detailed description of the simulation study conducted and the results obtained from both of the procedures for one of the pools are also provided. Results are summarized and further research needs in particular areas discussed.

AB - The relatively new shadow test approach (STA) to computerized adaptive testing (CAT) proposed by Wim van der Linden is a potentially attractive alternative to the weighted deviation algorithm (WDA) implemented at ETS. However, it has not been evaluated under testing conditions representative of current ETS testing programs. Of interest was whether STA would, under typical high-stakes on-demand testing situations, produce tests of comparable or better psychometric quality as those produced by the current weighted deviation algorithm. Based on simulated data, we found that the STA performed as well or slightly better than the WDA on two of the three commonly accepted testing objectives: measurement and content. The WDA appeared to perform slightly better than the STA when the issue is security or item exposure control. The paper provides a review of the rationale that led to the specific testing objectives employed, an outline of the test construction steps common to the two procedures investigated, and a description of the specific models and algorithms employed by both procedures. Detailed description of the simulation study conducted and the results obtained from both of the procedures for one of the pools are also provided. Results are summarized and further research needs in particular areas discussed.

KW - METIS-219564

KW - test security

KW - IR-103559

KW - Automated test assembly

KW - Computerized Adaptive Testing

KW - Item Response Theory

KW - item exposure

M3 - Report

T3 - ETS research report

BT - A comparison of two procedures for constrained adaptive test construction

PB - Educational Testing Service

CY - Princeton, New Jersey

ER -

Robin F, van der Linden WJ, Eignor DR, Steffen M, Stocking ML. A comparison of two procedures for constrained adaptive test construction. Princeton, New Jersey: Educational Testing Service, 2004. (ETS research report; 04-39).