Robust Automated Test Assembly for Testlet-Based Tests: An Illustration With the Analytical Reasoning Section of the LSAT

Bernard P. Veldkamp, Muirne Paap

Research output: Book/ReportReportProfessional

14 Downloads (Pure)

Abstract

In many high-stakes tests, subsets of questions (i.e., items) grouped around a common stimulus are often utilized to increase testing efficiency. These subsets of items are commonly called testlets. Since responses to items belonging to the same testlet not only depend on the test taker’s ability, but also on the correct reading, understanding, and interpretation of the stimulus, the assumption that the responses to these items are independent of one another does not always hold. A mathematical model called item response theory is often applied in automated test assembly (ATA) with testlets. Testlet response theory (TRT) models have been developed to deal with dependency among items within a testlet. This report addresses some of the questions that arise in the application of TRT models to ATA. Specifically, a robust ATA method is applied. The results obtained by this method, as well as the advantages it offers, are discussed. Finally, recommendations about the use of the new method are given.
Original languageEnglish
PublisherLSAC
Number of pages20
Publication statusPublished - 2013

Publication series

NameLSAC research reports
PublisherLSAC
No.RR 13-02

Fingerprint

model theory
stimulus
efficiency
interpretation
ability

Keywords

  • METIS-301209
  • IR-88753

Cite this

@book{d4ff1e63cff44806b84fd515585c5042,
title = "Robust Automated Test Assembly for Testlet-Based Tests: An Illustration With the Analytical Reasoning Section of the LSAT",
abstract = "In many high-stakes tests, subsets of questions (i.e., items) grouped around a common stimulus are often utilized to increase testing efficiency. These subsets of items are commonly called testlets. Since responses to items belonging to the same testlet not only depend on the test taker’s ability, but also on the correct reading, understanding, and interpretation of the stimulus, the assumption that the responses to these items are independent of one another does not always hold. A mathematical model called item response theory is often applied in automated test assembly (ATA) with testlets. Testlet response theory (TRT) models have been developed to deal with dependency among items within a testlet. This report addresses some of the questions that arise in the application of TRT models to ATA. Specifically, a robust ATA method is applied. The results obtained by this method, as well as the advantages it offers, are discussed. Finally, recommendations about the use of the new method are given.",
keywords = "METIS-301209, IR-88753",
author = "Veldkamp, {Bernard P.} and Muirne Paap",
year = "2013",
language = "English",
series = "LSAC research reports",
publisher = "LSAC",
number = "RR 13-02",

}

Robust Automated Test Assembly for Testlet-Based Tests: An Illustration With the Analytical Reasoning Section of the LSAT. / Veldkamp, Bernard P.; Paap, Muirne.

LSAC, 2013. 20 p. (LSAC research reports; No. RR 13-02).

Research output: Book/ReportReportProfessional

TY - BOOK

T1 - Robust Automated Test Assembly for Testlet-Based Tests: An Illustration With the Analytical Reasoning Section of the LSAT

AU - Veldkamp, Bernard P.

AU - Paap, Muirne

PY - 2013

Y1 - 2013

N2 - In many high-stakes tests, subsets of questions (i.e., items) grouped around a common stimulus are often utilized to increase testing efficiency. These subsets of items are commonly called testlets. Since responses to items belonging to the same testlet not only depend on the test taker’s ability, but also on the correct reading, understanding, and interpretation of the stimulus, the assumption that the responses to these items are independent of one another does not always hold. A mathematical model called item response theory is often applied in automated test assembly (ATA) with testlets. Testlet response theory (TRT) models have been developed to deal with dependency among items within a testlet. This report addresses some of the questions that arise in the application of TRT models to ATA. Specifically, a robust ATA method is applied. The results obtained by this method, as well as the advantages it offers, are discussed. Finally, recommendations about the use of the new method are given.

AB - In many high-stakes tests, subsets of questions (i.e., items) grouped around a common stimulus are often utilized to increase testing efficiency. These subsets of items are commonly called testlets. Since responses to items belonging to the same testlet not only depend on the test taker’s ability, but also on the correct reading, understanding, and interpretation of the stimulus, the assumption that the responses to these items are independent of one another does not always hold. A mathematical model called item response theory is often applied in automated test assembly (ATA) with testlets. Testlet response theory (TRT) models have been developed to deal with dependency among items within a testlet. This report addresses some of the questions that arise in the application of TRT models to ATA. Specifically, a robust ATA method is applied. The results obtained by this method, as well as the advantages it offers, are discussed. Finally, recommendations about the use of the new method are given.

KW - METIS-301209

KW - IR-88753

M3 - Report

T3 - LSAC research reports

BT - Robust Automated Test Assembly for Testlet-Based Tests: An Illustration With the Analytical Reasoning Section of the LSAT

PB - LSAC

ER -