Detecting intrajudge inconsistency in standard setting using test items with a selected-response format

Willem J. van der Linden, Hendrik J. Vos, Lei Chang

Research output: Book/ReportReportProfessional

17 Downloads (Pure)

Abstract

In judgmental standard setting experiments, it may be difficult to specify subjective probabilities that adequately take the properties of the items into account. As a result, these probabilities are not consistent with each other in the sense that they do not refer to the same borderline level of performance. Methods to check standard setting data for intrajudge inconsistencies are thus of paramount importance to setting meaningful standards. This paper presents a method of consistency analysis for standard setting experiments in which judges specify probabilities for each response alternative of the items. The method is based on a residual diagnosis of the subjective probabilities under the hypothesis of a consistent judge tothe probabilities. An empirical example shows how the method can be used to identify sources of inconsistency in response alternatives, items, or judges.
Original languageEnglish
Place of PublicationEnschede
PublisherUniversity of Twente, Faculty Educational Science and Technology
Number of pages19
Publication statusPublished - 2002

Publication series

NameOMD Research Report
PublisherUniversity of Twente, Faculty Educational Science and Technology
No.02-01

Keywords

  • IR-103552
  • METIS-209497

Fingerprint Dive into the research topics of 'Detecting intrajudge inconsistency in standard setting using test items with a selected-response format'. Together they form a unique fingerprint.

Cite this