Robustness of judgments in evaluation research

Willem J. van der Linden, Michel A. Zwarts

Research output: Book/ReportReportProfessional

5 Downloads (Pure)

Abstract

It is argued that judgments in evaluative research are ultimately subjective, but that good criteria are available to assess their quality. One of these criteria is the robustness of the judgments against incompleteness or uncertainty in the data used to describe the educational system. The use of the robustness criterion is demonstrated through the case of a recent evaluation project in which the state of elementary education in The Netherlands was evaluated. To test robustness, four different procedures were simulated for item removal: (1) scaling; (2) removal of easy items; (3) removal of difficult items; and (4) removal of extreme items. The robustness study demonstrated that the qualifications used in the evaluation project were quite stable under the removal of items from the pool by these four methods. Nearly all the qualifications met the rigorous criterion of robustness. An appendix discusses the independence of the mean observed score of covariation between abilities.
Original languageEnglish
Place of PublicationEnschede
PublisherUniversity of Twente, Faculty Educational Science and Technology
Number of pages33
Publication statusPublished - 1994

Publication series

NameOMD research report
PublisherUniversity of Twente, Faculty of Educational Science and Technology
No.94-10

Fingerprint

evaluation research
qualification
elementary education
evaluation
scaling
educational system
Netherlands
uncertainty
ability

Keywords

  • METIS-136728
  • IR-103736
  • Case Studies
  • Ability
  • Evaluation Methods
  • Evaluation Research
  • Foreign Countries
  • Difficulty Level
  • Criteria
  • Robustness (Statistics)
  • Decision Making
  • Scaling
  • Program Evaluation
  • Research Problems
  • Elementary Secondary Education
  • Test Items

Cite this

van der Linden, W. J., & Zwarts, M. A. (1994). Robustness of judgments in evaluation research. (OMD research report; No. 94-10). Enschede: University of Twente, Faculty Educational Science and Technology.
van der Linden, Willem J. ; Zwarts, Michel A. / Robustness of judgments in evaluation research. Enschede : University of Twente, Faculty Educational Science and Technology, 1994. 33 p. (OMD research report; 94-10).
@book{ab1c0f11a0524949969fff07024a3f7f,
title = "Robustness of judgments in evaluation research",
abstract = "It is argued that judgments in evaluative research are ultimately subjective, but that good criteria are available to assess their quality. One of these criteria is the robustness of the judgments against incompleteness or uncertainty in the data used to describe the educational system. The use of the robustness criterion is demonstrated through the case of a recent evaluation project in which the state of elementary education in The Netherlands was evaluated. To test robustness, four different procedures were simulated for item removal: (1) scaling; (2) removal of easy items; (3) removal of difficult items; and (4) removal of extreme items. The robustness study demonstrated that the qualifications used in the evaluation project were quite stable under the removal of items from the pool by these four methods. Nearly all the qualifications met the rigorous criterion of robustness. An appendix discusses the independence of the mean observed score of covariation between abilities.",
keywords = "METIS-136728, IR-103736, Case Studies, Ability, Evaluation Methods, Evaluation Research, Foreign Countries, Difficulty Level, Criteria, Robustness (Statistics), Decision Making, Scaling, Program Evaluation, Research Problems, Elementary Secondary Education, Test Items",
author = "{van der Linden}, {Willem J.} and Zwarts, {Michel A.}",
year = "1994",
language = "English",
series = "OMD research report",
publisher = "University of Twente, Faculty Educational Science and Technology",
number = "94-10",

}

van der Linden, WJ & Zwarts, MA 1994, Robustness of judgments in evaluation research. OMD research report, no. 94-10, University of Twente, Faculty Educational Science and Technology, Enschede.

Robustness of judgments in evaluation research. / van der Linden, Willem J.; Zwarts, Michel A.

Enschede : University of Twente, Faculty Educational Science and Technology, 1994. 33 p. (OMD research report; No. 94-10).

Research output: Book/ReportReportProfessional

TY - BOOK

T1 - Robustness of judgments in evaluation research

AU - van der Linden, Willem J.

AU - Zwarts, Michel A.

PY - 1994

Y1 - 1994

N2 - It is argued that judgments in evaluative research are ultimately subjective, but that good criteria are available to assess their quality. One of these criteria is the robustness of the judgments against incompleteness or uncertainty in the data used to describe the educational system. The use of the robustness criterion is demonstrated through the case of a recent evaluation project in which the state of elementary education in The Netherlands was evaluated. To test robustness, four different procedures were simulated for item removal: (1) scaling; (2) removal of easy items; (3) removal of difficult items; and (4) removal of extreme items. The robustness study demonstrated that the qualifications used in the evaluation project were quite stable under the removal of items from the pool by these four methods. Nearly all the qualifications met the rigorous criterion of robustness. An appendix discusses the independence of the mean observed score of covariation between abilities.

AB - It is argued that judgments in evaluative research are ultimately subjective, but that good criteria are available to assess their quality. One of these criteria is the robustness of the judgments against incompleteness or uncertainty in the data used to describe the educational system. The use of the robustness criterion is demonstrated through the case of a recent evaluation project in which the state of elementary education in The Netherlands was evaluated. To test robustness, four different procedures were simulated for item removal: (1) scaling; (2) removal of easy items; (3) removal of difficult items; and (4) removal of extreme items. The robustness study demonstrated that the qualifications used in the evaluation project were quite stable under the removal of items from the pool by these four methods. Nearly all the qualifications met the rigorous criterion of robustness. An appendix discusses the independence of the mean observed score of covariation between abilities.

KW - METIS-136728

KW - IR-103736

KW - Case Studies

KW - Ability

KW - Evaluation Methods

KW - Evaluation Research

KW - Foreign Countries

KW - Difficulty Level

KW - Criteria

KW - Robustness (Statistics)

KW - Decision Making

KW - Scaling

KW - Program Evaluation

KW - Research Problems

KW - Elementary Secondary Education

KW - Test Items

M3 - Report

T3 - OMD research report

BT - Robustness of judgments in evaluation research

PB - University of Twente, Faculty Educational Science and Technology

CY - Enschede

ER -

van der Linden WJ, Zwarts MA. Robustness of judgments in evaluation research. Enschede: University of Twente, Faculty Educational Science and Technology, 1994. 33 p. (OMD research report; 94-10).