The Psychometric Evaluation of a Summative Multimedia-Based Performance Assessment

Research output: Chapter in Book/Report/Conference proceedingChapterAcademicpeer-review

2 Citations (Scopus)

Abstract

In this article, a case study on the design, development, and evaluation of a multimedia-based performance assessment (MBPA) for measuring confined space guards’ skills is presented. A confined space guard (CSG) supervises operations that are carried out in a confined space (e.g. a tank or silo). Currently, individuals who want to become a certified CSG in The Netherlands have to participate in a one day training program and have to pass both a knowledge-based MC test and a practice-based performance-based assessment (PBA). Our goal is to measure the skills that are currently being assessed through the PBA, with the MBPA. We first discuss the design and development of the MBPA. Secondly, we present an empirical study which was used for assessing the quality of our measurement instrument. A representative sample of 55 CSG students, who had just completed the one day training program, has subsequently performed in the MC test, and then, depending on the condition they were assigned, the PBA or the MBPA. We report the psychometric properties of the MBPA. Furthermore, using correlations and regression analysis, we make an empirical comparison between students’ scores on the PBA and the MBPA. The results show that students’ scores on the PBA and the MBPA are significantly correlated and that students’ MBPA score is a good predictor for their score on the PBA. In the discussion, we provide implications and directions for future research and practice into the field of MBPA.
Original languageEnglish
Title of host publicationComputer Assisted Assessment. Research into E-Assessment: 18th International Conference, CAA 2015, Zeist, The Netherlands, June 22–23, 2015. Proceedings
EditorsEric Ras, Desiree Joosten-ten Brinke
PublisherSpringer
Pages1-11
ISBN (Print)978-3-319-27703-5
DOIs
Publication statusPublished - 2015

Publication series

NameCommunications in Computer and Information Science
PublisherSpringer
Number571

Fingerprint

performance assessment
psychometrics
multimedia
evaluation
performance
training program
student
measurement method
regression analysis
Netherlands

Keywords

  • IR-98998
  • METIS-314540

Cite this

de Klerk, S., Veldkamp, B. P., & Eggen, T. J. H. M. (2015). The Psychometric Evaluation of a Summative Multimedia-Based Performance Assessment. In E. Ras, & D. Joosten-ten Brinke (Eds.), Computer Assisted Assessment. Research into E-Assessment: 18th International Conference, CAA 2015, Zeist, The Netherlands, June 22–23, 2015. Proceedings (pp. 1-11). (Communications in Computer and Information Science; No. 571). Springer. https://doi.org/10.1007/978-3-319-27704-2_1
de Klerk, Sebastiaan ; Veldkamp, Bernard P. ; Eggen, Theodorus Johannes Hendrikus Maria. / The Psychometric Evaluation of a Summative Multimedia-Based Performance Assessment. Computer Assisted Assessment. Research into E-Assessment: 18th International Conference, CAA 2015, Zeist, The Netherlands, June 22–23, 2015. Proceedings. editor / Eric Ras ; Desiree Joosten-ten Brinke. Springer, 2015. pp. 1-11 (Communications in Computer and Information Science; 571).
@inbook{e47f995445324a16aa8088552538a812,
title = "The Psychometric Evaluation of a Summative Multimedia-Based Performance Assessment",
abstract = "In this article, a case study on the design, development, and evaluation of a multimedia-based performance assessment (MBPA) for measuring confined space guards’ skills is presented. A confined space guard (CSG) supervises operations that are carried out in a confined space (e.g. a tank or silo). Currently, individuals who want to become a certified CSG in The Netherlands have to participate in a one day training program and have to pass both a knowledge-based MC test and a practice-based performance-based assessment (PBA). Our goal is to measure the skills that are currently being assessed through the PBA, with the MBPA. We first discuss the design and development of the MBPA. Secondly, we present an empirical study which was used for assessing the quality of our measurement instrument. A representative sample of 55 CSG students, who had just completed the one day training program, has subsequently performed in the MC test, and then, depending on the condition they were assigned, the PBA or the MBPA. We report the psychometric properties of the MBPA. Furthermore, using correlations and regression analysis, we make an empirical comparison between students’ scores on the PBA and the MBPA. The results show that students’ scores on the PBA and the MBPA are significantly correlated and that students’ MBPA score is a good predictor for their score on the PBA. In the discussion, we provide implications and directions for future research and practice into the field of MBPA.",
keywords = "IR-98998, METIS-314540",
author = "{de Klerk}, Sebastiaan and Veldkamp, {Bernard P.} and Eggen, {Theodorus Johannes Hendrikus Maria}",
year = "2015",
doi = "10.1007/978-3-319-27704-2_1",
language = "English",
isbn = "978-3-319-27703-5",
series = "Communications in Computer and Information Science",
publisher = "Springer",
number = "571",
pages = "1--11",
editor = "Eric Ras and {Joosten-ten Brinke}, Desiree",
booktitle = "Computer Assisted Assessment. Research into E-Assessment: 18th International Conference, CAA 2015, Zeist, The Netherlands, June 22–23, 2015. Proceedings",

}

de Klerk, S, Veldkamp, BP & Eggen, TJHM 2015, The Psychometric Evaluation of a Summative Multimedia-Based Performance Assessment. in E Ras & D Joosten-ten Brinke (eds), Computer Assisted Assessment. Research into E-Assessment: 18th International Conference, CAA 2015, Zeist, The Netherlands, June 22–23, 2015. Proceedings. Communications in Computer and Information Science, no. 571, Springer, pp. 1-11. https://doi.org/10.1007/978-3-319-27704-2_1

The Psychometric Evaluation of a Summative Multimedia-Based Performance Assessment. / de Klerk, Sebastiaan; Veldkamp, Bernard P.; Eggen, Theodorus Johannes Hendrikus Maria.

Computer Assisted Assessment. Research into E-Assessment: 18th International Conference, CAA 2015, Zeist, The Netherlands, June 22–23, 2015. Proceedings. ed. / Eric Ras; Desiree Joosten-ten Brinke. Springer, 2015. p. 1-11 (Communications in Computer and Information Science; No. 571).

Research output: Chapter in Book/Report/Conference proceedingChapterAcademicpeer-review

TY - CHAP

T1 - The Psychometric Evaluation of a Summative Multimedia-Based Performance Assessment

AU - de Klerk, Sebastiaan

AU - Veldkamp, Bernard P.

AU - Eggen, Theodorus Johannes Hendrikus Maria

PY - 2015

Y1 - 2015

N2 - In this article, a case study on the design, development, and evaluation of a multimedia-based performance assessment (MBPA) for measuring confined space guards’ skills is presented. A confined space guard (CSG) supervises operations that are carried out in a confined space (e.g. a tank or silo). Currently, individuals who want to become a certified CSG in The Netherlands have to participate in a one day training program and have to pass both a knowledge-based MC test and a practice-based performance-based assessment (PBA). Our goal is to measure the skills that are currently being assessed through the PBA, with the MBPA. We first discuss the design and development of the MBPA. Secondly, we present an empirical study which was used for assessing the quality of our measurement instrument. A representative sample of 55 CSG students, who had just completed the one day training program, has subsequently performed in the MC test, and then, depending on the condition they were assigned, the PBA or the MBPA. We report the psychometric properties of the MBPA. Furthermore, using correlations and regression analysis, we make an empirical comparison between students’ scores on the PBA and the MBPA. The results show that students’ scores on the PBA and the MBPA are significantly correlated and that students’ MBPA score is a good predictor for their score on the PBA. In the discussion, we provide implications and directions for future research and practice into the field of MBPA.

AB - In this article, a case study on the design, development, and evaluation of a multimedia-based performance assessment (MBPA) for measuring confined space guards’ skills is presented. A confined space guard (CSG) supervises operations that are carried out in a confined space (e.g. a tank or silo). Currently, individuals who want to become a certified CSG in The Netherlands have to participate in a one day training program and have to pass both a knowledge-based MC test and a practice-based performance-based assessment (PBA). Our goal is to measure the skills that are currently being assessed through the PBA, with the MBPA. We first discuss the design and development of the MBPA. Secondly, we present an empirical study which was used for assessing the quality of our measurement instrument. A representative sample of 55 CSG students, who had just completed the one day training program, has subsequently performed in the MC test, and then, depending on the condition they were assigned, the PBA or the MBPA. We report the psychometric properties of the MBPA. Furthermore, using correlations and regression analysis, we make an empirical comparison between students’ scores on the PBA and the MBPA. The results show that students’ scores on the PBA and the MBPA are significantly correlated and that students’ MBPA score is a good predictor for their score on the PBA. In the discussion, we provide implications and directions for future research and practice into the field of MBPA.

KW - IR-98998

KW - METIS-314540

U2 - 10.1007/978-3-319-27704-2_1

DO - 10.1007/978-3-319-27704-2_1

M3 - Chapter

SN - 978-3-319-27703-5

T3 - Communications in Computer and Information Science

SP - 1

EP - 11

BT - Computer Assisted Assessment. Research into E-Assessment: 18th International Conference, CAA 2015, Zeist, The Netherlands, June 22–23, 2015. Proceedings

A2 - Ras, Eric

A2 - Joosten-ten Brinke, Desiree

PB - Springer

ER -

de Klerk S, Veldkamp BP, Eggen TJHM. The Psychometric Evaluation of a Summative Multimedia-Based Performance Assessment. In Ras E, Joosten-ten Brinke D, editors, Computer Assisted Assessment. Research into E-Assessment: 18th International Conference, CAA 2015, Zeist, The Netherlands, June 22–23, 2015. Proceedings. Springer. 2015. p. 1-11. (Communications in Computer and Information Science; 571). https://doi.org/10.1007/978-3-319-27704-2_1