A framework for designing and developing multimedia-based performance assessment in vocational education

Sebastiaan de Klerk (Corresponding Author), Bernard P. Veldkamp, Theo J.H.M. Eggen

Research output: Contribution to journalArticleAcademicpeer-review

4 Citations (Scopus)
46 Downloads (Pure)

Abstract

The development of any assessment should be an iterative and careful process. Ideally, this process is guided by a well-defined framework (see for example Downing in: Downing and Haladyna (eds) Handbook of test development, Lawrence Erlbaum Associates, Mahwah, 2006; Mislevy et al. in On the roles of task model variables in assessment design (CSE Technical Report 500), Educational Testing Service, Princeton, 1999; AERA et al. in Standards for educational and psychological testing, AERA, Washington, DC, 2004), but such a framework is not always available when the instrument to be developed is new or innovative. Frameworks for the development of traditional computer-based tests have been published and experimented with since the late 1990s, by which time CBT had already existed for more than a decade. In an earlier empirical pilot study, we described a new type of assessment for Dutch vocational education, called multimedia-based performance assessment (MBPA) (self-revealing reference 2014). This CBT uses multiple media formats and interactive tasks to measure skills that are currently measured by performance-based assessment. In conducting that pilot study, deficits in the existing literature made it difficult to ground all developmental steps in sound scientific theory. To remedy those deficits, this article presents and validates a framework for the design and development of MBPA, combining a search of the relevant literature from several subfields of educational assessment and consultation with assessment experts. The framework unites assessment development and multimedia development theory, focus solely on vocational education, and answers the call for a framework from the scientific community. The first step in validating the prototype framework involved five semi-structured interviews with Dutch assessment and multimedia experts to produce a final version of the framework. Second, the pilot MBPA was reconstructed in accordance with this finalized framework, resulting in an improved MBPA and demonstrating that the proposed framework is a useful and applicable tool for the design and development of MBPA in vocational education.

Original languageEnglish
Pages (from-to)147-171
Number of pages25
JournalEducational technology research and development
Volume66
Issue number1
Early online date20 Dec 2017
DOIs
Publication statusPublished - 1 Feb 2018

Fingerprint

performance assessment
Vocational Education
multimedia
Education
Testing
Acoustic waves
deficit
expert
self-reference
development theory
scientific community
remedies
interview

Keywords

  • UT-Hybrid-D
  • Assessment development
  • Computer-based assessment
  • Multimedia-based performance assessment
  • Vocational education
  • Assessment design

Cite this

@article{457ac5aee26d47b08bb9d2f34d37c27c,
title = "A framework for designing and developing multimedia-based performance assessment in vocational education",
abstract = "The development of any assessment should be an iterative and careful process. Ideally, this process is guided by a well-defined framework (see for example Downing in: Downing and Haladyna (eds) Handbook of test development, Lawrence Erlbaum Associates, Mahwah, 2006; Mislevy et al. in On the roles of task model variables in assessment design (CSE Technical Report 500), Educational Testing Service, Princeton, 1999; AERA et al. in Standards for educational and psychological testing, AERA, Washington, DC, 2004), but such a framework is not always available when the instrument to be developed is new or innovative. Frameworks for the development of traditional computer-based tests have been published and experimented with since the late 1990s, by which time CBT had already existed for more than a decade. In an earlier empirical pilot study, we described a new type of assessment for Dutch vocational education, called multimedia-based performance assessment (MBPA) (self-revealing reference 2014). This CBT uses multiple media formats and interactive tasks to measure skills that are currently measured by performance-based assessment. In conducting that pilot study, deficits in the existing literature made it difficult to ground all developmental steps in sound scientific theory. To remedy those deficits, this article presents and validates a framework for the design and development of MBPA, combining a search of the relevant literature from several subfields of educational assessment and consultation with assessment experts. The framework unites assessment development and multimedia development theory, focus solely on vocational education, and answers the call for a framework from the scientific community. The first step in validating the prototype framework involved five semi-structured interviews with Dutch assessment and multimedia experts to produce a final version of the framework. Second, the pilot MBPA was reconstructed in accordance with this finalized framework, resulting in an improved MBPA and demonstrating that the proposed framework is a useful and applicable tool for the design and development of MBPA in vocational education.",
keywords = "UT-Hybrid-D, Assessment development, Computer-based assessment, Multimedia-based performance assessment, Vocational education, Assessment design",
author = "{de Klerk}, Sebastiaan and Veldkamp, {Bernard P.} and Eggen, {Theo J.H.M.}",
note = "Springer deal",
year = "2018",
month = "2",
day = "1",
doi = "10.1007/s11423-017-9559-5",
language = "English",
volume = "66",
pages = "147--171",
journal = "Educational technology research and development",
issn = "1042-1629",
publisher = "Springer",
number = "1",

}

A framework for designing and developing multimedia-based performance assessment in vocational education. / de Klerk, Sebastiaan (Corresponding Author); Veldkamp, Bernard P.; Eggen, Theo J.H.M.

In: Educational technology research and development, Vol. 66, No. 1, 01.02.2018, p. 147-171.

Research output: Contribution to journalArticleAcademicpeer-review

TY - JOUR

T1 - A framework for designing and developing multimedia-based performance assessment in vocational education

AU - de Klerk, Sebastiaan

AU - Veldkamp, Bernard P.

AU - Eggen, Theo J.H.M.

N1 - Springer deal

PY - 2018/2/1

Y1 - 2018/2/1

N2 - The development of any assessment should be an iterative and careful process. Ideally, this process is guided by a well-defined framework (see for example Downing in: Downing and Haladyna (eds) Handbook of test development, Lawrence Erlbaum Associates, Mahwah, 2006; Mislevy et al. in On the roles of task model variables in assessment design (CSE Technical Report 500), Educational Testing Service, Princeton, 1999; AERA et al. in Standards for educational and psychological testing, AERA, Washington, DC, 2004), but such a framework is not always available when the instrument to be developed is new or innovative. Frameworks for the development of traditional computer-based tests have been published and experimented with since the late 1990s, by which time CBT had already existed for more than a decade. In an earlier empirical pilot study, we described a new type of assessment for Dutch vocational education, called multimedia-based performance assessment (MBPA) (self-revealing reference 2014). This CBT uses multiple media formats and interactive tasks to measure skills that are currently measured by performance-based assessment. In conducting that pilot study, deficits in the existing literature made it difficult to ground all developmental steps in sound scientific theory. To remedy those deficits, this article presents and validates a framework for the design and development of MBPA, combining a search of the relevant literature from several subfields of educational assessment and consultation with assessment experts. The framework unites assessment development and multimedia development theory, focus solely on vocational education, and answers the call for a framework from the scientific community. The first step in validating the prototype framework involved five semi-structured interviews with Dutch assessment and multimedia experts to produce a final version of the framework. Second, the pilot MBPA was reconstructed in accordance with this finalized framework, resulting in an improved MBPA and demonstrating that the proposed framework is a useful and applicable tool for the design and development of MBPA in vocational education.

AB - The development of any assessment should be an iterative and careful process. Ideally, this process is guided by a well-defined framework (see for example Downing in: Downing and Haladyna (eds) Handbook of test development, Lawrence Erlbaum Associates, Mahwah, 2006; Mislevy et al. in On the roles of task model variables in assessment design (CSE Technical Report 500), Educational Testing Service, Princeton, 1999; AERA et al. in Standards for educational and psychological testing, AERA, Washington, DC, 2004), but such a framework is not always available when the instrument to be developed is new or innovative. Frameworks for the development of traditional computer-based tests have been published and experimented with since the late 1990s, by which time CBT had already existed for more than a decade. In an earlier empirical pilot study, we described a new type of assessment for Dutch vocational education, called multimedia-based performance assessment (MBPA) (self-revealing reference 2014). This CBT uses multiple media formats and interactive tasks to measure skills that are currently measured by performance-based assessment. In conducting that pilot study, deficits in the existing literature made it difficult to ground all developmental steps in sound scientific theory. To remedy those deficits, this article presents and validates a framework for the design and development of MBPA, combining a search of the relevant literature from several subfields of educational assessment and consultation with assessment experts. The framework unites assessment development and multimedia development theory, focus solely on vocational education, and answers the call for a framework from the scientific community. The first step in validating the prototype framework involved five semi-structured interviews with Dutch assessment and multimedia experts to produce a final version of the framework. Second, the pilot MBPA was reconstructed in accordance with this finalized framework, resulting in an improved MBPA and demonstrating that the proposed framework is a useful and applicable tool for the design and development of MBPA in vocational education.

KW - UT-Hybrid-D

KW - Assessment development

KW - Computer-based assessment

KW - Multimedia-based performance assessment

KW - Vocational education

KW - Assessment design

UR - http://www.scopus.com/inward/record.url?scp=85038619033&partnerID=8YFLogxK

U2 - 10.1007/s11423-017-9559-5

DO - 10.1007/s11423-017-9559-5

M3 - Article

VL - 66

SP - 147

EP - 171

JO - Educational technology research and development

JF - Educational technology research and development

SN - 1042-1629

IS - 1

ER -