All about validity: an evaluation system for the quality of educational assessment

At all levels of education, tests and assessments are used to gather information about students’ skills and competences. This information can be used for decisions about groups of students or about individual students. When the stakes of these decisions are high, it becomes increasingly important that the assessments used are of good quality. In this dissertation, assessment quality is defined as the degree to which an assessment instrument is appropriate for its intended purpose. This definition entails a flexible view of quality: dependent on the intended interpretation and use of test scores, different quality criteria should be used for evaluation. This approach is also described as an argument-based approach to quality. It originates in validity theory, and our research project aimed to extend it to quality. This is done in a design-based research project whose aim was to develop an evaluation system, which incorporates an argument-based approach to quality, for the quality of educational assessments. In this design-based research project, a theoretical foundation is developed. This theoretical foundation describes the argument-based approach to quality for a single test. It also describes this approach in a context of an assessment program where multiple assessment instances are combined into one decision. From this theoretical foundation, design principles are formulated and then used to build a prototype of the evaluation system. This prototype is then evaluated and adjusted in an iterative process. This dissertation describes the theoretical foundation, the design principles, and several iterations in the development of the prototype.