Improving the modelling of response variation in international large-scale assessments

Renate Annemiek Punter

Research output: ThesisPhD Thesis - Research UT, graduation UTAcademic

48 Downloads (Pure)

Abstract

International large-scale assessments (ILSAs) play a major role in the evaluation of educational systems. These projects are characterized by the standardized assessment of student achievement and the collection of contextual data by means of curriculum, student, teacher, school, and home questionnaires. Together, the resulting high-quality data on student achievement and contextual factors provide great opportunities for more theory-oriented educational effectiveness research, particularly in international contexts. To ensure the validity of analyses based on these data, particularly relating to measurement invariance across (sub)populations, efforts must be made to evaluate response behaviour across (sub)populations of interest. A lack of measurement invariance characterized by these differences in response behaviour, is called differential item functioning (DIF).
This thesis presents five studies that contribute to research in the field of education by deploying ILSA data in research areas where the availability of standardized data from multiple countries offers new research opportunities. Topics addressed are: computer and information literacy, parental involvement and reading literacy, and language demand in testing mathematics. Also, in each chapter methods for identifying and handling potential DIF in the framework of item response theory are explored.
The studies in this thesis show how DIF analyses can be insightful by benefiting from the synergy between a methodological focus on validity and a focus on more substantive research questions. More than simply a task to tick off before the “real” questions are investigated, DIF analyses can lead to insights into effects underlying test results. Throughout the studies in this thesis it is therefore shown how, in studies with a substantive interest in comparing groups, the study of validity on both test and questionnaire items should be integrated into the methodology. Though no clear-cut one-method-fits-all strategy is presented here, the thesis shows that there are many ways to approach the issue.
Original languageEnglish
Awarding Institution
  • University of Twente
Supervisors/Advisors
  • Glas, Cees A.W., Supervisor
  • Eggen, Theo T.J.H.M., Supervisor
  • Meelissen, Martina R.M., Co-Supervisor
Award date19 Dec 2018
Place of PublicationEnschede
Publisher
Print ISBNs978-90-365-4686-7
DOIs
Publication statusPublished - 19 Dec 2018

Fingerprint

response behavior
literacy
questionnaire
data quality
synergy
educational system
student teacher
student
mathematics
curriculum
lack
demand
methodology
language
evaluation
school
education
Group

Cite this

Punter, Renate Annemiek. / Improving the modelling of response variation in international large-scale assessments. Enschede : University of Twente, 2018. 160 p.
@phdthesis{84d34bfbd38047c494f0297d717fae21,
title = "Improving the modelling of response variation in international large-scale assessments",
abstract = "International large-scale assessments (ILSAs) play a major role in the evaluation of educational systems. These projects are characterized by the standardized assessment of student achievement and the collection of contextual data by means of curriculum, student, teacher, school, and home questionnaires. Together, the resulting high-quality data on student achievement and contextual factors provide great opportunities for more theory-oriented educational effectiveness research, particularly in international contexts. To ensure the validity of analyses based on these data, particularly relating to measurement invariance across (sub)populations, efforts must be made to evaluate response behaviour across (sub)populations of interest. A lack of measurement invariance characterized by these differences in response behaviour, is called differential item functioning (DIF). This thesis presents five studies that contribute to research in the field of education by deploying ILSA data in research areas where the availability of standardized data from multiple countries offers new research opportunities. Topics addressed are: computer and information literacy, parental involvement and reading literacy, and language demand in testing mathematics. Also, in each chapter methods for identifying and handling potential DIF in the framework of item response theory are explored. The studies in this thesis show how DIF analyses can be insightful by benefiting from the synergy between a methodological focus on validity and a focus on more substantive research questions. More than simply a task to tick off before the “real” questions are investigated, DIF analyses can lead to insights into effects underlying test results. Throughout the studies in this thesis it is therefore shown how, in studies with a substantive interest in comparing groups, the study of validity on both test and questionnaire items should be integrated into the methodology. Though no clear-cut one-method-fits-all strategy is presented here, the thesis shows that there are many ways to approach the issue.",
author = "Punter, {Renate Annemiek}",
year = "2018",
month = "12",
day = "19",
doi = "10.3990/1.9789036546867",
language = "English",
isbn = "978-90-365-4686-7",
publisher = "University of Twente",
address = "Netherlands",
school = "University of Twente",

}

Improving the modelling of response variation in international large-scale assessments. / Punter, Renate Annemiek.

Enschede : University of Twente, 2018. 160 p.

Research output: ThesisPhD Thesis - Research UT, graduation UTAcademic

TY - THES

T1 - Improving the modelling of response variation in international large-scale assessments

AU - Punter, Renate Annemiek

PY - 2018/12/19

Y1 - 2018/12/19

N2 - International large-scale assessments (ILSAs) play a major role in the evaluation of educational systems. These projects are characterized by the standardized assessment of student achievement and the collection of contextual data by means of curriculum, student, teacher, school, and home questionnaires. Together, the resulting high-quality data on student achievement and contextual factors provide great opportunities for more theory-oriented educational effectiveness research, particularly in international contexts. To ensure the validity of analyses based on these data, particularly relating to measurement invariance across (sub)populations, efforts must be made to evaluate response behaviour across (sub)populations of interest. A lack of measurement invariance characterized by these differences in response behaviour, is called differential item functioning (DIF). This thesis presents five studies that contribute to research in the field of education by deploying ILSA data in research areas where the availability of standardized data from multiple countries offers new research opportunities. Topics addressed are: computer and information literacy, parental involvement and reading literacy, and language demand in testing mathematics. Also, in each chapter methods for identifying and handling potential DIF in the framework of item response theory are explored. The studies in this thesis show how DIF analyses can be insightful by benefiting from the synergy between a methodological focus on validity and a focus on more substantive research questions. More than simply a task to tick off before the “real” questions are investigated, DIF analyses can lead to insights into effects underlying test results. Throughout the studies in this thesis it is therefore shown how, in studies with a substantive interest in comparing groups, the study of validity on both test and questionnaire items should be integrated into the methodology. Though no clear-cut one-method-fits-all strategy is presented here, the thesis shows that there are many ways to approach the issue.

AB - International large-scale assessments (ILSAs) play a major role in the evaluation of educational systems. These projects are characterized by the standardized assessment of student achievement and the collection of contextual data by means of curriculum, student, teacher, school, and home questionnaires. Together, the resulting high-quality data on student achievement and contextual factors provide great opportunities for more theory-oriented educational effectiveness research, particularly in international contexts. To ensure the validity of analyses based on these data, particularly relating to measurement invariance across (sub)populations, efforts must be made to evaluate response behaviour across (sub)populations of interest. A lack of measurement invariance characterized by these differences in response behaviour, is called differential item functioning (DIF). This thesis presents five studies that contribute to research in the field of education by deploying ILSA data in research areas where the availability of standardized data from multiple countries offers new research opportunities. Topics addressed are: computer and information literacy, parental involvement and reading literacy, and language demand in testing mathematics. Also, in each chapter methods for identifying and handling potential DIF in the framework of item response theory are explored. The studies in this thesis show how DIF analyses can be insightful by benefiting from the synergy between a methodological focus on validity and a focus on more substantive research questions. More than simply a task to tick off before the “real” questions are investigated, DIF analyses can lead to insights into effects underlying test results. Throughout the studies in this thesis it is therefore shown how, in studies with a substantive interest in comparing groups, the study of validity on both test and questionnaire items should be integrated into the methodology. Though no clear-cut one-method-fits-all strategy is presented here, the thesis shows that there are many ways to approach the issue.

U2 - 10.3990/1.9789036546867

DO - 10.3990/1.9789036546867

M3 - PhD Thesis - Research UT, graduation UT

SN - 978-90-365-4686-7

PB - University of Twente

CY - Enschede

ER -