Methodological issues in large-scale educational surveys

This thesis focuses on the application of item response theory (IRT) in the context of large scale international educational surveys like PISA 2009 (OECD). Although IRT methodology has been widely used in educational applications such as test construction, norming of examinations, detection of item bias and computerized adaptive testing, large scale surveys present a number of specific problems. A number of these problems are addressed in this thesis using PISA student data of the 2006 and 2009 cycles. The first problem in international comparative educational tests relates to the detection of cultural bias over countries. In this thesis, we targeted a problem know as country-specific Differential Item Functioning (CDIF) or country-by-item-interaction. This problem is tackled by modeling CDIF using country specific item parameters. In Chapter 2 this methodology is applied to the background questionnaires of the PISA 2009 cycle. In Chapter 3, this methodology is outlined and applied to the Reading dataset of the PISA 2006 cycle. The results showed that the impact of CDIF on the ranking of countries is not prominent and becomes almost negligible when the statistical uncertainty regarding the country means is properly taken into account. In Chapter 4, the practical significance of modeling item bias on the background questionnaire scales was studied not only in terms of the ordering of countries on the respective scales but also its impact on the results of regression analyses with latent variables in survey research. The next topic relates to the combination of the results of IRT measurement models with multilevel structural models to relate cognitive outcomes to background variables. According to our findings, results obtained using separately estimated plausible values and latent covariates (as provided in large scale data sets like PISA) were comparable with the results obtained using a concurrent estimation with raw data for the various kinds of IRT measurement models embedded in the structural (regression) model. The last Chapter gives an example of an advanced latent regression model based on plausible value methodology that explores the relation between socio-economic status and reading achievement in PISA.