This study evaluates changes in teachers' instructional skills after participating in an intensive data-based decision making (DBDM) intervention for grade 4 teachers. Teachers were recorded three times prior to the intervention, and three times after the intervention, and all recordings were rated by four raters. The data was analyzed by means of advanced item response theory (IRT) techniques, combined with a generalizability model. Teachers significantly improved their DBDM related skills. Teachers’ initial basic teaching skills did not seem to matter for the extent to which teachers developed their DBDM related instructional skills. Suggestions for future research are presented.