A latent trait method for determining inconsistencies in the use of the Angoff and Nedelsky techniques of standard setting

It has often been argued that all techniques of standard setting are arbitrary and likely to yield different results for different techniques or persons. This paper deals with a related but hitherto ignored aspect of standard setting, namely, the possibility that Angoff or Nedelsky judges misspecify the probabilities of the borderline student's success on the items because they do not use the psychometric properties of the items consistently. A latent trait method is proposed to estimate such misspecifications, and an index of consistency is defined that can be used for deciding whether standards are set consistently enough for use in practice. Results from an empirical study are presented to illustrate the use of the method in a typical educational situation. The results indicate that serious errors of specification can be expected and that, on the whole, these will be considerably larger for the Nedelsky than for the Angoff technique.