Measuring agreement in ordered rating scales

Cees van der Eijk

Research output: Contribution to journalArticleAcademic

99 Citations (Scopus)

Abstract

Ordered rating scales are one of the most frequently used question formats in large-scale surveys. Analysts of the responses to such questions often find themselves in need of describing the degree of agreement (concentration, consensus) of the answers to such questions. For that purpose they commonly use standard deviations of the response distributions, or measures based on these (such as the coefficient of consensus defined by Granberg and Holmberg, 1988), or the coefficient of variability, etc. This paper demonstrates that such measures are inappropriate for this purpose because they misrepresent what they are supposed to measure: the `peakedness' of a distribution. As an alternative a measure of agreement A is proposed. This measure is a weighted average of the degree of agreement that exists in the simple component parts – layers – into which any frequency distribution can be disaggregated, and for which agreement can be expressed in a straightforward and unequivocal way.
Original languageUndefined
Pages (from-to)325-341
JournalQuality and quantity
Volume35
Issue number3
DOIs
Publication statusPublished - 2001
Externally publishedYes

Keywords

  • IR-61236

Cite this