The Designometric Perspective and the Psychometric Fallacy in Developing User Experience Rating Scales

Rating scales are widely used in user experience (UX) research to compare and rank design alternatives, yet their development typically relies on psychometric procedures originally created for assessing human traits. We argue that this practice commits the psychometric fallacy: evaluating designometric instruments—tools intended to discriminate between designs—using person × item response matrices that exclude the essential third dimension of design. Because design evaluation inherently forms a design × person × item data structure, valid scale development requires samples of designs large enough to assess how well items rank design alternatives. We show that collapsing the data cube along persons yields a design × item matrix that allows psychometric tools to be applied meaningfully, whereas collapsing along designs yields misleading psychometric results about person sensitivity rather than design discriminability. A simulation study demonstrates that scales can appear highly reliable psychometrically while being effectively unusable for ranking designs. Secondary analyses of eight commonly used UX scales across large design samples reveal unsystematic distortions in reliability estimates, item performance, and dimensionality when instruments are evaluated under the psychometric fallacy. We assess the impact of the psychometric fallacy for present users and designers of future rating scales and discuss more advanced methods in designometric modelling.