The effect of inefficient designs on the stability of individual preferences

Introduction In conjoint analysis, efficient experimental designs often require the respondent to answer many choice tasks. To reduce respondent burden, less efficient designs are used. We aim to investigate the effects of using inefficient designs on the stability of individual preferences. Methods Two convenience samples completed a best‐worst scaling case 2 experiment exploring preferences for treatment of localized prostate cancer (PC) or breast cancer (BC). Treatments were described using 5 attributes and 2 levels in both cases. An efficient design (balanced and orthogonal) was used to construct 16 profiles/questions. The sequence of profiles ensured that after 6 and 12 questions level‐balance was obtained (no orthogonality). After completion of 6, 12 and 16 questions, each respondent received real‐time feedback on the importance of attributes (graph) and a preferred treatment recommendation (words). Feedback was based on bestminus‐worst scores. Results In total, 68 women and 23 men completed the experiment. 82% of respondents received the same treatment recommendations after completion of 6, 12 or 16 questions (85% BC, 78% PC). In addition, for 12% (7% BC, 17% PC) only the treatment recommendation after six questions differed from the latter two. For the BC‐case the mean‐difference in importance score per attribute between 6 and 12 questions was 7.5% (3.1%) and between 12 and 16 questions 3.9% (2.0%), P<0.001. For PC, the differences were 6.9% (3.4%) and 3.5% (1.5%) respectively, P<0.001. Although only two respondents had the same rank order of attributes based on importance, 81% of respondents never changed their most important attribute (82%‐BC, 78%‐PC), and 45% never altered their second‐most important attribute (47%‐BC, 39%‐PC). Conclusions For these cases, individual preferences are quite stable after six questions. However, adding more questions results in higher stability of attribute importance. More research is needed to examine whether inefficient designs can be used in value clarification exercises.