The use of multi-criteria decision methods in health care: Does method used influence outcome?

OBJECTIVES: To investigate how the choice of multicriteria decision method influences outcome (ranking criteria and criteria weights). Population. A convenience sample of 28 subjects, 12 healthy and 16 cognitively impaired.

METHODS: Based on a literature review, 5 multicriteria methods were chosen for comparison including: Kepner-tregoe analysis (KTA), simple multi attribute rating technique (SMART), SMART using swing weights (SWING), Analytic Hierarchy Process (AHP) and Conjoint Analysis (CA). Four attributes of treatment were identified (impact, duration, and end-result of treatment and associated risks). Subjects were asked to both rank and rate the importance of these attributes with each method. The order of methods was randomized and the total length of the interview was restricted to one hour. Some subjects therefore did not use all methods. Subjects were interviewed either once (n = 14) or twice (n = 14) (Only the results of the first measurement are presented)

RESULTS: The highest percentages of rank reversals were found between CA and other methods (55–62%). The lowest percentage of rank reversals was between KTA and SMART (18%). The percentage of rank reversals was significantly higher in impaired population (An average of 54% compared to 36% in unimpaired population). When comparing actual weights, AHP and SMART correlate highly with all other methods except CA.

CONCLUSIONS: The high percentages in rank reversal and divergent correlation between individual weights (especially CA compared to other methods) show that the method chosen influences outcome. This has to be taken into account when the ranks or weights are used in multi-criteria decision analysis to make
actual treatment decisions. The dissimilar methodology of CA might explain the high percentages of rank-reversals and low correlation between this method and other. Also, the design of the survey might have influenced CA weights and ranking.