Assessing performance of a randomized versus a non-randomized study design

Maartje Raaijmakers, Hendrik Koffijberg, Jocelyne Posthumus, Ben van Hout, Herman van Engeland, Walter Matthys

Research output: Contribution to journalArticleAcademicpeer-review

8 Citations (Scopus)

Abstract

Introduction: Randomization is the most optimal design for evaluating program-effectiveness. In practice, however, conducting a randomized controlled trial is not always feasible. For a non-randomized study into the effect of a parent management training, predefined intervention and control groups of families were matched on six key characteristics. The quality of this match was then compared with the quality which is to be expected from a randomized study.

Methods: The performance of matching intervention and control families for predefined and randomized groups was evaluated by simulating new hypothetical intervention and control groups. The Mahalanobis metric was used to assess the distance between families in the intervention and the control groups and pairwise matching was performed. The global distance between these groups was used as measure of the balance of covariates in all matched pairs, with a smaller distance indicating a higher match quality. Results: In the ideal situation, when predefined groups are actually equal to randomized groups, the expected probability of a more equal balance of characteristics in the former groups than in the latter groups is 0.50. Using the data obtained in our study, and our predefined groups, this expected probability was 0.34.

Conclusion: Even when randomized groups are more balanced than predefined groups, using the latter groups for analyses might still be acceptable when the differences in group means are small. Findings suggest that matching can be a viable alternative to randomization for situations in which randomization is not feasible due to pragmatic constraints. However, a more accurate judgment on the value of the results obtained in this study requires results from similar analyses performed in other studies for comparison.

Original languageEnglish
Pages (from-to)293-303
Number of pages11
JournalContemporary clinical trials
Volume29
Issue number2
DOIs
Publication statusPublished - Mar 2008

Fingerprint

Random Allocation
Control Groups
Program Evaluation
Randomized Controlled Trials

Keywords

  • Equivalence
  • Mahalanobis distance
  • Matching
  • Randomized controlled trial

Cite this

Raaijmakers, M., Koffijberg, H., Posthumus, J., van Hout, B., van Engeland, H., & Matthys, W. (2008). Assessing performance of a randomized versus a non-randomized study design. Contemporary clinical trials, 29(2), 293-303. https://doi.org/10.1016/j.cct.2007.07.006
Raaijmakers, Maartje ; Koffijberg, Hendrik ; Posthumus, Jocelyne ; van Hout, Ben ; van Engeland, Herman ; Matthys, Walter. / Assessing performance of a randomized versus a non-randomized study design. In: Contemporary clinical trials. 2008 ; Vol. 29, No. 2. pp. 293-303.
@article{6b3f12188a5e46a288d1bef1ef650c72,
title = "Assessing performance of a randomized versus a non-randomized study design",
abstract = "Introduction: Randomization is the most optimal design for evaluating program-effectiveness. In practice, however, conducting a randomized controlled trial is not always feasible. For a non-randomized study into the effect of a parent management training, predefined intervention and control groups of families were matched on six key characteristics. The quality of this match was then compared with the quality which is to be expected from a randomized study.Methods: The performance of matching intervention and control families for predefined and randomized groups was evaluated by simulating new hypothetical intervention and control groups. The Mahalanobis metric was used to assess the distance between families in the intervention and the control groups and pairwise matching was performed. The global distance between these groups was used as measure of the balance of covariates in all matched pairs, with a smaller distance indicating a higher match quality. Results: In the ideal situation, when predefined groups are actually equal to randomized groups, the expected probability of a more equal balance of characteristics in the former groups than in the latter groups is 0.50. Using the data obtained in our study, and our predefined groups, this expected probability was 0.34.Conclusion: Even when randomized groups are more balanced than predefined groups, using the latter groups for analyses might still be acceptable when the differences in group means are small. Findings suggest that matching can be a viable alternative to randomization for situations in which randomization is not feasible due to pragmatic constraints. However, a more accurate judgment on the value of the results obtained in this study requires results from similar analyses performed in other studies for comparison.",
keywords = "Equivalence, Mahalanobis distance, Matching, Randomized controlled trial",
author = "Maartje Raaijmakers and Hendrik Koffijberg and Jocelyne Posthumus and {van Hout}, Ben and {van Engeland}, Herman and Walter Matthys",
year = "2008",
month = "3",
doi = "10.1016/j.cct.2007.07.006",
language = "English",
volume = "29",
pages = "293--303",
journal = "Contemporary clinical trials",
issn = "1551-7144",
publisher = "Elsevier",
number = "2",

}

Raaijmakers, M, Koffijberg, H, Posthumus, J, van Hout, B, van Engeland, H & Matthys, W 2008, 'Assessing performance of a randomized versus a non-randomized study design' Contemporary clinical trials, vol. 29, no. 2, pp. 293-303. https://doi.org/10.1016/j.cct.2007.07.006

Assessing performance of a randomized versus a non-randomized study design. / Raaijmakers, Maartje; Koffijberg, Hendrik; Posthumus, Jocelyne; van Hout, Ben; van Engeland, Herman; Matthys, Walter.

In: Contemporary clinical trials, Vol. 29, No. 2, 03.2008, p. 293-303.

Research output: Contribution to journalArticleAcademicpeer-review

TY - JOUR

T1 - Assessing performance of a randomized versus a non-randomized study design

AU - Raaijmakers, Maartje

AU - Koffijberg, Hendrik

AU - Posthumus, Jocelyne

AU - van Hout, Ben

AU - van Engeland, Herman

AU - Matthys, Walter

PY - 2008/3

Y1 - 2008/3

N2 - Introduction: Randomization is the most optimal design for evaluating program-effectiveness. In practice, however, conducting a randomized controlled trial is not always feasible. For a non-randomized study into the effect of a parent management training, predefined intervention and control groups of families were matched on six key characteristics. The quality of this match was then compared with the quality which is to be expected from a randomized study.Methods: The performance of matching intervention and control families for predefined and randomized groups was evaluated by simulating new hypothetical intervention and control groups. The Mahalanobis metric was used to assess the distance between families in the intervention and the control groups and pairwise matching was performed. The global distance between these groups was used as measure of the balance of covariates in all matched pairs, with a smaller distance indicating a higher match quality. Results: In the ideal situation, when predefined groups are actually equal to randomized groups, the expected probability of a more equal balance of characteristics in the former groups than in the latter groups is 0.50. Using the data obtained in our study, and our predefined groups, this expected probability was 0.34.Conclusion: Even when randomized groups are more balanced than predefined groups, using the latter groups for analyses might still be acceptable when the differences in group means are small. Findings suggest that matching can be a viable alternative to randomization for situations in which randomization is not feasible due to pragmatic constraints. However, a more accurate judgment on the value of the results obtained in this study requires results from similar analyses performed in other studies for comparison.

AB - Introduction: Randomization is the most optimal design for evaluating program-effectiveness. In practice, however, conducting a randomized controlled trial is not always feasible. For a non-randomized study into the effect of a parent management training, predefined intervention and control groups of families were matched on six key characteristics. The quality of this match was then compared with the quality which is to be expected from a randomized study.Methods: The performance of matching intervention and control families for predefined and randomized groups was evaluated by simulating new hypothetical intervention and control groups. The Mahalanobis metric was used to assess the distance between families in the intervention and the control groups and pairwise matching was performed. The global distance between these groups was used as measure of the balance of covariates in all matched pairs, with a smaller distance indicating a higher match quality. Results: In the ideal situation, when predefined groups are actually equal to randomized groups, the expected probability of a more equal balance of characteristics in the former groups than in the latter groups is 0.50. Using the data obtained in our study, and our predefined groups, this expected probability was 0.34.Conclusion: Even when randomized groups are more balanced than predefined groups, using the latter groups for analyses might still be acceptable when the differences in group means are small. Findings suggest that matching can be a viable alternative to randomization for situations in which randomization is not feasible due to pragmatic constraints. However, a more accurate judgment on the value of the results obtained in this study requires results from similar analyses performed in other studies for comparison.

KW - Equivalence

KW - Mahalanobis distance

KW - Matching

KW - Randomized controlled trial

UR - http://www.scopus.com/inward/record.url?scp=39049157284&partnerID=8YFLogxK

U2 - 10.1016/j.cct.2007.07.006

DO - 10.1016/j.cct.2007.07.006

M3 - Article

VL - 29

SP - 293

EP - 303

JO - Contemporary clinical trials

JF - Contemporary clinical trials

SN - 1551-7144

IS - 2

ER -