Experimental studies to improve the reliability and validity of regulatory judgments on health care in the Netherlands: a randomized controlled trial and before and after case study

Saskia M. Tuijn, Huub van den Bergh, Paul Robben, F.J.G. Janssens

    Research output: Contribution to journalArticleAcademicpeer-review

    3 Citations (Scopus)
    109 Downloads (Pure)

    Abstract

    Rationale, aims and objectives We examined the effect of two interventions on both the reliability and validity of regulatory judgments: adjusting the regulatory instrument and attending a consensus meeting. Method We adjusted the regulatory instrument. With a randomized controlled trial (RCT) we examined the effect of the adjustments we made to the instrument. In the consensus meeting inspectors discussed cases and had to reach consensus about the order of the cases. We used a before and after case study to assess the effect of the consensus meeting. We compared the judgments assigned in the RCT with the unadjusted instrument with the judgments assigned with the unadjusted instrument after the consensus meeting. Moreover we explored the effect of increasing the number of inspectors per regulatory visit based on the estimates of the two interventions. Results The consensus meeting improved the agreement between inspectors; the variance between inspectors was smallest (0.03) and the reliability coefficient was highest (0.59). Validity is assessed by examining the relation between the assigned judgments and the corporate standard and expressed by a correlation coefficient. This coefficient was highest after the consensus meeting (0.48). Adjustment of the instrument did not increase reliability and validity coefficients. Conclusions Participating in a consensus meeting improved reliability and validity. Increasing the number of inspectors resulted in both higher reliability and validity values. Organizing consensus meetings and increasing the number of inspectors per regulatory visit seem to be valuable interventions for improving regulatory judgments
    Original languageEnglish
    Pages (from-to)352-361
    JournalJournal of evaluation in clinical practice
    Volume20
    Issue number4
    DOIs
    Publication statusPublished - 2014

    Fingerprint

    Reproducibility of Results
    Netherlands
    Consensus
    Randomized Controlled Trials
    Delivery of Health Care
    Social Adjustment

    Keywords

    • METIS-304830
    • IR-91643

    Cite this

    @article{6f5d8ede3ec74f06b0f8a4f23c41ece4,
    title = "Experimental studies to improve the reliability and validity of regulatory judgments on health care in the Netherlands: a randomized controlled trial and before and after case study",
    abstract = "Rationale, aims and objectives We examined the effect of two interventions on both the reliability and validity of regulatory judgments: adjusting the regulatory instrument and attending a consensus meeting. Method We adjusted the regulatory instrument. With a randomized controlled trial (RCT) we examined the effect of the adjustments we made to the instrument. In the consensus meeting inspectors discussed cases and had to reach consensus about the order of the cases. We used a before and after case study to assess the effect of the consensus meeting. We compared the judgments assigned in the RCT with the unadjusted instrument with the judgments assigned with the unadjusted instrument after the consensus meeting. Moreover we explored the effect of increasing the number of inspectors per regulatory visit based on the estimates of the two interventions. Results The consensus meeting improved the agreement between inspectors; the variance between inspectors was smallest (0.03) and the reliability coefficient was highest (0.59). Validity is assessed by examining the relation between the assigned judgments and the corporate standard and expressed by a correlation coefficient. This coefficient was highest after the consensus meeting (0.48). Adjustment of the instrument did not increase reliability and validity coefficients. Conclusions Participating in a consensus meeting improved reliability and validity. Increasing the number of inspectors resulted in both higher reliability and validity values. Organizing consensus meetings and increasing the number of inspectors per regulatory visit seem to be valuable interventions for improving regulatory judgments",
    keywords = "METIS-304830, IR-91643",
    author = "Tuijn, {Saskia M.} and {van den Bergh}, Huub and Paul Robben and F.J.G. Janssens",
    note = "Open access",
    year = "2014",
    doi = "10.1111/jep.12136",
    language = "English",
    volume = "20",
    pages = "352--361",
    journal = "Journal of evaluation in clinical practice",
    issn = "1356-1294",
    publisher = "Wiley-Blackwell",
    number = "4",

    }

    Experimental studies to improve the reliability and validity of regulatory judgments on health care in the Netherlands: a randomized controlled trial and before and after case study. / Tuijn, Saskia M.; van den Bergh, Huub; Robben, Paul; Janssens, F.J.G.

    In: Journal of evaluation in clinical practice, Vol. 20, No. 4, 2014, p. 352-361.

    Research output: Contribution to journalArticleAcademicpeer-review

    TY - JOUR

    T1 - Experimental studies to improve the reliability and validity of regulatory judgments on health care in the Netherlands: a randomized controlled trial and before and after case study

    AU - Tuijn, Saskia M.

    AU - van den Bergh, Huub

    AU - Robben, Paul

    AU - Janssens, F.J.G.

    N1 - Open access

    PY - 2014

    Y1 - 2014

    N2 - Rationale, aims and objectives We examined the effect of two interventions on both the reliability and validity of regulatory judgments: adjusting the regulatory instrument and attending a consensus meeting. Method We adjusted the regulatory instrument. With a randomized controlled trial (RCT) we examined the effect of the adjustments we made to the instrument. In the consensus meeting inspectors discussed cases and had to reach consensus about the order of the cases. We used a before and after case study to assess the effect of the consensus meeting. We compared the judgments assigned in the RCT with the unadjusted instrument with the judgments assigned with the unadjusted instrument after the consensus meeting. Moreover we explored the effect of increasing the number of inspectors per regulatory visit based on the estimates of the two interventions. Results The consensus meeting improved the agreement between inspectors; the variance between inspectors was smallest (0.03) and the reliability coefficient was highest (0.59). Validity is assessed by examining the relation between the assigned judgments and the corporate standard and expressed by a correlation coefficient. This coefficient was highest after the consensus meeting (0.48). Adjustment of the instrument did not increase reliability and validity coefficients. Conclusions Participating in a consensus meeting improved reliability and validity. Increasing the number of inspectors resulted in both higher reliability and validity values. Organizing consensus meetings and increasing the number of inspectors per regulatory visit seem to be valuable interventions for improving regulatory judgments

    AB - Rationale, aims and objectives We examined the effect of two interventions on both the reliability and validity of regulatory judgments: adjusting the regulatory instrument and attending a consensus meeting. Method We adjusted the regulatory instrument. With a randomized controlled trial (RCT) we examined the effect of the adjustments we made to the instrument. In the consensus meeting inspectors discussed cases and had to reach consensus about the order of the cases. We used a before and after case study to assess the effect of the consensus meeting. We compared the judgments assigned in the RCT with the unadjusted instrument with the judgments assigned with the unadjusted instrument after the consensus meeting. Moreover we explored the effect of increasing the number of inspectors per regulatory visit based on the estimates of the two interventions. Results The consensus meeting improved the agreement between inspectors; the variance between inspectors was smallest (0.03) and the reliability coefficient was highest (0.59). Validity is assessed by examining the relation between the assigned judgments and the corporate standard and expressed by a correlation coefficient. This coefficient was highest after the consensus meeting (0.48). Adjustment of the instrument did not increase reliability and validity coefficients. Conclusions Participating in a consensus meeting improved reliability and validity. Increasing the number of inspectors resulted in both higher reliability and validity values. Organizing consensus meetings and increasing the number of inspectors per regulatory visit seem to be valuable interventions for improving regulatory judgments

    KW - METIS-304830

    KW - IR-91643

    U2 - 10.1111/jep.12136

    DO - 10.1111/jep.12136

    M3 - Article

    VL - 20

    SP - 352

    EP - 361

    JO - Journal of evaluation in clinical practice

    JF - Journal of evaluation in clinical practice

    SN - 1356-1294

    IS - 4

    ER -