Experiences in using practitioner's checklists to evaluate the industrial relevance of requirements engineering experiments

Maya Daneva, Klaas Sikkel, Nelly Condori-Fernandez, Andrea Herrmann

Research output: Chapter in Book/Report/Conference proceedingConference contributionAcademicpeer-review

Abstract

Background: A grand challenge for Requirement Engineering (RE) research is to help practitioners understand which RE methods work in what contexts and why. RE researchers recognize that for an RE method to be adopted in industry, RE practitioners should be able to evaluate the relevance of empirical studies to their practice. One possible approach to relevance evaluation is the set of perspective-based checklists proposed by Kitchenham et al. Specifically, the checklist from the practitioner's perspective seems to be a good candidate for evaluating the relevance of RE studies to RE practice. However, little is known about the applicability of this checklist to the RE field. Moreover, this checklist also requires a deeper analysis of its reliability. Aim: We propose a perspective-based checklist to the RE community that allows evaluating the relevance of experimental studies in RE from the practitioner's/consultant's viewpoint. Method: We followed an iterative design-science based approach in which we first analyzed the problems with a previously published checklist and then developed an operationalized proposal for a new checklist to counter these problems. We performed a reliability evaluation of this new checklist by having two practitioners apply the checklist on 24 papers that report experimental results on software requirements specifications' comprehensibility. Results: We report first-hand experiences of practitioners in evaluating the relevance of primary studies in RE, by using a perspective-based checklist. With respect to the reliability of the adjusted checklist, 9 of out 19 questions show an acceptable proportion of agreement (between two practitioners). Conclusions: Based on our experience, the contextualization and operationalization of a perspective-based checklist helps to make it more useful for the practitioners. However, to increase the reliability of the checklist, more reviewers and more discussion cycles are necessary.

Original languageEnglish
Title of host publicationProceedings 2018 ACM/IEEE 6th International Workshop on Conducting Empirical Studies in Industry, CESI 2018
PublisherIEEE
Pages5-12
Number of pages8
ISBN (Electronic)9781450357364
DOIs
Publication statusPublished - 28 May 2018
Event6th International Workshop on Conducting Empirical Studies in Industry 2018 - Gothenburg, Sweden
Duration: 28 May 201828 May 2018
Conference number: 6
http://cesi-workshops.org/cesi-2018/

Workshop

Workshop6th International Workshop on Conducting Empirical Studies in Industry 2018
Abbreviated titleCESI 2018
CountrySweden
CityGothenburg
Period28/05/1828/05/18
Internet address

Fingerprint

Requirements engineering
Experiments
Engineering research
Specifications

Cite this

Daneva, M., Sikkel, K., Condori-Fernandez, N., & Herrmann, A. (2018). Experiences in using practitioner's checklists to evaluate the industrial relevance of requirements engineering experiments. In Proceedings 2018 ACM/IEEE 6th International Workshop on Conducting Empirical Studies in Industry, CESI 2018 (pp. 5-12). IEEE. https://doi.org/10.1145/3193965.3193966
Daneva, Maya ; Sikkel, Klaas ; Condori-Fernandez, Nelly ; Herrmann, Andrea. / Experiences in using practitioner's checklists to evaluate the industrial relevance of requirements engineering experiments. Proceedings 2018 ACM/IEEE 6th International Workshop on Conducting Empirical Studies in Industry, CESI 2018. IEEE, 2018. pp. 5-12
@inproceedings{7e82780950584766a4a2bf3b807e49f4,
title = "Experiences in using practitioner's checklists to evaluate the industrial relevance of requirements engineering experiments",
abstract = "Background: A grand challenge for Requirement Engineering (RE) research is to help practitioners understand which RE methods work in what contexts and why. RE researchers recognize that for an RE method to be adopted in industry, RE practitioners should be able to evaluate the relevance of empirical studies to their practice. One possible approach to relevance evaluation is the set of perspective-based checklists proposed by Kitchenham et al. Specifically, the checklist from the practitioner's perspective seems to be a good candidate for evaluating the relevance of RE studies to RE practice. However, little is known about the applicability of this checklist to the RE field. Moreover, this checklist also requires a deeper analysis of its reliability. Aim: We propose a perspective-based checklist to the RE community that allows evaluating the relevance of experimental studies in RE from the practitioner's/consultant's viewpoint. Method: We followed an iterative design-science based approach in which we first analyzed the problems with a previously published checklist and then developed an operationalized proposal for a new checklist to counter these problems. We performed a reliability evaluation of this new checklist by having two practitioners apply the checklist on 24 papers that report experimental results on software requirements specifications' comprehensibility. Results: We report first-hand experiences of practitioners in evaluating the relevance of primary studies in RE, by using a perspective-based checklist. With respect to the reliability of the adjusted checklist, 9 of out 19 questions show an acceptable proportion of agreement (between two practitioners). Conclusions: Based on our experience, the contextualization and operationalization of a perspective-based checklist helps to make it more useful for the practitioners. However, to increase the reliability of the checklist, more reviewers and more discussion cycles are necessary.",
author = "Maya Daneva and Klaas Sikkel and Nelly Condori-Fernandez and Andrea Herrmann",
year = "2018",
month = "5",
day = "28",
doi = "10.1145/3193965.3193966",
language = "English",
pages = "5--12",
booktitle = "Proceedings 2018 ACM/IEEE 6th International Workshop on Conducting Empirical Studies in Industry, CESI 2018",
publisher = "IEEE",
address = "United States",

}

Daneva, M, Sikkel, K, Condori-Fernandez, N & Herrmann, A 2018, Experiences in using practitioner's checklists to evaluate the industrial relevance of requirements engineering experiments. in Proceedings 2018 ACM/IEEE 6th International Workshop on Conducting Empirical Studies in Industry, CESI 2018. IEEE, pp. 5-12, 6th International Workshop on Conducting Empirical Studies in Industry 2018, Gothenburg, Sweden, 28/05/18. https://doi.org/10.1145/3193965.3193966

Experiences in using practitioner's checklists to evaluate the industrial relevance of requirements engineering experiments. / Daneva, Maya; Sikkel, Klaas; Condori-Fernandez, Nelly; Herrmann, Andrea.

Proceedings 2018 ACM/IEEE 6th International Workshop on Conducting Empirical Studies in Industry, CESI 2018. IEEE, 2018. p. 5-12.

Research output: Chapter in Book/Report/Conference proceedingConference contributionAcademicpeer-review

TY - GEN

T1 - Experiences in using practitioner's checklists to evaluate the industrial relevance of requirements engineering experiments

AU - Daneva, Maya

AU - Sikkel, Klaas

AU - Condori-Fernandez, Nelly

AU - Herrmann, Andrea

PY - 2018/5/28

Y1 - 2018/5/28

N2 - Background: A grand challenge for Requirement Engineering (RE) research is to help practitioners understand which RE methods work in what contexts and why. RE researchers recognize that for an RE method to be adopted in industry, RE practitioners should be able to evaluate the relevance of empirical studies to their practice. One possible approach to relevance evaluation is the set of perspective-based checklists proposed by Kitchenham et al. Specifically, the checklist from the practitioner's perspective seems to be a good candidate for evaluating the relevance of RE studies to RE practice. However, little is known about the applicability of this checklist to the RE field. Moreover, this checklist also requires a deeper analysis of its reliability. Aim: We propose a perspective-based checklist to the RE community that allows evaluating the relevance of experimental studies in RE from the practitioner's/consultant's viewpoint. Method: We followed an iterative design-science based approach in which we first analyzed the problems with a previously published checklist and then developed an operationalized proposal for a new checklist to counter these problems. We performed a reliability evaluation of this new checklist by having two practitioners apply the checklist on 24 papers that report experimental results on software requirements specifications' comprehensibility. Results: We report first-hand experiences of practitioners in evaluating the relevance of primary studies in RE, by using a perspective-based checklist. With respect to the reliability of the adjusted checklist, 9 of out 19 questions show an acceptable proportion of agreement (between two practitioners). Conclusions: Based on our experience, the contextualization and operationalization of a perspective-based checklist helps to make it more useful for the practitioners. However, to increase the reliability of the checklist, more reviewers and more discussion cycles are necessary.

AB - Background: A grand challenge for Requirement Engineering (RE) research is to help practitioners understand which RE methods work in what contexts and why. RE researchers recognize that for an RE method to be adopted in industry, RE practitioners should be able to evaluate the relevance of empirical studies to their practice. One possible approach to relevance evaluation is the set of perspective-based checklists proposed by Kitchenham et al. Specifically, the checklist from the practitioner's perspective seems to be a good candidate for evaluating the relevance of RE studies to RE practice. However, little is known about the applicability of this checklist to the RE field. Moreover, this checklist also requires a deeper analysis of its reliability. Aim: We propose a perspective-based checklist to the RE community that allows evaluating the relevance of experimental studies in RE from the practitioner's/consultant's viewpoint. Method: We followed an iterative design-science based approach in which we first analyzed the problems with a previously published checklist and then developed an operationalized proposal for a new checklist to counter these problems. We performed a reliability evaluation of this new checklist by having two practitioners apply the checklist on 24 papers that report experimental results on software requirements specifications' comprehensibility. Results: We report first-hand experiences of practitioners in evaluating the relevance of primary studies in RE, by using a perspective-based checklist. With respect to the reliability of the adjusted checklist, 9 of out 19 questions show an acceptable proportion of agreement (between two practitioners). Conclusions: Based on our experience, the contextualization and operationalization of a perspective-based checklist helps to make it more useful for the practitioners. However, to increase the reliability of the checklist, more reviewers and more discussion cycles are necessary.

U2 - 10.1145/3193965.3193966

DO - 10.1145/3193965.3193966

M3 - Conference contribution

SP - 5

EP - 12

BT - Proceedings 2018 ACM/IEEE 6th International Workshop on Conducting Empirical Studies in Industry, CESI 2018

PB - IEEE

ER -

Daneva M, Sikkel K, Condori-Fernandez N, Herrmann A. Experiences in using practitioner's checklists to evaluate the industrial relevance of requirements engineering experiments. In Proceedings 2018 ACM/IEEE 6th International Workshop on Conducting Empirical Studies in Industry, CESI 2018. IEEE. 2018. p. 5-12 https://doi.org/10.1145/3193965.3193966