Experimental Evaluation of a Tool for Change Impact Prediction in Requirements Models: Design, Results and Lessons Learned

Arda Goknil; Roderick van Domburg; Ivan Kurtev; Klaas van den Berg; Fons Wijnhoven

doi:10.1109/MoDRE.2014.6890826

Experimental Evaluation of a Tool for Change Impact Prediction in Requirements Models: Design, Results and Lessons Learned

Arda Goknil, Roderick van Domburg, Ivan Kurtev, Klaas van den Berg, Fons Wijnhoven

Faculty of Behavioural, Management and Social Sciences

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › Academic › peer-review

7 Citations (Scopus)

193 Downloads (Pure)

Abstract

There are commercial tools like IBM Rational RequisitePro and DOORS that support semi-automatic change impact analysis for requirements. These tools capture the requirements relations and allow tracing the paths they form. In most of these tools, relation types do not say anything about the meaning of the relations except the direction. When a change is introduced to a requirement, the requirements engineer analyzes the impact of the change in related requirements. In case semantic information is missing to determine precisely how requirements are related to each other, the requirements engineer generally has to assume the worst case dependencies based on the available syntactic information only. We developed a tool that uses formal semantics of requirements relations to support change impact analysis and prediction in requirements models. The tool TRIC (Tool for Requirements Inferencing and Consistency checking) works on models that explicitly represent requirements and the relations among them with their formal semantics. In this paper we report on the evaluation of how TRIC improves the quality of change impact predictions. A quasiexperiment is systematically designed and executed to empirically validate the impact of TRIC. We conduct the quasi-experiment with 21 master’s degree students predicting change impact for five change scenarios in a real software requirements specification. The participants are assigned with Microsoft Excel, IBM RequisitePro or TRIC to perform change impact prediction for the change scenarios. It is hypothesized that using TRIC would positively impact the quality of change impact predictions. Two formal hypotheses are developed. As a result of the experiment, we are not able to reject the null hypotheses, and thus we are not able to show experimentally the effectiveness of our tool. In the paper we discuss reasons for the failure to reject the null hypotheses in the experiment.

Original language	English
Title of host publication	2014 IEEE 4th International Model-Driven Requirements Engineering Workshop (MoDRE)
Subtitle of host publication	Proceedings
Editors	Ana Moreira, Pablo Sánchez, Gunter Mussbacher, Joao Araújo
Place of Publication	Piscataway, NJ
Publisher	IEEE
Number of pages	10
ISBN (Print)	978-1-4799-6343-0
DOIs	https://doi.org/10.1109/MoDRE.2014.6890826
Publication status	Published - 25 Aug 2014
Event	2014 IEEE 4th International Model-Driven Requirements Engineering Workshop (MoDRE) - Karlskrona, Sweden Duration: 25 Aug 2014 → 25 Aug 2014

Conference

Conference	2014 IEEE 4th International Model-Driven Requirements Engineering Workshop (MoDRE)
Abbreviated title	MoDRE
Country/Territory	Sweden
City	Karlskrona
Period	25/08/14 → 25/08/14

Keywords

METIS-304279
IR-91418

Access to Document

10.1109/MoDRE.2014.6890826

Modre_PaperAccepted author version, 462 KB

Cite this

Goknil, A., van Domburg, R., Kurtev, I., van den Berg, K., & Wijnhoven, F. (2014). Experimental Evaluation of a Tool for Change Impact Prediction in Requirements Models: Design, Results and Lessons Learned. In A. Moreira, P. Sánchez, G. Mussbacher, & J. Araújo (Eds.), 2014 IEEE 4th International Model-Driven Requirements Engineering Workshop (MoDRE): Proceedings IEEE. https://doi.org/10.1109/MoDRE.2014.6890826

Goknil, Arda ; van Domburg, Roderick ; Kurtev, Ivan et al. / Experimental Evaluation of a Tool for Change Impact Prediction in Requirements Models : Design, Results and Lessons Learned. 2014 IEEE 4th International Model-Driven Requirements Engineering Workshop (MoDRE): Proceedings. editor / Ana Moreira ; Pablo Sánchez ; Gunter Mussbacher ; Joao Araújo. Piscataway, NJ : IEEE, 2014.

@inproceedings{ef70dda9e5f64b67a87919ce0182bbc6,

title = "Experimental Evaluation of a Tool for Change Impact Prediction in Requirements Models: Design, Results and Lessons Learned",

abstract = "There are commercial tools like IBM Rational RequisitePro and DOORS that support semi-automatic change impact analysis for requirements. These tools capture the requirements relations and allow tracing the paths they form. In most of these tools, relation types do not say anything about the meaning of the relations except the direction. When a change is introduced to a requirement, the requirements engineer analyzes the impact of the change in related requirements. In case semantic information is missing to determine precisely how requirements are related to each other, the requirements engineer generally has to assume the worst case dependencies based on the available syntactic information only. We developed a tool that uses formal semantics of requirements relations to support change impact analysis and prediction in requirements models. The tool TRIC (Tool for Requirements Inferencing and Consistency checking) works on models that explicitly represent requirements and the relations among them with their formal semantics. In this paper we report on the evaluation of how TRIC improves the quality of change impact predictions. A quasiexperiment is systematically designed and executed to empirically validate the impact of TRIC. We conduct the quasi-experiment with 21 master{\textquoteright}s degree students predicting change impact for five change scenarios in a real software requirements specification. The participants are assigned with Microsoft Excel, IBM RequisitePro or TRIC to perform change impact prediction for the change scenarios. It is hypothesized that using TRIC would positively impact the quality of change impact predictions. Two formal hypotheses are developed. As a result of the experiment, we are not able to reject the null hypotheses, and thus we are not able to show experimentally the effectiveness of our tool. In the paper we discuss reasons for the failure to reject the null hypotheses in the experiment.",

keywords = "METIS-304279, IR-91418",

author = "Arda Goknil and {van Domburg}, Roderick and Ivan Kurtev and {van den Berg}, Klaas and Fons Wijnhoven",

year = "2014",

month = aug,

day = "25",

doi = "10.1109/MoDRE.2014.6890826",

language = "English",

isbn = "978-1-4799-6343-0",

editor = "Ana Moreira and Pablo S{\'a}nchez and Gunter Mussbacher and Joao Ara{\'u}jo",

booktitle = "2014 IEEE 4th International Model-Driven Requirements Engineering Workshop (MoDRE)",

publisher = "IEEE",

address = "United States",

note = "2014 IEEE 4th International Model-Driven Requirements Engineering Workshop (MoDRE), MoDRE ; Conference date: 25-08-2014 Through 25-08-2014",

}

Goknil, A, van Domburg, R, Kurtev, I, van den Berg, K & Wijnhoven, F 2014, Experimental Evaluation of a Tool for Change Impact Prediction in Requirements Models: Design, Results and Lessons Learned. in A Moreira, P Sánchez, G Mussbacher & J Araújo (eds), 2014 IEEE 4th International Model-Driven Requirements Engineering Workshop (MoDRE): Proceedings. IEEE, Piscataway, NJ, 2014 IEEE 4th International Model-Driven Requirements Engineering Workshop (MoDRE), Karlskrona, Sweden, 25/08/14. https://doi.org/10.1109/MoDRE.2014.6890826

Experimental Evaluation of a Tool for Change Impact Prediction in Requirements Models: Design, Results and Lessons Learned. / Goknil, Arda; van Domburg, Roderick; Kurtev, Ivan et al.
2014 IEEE 4th International Model-Driven Requirements Engineering Workshop (MoDRE): Proceedings. ed. / Ana Moreira; Pablo Sánchez; Gunter Mussbacher; Joao Araújo. Piscataway, NJ: IEEE, 2014.

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › Academic › peer-review

TY - GEN

T1 - Experimental Evaluation of a Tool for Change Impact Prediction in Requirements Models

T2 - 2014 IEEE 4th International Model-Driven Requirements Engineering Workshop (MoDRE)

AU - Goknil, Arda

AU - van Domburg, Roderick

AU - Kurtev, Ivan

AU - van den Berg, Klaas

AU - Wijnhoven, Fons

PY - 2014/8/25

Y1 - 2014/8/25

N2 - There are commercial tools like IBM Rational RequisitePro and DOORS that support semi-automatic change impact analysis for requirements. These tools capture the requirements relations and allow tracing the paths they form. In most of these tools, relation types do not say anything about the meaning of the relations except the direction. When a change is introduced to a requirement, the requirements engineer analyzes the impact of the change in related requirements. In case semantic information is missing to determine precisely how requirements are related to each other, the requirements engineer generally has to assume the worst case dependencies based on the available syntactic information only. We developed a tool that uses formal semantics of requirements relations to support change impact analysis and prediction in requirements models. The tool TRIC (Tool for Requirements Inferencing and Consistency checking) works on models that explicitly represent requirements and the relations among them with their formal semantics. In this paper we report on the evaluation of how TRIC improves the quality of change impact predictions. A quasiexperiment is systematically designed and executed to empirically validate the impact of TRIC. We conduct the quasi-experiment with 21 master’s degree students predicting change impact for five change scenarios in a real software requirements specification. The participants are assigned with Microsoft Excel, IBM RequisitePro or TRIC to perform change impact prediction for the change scenarios. It is hypothesized that using TRIC would positively impact the quality of change impact predictions. Two formal hypotheses are developed. As a result of the experiment, we are not able to reject the null hypotheses, and thus we are not able to show experimentally the effectiveness of our tool. In the paper we discuss reasons for the failure to reject the null hypotheses in the experiment.

AB - There are commercial tools like IBM Rational RequisitePro and DOORS that support semi-automatic change impact analysis for requirements. These tools capture the requirements relations and allow tracing the paths they form. In most of these tools, relation types do not say anything about the meaning of the relations except the direction. When a change is introduced to a requirement, the requirements engineer analyzes the impact of the change in related requirements. In case semantic information is missing to determine precisely how requirements are related to each other, the requirements engineer generally has to assume the worst case dependencies based on the available syntactic information only. We developed a tool that uses formal semantics of requirements relations to support change impact analysis and prediction in requirements models. The tool TRIC (Tool for Requirements Inferencing and Consistency checking) works on models that explicitly represent requirements and the relations among them with their formal semantics. In this paper we report on the evaluation of how TRIC improves the quality of change impact predictions. A quasiexperiment is systematically designed and executed to empirically validate the impact of TRIC. We conduct the quasi-experiment with 21 master’s degree students predicting change impact for five change scenarios in a real software requirements specification. The participants are assigned with Microsoft Excel, IBM RequisitePro or TRIC to perform change impact prediction for the change scenarios. It is hypothesized that using TRIC would positively impact the quality of change impact predictions. Two formal hypotheses are developed. As a result of the experiment, we are not able to reject the null hypotheses, and thus we are not able to show experimentally the effectiveness of our tool. In the paper we discuss reasons for the failure to reject the null hypotheses in the experiment.

KW - METIS-304279

KW - IR-91418

U2 - 10.1109/MoDRE.2014.6890826

DO - 10.1109/MoDRE.2014.6890826

M3 - Conference contribution

SN - 978-1-4799-6343-0

BT - 2014 IEEE 4th International Model-Driven Requirements Engineering Workshop (MoDRE)

A2 - Moreira, Ana

A2 - Sánchez, Pablo

A2 - Mussbacher, Gunter

A2 - Araújo, Joao

PB - IEEE

CY - Piscataway, NJ

Y2 - 25 August 2014 through 25 August 2014

ER -

Goknil A, van Domburg R, Kurtev I, van den Berg K, Wijnhoven F. Experimental Evaluation of a Tool for Change Impact Prediction in Requirements Models: Design, Results and Lessons Learned. In Moreira A, Sánchez P, Mussbacher G, Araújo J, editors, 2014 IEEE 4th International Model-Driven Requirements Engineering Workshop (MoDRE): Proceedings. Piscataway, NJ: IEEE. 2014 doi: 10.1109/MoDRE.2014.6890826

Experimental Evaluation of a Tool for Change Impact Prediction in Requirements Models: Design, Results and Lessons Learned

Abstract

Conference

Keywords

Access to Document

Fingerprint

Cite this