Interpreting a Successful Testing Process: Risk and Actual Coverage

Research output: Chapter in Book/Report/Conference proceedingConference contributionAcademicpeer-review

Abstract

Testing is inherently incomplete; no test suite will ever be able to test all possible usage scenarios of a system. It is therefore vital to assess the implication of a system passing a test suite. This paper quantifies that implication by means of two distinct, but related, measures: the risk quantifies the confidence in a system after it passes a test suite, i.e., the number of faults still expected to be present (weighted by their severity); the actual coverage quantifies the extent to which faults have been shown absent, i.e., the fraction of possible faults that has been covered. We provide evaluation algorithms that calculate these metrics for a given test suite, as well as optimisation algorithms that yield the best test suite for a given optimisation criterion.
Original languageEnglish
Title of host publicationProceedings of the 3rd IEEE International Symposium on Theoretical Aspects of Software Engineering (TASE)
EditorsW.N. Chin, S. Qin
Place of PublicationLos Alamitos
PublisherIEEE Computer Society
Pages251-258
Number of pages8
ISBN (Print)978-0-7695-3757-3
DOIs
Publication statusPublished - Jul 2009
Event3rd IEEE International Symposium on Theoretical Aspects of Software Engineering, TASE 2009 - Tianjin, China
Duration: 29 Jul 200931 Jul 2009
Conference number: 3
https://www.scm.tees.ac.uk/s.qin/tase2009/

Conference

Conference3rd IEEE International Symposium on Theoretical Aspects of Software Engineering, TASE 2009
Abbreviated titleTASE
CountryChina
CityTianjin
Period29/07/0931/07/09
Internet address

Fingerprint

Testing

Keywords

  • METIS-263966
  • IR-67564
  • Risk
  • Coverage
  • EC Grant Agreement nr.: FP7-ICT-2007-1
  • Probabilistic
  • EWI-15837
  • EC Grant Agreement nr.: FP7/214755
  • Formal testing

Cite this

Stoelinga, M. I. A., & Timmer, M. (2009). Interpreting a Successful Testing Process: Risk and Actual Coverage. In W. N. Chin, & S. Qin (Eds.), Proceedings of the 3rd IEEE International Symposium on Theoretical Aspects of Software Engineering (TASE) (pp. 251-258). Los Alamitos: IEEE Computer Society. https://doi.org/10.1109/TASE.2009.26
Stoelinga, Mariëlle Ida Antoinette ; Timmer, Mark. / Interpreting a Successful Testing Process: Risk and Actual Coverage. Proceedings of the 3rd IEEE International Symposium on Theoretical Aspects of Software Engineering (TASE). editor / W.N. Chin ; S. Qin. Los Alamitos : IEEE Computer Society, 2009. pp. 251-258
@inproceedings{6f6e892322144c39a3b019ba7a79dc10,
title = "Interpreting a Successful Testing Process: Risk and Actual Coverage",
abstract = "Testing is inherently incomplete; no test suite will ever be able to test all possible usage scenarios of a system. It is therefore vital to assess the implication of a system passing a test suite. This paper quantifies that implication by means of two distinct, but related, measures: the risk quantifies the confidence in a system after it passes a test suite, i.e., the number of faults still expected to be present (weighted by their severity); the actual coverage quantifies the extent to which faults have been shown absent, i.e., the fraction of possible faults that has been covered. We provide evaluation algorithms that calculate these metrics for a given test suite, as well as optimisation algorithms that yield the best test suite for a given optimisation criterion.",
keywords = "METIS-263966, IR-67564, Risk, Coverage, EC Grant Agreement nr.: FP7-ICT-2007-1, Probabilistic, EWI-15837, EC Grant Agreement nr.: FP7/214755, Formal testing",
author = "Stoelinga, {Mari{\"e}lle Ida Antoinette} and Mark Timmer",
note = "eemcs-eprint-15837",
year = "2009",
month = "7",
doi = "10.1109/TASE.2009.26",
language = "English",
isbn = "978-0-7695-3757-3",
pages = "251--258",
editor = "W.N. Chin and S. Qin",
booktitle = "Proceedings of the 3rd IEEE International Symposium on Theoretical Aspects of Software Engineering (TASE)",
publisher = "IEEE Computer Society",
address = "United States",

}

Stoelinga, MIA & Timmer, M 2009, Interpreting a Successful Testing Process: Risk and Actual Coverage. in WN Chin & S Qin (eds), Proceedings of the 3rd IEEE International Symposium on Theoretical Aspects of Software Engineering (TASE). IEEE Computer Society, Los Alamitos, pp. 251-258, 3rd IEEE International Symposium on Theoretical Aspects of Software Engineering, TASE 2009, Tianjin, China, 29/07/09. https://doi.org/10.1109/TASE.2009.26

Interpreting a Successful Testing Process: Risk and Actual Coverage. / Stoelinga, Mariëlle Ida Antoinette; Timmer, Mark.

Proceedings of the 3rd IEEE International Symposium on Theoretical Aspects of Software Engineering (TASE). ed. / W.N. Chin; S. Qin. Los Alamitos : IEEE Computer Society, 2009. p. 251-258.

Research output: Chapter in Book/Report/Conference proceedingConference contributionAcademicpeer-review

TY - GEN

T1 - Interpreting a Successful Testing Process: Risk and Actual Coverage

AU - Stoelinga, Mariëlle Ida Antoinette

AU - Timmer, Mark

N1 - eemcs-eprint-15837

PY - 2009/7

Y1 - 2009/7

N2 - Testing is inherently incomplete; no test suite will ever be able to test all possible usage scenarios of a system. It is therefore vital to assess the implication of a system passing a test suite. This paper quantifies that implication by means of two distinct, but related, measures: the risk quantifies the confidence in a system after it passes a test suite, i.e., the number of faults still expected to be present (weighted by their severity); the actual coverage quantifies the extent to which faults have been shown absent, i.e., the fraction of possible faults that has been covered. We provide evaluation algorithms that calculate these metrics for a given test suite, as well as optimisation algorithms that yield the best test suite for a given optimisation criterion.

AB - Testing is inherently incomplete; no test suite will ever be able to test all possible usage scenarios of a system. It is therefore vital to assess the implication of a system passing a test suite. This paper quantifies that implication by means of two distinct, but related, measures: the risk quantifies the confidence in a system after it passes a test suite, i.e., the number of faults still expected to be present (weighted by their severity); the actual coverage quantifies the extent to which faults have been shown absent, i.e., the fraction of possible faults that has been covered. We provide evaluation algorithms that calculate these metrics for a given test suite, as well as optimisation algorithms that yield the best test suite for a given optimisation criterion.

KW - METIS-263966

KW - IR-67564

KW - Risk

KW - Coverage

KW - EC Grant Agreement nr.: FP7-ICT-2007-1

KW - Probabilistic

KW - EWI-15837

KW - EC Grant Agreement nr.: FP7/214755

KW - Formal testing

U2 - 10.1109/TASE.2009.26

DO - 10.1109/TASE.2009.26

M3 - Conference contribution

SN - 978-0-7695-3757-3

SP - 251

EP - 258

BT - Proceedings of the 3rd IEEE International Symposium on Theoretical Aspects of Software Engineering (TASE)

A2 - Chin, W.N.

A2 - Qin, S.

PB - IEEE Computer Society

CY - Los Alamitos

ER -

Stoelinga MIA, Timmer M. Interpreting a Successful Testing Process: Risk and Actual Coverage. In Chin WN, Qin S, editors, Proceedings of the 3rd IEEE International Symposium on Theoretical Aspects of Software Engineering (TASE). Los Alamitos: IEEE Computer Society. 2009. p. 251-258 https://doi.org/10.1109/TASE.2009.26