Interpreting a Successful Testing Process: Risk and Actual Coverage

Mariëlle Ida Antoinette Stoelinga; Mark Timmer

doi:10.1109/TASE.2009.26

Interpreting a Successful Testing Process: Risk and Actual Coverage

Mariëlle Ida Antoinette Stoelinga, Mark Timmer

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › Academic › peer-review

Abstract

Testing is inherently incomplete; no test suite will ever be able to test all possible usage scenarios of a system. It is therefore vital to assess the implication of a system passing a test suite. This paper quantifies that implication by means of two distinct, but related, measures: the risk quantifies the confidence in a system after it passes a test suite, i.e., the number of faults still expected to be present (weighted by their severity); the actual coverage quantifies the extent to which faults have been shown absent, i.e., the fraction of possible faults that has been covered. We provide evaluation algorithms that calculate these metrics for a given test suite, as well as optimisation algorithms that yield the best test suite for a given optimisation criterion.

Original language	English
Title of host publication	Proceedings of the 3rd IEEE International Symposium on Theoretical Aspects of Software Engineering (TASE)
Editors	W.N. Chin, S. Qin
Place of Publication	Los Alamitos
Publisher	IEEE Computer Society
Pages	251-258
Number of pages	8
ISBN (Print)	978-0-7695-3757-3
DOIs	https://doi.org/10.1109/TASE.2009.26
Publication status	Published - Jul 2009
Event	3rd IEEE International Symposium on Theoretical Aspects of Software Engineering, TASE 2009 - Tianjin, China Duration: 29 Jul 2009 → 31 Jul 2009 Conference number: 3 https://www.scm.tees.ac.uk/s.qin/tase2009/

Conference

Conference	3rd IEEE International Symposium on Theoretical Aspects of Software Engineering, TASE 2009
Abbreviated title	TASE
Country	China
City	Tianjin
Period	29/07/09 → 31/07/09
Internet address	https://www.scm.tees.ac.uk/s.qin/tase2009/

Keywords

METIS-263966
IR-67564
Risk
Coverage
EC Grant Agreement nr.: FP7-ICT-2007-1
Probabilistic
EWI-15837
EC Grant Agreement nr.: FP7/214755
Formal testing

Access to Document

10.1109/TASE.2009.26

Fingerprint Dive into the research topics of 'Interpreting a Successful Testing Process: Risk and Actual Coverage'. Together they form a unique fingerprint.

Cite this

@inproceedings{6f6e892322144c39a3b019ba7a79dc10,

title = "Interpreting a Successful Testing Process: Risk and Actual Coverage",

abstract = "Testing is inherently incomplete; no test suite will ever be able to test all possible usage scenarios of a system. It is therefore vital to assess the implication of a system passing a test suite. This paper quantifies that implication by means of two distinct, but related, measures: the risk quantifies the confidence in a system after it passes a test suite, i.e., the number of faults still expected to be present (weighted by their severity); the actual coverage quantifies the extent to which faults have been shown absent, i.e., the fraction of possible faults that has been covered. We provide evaluation algorithms that calculate these metrics for a given test suite, as well as optimisation algorithms that yield the best test suite for a given optimisation criterion.",

keywords = "METIS-263966, IR-67564, Risk, Coverage, EC Grant Agreement nr.: FP7-ICT-2007-1, Probabilistic, EWI-15837, EC Grant Agreement nr.: FP7/214755, Formal testing",

author = "Stoelinga, {Mari{\"e}lle Ida Antoinette} and Mark Timmer",

note = "eemcs-eprint-15837 ; 3rd IEEE International Symposium on Theoretical Aspects of Software Engineering, TASE 2009, TASE ; Conference date: 29-07-2009 Through 31-07-2009",

year = "2009",

month = jul,

doi = "10.1109/TASE.2009.26",

language = "English",

isbn = "978-0-7695-3757-3",

pages = "251--258",

editor = "W.N. Chin and S. Qin",

booktitle = "Proceedings of the 3rd IEEE International Symposium on Theoretical Aspects of Software Engineering (TASE)",

publisher = "IEEE Computer Society",

address = "United States",

url = "https://www.scm.tees.ac.uk/s.qin/tase2009/",

}

Stoelinga, MIA & Timmer, M 2009, Interpreting a Successful Testing Process: Risk and Actual Coverage. in WN Chin & S Qin (eds), Proceedings of the 3rd IEEE International Symposium on Theoretical Aspects of Software Engineering (TASE). IEEE Computer Society, Los Alamitos, pp. 251-258, 3rd IEEE International Symposium on Theoretical Aspects of Software Engineering, TASE 2009, Tianjin, China, 29/07/09. https://doi.org/10.1109/TASE.2009.26

Interpreting a Successful Testing Process: Risk and Actual Coverage. / Stoelinga, Mariëlle Ida Antoinette ; Timmer, Mark.

Proceedings of the 3rd IEEE International Symposium on Theoretical Aspects of Software Engineering (TASE). ed. / W.N. Chin; S. Qin. Los Alamitos : IEEE Computer Society, 2009. p. 251-258.

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › Academic › peer-review

TY - GEN

T1 - Interpreting a Successful Testing Process: Risk and Actual Coverage

AU - Stoelinga, Mariëlle Ida Antoinette

AU - Timmer, Mark

N1 - Conference code: 3

PY - 2009/7

Y1 - 2009/7

N2 - Testing is inherently incomplete; no test suite will ever be able to test all possible usage scenarios of a system. It is therefore vital to assess the implication of a system passing a test suite. This paper quantifies that implication by means of two distinct, but related, measures: the risk quantifies the confidence in a system after it passes a test suite, i.e., the number of faults still expected to be present (weighted by their severity); the actual coverage quantifies the extent to which faults have been shown absent, i.e., the fraction of possible faults that has been covered. We provide evaluation algorithms that calculate these metrics for a given test suite, as well as optimisation algorithms that yield the best test suite for a given optimisation criterion.

AB - Testing is inherently incomplete; no test suite will ever be able to test all possible usage scenarios of a system. It is therefore vital to assess the implication of a system passing a test suite. This paper quantifies that implication by means of two distinct, but related, measures: the risk quantifies the confidence in a system after it passes a test suite, i.e., the number of faults still expected to be present (weighted by their severity); the actual coverage quantifies the extent to which faults have been shown absent, i.e., the fraction of possible faults that has been covered. We provide evaluation algorithms that calculate these metrics for a given test suite, as well as optimisation algorithms that yield the best test suite for a given optimisation criterion.

KW - METIS-263966

KW - IR-67564

KW - Risk

KW - Coverage

KW - EC Grant Agreement nr.: FP7-ICT-2007-1

KW - Probabilistic

KW - EWI-15837

KW - EC Grant Agreement nr.: FP7/214755

KW - Formal testing

U2 - 10.1109/TASE.2009.26

DO - 10.1109/TASE.2009.26

M3 - Conference contribution

SN - 978-0-7695-3757-3

SP - 251

EP - 258

BT - Proceedings of the 3rd IEEE International Symposium on Theoretical Aspects of Software Engineering (TASE)

A2 - Chin, W.N.

A2 - Qin, S.

PB - IEEE Computer Society

CY - Los Alamitos

T2 - 3rd IEEE International Symposium on Theoretical Aspects of Software Engineering, TASE 2009

Y2 - 29 July 2009 through 31 July 2009

ER -