Interpreting a Successful Testing Process: Risk and Actual Coverage

    Research output: Chapter in Book/Report/Conference proceedingConference contributionAcademicpeer-review

    Abstract

    Testing is inherently incomplete; no test suite will ever be able to test all possible usage scenarios of a system. It is therefore vital to assess the implication of a system passing a test suite. This paper quantifies that implication by means of two distinct, but related, measures: the risk quantifies the confidence in a system after it passes a test suite, i.e., the number of faults still expected to be present (weighted by their severity); the actual coverage quantifies the extent to which faults have been shown absent, i.e., the fraction of possible faults that has been covered. We provide evaluation algorithms that calculate these metrics for a given test suite, as well as optimisation algorithms that yield the best test suite for a given optimisation criterion.
    Original languageEnglish
    Title of host publicationProceedings of the 3rd IEEE International Symposium on Theoretical Aspects of Software Engineering (TASE)
    EditorsW.N. Chin, S. Qin
    Place of PublicationLos Alamitos
    PublisherIEEE Computer Society
    Pages251-258
    Number of pages8
    ISBN (Print)978-0-7695-3757-3
    DOIs
    Publication statusPublished - Jul 2009
    Event3rd IEEE International Symposium on Theoretical Aspects of Software Engineering, TASE 2009 - Tianjin, China
    Duration: 29 Jul 200931 Jul 2009
    Conference number: 3
    https://www.scm.tees.ac.uk/s.qin/tase2009/

    Conference

    Conference3rd IEEE International Symposium on Theoretical Aspects of Software Engineering, TASE 2009
    Abbreviated titleTASE
    CountryChina
    CityTianjin
    Period29/07/0931/07/09
    Internet address

    Fingerprint

    Testing

    Keywords

    • METIS-263966
    • IR-67564
    • Risk
    • Coverage
    • EC Grant Agreement nr.: FP7-ICT-2007-1
    • Probabilistic
    • EWI-15837
    • EC Grant Agreement nr.: FP7/214755
    • Formal testing

    Cite this

    Stoelinga, M. I. A., & Timmer, M. (2009). Interpreting a Successful Testing Process: Risk and Actual Coverage. In W. N. Chin, & S. Qin (Eds.), Proceedings of the 3rd IEEE International Symposium on Theoretical Aspects of Software Engineering (TASE) (pp. 251-258). Los Alamitos: IEEE Computer Society. https://doi.org/10.1109/TASE.2009.26
    Stoelinga, Mariëlle Ida Antoinette ; Timmer, Mark. / Interpreting a Successful Testing Process: Risk and Actual Coverage. Proceedings of the 3rd IEEE International Symposium on Theoretical Aspects of Software Engineering (TASE). editor / W.N. Chin ; S. Qin. Los Alamitos : IEEE Computer Society, 2009. pp. 251-258
    @inproceedings{6f6e892322144c39a3b019ba7a79dc10,
    title = "Interpreting a Successful Testing Process: Risk and Actual Coverage",
    abstract = "Testing is inherently incomplete; no test suite will ever be able to test all possible usage scenarios of a system. It is therefore vital to assess the implication of a system passing a test suite. This paper quantifies that implication by means of two distinct, but related, measures: the risk quantifies the confidence in a system after it passes a test suite, i.e., the number of faults still expected to be present (weighted by their severity); the actual coverage quantifies the extent to which faults have been shown absent, i.e., the fraction of possible faults that has been covered. We provide evaluation algorithms that calculate these metrics for a given test suite, as well as optimisation algorithms that yield the best test suite for a given optimisation criterion.",
    keywords = "METIS-263966, IR-67564, Risk, Coverage, EC Grant Agreement nr.: FP7-ICT-2007-1, Probabilistic, EWI-15837, EC Grant Agreement nr.: FP7/214755, Formal testing",
    author = "Stoelinga, {Mari{\"e}lle Ida Antoinette} and Mark Timmer",
    note = "eemcs-eprint-15837",
    year = "2009",
    month = "7",
    doi = "10.1109/TASE.2009.26",
    language = "English",
    isbn = "978-0-7695-3757-3",
    pages = "251--258",
    editor = "W.N. Chin and S. Qin",
    booktitle = "Proceedings of the 3rd IEEE International Symposium on Theoretical Aspects of Software Engineering (TASE)",
    publisher = "IEEE Computer Society",
    address = "United States",

    }

    Stoelinga, MIA & Timmer, M 2009, Interpreting a Successful Testing Process: Risk and Actual Coverage. in WN Chin & S Qin (eds), Proceedings of the 3rd IEEE International Symposium on Theoretical Aspects of Software Engineering (TASE). IEEE Computer Society, Los Alamitos, pp. 251-258, 3rd IEEE International Symposium on Theoretical Aspects of Software Engineering, TASE 2009, Tianjin, China, 29/07/09. https://doi.org/10.1109/TASE.2009.26

    Interpreting a Successful Testing Process: Risk and Actual Coverage. / Stoelinga, Mariëlle Ida Antoinette; Timmer, Mark.

    Proceedings of the 3rd IEEE International Symposium on Theoretical Aspects of Software Engineering (TASE). ed. / W.N. Chin; S. Qin. Los Alamitos : IEEE Computer Society, 2009. p. 251-258.

    Research output: Chapter in Book/Report/Conference proceedingConference contributionAcademicpeer-review

    TY - GEN

    T1 - Interpreting a Successful Testing Process: Risk and Actual Coverage

    AU - Stoelinga, Mariëlle Ida Antoinette

    AU - Timmer, Mark

    N1 - eemcs-eprint-15837

    PY - 2009/7

    Y1 - 2009/7

    N2 - Testing is inherently incomplete; no test suite will ever be able to test all possible usage scenarios of a system. It is therefore vital to assess the implication of a system passing a test suite. This paper quantifies that implication by means of two distinct, but related, measures: the risk quantifies the confidence in a system after it passes a test suite, i.e., the number of faults still expected to be present (weighted by their severity); the actual coverage quantifies the extent to which faults have been shown absent, i.e., the fraction of possible faults that has been covered. We provide evaluation algorithms that calculate these metrics for a given test suite, as well as optimisation algorithms that yield the best test suite for a given optimisation criterion.

    AB - Testing is inherently incomplete; no test suite will ever be able to test all possible usage scenarios of a system. It is therefore vital to assess the implication of a system passing a test suite. This paper quantifies that implication by means of two distinct, but related, measures: the risk quantifies the confidence in a system after it passes a test suite, i.e., the number of faults still expected to be present (weighted by their severity); the actual coverage quantifies the extent to which faults have been shown absent, i.e., the fraction of possible faults that has been covered. We provide evaluation algorithms that calculate these metrics for a given test suite, as well as optimisation algorithms that yield the best test suite for a given optimisation criterion.

    KW - METIS-263966

    KW - IR-67564

    KW - Risk

    KW - Coverage

    KW - EC Grant Agreement nr.: FP7-ICT-2007-1

    KW - Probabilistic

    KW - EWI-15837

    KW - EC Grant Agreement nr.: FP7/214755

    KW - Formal testing

    U2 - 10.1109/TASE.2009.26

    DO - 10.1109/TASE.2009.26

    M3 - Conference contribution

    SN - 978-0-7695-3757-3

    SP - 251

    EP - 258

    BT - Proceedings of the 3rd IEEE International Symposium on Theoretical Aspects of Software Engineering (TASE)

    A2 - Chin, W.N.

    A2 - Qin, S.

    PB - IEEE Computer Society

    CY - Los Alamitos

    ER -

    Stoelinga MIA, Timmer M. Interpreting a Successful Testing Process: Risk and Actual Coverage. In Chin WN, Qin S, editors, Proceedings of the 3rd IEEE International Symposium on Theoretical Aspects of Software Engineering (TASE). Los Alamitos: IEEE Computer Society. 2009. p. 251-258 https://doi.org/10.1109/TASE.2009.26