Interpreting a Successful Testing Process: Risk and Actual Coverage

Research output: Book/ReportReportAcademic

2 Citations (Scopus)
30 Downloads (Pure)

Abstract

Testing is inherently incomplete; no test suite will ever be able to test all possible usage scenarios of a system. It is therefore vital to assess the implication of a system passing a test suite. This paper quantifies that implication by means of two distinct, but related, measures: the risk quantifies the confidence in a system after it passes a test suite, i.e., the number of faults still expected to be present (weighted by their severity); the actual coverage quantifies the extent to which faults have been shown absent, i.e., the fraction of possible faults that has been covered. We provide evaluation algorithms that calculate these metrics for a given test suite, as well as optimisation algorithms that yield the best test suite for a given optimisation criterion.
Original languageUndefined
Place of PublicationEnschede
PublisherCentre for Telematics and Information Technology (CTIT)
Number of pages24
Publication statusPublished - 7 May 2009

Publication series

NameCTIT Technical Report Series
PublisherUniversity of Twente, Centre for Telematica and Information Technology (CTIT)
No.TR-CTIT-09-17
ISSN (Print)1381-3625

Keywords

  • EWI-15330
  • Risk
  • Formal testing
  • METIS-263837
  • Coverage
  • Probabilistic
  • IR-65483

Cite this

Stoelinga, M. I. A., & Timmer, M. (2009). Interpreting a Successful Testing Process: Risk and Actual Coverage. (CTIT Technical Report Series; No. TR-CTIT-09-17). Enschede: Centre for Telematics and Information Technology (CTIT).
Stoelinga, Mariëlle Ida Antoinette ; Timmer, Mark. / Interpreting a Successful Testing Process: Risk and Actual Coverage. Enschede : Centre for Telematics and Information Technology (CTIT), 2009. 24 p. (CTIT Technical Report Series; TR-CTIT-09-17).
@book{c3e2d1d8cf404e1590c342ab5f6d48d0,
title = "Interpreting a Successful Testing Process: Risk and Actual Coverage",
abstract = "Testing is inherently incomplete; no test suite will ever be able to test all possible usage scenarios of a system. It is therefore vital to assess the implication of a system passing a test suite. This paper quantifies that implication by means of two distinct, but related, measures: the risk quantifies the confidence in a system after it passes a test suite, i.e., the number of faults still expected to be present (weighted by their severity); the actual coverage quantifies the extent to which faults have been shown absent, i.e., the fraction of possible faults that has been covered. We provide evaluation algorithms that calculate these metrics for a given test suite, as well as optimisation algorithms that yield the best test suite for a given optimisation criterion.",
keywords = "EWI-15330, Risk, Formal testing, METIS-263837, Coverage, Probabilistic, IR-65483",
author = "Stoelinga, {Mari{\"e}lle Ida Antoinette} and Mark Timmer",
note = "eemcs-eprint-15330",
year = "2009",
month = "5",
day = "7",
language = "Undefined",
series = "CTIT Technical Report Series",
publisher = "Centre for Telematics and Information Technology (CTIT)",
number = "TR-CTIT-09-17",
address = "Netherlands",

}

Stoelinga, MIA & Timmer, M 2009, Interpreting a Successful Testing Process: Risk and Actual Coverage. CTIT Technical Report Series, no. TR-CTIT-09-17, Centre for Telematics and Information Technology (CTIT), Enschede.

Interpreting a Successful Testing Process: Risk and Actual Coverage. / Stoelinga, Mariëlle Ida Antoinette; Timmer, Mark.

Enschede : Centre for Telematics and Information Technology (CTIT), 2009. 24 p. (CTIT Technical Report Series; No. TR-CTIT-09-17).

Research output: Book/ReportReportAcademic

TY - BOOK

T1 - Interpreting a Successful Testing Process: Risk and Actual Coverage

AU - Stoelinga, Mariëlle Ida Antoinette

AU - Timmer, Mark

N1 - eemcs-eprint-15330

PY - 2009/5/7

Y1 - 2009/5/7

N2 - Testing is inherently incomplete; no test suite will ever be able to test all possible usage scenarios of a system. It is therefore vital to assess the implication of a system passing a test suite. This paper quantifies that implication by means of two distinct, but related, measures: the risk quantifies the confidence in a system after it passes a test suite, i.e., the number of faults still expected to be present (weighted by their severity); the actual coverage quantifies the extent to which faults have been shown absent, i.e., the fraction of possible faults that has been covered. We provide evaluation algorithms that calculate these metrics for a given test suite, as well as optimisation algorithms that yield the best test suite for a given optimisation criterion.

AB - Testing is inherently incomplete; no test suite will ever be able to test all possible usage scenarios of a system. It is therefore vital to assess the implication of a system passing a test suite. This paper quantifies that implication by means of two distinct, but related, measures: the risk quantifies the confidence in a system after it passes a test suite, i.e., the number of faults still expected to be present (weighted by their severity); the actual coverage quantifies the extent to which faults have been shown absent, i.e., the fraction of possible faults that has been covered. We provide evaluation algorithms that calculate these metrics for a given test suite, as well as optimisation algorithms that yield the best test suite for a given optimisation criterion.

KW - EWI-15330

KW - Risk

KW - Formal testing

KW - METIS-263837

KW - Coverage

KW - Probabilistic

KW - IR-65483

M3 - Report

T3 - CTIT Technical Report Series

BT - Interpreting a Successful Testing Process: Risk and Actual Coverage

PB - Centre for Telematics and Information Technology (CTIT)

CY - Enschede

ER -

Stoelinga MIA, Timmer M. Interpreting a Successful Testing Process: Risk and Actual Coverage. Enschede: Centre for Telematics and Information Technology (CTIT), 2009. 24 p. (CTIT Technical Report Series; TR-CTIT-09-17).