A Semantic Framework for Test Coverage

Laura Brandan Briones, Ed Brinksma, Mariëlle Stoelinga

Research output: Book/ReportReport

  • 10 Citations

Abstract

Since testing is inherently incomplete, test selection has vital importance. Coverage measures evaluate the quality of a test suite and help the tester select test cases with maximal impact at minimum cost. Existing coverage criteria for test suites are usually defined in terms of syntactic characteristics of the implementation under test or its specification. Typical black-box coverage metrics are state and transition coverage of the specification. White-box testing often considers statement, condition and path coverage. A disadvantage of this syntactic approach is that different coverage figures are assigned to systems that are behaviorally equivalent, but syntactically different. Moreover, those coverage metrics do not take into account that certain failures are more severe than others, and that more testing effort should be devoted to uncover the most important bugs, while less critical system parts can be tested less thoroughly. This paper introduces a semantic approach to black box test coverage. Our starting point is a weighted fault model (or WFM), which augments a specification by assigning a weight to each error that may occur in an implementation. We define a framework to express coverage measures that express how well a test suite covers such a specification, taking into account the error weight. Since our notions are semantic, they are insensitive to replacing a specification by one with equivalent behaviour. We present several algorithms that, given a certain inimality criterion, compute a minimal test suite with maximal coverage. These algorithms work on a syntactic representation of WFMs as fault automata. They are based on existing and novel optimization problems. Finally, we illustrate our approach by analyzing and comparing a number of test suites for a chat protocol.
LanguageUndefined
Place of PublicationEnschede
PublisherCentre for Telematics and Information Technology (CTIT)
Number of pages31
StatePublished - 10 Dec 2006

Publication series

NameCTIT Technical Report Series
PublisherCentre for Telematics and Information Technology (CTIT), University of Twente
No.06-24
ISSN (Print)1381-3625

Keywords

  • IR-59907
  • EWI-9435
  • METIS-238789

Cite this

Brandan Briones, L., Brinksma, E., & Stoelinga, M. (2006). A Semantic Framework for Test Coverage. (CTIT Technical Report Series; No. 06-24). Enschede: Centre for Telematics and Information Technology (CTIT).
Brandan Briones, Laura ; Brinksma, Ed ; Stoelinga, Mariëlle. / A Semantic Framework for Test Coverage. Enschede : Centre for Telematics and Information Technology (CTIT), 2006. 31 p. (CTIT Technical Report Series; 06-24).
@book{c368385d6950490f9f960930b26891a2,
title = "A Semantic Framework for Test Coverage",
abstract = "Since testing is inherently incomplete, test selection has vital importance. Coverage measures evaluate the quality of a test suite and help the tester select test cases with maximal impact at minimum cost. Existing coverage criteria for test suites are usually defined in terms of syntactic characteristics of the implementation under test or its specification. Typical black-box coverage metrics are state and transition coverage of the specification. White-box testing often considers statement, condition and path coverage. A disadvantage of this syntactic approach is that different coverage figures are assigned to systems that are behaviorally equivalent, but syntactically different. Moreover, those coverage metrics do not take into account that certain failures are more severe than others, and that more testing effort should be devoted to uncover the most important bugs, while less critical system parts can be tested less thoroughly. This paper introduces a semantic approach to black box test coverage. Our starting point is a weighted fault model (or WFM), which augments a specification by assigning a weight to each error that may occur in an implementation. We define a framework to express coverage measures that express how well a test suite covers such a specification, taking into account the error weight. Since our notions are semantic, they are insensitive to replacing a specification by one with equivalent behaviour. We present several algorithms that, given a certain inimality criterion, compute a minimal test suite with maximal coverage. These algorithms work on a syntactic representation of WFMs as fault automata. They are based on existing and novel optimization problems. Finally, we illustrate our approach by analyzing and comparing a number of test suites for a chat protocol.",
keywords = "IR-59907, EWI-9435, METIS-238789",
author = "{Brandan Briones}, Laura and Ed Brinksma and Mari{\"e}lle Stoelinga",
year = "2006",
month = "12",
day = "10",
language = "Undefined",
series = "CTIT Technical Report Series",
publisher = "Centre for Telematics and Information Technology (CTIT)",
number = "06-24",
address = "Netherlands",

}

Brandan Briones, L, Brinksma, E & Stoelinga, M 2006, A Semantic Framework for Test Coverage. CTIT Technical Report Series, no. 06-24, Centre for Telematics and Information Technology (CTIT), Enschede.

A Semantic Framework for Test Coverage. / Brandan Briones, Laura; Brinksma, Ed; Stoelinga, Mariëlle.

Enschede : Centre for Telematics and Information Technology (CTIT), 2006. 31 p. (CTIT Technical Report Series; No. 06-24).

Research output: Book/ReportReport

TY - BOOK

T1 - A Semantic Framework for Test Coverage

AU - Brandan Briones,Laura

AU - Brinksma,Ed

AU - Stoelinga,Mariëlle

PY - 2006/12/10

Y1 - 2006/12/10

N2 - Since testing is inherently incomplete, test selection has vital importance. Coverage measures evaluate the quality of a test suite and help the tester select test cases with maximal impact at minimum cost. Existing coverage criteria for test suites are usually defined in terms of syntactic characteristics of the implementation under test or its specification. Typical black-box coverage metrics are state and transition coverage of the specification. White-box testing often considers statement, condition and path coverage. A disadvantage of this syntactic approach is that different coverage figures are assigned to systems that are behaviorally equivalent, but syntactically different. Moreover, those coverage metrics do not take into account that certain failures are more severe than others, and that more testing effort should be devoted to uncover the most important bugs, while less critical system parts can be tested less thoroughly. This paper introduces a semantic approach to black box test coverage. Our starting point is a weighted fault model (or WFM), which augments a specification by assigning a weight to each error that may occur in an implementation. We define a framework to express coverage measures that express how well a test suite covers such a specification, taking into account the error weight. Since our notions are semantic, they are insensitive to replacing a specification by one with equivalent behaviour. We present several algorithms that, given a certain inimality criterion, compute a minimal test suite with maximal coverage. These algorithms work on a syntactic representation of WFMs as fault automata. They are based on existing and novel optimization problems. Finally, we illustrate our approach by analyzing and comparing a number of test suites for a chat protocol.

AB - Since testing is inherently incomplete, test selection has vital importance. Coverage measures evaluate the quality of a test suite and help the tester select test cases with maximal impact at minimum cost. Existing coverage criteria for test suites are usually defined in terms of syntactic characteristics of the implementation under test or its specification. Typical black-box coverage metrics are state and transition coverage of the specification. White-box testing often considers statement, condition and path coverage. A disadvantage of this syntactic approach is that different coverage figures are assigned to systems that are behaviorally equivalent, but syntactically different. Moreover, those coverage metrics do not take into account that certain failures are more severe than others, and that more testing effort should be devoted to uncover the most important bugs, while less critical system parts can be tested less thoroughly. This paper introduces a semantic approach to black box test coverage. Our starting point is a weighted fault model (or WFM), which augments a specification by assigning a weight to each error that may occur in an implementation. We define a framework to express coverage measures that express how well a test suite covers such a specification, taking into account the error weight. Since our notions are semantic, they are insensitive to replacing a specification by one with equivalent behaviour. We present several algorithms that, given a certain inimality criterion, compute a minimal test suite with maximal coverage. These algorithms work on a syntactic representation of WFMs as fault automata. They are based on existing and novel optimization problems. Finally, we illustrate our approach by analyzing and comparing a number of test suites for a chat protocol.

KW - IR-59907

KW - EWI-9435

KW - METIS-238789

M3 - Report

T3 - CTIT Technical Report Series

BT - A Semantic Framework for Test Coverage

PB - Centre for Telematics and Information Technology (CTIT)

CY - Enschede

ER -

Brandan Briones L, Brinksma E, Stoelinga M. A Semantic Framework for Test Coverage. Enschede: Centre for Telematics and Information Technology (CTIT), 2006. 31 p. (CTIT Technical Report Series; 06-24).