A Semantic Framework for Test Coverage

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

Since testing is inherently incomplete, test selection has vital importance. Coverage measures evaluate the quality of a test suite and help the tester select test cases with maximal impact at minimum cost. Existing coverage criteria for test suites are usually defined in terms of syntactic characteristics of the implementation under test or its specification. Typical black-box coverage metrics are state and transition coverage of the specification. White-box testing often considers statement, condition and path coverage. A disadvantage of this syntactic approach is that different coverage figures are assigned to systems that are behaviorally equivalent, but syntactically different. Moreover, those coverage metrics do not take into account that certain failures are more severe than others, and that more testing effort should be devoted to uncover the most important bugs, while less critical system parts can be tested less thoroughly. This paper introduces a semantic approach to black box test coverage. Our starting point is a weighted fault model (or WFM), which augments a specification by assigning a weight to each error that may occur in an implementation. We define a framework to express coverage measures that express how well a test suite covers such a specification, taking into account the error weight. Since our notions are semantic, they are insensitive to replacing a specification by one with equivalent behaviour. We present several algorithms that, given a certain minimality criterion, compute a minimal test suite with maximal coverage. These algorithms work on a syntactic representation of WFMs as fault automata. They are based on existing and novel optimization problems. Finally, we illustrate our approach by analyzing and comparing a number of test suites for a chat protocol.
LanguageEnglish
Title of host publicationAutomated Technology for Verification and Analysis
Subtitle of host publication4th International Symposium, ATVA 2006, Beijing, China, October 23-26, 2006. Proceedings
EditorsSusanne Graf, Wenhui Zhang
Place of PublicationBerlin
PublisherSpringer Verlag
Pages399-414
Number of pages16
ISBN (Print)978-3-540-47237-7
DOIs
StatePublished - 2006
Event4th International Symposium on Automated Technology for Verification and Analysis, ATVA 2006 - Beijing, China
Duration: 23 Oct 200626 Oct 2006
Conference number: 4

Publication series

NameLecture Notes in Computer Science
PublisherSpringer Verlag
Volume4218
ISSN (Print)0302-9743
ISSN (Electronic)1611-3349

Other

Other4th International Symposium on Automated Technology for Verification and Analysis, ATVA 2006
Abbreviated titleATVA
CountryChina
CityBeijing
Period23/10/0626/10/06

Fingerprint

Semantics
Specifications
Syntactics
Testing
Costs

Keywords

  • EWI-6937
  • METIS-238175
  • IR-59905

Cite this

Brandan Briones, L., Brinksma, H., & Stoelinga, M. I. A. (2006). A Semantic Framework for Test Coverage. In S. Graf, & W. Zhang (Eds.), Automated Technology for Verification and Analysis: 4th International Symposium, ATVA 2006, Beijing, China, October 23-26, 2006. Proceedings (pp. 399-414). (Lecture Notes in Computer Science; Vol. 4218). Berlin: Springer Verlag. DOI: 10.1007/11901914_30
Brandan Briones, L. ; Brinksma, Hendrik ; Stoelinga, Mariëlle Ida Antoinette. / A Semantic Framework for Test Coverage. Automated Technology for Verification and Analysis: 4th International Symposium, ATVA 2006, Beijing, China, October 23-26, 2006. Proceedings. editor / Susanne Graf ; Wenhui Zhang. Berlin : Springer Verlag, 2006. pp. 399-414 (Lecture Notes in Computer Science).
@inproceedings{e20ea2f0ef0b4bbbba446b0efb58cd85,
title = "A Semantic Framework for Test Coverage",
abstract = "Since testing is inherently incomplete, test selection has vital importance. Coverage measures evaluate the quality of a test suite and help the tester select test cases with maximal impact at minimum cost. Existing coverage criteria for test suites are usually defined in terms of syntactic characteristics of the implementation under test or its specification. Typical black-box coverage metrics are state and transition coverage of the specification. White-box testing often considers statement, condition and path coverage. A disadvantage of this syntactic approach is that different coverage figures are assigned to systems that are behaviorally equivalent, but syntactically different. Moreover, those coverage metrics do not take into account that certain failures are more severe than others, and that more testing effort should be devoted to uncover the most important bugs, while less critical system parts can be tested less thoroughly. This paper introduces a semantic approach to black box test coverage. Our starting point is a weighted fault model (or WFM), which augments a specification by assigning a weight to each error that may occur in an implementation. We define a framework to express coverage measures that express how well a test suite covers such a specification, taking into account the error weight. Since our notions are semantic, they are insensitive to replacing a specification by one with equivalent behaviour. We present several algorithms that, given a certain minimality criterion, compute a minimal test suite with maximal coverage. These algorithms work on a syntactic representation of WFMs as fault automata. They are based on existing and novel optimization problems. Finally, we illustrate our approach by analyzing and comparing a number of test suites for a chat protocol.",
keywords = "EWI-6937, METIS-238175, IR-59905",
author = "{Brandan Briones}, L. and Hendrik Brinksma and Stoelinga, {Mari{\"e}lle Ida Antoinette}",
year = "2006",
doi = "10.1007/11901914_30",
language = "English",
isbn = "978-3-540-47237-7",
series = "Lecture Notes in Computer Science",
publisher = "Springer Verlag",
pages = "399--414",
editor = "Susanne Graf and Wenhui Zhang",
booktitle = "Automated Technology for Verification and Analysis",
address = "Germany",

}

Brandan Briones, L, Brinksma, H & Stoelinga, MIA 2006, A Semantic Framework for Test Coverage. in S Graf & W Zhang (eds), Automated Technology for Verification and Analysis: 4th International Symposium, ATVA 2006, Beijing, China, October 23-26, 2006. Proceedings. Lecture Notes in Computer Science, vol. 4218, Springer Verlag, Berlin, pp. 399-414, 4th International Symposium on Automated Technology for Verification and Analysis, ATVA 2006, Beijing, China, 23/10/06. DOI: 10.1007/11901914_30

A Semantic Framework for Test Coverage. / Brandan Briones, L.; Brinksma, Hendrik; Stoelinga, Mariëlle Ida Antoinette.

Automated Technology for Verification and Analysis: 4th International Symposium, ATVA 2006, Beijing, China, October 23-26, 2006. Proceedings. ed. / Susanne Graf; Wenhui Zhang. Berlin : Springer Verlag, 2006. p. 399-414 (Lecture Notes in Computer Science; Vol. 4218).

Research output: Chapter in Book/Report/Conference proceedingConference contribution

TY - GEN

T1 - A Semantic Framework for Test Coverage

AU - Brandan Briones,L.

AU - Brinksma,Hendrik

AU - Stoelinga,Mariëlle Ida Antoinette

PY - 2006

Y1 - 2006

N2 - Since testing is inherently incomplete, test selection has vital importance. Coverage measures evaluate the quality of a test suite and help the tester select test cases with maximal impact at minimum cost. Existing coverage criteria for test suites are usually defined in terms of syntactic characteristics of the implementation under test or its specification. Typical black-box coverage metrics are state and transition coverage of the specification. White-box testing often considers statement, condition and path coverage. A disadvantage of this syntactic approach is that different coverage figures are assigned to systems that are behaviorally equivalent, but syntactically different. Moreover, those coverage metrics do not take into account that certain failures are more severe than others, and that more testing effort should be devoted to uncover the most important bugs, while less critical system parts can be tested less thoroughly. This paper introduces a semantic approach to black box test coverage. Our starting point is a weighted fault model (or WFM), which augments a specification by assigning a weight to each error that may occur in an implementation. We define a framework to express coverage measures that express how well a test suite covers such a specification, taking into account the error weight. Since our notions are semantic, they are insensitive to replacing a specification by one with equivalent behaviour. We present several algorithms that, given a certain minimality criterion, compute a minimal test suite with maximal coverage. These algorithms work on a syntactic representation of WFMs as fault automata. They are based on existing and novel optimization problems. Finally, we illustrate our approach by analyzing and comparing a number of test suites for a chat protocol.

AB - Since testing is inherently incomplete, test selection has vital importance. Coverage measures evaluate the quality of a test suite and help the tester select test cases with maximal impact at minimum cost. Existing coverage criteria for test suites are usually defined in terms of syntactic characteristics of the implementation under test or its specification. Typical black-box coverage metrics are state and transition coverage of the specification. White-box testing often considers statement, condition and path coverage. A disadvantage of this syntactic approach is that different coverage figures are assigned to systems that are behaviorally equivalent, but syntactically different. Moreover, those coverage metrics do not take into account that certain failures are more severe than others, and that more testing effort should be devoted to uncover the most important bugs, while less critical system parts can be tested less thoroughly. This paper introduces a semantic approach to black box test coverage. Our starting point is a weighted fault model (or WFM), which augments a specification by assigning a weight to each error that may occur in an implementation. We define a framework to express coverage measures that express how well a test suite covers such a specification, taking into account the error weight. Since our notions are semantic, they are insensitive to replacing a specification by one with equivalent behaviour. We present several algorithms that, given a certain minimality criterion, compute a minimal test suite with maximal coverage. These algorithms work on a syntactic representation of WFMs as fault automata. They are based on existing and novel optimization problems. Finally, we illustrate our approach by analyzing and comparing a number of test suites for a chat protocol.

KW - EWI-6937

KW - METIS-238175

KW - IR-59905

U2 - 10.1007/11901914_30

DO - 10.1007/11901914_30

M3 - Conference contribution

SN - 978-3-540-47237-7

T3 - Lecture Notes in Computer Science

SP - 399

EP - 414

BT - Automated Technology for Verification and Analysis

PB - Springer Verlag

CY - Berlin

ER -

Brandan Briones L, Brinksma H, Stoelinga MIA. A Semantic Framework for Test Coverage. In Graf S, Zhang W, editors, Automated Technology for Verification and Analysis: 4th International Symposium, ATVA 2006, Beijing, China, October 23-26, 2006. Proceedings. Berlin: Springer Verlag. 2006. p. 399-414. (Lecture Notes in Computer Science). Available from, DOI: 10.1007/11901914_30