On the organisation of program verification competitions

Marieke Huisman, Vladimir Klebanov, Rosemary Monahan

Research output: Chapter in Book/Report/Conference proceedingConference contributionAcademicpeer-review

5 Citations (Scopus)
14 Downloads (Pure)

Abstract

In this paper, we discuss the challenges that have to be addressed when organising program verification competitions. Our focus is on competitions for verification systems where the participants both formalise an informally stated requirement and (typically) provide some guidance for the tool to show it. The paper draws its insights from our experiences with organising a program verification competition at FoVeOOS 2011. We discuss in particular the following aspects: challenge selection, on-site versus online organisation, team composition and judging. We conclude with a list of recommendations for future competition organisers.
Original languageUndefined
Title of host publication1st International Workshop on Comparative Empirical Evaluation of Reasoning Systems (COMPARE 2012)
EditorsVladimir Klebanov, Bernhard Beckert, Armin Biere, Geoff Sutcliffe
Place of PublicationAachen
PublisherSun SITE Central Europe
Pages50-59
Number of pages10
Publication statusPublished - Jun 2012

Publication series

NameCEUR-WS Proceedings
PublisherSun SITE Central Europe
Volume873
ISSN (Print)1613-0073

Keywords

  • EWI-22675
  • IR-83451
  • METIS-293239

Cite this

Huisman, M., Klebanov, V., & Monahan, R. (2012). On the organisation of program verification competitions. In V. Klebanov, B. Beckert, A. Biere, & G. Sutcliffe (Eds.), 1st International Workshop on Comparative Empirical Evaluation of Reasoning Systems (COMPARE 2012) (pp. 50-59). (CEUR-WS Proceedings; Vol. 873). Aachen: Sun SITE Central Europe.
Huisman, Marieke ; Klebanov, Vladimir ; Monahan, Rosemary. / On the organisation of program verification competitions. 1st International Workshop on Comparative Empirical Evaluation of Reasoning Systems (COMPARE 2012). editor / Vladimir Klebanov ; Bernhard Beckert ; Armin Biere ; Geoff Sutcliffe. Aachen : Sun SITE Central Europe, 2012. pp. 50-59 (CEUR-WS Proceedings).
@inproceedings{9c3f0d2deaef49e8962954013afc1a7a,
title = "On the organisation of program verification competitions",
abstract = "In this paper, we discuss the challenges that have to be addressed when organising program verification competitions. Our focus is on competitions for verification systems where the participants both formalise an informally stated requirement and (typically) provide some guidance for the tool to show it. The paper draws its insights from our experiences with organising a program verification competition at FoVeOOS 2011. We discuss in particular the following aspects: challenge selection, on-site versus online organisation, team composition and judging. We conclude with a list of recommendations for future competition organisers.",
keywords = "EWI-22675, IR-83451, METIS-293239",
author = "Marieke Huisman and Vladimir Klebanov and Rosemary Monahan",
year = "2012",
month = "6",
language = "Undefined",
series = "CEUR-WS Proceedings",
publisher = "Sun SITE Central Europe",
pages = "50--59",
editor = "Vladimir Klebanov and Bernhard Beckert and Armin Biere and Geoff Sutcliffe",
booktitle = "1st International Workshop on Comparative Empirical Evaluation of Reasoning Systems (COMPARE 2012)",

}

Huisman, M, Klebanov, V & Monahan, R 2012, On the organisation of program verification competitions. in V Klebanov, B Beckert, A Biere & G Sutcliffe (eds), 1st International Workshop on Comparative Empirical Evaluation of Reasoning Systems (COMPARE 2012). CEUR-WS Proceedings, vol. 873, Sun SITE Central Europe, Aachen, pp. 50-59.

On the organisation of program verification competitions. / Huisman, Marieke; Klebanov, Vladimir; Monahan, Rosemary.

1st International Workshop on Comparative Empirical Evaluation of Reasoning Systems (COMPARE 2012). ed. / Vladimir Klebanov; Bernhard Beckert; Armin Biere; Geoff Sutcliffe. Aachen : Sun SITE Central Europe, 2012. p. 50-59 (CEUR-WS Proceedings; Vol. 873).

Research output: Chapter in Book/Report/Conference proceedingConference contributionAcademicpeer-review

TY - GEN

T1 - On the organisation of program verification competitions

AU - Huisman, Marieke

AU - Klebanov, Vladimir

AU - Monahan, Rosemary

PY - 2012/6

Y1 - 2012/6

N2 - In this paper, we discuss the challenges that have to be addressed when organising program verification competitions. Our focus is on competitions for verification systems where the participants both formalise an informally stated requirement and (typically) provide some guidance for the tool to show it. The paper draws its insights from our experiences with organising a program verification competition at FoVeOOS 2011. We discuss in particular the following aspects: challenge selection, on-site versus online organisation, team composition and judging. We conclude with a list of recommendations for future competition organisers.

AB - In this paper, we discuss the challenges that have to be addressed when organising program verification competitions. Our focus is on competitions for verification systems where the participants both formalise an informally stated requirement and (typically) provide some guidance for the tool to show it. The paper draws its insights from our experiences with organising a program verification competition at FoVeOOS 2011. We discuss in particular the following aspects: challenge selection, on-site versus online organisation, team composition and judging. We conclude with a list of recommendations for future competition organisers.

KW - EWI-22675

KW - IR-83451

KW - METIS-293239

M3 - Conference contribution

T3 - CEUR-WS Proceedings

SP - 50

EP - 59

BT - 1st International Workshop on Comparative Empirical Evaluation of Reasoning Systems (COMPARE 2012)

A2 - Klebanov, Vladimir

A2 - Beckert, Bernhard

A2 - Biere, Armin

A2 - Sutcliffe, Geoff

PB - Sun SITE Central Europe

CY - Aachen

ER -

Huisman M, Klebanov V, Monahan R. On the organisation of program verification competitions. In Klebanov V, Beckert B, Biere A, Sutcliffe G, editors, 1st International Workshop on Comparative Empirical Evaluation of Reasoning Systems (COMPARE 2012). Aachen: Sun SITE Central Europe. 2012. p. 50-59. (CEUR-WS Proceedings).