On the organisation of program verification competitions

Marieke Huisman, Vladimir Klebanov, Rosemary Monahan

    Research output: Chapter in Book/Report/Conference proceedingConference contributionAcademicpeer-review

    5 Citations (Scopus)
    19 Downloads (Pure)

    Abstract

    In this paper, we discuss the challenges that have to be addressed when organising program verification competitions. Our focus is on competitions for verification systems where the participants both formalise an informally stated requirement and (typically) provide some guidance for the tool to show it. The paper draws its insights from our experiences with organising a program verification competition at FoVeOOS 2011. We discuss in particular the following aspects: challenge selection, on-site versus online organisation, team composition and judging. We conclude with a list of recommendations for future competition organisers.
    Original languageUndefined
    Title of host publication1st International Workshop on Comparative Empirical Evaluation of Reasoning Systems (COMPARE 2012)
    EditorsVladimir Klebanov, Bernhard Beckert, Armin Biere, Geoff Sutcliffe
    Place of PublicationAachen
    PublisherSun SITE Central Europe
    Pages50-59
    Number of pages10
    Publication statusPublished - Jun 2012

    Publication series

    NameCEUR-WS Proceedings
    PublisherSun SITE Central Europe
    Volume873
    ISSN (Print)1613-0073

    Keywords

    • EWI-22675
    • IR-83451
    • METIS-293239

    Cite this

    Huisman, M., Klebanov, V., & Monahan, R. (2012). On the organisation of program verification competitions. In V. Klebanov, B. Beckert, A. Biere, & G. Sutcliffe (Eds.), 1st International Workshop on Comparative Empirical Evaluation of Reasoning Systems (COMPARE 2012) (pp. 50-59). (CEUR-WS Proceedings; Vol. 873). Aachen: Sun SITE Central Europe.
    Huisman, Marieke ; Klebanov, Vladimir ; Monahan, Rosemary. / On the organisation of program verification competitions. 1st International Workshop on Comparative Empirical Evaluation of Reasoning Systems (COMPARE 2012). editor / Vladimir Klebanov ; Bernhard Beckert ; Armin Biere ; Geoff Sutcliffe. Aachen : Sun SITE Central Europe, 2012. pp. 50-59 (CEUR-WS Proceedings).
    @inproceedings{9c3f0d2deaef49e8962954013afc1a7a,
    title = "On the organisation of program verification competitions",
    abstract = "In this paper, we discuss the challenges that have to be addressed when organising program verification competitions. Our focus is on competitions for verification systems where the participants both formalise an informally stated requirement and (typically) provide some guidance for the tool to show it. The paper draws its insights from our experiences with organising a program verification competition at FoVeOOS 2011. We discuss in particular the following aspects: challenge selection, on-site versus online organisation, team composition and judging. We conclude with a list of recommendations for future competition organisers.",
    keywords = "EWI-22675, IR-83451, METIS-293239",
    author = "Marieke Huisman and Vladimir Klebanov and Rosemary Monahan",
    year = "2012",
    month = "6",
    language = "Undefined",
    series = "CEUR-WS Proceedings",
    publisher = "Sun SITE Central Europe",
    pages = "50--59",
    editor = "Vladimir Klebanov and Bernhard Beckert and Armin Biere and Geoff Sutcliffe",
    booktitle = "1st International Workshop on Comparative Empirical Evaluation of Reasoning Systems (COMPARE 2012)",

    }

    Huisman, M, Klebanov, V & Monahan, R 2012, On the organisation of program verification competitions. in V Klebanov, B Beckert, A Biere & G Sutcliffe (eds), 1st International Workshop on Comparative Empirical Evaluation of Reasoning Systems (COMPARE 2012). CEUR-WS Proceedings, vol. 873, Sun SITE Central Europe, Aachen, pp. 50-59.

    On the organisation of program verification competitions. / Huisman, Marieke; Klebanov, Vladimir; Monahan, Rosemary.

    1st International Workshop on Comparative Empirical Evaluation of Reasoning Systems (COMPARE 2012). ed. / Vladimir Klebanov; Bernhard Beckert; Armin Biere; Geoff Sutcliffe. Aachen : Sun SITE Central Europe, 2012. p. 50-59 (CEUR-WS Proceedings; Vol. 873).

    Research output: Chapter in Book/Report/Conference proceedingConference contributionAcademicpeer-review

    TY - GEN

    T1 - On the organisation of program verification competitions

    AU - Huisman, Marieke

    AU - Klebanov, Vladimir

    AU - Monahan, Rosemary

    PY - 2012/6

    Y1 - 2012/6

    N2 - In this paper, we discuss the challenges that have to be addressed when organising program verification competitions. Our focus is on competitions for verification systems where the participants both formalise an informally stated requirement and (typically) provide some guidance for the tool to show it. The paper draws its insights from our experiences with organising a program verification competition at FoVeOOS 2011. We discuss in particular the following aspects: challenge selection, on-site versus online organisation, team composition and judging. We conclude with a list of recommendations for future competition organisers.

    AB - In this paper, we discuss the challenges that have to be addressed when organising program verification competitions. Our focus is on competitions for verification systems where the participants both formalise an informally stated requirement and (typically) provide some guidance for the tool to show it. The paper draws its insights from our experiences with organising a program verification competition at FoVeOOS 2011. We discuss in particular the following aspects: challenge selection, on-site versus online organisation, team composition and judging. We conclude with a list of recommendations for future competition organisers.

    KW - EWI-22675

    KW - IR-83451

    KW - METIS-293239

    M3 - Conference contribution

    T3 - CEUR-WS Proceedings

    SP - 50

    EP - 59

    BT - 1st International Workshop on Comparative Empirical Evaluation of Reasoning Systems (COMPARE 2012)

    A2 - Klebanov, Vladimir

    A2 - Beckert, Bernhard

    A2 - Biere, Armin

    A2 - Sutcliffe, Geoff

    PB - Sun SITE Central Europe

    CY - Aachen

    ER -

    Huisman M, Klebanov V, Monahan R. On the organisation of program verification competitions. In Klebanov V, Beckert B, Biere A, Sutcliffe G, editors, 1st International Workshop on Comparative Empirical Evaluation of Reasoning Systems (COMPARE 2012). Aachen: Sun SITE Central Europe. 2012. p. 50-59. (CEUR-WS Proceedings).