Choice and chance: model-based testing of stochastic behaviour

    Research output: ThesisPhD Thesis - Research UT, graduation UT

    92 Downloads (Pure)

    Abstract

    Probability plays an important role in many computer applications. A vast number of algorithms, protocols and computation methods uses randomisation to achieve their goals. A crucial question then becomes whether such probabilistic systems work as intended. To investigate this, such systems are often subjected to a large number of well-designed test cases, that compare a observed behaviour to a requirements specification. Model-based testing is an innovative testing technique rooted in formal methods, that aims at automating this labour intense and often error-prone manual task. By providing faster and more thorough testing at lower cost, it has gained rapid popularity in industry and academia alike. However, classic model-based testing methods are insufficient when dealing with inherently stochastic systems.

    This thesis introduces a rigorous model-based testing framework, that is capable to automatically test such systems. The presented methods are capable of judging functional correctness, discrete probability choices, and hard and soft-real time constraints. The framework is constructed in a clear step-by-step approach. First, the model-based testing landscape is laid out, and related work is discussed. Next, we instantiate a model-based testing framework to highlight the purpose of individual theoretical components like, e.g., a conformance relation, test cases, and practical test generation algorithms. This framework is then conservatively extended by introducing discrete probability choices to the specification language. A last step further extends this probabilistic framework by adding hard and soft real time constraints. Classical functional correctness verdicts are thus extended with goodness of fit methods known from statistics. Proofs of the framework’s correctness are presented before its capabilities are exemplified by studying smaller scale case studies known from the literature.

    The framework reconciles non-deterministic and probabilistic choices in a fully-fledged way via the use of schedulers. Schedulers then become a subject worthy to study in their own rights. This is done in the second part of this thesis; we introduce a most natural equivalence relation based on schedulers for Markov automata, and compare its distinguishing power to notions of trace distributions and bisimulation relations. Lastly, the power of different scheduler classes of stochastic automata is investigated. We compare reachability probabilities of different schedulers by altering the information available to them. A hierarchy of scheduler classes is established, with the intent to reduce complexity of related problems by gaining near optimal results for smaller scheduler classes.
    Original languageEnglish
    Awarding Institution
    • University of Twente
    Supervisors/Advisors
    • Stoelinga, Mariëlle Ida Antoinette, Supervisor
    • van de Pol, Jaco , Supervisor
    Award date12 Dec 2018
    Place of PublicationEnschede
    Publisher
    Print ISBNs978-90-365-4695-9
    DOIs
    Publication statusPublished - 12 Dec 2018

    Fingerprint

    Testing
    Specification languages
    Computer applications
    Formal methods
    Stochastic systems
    Statistics
    Personnel
    Specifications
    Network protocols
    Costs
    Industry

    Cite this

    @phdthesis{99f55372feeb4f4aba88620696936a54,
    title = "Choice and chance: model-based testing of stochastic behaviour",
    abstract = "Probability plays an important role in many computer applications. A vast number of algorithms, protocols and computation methods uses randomisation to achieve their goals. A crucial question then becomes whether such probabilistic systems work as intended. To investigate this, such systems are often subjected to a large number of well-designed test cases, that compare a observed behaviour to a requirements specification. Model-based testing is an innovative testing technique rooted in formal methods, that aims at automating this labour intense and often error-prone manual task. By providing faster and more thorough testing at lower cost, it has gained rapid popularity in industry and academia alike. However, classic model-based testing methods are insufficient when dealing with inherently stochastic systems.This thesis introduces a rigorous model-based testing framework, that is capable to automatically test such systems. The presented methods are capable of judging functional correctness, discrete probability choices, and hard and soft-real time constraints. The framework is constructed in a clear step-by-step approach. First, the model-based testing landscape is laid out, and related work is discussed. Next, we instantiate a model-based testing framework to highlight the purpose of individual theoretical components like, e.g., a conformance relation, test cases, and practical test generation algorithms. This framework is then conservatively extended by introducing discrete probability choices to the specification language. A last step further extends this probabilistic framework by adding hard and soft real time constraints. Classical functional correctness verdicts are thus extended with goodness of fit methods known from statistics. Proofs of the framework’s correctness are presented before its capabilities are exemplified by studying smaller scale case studies known from the literature.The framework reconciles non-deterministic and probabilistic choices in a fully-fledged way via the use of schedulers. Schedulers then become a subject worthy to study in their own rights. This is done in the second part of this thesis; we introduce a most natural equivalence relation based on schedulers for Markov automata, and compare its distinguishing power to notions of trace distributions and bisimulation relations. Lastly, the power of different scheduler classes of stochastic automata is investigated. We compare reachability probabilities of different schedulers by altering the information available to them. A hierarchy of scheduler classes is established, with the intent to reduce complexity of related problems by gaining near optimal results for smaller scheduler classes.",
    author = "Marcus Gerhold",
    year = "2018",
    month = "12",
    day = "12",
    doi = "10.3990/1.9789036546959",
    language = "English",
    isbn = "978-90-365-4695-9",
    series = "DSI Ph.D. thesis series",
    publisher = "University of Twente",
    number = "18-022",
    address = "Netherlands",
    school = "University of Twente",

    }

    Choice and chance : model-based testing of stochastic behaviour. / Gerhold, Marcus .

    Enschede : University of Twente, 2018. 263 p.

    Research output: ThesisPhD Thesis - Research UT, graduation UT

    TY - THES

    T1 - Choice and chance

    T2 - model-based testing of stochastic behaviour

    AU - Gerhold, Marcus

    PY - 2018/12/12

    Y1 - 2018/12/12

    N2 - Probability plays an important role in many computer applications. A vast number of algorithms, protocols and computation methods uses randomisation to achieve their goals. A crucial question then becomes whether such probabilistic systems work as intended. To investigate this, such systems are often subjected to a large number of well-designed test cases, that compare a observed behaviour to a requirements specification. Model-based testing is an innovative testing technique rooted in formal methods, that aims at automating this labour intense and often error-prone manual task. By providing faster and more thorough testing at lower cost, it has gained rapid popularity in industry and academia alike. However, classic model-based testing methods are insufficient when dealing with inherently stochastic systems.This thesis introduces a rigorous model-based testing framework, that is capable to automatically test such systems. The presented methods are capable of judging functional correctness, discrete probability choices, and hard and soft-real time constraints. The framework is constructed in a clear step-by-step approach. First, the model-based testing landscape is laid out, and related work is discussed. Next, we instantiate a model-based testing framework to highlight the purpose of individual theoretical components like, e.g., a conformance relation, test cases, and practical test generation algorithms. This framework is then conservatively extended by introducing discrete probability choices to the specification language. A last step further extends this probabilistic framework by adding hard and soft real time constraints. Classical functional correctness verdicts are thus extended with goodness of fit methods known from statistics. Proofs of the framework’s correctness are presented before its capabilities are exemplified by studying smaller scale case studies known from the literature.The framework reconciles non-deterministic and probabilistic choices in a fully-fledged way via the use of schedulers. Schedulers then become a subject worthy to study in their own rights. This is done in the second part of this thesis; we introduce a most natural equivalence relation based on schedulers for Markov automata, and compare its distinguishing power to notions of trace distributions and bisimulation relations. Lastly, the power of different scheduler classes of stochastic automata is investigated. We compare reachability probabilities of different schedulers by altering the information available to them. A hierarchy of scheduler classes is established, with the intent to reduce complexity of related problems by gaining near optimal results for smaller scheduler classes.

    AB - Probability plays an important role in many computer applications. A vast number of algorithms, protocols and computation methods uses randomisation to achieve their goals. A crucial question then becomes whether such probabilistic systems work as intended. To investigate this, such systems are often subjected to a large number of well-designed test cases, that compare a observed behaviour to a requirements specification. Model-based testing is an innovative testing technique rooted in formal methods, that aims at automating this labour intense and often error-prone manual task. By providing faster and more thorough testing at lower cost, it has gained rapid popularity in industry and academia alike. However, classic model-based testing methods are insufficient when dealing with inherently stochastic systems.This thesis introduces a rigorous model-based testing framework, that is capable to automatically test such systems. The presented methods are capable of judging functional correctness, discrete probability choices, and hard and soft-real time constraints. The framework is constructed in a clear step-by-step approach. First, the model-based testing landscape is laid out, and related work is discussed. Next, we instantiate a model-based testing framework to highlight the purpose of individual theoretical components like, e.g., a conformance relation, test cases, and practical test generation algorithms. This framework is then conservatively extended by introducing discrete probability choices to the specification language. A last step further extends this probabilistic framework by adding hard and soft real time constraints. Classical functional correctness verdicts are thus extended with goodness of fit methods known from statistics. Proofs of the framework’s correctness are presented before its capabilities are exemplified by studying smaller scale case studies known from the literature.The framework reconciles non-deterministic and probabilistic choices in a fully-fledged way via the use of schedulers. Schedulers then become a subject worthy to study in their own rights. This is done in the second part of this thesis; we introduce a most natural equivalence relation based on schedulers for Markov automata, and compare its distinguishing power to notions of trace distributions and bisimulation relations. Lastly, the power of different scheduler classes of stochastic automata is investigated. We compare reachability probabilities of different schedulers by altering the information available to them. A hierarchy of scheduler classes is established, with the intent to reduce complexity of related problems by gaining near optimal results for smaller scheduler classes.

    U2 - 10.3990/1.9789036546959

    DO - 10.3990/1.9789036546959

    M3 - PhD Thesis - Research UT, graduation UT

    SN - 978-90-365-4695-9

    T3 - DSI Ph.D. thesis series

    PB - University of Twente

    CY - Enschede

    ER -

    Gerhold M. Choice and chance: model-based testing of stochastic behaviour. Enschede: University of Twente, 2018. 263 p. (DSI Ph.D. thesis series; 18-022). (IPA Dissertation Series; 2018-20). https://doi.org/10.3990/1.9789036546959