Experimental Validation of a Risk Assessment Method

[Context and motivation] It is desirable that require- ment engineering methods are reliable, that is, that methods can be repeated with the same results. Risk assessments methods, however, often have low reliability when they identify risk mitigations for a sys- tem based on expert judgement. [Question/problem] Our goal is to assess the reliability of an availability risk assessment method for telecom infrastructures, and to identify possibilities for improvement of its reli- ability. [Principal ideas/results] We propose an experimental valida- tion of reliability, and report on its application. We give a detailed analy- sis of sources of variation, explain how we controlled them and validated their mitigations, and motivate the statistical procedure used to analyse the outcome. [Contribution] Our results can be used to improve the reliability of risk assessment methods. Our approach to validating relia- bility can be useful for the assessment of the reliability of other methods.