• 4 Citations

Abstract

Existing penetration testing approaches assess the vulnerability of a system by determining whether certain attack paths are possible in practice. Thus, penetration testing has so far been used as a qualitative research method. To enable quantitative approaches to security risk management, including decision support based on the cost-effectiveness of countermeasures, one needs quantitative measures of the feasibility of an attack. Also, when physical or social attack steps are involved, the binary view on whether a vulnerability is present or not is insufficient, and one needs some viability metric. When penetration tests are performed anyway, it is very easy for the testers to keep track of, for example, the time they spend on each attack step. Therefore, this paper proposes the concept of quantitative penetration testing to determine the difficulty rather than the possibility of attacks based on such measurements. We do this by step-wise updates of expected time and probability of success for all steps in an attack scenario. In addition, we show how the skill of the testers can be included to improve the accuracy of the metrics, based on the framework of item response theory (Elo ratings). We prove the feasibility of the approach by means of simulations, and discuss application possibilities.
Original languageUndefined
Title of host publication9th International Conference on Information Assurance and Security, IAS 2013
Place of PublicationPiscataway, New Jersey
PublisherIEEE
Pages49-54
Number of pages6
ISBN (Print)978-1-4799-2989-4
DOIs
StatePublished - 6 Dec 2013

Publication series

Name
PublisherIEEE

Fingerprint

Testing
Cost effectiveness
Risk management

Keywords

  • quantitative security
  • EWI-25270
  • EC Grant Agreement nr.: FP7/2007-2013
  • EC Grant Agreement nr.: FP7/318003
  • IR-92537
  • Socio-technical security
  • Security Metrics
  • Item Response Theory
  • METIS-309645
  • Penetration Testing

Cite this

Arnold, F., Pieters, W., & Stoelinga, M. I. A. (2013). Quantitative penetration testing with item response theory. In 9th International Conference on Information Assurance and Security, IAS 2013 (pp. 49-54). Piscataway, New Jersey: IEEE. DOI: 10.1109/ISIAS.2013.6947732

Arnold, Florian; Pieters, Wolter; Stoelinga, Mariëlle Ida Antoinette / Quantitative penetration testing with item response theory.

9th International Conference on Information Assurance and Security, IAS 2013. Piscataway, New Jersey : IEEE, 2013. p. 49-54.

Research output: Scientific - peer-reviewConference contribution

@inbook{8317fff7462444b99d3c77dd88fd2957,
title = "Quantitative penetration testing with item response theory",
abstract = "Existing penetration testing approaches assess the vulnerability of a system by determining whether certain attack paths are possible in practice. Thus, penetration testing has so far been used as a qualitative research method. To enable quantitative approaches to security risk management, including decision support based on the cost-effectiveness of countermeasures, one needs quantitative measures of the feasibility of an attack. Also, when physical or social attack steps are involved, the binary view on whether a vulnerability is present or not is insufficient, and one needs some viability metric. When penetration tests are performed anyway, it is very easy for the testers to keep track of, for example, the time they spend on each attack step. Therefore, this paper proposes the concept of quantitative penetration testing to determine the difficulty rather than the possibility of attacks based on such measurements. We do this by step-wise updates of expected time and probability of success for all steps in an attack scenario. In addition, we show how the skill of the testers can be included to improve the accuracy of the metrics, based on the framework of item response theory (Elo ratings). We prove the feasibility of the approach by means of simulations, and discuss application possibilities.",
keywords = "quantitative security, EWI-25270, EC Grant Agreement nr.: FP7/2007-2013, EC Grant Agreement nr.: FP7/318003, IR-92537, Socio-technical security, Security Metrics, Item Response Theory, METIS-309645, Penetration Testing",
author = "Florian Arnold and Wolter Pieters and Stoelinga, {Mariëlle Ida Antoinette}",
note = "Foreground = 100%; Type of activity = publication, presentation; Main leader = UT; Type of audience = scientific community, industry; Size of audience = 30; Countries addressed = international;",
year = "2013",
month = "12",
doi = "10.1109/ISIAS.2013.6947732",
isbn = "978-1-4799-2989-4",
publisher = "IEEE",
pages = "49--54",
booktitle = "9th International Conference on Information Assurance and Security, IAS 2013",

}

Arnold, F, Pieters, W & Stoelinga, MIA 2013, Quantitative penetration testing with item response theory. in 9th International Conference on Information Assurance and Security, IAS 2013. IEEE, Piscataway, New Jersey, pp. 49-54. DOI: 10.1109/ISIAS.2013.6947732

Quantitative penetration testing with item response theory. / Arnold, Florian; Pieters, Wolter; Stoelinga, Mariëlle Ida Antoinette.

9th International Conference on Information Assurance and Security, IAS 2013. Piscataway, New Jersey : IEEE, 2013. p. 49-54.

Research output: Scientific - peer-reviewConference contribution

TY - CHAP

T1 - Quantitative penetration testing with item response theory

AU - Arnold,Florian

AU - Pieters,Wolter

AU - Stoelinga,Mariëlle Ida Antoinette

N1 - Foreground = 100%; Type of activity = publication, presentation; Main leader = UT; Type of audience = scientific community, industry; Size of audience = 30; Countries addressed = international;

PY - 2013/12/6

Y1 - 2013/12/6

N2 - Existing penetration testing approaches assess the vulnerability of a system by determining whether certain attack paths are possible in practice. Thus, penetration testing has so far been used as a qualitative research method. To enable quantitative approaches to security risk management, including decision support based on the cost-effectiveness of countermeasures, one needs quantitative measures of the feasibility of an attack. Also, when physical or social attack steps are involved, the binary view on whether a vulnerability is present or not is insufficient, and one needs some viability metric. When penetration tests are performed anyway, it is very easy for the testers to keep track of, for example, the time they spend on each attack step. Therefore, this paper proposes the concept of quantitative penetration testing to determine the difficulty rather than the possibility of attacks based on such measurements. We do this by step-wise updates of expected time and probability of success for all steps in an attack scenario. In addition, we show how the skill of the testers can be included to improve the accuracy of the metrics, based on the framework of item response theory (Elo ratings). We prove the feasibility of the approach by means of simulations, and discuss application possibilities.

AB - Existing penetration testing approaches assess the vulnerability of a system by determining whether certain attack paths are possible in practice. Thus, penetration testing has so far been used as a qualitative research method. To enable quantitative approaches to security risk management, including decision support based on the cost-effectiveness of countermeasures, one needs quantitative measures of the feasibility of an attack. Also, when physical or social attack steps are involved, the binary view on whether a vulnerability is present or not is insufficient, and one needs some viability metric. When penetration tests are performed anyway, it is very easy for the testers to keep track of, for example, the time they spend on each attack step. Therefore, this paper proposes the concept of quantitative penetration testing to determine the difficulty rather than the possibility of attacks based on such measurements. We do this by step-wise updates of expected time and probability of success for all steps in an attack scenario. In addition, we show how the skill of the testers can be included to improve the accuracy of the metrics, based on the framework of item response theory (Elo ratings). We prove the feasibility of the approach by means of simulations, and discuss application possibilities.

KW - quantitative security

KW - EWI-25270

KW - EC Grant Agreement nr.: FP7/2007-2013

KW - EC Grant Agreement nr.: FP7/318003

KW - IR-92537

KW - Socio-technical security

KW - Security Metrics

KW - Item Response Theory

KW - METIS-309645

KW - Penetration Testing

U2 - 10.1109/ISIAS.2013.6947732

DO - 10.1109/ISIAS.2013.6947732

M3 - Conference contribution

SN - 978-1-4799-2989-4

SP - 49

EP - 54

BT - 9th International Conference on Information Assurance and Security, IAS 2013

PB - IEEE

ER -

Arnold F, Pieters W, Stoelinga MIA. Quantitative penetration testing with item response theory. In 9th International Conference on Information Assurance and Security, IAS 2013. Piscataway, New Jersey: IEEE. 2013. p. 49-54. Available from, DOI: 10.1109/ISIAS.2013.6947732