Prudent practices for designing malware experiments: Status quo and outlook

Christian Rossow, Christian J. Dietrich, Chris Grier, Christian Kreibich, Vern Paxson, Norbert Pohlmann, Herbert Bos, Maarten Van Steen

Research output: Chapter in Book/Report/Conference proceedingConference contributionAcademicpeer-review

131 Citations (Scopus)


Malware researchers rely on the observation of malicious code in execution to collect datasets for a wide array of experiments, including generation of detection models, study of longitudinal behavior, and validation of prior research. For such research to reflect prudent science, the work needs to address a number of concerns relating to the correct and representative use of the datasets, presentation of methodology in a fashion sufficiently transparent to enable reproducibility, and due consideration of the need not to harm others. In this paper we study the methodological rigor and prudence in 36 academic publications from 2006 - 2011 that rely on malware execution. 40% of these papers appeared in the 6 highest-ranked academic security conferences. We find frequent shortcomings, including problematic assumptions regarding the use of execution-driven datasets (25% of the papers), absence of description of security precautions taken during experiments (71% of the articles), and oftentimes insufficient description of the experimental setup. Deficiencies occur in top-tier venues and elsewhere alike, highlighting a need for the community to improve its handling of malware datasets. In the hope of aiding authors, reviewers, and readers, we frame guidelines regarding transparency, realism, correctness, and safety for collecting and using malware datasets.

Original languageEnglish
Title of host publication2012 IEEE Symposium on Security and Privacy, S and P 2012
Number of pages15
ISBN (Electronic)978-0-7695-4681-0
ISBN (Print)978-1-4673-1244-8
Publication statusPublished - 1 Dec 2012
Externally publishedYes
Event33rd IEEE Symposium on Security and Privacy, 2012 - San Francisco, United States
Duration: 20 May 201223 May 2012
Conference number: 33

Publication series

NameIEEE Symposium on Security and Privacy
ISSN (Print)1081-6011
ISSN (Electronic)2375-1207


Conference33rd IEEE Symposium on Security and Privacy, 2012
Country/TerritoryUnited States
CitySan Francisco
Internet address


  • Datasets
  • Dynamic analysis
  • Experiments
  • Malware


Dive into the research topics of 'Prudent practices for designing malware experiments: Status quo and outlook'. Together they form a unique fingerprint.

Cite this