Principles for Constructing Datasets with Vulnerable Software – For Evaluation of Tools that Identify Software Vulnerabilities

Authors:

  • Christian Vestlund
  • Christian Gustavsson
  • Daniel Eidenskog

Publish date: 2026-02-02

Report number: FOI-R--5891--SE

Pages: 48

Written in: Swedish

Keywords:

  • software security
  • tools
  • datasets
  • vulnerabilities

Abstract

Tools that can detect software vulnerabilities play a critical role in creating secure software. However, it is not clear how effective such tools are at identifying different types of vulnerabilities or how these tools should be evaluated. In research literature, various types of datasets with vulnerable code are used to evaluate and compare such tools. This report presents a literature study that identifies datasets and examines how they are presented in research literature. This study also examines the criticism against datasets and the characteristics highlighted in the publications. The study has identified 54 different datasets or tools that can generate datasets. Additionally, ten types of criticisms and 22 desirable characteristics have been identified. The results show a trend toward datasets based on vulnerabilities in real software becoming more common, while fewer datasets with constructed or injected vulnerabilities are being created. Moreover, lack of realism is a common criticism against datasets, which is also reflected in realism being one of the most common desirable characteristics. This study's results indicate a lack of an established common understanding of characteristics needed to create high-quality datasets.