Attacking and Deceiving Military AI Systems


  • Farzad Kamrani
  • Linus Kanestad
  • Linus Luotsinen
  • Björn Pelzer
  • Johan Sabel
  • Viktor Sandström
  • Agnes Tegen

Publish date: 2023-04-12

Report number: FOI-R--5396--SE

Pages: 46

Written in: English


  • artificial intelligence
  • machine learning
  • deep learning
  • deep neural networks
  • deception
  • cyber attacks
  • attack vectors
  • vulnerabilities
  • adversarial examples
  • data poisoning
  • data extraction
  • adversarial policy


This report investigates adversarial machine learning (AML), the research into methods of exploiting weaknesses in AI systems based on machine learning (ML). In recent years, machine learning, especially deep learning (DL), has allowed rapid progress in diverse fields like image classification, natural language processing and autonomous agents. As such DL is also of interest in military contexts. Yet, alongside the progress there has been a rising interest in AML methods, with new attack variations being published constantly. Practically all DL-systems are susceptible in some way, whether it is to confuse them, to avoid being detected by them, or to extract secret information they may hold. From a military perspective it is important to be aware of the possibility of such exploits, both against the own AI systems and against those used by an adversary. The report provides an overview of AML research, and then showcases a selection of attack methods against different types of AI systems:  poisoning of image classification systems, enabling military vehicles to avoid detection;  extraction attacks that can retrieve secret information from large generative models;  adversarial policy attacks where an adversary behaves in a manner that confound autonomous agents. Each case describes and discusses the attacks and evaluates implementations. The focus of this report is on the attacks. While defence against AML methods is discussed briey where applicable, a more in-depth study of AML defence is the subject of a follow-up report.