Natural Language Processing using Deep Neural Networks


  • Maja Karasalo
  • Fredrik Johansson
  • Magnus Rosell
  • Andreas Horndahl
  • Lukas Lundmark
  • Hanna Lilja
  • Ulrika Wickenberg Bolin
  • Harald Stiff

Publish date: 2020-05-08

Report number: FOI-R--4948--SE

Pages: 46

Written in: English


  • NLP
  • deep learning
  • language models
  • Transformers


In recent years, the research field of natural language processing (NLP) has improved significantly on a variety of tasks ranging from machine translation and sentiment classification to question answering, text generation and summarization. This progress has been made possible by the introduction of novel deep neural network architectures that can be trained on massive amounts of unstructured textual data. The advances within NLP brings a range of opportunities as well as challenges for society in general, and specifically for the areas of security and defence. This report aims to give an insight into the rapid development of NLP techniques, what can be achieved with the technology today, as well as current and future consequences. An overview of state of the art techniques from recent literature is provided in this report, along with demonstrations of NLP methods on example applications for text generation and text analysis. The examples show that, for many NLP tasks, impressive results can be obtained by applying openly available methods and algorithms. However, for sensitive activities, or in areas where data is scarce, the capacity to develop and maintain in-house NLP software will be increasingly critical. For the security and defence domain, large-scale introduction of these types of techniques requires resources for data acquisition and labeling, AI competence, and computational power.