Methods for evaluation of MHI design in a simulator environment

Authors:

  • Per-Anders Oskarsson
  • Patrik Lif
  • Britta Levin

Publish date: 2017-02-17

Report number: FOI-R--4391--SE

Pages: 54

Written in: Swedish

Keywords:

  • usability
  • workload
  • mental workload
  • situation awareness
  • psychophysiology
  • evaluation
  • validation
  • display

Abstract

Modern fighter aircraft are very complex, operate in dynamic environments and require user interfaces and system solutions that are well adapted to users' capacity. It is important to account for pilots' ability to effectively handle the aircraft, and utilize all system functionality, during the entire aircraft development process. A new interface is a major undertaking that requires both thorough design work and testing before it can be implemented in a real aircraft. A large part of the user interface development is conducted by evaluation and validation in the simulator environment. One question is: How can the methodology for evaluation and validation of operator user interfaces, and related functionalities, be further developed? Are there methods that could facilitate the process, increase support for evaluators and help to determine if interfaces and functions are ready for implementation? FOI has been asked to review existing methods and assess their applicability for the simulator environment. The focus of the work has been on methods for measurement of mental workload, situation awareness, psycho-physiological reactions, and usability. The work has included both subjective measures, which often involves discrete estimates, and objective measures that often are dynamic. The latter type of measurement can be done continuously over time and provide information for an entire session. The need for simulation activities vary depending on where in the development cycle a function currently is. Similarly, the applicability of valuation methods will vary over time. Irrespective of development stage, there is a basic need for a structured approach. A general prerequisite for effective evaluation and validation is that all personnel involved are well prepared and aware of the expectations. This report contains recommendations for methodology regarding evaluation and validation of user interfaces and underlying functionality. The methodology has a focus on evaluation of usability and how to systematically capture the participants' perceptions of the system by use of questions. The suggestions is a combination of questions and ratings and, when applicable, use of eye-point-of-gaze registrations and other psycho-physiological measurements. In order to facilitate, simplify and speed up the process around the management of questionnaires, it is recommended to introduce a digital tool.