Exercise activities in log analysis for cybersecurity

The current report describes a cyber security log analysis exercise. The participants practiced log analysis in the roles of manager, scout and analyst, and practice individually as well as in teams. Furthermore, the exercise studied cyber situation awareness (CSA) and effects of receiving feedback. The exercise was conducted in FOI:s cyber range CRATE with web-, email-, and file servers, network equipment and more than 200 computer clients. The participants' task was to identify and analyze various attacks, including network scans from inside and outside the network, password guesses on network services, infected USB sticks and overload attacks. The first day of the exercise, the participants worked as a team and data was collected in order to study working methods and validate a measurement technique for CSA. The measurement instrument was developed by literature analysis as well as interviews with log analysts before the exercise. Also, data collection included performance measures, communication analysis, hierarchical task analysis and subjective assessments of learning effects, usefulness and complexity of the given tasks. The results concerning working methods gave a good input to the participating organization regarding division of labor and workload. The CSA instrument was well received and only minor changes are needed for future studies. During the second day, the participants worked individually, solving three different tasks. After each task, half of the participants received oral feedback. The data collection aimed to study the possible benefits of feedback. The results indicate that feedback had a positive effect on the accomplishment of the subsequent task but a drawback was a floor effect was obtained due to the level of difficulty of the tasks and that the available monitoring tools were not optimally configured for the tasks. Overall, the exercise setup and data collection tools proved successful. For future exercises, the complexity in the exercise environment should be reduced and configuration of monitoring tools revised. The exercise was very appreciated by the participants and forms a good basis for future project activities.