Making correct assessments of probabilities and consequences of threats against IT systems is a daunting challenge. An FOI study has arrived at several important conclusions about what the Armed Forces should consider in its assessments

To ensure effective protection against IT attacks, the Armed Forces must rely on assessments of the threats against the current system. A new FOI report considers, though, that it is more difficult than many believe to assess the probability and consequences of threats against IT systems.

“When insurance companies assess the probability and consequences of damage, they have extensive statistics to rely on. Corresponding material on which to base assessments of probabilities and consequences of threats against IT systems are lacking,” says Jonas Hallberg, FOI, one of three researchers who produced the report.

“The Armed Forces must instead rely on experts for these assessments. At the same time, it’s well-known that threat assessments often vary according to who does them,” he continues.

Knowledge and experience are two important causal factors. Researchers have also investigated whether access to briefing material and the form it is delivered in affect consistency between different individuals. In one experiment, a number of experts were provided with threat descriptions, both in the form of structured tables and as running text. The differences in consistency due to the form of the material were not so significant. On the other hand, the researchers concluded that tables appear to be the best alternative when probability and consequence assessments are performed by experienced persons, whereas ordinary language was more suitable for non-experts.

Another of the study’s experiments compared different threat assessment methods. The experiment demonstrated that the current assessment method, where every threat is assessed individually, is better for novices, while paired threat comparisons are more suited to expert assessments.

In order to become less person-dependent, and increase the likelihood of consistent assessments, the Armed Forces should clearly specify how threats to IT systems should be described.

“It is extremely important to put more energy into producing clear threat descriptions that can then form the basis for working on the security of IT systems,” says Jonas Hallberg.