Look – but preserve integrity

(From Framsyn Magazine 1-2006)

The people in the picture are tracked by the system and their positions are marked. In the right hand picture the people have been removed with only the markers remaining.

The rapid development of computers, sensors and methods makes possible entirely new types of surveillance systems with new capabilities. The new technology can be used to create automatic human-aware systems but it can also be used to help protect the integrity and privacy of the individual.

By Jörgen Ahlberg

Camera-based surveillance systems are becoming increasingly common in today’s society. It is hardly possible to take a step in a modern town centre without being captured on one or more CCTV cameras, especially in stores, shopping malls and underground stations.

Surveillance cameras obviously have military applications as well; intrusion alarms, tactical decision support and the documentation and monitoring of  peace operations are just a few examples. At the Swedish Defence Research Agency (FOI) there is growing interest in surveillance systems and research is being conducted into applications such as antiterrorism, airport security and various forms of crisis in urban environments.

The growing number of surveillance sensors gives rise to a number of problems, some of the main ones being:

• How can we best process and present the gathered data in order to give a picture of the situation?
• How can we avoid impinging on the integrity, or privacy, of the individual?

The first of these is, of course, a classic problem in both military and civil contexts, but the second has not been a topic of great concern in the past. In the context of future international operations, however, we believe that it will represent a problem of increasing urgency.

In this article we present four possible approaches to the resolution of the problems mentioned above while also enhancing the capability of the relevant surveillance.

• Three-dimensional presentation
• Combining different types of sensor
• Integrity preserving surveillance
• Human-aware systems

The greatest problem associated with traditional camera-based surveillance is that it is difficult to get the overall picture. In today’s CCTV operations centres, each operator will be watching a number of screens which may each be showing four, or even more, views from different cameras.

Even a very simple sequence of events, such as a single individual moving round inside a building, requires considerable mental effort to interpret. What is the relationship between the various camera views. If the person moves out of view in that direction, in which direction is he actually moving, and in which camera view is he likely to appear next? In order to be able to answer these questions quickly, the operator needs to keep a plan of the building in his head as well as remembering where the various cameras are situated.

An alternative approach is to present the camera views in a three-dimensional computer graphics model of the area under surveillance. In the 3D model the cameras are replaced by projectors and what the cameras see is projected onto the 3-D model (see left hand picture below). This approach has several advantages:

• It is obvious where each of the cameras is positioned.
• The relationship between the different camera views is obvious.

Pictures from several cameras can be shown at the same time and an overview covering the whole of the area under surveillance can be given. A trials platform has been constructed at FOI and tested using data from an exercise in Norrköping in 2004. The exercise, which covered  a number of scenarios, both civil and military, was conducted in collaboration with the Norrköping police, the Home Guard and the Army Combat School. Methods for the automatic building of 3-D models, including the classification of buildings and vegetation, using laser scanner data have been developed by FOI and tested in Norrköping. Digital Norrköping was illustrated in Framsyn 2-2005.

Combining different types of sensor
In a civil setting, the video camera is the most common form of surveillance sensor. For special purposes, other types of sensor can contribute important information, the essential point being the possibility of combining sensors of different types.

Suppose, for example, that we have sensors which can pinpoint the position from which a shot has been fired. The position can then be marked in the 3-D model which then leads to a number of  interesting possibilities:

• If the firer of the shot is in view of one of the cameras, his position can be marked so that it can then be traced forwards and backwards in time so allowing, for example, suitable images to be selected for identification purposes.
• Even if the firer of the shot is not in view of a camera, his field of fire can still be marked in the model.

This latter possibility can also be useful when the aim is to position either marksmen or sensors.

Other useful functions include passage detection, the tracking or classification of vehicles and the detection of suspicious events or behaviour.

Examples of sensors and their capabilities:

• For passage detection there is an abundance of sensors to choose from. Laser beams or so-called IR barriers can be used as invisible fences. There are different kinds of warning systems in which buried cables react when walked or driven over, as well as seismic sensors which detect ground vibrations. The sensors can also be used for classification and can detect whether either persons or a particular type of vehicle have passed.
• An acoustic sensor network consists of a number of nodes where each node consists of two or three microphones. Each node can give the direction of a sound source and the network can pinpoint the position of sound sources such as shots or the sound of a mob.
• Radar sensors can produce images of people through walls or detect movements on the other side of a wall. To detect anyone passing between two buildings, a small radar sensor can be placed inside one building, totally invisible to anyone passing. A prototype with dimensions of less than 10 cm (including batteries) has been built at FOI.

Integrity preserving surveillance
Integrity preserving surveillance (IPS) is based on a combination of technologies which enable surveillance to be conducted without revealing the identities of the subjects. The reason for using IPS is of course that people in general do not like being watched and that consequently there are strict restrictions regarding the use of surveillance cameras. Here are two examples of the way in which such systems could be used.

Civil scenario: A shop owner wants to know how many people pass his display window, how many of these stop and look at the display and what particular things they look at. A surveillance camera with a view out through the window is difficult to obtain approval for, and it only provides a partial solution since someone has to view all the image material in order to answer the original questions. An IPS camera, on the other hand, shows no images but provides a daily report of the type “14 persons passed, five of them stopped and looked, three of them looked at object X”.

One morning the shop owner comes in and finds his shop window broken. The police arrive, unlock the camera and view the encrypted images. The images the police see are still integrity preserving since all the people are masked or replaced, for example, by matchstick men. It is of course easy to see which of the matchstick men smashed the window and after a due process of approval, the particular person committing the crime can be unmasked. Innocent passers-by and window shoppers continue to have their privacy protected.

Military scenario: In a peacekeeping operation we wish to maintain surveillance over an area of a town. The problem is that we know it is unpopular amongst the local inhabitants and so the solution is an IPS system for surveillance cameras. The images from the cameras are projected onto a 3D model of the area but with people and vehicles masked or replaced by ‘matchstick men’ (see the picture on the left below). The unmanipulated image material is encrypted and stored with a local institution trusted by the inhabitants. The manipulated, integrity preserving image material can also be shown openly (for example on a website). The information extracted from the image material in order to manipulate the images can also be used for behaviour analysis and warning purposes.

Human-aware systems
The technology required to create an integrity preserving surveillance system is a part of what we call human-aware systems. Human-aware systems are systems which can detect, track and recognise humans and human activity. The term is used generically to cover a wide range of systems with a broad range of applications. Existing examples include systems which can detect when a driver is tired, or even falls asleep, or systems which can count the number of people passing.

In surveillance situations such systems are of interest for such purposes as giving warning of abnormal events or behaviour, for example violence, crowds or running people. Such a system can help an operator by indicating interesting parts of a mass of image material, while more advanced systems are even able to draw conclusions from different groupings. A group of five people moving around in a coordinated way among a crowd could be very difficult for an operator to detect bur could be followed by an automatic system.

Human-aware systems are technically possible today. In combination with cheaper and smaller sensors they can also be made more inconspicuous and for that reason it is all the more important that ethical aspects such as integrity preservation should be built into the system from the start.

Jörgen Ahlberg is engaged in research into image analysis in surveillance and reconnaissance systems with the IR Systems Department of FOI’s Sensor Technology Division.