Scenarios and data acquisition
The P5 scenarios are based on the needs of the users and they have a focus on critical infrastructure intrusion attempts, and an important effort has been paid to identify these needs. The scenarios are used to give a common and tangible context for both the technical development and for the final demonstrations. For instance, in one of the scenarios, we aim to detect suspicious movements by potential intruders in an area with otherwise innocent people or authorized personnel. The scenario also includes transportation of goods (weaponry) with an aerial drone, as well as movements of cars and animals. The scenarios have been staged for data acquisition at both the OKG nuclear plant outside Oskarshamn, Sweden, and at the CAST test and training facility outside Horsham, UK. At OKG we staged a major part of an intrusion scenario as well as different bits of scenarios to get much variation needed for development and evaluation of algorithms The photographs below is from one of the staged scenaria at OKG, where two potential perpetrators land suspicios goods on the shore.
Surveillance radar development
A radar module that fits into the ISM band specification at 24 GHz has been developed and optimized to fulfil the requirements of detection and tracking of persons/targets within a defined area. Those targets can be measured in distance and angle. Furthermore, different radar modules can be combined to cover a larger area. Hardware and software for the radar modules have been modified so that they can measure not only distance and angle, but also record µ-Doppler data, which facilitates target classification. The radar tracking algorithm will be further improved and the radar module with µ-Doppler capability will be used to evaluate the classification tests and develop algorithms for target classification or characterisation.
The P5 scope of privacy has been established from both the ethics and legal sides in order to give a presentation to the whole consortium. In the legal field, the work within P5 has identified and analysed several relevant legislations coming from the European Union (Directive 95/46, Project of Regulation on data protection, article 29 Working group), Council of Europe (convention 108, project of modification of Convention 108, etc.) and Belgium (law on the cameras). In the ethical field, we haveevaluated the social acceptability of the project and completed a social acceptability study that has been particularly devoted to the political and collective dimensions of "virtual fences". It has unfolded a variety of political problems posed by virtual fence based on state-of-the-art social science literature. The study also addressed different prospective publics for this technology, through "deliberative arenas", to explore collectively the problems identified in the first place. The privacy work continues in P5 to find ways to handle the privacy aspects and also to implement and demonstrate them in the integrated surveillance system. P5 has contributed to the knowledge within the field through.
A significant amount of research and development has been conducted into real time analysis of video data from both visual and thermal sensors. The work has been focussed on detection and tracking of targets from air, sea, and land environments, ranging from single sensors for single target detection to multi-target tracking using multiple sensors. A comprehensive review of Background Subtraction (BGS) techniques, specifically focused on the challenges and environmental conditions considered in P5 has been conducted. The review focuses on the classical methods, as well as the most recent developments in the field (2010 onwards). The documented review covers more than 30 state-of-the-art algorithms and their variants. Also methods for tracking aerial drones (UAVs) in thermal images are being developed. Furthermore, new image tracking algorithms have been developed, that are especially designed for thermal imagery and clearly outperforms previously published methods. Moreover, two methods for reducing background contamination effects of detection noise have been invented. The reduced sensitivity to detection noise improves the practical usability of the tracker, and the method also improves the accuracy and, in particular, the robustness. An example is given in the figure to the right where the state-of-the-art trackers lose track when the object enters an area with colder background and is partially occluded. This new level of image tracking performance together with multi sensor fusion and automatic threat detection enables new surveillance abilities beyond state-of-the-art.
The fusion of sensors in a surveillance network relies on accurate spatial calibration. The calibration effectively means that all sensor data, images for instance, can be related to a common coordinate system. In P5, both a best practice for multi-sensor calibration is developed, as well as an approach based on machine learning that automatically deduces correspondences and spatial mappings between sensors. A grand effort has been made to very accurately calibrate the sensors used at the abovementioned data acquisitions at CAST and OKG. The calibration work includes measurements at site, annotation of measured points and computation of extrinsic and intrinsic parameters of the sensor models. The accurate calibration is crucial when developing and evaluating detection and tracking algorithms. An example of the calibration results from the CAST data acquisition is shown in the photograph besides. The red crosses are marked calibration points on the ground. The green crosses are the same projected coordinates using the calibrated model. The yellow circles are projections one meter above the calibration points. The checkerboard is used to identify the internal (intrinsic) sensor model.
Architecture and integration
A so far preliminary surveillance system architecture has been developed that aims at a future platform facilitating sensor fusion, sensor feed-back, privacy preserving design and advanced human-machine interaction. An integration platform that will play a crucial role in the final demonstrations is under development. Data formats are being established and a middleware based on existing software libraries (ZeroMQ and ProtoBuf) has been decided, and communication prototypes have been proposed.