Unmanned ground and aerial systems for hidden threats detection – Organisation of a technological challenge

Topic description

Objective:

IED and landmine detection has been a research topic for many years. However, progress is hindered by the lack of standardised benchmarks. There is a need to rely on representative testing environments enabling an objective and comparable evaluation of developed systems.

Furthermore, field tests cannot be repeated at will and are not perfectly reproducible, especially for detection systems that involve artificial intelligence. Online tests of software components, for which measurements are easily reproducible and which enable short development cycles, should therefore also be organised. Since little data is readily available, data for online tests need to be collected during field tests organised previously during the challenge. This combination of field tests and online tests is needed to steer fast progress toward operational goals.

Scope and types of activities

Scope

Proposals should address the organisation of a technological challenge on IED and landmine detection based on the preliminary evaluation plan provided as part of the call documents. This includes the collection of data recorded by the participating teams during field tests, the annotation of this data and the sharing of the resulting databases.

Proposals should include clear descriptions of criteria to assess work package completion. Criteria should include the production of detailed evaluation plans agreed upon by all stakeholders, the production of the annotated databases needed for the evaluations, the production of measurements for all systems submitted to the tests by the participating teams following these plans, and the organisation of the needed events.

Types of activities

The following types of activities are eligible for this topic:

The proposals must address in particular the following as part of the mandatory activities:

• Setting up of the hardware and software infrastructures for testing hidden threat detection and characterisation technologies in the framework of the technological challenge
• Collection of sensor data from the participating teams, annotation of the data with ground truth information, and quality assessment, distribution and curation of databases
• Organisation of the evaluation campaigns, and in particular
  • Coordination of the exchanges with other stakeholders on the evaluation plans and elaboration of these plans
  • Management of the experimental hardware and software test campaigns and of the objective measurements of the performances of the systems submitted to the tests by the participating teams according to the protocols and metrics described in the evaluation plans
  • Organisation of the debriefing workshops

Functional requirements

The proposed solutions should enable to measure the performances of the tested systems according to detailed evaluation plans based on the preliminary evaluation plan provided as part of the call documents. Key aspects of the foreseen detailed evaluation plans and associated data management should be described in the proposals. Proposals should in particular describe:

• scenarios, nature and size of test ranges, and environmental conditions,
• types of devices, concealment, attack geography,
• nature and volume of data annotation,
• the framework for trusted sharing of data,
• the detailed planning of the test campaigns, including how runs can be organised in parallel on several test ranges,
• evaluation procedures (rules and tools to implement the metrics) and significance tests performed on measurements.

The testing environment should be able to accommodate for up to six participating teams.

During the challenge, drafts of the detailed evaluation plans should be submitted for discussion to the participating teams and to any stakeholder designated by the funding authority, early enough to take into account the feedback for the actual evaluation campaigns. Any evolution of the evaluation plans should take into account several factors: technical possibilities and cost, scientific relevance of the measurement, and representativeness of the metrics and protocols with respect to military needs. The justification of any change that is not subject to a consensus should be documented.

The expected impacts are

• Enhanced metrics and protocols to measure progress of R&D on IED and landmine detection and characterisation
• Standardisation of combined online and field testing for IED and landmine detection and characterisation
• Availability of databases to further develop and test equipment
• Enhanced clarity of system performances for all stakeholders, including system developers, funders and users
• Enhanced community building for the topic.