Enhanced pilot environment for air combat

Topic description

Objective:

The future warfare is largely characterised by weapon system automation and networking. While being implemented in all military domains, the concept of swarming and autonomy is in particular evolving in the air domain. Such evolutions have the potential to increase next generation air combat assets effectiveness because connectivity would allow accessing an increased amount of information thus contributing to build a more comprehensive operational picture and UAS assets contributing to the execution of specific mission tasks would multiply the operational impact.

As a result, a large number of actors, effectors, and sensors will be connected, generating an amazing collection of information and data. This induces a great challenge to put the pilots at the centre of missions.

From an air combat pilot perspective, the above evolution progressively adds information for situation awareness building and mission management tasks going beyond the ownship to supervise other platforms under his/her responsibility. The human-in-control principle would imply the risk of information overload or that key aspects of the mission are overlooked.

In order to match the capabilities brought by the above enablers with pilot effectiveness, interfaces need to become more flexible and be able to drive pilot attention to the best course of action. In other words, the cockpit HMI ^[1], despite the larger amount of information available and the management tasks going beyond the ownship, should evolve to enhance the pilot decision making and action process and timing. Main characteristics would be the capability to delegate under human supervision, an increasing number of tasks to more and more autonomous systems and the capability to adjust to new and unexpected situations that will enable to cut short the cognitive load of operators within a specific framework.

This context will require the development of new sets of equipment and software more and more sophisticated which could take advantage from new technologies like wearable, visionics, haptics, vocal command, virtual operator assistant, augmented reality, 3D holography, implementation concepts, artificial intelligence and autonomy. This will free men from repetitive tasks so they can focus their resources on high value fields of action, thereby improving combat effectiveness.

Specific challenge

From the human-machine relationship point of view, new generation military aircraft inserted in this collaborative air combat will require a new generation of man-machine relationship that allows an ergonomic cooperation between the crew and the machine, a performed effectiveness and a safe flight, as well as the cooperation with other assets, including unmanned ones. The new technologies will allow gaining tactical advantage by assisting the crew as a real co-pilot that answers the crew requests, proposes tactics and procedures and adapts the interfaces to the crew.

The definition of a novel design and interaction principles for managing automated/autonomous aircraft functions and cooperating with System-of-Systems team mates, including adaptive interfaces can be defined as man-machine teaming (MMT).

Taking into account the new paradigm, the following subjects could be addressed:

- New or disruptive HMI technologies including for instance displays, wearables, vocal dialog, augmented reality, stereoscopy;

- Pilot state monitoring in relation to the mission and systems status;

- Services for assisted decision- making support (based on advances techniques like Artificial Intelligence not excluding other approaches).

Preliminary analyses show that in order to pursue those challenges, the future European aerial combat systems will need to be equipped with an innovative cockpit offering the pilot breakthrough display and interaction capabilities. In this context, it seems clear that new products (head-down, eyes-out, interface modalities, virtual assistant...) have to be developed.

Hence, this topic addresses the rise in maturity, with the objective to reach TRL ^[2] 4, supported by demonstrations, of technological and technical solutions necessary for future enhanced products.

The proposals may consider existing manned and unmanned air platforms and future ones under development, including training aircraft in a long term perspective or as quick-win.

Against the background of the design of new generation air combat platforms in Europe, or upgrades of those today in service, the following themes have to be considered:

- Adaptive human system collaboration: adaptive collaborative HMI for operations in a distributed environment with multiplatform assets and the definition of novel design and interaction principles for managing automated/autonomous aircraft functions and cooperating with System-of-Systems team mates, including adaptive interfaces;

- Visualisation: both visualisation products and advanced pilot information presentation capabilities;

- Crew monitoring system: systems and techniques able to support and assist pilots, and in general human operators in performing the flight and mission control in a more demanding operational environment;

- Interaction modalities: the need for innovative HMI technologies including e.g. wearable, visionics, haptics, vocal command, virtual operator assistant, Augmented Reality, 3D holography and implementation concepts;

Emulation of the pilot interactions with its environment might be addressed when needed in a transversal way within the studied areas.

All of those themes will be able to rely, at different levels, on different technology building blocks exploiting the opportunity to use advanced research techniques such as artificial intelligence, machine learning or others that can enable more advanced capabilities for the overall mission performance. This topic is therefore transversal. However, as a complement, it may be interesting to study the theme of "Decision-making system" whose objective would be to prioritize, order and present, whatever the situation during the mission, the most relevant information to the pilot with an objective of efficiency and safety.

Whatever the theme considered, quick wins must be identified, evaluated and tested so as to prepare their implementation on current or upcoming systems.

Targeted activities

The proposals must cover the following activities as referred in article 10.3 of the EDF Regulation, not excluding upstream and downstream activities eligible for development actions if deemed useful to reach the objectives:

- Studies, such as feasibility studies to explore the feasibility of new or improved technologies, products, processes, services and solutions, related to advanced air combat cockpit HMI functions and related technologies enabling effective multi-role and networked operations including MUM-T in a highly contested environment.

In particular, the targeted activities are:

- Operational and training use-cases definition to elaborate specifications (performances, safety objectives…). For that purpose, workshops will be implemented with participating Member States and associated countries ministry of defence representatives, including end-users, to establish high-level operational requirements and relevant scenarios.

- For each of the abovementioned themes, relevant technology identification, analysis, including quick wins opportunities identification and analysis leading to the demonstration and development activities dedicated to these opportunities to be performed in a shorter timeframe in order to enable quick implementation process.

- Technologies evaluations and demonstrations. These activities will help on to select the relevant technologies to upgrade and to demonstrate further the efficiently of these upgraded technologies through operational scenarios.

- Relevant technologies maturation and development.

In order to maximize inter-domain synergies and take advantage of distributed expertise, proposals must address the abovementioned themes according to their following respective description, notwithstanding others fields of interest leading to identify new technologies to be explored, preliminary developed and demonstrated:

Theme N°1: Adaptive human system collaboration

This theme addresses the definition of a new paradigm of human-machine teaming in future collaborative and connected air warfare. New generation military aircraft participating in collaborative air combat will require a human system interface that enhances the awareness of the tactical situation and allows an ergonomic cooperation between crews and machines for a safe flight and high performance in cooperation with both manned and unmanned assets. This will cover human-machine/human-human/machine-machine Teaming, but will not include functional algorithms.

Legacy human-machine interfaces lack the necessary flexibility and adaptability to meet the demands of future combat systems. To avoid compromising the effectiveness of human operators in the future, applied research is required to address topics such as:

- HMI principles for cross platform mission management considering (human-machine/human-human/machine-machine teaming, not including the specific functional algorithms;

- Adaptive HMI mechanisms e.g. based on crew management system (CMS) data and in accordance with the specific operational context.

This will then lead to main characteristics of the Human Machine Interface as follows:

- Strengthened and adaptive cooperation between all systems, either manned or unmanned, involved in an operation;

- Human supervised delegation of tasks to more and more autonomous systems;

- “Real co-pilot” like assistance to provide the crew with system proposals and to adapt interfaces.

Theme N°2: Visualisation

This theme addresses both visualization and advanced pilot information presentation capabilities, including 3D presentation, and other novel presentations that could be implemented in the next generation of aircraft, through:

- Augmented reality, large area displays (free form, multi touch, auto-stereoscopy), 3D holography and implementation concept;

- Helmet mounted display (HMD) solutions crucial for the next generation cockpit. Technological solutions exploration should be carried out for increasing technical characteristics in terms of presentation field and functional capabilities (integrated night vision, primary flight display function, and support for CMS sensors, target designation and view through the cockpit. It will also have to take into account the control of its inertia characteristics (mass and centre of gravity of the HMD carried by the pilot's head).

Therefore, the following areas could be investigated:

- Digital integrated night vision;

- HMD wireless link;

- Enhanced synthetic vision system (including live virtual constructive visual integration).

Under the scope of this theme, there will be demonstrations with physical, digital mock-up and/or simulations on the basis of operational “use-cases”. An iterative implementation of research findings will be conducted to continuously optimise the performance of the demonstration also based on the initial user requirements.

Theme N°3: Crew monitoring system

This theme concerns the real-time monitoring of the physiological and cognitive states of the crew. The elements of interest, or deleterious capacities, to be monitored are, for example, operator incapacity (G-LoC ^[3], hypoxia, spatial disorientation...), hypovigilance, attentional tunnelling, mental workload, stress and situational awareness. These aspects are crucial in particular for the future air combat systems where the operational environment and the way of operate are significantly more complex than the current ones. Crew monitoring system can be applied to operational embedded systems as well as to training systems (embedded or on ground).

In order to more specifically mature the CMS models the following areas could be investigated:

- Mental workload;

- Ability to collaborate;

- Situational Awareness;

- Stress.

The validation of CMS models is a crucial point in the CMS chain's rise to maturity also based on a pilot behavioural knowledge base (PBKB) that needs to be contextualised in accordance to the diversity of human, missions and tasks, including through AI and ML-based techniques.

Under the scope of this theme, there will be demonstrations with physical, digital mock-up and/or simulations on the basis of operational use-cases. An iterative implementation of research findings will be conducted to continuously optimise the performance of the demonstration.

Theme N°4: Interaction modalities

This theme addresses both the modalities of interaction as well as their combination in the field of e.g. wearable, visionics, haptics, vocal command, touch, gesture, etc.

In terms of means, it takes into account:

- Audio, in terms of input/outputs: voice command, natural language processing, in a very constrained environment such as that of a fighter, voice synthesis and advanced audio functions such as 3D Sound for example;

- Eye: the eye tracker which is used as a CMS sensor is here dedicated to interaction. Coupled with another modality such as voice, it is a vector of efficiency for target designation in an eyes-in or eyes-out use;

- Touch: a particular objective will be to study multi-touch (up to 5 fingers) technologies to interact with the displays;

- Gesture controls;

- Haptic/Tactile display of information;

- Handwriting recognition and more generally the ability to interact naturally with a "white board".

Multimodality would provide greater security, resilience and accuracy by removing ambiguity about the operator's intentions.

Under the scope of this theme, there will be demonstrations with physical/digital mock-up and/or simulations on the basis of operational use-cases. An iterative implementation of research findings will be conducted to continuously optimise the performance of the demonstration.

Functional requirements

The proposals must fulfil the following requirements:

(1) In order to support the future air collaborative combat:

- Take into account the new paradigm of human-machine teaming in the future collaborative and connected air warfare and adaptive cooperation between all systems, either manned or unmanned, involved in multi-assets operation:

• through a strengthened and adaptive cooperation between all systems, either manned or unmanned, involved in an operation,
• based on a human-supervised delegation of tasks leading to more and more autonomous systems,
• using a “real co-pilot”-like assistance providing the crew with system proposals,
• piloting performance monitoring for adaptive HMI.

(2) In order to improve human-machine performances:

- Human-machine performance must be evaluated according to different criteria to be defined in the proposals;

- Human factor aspects must be taken into account to develop the technologies (especially physical and cognitive ergonomics);

- Physical ergonomics must take into account the anthropometrics data considered for the air combat domain.

(3) The proposals must comply with the following specific technical requirements:

- The technologies have to be scalable for existing fighters or future fighters in order to apply the “quick-win” principle;

- Each technological building and capability blocks must be evaluated and demonstrated through physical or digital mock-ups and simulations based on the studied “use-cases” scope.

(4) The proposals must ensure that the treatments of data collection about humans to build models or algorithms are compliant with the European GDPR (General Data Protection Regulation) rulemaking.

- European platforms for enhanced combat pilot technologies tests and demonstrations to welcome joint or national tests and demonstration needs.

- Consolidation of a sector of excellence in Europe for enhanced combat pilot based on innovative technologies.

- Generation of inputs for the mid-term and long-term development of next generation air combat cockpit HMI.

- Definition and “demonstrator development” of novel cockpit HMI technologies.

- Increase mission capability, efficiency, effectiveness and performance in air combat missions (more safe for pilot and limiting the collateral damage) exploiting the emerging technologies.

- Provision of a potential starting point for developing EU guidelines in the frame of advanced HMI design for managing systems-of-systems operations.

- Provision of an opportunity for cross-ministries of defence and cross-industries exchanges in the subject of cockpit design and pilot operating procedures.

- Strengthen European industry in advanced air combat cockpit technologies independent of third countries.

- Quick wins identification to be implemented on current or upcoming systems.

[1]Human machine interface

[2]Technology readiness level

[3]G-force induced loss of consciousness