Multimodal and Natural computer interaction

Topic description

Scope:

Specific Challenge: As devices and systems are becoming increasingly powerful, the interface between human and computer is often lagging behind and constitutes a bottleneck for seamless and efficient use.  Leveraging on multidisciplinary expertise combining knowledge from both the technological and human sciences, new technologies need to offer interactions which are closer to the communication patterns of human beings and allow a simple, intuitive and hence more "natural" communication with the system.

Scope: The topic will be addressed by the following focused actions: 

a.      Research & Innovation Actions: Provide interactive information retrieval systems with more efficient and natural ways of delivering answers to users' queries especially in unexpected and/or difficult circumstances. This should be supported by research on knowledge-based autonomous human-like social agents that can handle and learn from conversational spoken and multimodal interaction as well as react proactively to new communicative situations. Systems should cope with spontaneous spoken dialogue and gestural interaction, in multiple languages, and exhibit adequate communicative, conversational, affective and social capabilities in relation to the domain/task under consideration and the needs and abilities of the user. Technologies should be designed to match multiple delivery platforms and be demonstrated in real environments, while research is expected to be based on and/or produce freely available and re-usable resources.

b.      Research & Innovation Actions: Develop novel multi-modal, adaptive interfaces, including Brain Computer Interfaces, assisting people with disabilities. Research should explore: how users interact and cooperate with (intelligent) systems, including user modelling aspects for the identification of necessary abilities for different functions and environments; how to detect behaviours, emotions and intentions of the user; how to sense and understand the environment and other context factors; how multimodal (including nonverbal) interaction is used in ambient environments. Activities may cover also interoperability standards (for software and devices) as well as interaction and cooperation between machine intelligence in environments and human intelligence.

c.       Innovation Actions: Develop and validate innovative multimodal interfaces to provide more efficient and natural ways of interacting with computers and improve users' experience. Leveraging on one or multiple smart devices and sensors with capabilities such as scene analysis, voice recognition, human position, gestures and body language detection capabilities, such systems must provide non-intrusive interaction with human where real and virtual content are blended. Built with a user centric approach, solutions should be cost effective; address clear market needs and be validated in domains such as those of the creative industries fields.

Expected impact:

a.      Research & Innovation Actions

         Improve multilingual speech processing and bridging the gap between recognition and synthesis, exploiting metadata and other contextual data.

         Increase the automatic inferences capacities from rich context thanks to improved language understanding, sensed environments/objects, use of social media and agent’s experience.

b.      Research & Innovation Actions

         Advance the capacity of human-machine interaction technologies to enable disabled and elderly people to fully participate in society.

c.       Innovation Actions

         Enable better uses of ICT technologies within the creative industries by providing directly usable solutions addressing their specific needs.

         Provide a large spill over of the knowledge acquired to a maximum of European industries.

         Improve the competitive position of the European industries through the provision of cost effective, innovative and high-value products and services.

Types of action:

a.       Research & Innovation Actions – Proposals requesting a Small contribution are expected

b.      Research & Innovation Actions – Proposals requesting a Small contribution are expected

c.       Innovation Actions – Proposals requesting a Small contribution are expected

Socio-economic science and humanities