Current Projects

  1. Improving the specificity of affective computing via multimodal analysis
    ARC Discovery Project (22% Acceptance Rate in 2nd Round of Call)
    Runtime: 3 years
    Role: Principal Investigator, Co-author Proposal
    Partners: University of Canberra, University of Pittsburgh, CMU, Imperial College London
    Being able to have computational models and approaches to sense and understand a person’s emotion or mood is a core component of affective computing. While much research over the last two decades has tried to address the question of sensitivity – the correct recognition of affect classes – the equally important issue of specificity – the correct recognition of true negatives – has been neglected. This highly inter-disciplinary project aims to address this issue and to solve the fundamental affective computing problem of developing robust non-invasive multimodal approaches for accurately sensing a person’s affective state. Of course, neither sensitivity, nor specificity should be seen in isolation. The underlying issue is one of conceptualising affective states as areas within a continuous space, of determining the affect intensity on a continuous scale and of being able to analyse very subtle expressions of affect.
  2. ERIK: Entwicklung einer Roboterplattform zur Unterstützung neuer Interaktionsstrategien bei Kindern mit eingeschränkten sozioemotionalen Fähigkeiten
    BMBF IKT2020-Grant (Forschungsprogramm Roboter für Assistenzfunktionen: Interaktionsstrategien)
    Runtime: 01.11.2018 – 31.10.2021
    Role: Beneficiary, Scientific and Technical Manager (STM)
    Partners: Fraunhofer IIS, ASTRUM IT GmbH, Humboldt-Universität zu Berlin, Friedrich-Alexander-Universität Erlangen-Nürnberg, audEERING GmbH
    Das Verstehen und Ausdrücken von sozio-emotionalen Signalen, wie z. B. Gesichtsausdruck und Stimmenmodulation, ist bei Kindern mit Autismus beeinträchtigt. Während menschliche Interaktionspartner für sie schwer einzuschätzen sind, nehmen diese Kinder Roboter als vorhersehbarer und weniger komplex wahr. Häufig sind sie zudem technisch interessiert und aufgeschlossen. Zur Entwicklung der sozio-emotionalen Kommunikationsfähigkeiten autistischer Kinder wird im Projekt ERIK eine neue Therapieform mit Hilfe eines robotischen Systems entwickelt und erprobt. Der Roboter „Pepper“ erfasst in der Interaktion mit dem Kind die Mimik und Sprache. Durch das Spielen mit dem Roboter-Ball „Leka“ kann zusätzlich über Elektroden der Puls ermittelt werden. „Pepper“ interpretiert diese Signale und leitet in Echtzeit Emotionen ab. Kombiniert mit der Therapie-App „Zirkus Empatico“ können alltagsrelevante emotionale und soziale Fähigkeiten trainiert werden. Durch das Erkennen von Interesse, Frustration und Langweile des Kindes können die Therapieszenarien individuell angepasst werden. Mittels Gesten und Augenbewegungen kann „Pepper“ lebensnah mit Kindern interagieren, wobei Ängste im Umgang mit Menschen reduziert werden können. Der innovative Therapieansatz erlaubt Therapeuten, Interaktionen genauer zu beobachten und auszuwerten, da sie selbst nicht mehr Teil der Interaktion sind. Die emotionssensitive Robotik kann außerdem erstmalig auch mit Gruppen von Kindern interagieren.
  3. sustAGESmart environments for person-centered sustainable work and well-being (#826506)
    EU Horizon 2020 Research & Innovation Action (RIA)
    Runtime: 36 months
    Role: Principal Investigator, Scientific and Technical Manager (STM), Workpackage Leader, Co-Author Proposal
    Partners: Foundation for Research and Technology Hellas, Centro Ricerche Fiat SCPA, Software AG, Imaginary SRL, Forschungsgesellschaft für Arbeitsphysiologie und Arbeitsschutz e.V., Heraklion Port Authority S.A., Aegis IT Research UG, University of Augsburg, Aristotelio Panepistimio Thessalonikis, Universidad Nacional de Educacion a Distancia
    sustAGE aims to develop a person-centered solution for promoting the concept of “sustainable work” for EU industries.
    The project provides a paradigm shift in human machine interaction, building upon seven strategic technology trends, IoT, Machine learning, micro-moments, temporal reasoning, recommender systems, data analytics and gamification to deliver a composite system integrated with the daily activities at work and outside, to support employers and ageing employees to jointly increase well-being, wellness at work and productivity. The manifold contribution focuses on the support of the employment and later retirement of older adults from work and the optimization of the workforce management. The sustAGE platform guides workers on work-related tasks, recommends personalized cognitive and physical training activities with emphasis on game and social aspects, delivers warnings regarding occupational risks and cares for their proper positioning in work tasks that will maximize team performance.
    By combining a broad range of the innovation chain activities namely, technology R&D, demonstration, prototyping, pilots, and extensive validation, the project aims to explore how health and safety at work, continuous training and proper workforce management can prolongue older workers’ competitiveness at work. The deployment of the proposed technologies in two critical industrial sectors and their extensive evaluation will lead to a ground-breaking contribution that will improve the performance and quality of life at work and beyond for many ageing adult workers.
  4. WorkingAge: Smart Working environments for all Ages (#210487208)
    EU Horizon 2020 Research & Innovation Action (RIA)
    Runtime: 36 months
    Role: Principal Investigator, Co-Author Proposal
    Partners:  Instituto Tecnológico de Castilla y Leon, Exodus Anonymos Etaireia Pliroforikis, University of Cambridge, Politecnico di Milano, Green Communications SAS, Brainsigns SRL, RWTH Aachen, Telespazio France SAS, audEERING GmbH, European Emergency Number Association ASBL, Fundacion Intras, Telematic Medical Applications MEPE
    WorkingAge will use innovative HCI methods (augmented reality, virtual reality, gesture/voice recognition and eye tracking) to measure the user emotional/cognitive/health state and create communication paths. At the same time with the use of IoT sensors it will be able to detect environmental conditions. The purpose is to promote healthy habits of users in their working environment and daily living activities in order to improve their working and living conditions. By studying the profile of the >50 (year old) workers and the working place requirements in three different working environments (Office, Driving and Manufacturing), both profiles (user and environment) will be considered. Information obtained will be used for the creation of interventions that will lead to healthy aging inside and outside the working environment. WorkingAge will test and validate an integrated solution that will learn the user’s behaviour, health data and preferences and through continue data collection and analysis will interact naturally with the user. This innovative system will provide workers assistance in their everyday routine in the form of reminders, risks avoidance and recommendations. In this way the WorkingAge project will create a sustainable and scalable product that will empower their users’ easing their life by attenuating the impact of aging in their autonomy, work conditions, health and well-being.
  5. HOL-DEEP-SENSE: Holistic Deep Modelling for User Recognition and Affective Social Behaviour Sensing
    EU Horizon 2020 Marie Skłodowska-Curie action Individual Fellowship (MASCA-IF 2017)
    Runtime: 30 months
    Role: Coauthor Proposal, Coordinator, Beneficiary, Supervisor
    Partners: University of Augsburg, Massachussetts Insititute of Technology
    The “Holistic Deep Modelling for User Recognition and Affective Social Behaviour Sensing” (HOL-DEEP-SENSE) project aims at augmenting affective machines such as virtual assistants and social robots with human-like acumen based on holistic perception and understanding abilities. Social competencies comprising context awareness, salience detection and affective sensitivity present a central aspect of human communication, and thus are indispensable for enabling natural and spontaneous human-machine interaction. Therefore, with the aim to advance affective computing and social signal processing, we envision a “Social Intelligent Multi-modal Ontological Net” (SIMON) that builds on technologies at the leading edge of deep learning for pattern recognition. In particular, our approach is driven by multi-modal information fusion using end-to-end deep neural networks trained on large datasets, allowing SIMON to exploit combined auditory, visual and physiological analysis. In contrast to standard machine learning systems, SIMON makes use of task relatedness to adapt its topology within a novel construct of subdivided neural networks. Through deep affective feature transformation, SIMON is able to perform associative domain adaptation via transfer and multi-task learning, and thus can infer user characteristics and social cues in a holistic context. This new unified sensing architecture will enable affective computers to assimilate ontological human phenomena, leading to a step change in machine perception. This will offer a wide range of applications for health and wellbeing in future IoT-inspired environments, connected to dedicated sensors and consumer electronics. By verifying the gains through holistic sensing, the project will show the true potential of the much sought-after emotionally and socially intelligent AI, and herald a new generation of machines with hitherto unseen skills to interact with humans via universal communication channels.
  6. Sentiment Analyse
    Industry Cooperation with BMW AG
    Runtime: 01.05.2018 – 31.05.2021
    Role: Principal Investigator
    Partners: University of Augsburg, BMW AG
    The project aims at real-time internet-scale sentiment analysis in unstructured multimodal data in the wild.
  7. An Embedded Soundscape System for Personalised Wellness via Multimodal Bio-Signal and Speech Monitoring – 7% acceptance rate in the call
    ZD.B Fellowship
    Runtime: 01.01.2018 – 31.12.2020
    Role: Supervisor, Co-Author Proposal
    Partners:  University of Augsburg
    The main research aim is to explore how diverse multimodal data can inform the production of personalised embedded soundscapes, and how such digitally produced soundscapes can improve human wellness. As highlighted by ZD.B Digital Health / Medicine, digitisation in health care shows great potential. The proposed could be effective in a variety of scenarios, including nervousness. Imagine the hours before an important presentation and the presenter’s nerves are building. The presenter could use a smart-device application, to provide a speech instance (whilst monitoring pulse). The application returns a (user dependent) soundscape which clinically reduces the negative feeling. To explore this, the project will be divided into 3 phases (detailed in section 5), each a fundamental part for development of such wellness systems. Questions will arise, pertaining to both human audible, and speech perception with observations of current ‘norms’ in data science, contributing to the ethics involved in artificial intelligence.
  8. ECoWeB: Assessing and Enhancing Emotional Competence for Well-Being (ECoWeB) in the Young: A principled, evidence-based, mobile-health approach to prevent mental disorders and promote mental wellbeing (#754657)
    EU Horizon 2020 Research & Innovation Action (RIA)
    : 01.01.2018 – 31.12.2021
    Role: Principal Investigator, Innovation Manager, Innovation Management Board Chair, Workpackage Leader, Co-Author Proposal
    Partners:  University of Exeter, audEERING GmbH, Vysoke Uceni Technicke v Brne, Institute of Communication and Computing Systems, Universitat Jaume i de Castellon, Fraunhofer Gesellschaft, University of Oxford, University of Geneva, LMU Munich, University of Gent, Monsenso ApS, University of Copenhagen, Deutsches Jugendinstitut eV
    Although there are effective mental well-being promotion and mental disorder prevention interventions for young people, there is a need for more robust evidence on resilience factors, for more effective interventions, and for approaches that can be scalable and accessible at a population level. To tackle these challenges and move beyond the state-of-the-art, ECoWeB uniquely integrates three multidisciplinary approaches: (a) For the first time to our knowledge, we will systematically use an established theoretical model of normal emotional functioning (Emotional Competence Process) to guide the identification and targeting of mechanisms robustly implicated in well-being and psychopathology in young people; (b) A personalized medicine approach: systematic assessment of personal Emotional Competence (EC) profiles is used to select targeted interventions to promote well-being: (c) Mobile application delivery to target scalability, accessibility and acceptability in young people. Our aim is to improve mental health promotion by developing, evaluating, and disseminating a comprehensive mobile app to assess deficits in three major components of EC (production, regulation, knowledge) and to selectively augment pertinent EC abilities in adolescents and young adults. It is hypothesized that the targeted interventions, based on state-of-the-art assessment, will efficiently increase resilience toward adversity, promote mental well-being, and act as primary prevention for mental disorders. The EC intervention will be tested in cohort multiple randomized trials with young people from many European countries against a usual care control and an established, non-personalized socio-emotional learning digital intervention. Building directly from a fundamental understanding of emotion in combination with a personalized approach and leading edge digital technology is a novel and innovative approach, with potential to deliver a breakthrough in effective prevention of mental disorder.
  9. TAPAS: Training network on Automatic Processing of PAthological Speech (#766287)
    EU H2020 Marie Sklodowska-Curie Innovative Training Networks European Training Networks (MSCA-ITN-ETN:ENG)
    TAPAS: Training Network on Automatic Processing of PAthological Speech
    Runtime: 01.11.2017 – 31.10.2021
    Role: Principal Investigator, Co-Author Proposal
    Partners: IDIAP, Université Paul Sabatier Toulouse III, Universitair Ziekenhuis Antwerpen, FAU Erlangen-Nürnberg, Stichting Katholieke Universiteit, INESC ID, LMU Munich, Interuniersitair Micro-Electronicacentrum IMEC, Stichting het Nederlands Kanker Instituutantoni van Leeuwenhoek Ziekenhuis, University of Augsburg, University of Sheffield, audEERING GmbH
    There are an increasing number of people across Europe with debilitating speech pathologies (e.g., due to stroke, Parkinson’s, etc). These groups face communication problems that can lead to social exclusion. They are now being further marginalised by a new wave of speech technology that is increasingly woven into everyday life but which is not robust to atypical speech. TAPAS is proposing a programme of pathological speech research, that aims to transform the well-being of these people. The TAPAS work programme targets three key research problems: (a) Detection: We will develop speech processing techniques for early detection of conditions that impact on speech production. The outcomes will be cheap and non-invasive diagnostic tools that provide early warning of the onset of progressive conditions such as Alzheimer’s and Parkinson’s. (b) Therapy: We will use newly-emerging speech processing techniques to produce automated speech therapy tools. These tools will make therapy more accessible and more individually targeted. Better therapy can increase the chances of recovering intelligible speech after traumatic events such a stroke or oral surgery. (c) Assisted Living: We will re-design current speech technology so that it works well for people with speech impairments and also helps in making informed clinical choices. People with speech impairments often have other co-occurring conditions making them reliant on carers. Speech-driven tools for assisted-living are a way to allow such people to live more independently. TAPAS adopts an inter-disciplinary and multi-sectorial approach. The consortium includes clinical practitioners, academic researchers and industrial partners, with expertise spanning speech engineering, linguistics and clinical science. All members have expertise in some element of pathological speech. This rich network will train a new generation of 15 researchers, equipping them with the skills and resources necessary for lasting success.
  10. OPTAPEB: Optimierung der Psychotherapie durch Agentengeleitete Patientenzentrierte Emotionsbewältigung (#V5IKM010)
    BMBF IKT2020-Grant (Forschungsprogramm zur Mensch-Technik-Interaktion: Technik zum Menschen bringen – Interaktive körpernahe Medizintechnik)
    Runtime: 01.11.2017 – 31.10.2020
    Role: Beneficiary
    Partners: Universität Regensburg, Fraunhofer IIS, VTplus GmbH, Ambiotex GmbH, NTT GmbH, eHealthLabs, audEERING GmbH
    OPTAPEB aims to develop an immersive and interactive virtual reality system that assists users in curing phobia. The system will allow to experience situations of phobia and protocol this emotional experience and the user’s behaviour. Various levels of emotional reactions will be monitored continuously and in real time by the system that applies sensors based on innovative e-wear technology, speech signals, and other pervasive technologies (e.g. accelerometres). A further goal of the project is the development of a game-like algorithm to control the user experience of anxieties through exposure therapy and to adapt the course of the therapy to the user needs and the current situation automatically.
  11. Curiosity and Data-Efficiency in Reinforcement Learning via Bayesian Deep Networks
    HiPEDS EPSRC / Presidential Scholarship, Imperial College, Industry integrated Centre for Doctoral Training
    Runtime: 01.10.2017 – 30.09.2021
    Role: Supervisor, Co-author Proposal
    Partners: Imperial College London
    In reinforcement learning, an agent tries to maximise some notion of expected future cumulative reward by sequentially selecting actions that affect its environment. Its applications are many and range from embodied robotics, through biomedical engineering, to human-robot interaction via speech and emotion recognition. However, in many such applications the great amounts of real data required to effectively train the agent may be difficult or expensive to obtain and an inadequate agent policy can be catastrophic. Here I propose the combination of a) environment models that make more efficient use of the agent’s past experience and b) directed exploration that encourages the exploration of more informative locations in the environment – both by means of Bayesian deep neural networks. Recent advances in the latter show them to combine the ability of deep neural networks to learn from complex, high-dimensional data and also the aversion to overfitting and learning with uncertainty of Bayesian methods. With this approach, I expect to be able to design reinforcement learning solutions that can exhibit an improved usage of data both at the beginning and at later stages of the agent’s exploration.
  12. ACLEW: Analyzing Child Language Experiences Around the World (HJ-253479) – 14 winning projects in total
    T-AP (Trans-Atlantic Platform for the Social Sciences and Humanities along with Argentina (MINCyT), Canada (SSHRC, NSERC), Finland (AKA), France (ANR), United Kingdom (ESRC/AHRC), United States (NEH)) Digging into Data Challenge 4th round

    Runtime: 01.06.2017 – 31.05.2020
    Role: Principal Investigator, Co-Author Proposal
    Partners: Duke University, École Normale Supérieure, Aalto University, CONICET, Imperial College London, University of Manitoba, Carnegie Mellon University, University of Toronto
    An international collaboration among linguists and speech experts to study child language development across nations and cultures to gain a better understanding of how an infant’s environment affects subsequent language ability.
  13. Evolutionary Computing: The Changing Mind
    HiPEDS EPSRC, Imperial College, Industry integrated Centre for Doctoral Training (CDT)
    Runtime: 01.04.2017 – 31.03.2021
    Role: Supervisor
    Partners: Imperial College London
    This project aims to (1) innovate upon NeuroEvolution of Augmenting Topologies (NEAT), (2) permit function extraction for Transfer Learning, (3) find ways to merge evolutionary computation with broader systems and (4) deploy methods using the latest processing technology – “NeuroMorphic” computing chips.
  14. RADAR CNS: Remote Assessment of Disease and Relaps in Central Nervous System Disorders (#115902) – 15.8% acceptance rate in the call
    EU H2020 / EFPIA Innovative Medicines Initiative (IMI) 2 Call 3

    Runtime: 01.04.2016 – 31.03.2021
    Role: Coauthor Proposal, Beneficiary, Principal Investigator, Workpackage Leader
    Partners: King’s College London, Provincia Lombardo-Veneta – Ordine Ospedaliero di San Giovanni di Dio— Fatebenefratelli Lygature, Università Vita-Salute San Raffaele, Fundacio Hospital Universitari Vall D’Hebron, University of Nottingham, Centro de Investigacion Biomedica en Red, Software AG, Region Hovedstaden, Stichting VU-Vumc, University Hospital Freiburg, Stichting IMEC Nederland, Katholieke Universiteit Leuven, Northwestern University, Stockholm Universitet, University of AugsburgUniversity of Passau, Università degli Studi di Bergamo, Charité – Universitätsmedizin Berlin, Intel Corporation (UK) Ltd, GABO:mi, Janssen Pharmaceutica NV, H. Lundbeck A/S, UCB Biopharma SPRL, MSD IT Global Innovation Center
    The general aim is to develop and test a transformative platform of remote monitoring (RMT) of disease state in three CNS diseases: epilepsy, multiple sclerosis and depression. Other aims are: (i) to build an infrastructure to identify clinically useful RMT measured biosignatures to assist in the early identification of relapse or deterioration;  (ii) to develop a platform to identify these biosignatures;  (iii) to anticipate potential barriers to translation by initiating a dialogue with key stakeholders (patients, clinicians, regulators and healthcare providers).
  15. EngageME: Automated Measurement of Engagement Level of Children with Autism Spectrum Conditions during Human-robot Interaction (#701236) – 14.4% acceptance rate in the call
    EU Horizon 2020 Marie Skłodowska-Curie action Individual Fellowship (MASCA-IF 2015)

    Runtime: 01.09.2016 – 31.08.2019
    Role: Coauthor Proposal, Coordinator, Beneficiary, Supervisor
    Partners: University of AugsburgUniversity of Passau, Massachussetts Insititute of Technology
    Engaging children with ASC (Autism Spectrum Conditions) in communication centred activities during educational therapy is one of the cardinal challenges by ASC and contributes to its poor outcome. To this end, therapists recently started using humanoid robots (e.g., NAO) as assistive tools. However, this technology lacks the ability to autonomously engage with children, which is the key for improving the therapy and, thus, learning opportunities. Existing approaches typically use machine learning algorithms to estimate the engagement of children with ASC from their head-pose or eye-gaze inferred from face-videos. These approaches are rather limited for modeling atypical behavioral displays of engagement of children with ASC, which can vary considerably across the children. The first objective of EngageME is to bring novel machine learning models that can for the first time effectively leverage multi-modal behavioural cues, including facial expressions, head pose, vocal and physiological cues, to realize fully automated context-sensitive estimation of engagement levels of children with ASC. These models build upon dynamic graph models for multi-modal ordinal data, based on state-of-the-art machine learning approaches to sequence classification and domain adaptation, which can adapt to each child, while still being able to generalize across children and cultures. To realize this, the second objective of EngageME is to provide the candidate with the cutting-edge training aimed at expanding his current expertise in visual processing with expertise in wearable/physiological, and audio technologies, from leading experts in these fields. EngageME is expected to bring novel technology/models for endowing assistive robots with ability to accurately ‘sense’ engagement levels of children with ASC during robot-assisted therapy, while providing the candidate with a set of skills needed to become one of the frontiers in the emerging field of affect-sensitive assistive technology.
  16. DE-ENIGMA: Multi-Modal Human-Robot Interaction for Teaching and Expanding Social Imagination in Autistic Children (#688835) – 6.9% acceptance rate in the call

    EU Horizon 2020 Research & Innovation Action (RIA)
    Runtime: 01.02.2016 – 31.07.2020
    Role: Coauthor Proposal, Beneficiary, Principal Investigator, WP Leader
    Partners: University of Twente, Savez udruzenja Srbije za pomoc osobama sa autizmom, Autism-Europe, IDMIND, University College London, University of AugsburgUniversity of Passau, Romane Institute of Mathematics Simion Stoilow of the Romanian Academy, Imperial College London
    Autism Spectrum Conditions (ASC, frequently defined as ASD — Autism Spectrum Disorders) are neurodevelopmental conditions, characterized by social communication difficulties and restricted and repetitive behaviour patterns. There are over 5 million people with autism in Europe – around 1 in every 100 people, affecting lives of over 20 million people each day. Alongside their difficulties, individuals with ASC tend to have intact and sometimes superior abilities to comprehend and manipulate closed, rule-based, predictable systems, such as robotbased technology. Over the last couple of years, this has led to several attempts to teach emotion recognition and expression to individuals with ASC, using humanoid robots. This has been shown to be very effective as an integral part of the psychoeducational therapy for children with ASC. The main reason for this is that humanoid robots are perceived by children with autism as being more predictable, less complicated, less threatening, and more comfortable to communicate with than humans, with all their complex and frightening subtleties and nuances. The proposed project aims to create and evaluate the effectiveness of such a robot-based technology, directed for children with ASC. This technology will enable to realise robust, context-sensitive (such as user- and culture-specific), multimodal (including facial, bodily, vocal and verbal cues) and naturalistic human-robot interaction (HRI) aimed at enhancing the social imagination skills of children with autism. The proposed will include the design of effective and user-adaptable robot behaviours for the target user group, leading to more personalised and effective therapies than previously realised. Carers will be offered their own supportive environment, including professional information, reports of child’s progress and use of the system and forums for parents and therapists.
  17. Promoting Early Diagnosis of Rett Syndrome through Speech-Language Pathology
    (Akustische Parameter als diagnostische Marker zur Früherkennung von Rett-Syndrom) (#16430)
    Österreichische Nationalbank (OeNB) Jubiläumsfonds
    Runtime: 01.11.2015 – 31.10.2019
    Role: Main Cooperation Partner
    Partners: Medical University of Graz, Karolinska Institutet, Boston Children’s Hospital and Harvard Medical School, University of Passau, Imperial College London, Victoria University of Wellington


Comments are closed.