Visual emotion recognition and physiological feedback during therapy

Development of a robotic platform to aid new interactive strategies for children with impaired socioemotional functioning


Development of an emotion-sensitive robotic platform featuring video-based emotion recognition and multimodal analysis and evaluation of biosignals as visual feedback support for children on the autism spectrum. Acting as a social reaction and interaction partner between a child and their therapist, the new platform uses sensor-based technologies (facial expressions, psycho-physiological signals, language) to enhance participants’ communication skills. Direct, real-time feedback from the robot helps children learn to correctly gauge others’ feelings, as well as become aware of their own emotions.

The planned open framework and interface design of the platform mean that other use cases are conceivable, too, such as the treatment of dementia patients.


Some things that most people do entirely at the unconscious level pose enormous challenges for autistic children: correctly recognizing and interpreting the emotions of the person they are interacting with and responding accordingly. Using robots as an interactive communication tool in therapy allows children on the autism spectrum to get the help they need early on.1

The joint ERIK project therefore aims to develop a robotic platform that addresses new interaction strategies in treatments for children with impaired socioemotional functioning, such as those on the autism spectrum.


1 Cabibihan et al. “Why robots? A survey on the roles and benefits of social robots in the therapy of children with autism.” Int J Soc Robot 5.4 (2013): 593-618.


Emotion-sensitive robots ERIK
© Frank -
Interactive strategies for children with impaired socioemotional functioning

Fraunhofer IIS | Bereich Smart Sensing and Electronics (project coordinator)

  • Physiological and video-based emotion recognition

Humboldt Universität Berlin | Clinical Psychology of Social Interaction research group(German)

Friedrich-Alexander-Universität Erlangen-Nürnberg | Chair of Factory Automation and Production Systems (FAPS)

  • Technical development for the robotic system and components


  • Integration, evaluation and preparation of interaction data


  • Speech-based emotion recognition

Human-machine interaction based on real-time-capable emotion recognition

Our primary research areas are

  • Expanding and optimizing SHORE®, our real-time-capable analysis software, for reliable classification based on video data of younger test subjects
    • Basic emotions such as happiness, sadness, surprise, anger
    • Positive and negative valency
    • Action units
  • Reliable implementation and integration of emotion-sensitive sensor technology on the platform

The ERIK project complies with both the European General Data Protection Regulation (GDPR) and the ethical, legal and social implications (ELSI) relating to the target group (children) and incorporates these guidelines into its overall work process.




© Fraunhofer IIS/Bianca Möller
Entwicklung einer Roboterplattform zur Unterstützung neuer Interaktionsstrategien bei Kindern mit eingeschränkten sozio-emotionalen Fähigkeiten