PainFaceReader – Automatic pain detection

Autonomous long-term monitoring system that uses action units to automatically detect pain


Development of a monitoring system for automatic pain detection using action units.
A special focus in pain research is on reliable pain detection based on changing facial expressions as a form of non-verbal communication and how to clearly differentiate them from those produced by other emotional states. Here we use our SHORE® emotion analysis software and the Facial Action Coding System (FACS) to reliably detect action units.

In collaboration with the University of Bamberg, our goal is to create an autonomous system that can automatically detect pain in patients who are unable to communicate – and do so in a timely manner and when medical staff are not present.


Chair of Applied Computer Sciences / Cognitive Systems at the University of Bamberg

  • Using machine learning approaches to interpret preprocessed action units

Chair of Physiological Psychology at the University of Bamberg

  • Database for machine learning
  • Evaluation of results

Fraunhofer IIS | Facial Analysis Solutions

  • Emotion recognition in a variety of environmental, lighting and perspective contexts, based on reliable detection and identification of action units with the help of machine learning

The project is being funded by the Deutsche Forschungsgemeinschaft DFG (project number 405630557) since 2018.


The unambiguous classification of facial expressions using action units plays a major role in pain research. People with cognitive impairments that leave them unable to articulate their pain, such as those suffering from dementia or undergoing acute post-operative care, have only their facial expressions to communicate with others.

The challenge

Using such a system to help make diagnoses and decide when to prescribe pain medication faces two challenges: one, how to reliably identify a person’s state of mind; and two, how to explain classification decisions. Facial responses tied to annoyance, anger, disgust and pain exhibit similar action units or even the same action units appearing in a different order. The action units detected point directly to a particular state of mind.
The proposed system is to function autonomously so that medical staff are called only in an emergency, which means more targeted use of personnel while reducing costs.

Autonomous system for pain detection based on characteristic combinations and timing of action units

To achieve a high degree of sensitivity as well as specificity in the classification of pain, we are working on a way to reliably detect action units that takes into account combinations of those units while integrating time-variant models.

  • Co-occurrence of action units delivers important additional information
  • The characteristics related to the time sequence of action units provide another source of information

We are also working to further develop our SHORE® software library in terms of valid detection of concurrent action units and consideration of time sequences. To this end, we apply machine learning methods with a focus on computer vision (deep neural networks).

Further fields of application for the PainFaceReader

  • Intensive care
  • Palliative medicine
  • Facial expression analysis in depression