A question of data protection – SHORE image analysis software

Mr. Garbas, you are an expert in the image analysis software SHORE. What does this software do?  

SHORE offers real-time facial detection and analysis, which means that it can find and analyze faces in images and videos very quickly. It can identify a person’s sex and age, as well as determine emotions by looking at their facial expressions.

The AVARD system builds on SHORE. What exactly does AVARD do?

AVARD is SHORE implemented in an intelligent sensor. This enables us to build very compact cameras that can perform analyses without a cloud connection. With AVARD, analysis takes place in the camera and data does not need to be transferred to a computer, cloud, or service provider. The data remains in the camera and therefore under the user’s control. Only anonymous meta-information is issued from the system.

What kind of metadata?

The metadata encompasses the number of people, their age and sex, their facial expression, and therefore their mood. The time a person spends in a particular place can also be determined.

Which data is not recorded?

The system cannot be used to identify people or recognize them again. It is also not possible to determine the presence of a specific person or that they have returned to a particular place.

How can this facial analysis be used?

There are many potential applications; in outdoor advertising, for example, adverts can be displayed that are relevant to specific customer groups such as youngsters, older people, men, or women. The AVARD system detects, for example, that three men or two women are standing in front of a display. This information can then be used to display adverts tailored to these customer groups.

There is currently a lot of talk about the use of such software in German retail. How is it being used here?

You can determine how many people come into a shop at what time and which groups of people are interested in which offers.

We have all seen personalized adverts on the internet that are tailored to the sites we have visited. Which data does SHORE analyze?

Yes, it’s part of online retail, where banners are also displayed that are customized for users. With AVARD, advertising can be adapted in stationary retail too.

SHORE analyzes a person’s sex, age, and facial expression. You can also analyze the amount of time they spend somewhere; this is particularly important for the advertising industry to determine how long someone has looked at an advert. If, for example, all men stop watching a commercial after three seconds, it is probably of no relevance or interest to them. Or if all older men watch a commercial through to the end, they probably find it interesting. This provides key data for measuring and optimizing the reach of adverts.

What happens to the data and what exactly is the data protection situation? Does it all conform to data protection guidelines, and has it been checked by the authorities?

It’s important to note that no personal data is recorded because the image data is discarded and anonymized as soon as it is analyzed. This prevents it from being traced back to specific people. The system therefore conforms to data protection requirements and has been certified with the “Privacy Seal” following a technical and legal assessment. The “Privacy Seal” is awarded by ePrivacy GmbH in Hamburg, which checks and certifies conformity with and the applicability of the German General Data Protection Regulation.

In addition, the Bavarian Data Protection Authority has checked how the system is used by German retailers and found that it conforms to data protection standards.

So data cannot be traced back to individuals?

Exactly, it’s impossible.

What exactly does “Privacy by Design” mean and how were these principles taken into account when developing SHORE?

»Privacy by Design« means that data protection is taken into account from the outset when creating and developing a technical system. The concept dates back to a white paper by a group of Canadian and Dutch researchers. In terms of AVARD, it means that SHORE is implemented so efficiently and so well that it can run on a small, intelligent camera without the need for a large computer in the cloud – so it can be implemented while maintaining data protection.

Mr. Garbas, where do you see applications like SHORE and AVARD being used in the future?

At the moment, it’s important for us to expand facial analysis, and in particular to strengthen the recognition of facial expressions. This can be used within robotics to create empathetic robots that can start to understand human feelings and behave accordingly. In the automotive sector, it is important for detecting how a motorist is feeling, whether they’re stressed or feeling well. It can also be used in medicine to help people with autism understand and interpret facial expressions.

SHORE can be of assistance wherever people interact with technical systems, be it a computer or a car; it is helpful if the machine understands the people and adapts itself to them, rather than the other way round. Technology can help people to improve their work or their driving and, ultimately, to improve their lives.

Our aim is to make interacting with machines a more pleasant experience. We endeavor to make empathetic machines that adapt to people and understand how to provide the best possible assistance.