Fraunhofer Institute for Integrated Circuits IIS
Fraunhofer Institute for Integrated Circuits IIS

Fraunhofer upHear Spatial Audio Microphone Processing

About#

Today’s consumers expect natural, even immersive sound from any recording – no matter if it is made with a professional or a consumer device. This means that even the built-in microphones of 360°-cameras and other mobile devices require high-end sound capture capabilities. That’s why Fraunhofer developed upHear Spatial Audio Microphone Processing with its spatial audio capturing algorithm. The software uses spatial sound processing to create high-resolution, immersive spatial sound that recreates the original experience.

This is achieved through three main features: Directional sound sources such as speech sources, distinct instruments, and prominent sound events are localized with high precision. At the same time, ambient sounds like the background ambience of a stadium or the reverberation of a concert hall are recorded. Plus, the high quality of the sound processing means that there are no audible artifacts. Together, these features make for high-end sound even from very compact devices and microphone setups.

Fraunhofer upHear Spatial Audio Microphone Processing is the first audio technology under the Fraunhofer upHear brand of immersive audio innovations. The algorithm preserves the authenticity of the audio scene when automatically transforming the captured sound in real time to any popular surround or immersive audio reproduction format, such as first-order ambisonic (FOA) and higher-order ambisonic (HOA), 5.1 channels, and 7.1+4 height channels.

Features#

Noise reduction

Automatic noise reduction removes background noises and ensures the optimal balance between speech quality and noise reduction performance

Beamforming

Signal-adaptive beamforming extracts the desired sounds with the best possible quality while attenuating undesired interfering sounds

Dereverberation

Advanced dereverberation algorithms reduce reverberant sounds to improve speech intelligibility when capturing distant speech sources indoors

Sound equalization (EQ)

Improved sound coloration and individual sound tuning by experienced Fraunhofer sound engineers

Wind noise reduction

Automatic wind noise reduction detects and reduces wind noise while preserving optimum speech quality

Automatic gain control (AGC) and dynamic range control (DRC)

AGC and DRC ensure optimal output loudness for different signal content and independent of the distance of the sound sources

Audio zoom

The audio zoom technology enables real-time focus on specific sources (when zooming in) or generates an immersive spatial sound impression (when zooming out)

Customized audio

Customized algorithms and real-time tuning options make it possible to adjust almost any aspect of sound processing depending on the need of the audience or user

upHear Product Family

upHear microphone processing technology is used in various products across the upHear product family. Find out more here:

upHear VQE

Benefits#

Fraunhofer’s Spatial Audio Microphone Processing technology makes it possible for everyone to enhance video content with spatial audio in a simple and robust way.

 

Professional content creators

Professional content creators benefit from Fraunhofer’s audio expertise: It makes it easier to capture high-quality, position-accurate 3D audio and a natural ambience that carries the atmosphere of the set. In post-production, the captured audio scenes can be combined with additional sound elements such as voice overs or dynamic objects.

 

Consumers

In the consumer area, there is a demand for devices that combine ease of use with high-quality recordings. The integration of Fraunhofer’s spatial audio capturing technology into consumer cameras delivers on this demand: immersive sound capture to match the video at the push of a button.

 

Manufacturers

Fraunhofer’s Spatial Audio Microphone Processing technology provides the flexibility to choose the input and output sources to consumer electronics manufacturers.

Be it a typical custom microphone arrangement (found in devices such as smartphones or action cameras) or multi-microphone geometry (such as linear microphone arrays, circular, and spherical microphone arrays as well as B-format microphones) the Fraunhofer algorithms can be adjusted to any specific microphone geometry.

The upHear technology is also flexible in terms of expected output: it fits both spatial sound output of various formats including channel-based audio (like 2.0, 5.1, 22.2) and Ambisonics (FOA, HOA). Plus, the audio processing can even be applied if the microphone geometry is unknown and if only one microphone is available (with some limitations, of course).

Applications#

© Monkey Business - stock.adobe.com

Social media content

  • Recording of audio blogs for social media platforms
  • Superior speech quality and speech intelligibility even when using relatively basic microphones and when recording from a distance
  • No external microphones needed since the ones built into the camera or smartphone can provide sufficient sound quality when using Fraunhofer processing

 

Mobile phone video recordings

  • Audio zoom for real-time focus on the desired sound source in front of the camera and attenuation of undesired interfering sounds from the side or back
  • Enables zooming out to capture immersive spatial sound
  • SAMP mono up-mix enables high-quality immersive music recordings with any smartphone

Concert recordings

  • High-quality, immersive spatial sound capturing with compact devices
  • Enables recreation of the original spatial impression as it was during the concert
  • Enables clear localization of the music as well as an immersive impression of the ambience (such as the cheering noise of the crowd, applause, and the reverberation of the concert hall)

 

Sport event recordings

  • Wind noise reduction enables capturing outside sport events
  • Immersive sound capturing ensures that the original impression (such as the stadium atmosphere) is recreated
© Евгений Вдовин - stock.adobe.com
© pikselstock - stock.adobe.com

VR and AR audio

  • Capture immersive spatial 3D sound for VR applications
  • Customized mic setups make it possible to hide the microphones on the video camera so that they are invisible in the field-of-view of the camera, while still being able to capture high-resolution spatial sound

 

Speech recordings

  • Recording of interviews with high speech intelligibility
  • Can be used as front-end processing for video-conferencing systems
  • upHear SAMP speech enhancement technologies are used in upHear VQE

More Information#

News

Interested in upHear?

For a consultation regarding your device’s microphone placement and spatial audio capability:

Product Brochures

Latest research

We keep our signal processing state-of-the art by always integrating our latest innovations in related signal processing fields. For the latest demos, check the AudioLabs:

Back to Consumer Electronics

back