3D sound that transports viewers to the heart of the action.

3D Sound for Everyone

Until now, recording high-quality 360° sound or enjoying immersive audio in a home theater was exclusive to users willing to install and operate a multitude of microphones or loudspeakers. Such complex setups will, however, soon be a thing of the past. Thanks to the special algorithms of the Fraunhofer upHear family of technologies capturing and playback of 3D sound is now possible just with a single device.

 

Enthusiastic viewers use their TV and soundbar to view footage of an automobile race that they recorded with their camera from the side of the race track. The roar of the motors follows the cars on the screen, thunderously panning from left to right across the room. Above them, they hear the stadium announcer commenting on the race, while all around them the spectators cheer the drivers on. It feels as if they are back on the race track. What used to be a science fiction scenario is now reality.

Cameras able to generate 360° images and videos have been around for some time, but their audio recording capabilities have left a great deal to be desired. Compelling 360° sound, unless captured using an elaborate arrangement of microphones, has hardly been an option for everyday users. Meanwhile, soundbars have also been around for some time, but they have only been able to play back basic surround sound. Truly immersive audio as experienced in a movie theater has only been available to Hi-Fi enthusiasts willing to make the effort to equip their living room with an assortment of ear-height and ceiling loudspeakers. For most people, the obstacles of recording true 3D sound or enjoying it at home are simply too big. But even professional producers of 3D (immersive) media content welcome solutions which simplify their work. This is where the upHear technologies developed in our Audio and Media Technologies division come into play, providing user-friendly solutions for a convincing 3D sound experience. All that is needed is a single end device – a microphone or a camera on the production side, and a soundbar on the reproduction side. This way, immersive sound can be enjoyed easily.

 

© Fraunhofer IIS/Ulli Scuda

Capturing 3D sound with a highly compact microphone setup.

3D sound capture made easy

Our engineers developed the Fraunhofer upHear Spatial Audio Microphone Processing technology to enable high-quality 3D sound capture using spatial audio microphones or the built-in microphones of professional and consumer-level 360° cameras and mobile devices at the push of a button. The algorithm can be implemented directly in devices, mobile applications, or post-production software, and can be fl exibly adapted to meet manufacturers’ requirements or the particular design of cameras and mobile devices. Integration in devices with three or more microphones in a variety of confi gurations, or even more demanding applications, is also possible.

The technology analyzes the exact position of audio signals, and automatically converts them into any popular surround or immersive playback format in real time, faithfully preserving the original sound atmosphere of the recorded scene. The information collected during this analysis allows the sound sources to be subsequently reproduced on a loudspeaker setup, thanks to the creation of audio channels. Typical formats include 5.1 surround sound or 7.1+4 for 3D sound. Additionally, an atmospheric soundscape can be created using ambisonics. For professional applications, when implemented in microphones, the technology also allows manual editing. This makes the data easier to work with and offers greater fl exibility during post-production: sound designers can combine the recorded audio scenes with additional sound elements such as voice-overs or dynamic objects. This enables them to create immersive sound experiences for virtual reality applications or enhance movies with 3D sound.

The Emmy award-winning producer of virtual reality content Felix & Paul Studios uses Fraunhofer upHear Spatial Audio Microphone Processing in their 360° 3D camera system and associated production tools to faithfully and precisely capture scene-based audio. The captured sound can subsequently be combined with other sources to create a fully immersive audio experience.

The algorithm automatically converts the captured sound in real time to any popular surround or immersive audio reproduction format, such as FOA, HOA, 5.1 channels, and 7.1+4 height channels.

»3D soundscapes can also unfold in your own living room.«

Smart soundbar designs for immersive playback

Thanks to premium streaming services, TV broadcasts in UHD format or series and fi lms on Blu-ray, it is theoretically possible to enjoy immersive theater-quality audio at home. However, conventional approaches to implementing three-dimensional sound in the living room involve multiple loudspeakers and cables and considerable technical expertise for their installation. In the future, none of this will be necessary: The Immersive Audio Virtualizer technology from the Fraunhofer upHear family will enable playback of high-quality 3D sound on soundbars or television sets.

Besides sophisticated signal analysis and perfectly matched hardware and software, the benefi ts of upHear for consumer electronics manufacturers also include personalized guidance on hardware specifi cations and individual tuning by experienced Fraunhofer Tonmeisters during product development. This allows advantageous positioning of the drivers within the device – resulting in smart, customized soundbar designs for immersive playback with no need for satellite loudspeakers. For the consumer at home, this saves time, effort, and hassle.

By employing intelligent spatial audio, a soundbar equipped with the upHear algorithm is able to render 3D sound in a way that would normally require seven loudspeakers, a subwoofer, and four height loudspeakers. The algorithm can process any audio format, and legacy formats such as surround or stereo can be given added spatiality by the integrated upmix functionality. The perfect partner for highly authentic 3D sound is the next-generation audio codec MPEG-H Audio. MPEG-H allows effi cient transmission of immersive sound for TV or streaming at bitrates typically required to transmit 5.1 surround sound today. Furthermore, the technology offers a range of features for a new, exciting TV audio experience: in addition to the immersive sound provided by 3D audio components, the sound mix can also be adjusted to meet individual requirements – allowing users to enjoy audio tailored to the device it is being played on. In collaboration with our partners, we will soon be delivering authentic 3D audio to living rooms everywhere with the help of the Fraunhofer upHear Immersive Audio Virtualizer.

upHear: Outstanding solutions for immersive soundscapes

Life is not two-dimensional, and 3D sound is an important gateway into new media environments, giving users the impression of being in the midst of the action. Accordingly, interdisciplinary teams in our Audio and Media Technologies division are working on the upHear family of technologies to provide just that. Our goal is to create technically and artistically outstanding solutions designed to bring authentic, immersive soundscapes to life.

Thanks to sophisticated signal analysis, perfectly matched hardware and software and individual tuning during product development, the Fraunhofer upHear Immersive Audio Virtualizer allows soundbars to deliver a convincing 3D sound experience, regardless of the input source used – with no need for special speaker setups.