Immersed in Sound

Fraunhofer IIS has been successfully developing audio coding technologies for many years. The goal of these technologies is to store and transmit audio signals in optimum quality. But what good is having the best audio quality if devices can only reproduce it inadequately? To address this prob- lem, scientists at Fraunhofer IIS have developed the Cingo® and Symphoria® technologies to improve playback sound quality.

3D sound is already in movie theaters near you. But incredibly, listening to 3D sound on cell- phones through headphones will soon be an everyday reality too. Although it might sound like a distant vision or even a pipe dream, it is actually already common practice for users of virtual reality headsets equipped with Fraunhofer technology. And Fraunhofer 3D sound is also providing more realistic music experiences in cars.

To bring big sound to small devices such as smartphones, tablets, and VR headsets and into cars, developers at Fraunhofer IIS have been working in the area of audio coding and audio signal processing for many years. In audio coding, the focus is on reducing the volume of data to be transmitted. Fraunhofer IIS played a major role in developing the best-known examples of this technology: mp3 and AAC. In audio signal processing, the focus is on improving the sound on the respective device.

Whether on their smartphones or in home cinemas, users expect best-quality sound

The days when movies, music, and games were consumed only on TV sets or PCs have long gone. Nowadays people consume entertainment on a whole range of devices, such as smartphones and tablet computers, and in their cars. This calls for a seamless crossover between platforms. To give an example: a user might buy a movie on their PC and watch the first half on TV and then watch the rest on the way to work the next morning. In spite of the different video and audio playback systems, buyers expect excellent picture and sound quality on all devices – even though the technical preconditions to make this possible are not always fulfilled. For instance, smartphones have small screens and often poor loudspeakers and cheap headphones. And although cars often feature high-end sound systems, the acoustic conditions inside vehicles are difficult on account of the loudspeaker positions, driving noises, and narrow passenger compartments.

Intelligent algorithms improve sound reproduction

Researchers at Fraunhofer IIS are developing intelligent algorithms to process audio signals, analyze music, and break it up into its constituent parts in order to optimize its reproduction for the specific playback system. For example, the music in a concert hall comes directly from the stage, but also bounces back from the ceiling and the walls. Through the combination of direct sound waves from the stage and reverberant sound, the audience at a concert enjoys a natural and immersive sound experience. If you want to reproduce this experience as closely as possible in a car or over headphones, first the direct and reflected sound waves in the music signal must be identified and separated from each other. Once that is done, the individual elements can be reassembled for playback to form a 3D soundscape that recreates the live concert experience as accurately and faithfully as possible. In this way the listener hears the music as it was originally intended, irrespective of the quality of the actual acoustic environment or whether loudspeakers or headphones are being used. Supporting algorithms balance out weaknesses in the hardware, so that users can enjoy music and other audio content in precise, natural sound quality even over inexpensive loudspeakers or headphones.

Engineers and sound designers work hand in hand

It would not be possible to develop these intelligent algorithms if engineers and scientists did not work closely together with sound technicians and designers. Only by marryig the worlds of sound and technology – and combining their specific know­how – can the optimum sound be achieved for the various playback scenarios. While engineers develop the algorithms for analyzing and processing the audio signals, the specially trained Tonmeisters and sound designers evaluate the sound quality and use the tools provided by the engineers to optimize the sound for reproduction on different devices. This interplay of technology and sound design creates a significantly improved acoustic experience for users of the optimized devices.

Fraunhofer Cingo® brings cinema sound to mobile devices

We are familiar with surround or 3D sound from the movie theater or our home cinema systems. In both cases, a large number of loudspeakers ensure perfect sound. But today most people watch movies on the move on smartphones or tablet computers, which do not have sophisticated sound systems. Yet users still expect an experience that is close to cinema quality. This is where Fraunhofer Cingo comes in: Cingo allows the reproduction of authentic stereo, surround, and 3D sound over the headphones and stereo loudspeakers of smartphones and tablet computers.

When smartphone users, for example, buy movies from a streaming service the soundtrack is typically played back over headphones. But thanks to Cingo this does not necessarily mean having to forego surround or 3D sound experience. Cingo carries out the multi­channel processing directly on the device. Special headphones are not required: the technology is designed to yield optimum results with any type of headphones. As a result, sound effects such as a helicopter passing overhead or rain beating down from above are perfectly repro- duced. Moreover, Cingo provides dynamic volume control to improve intelligibility, boosting the loudness of quiet passages of speech and smoothing the contrast between alternating scenes with loud and soft noise levels. This feature also allows spoken dialog to be amplified, which means that the words spoken by movie characters stand out clearly from any sources of background noise – for instance if the viewer is watching the movie on their smartphone in a noisy train station.

© Photo Fraunhofer IIS/David Hartfiel

Increasingly, multimedia con- tent is being played or streamed on mobile devices.

© Photo Fraunhofer IIS/David Hartfiel

Authentic 3D sound also en- sures that tablet viewers feel im- mersed in the film’s soundscape.

© Photo Dirk Mahler/Fraunhofer

Harald Popp, Oliver Hellmuth, and Jan Plogsties (from left) were awarded the 2015 Joseph von Fraunhofer Prize for the develop- ment and marketing of Cingo® and Symphoria®.

Cingo® makes the virtual world audible

Reproducing realistic 3D sound is especially important for virtual reality headsets. This new class of consumer electronics devices allows users to immerse themselves completely in a virtual world. To enter this world, they put on a headset with a built-in video screen or smartphone display and special optical components that give wearers the impression of being in a different environment. However, this first step toward the holodeck featured in the Star Trek universe is only convincing when the 360-degree video images are matched with sound that also seems to come from all directions. After all, if you are looking around in a tropical rainforest, you should be able to hear the birds in the treetops above and not just at head level. This exact localization of sound sources in space via conventional stereo head- phones is precisely what Cingo facilitates, making it a perfect technology for all virtual reality applications. To portray 3D sound as convincingly as possible in the virtual world, Cingo also takes the movements of the head into account. If the VR user turns around, Cingo ensures that the sound image remains fixed in the same place – just as it does in reality – instead of shifting with the head movement as happens with the conventional reproduction of audio signals through headphones.

»CINGO® COMPLEMENTS VR APPLICATIONS PERFECTLY BECAUSE ONLY NATURAL 360° SOUND CAN COMPLETE THE VIRTUAL EXPERIENCE.«

Cingo is already used in numerous devices worldwide. Since 2013, for instance, all devices in Google’s Nexus range feature Cingo. This allows users to play movies that they have bought in the Google Play Store in surround sound on devices. In addition, numerous video-on-demand providers use Cingo in their various apps in order to offer their customers an impressive sound experience even on mobile devices. And last but not least, Cingo is integrated into the Samsung Gear VR, the LG 360 VR and the Hulu VR app.

Fraunhofer Symphoria® creates a harmonious soundscape in vehicles

Whereas the audio effect is often the priority when reproducing surround or 3D sound on smartphones or virtual reality headsets, ensuring that the music reproduction is as natural as possible is essential inside of vehicles. This is why Fraunhofer IIS developed a 3D surround sound technology called Symphoria, which conveys a new sense of spaciousness inside cars. Symphoria expands the soundscape by giving it greater width, depth, and also height. The physical limitations of the passenger cabin seem to disappear. A broad, clearly defined sound stage and a flawlessly balanced surround sound field can be created for each seat in the vehicle. As a result, each passenger enjoys outstanding audio quality, irrespective of whether the sound comes from the radio, a CD, or an mp3 player or whether the audio material is in stereo, 5.1 surround, or 3D audio format. Symphoria creates a harmonious sound pattern for each incoming signal.

Audi is the first vehicle manufacturer to use the Symphoria system. First, the new Audi TT, which features a Bang & Olufsen sound system and Symphoria, was unveiled in the summer of 2014. This was followed by the Audi Q7, R8, and A4 in 2015. In these vehicles, end customers can order the Bang & Olufsen sound system as an optional extra. When they do, they receive a package of around twenty loudspeakers and Fraunhofer Symphoria sound for a unique in­vehicle sound experience unlike anything that has come before.

Award-winning sound: Joseph von Fraunhofer Prize for Cingo® and Symphoria®

Through its technologies for audio signal processing, the institute is systematically continuing its successful work in the field of audio research. Whereas previously the main focus was on the efficient, high­quality storage and transmission of audio data, Cingo and Symphoria optimize sound for playback on the specific device. As a consequence, Fraunhofer IIS now covers the entire chain from production to reproduction. Using the Sonnox plug­ins co-developed by Fraunhofer IIS, music producers can optimize their music for online distribu- tion. Streaming services and broadcasting companies worldwide use MPEG audio codecs, in the development of which Fraunhofer IIS played a major role, for the efficient transmission of data. And finally, Cingo and Symphoria ensure that good sound is available beyond the confines of living­room hi­fi systems. For this last step, the 2015 Joseph von Fraunhofer Prize was awarded to Oliver Hellmuth, Jan Plogsties, and Harald Popp.

Please note: Starting the videos transfers usage data to youtube. Video in German language.