Fraunhofer Institute for Integrated Circuits IIS
Fraunhofer Institute for Integrated Circuits IIS

Light-field Technology

What is light-field technology?

A light-field is defined by the light rays within a specific area. If all light rays are captured within a scene, it is possible to generate a perspective from any position. Although it is not currently possible to completely capture a light-field for a full scene, a large portion can still be acquired under special conditions. This is a major advantage when editing scenes during the post-production process.

Researchers in the Moving Picture Technologies department at Fraunhofer IIS are already demonstrating how scenes can be optimized using a setup with professional cameras in a planar arrangement that capture slightly different scene perspectives. This opens up new creative opportunities such as virtual camera panning and zooming, all with a single light-field recording.

 

© Fraunhofer IIS

Business Area Digital Media: Digital Events

Check out our digital formats at the Digital Media Business Area. We offer regular Weekly Specials revolving around light-field and other technological innovations for digital media workflows.

To the Business Area

Weekly Special: Light-Field - Important facts to know

Watch the Video on Vimeo

Realception® Plug-In Suite

 

© Fraunhofer IIS

Realception® is a light-field and multi camera data based Plug-in Suite for the professional movie post-production.

Realception® Plug-in Suite

Realception® is a light-field and multi-camera data-based Plug-in Suite for  professional movie post-production.

It is available as a Plug-in Suite for Foundry's Nuke as well as a Plug-In for Unreal Engine.

Find out more:

Realception® Plug-in Suite
 

© Fraunhofer IIS

Realception® is a light-field and multi camera data based Plug-in Suite for the professional movie post-production.

Realception® Plug-in Suite

New possibilities in movie production

If the focus was incorrectly set during shooting or the perspective has to be changed, time-consuming and expensive retakes would usually be necessary. However, the use of intelligent algorithms and high-quality cameras set up in a planar arrangement to provide multiple simultaneous views opens the door to new post-production possibilities.

The algorithms developed by Fraunhofer IIS offer a wide range of editing features: the camera perspective can, for example, be virtually shifted or expanded, as if the camera were actually moved, even though it was stationary during shooting. Another example would be the so-called dolly zoom effect, which works without moving the physical light-field camera.

The system can create 3D views from the existing depth information and plug-ins can be integrated into the post-production software for added creativity during the editing process, such as the Realception® Plug-ins for Nuke and Unreal Engine. The light-field recording system is also suitable for the production of stop-motion films where the camera remains stationary.

Current projects

Virtual LiVe

The Virtual LiVe project aims to digitize physical event formats through the use of new immersive media formats (e.g. 3D audio, 360° video, light fields, volumetric video). Virtual LiVe is funded as part of the Fraunhofer-Gesellschaft's SME acute program "Research for SMEs".
Fraunhofer FOKUS, Fraunhofer HHI and Fraunhofer IIS are working together to find applicable high-end solutions for the event streaming needs of various players from different industries.

XR-INTERACTION - German Research Alliance for Extended Reality

The innovation network XR-INTERACTION was initiated by the Professorship Ergonomics and Innovation Management at Chemnitz University of Technology. XR-INTERACTION is the first German innovation network doing research and development of new technologies and applications for Extended Reality. Incorporated by Virtual, Augmented and Mixed Reality applications, Extended Reality represents a new form of a digital world. We can immerse ourselves in this world and share the experience with others who are spatially separated from us. This technology connects people and brings them closer together.

As the world´s leading applied research organization, we have the expertise in key technologies for 3D digitation of assets and environments that form the basis for innovations in the field of Extended Reality. In collaboration with German research institutes and technology companies, the research project aims to develop and explore scalable Extended Reality systems for effective socio-cultural applications.

SocialSTAGE-VR

 

Public events are a vital part of cultural life as they provide significantly more sensory sensations than just watching and listening to what happens on the stage. An important part of the individual experience is the presence of others in the audience with whom the evoked emotions can be shared.

Whether for personal reasons, such as physical distance, or global reasons such as a pandemic, this is not always possible, so a contemporary technical solution that goes beyond traditional formats is needed. SocialSTAGE-VR focuses on the aspect of cultural participation as a virtual social experience.

Funded by the Federal Ministry of Education and Research, the project is coordinated by the Technische Universität Chemnitz with Fraunhofer IIS, vr-on GmbH, Die Etagen GmbH, point omega Aktiengesellschaft and YOUSE GmbH as its partners.

 

 

EU Project RealVision - hyperrealistic imaging experience

The RealVision network brings together leading universities, research centers focused on industrial development and companies to train a new generation of highly-skilled scientists and entrepreneurs in the area of hyper-realistic imaging - as well as coding and display technologies. The goal is to create of high-quality realistic imagery that faithfully represents the real scene.

The video below summarizes the team building and training school hosted by Fraunhofer IIS in the context of the RealVision project. The aim of this training school was to introduce the early stage researchers (ESRs) with topics related to light-field capturing and processing through a series of lectures from the experts of the field, complemented by practical hands-on sessions.

 

Notice: Starting the video transfers usage data to vimeo.

Concluded Projects

ETN FPI - European Training Network on Full Parallax Imaging

ETN-FPI was a four-year (2015-2019) Innovative Training Network, which received funding from the European Union’s Horizon 2020 research and innovation program under the Marie Sklodowska-Curie Grant Agreement. It brought together eight beneficiaries and partner organizations from Finland, Sweden, Germany, the United Kingdom, Spain, Hungary and Singapore to train a new generation of researchers in the area of full parallax imaging.  

 

© Fraunhofer IIS

On-line light-field video processing

Fraunhofer IIS and the Max-Planck Institute for Informatics introduce a real-time multi-view correspondence algorithm that extracts multi-view depth maps from sparse, wide-baseline light-field video to produce high-quality novel views for virtual effects such as apertures or virtual camera positions.



“Coming Home” − First live-action-clip with light-field processing

On set more and more cameras and camera arrays are used to generate additional views and data for visual effects or depth maps information. Together with the German Stuttgart Media University HDM, the Computational Imaging team from Fraunhofer´s Department of Moving Picture Technologies set up a project to evaluate and demonstrates the potential of shooting with multi-camera arrays and light-field processing.

 

Publications

"Light-field acquisition system allowing camera viewpoint and depth of field compositing in post-production"


Frederik Zilly, Michael Schöberl, Peter Schäfer, Matthias Ziegler, Joachim Keinert, Siegfried Foessel


IBC (International Broadcasting Convention), Amsterdam, Netherlands, September 13-17, 2013.

"Acquisition system for dense lightfield of large scenes"





Matthias Ziegler, Ron op het Veld, Joachim Keinert, Frederik Zilly



3DTV-Con, Copenhagen, Denmark, June 7-9, 2017.

"Immersive Virtual Reality for live-action video using camera arrays"



 

Matthias Ziegler, Joachim Keinert, Nina Holzer, Thorsten Wolf, Tobias Jaschke, Ron op het Veld, Faezeh Sadat Zakeri, Siegfried Foessel


IBC (International Broadcasting Convention), Amsterdam, Netherlands, September 15-19, 2017.

"A novel confidence measure for disparity maps by pixel-wise cost function analysis"

 

 

Ron Op het Veld, Tobias Jaschke, Michel   Bätz, Luca Palmieri (Department of Computer Science, Kiel University), Joachim Keinert

 

 

ICIP (International Conference on Image Processing), Athens, Greece, October 7-10, 2018.

"Benchmarking of several disparity estimation algorithms for light field processing"

 

 

Faezeh Sadat Zakeri, Michel Bätz, Tobias Jaschke, Joachim Keinert, Alexandra Chuchvara

 

 

Fourteenth International Conference on Quality Control by Artificial Vision, Mulhouse, France, May 15-17, 2019.

"Non-Planar Inside-Out Dense Light-Field Dataset and Reconstruction Pipeline"

 

 

Faezeh Sadat Zakeri, Ahmed Durmush (3D Media Group, Tampere University, Tampere, Finland), Matthias Ziegler, Michel Bätz, and Joachim Keinert

 

IEEE International Conference on Image Processing (ICIP), Taipei, Taiwan, September 22-25, 2019.

"Learning to Predict Image-based Rendering Artifacts with Respect to a Hidden Reference Image"

 

 

Mojtaba Bemana (MPI Informatik), Joachim Keinert, Karol Myszkowski, Michel Bätz, Matthias Ziegler, Hans-Peter Seidel (MPI Informatik), Tobias Ritschel (University College London)

 

 

 

"Near Real-Time Light Field Reconstruction and Rendering for on-Line Light Field Evaluation"

 

 

Mojtaba Bemana (MPI Informatik), Joachim Keinert, Karol Myszkowski, Michel Bätz, Matthias Ziegler, Hans-Peter Seidel (MPI Informatik), Tobias Ritschel (University College London)

 

 

 

"Light-field view synthesis using a convolutional block attention module"

 

 

 

Mojtaba Bemana (MPI Informatik), Joachim Keinert, Karol Myszkowski, Michel Bätz, Matthias Ziegler, Hans-Peter Seidel (MPI Informatik), Tobias Ritschel (University College London)

 

 

 

"A High-Resolution High Dynamic Range Light-Field Dataset with an Application to View Synthesis and Tone-Mapping"

More Information

News

Realception® Plug-In Suite

Find out more about our light-field and multi camera data based Plug-in Suite for the professional movie post-production:

Realception® Plug-in Suite

Download of Static Planar Light-field Data

Are you interested in Light-field?
Fraunhofer IIS now offers a Light-field data-set for free download.

Continue to download

Download of Static Non-planar Light-field Data

Are you interested in Light-field?
Fraunhofer IIS now offers a Light-field data-set for free download based on the paper "Non-planar inside-out dense Light-field dataset and reconstruction pipeline"

Continue to download

Back to Research in the Field of Moving Picture Technologies

back