Ego-motion Aware Immersive Rendering from Real-World Recorded Panorama Videos

Capturing moments from our lives in the form of photos and videos has become increasingly easier since the invention of photography. Today, nearly everyone carries a high-resolution camera in the form of a smartphone. However, standard imagery is limited to the predefined viewing directions and positions during recording, which does not allow spectators to freely explore and immerse themselves into the whole scene. While omnidirectional panorama recordings capture all directions and allow exploration, they are still restricted in terms of positional movement. This is especially limiting in virtual reality (VR), where a fully immersive experience requires not just the ability to look around from a stationary viewpoint, but also to respond naturally to the head movement of the user by providing parallax motion within the virtual space.

In this thesis, I address the challenge of incorporating ego-motion-aware parallax effects into real-world panoramic videos, focusing on enhancing the immersive experience for VR applications. The research is divided into two main parts: the first focuses on a multi-panorama setup, while the second explores the use of a single stationary omnidirectional stereo (ODS) camera. In the first part, I develop necessary pre-processing steps to enable parallax effects for multi-panorama setups. Combined, the techniques allow for the direct application of image-warping techniques to simulate head-motion parallax. Although this only approximates the real parallax effect, it enhances the immersive quality of VR experiences. In the second part, I explore and utilize the advantages of ODS cameras which simultaneously capture images for the left and right eye, directly providing spectators with a sense of depth when viewed in VR. By focusing on a setup that requires only a single recording, my work allows for easier acquisition without the need to synchronize multiple cameras. The methods developed offer flexible solutions for creating novel views and parallax effects with minimal input data.

Overall, this thesis contributes to the field of immersive VR by providing innovative solutions for adding parallax effects to panoramic videos, improving depth perception and realism in VR content. The results of the experiments indicate the potential of the developed methods to enhance various VR applications — from entertainment and gaming to education and training — by offering a more engaging and lifelike experience.