Wide Angle Virtual View Synthesis using Two-by-Two Kinect V2

ProfessorHsueh-Ming Hang
ProjectConstruction of Intelligent Situation Systems by Advanced Computer Graphics, 3D Video, and Cloud Storage Technologies
ApplicationEntertainment and Virtual reality and augmented reality
Function3D localization, Background reconstruction, Image enhancement, Information of depth sensor, Stereo vision, and VR (Virtual reality)
Technical BenefitImprove system integration, Improve the quality of synthetic images, and Working on multi-camera at the same time
Technology StatusCan be transferred

We design a wide-angle view synthesis system based on captured multiple images and depth maps. We use a two-by–two array of Kinect v2 cameras to facilitate virtual camera tilt and zoom-in cases. We integrate two essential components developed by our lab colleagues to form a complete and practical system. Two additional elements are designed and implemented in this study. The first element is an improved multi-view blending algorithm that provides a clear improvement on the synthesized image quality even though the warping process induces some artifacts. By using four RGB-D cameras to capture the reference views, we have sufficient information to decide which pixel in the warped views is erroneous and we replace it by choosing the correct virtual color pixel from the other warped views to generate the synthesized pixel. The second element is solving the asynchronization problem among multiple Kinects in capturing images/videos. For simplicity, our first attempt is implementing a clock calibration system based on PC clock synchronization software. We examine the quality of synthesized views for various camera tilt and zoom-in/out cases. Experimental results show that the proposed multi-view blending algorithm achieves good synthesized image subjective quality on the real world images captured by four Kinect v2 sensors. To our knowledge, our system is the first one using four RGB-D cameras to synthesize wide-angle views.

Fig. 1. Setup of four Kinect v2 sensors in our experiments.

Fig. 2. Flowchart of proposed view synthesis system

Fig. 3. Geometric location of the virtual camera (with respect to four reference cameras) with a 5-degrees-tilt-down and 100mm zoom-in.

Fig. 4. [The virtual view has 5-degrees-tilt-down and 100mm zoom-in] Blending results of (Left column) adaptive blending algorithm [1] with upper two cameras; and (Right column) proposed blending algorithm with four cameras.

  • [1] -C. Lee, C.-L. Chien, and H.-M. Hang, “Virtual View Synthesis Quality Refinement,” in 3D-TV Conference: The True Vision – Capture, Transmission and Display of 3D Video (3DTV-CON), 2016.