List of Past Events
Temporally Consistent 3D Reconstruction from Video
Dr. Phillippos Mordohai
Monday, March 30, 2009, 12:00pm - 07:00pm
Department of Computer Science, Stevens Institute of Technology
Conventional video-based 3D reconstruction is performed either by assuming that the scene remains static or by reconstructing the scene at each time frame from multiple video inputs. When the scene is dynamic, the former approach is invalid and the latter leaves a lot to be desired. I will present ways to achieve temporal consistency in the context of video-based reconstruction of dynamic scenes in the form of two separate projects.
In the first project, we attempt to enforce temporal consistency onto the reconstruction of scenes with fast motion, large occlusions and self-occlusions. We formulate the problem as energy minimization in a Markov Random Field whose nodes are the pixels of the current, previous and next frame. The energy is optimized using a modified form of belief propagation, which includes mechanisms for noise reduction and background modeling. This is joint work with Scott Larsen, Marc Pollefeys and Henry Fuchs.
The second project aims at inserting simulated fluids into real video in order to generate composite videos in which one-way interactions between the fluids and dynamic objects in the scene appear realistic. We capture binocular video streams of the real scene, which we reconstruct using standard techniques from computer vision with an emphasis on obtaining accurate object boundaries. We, then, apply novel techniques for completing the shape and velocity of the dynamic objects in plausible ways. By plausible we refer to completions that interact naturally with the fluid and also hallucinate the invisible parts of the objects. These completions provide the boundary conditions for one-way coupling between the reconstructed geometry and the simulated fluid. This is joint work with Vivek Kwatra, Rahul Narain, Sashi Kumar Penta, Mark Carlson, Marc Pollefeys and Ming Lin.
Readings:
http://www.cs.stevens.edu/~mordohai/public/Larsen_TemporallyConsistentReconstruction07.pdf">http://www.cs.stevens.edu/~mordohai/public/Larsen_TemporallyConsistentReconstruction07.pdf
http://www.cs.stevens.edu/~mordohai/public/Kwatra_FluidInVideo08.pdf">http://www.cs.stevens.edu/~mordohai/public/Kwatra_FluidInVideo08.pdf
