Neural Dynamic Image-Based Rendering

1Google Research   2Cornell Tech, Cornell University  

CVPR 2023 (Best Paper Honorable Mention)

State-of-the-art monocular dynamic view synthesis methods struggle to render high-quality views from long videos featuring uncontrolled camera paths and complex scene motion. We present a new approach that addresses these limitations, illustrated above via an application to 3D stabilization, where we apply our approach and prior methods to a shaky, 30-second video, and compare novel views rendered along a smoothed camera path.


We address the problem of synthesizing novel views from a monocular video depicting a complex dynamic scene. State-of-the-art methods based on temporally varying Neural Radiance Fields (aka dynamic NeRFs) have shown impressive results on this task. However, for long videos with complex object motions and uncontrolled camera trajectories, these methods can produce blurry or inaccurate renderings, hampering their use in real-world applications. Instead of encoding the entire dynamic scene within the weights of MLPs, we present a new approach that addresses these limitations by adopting a volumetric image-based rendering framework that synthesizes new viewpoints by aggregating features from nearby views in a scene-motion-aware manner. Our system retains the advantages of prior methods in its ability to model complex scenes and view-dependent effects, but also enables synthesizing photo-realistic novel views from long videos featuring complex scene dynamics with challenging camera and object motion, where prior methods fail to produce high-quality renderings.



Thanks to Andrew Liu, Richard Bowen and Lucy Chai for the fruitful discussions, and thanks to Richard Szeliski and Ricardo Martin-Brualla for their helpful proofreading.


  title         = {DynIBaR: Neural Dynamic Image-Based Rendering},
  author        = {Li, Zhengqi and Wang, Qianqian and Cole, Forrester and Tucker, Richard and Snavely, Noah},
  booktitle     = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
  year          = {2023},