SunStage: Portrait Reconstruction and Relighting using the Sun as a Light Stage (Talk)
A light stage acquires the shape and material properties of a face in high detail using a series of images captured under synchronized cameras and lights. This captured information can be used to synthesize novel images of the subject under arbitrary lighting conditions or from arbitrary viewpoints. This process enables a number of visual effects, such as creating digital replicas of actors that can be used in movies or high-quality postproduction relighting. In many cases, however, it is often infeasible to get access to a light stage for capturing a particular subject, because light stages are not easy to find: they are expensive and require significant technical expertise (often teams of people) to build and operate. In this talk, we will delve into a lightweight alternative to a light stage that captures comparable data using only a smartphone camera and the sun, which we dub SunStage. Our method only requires the user to capture a selfie video outdoors, rotating in place, and uses the varying angles between the sun and the face as guidance in joint reconstruction of facial geometry, reflectance, camera pose, and lighting parameters. Despite the in-the-wild un-calibrated setting, SunStage is able to reconstruct detailed facial appearance and geometry, enabling compelling effects such as relighting, novel view synthesis, and reflectance editing.
Biography: Yifan is a final-year PhD student at the University of Washington, working with Brian Curless and Steve Seitz. He has a broad interest in 3D vision and view synthesis. His work focuses on modeling special lighting effects, such as shadows, reflections, and refractions, to enable compelling effects like relighting and novel view synthesis. He received his Bachelor's degree from ShanghaiTech University, where he collaborated with Yi Ma and Manolis Tsakiris. Yifan has also spent time as a research intern at ByteDance, Google, and Adobe.
