Perceiving Systems, Computer Vision

Generating 3D People in Scenes without People

2020

Conference Paper

ps


We present a fully automatic system that takes a 3D scene and generates plausible 3D human bodies that are posed naturally in that 3D scene. Given a 3D scene without people, humans can easily imagine how people could interact with the scene and the objects in it. However, this is a challenging task for a computer as solving it requires that (1) the generated human bodies to be semantically plausible within the 3D environment (e.g. people sitting on the sofa or cooking near the stove), and (2) the generated human-scene interaction to be physically feasible such that the human body and scene do not interpenetrate while, at the same time, body-scene contact supports physical interactions. To that end, we make use of the surface-based 3D human model SMPL-X. We first train a conditional variational autoencoder to predict semantically plausible 3D human poses conditioned on latent scene representations, then we further refine the generated 3D bodies using scene constraints to enforce feasible physical interaction. We show that our approach is able to synthesize realistic and expressive 3D human bodies that naturally interact with 3D environment. We perform extensive experiments demonstrating that our generative framework compares favorably with existing methods, both qualitatively and quantitatively. We believe that our scene-conditioned 3D human generation pipeline will be useful for numerous applications; e.g. to generate training data for human pose estimation, in video games and in VR/AR. Our project page for data and code can be seen at: \url{https://vlg.inf.ethz.ch/projects/PSI/}.

Author(s): Yan Zhang and Mohamed Hassan and Heiko Neumann and Michael J. Black and Siyu Tang
Book Title: 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR 2020)
Pages: 6193--6203
Year: 2020
Month: June
Publisher: IEEE

Department(s): Perceiving Systems
Research Project(s): Putting People into Scenes
Bibtex Type: Conference Paper (inproceedings)
Paper Type: Conference

DOI: 10.1109/CVPR42600.2020.00623
Event Name: IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR 2020)
Event Place: Seattle, WA, USA

Address: Piscataway, NJ
ISBN: 978-1-7281-7168-5
State: Published

Links: Code
Video
Video:
Attachments: PDF

BibTex

@inproceedings{PSI:2019,
  title = {Generating {3D} People in Scenes without People},
  author = {Zhang, Yan and Hassan, Mohamed and Neumann, Heiko and Black, Michael J. and Tang, Siyu},
  booktitle = {2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR 2020)},
  pages = {6193--6203},
  publisher = {IEEE},
  address = {Piscataway, NJ},
  month = jun,
  year = {2020},
  doi = {10.1109/CVPR42600.2020.00623},
  month_numeric = {6}
}