Reconstructing Signing Avatars from Video Using Linguistic Priors
2023
Conference Paper
hi
ps
Sign language (SL) is the primary method of communication for the 70 million Deaf people around the world. Video dictionaries of isolated signs are a core SL learning tool. Replacing these with 3D avatars can aid learning and enable AR/VR applications, improving access to technology and online media. However, little work has attempted to estimate expressive 3D avatars from SL video; occlusion, noise, and motion blur make this task difficult. We address this by introducing novel linguistic priors that are universally applicable to SL and provide constraints on 3D hand pose that help resolve ambiguities within isolated signs. Our method, SGNify, captures fine-grained hand pose, facial expression, and body movement fully automatically from in-the-wild monocular SL videos. We evaluate SGNify quantitatively by using a commercial motion-capture system to compute 3D avatars synchronized with monocular video. SGNify outperforms state-of-the-art 3D body-pose- and shape-estimation methods on SL videos. A perceptual study shows that SGNify's 3D reconstructions are significantly more comprehensible and natural than those of previous methods and are on par with the source videos. Code and data are available at sgnify.is.tue.mpg.de.
| Author(s): | Maria-Paola Forte and Peter Kulits and Chun-Hao Paul Huang and Vasileios Choutas and Dimitrios Tzionas and Katherine J. Kuchenbecker and Michael J. Black |
| Book Title: | IEEE/CVF Conf. on Computer Vision and Pattern Recognition (CVPR) |
| Pages: | 12791--12801 |
| Year: | 2023 |
| Month: | June |
| Department(s): | Haptic Intelligence, Perceiving Systems |
| Bibtex Type: | Conference Paper (inproceedings) |
| Paper Type: | Conference |
| DOI: | 10.1109/CVPR52729.2023.01230 |
| Event Name: | CVPR 2023 |
| Event Place: | Vancouver |
| State: | Published |
| Links: |
pdf
arXiv project code |
|
BibTex @inproceedings{Forte23-CVPR-SGNify,
title = {Reconstructing Signing Avatars from Video Using Linguistic Priors},
author = {Forte, Maria-Paola and Kulits, Peter and Huang, Chun-Hao Paul and Choutas, Vasileios and Tzionas, Dimitrios and Kuchenbecker, Katherine J. and Black, Michael J.},
booktitle = {IEEE/CVF Conf.~on Computer Vision and Pattern Recognition (CVPR)},
pages = {12791--12801},
month = jun,
year = {2023},
doi = {10.1109/CVPR52729.2023.01230},
month_numeric = {6}
}
|
|
