Hands are important for humans for signaling and communication, as well as for interacting with the physical world. Capturing the motion of hands is a very challenging computer vision problem that is also highly relevant for other areas like computer graphics, human-computer interfaces, or robotics.
We focus on building an accurate and realistic model of the human hand [ ], capturing the pose and shape variation across a human population. For this we collect many examples of human hands with our 3D scanner, following a systematic grasp taxonomy [ ]. We then combine the hand model with our body model [ ] to build a joint, seamless model of the body together with hands. This allows us to naturally capture the motion of people with expressive body and hands motion using our 4D scanner.
A strong hand model can be used to regularise fitting to noisy input data to reconstruct hands and/or objects [ ]. We focus on hands that interact with other hands or known objects [ ], using either a single RGB-D camera or multiple synchronized RGB cameras. Interaction cues can also reveal information that helps to reconstruct unknown properties of the object, like the shape [ ] or kinematic skeleton [ ].
Our current work focuses on moving towards a more holistic model, by adding more expressiveness to the current body and hands model. Furthermore, we use deep learning to reconstruct hand-object configurations jointly from a single RGB image. Existing work reconstructs either the hand pose alone in the form of 2D or 3D joint positions, or the object pose or shape. We want to jointly reconstruct hand-object configurations by estimating 3D meshes for both, so that we can reason about interactions, e.g. proximity, contact, grasp stability, forces. We then plan to move towards more realistic in-the-wild interactions.
SIGGRAPH-Asia 2017 (TOG) models/dataset (link)
Models & Alignments & Scans, for:
- hand-only (MANO)
- body+hand (SMPL+H)
ECCVw 2016 dataset (link)
RGB-D dataset of an object under manipulation.
The dataset also contains input 3D template meshes
for each object and output articulated models.
IJCV 2016 dataset (link)
Annotated RGB-D + multicamera-RGB dataset of one or two hands
interacting with each other and/or with a rigid or an articulated object
ICCV 2015 dataset (link)
RGB-D dataset of a hand rotating a rigid object for 3d scanning
GCPR 2014 dataset (link)
Annotated RGB-D dataset of one or two hands interacting with each other
GCPR 2013 dataset (link)
Synthetic dataset of two hands interacting with each other
Romero, J., Tzionas, D., Black, M. J.
ACM Transactions on Graphics, (Proc. SIGGRAPH Asia), 36(6):245:1-245:17, 245:1–245:17, ACM, November 2017 (article)
Feix, T., Romero, J., Schmiedmayer, H., Dollar, A., Kragic, D.
Human-Machine Systems, IEEE Transactions on, 46(1):66-77, 2016 (article)
Tzionas, D., Ballan, L., Srikantha, A., Aponte, P., Pollefeys, M., Gall, J.
International Journal of Computer Vision (IJCV), 118(2):172-193, June 2016 (article)
Tzionas, D., Srikantha, A., Aponte, P., Gall, J.
In German Conference on Pattern Recognition (GCPR), pages: 1-13, Lecture Notes in Computer Science, Springer, September 2014 (inproceedings)
Ballan, L., Taneja, A., Gall, J., van Gool, L., Pollefeys, M.
In European Conference on Computer Vision (ECCV), 7577, pages: 640-653, LNCS, Springer, 2012 (inproceedings)
Romero, J., Feix, T., Kjellström, H., Kragic, D.
In IEEE/RSJ International Conference on Intelligent Robots and Systems, IROS, pages: 2103-2108, 2010 (inproceedings)
Romero, J., Kjellström, H., Kragic, D.
In IEEE International Conference on Robotics and Automation (ICRA), pages: 458-463, 2010 (inproceedings)
For website questions and technical problems please contact:
© 2018 Max-Planck-Gesellschaft - Imprint | Privacy Policy
