Since humans are often the subject of photographs, detecting them and analyzing their pose is critical for image understanding. The photographic study of human pose and motion dates from the late 1800's with the work of Muybridge and Marey. Our research continues this tradition but with advanced graphics models of the body, new algorithms for pose and shape estimation, machine learning methods, and quantitative analysis of human motion and pose on ground-truth datasets.
In the last five years we have made significant progress on automatically estimating 2D human pose from images by leveraging training datasets and machine learning methods. We also leverage our expertise in optical flow estimation to extend 2D pose estimation over time, resulting in increased accuracy.
Beyond 2D pose, we are pushing the notion of "motion capture" in new directions. The goal is always to leverage what we know about bodies to get more from less - more accuracy and more shape detail from a small number of simple sensors. From 3D mocap markers, we recover detailed shape, pose, and soft tissue motion. Using a single RGB-D sensor and a parametric model of the human body, we are able to estimate human body shapes and poses from complex sequences of unconstrained motion.
Our work on human pose includes:
- 2D pose estimation from images.
- 2D pose from video and optical flow.
- 3D pose form multi-camera data.
- 3D pose from images and monocular video.
- 3D pose from RGB-D sequences.
- 3D pose from IUMs and other non-vision sensors.
- Marker-based and markerless motion capture.
- Learning pose priors.
- Tracking people interacting with objects.
- Datasets for quantitative evaluation.
- Human activity recognition.
Our current work is pushing the state of the art in monocular pose and motion capture to automatically go from 2D images or monocular video to 3D pose and shape of the human body. We are also expanding our research from tracking humans to tracking animals of many kinds.