30 results
(View BibTeX file of all listed publications)
2022
Reconstructing Expressive 3D Humans from RGB Images
ETH Zurich, Max Planck Institute for Intelligent Systems and ETH Zurich, December 2022 (thesis)
To interact with our environment, we need to adapt our body posture
and grasp objects with our hands. During a conversation our facial expressions
and hand gestures convey important non-verbal cues about our
emotional state and intentions towards our fellow speakers. Thus, modeling
and capturing 3D full-body shape and pose, hand articulation and facial
expressions are necessary to create realistic human avatars for augmented
and virtual reality. This is a complex task, due to the large number of
degrees of freedom for articulation, body shape variance, occlusions from
objects and self-occlusions from body parts, e.g. crossing our hands, and
subject appearance. The community has thus far relied on expensive and
cumbersome equipment, such as multi-view cameras or motion capture
markers, to capture the 3D human body. While this approach is effective,
it is limited to a small number of subjects and indoor scenarios. Using
monocular RGB cameras would greatly simplify the avatar creation process,
thanks to their lower cost and ease of use. These advantages come at a price
though, since RGB capture methods need to deal with occlusions, perspective
ambiguity and large variations in subject appearance, in addition to
all the challenges posed by full-body capture. In an attempt to simplify the
problem, researchers generally adopt a divide-and-conquer strategy, estimating
the body, face and hands with distinct methods using part-specific
datasets and benchmarks. However, the hands and face constrain the body
and vice-versa, e.g. the position of the wrist depends on the elbow, shoulder,
etc.; the divide-and-conquer approach can not utilize this constraint.
In this thesis, we aim to reconstruct the full 3D human body, using only
readily accessible monocular RGB images. In a first step, we introduce a
parametric 3D body model, called SMPL-X, that can represent full-body
shape and pose, hand articulation and facial expression. Next, we present
an iterative optimization method, named SMPLify-X, that fits SMPL-X to
2D image keypoints. While SMPLify-X can produce plausible results if
the 2D observations are sufficiently reliable, it is slow and susceptible
to initialization. To overcome these limitations, we introduce ExPose, a
neural network regressor, that predicts SMPL-X parameters from an image
using body-driven attention, i.e. by zooming in on the hands and face,
after predicting the body. From the zoomed-in part images, dedicated
part networks predict the hand and face parameters. ExPose combines
the independent body, hand, and face estimates by trusting them equally.
This approach though does not fully exploit the correlation between parts
and fails in the presence of challenges such as occlusion or motion blur.
Thus, we need a better mechanism to aggregate information from the full
body and part images. PIXIE uses neural networks called moderators that
learn to fuse information from these two image sets before predicting the
final part parameters. Overall, the addition of the hands and face leads to
noticeably more natural and expressive reconstructions.
Creating high fidelity avatars from RGB images requires accurate estimation
of 3D body shape. Although existing methods are effective at
predicting body pose, they struggle with body shape. We identify the lack
of proper training data as the cause. To overcome this obstacle, we propose
to collect internet images from fashion models websites, together with
anthropometric measurements. At the same time, we ask human annotators
to rate images and meshes according to a pre-defined set of linguistic attributes.
We then define mappings between measurements, linguistic shape
attributes and 3D body shape. Equipped with these mappings, we train a
neural network regressor, SHAPY, that predicts accurate 3D body shapes
from a single RGB image. We observe that existing 3D shape benchmarks
lack subject variety and/or ground-truth shape. Thus, we introduce a new
benchmark, Human Bodies in the Wild (HBW), which contains images of
humans and their corresponding 3D ground-truth body shape. SHAPY
shows how we can overcome the lack of in-the-wild images with 3D shape
annotations through easy-to-obtain anthropometric measurements and linguistic
shape attributes.
Regressors that estimate 3D model parameters are robust and accurate,
but often fail to tightly fit the observations. Optimization-based approaches
tightly fit the data, by minimizing an energy function composed of a data
term that penalizes deviations from the observations and priors that encode
our knowledge of the problem. Finding the balance between these terms
and implementing a performant version of the solver is a time-consuming
and non-trivial task. Machine-learned continuous optimizers combine the
benefits of both regression and optimization approaches. They learn the
priors directly from data, avoiding the need for hand-crafted heuristics and
loss term balancing, and benefit from optimized neural network frameworks
for fast inference. Inspired from the classic Levenberg-Marquardt
algorithm, we propose a neural optimizer that outperforms classic optimization,
regression and hybrid optimization-regression approaches. Our
proposed update rule uses a weighted combination of gradient descent
and a network-predicted update. To show the versatility of the proposed
method, we apply it on three other problems, namely full body estimation
from (i) 2D keypoints, (ii) head and hand location from a head-mounted
device and (iii) face tracking from dense 2D landmarks. Our method can
easily be applied to new model fitting problems and offers a competitive
alternative to well-tuned traditional model fitting pipelines, both in terms
of accuracy and speed.
To summarize, we propose a new and richer representation of the human
body, SMPL-X, that is able to jointly model the 3D human body pose
and shape, facial expressions and hand articulation. We propose methods,
SMPLify-X, ExPose and PIXIE that estimate SMPL-X parameters from
monocular RGB images, progressively improving the accuracy and realism
of the predictions. To further improve reconstruction fidelity, we demonstrate
how we can use easy-to-collect internet data and human annotations
to overcome the lack of 3D shape data and train a model, SHAPY, that
predicts accurate 3D body shape from a single RGB image. Finally, we
propose a flexible learnable update rule for parametric human model fitting
that outperforms both classic optimization and neural network approaches.
This approach is easily applicable to a variety of problems, unlocking new
applications in AR/VR scenarios.
2021
Skinned multi-infant linear body model
Hesse, N., Pujades, S., Romero, J., Black, M.
(US Patent 11,127,163, 2021), September 2021 (patent)
A computer-implemented method for automatically obtaining pose and shape parameters of a human body. The method includes obtaining a sequence of digital 3D images of the body, recorded by at least one depth camera; automatically obtaining pose and shape parameters of the body, based on images of the sequence and a statistical body model; and outputting the pose and shape parameters. The body may be an infant body.
2020
Machine learning systems and methods of estimating body shape from images
Black, M., Rachlin, E., Heron, N., Loper, M., Weiss, A., Hu, K., Hinkle, T., Kristiansen, M.
(US Patent 10,679,046), June 2020 (patent)
Disclosed is a method including receiving an input image including a human, predicting, based on a convolutional neural network that is trained using examples consisting of pairs of sensor data, a corresponding body shape of the human and utilizing the corresponding body shape predicted from the convolutional neural network as input to another convolutional neural network to predict additional body shape metrics.
Machine learning systems and methods for augmenting images
Black, M., Rachlin, E., Lee, E., Heron, N., Loper, M., Weiss, A., Smith, D.
(US Patent 10,529,137 B1), January 2020 (patent)
Disclosed is a method including receiving visual input comprising a human within a scene, detecting a pose associated with the human using a trained machine learning model that detects human poses to yield a first output, estimating a shape (and optionally a motion) associated with the human using a trained machine learning model associated that detects shape (and optionally motion) to yield a second output, recognizing the scene associated with the visual input using a trained convolutional neural network which determines information about the human and other objects in the scene to yield a third output, and augmenting reality within the scene by leveraging one or more of the first output, the second output, and the third output to place 2D and/or 3D graphics in the scene.
2019
Method for providing a three dimensional body model
Loper, M., Mahmood, N., Black, M.
September 2019, U.S.~Patent 10,417,818 (patent)
A method for providing a three-dimensional body model which may be applied for an animation, based on a moving body, wherein the method comprises providing a parametric three-dimensional body model, which allows shape and pose variations; applying a standard set of body markers; optimizing the set of body markers by generating an additional set of body markers and applying the same for providing 3D coordinate marker signals for capturing shape and pose of the body and dynamics of soft tissue; and automatically providing an animation by processing the 3D coordinate marker signals in order to provide a personalized three-dimensional body model, based on estimated shape and an estimated pose of the body by means of predicted marker locations.
2018
Method and Apparatus for Estimating Body Shape
Black, M. J., Balan, A., Weiss, A., Sigal, L., Loper, M., St Clair, T.
June 2018, U.S.~Patent 10,002,460 (patent)
A system and method of estimating the body shape of an individual from input data such as images or range maps. The body may appear in one or more poses captured at different times and a consistent body shape is computed for all poses. The body may appear in minimal tight-fitting clothing or in normal clothing wherein the described method produces an estimate of the body shape under the clothing. Clothed or bare regions of the body are detected via image classification and the fitting method is adapted to treat each region differently. Body shapes are represented parametrically and are matched to other bodies based on shape similarity and other features. Standard measurements are extracted using parametric or non-parametric functions of body shape. The system components support many applications in body scanning, advertising, social networking, collaborative filtering and Internet clothing shopping.
Co-Registration – Simultaneous Alignment and Modeling of Articulated 3D Shapes
Black, M., Hirshberg, D., Loper, M., Rachlin, E., Weiss, A.
February 2018, U.S.~Patent 9,898,848 (patent)
Present application refers to a method, a model generation unit and a computer program (product) for generating trained models (M) of moving persons, based on physically measured person scan data (S). The approach is based on a common template (T) for the respective person and on the measured person scan data (S) in different shapes and different poses. Scan data are measured with a 3D laser scanner. A generic personal model is used for co-registering a set of person scan data (S) aligning the template (T) to the set of person scans (S) while simultaneously training the generic personal model to become a trained person model (M) by constraining the generic person model to be scan-specific, person-specific and pose-specific and providing the trained model (M), based on the co registering of the measured object scan data (S).
2017
Parameterized Model of 2D Articulated Human Shape
Black, M. J., Freifeld, O., Weiss, A., Loper, M., Guan, P.
September 2017, U.S.~Patent 9,761,060 (patent)
Disclosed are computer-readable devices, systems and methods for generating a model of a clothed body. The method includes generating a model of an unclothed human body, the model capturing a shape or a pose of the unclothed human body, determining two-dimensional contours associated with the model, and computing deformations by aligning a contour of a clothed human body with a contour of the unclothed human body. Based on the two-dimensional contours and the deformations, the method includes generating a first two-dimensional model of the unclothed human body, the first two-dimensional model factoring the deformations of the unclothed human body into one or more of a shape variation component, a viewpoint change, and a pose variation and learning an eigen-clothing model using principal component analysis applied to the deformations, wherein the eigen-clothing model classifies different types of clothing, to yield a second two-dimensional model of a clothed human body.
Crowdshaping Realistic 3D Avatars with Words
Streuber, S., Ramirez, M. Q., Black, M., Zuffi, S., O’Toole, A., Hill, M. Q., Hahn, C. A.
August 2017, Application PCT/EP2017/051954 (patent)
System and method for simulating realistic clothing
June 2017, U.S.~Patent 9,679,409 B2 (patent)
Systems, methods, and computer-readable storage media for simulating realistic clothing. The system generates a clothing deformation model for a clothing type, wherein the clothing deformation model factors a change of clothing shape due to rigid limb rotation, pose-independent body shape, and pose-dependent deformations. Next, the system generates a custom-shaped garment for a given body by mapping, via the clothing deformation model, body shape parameters to clothing shape parameters. The system then automatically dresses the given body with the custom- shaped garment.
2016
Skinned multi-person linear model
Black, M.J., Loper, M., Mahmood, N., Pons-Moll, G., Romero, J.
December 2016, Application PCT/EP2016/064610 (patent)
The invention comprises a learned model of human body shape and pose dependent shape variation that is more accurate than previous models and is compatible with existing graphics pipelines. Our Skinned Multi-Person Linear model (SMPL) is a skinned vertex based model that accurately represents a wide variety of body shapes in natural human poses. The parameters of the model are learned from data including the rest pose template, blend weights, pose-dependent blend shapes, identity- dependent blend shapes, and a regressor from vertices to joint locations. Unlike previous models, the pose-dependent blend shapes are a linear function of the elements of the pose rotation matrices. This simple formulation enables training the entire model from a relatively large number of aligned 3D meshes of different people in different poses. The invention quantitatively evaluates variants of SMPL using linear or dual- quaternion blend skinning and show that both are more accurate than a Blend SCAPE model trained on the same data. In a further embodiment, the invention realistically models dynamic soft-tissue deformations. Because it is based on blend skinning, SMPL is compatible with existing rendering engines and we make it available for research purposes.
2015
Proceedings of the 37th German Conference on Pattern Recognition
Gall, J., Gehler, P., Leibe, B.
Springer, German Conference on Pattern Recognition, October 2015 (proceedings)
2014
Advanced Structured Prediction
Nowozin, S., Gehler, P. V., Jancsary, J., Lampert, C. H.
Advanced Structured Prediction, pages: 432, Neural Information Processing Series, MIT Press, November 2014 (book)
The goal of structured prediction is to build machine learning models that predict relational information that itself has structure, such as being composed of multiple interrelated parts. These models, which reflect prior knowledge, task-specific relations, and constraints, are used in fields including computer vision, speech recognition, natural language processing, and computational biology. They can carry out such tasks as predicting a natural language sentence, or segmenting an image into meaningful components.
These models are expressive and powerful, but exact computation is often intractable. A broad research effort in recent years has aimed at designing structured prediction models and approximate inference and learning procedures that are computationally efficient. This volume offers an overview of this recent research in order to make the work accessible to a broader research community. The chapters, by leading researchers in the field, cover a range of topics, including research trends, the linear programming relaxation approach, innovations in probabilistic modeling, recent theoretical progress, and resource-aware learning.
Learning People Detectors for Tracking in Crowded Scenes.
Tang, S., Andriluka, M., Milan, A., Schindler, K., Roth, S., Schiele, B.
2014, Scene Understanding Workshop (SUNw, CVPR workshop) (unpublished)
Human Pose Estimation from Video and Inertial Sensors
Ph.D Thesis, -, 2014 (book)
The analysis and understanding of human movement is central to many applications
such as sports science, medical diagnosis and movie production. The ability to
automatically monitor human activity in security sensitive areas such as airports,
lobbies or borders is of great practical importance. Furthermore, automatic
pose estimation from images leverages the processing
and understanding of massive digital libraries available on the Internet.
We build upon a model based approach where the human shape is modelled with a surface mesh
and the motion is parametrized by a kinematic chain. We then seek for the pose
of the model that best explains the available observations coming from different sensors.
In a first scenario, we consider a calibrated mult-iview setup in an indoor studio. To obtain very accurate
results, we propose a novel tracker that combines information coming from video and a
small set of Inertial Measurement Units (IMUs). We do so by locally optimizing a joint
energy consisting of a term that measures the likelihood of the video data and a term
for the IMU data. This is the first work to successfully combine video and IMUs
information for full body pose estimation. When compared to commercial marker based systems
the proposed solution is more cost efficient and less intrusive for the user.
In a second scenario, we relax the assumption of an indoor studio and we tackle outdoor scenes
with background clutter, illumination changes, large recording volumes and difficult motions
of people interacting with objects. Again, we combine information from video and IMUs.
Here we employ a particle based optimization approach
that allows us to be more robust to tracking failures. To satisfy the orientation constraints
imposed by the IMUs, we derive an analytic Inverse Kinematics (IK) procedure to sample from the manifold
of valid poses. The generated hypothesis come from a lower dimensional manifold and therefore the computational
cost can be reduced. Experiments on challenging sequences suggest the proposed tracker can be applied
to capture in outdoor scenarios. Furthermore, the proposed IK sampling procedure can be used
to integrate any kind of constraints derived from the environment.
Finally, we consider the most challenging possible scenario: pose estimation of monocular images.
Here, we argue that estimating the pose to the degree of accuracy as in an engineered environment is
too ambitious with the current technology. Therefore, we propose to extract meaningful semantic information about
the pose directly from image features in a discriminative fashion. In particular, we introduce posebits
which are semantic pose descriptors about the geometric relationships between parts in the body.
The experiments
show that the intermediate step of inferring posebits from images can improve pose estimation from
monocular imagery. Furthermore, posebits can be very useful as input feature for many computer vision
algorithms.
2013
Human Pose Calculation from Optical Flow Data
European Patent Application EP 2843621 , August 2013 (patent)
System and method for generating bilinear spatiotemporal basis models
Matthews, I. A. I. S. T. S. K. S. Y.
US Patent Application 13/425,369, March 2013 (patent)
Techniques are disclosed for generating a bilinear spatiotemporal basis model. A method includes the steps of predefining a trajectory basis for the bilinear spatiotemporal basis model, receiving three-dimensional spatiotemporal data for a training sequence, estimating a shape basis for the bilinear spatiotemporal basis model using the three-dimensional spatiotemporal data, and computing coefficients for the bilinear spatiotemporal basis model using the trajectory basis and the shape basis.
A Study of X-Ray Image Perception for Pneumoconiosis Detection
IIIT-Hyderabad, Hyderabad, India, January 2013 (mastersthesis)
Pneumoconiosis is an occupational lung disease caused by the inhalation of industrial dust. Despite the increasing safety measures and better work place environments, pneumoconiosis is deemed to be the most common occupational disease in the developing countries like India and China. Screening and assessment of this disease is done through radiological observation of chest x-rays. Several studies have shown the significant inter and intra reader observer variation in the diagnosis of this disease, showing the complexity of the task and importance of the expertise in diagnosis.
The present study is aimed at understanding the perceptual and cognitive factors affecting the reading of chest x-rays of pneumoconiosis patients. Understanding these factors helps in developing better image acquisition systems, better training regimen for radiologists and development of better computer aided diagnostic (CAD) systems. We used an eye tracking experiment to study the various factors affecting the assessment of this diffused lung disease. Specifically, we aimed at understanding the role of expertize, contralateral symmetric (CS) information present in chest x-rays on the diagnosis and the eye movements of the observers. We also studied the inter and intra observer fixation consistency along with the role of anatomical and bottom up saliency features in attracting the gaze of observers of different expertize levels, to get better insights into the effect of bottom up and top down visual saliency on the eye movements of observers.
The experiment is conducted in a room dedicated to eye tracking experiments. Participants consisting of novices (3), medical students (12), residents (4) and staff radiologists (4) were presented with good quality PA chest X-rays, and were asked to give profusion ratings for each of the 6 lung zones. Image set consisting of 17 normal full chest x-rays and 16 single lung images are shown to the participants in random order. Time of the diagnosis and the eye movements are also recorded using a remote head free eye tracker.
Results indicated that Expertise and CS play important roles in the diagnosis of pneumoconiosis. Novices and medical students are slow and inefficient whereas, residents and staff are quick and efficient. A key finding of our study is that the presence of CS information alone does not help improve diagnosis as much as learning how to use the information. This learning appears to be gained from focused training and years of experience. Hence, good training for radiologists and careful observation of each lung zone may improve the quality of diagnostic results. For residents, the eye scanning strategies play an important role in using the CS information present in chest radiographs; however, in staff radiologists, peripheral vision or higher-level cognitive processes seems to play role in using the CS information.
There is a reasonably good inter and intra observer fixation consistency suggesting the use of similar viewing strategies. Experience is helping the observers to develop new visual strategies based on the image content so that they can quickly and efficiently assess the disease level. First few fixations seem to be playing an important role in choosing the visual strategy, appropriate for the given image.
Both inter-rib and rib regions are given equal importance by the observers. Despite reading of chest x-rays being highly task dependent, bottom up saliency is shown to have played an important role in attracting the fixations of the observers. This role of bottom up saliency seems to be more in lower expertize groups compared to that of higher expertize groups. Both bottom up and top down influence of visual fixations seems to change with time. The relative role of top down and bottom up influences of visual attention is still not completely understood and it remains the part of future work.
Based on our experimental results, we have developed an extended saliency model by combining the bottom up saliency and the saliency of lung regions in a chest x-ray. This new saliency model performed significantly better than bottom-up saliency in predicting the gaze of the observers in our experiment. Even though, the model is a simple combination of bottom-up saliency maps and segmented lung masks, this demonstrates that even basic models using simple image features can predict the fixations of the observers to a good accuracy.
Experimental analysis suggested that the factors affecting the reading of chest x-rays of pneumoconiosis are complex and varied. A good understanding of these factors definitely helps in the development of better radiological screening of pneumoconiosis through improved training and also through the use of improved CAD tools. The presented work is an attempt to get insights into what these factors are and how they modify the behavior of the observers.
2012
An Analysis of Successful Approaches to Human Pose Estimation
An Analysis of Successful Approaches to Human Pose Estimation, University of Augsburg, University of Augsburg, May 2012 (mastersthesis)
The field of Human Pose Estimation is developing fast and lately leaped forward
with the release of the Kinect system. That system reaches a very good perfor-
mance for pose estimation using 3D scene information, however pose estimation
from 2D color images is not solved reliably yet. There is a vast amount of pub-
lications trying to reach this aim, but no compilation of important methods and
solution strategies. The aim of this thesis is to fill this gap: it gives an introductory
overview over important techniques by analyzing four current (2012) publications
in detail. They are chosen such, that during their analysis many frequently used
techniques for Human Pose Estimation can be explained. The thesis includes two
introductory chapters with a definition of Human Pose Estimation and exploration
of the main difficulties, as well as a detailed explanation of frequently used methods.
A final chapter presents some ideas on how parts of the analyzed approaches can
be recombined and shows some open questions that can be tackled in future work.
The thesis is therefore a good entry point to the field of Human Pose Estimation
and enables the reader to get an impression of the current state-of-the-art.
Consumer Depth Cameras for Computer Vision - Research Topics and Applications
Fossati, A., Gall, J., Grabner, H., Ren, X., Konolige, K.
Advances in Computer Vision and Pattern Recognition, Springer, 2012 (book)
2008
GNU Octave Manual Version 3
John W. Eaton, David Bateman, Soren Hauberg
Network Theory Ltd., October 2008 (book)
2005
Visual motion analysis method for detecting arbitrary numbers of moving objects in image sequences
Jepson, A. D., Fleet, D. J., Black, M. J.
US Pat. 6,954,544, October 2005 (patent)
A Flow-Based Approach to Vehicle Detection and Background Mosaicking in Airborne Video
IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), Video Proceedings,, pages: 1202, 2005 (patent)
2003
Method and apparatus for generating a condensed version of a video sequence including desired affordances
Black, M. J., Ju, S., Minneman, S., Kimber, D.
US Pat. 6,560,281, May 2003 (patent)
Apparatus and method for identifying and tracking objects with view-based representations
Black, M. J., Jepson, A.
US Pat. 6,526,156, February 2003 (patent)
1995
Apparatus and method for tracking facial motion through a sequence of images
Black, M. J., Yacoob, Y.
US Pat. 5,802,220, December 1995 (patent)
Apparatus and method for recognizing facial expressions and facial gestures in a sequence of images
Black, M. J., Yacoob, Y.
US Pat. 5,774,591, December 1995 (patent)
Image segmentation using robust mixture models
Black, M. J., Jepson, A. D.
US Pat. 5,802,203, June 1995 (patent)