Understanding human action requires modeling and understanding human movement. While we mostly focus on 3D human movement, what is directly observable in videos is the 2D optical flow. Previous work has shown that flow is useful for action recognition and, consequently, we explore how to better estimate human flow and improve action recognition.
Specifically, we trained a neural network to compute human optical flow [ ]. To enable this we created a new synthetic training database of image sequences with ground truth human flow. For this we use the 3D SMPL body model and motion capture data to synthesize realistic flow fields; this effectively extends the SURREAL dataset [ ]. We then train a convolutional neural network (SpyNet [ ] with some modifications) to estimate human flow from pairs of images.
The new network is more accurate than a wide range of top methods on held-out test data and generalizes well to real image sequences. When combined with a person detector/tracker, the approach provides a full solution to the problem of 2D human flow estimation.
Most of the top performing action recognition methods use optical flow as a ``black box'' input. In [ ], we take a deeper look at the combination of flow and action recognition, and investigate why optical flow is helpful, what makes a flow method good for action recognition, and how we can make it better.
Specifically, we fine tune two neural-network flow methods end-to-end on the UCF101 action recognition dataset. Based on these experiments, we make the following five observations: 1) optical flow is useful for action recognition because it is invariant to appearance, 2) optical flow methods are optimized to minimize end-point-error (EPE), but the EPE of current methods is not well correlated with action recognition performance, 3) flow accuracy at boundaries and for small displacements is most correlated with action recognition performance, 4) training optical flow to minimize classification error instead of EPE improves recognition performance, and 5) optical flow learned for the task of action recognition mostly differs from traditional optical flow inside and at the boundary of the human body.