Header logo is

On the Integration of Optical Flow and Action Recognition


Conference Paper



Most of the top performing action recognition methods use optical flow as a "black box" input. Here we take a deeper look at the combination of flow and action recognition, and investigate why optical flow is helpful, what makes a flow method good for action recognition, and how we can make it better. In particular, we investigate the impact of different flow algorithms and input transformations to better understand how these affect a state-of-the-art action recognition method. Furthermore, we fine tune two neural-network flow methods end-to-end on the most widely used action recognition dataset (UCF101). Based on these experiments, we make the following five observations: 1) optical flow is useful for action recognition because it is invariant to appearance, 2) optical flow methods are optimized to minimize end-point-error (EPE), but the EPE of current methods is not well correlated with action recognition performance, 3) for the flow methods tested, accuracy at boundaries and at small displacements is most correlated with action recognition performance, 4) training optical flow to minimize classification error instead of minimizing EPE improves recognition performance, and 5) optical flow learned for the task of action recognition differs from traditional optical flow especially inside the human body and at the boundary of the body. These observations may encourage optical flow researchers to look beyond EPE as a goal and guide action recognition researchers to seek better motion cues, leading to a tighter integration of the optical flow and action recognition communities.

Author(s): Laura Sevilla-Lara and Yiyi Liao and Fatma Güney and Varun Jampani and Andreas Geiger and Michael J. Black
Book Title: German Conference on Pattern Recognition (GCPR)
Volume: LNCS 11269
Pages: 281--297
Year: 2018
Month: October
Publisher: Springer, Cham

Department(s): Autonomous Vision, Perceiving Systems
Bibtex Type: Conference Paper (inproceedings)
Paper Type: Conference

DOI: https://doi.org/10.1007/978-3-030-12939-2_20

Links: arXiv


  title = {On the Integration of Optical Flow and Action Recognition},
  author = {Sevilla-Lara, Laura and Liao, Yiyi and G{\"u}ney, Fatma and Jampani, Varun and Geiger, Andreas and Black, Michael J.},
  booktitle = {German Conference on Pattern Recognition (GCPR)},
  volume = {LNCS 11269},
  pages = {281--297},
  publisher = {Springer, Cham},
  month = oct,
  year = {2018},
  month_numeric = {10}