Multi-Task Learning Using Uncertainty for Realtime Multi-Person Pose Estimation

Ghasemi-Naraghi, Z.; Nickabadi, A.; Safabakhsh, R.

doi:10.22061/jecei.2023.9848.657

Document Type : Original Research Paper

Authors

Artificial Intelligence and Robotics Department, Faculty of Computer Engineering and Information Technology, Amirkabir University of Technology, Tehran , Iran.

https://doi.org/10.22061/jecei.2023.9848.657

Abstract

Background and Obejctives: Multi-task learning is a widespread mechanism to improve the learning of multiple objectives with a shared representation in one deep neural network. In multi-task learning, it is critical to determine how to combine the tasks loss functions. The straightforward way is to optimize the weighted linear sum of multiple objectives with equal weights. Despite some studies that have attempted to solve the realtime multi-person pose estimation problem from a 2D image, major challenges still remain unresolved.
Methods: The prevailing solutions are two-stream, learning two tasks simultaneously. They intrinsically use a multi-task learning approach for predicting the confidence maps of body parts and the part affinity fields to associate the parts to each other. They optimize the average of the two tasks loss functions, while the two tasks have different levels of difficulty and uncertainty. In this work, we overcome this problem by applying a multi-task objective that captures task-based uncertainties without any additional parameters. Since the estimated poses can be more certain, the proposed method is called “CertainPose”.
Results: Experiments are carried out on the COCO keypoints data sets. The results show that capturing the task-dependent uncertainty makes the training procedure faster and causes some improvements in human pose estimation.
Conclusion: The highlight advantage of our method is improving the realtime multi-person pose estimation without increasing computational complexity.

Keywords

Main Subjects

Artificial Intelligence

Open Access

This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this license, visit: http://creativecommons.org/licenses/by/4.0/

Publisher’s Note

JECEI Publisher remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Publisher

Shahid Rajaee Teacher Training University

References

[1] Y. Yang, D. Ramanan, “Articulated pose estimation with flexible mixtures-of-parts,” in Proc. CVPR 2011: 1385–1392, 2011.

[2] P. F. Felzenszwalb, D. P. Huttenlocher, “Pictorial structures for object recognition,” Int. J. Comput. Vision, 61(1): 55–79, 2005.

[3] X. Chen, A. L. Yuille, “Articulated pose estimation by a graphical model with image dependent pairwise relations,” in Proc. NIPS, 2014.

[4] K. He, X. Zhang, S. Ren, J. Sun, “Deep residual learning for image recognition,” in Proc. the IEEE Conference on Computer Vision and Pattern Recognition: 770–778, 2016.

[5] S. Ren, K. He, R. Girshick, J. Sun, “Faster r-cnn: Towards real-time object detection with region proposal networks,” in Proc. NIPS, 2015.

[6] G. Papandreou, T. Zhu, N. Kanazawa, A. Toshev, J. Tompson, C. Bregler, K. Murphy, “Towards accurate multi-person pose estimation in the wild,” in Proc. the IEEE Conference on Computer Vision and Pattern Recognition: 4903–4911, 2017.

[7] K. He, G. Gkioxari, P. Dollár, R. Girshick, “Mask r-cnn,” in Proc. the IEEE International Conf. on Computer Vision: 2961–2969, 2017.

[8] S. E. Wei, V. Ramakrishna, T. Kanade, Y. Sheikh, ” Convolutional pose machines,” in Proc. the IEEE Conference on Computer Vision and Pattern Recognition: 4724–4732, 2016.

[9] H. S. Fang, S. Xie, Y. W. Tai, C. Lu, “Rmpe: Regional multi-person pose estimation,” in Proc. the IEEE International Conference on Computer Vision: 2334–2343, 2017.

[10] L. Ladicky, P. H. Torr, A. Zisserman, “Human pose estimation using a joint pixel-wise and part-wise formulation,” in Proc. IEEE Conf. on Computer Vision and Pattern Recognition: 3578– 585, 2013.

[11] U. Iqbal, J. Gall, “Multi-person pose estimation with local joint-to-person associations,” in Proc. European Conference on Computer Vision: 627–642, 2016.

[12] L. Pishchulin, E. Insafutdinov, S. Tang, B. Andres, M. Andriluka, P. V. Gehler, B. Schiele, “Deepcut: Joint subset partition and labeling formulti person pose estimation,” in Proc. the IEEE Conference on Computer Vision and Pattern Recognition: 4929–4937, 2016.

[13] E. Insafutdinov, L. Pishchulin, B. Andres, M. Andriluka, B. Schiele, “Deepercut: A deeper, stronger, and faster multi-person pose estimation model,” in Proc. European Conference on Computer Vision: 34–50, 2016.

[14] Z. Cao, T. Simon, S. E. Wei, Y. Sheikh, “Realtime multi-person 2dpose estimation using part affinity fields,” in Proc. the IEEE Conf. Computer Vision and Pattern Recognition: 7291–7299, 2017.

[15] X. Zhu, Y. Jiang, Z. Luo, “Multi-person pose estimation for posetrack with enhanced part affinity fields,” in Proc. ICCV PoseTrack Workshop, 7, 2017.

[16] D. Osokin, “Real-time 2d multi-person pose estimation on cpu: Lightweight openpose,” in Proc. the 8th International Conference on Pattern Recognition Applications and Methods: 744–748, 2019.

[17] Z. Cao, G. Hidalgo, T. Simon, S. E. Wei, Y. Sheikh, “Openpose: real- time multi-person 2d pose estimation using part affinity fields,” IEEE Trans. Pattern Anal. Mach. Intell., 43(1): 172–186, 2019.

[18] OpenPose library. https://github.com/CMU-Perceptual-Computing-Lab/openpose.

[19] H. Liu, D. Luo, S. Du, T. Ikenaga, “Resolution irrelevant encoding and difficulty balanced loss based network independent supervision for multi-person pose estimation,” in Proc. 13th International Conf. Human System Interaction (HSI): 112–117, 2020.

[20] G. H. Martınez, “OpenPose: Whole-Body Pose Estimation,” April, 2019.

[21] T. Gong, T. Lee, C. Stephenson, V. Renduchintala, S. Padhy, A. Ndi-rango, G. Keskin, O. H. Elibol, “A comparison of loss weighting strategies for multi task learning in deep neural networks,” IEEE Access, 7: 141627–141632, 2019.

[22] Q. Dang, J. Yin, B. Wang, W. Zheng, “Deep learning based 2d human pose estimation: A survey,” Tsinghua Sci. Technol., 24(6): 663– 676, 2019.

[23] C. Zheng, W. Wu, T. Yang, S. Zhu, C. Chen, R. Liu, J. Shen, N. Kehtarnavaz, M. Shah, “Deep learning-based human pose estimation: A survey,” arXiv preprint arXiv:2012.13392, 2020.

[24] T. L. Munea, Y. Z. Jembre, H. T. Weldegebriel, L. Chen, C. Huang, C. Yang, “The progress of human pose estimation: A survey and taxonomy of models applied in 2d human pose estimation,” IEEE Access, 8: 133330–133348, 2020.

[25] W. Gong, X. Zhang, J. Gonzàlez, A. Sobral, T. Bouwmans, C. Tu, E. H. Zahzah, “Human pose estimation from monocular images: A comprehensive survey,” Sensors, 16(12): 1966, 2016.

[26] G. Rogez, C. Schmid, “Mocap-guided data augmentation for 3d pose estimation in the wild,” in Proc. Advances in Neural Information Processing Systems (NIPS): 3108–3116, 2016.

[27] H. Jiang, “Finding human poses in videos using concurrent matching and segmentation,” in Proc. Asian Conference on Computer Vision: 228–243, 2010.

[28] H. Sidenbladh, F. De la Torre, M. J. Black, “A framework for modeling the appearance of 3d articulated figures,” in Proc. Fourth IEEE International Conference on Automatic Face and Gesture Recognition (Cat. No. PR00580): 368–375, 2000.

[29] J. J. Tompson, A. Jain, Y. LeCun, C. Bregler, “Joint training of a convolutional network and a graphical model for human pose estimation,” in Proc. Advances in Neural Information Processing Systems, 27, 2014.

[30] V. Ramakrishna, D. Munoz, M. Hebert, J. A. Bagnell, Y. Sheikh, “Pose machines: Articulated pose estimation via inference machines,” in Proc. European Conf. Computer Vision: 33–47, 2014.

[31]MSCOCO Dataset, https://cocodataset.org/#home.

[32] T. Simon, H. Joo, I. Matthews, Y. Sheikh, “Hand keypoint detection in single images using multiview bootstrapping,” in Proc. the IEEE Conf. Computer Vision and Pattern Recognition: 1145–1153, 2017.

[33] S. Kreiss, L. Bertoni, A. Alahi, “Pifpaf: Composite fields for human pose estimation,” in Proc. the IEEE/CVF Conference on Computer Vision and Pattern Recognition: 11977–11986, 2019.

[34] N. Nakano, T. Sakura, K. Ueda, L. Omura, A. Kimura, Y. Iino, S. Fukashiro, S. Yoshioka, “Evaluation of 3d markerless motion capture accuracy using openpose with multiple video cameras,” Fron. sports Active Living, 2(50), 2020.

[35] N. D. Reddy, L. Guigues, L. Pishchulin, J. Eledath, S. G. Narasimhan, “Tessetrack: End-to-end learnable multi-person articulated 3d pose tracking,” in Proc. the IEEE/CVF Conference on Computer Vision and Pattern Recognition: 15190–15200, 2021.

[36] N. D. Reddy, M. Vo, S. G. Narasimhan, “Occlusion-net: 2d/3d occluded keypoint localization using graph networks,” in Proc. the IEEE/CVF Conference on Computer Vision and Pattern Recognition: 7326–7335, 2019.

[37] Y. Cheng, B. Wang, B. Yang, R. T. Tan, “Monocular 3d multi-person pose estimation by integrating top-down and bottom-up networks,” in Proc. the IEEE/CVF Conference on Computer Vision and Pattern Recognition: 7649–7659, 2021.

[38] H. Tu, C. Wang, W. Zeng, “Voxelpose: Towards multi-camera 3d human pose estimation in wild environment,” in Proc. 16th European Conference on Computer Vision–ECCV 2020, Part I 16: 197–212, 2020.

[39] G. Zhang, J. Liu, H. Li, Y. Q. Chen, L. S. Davis, “Joint human detection and head pose estimation via multistream networks for rgb-d videos,” IEEE Signal Process. Lett., 24(11): 1666–1670, 2017.

[40] M. Schwarz, H. Schulz, S. Behnke, “Rgb-d object recognition and pose estimation based on pre-trained convolutional neural network features,” in Proc. 2015 IEEE International Conference on Robotics and Automation (ICRA): 1329–1335, 2015.

[41] A. Krull, E. Brachmann, F. Michel, M. Y. Yang, S. Gumhold, C. Rother, “Learning analysis-by-synthesis for 6d pose estimation in rgb-d images,” in Proc. the IEEE International Conference on Computer Vision: 954–962, 2015.

[42] Y. Gal, :Uncertainty in deep learning,” University of Cambridge 1(3), 2016.

[43] A. Kendall, Y. Gal, “What uncertainties do we need in bayesian deep learning for computer vision?” in Proc. Advances in Neural Information Processing Systems: 5574–5584, 2017.

[44] F. K. Gustafsson, M. Danelljan, T. B. Schon, “Evaluating scalable bayesian deep learning methods for robust computer vision,” in Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops: 318–319, 2020.

[45] I. Misra, A. Shrivastava, A. Gupta, M. Hebert, “Cross-stitch networks for multi-task learning,” in Proc. the IEEE Conference on Computer Vision and Pattern Recognition: 3994–4003, 2016.

[46] Z. Chen, V. Badrinarayanan, C. Y. Lee, A. Rabinovich, “Gradnorm: Gradient normalization for adaptive loss balancing in deep multitask networks,” in Proc. International Conference on Machine Learning: 794–803, 2018.

[47] A. Kendall, Y. Gal, R. Cipolla, “Multi-task learning using uncertainty to weigh losses for scene geometry and semantics,” in Proc. the IEEE Conference on Computer Vision and Pattern Recognition: 7482–7491, 2018.

[48] T. Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, C.L. Zitnick, “Microsoft coco: Common objects in context. In Proc. European Conf. Computer Vision: 740–755, 2014.

[49] M. Ruggero Ronchi, P. Perona, “Benchmarking and error diagnosis in multi-instance pose estimation,” in Proc. the IEEE International Conference on Computer Vision: 369–378, 2017.

[50] A. Newell, Z. Huang, J. Deng, “Associative embedding: End-to-end learning for joint detection and grouping,” in Proc. Advances in Neural Information Processing Systems: 2278–2288, 2017.

[51] M. Kocabas, S. Karagoz, E. Akbas, “Multiposenet: Fast multi-person pose estimation using pose residual network,” in Proc. the European Conference on Computer Vision (ECCV): 417–433, 2018.

LETTERS TO EDITOR

Journal of Electrical and Computer Engineering Innovations (JECEI) welcomes letters to the editor for the post-publication discussions and corrections which allows debate post publication on its site, through the Letters to Editor. Letters pertaining to manuscript published in JECEI should be sent to the editorial office of JECEI within three months of either online publication or before printed publication, except for critiques of original research. Following points are to be considering before sending the letters (comments) to the editor.

[1] Letters that include statements of statistics, facts, research, or theories should include appropriate references, although more than three are discouraged.

[2] Letters that are personal attacks on an author rather than thoughtful criticism of the author’s ideas will not be considered for publication.

[3] Letters can be no more than 300 words in length.

[4] Letter writers should include a statement at the beginning of the letter stating that it is being submitted either for publication or not.

[5] Anonymous letters will not be considered.

[6] Letter writers must include their city and state of residence or work.

[7] Letters will be edited for clarity and length.

Name *

Email Address *

Affiliation *

Comments *

Security Code *

Journal of Electrical and Computer Engineering Innovations (JECEI)

Multi-Task Learning Using Uncertainty for Realtime Multi-Person Pose Estimation

References

References

Send comment about this article

Volume 12, Issue 1
January 2024
Pages 147-162

Multi-Task Learning Using Uncertainty for Realtime Multi-Person Pose Estimation

References

References

Send comment about this article

Volume 12, Issue 1January 2024Pages 147-162

Volume 12, Issue 1
January 2024
Pages 147-162