Depth Estimation and Deblurring from a Single Image using an Optimized-throughput Coded Aperture

Masoudifar, M.; Pourreza, H. R.

doi:10.22061/jecei.2022.8630.537

Document Type : Original Research Paper

Authors

M. Masoudifar ¹
H. R. Pourreza ²

¹ Department of Computer Engineering, Hakim Sabzevari University, Sabzevar, Iran.

² Computer Engineering Department, Ferdowsi University of Mashhad, Mashhad, Iran.

https://doi.org/10.22061/jecei.2022.8630.537

Abstract

Background and Objectives: Depth from defocus and defocus deblurring from a single image are two challenging problems caused by the finite depth of field in conventional cameras. Coded aperture imaging is a branch of computational imaging, which is used to overcome these two problems. Up to now, different methods have been proposed for improving the results of either defocus deblurring or depth estimation. In this paper, an asymmetric coded aperture is proposed which improves results of depth estimation and defocus deblurring from a single input image.
Methods: To this aim, a multi-objective optimization function taking into consideration both deblurring results and depth discrimination ability is proposed. Since aperture throughput affects on image quality, our optimization function is defined based on illumination conditions and camera specifications which yields an optimized throughput aperture. Because the designed pattern is asymmetric, defocused objects on two sides of the focal plane can be distinguished. Depth estimation is performed using a new algorithm, which is based on perceptual image quality assessment criteria and can discern blurred objects lying in front or behind the focal plane.
Results: Extensive simulations as well as experiments on a variety of real scenes are conducted to compare the performance of our aperture with previously proposed ones.
Conclusion: Our aperture has been designed for indoor illumination settings. However, the proposed method can be utilized for designing and evaluating appropriate aperture patterns for different imaging conditions.

Keywords

Main Subjects

Computational Imaging

Open Access

This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this license, visit: http://creativecommons.org/licenses/by/4.0/

Publisher’s Note

JECEI Publisher remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Publisher

Shahid Rajaee Teacher Training University

References

[1] A. P. Pentland, "A new sense for depth of field," IEEE Trans. Pattern Anal. Mach. Intell., 9(4): 523-531, 1987.

[2] M. Subbarao, N. Gurumoorthy, "Depth recovery from blurred edges," in Proc. Computer Society Conference on Computer Vision and Pattern Recognition, CVPR'88: 498-503, 1988.

[3] A. Sellent, P. Favaro, "Which side of the focal plane are you on?," in Proc. 2014 IEEE International Conference on Computational Photography (ICCP): 1-8, 2014.

[4] A. Levin, R. Fergus, F. Durand, W. T. Freeman, "Image and depth from a conventional camera with a coded aperture," ACM Trans. Graphics, 26(3): 70, 2007.

[5] A. Sellent, P. Favaro, "Optimized aperture shapes for depth estimation," Pattern Recognit. Lett., 40: 96-103, 2014.

[6] A. Veeraraghavan, R. Raskar, A. Agrawal, A. Mohan, J. Tumblin, "Dappled photography: Mask enhanced cameras for heterodyned light fields and coded aperture refocusing," ACM Trans. Graphics, 26(3): 1-12, 2007.

[7] C. Zhou, S. Nayar, "What are good apertures for defocus deblurring?" in Proc. 2009 IEEE International Conference on Computational Photography (ICCP): 1-8, 2009.

[8] B. Masia, L. Presa, A. Corrales, D. Gutierrez, "Perceptually optimized coded apertures for defocus deblurring," Comput. Graphics Forum, 31(6): 1867-1879, 2012.

[9] Y. Takeda, S. Hiura, K. Sato, "Fusing depth from defocus and stereo with coded apertures," in Proc 2013 IEEE Conference on Computer Vision and Pattern Recognition (CVPR): 209-216, 2013.

[10] C. Zhou, S. Lin, S. K. Nayar, "Coded aperture pairs for depth from defocus and defocus deblurring," Int. J. Comput. Vision, 93(1): 53-72, 2011.

[11] A. Konak, D. W. Coit, A. E. Smith, "Multi-objective optimization using genetic algorithms: A tutorial," Reliab. Eng. Syst. Saf., 91(9): 992-1007, 2006.

[12] K. Mitra, O. S. Cossairt, A. Veeraraghavan, "A framework for analysis of computational imaging systems: Role of signal prior, sensor noise and multiplexing," IEEE Trans. Pattern Anal. Mach. Intell., 36(10): 1909-1921, 2014.

[13] V. Paramonov, I. Panchenko, V. Bucha, A. Drogolyub, S. Zagoruyko, "Depth camera based on color-coded aperture," in Proc. the IEEE Conference on Computer Vision and Pattern Recognition Workshops: 1-9, 2016.

[14] V. Aslantas, "A depth estimation algorithm with a single image," Opt. express, 15(8): 5024-5029, 2007.

[15] S. Zhuo, T. Sim, "Defocus map estimation from a single image," Pattern Recognit., 44(9): 1852-1858, 2011.

[16] J. Lin, X. Ji, W. Xu, Q. Dai, "Absolute depth estimation from a single defocused image," IEEE Trans. Image Process., 22(11): 4545-4550, 2013.

[17] X. Zhu, S. Cohen, S. Schiller, P. Milanfar, "Estimating spatially varying defocus blur from a single image," IEEE Trans. Image Process., 22(12): 4879-4891, 2013.

[18] S. Gur, L. Wolf, "Single image depth estimation trained via depth from defocus cues," in Proc. the IEEE/CVF Conference on Computer Vision and Pattern Recognition: 7683-7692, 2019.

[19] P. Hambarde, S. Murala, "S2DNet: Depth estimation from single image and sparse samples," IEEE Trans. Comput. Imaging, 6: 806-817, 2020.

[20] A. N. Rajagopalan, S. Chaudhuri, "Optimal selection of camera parameters for recovery of depth from defocused images," in Proc. IEEE Computer Society Conference on Computer Vision and Pattern Recognition: 219-224, 1997.

[21] M. Watanabe, S. K. Nayar, "Rational filters for passive depth from defocus," Int. J. Comput. Vision, 27(3): 203-225, 1998.

[22] P. Favaro, S. Soatto, "A geometric approach to shape from defocus," IEEE Trans. Pattern Anal. Mach. Intell., 27(3): 406-417, 2005.

[23] Matsui, H. Nagahara, R.I. Taniguchi, "Half-sweep imaging for depth from defocus," Image Vision Comput., 32(11): 954-964, 2014.

[24] M. Ye, X. Chen, Q. Li, J. Zeng, S. Yu, "Depth from defocus measurement method based on liquid crystal lens," Opt. Express, 26(2): 28413-28420, 2018.

[25] S. W. Hasinoff, K. N. Kutulakos, "Confocal stereo," Int. J. comput. vision, 81(1): 82-104, 2009.

[26] A. Rajagopalan, S. Chaudhuri, U. Mudenagudi, "Depth estimation and image restoration using defocused stereo pairs," IEEE Trans. Pattern Anal. Mach. Intell., 26(11): 1521-1525, 2004.

[27] S. Hiura, T. Matsuyama, "Depth measurement by the multi-focus camera," in Proc. 1998 IEEE Computer Society Conference on Computer Vision and Pattern Recognition: 953-959, 1998.

[28] O. Cossairt, M. Gupta, S. K. Nayar, "When does computational imaging improve performance?," IEEE Trans. Image Process., 22(2): 447-458, 2013.

[29] Y. Weiss, W. T. Freeman, "What makes a good model of natural images?," in Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR'07): 1-8, 2007.

[30] P. E. Debevec, J. Malik, "Recovering high dynamic range radiance maps from photographs," in ACM SIGGRAPH 2008 classes: 1-10, 2008.

[31] O. Cossairt, "Tradeoffs and limits in computational imaging," Columbia University, 2011.

[32] K. Deb, A. Pratap, S. Agarwal, T. Meyarivan, "A fast and elitist multiobjective genetic algorithm: NSGA-II," IEEE Trans. Evol. Comput., 6(2): 182-197, 2002.

[33] Y. Gao, "Population size and sampling complexity in genetic algorithms," in Proc. the Bird of a Feather Workshops: 178-181, 2003.

[34] S. Theodoridis, R. Chellappa, Academic Press Library in Signal Processing: Image, Video Processing and Analysis, Hardware, Audio, Acoustic and Speech Processing: Elsevier Science, 2013.

[35] M. Martinello, "Coded aperture imaging," Heriot-Watt University, Edinburgh, Scotland, 2012.

[36] Y. Liu, J. Wang, S. Cho, A. Finkelstein, S. Rusinkiewicz, "A no-reference metric for evaluating the quality of motion deblurring," ACM Trans. Graph., 32(6): 175, 2013.

[37] B. Hu, L. Li, J. Qian, "Perceptual quality evaluation for motion deblurring," IET Comput. Vision, 12(6): 796-805, 2018.

[38] D. Krishnan, T. Tay, R. Fergus, "Blind deconvolution using a normalized sparsity measure," in Proc. 2011 IEEE Conference on Computer Vision and Pattern Recognition (CVPR): 233-240, 2011.

[39] G. Blanchet, L. Moisan, "An explicit sharpness index related to global phase coherence," in Proc. 2012 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP): 1065-1068, 2012.

[40] M. Masoudifar, H. R. Pourreza, "Depth estimation from a single defocused image using no reference image quality assessment metrics," presented at the Fifth International Conference on Technology Development in Iranian Electrical Engineering, 2021.

[41] A. Delong, A. Osokin, H. N. Isack, Y. Boykov, "Fast approximate energy minimization with label costs," Int. J. comput. vision, 96(1): 1-27, 2012.

LETTERS TO EDITOR

Journal of Electrical and Computer Engineering Innovations (JECEI) welcomes letters to the editor for the post-publication discussions and corrections which allows debate post publication on its site, through the Letters to Editor. Letters pertaining to manuscript published in JECEI should be sent to the editorial office of JECEI within three months of either online publication or before printed publication, except for critiques of original research. Following points are to be considering before sending the letters (comments) to the editor.

[1] Letters that include statements of statistics, facts, research, or theories should include appropriate references, although more than three are discouraged.

[2] Letters that are personal attacks on an author rather than thoughtful criticism of the author’s ideas will not be considered for publication.

[3] Letters can be no more than 300 words in length.

[4] Letter writers should include a statement at the beginning of the letter stating that it is being submitted either for publication or not.

[5] Anonymous letters will not be considered.

[6] Letter writers must include their city and state of residence or work.

[7] Letters will be edited for clarity and length.

Name *

Email Address *

Affiliation *

Comments *

Security Code *

Journal of Electrical and Computer Engineering Innovations (JECEI)

Depth Estimation and Deblurring from a Single Image using an Optimized-throughput Coded Aperture

References

References

Send comment about this article

Volume 11, Issue 1
January 2023
Pages 51-64

Depth Estimation and Deblurring from a Single Image using an Optimized-throughput Coded Aperture

References

References

Send comment about this article

Volume 11, Issue 1January 2023Pages 51-64

Volume 11, Issue 1
January 2023
Pages 51-64