Hybrid Convolutional Neural Network with Domain adaptation for Sketch based Image Retrieval

Gheitasi, A.; Farsi, H.; Mohamadzadeh, S.

doi:10.22061/jecei.2024.10778.735

Document Type : Original Research Paper

Authors

Department of Electrical and Computer Engineering, University of Birjand, Birjand, Iran.

https://doi.org/10.22061/jecei.2024.10778.735

Abstract

Background and Objectives: Freehand sketching is an easy-to-use but effective instrument for computer-human connection. Sketches are highly abstract to the domain gap, that exists between the intended sketch and real image. In addition to appearance information, it is believed that shape information is also very efficient in sketch recognition and retrieval.
Methods: In the realm of machine vision, comprehending Freehand Sketches has grown more crucial due to the widespread use of touchscreen devices. In addition to appearance information, it is believed that shape information is also very efficient in sketch recognition and retrieval. The majority of sketch recognition and retrieval methods utilize appearance information-based tactics. A hybrid network architecture comprising two networks—S-Net (Sketch Network) and A-Net (Appearance Network)—is shown in this article under the heading of hybrid convolution. These subnetworks, in turn, describe appearance and shape information. Conversely, a module known as the Conventional Correlation Analysis (CCA) technique module is utilized to match the range and enhance the sketch retrieval performance to decrease the range gap distance. Finally, sketch retrieval using the hybrid Convolutional Neural Network (CNN) and CCA domain adaptation module is tested using many datasets, including Sketchy, Tu-Berlin, and Flickr-15k. The final experimental results demonstrated that compared to more sophisticated methods, the hybrid CNN and CCA module produced high accuracy and results.
Results: The proposed method has been evaluated in the two fields of image classification and Sketch Based Image Retrieval (SBIR). The proposed hybrid convolution works better than other basic networks. It achieves a classification score of 84.44% for the TU-Berlin dataset and 82.76% for the sketchy dataset. Additionally, in SBIR, the proposed method stands out among methods based on deep learning, outperforming non-deep methods by a significant margin.
Conclusion: This research presented the hybrid convolutional framework, which is based on deep learning for pattern recognition. Compared to the best available methods, hybrid network convolution has increased recognition and retrieval accuracy by around 5%. It is an efficient and thorough method which demonstrated valid results in Sketch-based image classification and retrieval on TU-Berlin, Flickr 15k, and sketchy datasets.

Keywords

Main Subjects

Image Annotation and Retrieval

Open Access

This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this license, visit: http://creativecommons.org/licenses/by/4.0/

Publisher’s Note

JECEI Publisher remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Publisher

Shahid Rajaee Teacher Training University

References

[1] D. Birari, D. Hiran, V. Narawade, “Survey on sketch based image and data retrieval,” in Proc. 2nd International Conference on Communications and Cyber Physical Engineering (ICCCE 2019): 285-290, 2020.

[2] A. Chaudhuri, A. K. Bhunia, Y. Z. Song, A. Dutta, “Data-free sketch-based image retrieval,” arXiv preprint arXiv:2303.07775, 2023.

[3] A. Sain, A. K. Bhunia, Y. Yang, T. Xiang, Y. Z. Song, “Stylemeup: Towards style-agnostic sketch-based image retrieval,” in Proc. the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021.

[4] P. Xu, T. M. Hospedales, Q. Yin, Y. Zh. Song, T. Xiang, L. Wang, “Deep learning for free-hand sketch: A survey,” IEEE Trans. Pattern Anal. Mach. Intell., 45(1): 285-312, 2023.

[5] Z. Hossein-Nejad, H. Agahi, A. Mahmoodzadeh, “Remote sensing image registration based on a geometrical model matching,” J. Inf. Syst. Telecommun. (JIST), 5(36): 41, 2021.

[6] R. Hu, S. James, T. Wang, J. Collomosse, “Markov random fields for sketch based video retrieval,” in Proc. the 3rd ACM conference on International Conference on Multimedia Retrieval (ICMR): 279-286, 2013.

[7] Y. Li, W. Li, “A survey of sketch-based image retrieval,” Mach. Vision Appl., 29(7): 1083-1100, 2018.

[8] S. Mohamadzadeh, S. Pasban, J. Zeraatkar-Moghadam, A. K. Shafiei, “Parkinson’s disease detection by using feature selection and sparse representation,” J. Med. Biol. Eng., 41(4): 412-421, 2021.

[9] X. Zhang, Y. Huang, Q. Zou, Y. Pei, R. Zhang, S. Wang, “A hybrid convolutional neural network for sketch recognition,” Pattern Recognit. Lett., 130: 73-82, 2020.

[10] P. Xu, T. M. Hospedales, Q. Yin, Y. Zh. Song, T. Xiang, L. Wang, “Deep learning for free-hand sketch: A survey,” IEEE Trans. Pattern Anal. Mach. Intell., 45(1): 285-312, 2020.

[11] P. Xu, Y. Huang, T. Yuan, K. Pang, Y. Zh. Song, T. Xiang, T. M. Hospedales, Zh. Ma, J. Guo, “Sketchmate: Deep hashing for million-scale human sketch retrieval,” in Proc. the IEEE Conference on Computer Vision and Pattern Recognition (CVPR 2018): 8090-8098, 2018.

[12] A. Dutta, Z. Akata, “Semantically tied paired cycle consistency for zero-shot sketch-based image retrieval,” in Proc. the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR 2019): 5089-5098, 2019.

[13] H. Yu, M. Huang, J. J. Zhang, “Domain adaptation problem in sketch based image retrieval,” ACM Trans. Multimidia Comput. Commun. Appl., 19(3): 1-17, 2022.

[14] A. K. Bhunia, A. Sain, P. H. Shah, A. Gupta, P. N. Chowdhury, T. Xiang, Y. Zh. Song, “Adaptive fine-grained sketch-based image retrieval,” in Proc. 17th European Conference Computer Vision (ECCV 2022), Part XXXVII: 163-181, 2022.

[15] E. Shechtman, M. Irani, “Matching local self-similarities across images and videos,” in Proc. 2007 IEEE Conference on Computer Vision and Pattern Recognition: 1-8, 2007.

[16] J. M. Saavedra, “Rst-shelo: Sketch-based image retrieval using sketch tokens and square root normalization,” Multimedia Tools Appl., 76(1): 931-951, 2017.

[17] R. G. Schneider, T. Tuytelaars, “Sketch classification and classification-driven analysis using fisher vectors,” ACM Trans. Graphics (TOG), 33(6): 1-9, 2014.

[18] V. Sabeti, “An improved approach to blind image steganalysis using an overlapping blocks idea,” J. Electr. Comput. Eng. Innovations, 11(2): 263-276, 2023.

[19] M. Imani, “Target detection using multispectral images, A case study: Wheat detection in Chenaran County in Iran,” J. Electr. Comput. Eng. Innovations, 9(1): 11-24, 2020.

[20] M. Rezaei, M. Rezaei, “Foreground-back ground segmentation using k-means clustering algorithm and support vector machine,” J. Inf. Syst. Telecommun. (JIST), 1(41): 65, 2023.

[21] S. Fooladi, H. Farsi, S. Mohamadzadeh, “Segmenting the lesion area of brain tumor using convolutional neural networks and fuzzy K-means clustering,” Int. J. Eng., 36(8): 1556-1568, 2023.

[22] A. Sain, A. K. Bhunia, Y. Yang, T. Xiang, Y. Zh. Song, “Cross-modal hierarchical modelling for fine-grained sketch based image retrieval,” arXiv preprint arXiv:2007.15103, 2020.

[23] A. K. Bhunia, P. N. Chowdhury, A. Sain, Y. Yang, T. Xiang, Y. Zh. Song, “More photos are all you need: Semi-supervised learning for fine-grained sketch based image retrieval,” in Proc. the IEEE/CVF Conference on Computer Vision and Pattern Recognition: 4247-4256, 2021.

[24] A. K. Bhunia, S. Koley, A. F. U. R. Khilji, A. Sain, P. N. Chowdhury, T. Xiang, Y. Zh. Song, “Sketching without worrying: Noise-tolerant sketch-based image retrieval,” in Proc. the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR 2019): 999-1008, 2022.

[25] A. Gheitasi, H. Farsi, S. Mohamadzadeh, “Estimation of hand skeletal postures by using deep convolutional neural networks,” Int. J. Eng., 33(4): 552-559, 2020.

[26] Q. Yu, F. Liu, Y. Zh. Song, T. Xiang, T. M. Hospedales, Ch. Ch. Loy, “Sketch me that shoe,” in Proc. the IEEE Conference on Computer Vision and Pattern Recognition (CVPR 2016): 799-807, 2016.

[27] C. Szegedy, W. Liu, Y. Jia, P. Sermanet, S. Reed, D. Anguelov, D. Erhan, V. Vanhoucke, A. Rabinovich, “Going deeper with convolutions,” in Proc. the IEEE Conference on Computer Vision and Pattern Recognition: 1-9, 2015.

[28] N. Kumar, R. Ahmed, V. B. Honnakasturi, S. Sowmya Kamath, V. Mayya, “Sketch-based image retrieval using convolutional neural networks based on feature adaptation and relevance feedback,” in Proc. International Conference on Emerging Applications of Information Technology: 103-113, 2022.

[29] H. Zhang, S. Liu, Ch. Zhang, W. Ren, R. Wang, X. Cao, “Sketchnet: Sketch classification with web images,” in Proc. the IEEE Conference on Computer Vision and Pattern Recognition: 1105-1113, 2016.

[30] S. D. Bhattacharjee, J. Yuan, W. Hong, X, Ruan, “Query adaptive instance search using object sketches,” in Proc. the 24th ACM International Conference on Multimedia: 1306-1315, 2016.

[31] F. Lin, M. Li, D. Li, T. Hospedales, Y. Zh. Song, Y. Qi, “Zero-shot everything sketch-based image retrieval,” in Proc. the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR 2023): 23349-23358, 2023.

[32] R. Chavhan, Zero-Shot Sketch Based Image Retrieval, Indian Institute of Technology Bombay, 2021.

[33] J. Y. He, X. Wu, Y. G. Jiang, B. Zhao, Q. Peng, “Sketch recognition with deep visual-sequential fusion model,” in Proc. the 25th ACM International Conference on Multimedia: 448-456, 2017.

[34] H. Zhao, M. Liu, M. Li, “Feature fusion and metric learning network for zero-shot sketch-based image retrieval,” Entropy, 25(3): 502, 2023.

[35] E. Askari, S. Motamed, “Computational model for image processing in the minds of people with visual agnosia using fuzzy cognitive map,” J. Inf. Syst. Telecommun. (JIST), 2(42): 102, 2023.

[36] M. Lin, Q. Chen, S. Yan, “Network in network,” arXiv preprint arXiv:1312.4400, 2013.

[37] F. Wang, L. Kang, Y. Li, “Sketch-based 3d shape retrieval using convolutional neural networks,” in Proc. the IEEE Conference on Computer Vision and Pattern Recognition: 1875-1883, 2015.

[38] K. Chatfield, K. Simonyan, A. Vedaldi, A. Zisserman, “Return of the devil in the details: Delving deep into convolutional nets,” arXiv preprint arXiv:1405.3531, 2014.

[39] Y. Qi, Y. Zh. Song, H. Zhang, J. Liu, “Sketch-based image retrieval via siamese convolutional neural network,” in Proc. 2016 IEEE International Conference on Image Processing (ICIP): 2460-2464, 2016.

[40] T. Bui, L. Ribeiro, M. Ponti, J. Collomosse, “Generalisation and sharing in triplet convnets for sketch based visual search,” arXiv preprint arXiv:1611.05301, 2016.

[41] M. Rohani, H. Farsi, S. Mohamadzadeh, “Deep multi-task convolutional neural networks for efficient classification of face attributes,” Int. J. Eng., 36(11): 2102-2111, 2023.

[42] P. Sangkloy, N. Burnell, C. Ham, J. Hays, “The sketchy database: learning to retrieve badly drawn bunnies,” ACM Trans. Graphics (TOG), 35(4): 1-12, 2016.

[43] T. Bui, L. Ribiro, M. Ponti, J. Collomosse, “Sketching out the details: Sketch-based image retrieval using convolutional neural networks with multi-stage regression,” Computers & Graphics, 71: 77-87, 2018.

[44] O. Tursun, S. Denman, S. Sridharan, E. Goan, C. Fookes, “An efficient framework for zero-shot sketch-based image retrieval,” Pattern Recognit., 126: 108528, 2022.

[45] T. Bui, L. Ribeiro, M. Ponti, J. Collomosse, “Compact descriptors for sketch-based image retrieval using a triplet loss convolutional neural network,” Comput. Vision Image Understanding, 164: 27-37, 2017.

[46] C. Bai, J. Chen, Q. Ma, P. Hao, Sh. Chen, “Cross-domain representation learning by domain-migration generative adversarial network for sketch based image retrieval,” J. Visual Commun. Image Represent., 71: 102835, 2020.

[47] A. P. R. G. G. Rajput, “Sketch based image retrieval in large databases using edge features,” Int. J. Recent Technol. Eng. (IJRTE), 08: 2277-3878, 2020.

[48] M. Eitz, J. Hays, M. Alexa, “How do humans sketch objects?,” ACM Trans. Graphics (TOG), 31(4): 1-10, 2012.

[49] N. Dalal, B. Triggs, “Histograms of oriented gradients for human detection,” in Proc. 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05), 1: 886-893, 2005.

[50] J. M. Saavedra, “Sketch based image retrieval using a soft computation of the histogram of edge local orientations (shelo),” in Proc. 2014 IEEE International Conference on Image Processing (ICIP): 2998-3002, 2014.

[51] D. G. Lowe, “Distinctive image features from scale-invariant keypoints,” Int. J. Comput. Vision, 60: 91-110, 2004.

LETTERS TO EDITOR

Journal of Electrical and Computer Engineering Innovations (JECEI) welcomes letters to the editor for the post-publication discussions and corrections which allows debate post publication on its site, through the Letters to Editor. Letters pertaining to manuscript published in JECEI should be sent to the editorial office of JECEI within three months of either online publication or before printed publication, except for critiques of original research. Following points are to be considering before sending the letters (comments) to the editor.

[1] Letters that include statements of statistics, facts, research, or theories should include appropriate references, although more than three are discouraged.

[2] Letters that are personal attacks on an author rather than thoughtful criticism of the author’s ideas will not be considered for publication.

[3] Letters can be no more than 300 words in length.

[4] Letter writers should include a statement at the beginning of the letter stating that it is being submitted either for publication or not.

[5] Anonymous letters will not be considered.

[6] Letter writers must include their city and state of residence or work.

[7] Letters will be edited for clarity and length.

Name *

Email Address *

Affiliation *

Comments *

Security Code *

Journal of Electrical and Computer Engineering Innovations (JECEI)

Hybrid Convolutional Neural Network with Domain adaptation for Sketch based Image Retrieval

References

References

Send comment about this article

Volume 12, Issue 2
July 2024
Pages 497-510

Hybrid Convolutional Neural Network with Domain adaptation for Sketch based Image Retrieval

References

References

Send comment about this article

Volume 12, Issue 2July 2024Pages 497-510

Volume 12, Issue 2
July 2024
Pages 497-510