Document Type : Original Research Paper

Authors

• J. Salimi Sartakhti
• A. Beirnvand
• M. Sarhadi

Department of Computer Engineering, University of Kashan, Kashan, Iran.

Abstract

Background and Objectives: Large Language Models have demonstrated ‎exceptional performance across various NLP tasks, especially when fine-tuned for ‎specific applications.‎‏ ‏Full fine-tuning of large language models requires extensive ‎computational resources, which are often unavailable in real-world settings. While ‎Low-Rank Adaptation (LoRA) has emerged as a promising solution to mitigate these ‎challenges, its potential remains largely untapped in multi-task scenarios. This ‎study addresses this gap by introducing a novel hybrid approach that combines ‎LoRA with an attention-based mechanism, enabling fine-tuning across tasks while ‎facilitating knowledge sharing to improve generalization and efficiency.‎‏ ‏‎ This study ‎aims to address this gap by introducing a novel hybrid fine-tuning approach using ‎LoRA for multi-task text classification, with a focus on inter-task knowledge sharing ‎to enhance overall model performance.‎
Methods: We proposed a hybrid fine-tuning method that utilizes LoRA to fine-tune ‎LLMs across multiple tasks simultaneously. By employing an attention mechanism, ‎this approach integrates outputs from various task-specific models, facilitating ‎cross-task knowledge sharing. The attention layer dynamically prioritizes relevant ‎information from different tasks, enabling the model to benefit from ‎complementary insights. ‎
Results: The hybrid fine-tuning approach demonstrated significant improvements ‎in accuracy across multiple text classification tasks. On different NLP tasks, the ‎model showed superior generalization and precision compared to conventional ‎single-task LoRA fine-tuning. Additionally, the model exhibited better scalability ‎and computational efficiency, as it required fewer resources to achieve comparable ‎or better performance. Cross-task knowledge sharing through the attention ‎mechanism was found to be a critical factor in achieving these performance gains.‎
Conclusion: The proposed hybrid fine-tuning method enhances the accuracy and ‎efficiency of LLMs in multi-task settings by enabling effective knowledge sharing ‎between tasks. This approach offers a scalable and resource-efficient solution for ‎real-world applications requiring multi-task learning, paving the way for more ‎robust and generalized NLP models. ‎

Keywords

• Large Language Model
• ‎Fine-Tuning
• ‎LoRA
• Knowledge ‎Sharing
• Attention ‎Mechanism

Main Subjects

• Text Classification

Open Access

This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this license, visit: http://creativecommons.org/licenses/by/4.0/

 

Publisher’s Note

JECEI Publisher remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

 

Publisher

Shahid Rajaee Teacher Training University