Sharelora: Parameter Efficient And Robust Large Language Model Fine-tuning Via Shared Low-rank Adaptation

Abstract

In this paper, we introduce \textbf\{Share\}d \textbf\{Lo\}w \textbf\{R\}ank \textbf\{A\}daptation (ShareLoRA), a Large Language Model (LLM) fine-tuning technique that balances parameter efficiency, adaptability, and robustness without compromising performance. By strategically sharing the low-rank weight matrices across different layers, ShareLoRA achieves 44% to 96% reduction in trainable parame