Delta-lora: Fine-tuning High-rank Parameters With The Delta Of Low-rank Matrices

Abstract

In this paper, we present Delta-LoRA, which is a novel parameter-efficient approach to fine-tune large language models (LLMs). In contrast to LoRA and other low-rank adaptation methods such as AdaLoRA, Delta-LoRA not only updates the low-rank matrices $\text{bA}$ and $\text{bB}$, but also propagate the learning to the pre-trained weights $\text{bW}$ via updates utilizing the delta of the product of two low-ra