BEFT: Bias-Efficient Fine-Tuning of Language Models in Low-Data Regimes

Abstract

Fine-tuning the bias terms of large language models (LLMs) has the potential to achieve unprecedented parameter efficiency while maintaining competitive performance, particularly in low-data regimes. However, the link between fine-tuning different bias terms (i.e., ${\mathbf{b}}_q$, ${\mathbf{b}}_k$, and ${\mathbf{b}}_v$ in the query, key, or value projections) and downstream performance remains largely unclear to date. In this paper, we investigate the link between fine-tuning ${\mathbf{b}}_q$, ${\mathbf{b}}_k$, and ${\mathbf{b}}_v$ with the performance of the downstream task. Our key finding is that directly fine-tuning ${\mathbf{b}}_v$ generally leads to higher downstream performance in low-data regimes, in comparison to ${\mathbf{b}}_q$ and ${\mathbf{b}}_k$. We extensively evaluate this unique property across a wide range of LLMs spanning encoder-only and decoder-only architectures up to 6.7B parameters (including bias-free LLMs). Our results provide strong evidence for the effectiveness of directly fine-tuning ${\mathbf{b}}_v$ across various downstream tasks. The implementation code is available at https://github.com/whubaichuan/BEFT.