Chunkwise Lora: Adaptive Sequence Partitioning For Memory-efficient Low-rank Adaptation And Accelerated LLM Inference

Abstract

Recent advances in low-rank adaptation (LoRA) have enabled efficient fine-tuning of large language models (LLMs) with minimal additional parameters. However, existing LoRA methods apply static rank configurations uniformly across all input tokens, ignoring variation in token complexity and computational requirements. In this work, we propose ChunkWise LoRA, a dynamic and adaptive approach that par