Dual Grained Quantization: Efficient Fine-grained Quantization For LLM

Abstract

Large Language Models (LLMs) pose significant hardware challenges related to memory requirements and computational ability. There are two mainstream quantization schemes for LLMs: coarse-grained ($\text{{}e.g.,\}$ channel-wise) quantization and fine-grained ($\text{{}e.g.,\}$ group-wise) quantization. Fine-grained quantization has smaller quantization loss, consequently achieving superior pe