Generalizability Of Mixture Of Domain-specific Adapters From The Lens Of Signed Weight Directions And Its Application To Effective Model Pruning

Abstract

Several parameter-efficient fine-tuning methods based on adapters have been proposed as a streamlined approach to incorporate not only a single specialized knowledge into existing Pre-Trained Language Models (PLMs) but also multiple of them at once. Recent works such as AdapterSoup propose to mix not all but only a selective sub-set of domain-specific adapters during inference via model weight ave