Steering Into New Embedding Spaces: Analyzing Cross-lingual Alignment Induced By Model Interventions In Multilingual Language Models

Abstract

Aligned representations across languages is a desired property in multilingual large language models (mLLMs), as alignment can improve performance in cross-lingual tasks. Typically alignment requires fine-tuning a model, which is computationally expensive, and sizable language data, which often may not be available. A data-efficient alternative to fine-tuning is model interventions -- a method for manipulating model activations to steer generation into the desired direction. We analyze the effect of a popular intervention (finding experts) on the alignment of cross-lingual representations in mLLMs. We identify the neurons to manipulate for a given language and introspect the embedding space of mLLMs pre- and post-manipulation. We show that modifying the mLLM's activations changes its embedding space such that cross-lingual alignment is enhanced. Further, we show that the changes to the embedding space translate into improved downstream performance on retrieval tasks, with up to 2x impr

Stats

• citations2
• S2 citations—
• github stars0
• HF likes0
• heat score3.58
• arxiv keysundar2025steering

Related papers

• CLEAR: Cross-lingual Enhancement In Alignment Via Reverse-training (2026)0.78
• What Drives Cross-lingual Ranking? Retrieval Approaches With Multilingual Language Models (2025)0.00
• Transforming Llms Into Cross-modal And Cross-lingual Retrieval Systems (2024)4.52
• Guiding Cross-modal Representations With MLLM Priors Via Preference Alignment (2025)0.00
• Blind To Position, Biased In Language: Probing Mid-layer Representational Bias In Vision-language Encoders For Zero-shot Language-grounded Spatial Understanding (2025)0.00
• Hyperdimensional Cross-modal Alignment Of Frozen Language And Image Models For Efficient Image Captioning (2026)0.00
• Context-adaptive Multi-prompt Embedding With Large Language Models For Vision-language Alignment (2025)0.00
• Multiway-adapater: Adapting Large-scale Multi-modal Models For Scalable Image-text Retrieval (2023)0.00