Abstract
Vision Language Models (VLMs) have demonstrated impressive capabilities in integrating visual and textual information for understanding and reasoning, but remain highly vulnerable to adversarial attacks. While activation steering has emerged as a promising defence, existing approaches often rely on task-specific contrastive prompts to extract harmful directions, which exhibit suboptimal performance and can degrade visual grounding performance. To address these limitations, we propose \textit\{Sequence-Level Preference Optimization\} for VLM (\textit\{SPO-VLM\}), a novel two-stage defense framework that combines activation-level intervention with policy-level optimization to enhance model robustness. In \textit\{Stage I\}, we compute adaptive layer-specific steering vectors from diverse data sources, enabling generalized suppression of harmful behaviors during inference. In \textit\{Stage II\}, we refine these steering vectors through a sequence-level preference optimization process. Th