Hypergraph Vision Transformers: Images Are More Than Nodes, More Than Edges

Abstract

Recent advancements in computer vision have highlighted the scalability of Vision Transformers (ViTs) across various tasks, yet challenges remain in balancing adaptability, computational efficiency, and the ability to model higher-order relationships. Vision Graph Neural Networks (ViGs) offer an alternative by leveraging graph-based methodologies but are hindered by the computational bottlenecks of clustering algorithms used for edge generation. To address these issues, we propose the Hypergraph Vision Transformer (HgVT), which incorporates a hierarchical bipartite hypergraph structure into the vision transformer framework to capture higher-order semantic relationships while maintaining computational efficiency. HgVT leverages population and diversity regularization for dynamic hypergraph construction without clustering, and expert edge pooling to enhance semantic extraction and facilitate graph-based image retrieval. Empirical results demonstrate that HgVT achieves strong performance

Stats

• citations3
• S2 citations—
• github stars0
• HF likes0
• heat score4.52
• arxiv keyfixelle2025hypergraph

Related papers

• Boosting Vision Transformers For Image Retrieval (2022)15.28
• Vision Transformer Hashing For Image Retrieval (2021)17.01
• VITR: Augmenting Vision Transformers With Relation-focused Learning For Cross-modal Information Retrieval (2023)4.52
• Analyzing Local Representations Of Self-supervised Vision Transformers (2023)0.00
• Transhash: Transformer-based Hamming Hashing For Efficient Image Retrieval (2021)13.44
• Decomposing And Interpreting Image Representations Via Text In Vits Beyond CLIP (2024)7.28
• Graph Neural Networks In Vision-language Image Understanding: A Survey (2023)10.35
• Training Vision Transformers For Image Retrieval (2021)0.00