Abstract
Vision transformers face significant computational overheads in high-resolution dense prediction due to the quadratic complexity of self-attention. Linear attention offers efficiency but sacrifices local context modeling. We propose \textbf{HSMLA (Hierarchical Softmax Multi-scale Linear Attention)}, which combines ReLU-based linear attention for global context, selective softmax refinement for critical local features, and multi-scale token representations via depthwise convolutions. HSMLA achieves superior accuracy-efficiency trade-offs: up to 4.2× inference-time speedup across dense prediction tasks, 87.3% Dice with 3.2× speedup on CT organ segmentation, and 94.2% AUC with 4.1× speedup on pathology WSI.