Memory-efficient cross-modal attention for RGB-X segmentation and crowd counting

Abstract

In multimodal visual understanding, fusing RGB images with additional modalities like depth or thermal data is essential for improving both accuracy and robustness. However, traditional approaches often rely on task-specific architectures that are difficult to generalize across different multimodal scenarios. To address this limitation, we propose the Cross-modal Spatio-Channel Attention (CSCA) module, designed to flexibly integrate diverse modalities into various model architectures while enhancing performance. CSCA employs spatial attention to capture interactions between modalities effectively, improving model adaptability. Additionally, we introduce a patch-based cross-modal interaction mechanism that optimizes the processing of spatial and channel features, reducing memory overhead while preserving critical spatial information. These refinements significantly simplify cross-modal interactions, increasing computational efficiency. Extensive experiments demonstrate that CSCA generalizes well across various multimodal combinations, achieving promising performance in crowd counting and image segmentation tasks, particularly in RGB-Depth, RGB-Thermal, and RGB-Polarization scenarios. Our approach provides a scalable and efficient solution for multimodal integration, with the potential for broader applications in future work. © 2025 Elsevier Ltd