DICE: Disentangled Instance-Class KnowlEdge Prompt Tuning via SAE for Vision-Language Models

Citations

SCOPUS

0

초록

CLIP, known for its strong zero-shot generalization, has received significant attention in prompt tuning for its ability to adapt effectively to new tasks in few-shot settings. Recent studies have employed descriptions generated by Large Language Models (LLMs) as predefined prompts to obtain class-level semantics. However, because these predefined descriptions are not directly grounded in visual information, they often misalign with an image’s visual semantics. They tend to generate generic class-level descriptions rather than instance-specific ones. As a result, LLM-based prompts are structurally biased toward class-level semantics, making it difficult to capture and reflect instance-level cues, which in turn limits generalization in few-shot settings where both levels of knowledge are crucial. To address this limitation, we propose Disentangled Instance-Class knowlEdge (DICE) prompt tuning, a framework that restructures LLM-derived class-level priors to capture instance-level semantics. Specifically, DICE decomposes LLM-derived priors into instance-level components using a Sparse Autoencoder (SAE), which selects instance-specific concept vectors. These concept vectors are then fused with class embeddings to form enriched representations. This synergy preserves class-level semantic coherence while capturing instance-level details, improving generalization to unseen classes. Our approach achieves competitive performance across 11 few-shot recognition benchmarks, while additionally offering a plug-and-play solution that enhances interpretability through SAE. © The Author(s), under exclusive license to Springer Nature Switzerland AG 2026.

키워드

Plug-and-play; Prompt learning; Sparse autoencoder
제목
DICE: Disentangled Instance-Class KnowlEdge Prompt Tuning via SAE for Vision-Language Models
저자
Lee, Daeun; Jang, Seungwoo; Kim, Kwangsu
DOI
10.1007/978-3-032-36846-1_35
발행일
2026
유형
Conference paper
저널명
Lecture Notes in Computer Science
권
17003 LNCS
페이지
593 ~ 611