상세 보기
DICE: Disentangled Instance-Class KnowlEdge Prompt Tuning via SAE for Vision-Language Models
- Lee, Daeun;
- Jang, Seungwoo;
- Kim, Kwangsu
SCOPUS
0초록
CLIP, known for its strong zero-shot generalization, has received significant attention in prompt tuning for its ability to adapt effectively to new tasks in few-shot settings. Recent studies have employed descriptions generated by Large Language Models (LLMs) as predefined prompts to obtain class-level semantics. However, because these predefined descriptions are not directly grounded in visual information, they often misalign with an image’s visual semantics. They tend to generate generic class-level descriptions rather than instance-specific ones. As a result, LLM-based prompts are structurally biased toward class-level semantics, making it difficult to capture and reflect instance-level cues, which in turn limits generalization in few-shot settings where both levels of knowledge are crucial. To address this limitation, we propose Disentangled Instance-Class knowlEdge (DICE) prompt tuning, a framework that restructures LLM-derived class-level priors to capture instance-level semantics. Specifically, DICE decomposes LLM-derived priors into instance-level components using a Sparse Autoencoder (SAE), which selects instance-specific concept vectors. These concept vectors are then fused with class embeddings to form enriched representations. This synergy preserves class-level semantic coherence while capturing instance-level details, improving generalization to unseen classes. Our approach achieves competitive performance across 11 few-shot recognition benchmarks, while additionally offering a plug-and-play solution that enhances interpretability through SAE. © The Author(s), under exclusive license to Springer Nature Switzerland AG 2026.
키워드
- 제목
- DICE: Disentangled Instance-Class KnowlEdge Prompt Tuning via SAE for Vision-Language Models
- 저자
- Lee, Daeun; Jang, Seungwoo; Kim, Kwangsu
- 발행일
- 2026
- 유형
- Conference paper
- 권
- 17003 LNCS
- 페이지
- 593 ~ 611
- 언어
- ENG
- 출판사
- Springer Science and Business Media Deutschland GmbH
- 발행국가
- 미국
- 분량
- 19 페이지
- ISSN
- E 1611-3349
P 0302-9743