Learning Clustering-based Prototypes for Compositional Zero-Shot Learning

We thank reviewer Q4uv for the valuable time and constructive feedback. We provide point-to-point response below.

---

Q1: Lack of discussion and citation of some related works.

A1: Thank you for pointing this out! To better address your concern and avoid potential misunderstandings, we expand and further discuss the related work [A, B, ref1, ref2] on prototype learning for ZSL/CZSL fields in L131-134 and the difference between our local-aware prototype assignment module and previous work [C, ref3] in L251-255 to enhance the contextual relevance and depth of our review, as follows:

L131-134: In the (compositional) zero-shot learning field, these works [A, B, ref1, ref2] extensively explore prototype learning to enhance feature representation. However, they typically model each class with only one prototype and their prototypes are often flexible parameters, ignoring intra-class diversity.

L251-255: Different from the classical formulation in [C, ref3], i.e., Optimal Transport with entropic constraints, our local-aware prototype assignment can produce superior assignments by fully considering the intrinsic coherence structure of attribute feature distribution, i.e., intra-distribution coherence.

[A] ProtoCLIP: Prototypical Contrastive Language Image Pretraining. TNNLS 2023.

[B] Dual progressive prototype network for generalized zero-shot learning. NeurIPS 2021.

[C] Rethinking semantic segmentation: A prototype view. CVPR 2022.

[ref1] Visual-augmented dynamic semantic prototype for generative zero-shot learning. CVPR 2024.

[ref2] Attribute prototype network for zero-shot learning. NeurIPS 2020.

[ref3] Prompt learning with optimal transport for vision-language model. arXiv 2022.

---

Q2: Novelty of ClusPro.

A2: Sorry for this confusion.

First, although ClusPro and [C] both reformulate the prototype assignment problem as an optimal transport problem, as proposed by [ref4], there are notable differences in their implementations.

[C] utilizes the classical formulation, i.e., Optimal Transport with entropic constraints, defined as $\Omega({L}^{a\top})=-\sum l^a_{n,k}\log(l^a_{n,k})$, which typically yields denser transportation plans. The entropic regularization term enables [C] to approximate the optimal transport problem by the Sinkhorn algorithm.

Conversely, ClusPro adopts another formulation with stronger priors, i.e., Optimal Transport with a structure-aware constraint, defined as $\Omega({L}^{a\top})=-\langle{S}^{a},({L}^{a}\odot{Q}^{a}{)^\top}{({L}^{a}\odot{Q}^{a})} \rangle$, which promotes intra-distribution coherence by encouraging local consistency (see L254-257). Given that this objective function is non-convex, a Generalized Conditional Gradient (GCG) algorithm [ref5] is utilized to effectively solve the problem. Moreover, as illustrated in Table 4d (L446-450), it can be observed that our local-aware prototype assignment demonstrates superior performance compared to classical Optimal Transport [C].

Second, [A] only employs instance-level contrastive learning, which aligns with the standard CLIP framework. For prototype, it adopts K-means clustering akin to DeepCluster, where cluster assignments serve as pseudo-labels, and a cross-entropy objective is optimized to learn robust representations.

In contrast, ClusPro implements prototype-level contrastive learning (PCL) to shape well-structured embedding space. It optimizes an infoNCE objective to maximize the prototype assignment posterior probability.

As outlined in L303-311, our prototype-based contrastive learning offers two key advantages over previous methods like [A]:

1. It constructs positive and negative pairs using clustering-based prototypes, thereby avoiding the need for sophisticated negative sampling strategies.

2. It leverages pre-constructed prototypes for contrast computation, thus avoiding additional computational and storage costs.

Moreover, Prototype-anchored Decorrelation Learning is also devised to complement prototype-based contrastive learning, working together to promote intra-primitive separation and inter-primitive decorrelation.

In summary, motivated by the objective to "model each primitive with a set of prototypes", we propose a framework for CZSL that includes prototype construction and prototype-based self-supervised learning to achieve intra-primitive separation and inter-primitive correlation in the embedding space. Thus, our approach stands out from previous works with clear distinctions in both the core motivation (acknowledged by the reviewers siAz and 1rZT) and techniques (acknowledged by the reviewer siAz) employed.

[ref4] Self-labelling via simultaneous clustering and representation learning. ICLR 2020.

[ref5] Generalized conditional gradient: analysis of convergence and applications. arXiv 2015.

Q3: Writing of Sec. 3.3 could be improved.

Our apologies. We make detailed clarification below and revise Section 3.3 to improve clarity.

---

Q3.1: Clarify how the K prototypes are obtained.

A3.1: Sorry for this confusion. In fact, we do not obtain the prototypes through the off-the-shelf K-means algorithm [ref6], which is based on data reconstruction; instead, we cast the prototype assignments as an optimal transport problem, following the learning formulation proposed by [ref7]. Compared to methods such as K-means, the new formulation [ref7] is more principled and allows to more easily demonstrate properties of the method such as convergence.

As detailed in L235-268, given attribute features ${F}^{a} \in \mathbb{R}^{D\times N^{a}}$, our goal is to seek the optimal assignment matrix ${L}^{a} \in \mathbb{R}^{K\times N^{a}}$, thereby deriving the prototypes ${P}^{a}={F}^{a}{{L}^{a}}^\top \in \mathbb{R}^{D\times K}$. The clustering within each attribute can be achieved by the optimization of the assignment matrix ${L}^{a}$:

$\min\_{{L}^{a}\in \mathcal{L}^{a}}\langle{L}^{a\top},-\log{Q}^{a}\rangle + \kappa\Omega({L}^{a\top})$.

The relevant contents in Sec. 3.3 in L258-260 have been modified to reduce confusion and improve readability. Thanks.

L258-260: Unlike offline clustering [73, 74] requiring multiple passes over the entire dataset for feature computation, we cast local-aware prototype assignment as an optimal transport problem, so as to scale our algorithm to massive data by online clustering.

[ref6] Deep clustering for unsupervised learning of visual features. ECCV 2018.

[ref7] Self-labelling via simultaneous clustering and representation learning. ICLR 2020.

---

Q3.2: Clarify the variable notations.

A3.2: Thank you so much for your careful review! We make an obvious comment in the revised version for the variable notations to improve clarity, e.g., ${p}^{a}\_k$, ${f}^{a}\_n$ and ${f}\_n$.

1. ${p}^{a}\_k$: "${p}^{a}\_k$ is $k$-th prototypes of attribute $a$"
2. ${f}^{a}\_n$: "${f}^{a}\_n$ is $n$-th attribute features of attribute $a$"
3. ${f}\_n$: "${f}\_n$ is $n$-th primitive features"

---

Q4: Use $K(M+N-1)$ irrelevant prototypes from other primitives as negative samples.

A4: Thanks for your thoughtful comments on our manuscript. As outlined in L293-296, our prototype-based contrastive learning is designed to capture intra-primitive separation. To this end, it is pivotal to ensure the discriminativeness of prototypes within each primitive, i.e., to enforce each feature to be similar with its assigned prototype $p_+$ and dissimilar with other $K-1$ irrelevant prototypes. To increase the scale of negative samples, we add $K(M+N-1)$ irrelevant prototypes from other primitives, ultimately resulting in a total of $K(M+N)-1$ irrelevant prototypes $\mathcal{P}_{-}$. Additionally, using only $K(M+N-1)$ irrelevant prototypes from other primitives as negative samples can ensure inter-primitive separation to some extent, but it does not guarantee intra-primitive separation. We incorporate the discussions into Sec. 3.4. Thanks.

---

Finally, we would like to sincerely thank you for your careful review and valuable comments, which really help us to improve this work. We hope we addressed your concerns. Please let us know if you'd like any further information.

Q3: Non-CLIP version of ClusPro vs. Non-CLIP methods.

A3: Thanks for your comment. We would like to respectfully clarify that we had provided comparison results with non-CLIP methods in Table 7 and 8. It is worth noting that some CLIP-based methods (i.e., coop, CSP) perform worse than non-CLIP methods under certain settings. For example, CSP performs worse than non-CLIP method kG-SP on UT-zappos under OW setting.

In addition, for the sake of fair comparison with non-CLIP SOTAs, we also present results that do not utilize CLIP model. Please see A1.2 for experimental results. The results are added in Appendix Sec.D and Table 9. Thanks.

---

Q4: The diversities of primitive (i.e., attribute and object).

A4: Sorry for this confusion. In fact, we consider both the diversity of attributes and objects:

• As stated in L228-230: Notably, we present the online primitive-wise clustering process within both the attribute and object embedding spaces. For clarity, we only explain the prototype construction in the attribute branch, while the object branch follows the same process.

• As detailed in L282-283: we construct a set of prototypes for each attribute, and each object, to represent diverse sub-primitive patterns.

But, for simplicity, we primarily highlight attribute diversity in our exposition. To improve clarity, the following updates have been made to the manuscript:

L228-230: Notably, we present the online primitive-wise clustering process within both the attribute and object embedding spaces, so as to well represent rich and diverse patterns within each primitive.

---

Finally, we would like to sincerely thank you for your careful review and valuable comments. We have revised our manuscript with discussions and experimental results according to your review. It is welcome if there are any further questions or suggestions.

We thank reviewer Aocv for the valuable time and constructive feedback. We provide point-to-point response below.

---

Q1.1: Clarify the definition of Compositional Zero-shot Learning (CZSL).

A1.1: Sorry for this confusion. Actually, most current researches [ref1, ref2, ref3, ref4, ref5] in compositional zero-shot learning (CZSL), including our work (see L150-157), report the performance in the Generalized Zero-shot Learning [ref6] for both the closed-world and the open-world settings, where test samples include both seen and unseen compositions.

Given that CLIP is trained on millions of text-image pairs sourced from the web, it is hard to know whether CLIP has been exposed to certain unseen compositions during its training. However, we noted that the CLIP paper performed a data overlap analysis (Sec. 5 of [ref7]) w.r.t. 35 datasets on zero-shot transfer experiments. Finally, they find there is a median overlap of 2.2% and an average overlap of 3.2%. According to [ref7], if overlap exists, overall accuracy shift is less than 0.1% with the largest shift as 0.6%.

In light of above discussion, we argue that CLIP-based model still aligns with the definition of CZSL, even though CLIP might may be exposed to a small number of unseen compositions during pre-training. Besides, given the prevalence of CLIP-based approaches in the CZSL, it seems a little unfair that treating this as a specific limitation of our approach.

Moreover, the experiment results in Table 7 and 8 shows that CLIP without any fine-tuning underperforms task-specific architectures (i.e., non-CLIP methods) in the CZSL task, even though it has been pre-trained on vastly more data. We suspect that this is because contrastive pre-training on Web image captions does not require the model to learn precise compositional semantics. Thus, it is non-trivial to explore how to explicitly enforce compositional semantics by fine-tuning CLIP.

[ref1] Decomposed soft prompt guided fusion enhancing for compositional zero-shot learning. CVPR 2023.

[ref2] Learning to compose soft prompts for compositional zero-shot learning. ICLR 2023.

[ref3] Context-based and Diversity-driven Specificity in Compositional Zero-Shot Learning. CVPR 2024.

[ref4] Prompting language-informed distribution for compositional zero-shot learning. ECCV 2024.

[ref5] Retrieval-Augmented Primitive Representations for Compositional Zero-Shot Learning. AAAI 2024.

[ref6] Open world compositional zero-shot learning. CVPR 2021.

[ref7] Learning Transferable Visual Models From Natural Language Supervision. ICML 2021.

---

Q1.2: Evaluation results for models pre-trained on datasets with no overlap.

A1.2: To further highlight the robustness and superiority of our approach, we additionally present results on CW setting that utilizing ViT-B backbone pre-trained with DINO [ref8] on ImageNet in a self-supervised manner as ADE [ref9] instead of CLIP model, as follows:

As seen, our model also demonstrates better performance than the baseline and SOTA non-CLIP methods (e.g., CANet, CGE, OADis and ADE ),when using the backbone pre-trained on datasets with no overlap. We incorporate the above results into the Appendix Sec.D and Table 9. Thanks.

[ref8] Emerging properties in self-supervised vision transformers. ICCV 2021.

[ref9] Learning Attention as Disentangler for Compositional Zero-shot Learning. CVPR 2023.

---

Q2: The usage of CLIP contradicts the definition of CZSL.

A2: Thanks for your careful review. Please see A1.1 for detailed clarification. Thus, we safely conclude that CLIP-based methods still align with the definition of CZSL. Thanks.

I appreciate the response from authors. It addresses my partial concerns. If authors can resolve the remained issues, I am willing to increase my score.

1. Although the visual encoder is replaced, the text encoder still is from CLIP. It still might bring potential risk for composition leaking. Could you provide the results that use other text encoder, even just on one dataset, before the dicussion close?

2. Although CLIP provides a data overlap analysis, but it violated the zero-shot learning setting factually. To prevent future researchers from having similar issues, could you provide a highlighted brief claim (e.g. claim in footnote) about the overlap discussion and link the detailed discussion and results in appendix？

We thank reviewer 1rZT for the valuable time and constructive feedback. We provide point-to-point response below.

---

Q1: Number of prototypes for each primitive.

A1: Thanks for pointing out this promising direction. Intra-primitive variability indeed varies across primitive. Considering this, we conduct the experiment by setting the number $K$ of prototypes based on the proportion of training samples for each primitive. In UT-Zappos dataset, training samples per primitive range from 0.2% to over 20%. Thus, we assign $K=1$ to the primitive with 0.2-5% training samples, $K=2$ to the primitive with 5-10% training samples, $K=3$ to the primitive with 10-15% training samples, $K=4$ to the primitive with 15-20% training samples, and $K=5$ to the primitive with over 20% training samples. As seen, this strategy results in slightly better performance than setting a fixed value for all the primitives. However, in our current version, we directly set $K=5$ for all the primitives for simplicity. This experiment is added in Appendix Sec.D and Table 11. Thanks.

Also, as you noticed, it is interesting to explore how to automatically determine $K$ for different primitives. We found that there are some clustering techniques (like [ref1]), which can automatically determine the number of cluster centers. However, after running their code, we find that their algorithms are complex and time-consuming. We consider this as the basis of our future work. The discussions are incorporated into the Appendix Sec.G.

[ref1] Deepdpm: Deep clustering with an unknown number of clusters. CVPR 2022.

Q2: Resource limitations on large datasets.

A2: Sorry for this confusion. Unlike offline clustering [ref2] requiring recomputing sub-primitive prototypes over the entire dataset after each batch, our algorithm adopts online clustering approach with momentum updates, where prototypes are dynamically updated using the embeddings within the current batch (stated in L267-271). This design ensures our approach is scalable to large datasets, with minimal and consistent resource overhead regardless of dataset size.

Also, we would like to clarify that we had provided the resource usage analysis below about UT-Zappos in Table 10. As seen, ClusPro hardly incurs any additional GPU memory overhead or trainable parameters, with only a slight training delay due to additional clustering loops. Note that the memory and clustering are discarded after training, and do not introduce extra overheads in inference.

[ref2] Deep clustering for unsupervised learning of visual features. ECCV 2018.

---

We sincerely appreciate your thorough review and invaluable feedback, which have significantly contributed to the improvement of our work. We hope we have addressed all of your concerns. Please let us know if you require any additional information.

Dear Reviewer,

We sincerely appreciate the time and effort you have dedicated to reviewing our submission. We have submitted our rebuttal and would like to follow up to inquire whether our responses have sufficiently addressed your concerns.

Please let us know if you have any remaining questions or require additional clarification. We value your feedback and are eager to ensure our work meets the highest standards.

Authors

We thank reviewer siAz for the valuable time and constructive feedback. We provide point-to-point response below.

---

Q1: Clarification of the whole pipeline.

A1: Thanks for your suggestion. We have further provided the complete pipeline below of our algorithm in Appendix Sec.B to make readers fully understand our framework. Thank you.

Fig. 2 presents the architecture of our ClusPro. It takes a batch of images and all the semantic labels (i.e., attributes, objects, and compositions) as input. ClusPro first utilizes the visual encoder of CLIP [38] along with attribute and object adapters to obtain attribute features $F^a$, object features $F^o$, and composition features $F$ (Eq. 1). Besides, ClusPro constructs attribute, object, and composition prompt representations (Eq. 2) via a soft learnable prompt strategy [14] based on pre-given semantic labels. $\cdots\cdots$ Finally, we assemble the three-path classification loss $\mathcal{L}^{\text{BAS}}$ and our proposed prototype-based loss constraints (i.e., $\mathcal{L}^{\text{PCL}}$ and $\mathcal{L}^{\text{PDL}}$) as our final learning objective.

---

Q2: Comparison with the same backbone.

A2: Apologies for missing the backbone information; we have updated it in Table 1 and 2.

As seen, all the compared methods including our approach employ the same ViT-L backbone. Based on the above, we believe we render a fair comparison to existing work. Thanks.

---

Q3: More details about the baseline model.

A3: Following your suggestion, we provide more details below about the baseline model in Appendix Sec.C for clarity, as:

Visual Feature Extraction. Following [14, 15, 23], we adopt the visual encoder ${\phi}^{\text{vis}}$ of CLIP [38] to splits the input image $X\in\mathbb{R}^{H \times W \times 3}$ into $N_{p}=HW/P$ patches, where $P$ is the resolution of each patch. $\cdots\cdots$ We adopt attribute adapter ${h}^{a}$ and object adapter ${h}^{o}$, each implemented as a separate MLP, to project ${f}^{c}$ into the discriminative attribute feature ${f}^{a}$ and object feature ${f}^{o}$, respectively. Prompt Feature Extraction. We follow existing CZSL [18, 14] to employ an independent prompt prefix for each branch. $\cdots\cdots$ Then these prompts are then fed into the frozen text encoder of CLIP [38] to obtain prompt features. Training Loss $\mathcal{L}^{\text{BAS}}$. Following previous CZSL approaches [14, 15, 23], the parameters $\theta$ of the baseline model are learned by minimizing the three-path classification loss (Eq.14) on the training dataset. Note that we have omitted the weight decay in Eq.14 for simplicity. We just follow [14] set the weight decay as 5e−5 for all our experiments. Feasibility Calibration for Open-World Setting. Following [17, 14], we adopt post-training feasibility calibration to filter out infeasible compositions that might be present in the open-world setting. The calibration relies on the assumption that similar objects tend to share similar attributes, while dissimilar objects are unlikely to exhibit shared attributes.$\cdots\cdots$

---

We appreciate again your thoughtful review and we hope we addressed your concerns. Please let us know if you'd like any further information.

Dear Reviewer,

We sincerely appreciate the time and effort you have dedicated to reviewing our submission. We have submitted our rebuttal and would like to follow up to inquire whether our responses have sufficiently addressed your concerns.

Please let us know if you have any remaining questions or require additional clarification. We value your feedback and are eager to ensure our work meets the highest standards.

Authors