Thank you for your valuable feedback! We hope the clarifications below effectively address your concerns.

W1 (Embedding ZSLs with limited training data):

Thank you for your suggestion. We use the AttentionNet (Rebalanced Zero-shot Learning, TIP23) to compute the class-averaged Mean-Squared Error (MSE) for semantic prediction.
The experiments show that semantic errors on test samples from both unseen and seen classes do not increase significantly during training.
This suggests that spurious visual-semantic correlation might be a unique issue for generative ZSL methods.

At the very least, fewer training samples do not significantly deteriorate semantic prediction in embedding methods.
However, embedding methods still perform substantially worse, with their performance being up to 16.4% lower (AWA2 ZSL T1 accuracy) compared to ours.

W2 (Relationship between $\Delta_{adv}$ and performance):

Figure 2 in the original paper corresponds to the curves from VAEGAN. Our Figure 4 shows the effect of $\mathcal{L}_{mu}$ on the curve.

The $\Delta_{adv}$ is the critic score differences between training real samples and validating real samples. If $\Delta_{adv}$ is larger than 0, $D_{adv}$ thinks training real samples are more realistic than validating real samples, vice versa. So it can reflect the $D_{adv}$ is over-fitting or under-fitting. However, it is hard to use $\Delta_{adv}$ indicate the ZSL performance. The reasons are mainly from two points:

(a) In generative ZSL methods, the ZSL performance mainly relies on $G$ generation ability. If $G$ only can generate poor samples and $D_{adv}$ can easily distinguish fake and real samples, $D_{adv}$ has no need to remember training samples. Especially, in the beginning of training (e.g. the first 10 epoches), the ZSL performance is far lower than in the best performance epoch, while the $\Delta_{adv}$ is also low.

(b) In other way, even $\Delta_{adv}$ is very high, $D_{adv}$ still has a basic visual-semantic correlation for training samples, which guarantees a low bound of ZSL performance. Thus, when the training goes to late, although $\Delta_{adv}$ is increasing, the performance still does not drop significantly.

The results from our ablation study effectively shows that regularizing $\Delta_{adv}$ could improve ZSL performance. After our regularization term $L_{\text{mu}}$, $\Delta_{adv}$ is more close to the ideal situation: should always around 0 during training. It indicates that $D_{adv}$ is neither over-fitting nor under-fitting.

W3 (Experiment Fairness):

We apologize for any confusion and would like to clarify that our evaluation is fair. Below, we address this concern from two perspectives:

(1) Clarification of Results in Table 1 in paper. In Table 1, the results of all other ZSL generative methods use fine-tuned features. We have used the marker $\dagger$ to denote results obtained using our CE-based features. For the remaining methods, DSP and VADS use fine-tuned features from PSVMA, while CoOp+SHIP utilize their own fine-tuned features.
To avoid similar concerns, we have added the marker $\ddagger$ to denote methods that use other fine-tuned features.

(2) The Fine-tuning Step as an Indivisible Innovation.
Our method distinguishes itself by deeply integrating fine-tuning into the overall approach.
While previous methods (e.g., f-VAEGAN, TF-VAEGAN) provide results using fine-tuned features, they only replace non-fine-tuned features without exploring how to fine-tune or utilize them effectively in subsequent generation steps.

In contrast, we uncover a key insight: Fine-tuning with SC-loss introduces instance-level semantics.
This is a critical problem in recent ZSL research (e.g., VADS, DSP), as existing semantic labels are class-level and often misaligned with instances, leading to spurious visual-semantic correlations.
From domain adaptation studies, it is known that contrastive loss fine-tuned in source domains can capture low- and mid-level visual information (e.g., color).
Building on this, we hypothesize that it can also capture instance-level semantics.

To verify this, we provide a t-SNE comparison in Fig. 6, showing that SC-based representations have larger intra-class variation than CE-based features.
This indicates that SC-based representations reflect instance-level variation.
Further visualization in Fig. 7 reveals that SC-loss can separate the fox class into distinct sub-classes (e.g., red fox, white fox, and grey fox).
These findings strongly support our hypothesis.

Thus, we use SC-based representations as an additional condition to provide instance-level semantics and mitigate spurious visual-semantic correlations.

Dear Reviewer ZpvL, We appreciate your constructive feedback provided in your earlier positive review. It indeed inspired us to reflect and improve our paper. As the deadline draws close, could you please kindly let us know if our rebuttal may properly address your concerns? We eagerly look forward to any further thoughts you may wish to share.

We thank you for the time and effort you have invested in reviewing this submission and for your positive comments on our work. We hope the following clarifications effectively address your concerns.

W1 (Overall Presentation):

We have thoroughly refined our manuscript and uploaded the revised PDF, with updates highlighted in $\color{red}{red}$. Following the suggestions from you and reviewer KQOZ, we now provide a step-by-step explanation of our three key insights and their connection to the corresponding modules in our methodology.
In this revised version, readers can seamlessly transition from the baseline f-VAEGAN to our proposed ZeroDiff.
You can review these changes in the updated manuscript PDF and our summarized Common response 1. We hope these revisions enhance readability and that our clarifications effectively address your concerns.

W2&Q1 (More Comprehensive Ablation Study):

The ablation study is also a common consideration.

We have supplemented additional ablation studies in Common response 2 from two perspectives:

(1) Following your suggestion, we have added results on more compositions of modules and losses.
In addition to the previously mentioned $G+R+D_{\text{adv}}$ and $G+D_{\text{adv}}+D_{\text{diff}}+L_{\text{mu}}$, we have also included $G+R+D_{\text{diff}}$, $G+R+D_{\text{adv}}+D_{\text{diff}}$, and $G+R+D_{\text{adv}}+D_{\text{diff}}+L_{\text{mu}}$.
A noteworthy observation from these new compositions is that adding $R$ to support $G$, without using $D_{\text{rep}}$ to discriminate from the SC representation perspective, does not significantly improve performance.

(2) Additionally, we followed the suggestion from reviewer KQOZ to conduct an ablation study on DFG inputs.
The results demonstrate that all inputs are necessary, particularly the diffusion time $t$.

The completed ablation study has also been updated in the revised PDF.

Q2 (LMMs):

Thank you for your insightful comment. We acknowledge the impressive performance of GPT-series models and large multi-modal models (LMMs) on zero-shot learning (ZSL) tasks. However, specifically designed ZSL models remain relevant for several reasons:

2-1. ZSL models v.s. LMMs:

Existing LMMs including CLIP actually change the definition of zero-shot learning from traditional unseen classes to unseen datasets, which has been clearly discussed in Section 3.2 of the CLIP paper [1]. In other word, the ZSL ability of LMMs is zero-shot dataset learning, while our specifically designed ZSL models are still zero-shot class learning. We humbly believe that such difference is fundamental. LMMs suggest a large-scale pre-training hypothesis: only if the pre-trained data are sufficient enough, LMMs can generalize to any unseen datasets or tasks. We argue that this may have four potential issues.

(1) LMMs may not adhere to the ZSL setting. LMMs generally do not split classes, potentially violating the ZSL premise. Although CLIP [1] has provided a comprehensive data overlap analysis in Section 4 stating overlapped data presents minimum influences on the overall performance, such analysis is not rigorous. The reason is that this analysis is drawn upon the evaluations on every dataset individually. For example, in CLIP's Appendix 4, the analysis was made on a bird dataset, Birdsnap, merely from which overlapped data were detected. It however ignores the fact that other datasets also contain many bird data, e.g. CIFAR-100. This relaxed evaluation might violate the ZSL setting. In contrast, ZSL models explicitly adhere to this ZSL principle, ensuring a more strict evaluation on the zero-shot performance.

(2) Data used for training certain LMMs may be limited and/or biased [2]. This may compromise LMMs' zero-shot learning ability. In this case, we still need explore the methods that do not rely on the large-scale pre-training hypothesis, i.e. the specifically designed ZSL methods. In other words, we believe these two different paradigms may be complementary.

(3) LMMs might perform worse than ZSL models in many cases.

In fact, we compare the most basic LMM model CLIP [1] and its derivative CoOp and SHIP in Table 1 of our paper. CLIP performs worse even if it enjoys the huge parameters and training set. Such observations have also been verified in other ZSL works, e.g. ZSLViT [2], VADS [3]. Besides, ZSL models may perform better when unseen classes are truly novel or in domains requiring strict adherence to the ZSL constraints. Again, in many scenarios, there might be limited data for pre-training. A typical example can be seen in medical domain where existing LMMs work not as well as specifically designed ZSL methods [4].

(4) Resource-efficient.

Specifically designed ZSL models are often more resource-efficient, interpretable, and modular, making them suitable for fine-grained or domain-specific tasks, especially in resource-constrained environments.

2-2. Combining ZSL Models and LMMs:

We believe integrating ZSL models with LMMs could leverage the strengths of both approaches, achieving enhanced generalization and performance.

How LMMs can used to improve ZSL Models

There are a few ways that LMMs can be used for boosting the performance of ZSL models. On one hand, ZSL Models can take advantages of the powerful representation of LLMs (if the LMM pretraining follows the rigorous class splitting). On the other hand, many existing ZSL works can leverage language models to generate class information. For example, I2MVFormer [5] utilizes GPT-3 that is a fully language model to produce class documents to help ZSL recognition. Such a way does not only enjoy the powerful knowledge from LLM, but also guarantees that unseen class images are not leaked.

How ZSL Models can used to improve LMMs

Existing ZSL framework can be used to help LLMs to adapt on unseen datasets. For example, SHIP just utilized the generative framework f-VAEGAN to help CLIP adapting on many unseen datasets.

In summary, thanks for your suggestions. We are now working on the experiments using LLaMA3 on the test sets of all the three benchmarks, which is however a bit time-consuming. We will be happy to report these results in our revision once we have any progress.

[1] Learning transferable visual models from natural language supervision, ICML, 2021.

[2] Progressive Semantic-Guided Vision Transformer for Zero-Shot Learning, CVPR, 2024.

[3] Visual-Augmented Dynamic Semantic Prototype for Generative Zero-Shot Learning, CVPR, 2024.

[4] Generative multi-label zero-shot learning, IEEE TPAMI, 2023.

[5] I2mvformer: Large language model generated multi-view document supervision for zero-shot image classification, CVPR, 2023.

Thank you for your valuable feedback! We hope that the following clarifications effectively address your concerns.

W1&W2&Q2 (Overall Presentation, Key module motivations):

We apologize for any confusion caused by the complexity of our components and would like to clarify that our key insights are based on three simple ideas presented in Fig.1.

Following suggestions from you and reviewer FPAJ, we now provide a step-by-step explanation of our three key insights and their connections to the corresponding modules in our methodology. In this version, readers can seamlessly transition from the baseline f-VAEGAN to our proposed ZeroDiff.

We have thoroughly refined our manuscript and uploaded the revised PDF, with updates highlighted in $\color{red}{red}$. You can review these changes in the revised manuscript PDF and our summarized Common response 1.

We hope these revisions improve readability and effectively address your concerns.

W2 (CE v.s. SC):

Since SC-based representations exhibit larger intra-class variation compared to CE-based features (Fig. 6 and Fig. 7), they contain more instance-level semantics.
However, due to the compactness of CE-based features, classification using CE-based features generally outperforms classification using SC-based representations.
Therefore, utilizing SC-based representations to assist feature generation while classifying in CE space is the optimal approach, compared to using either feature/representation alone for classification.

Fig. 8 includes results from other SOTA methods using SC-based or CE-based features, further demonstrating that CE-based classification generally achieves better performance than SC-based classification.

W3&Q3 (More Ablation Study and Module Evaluation):

We have supplemented additional ablation studies in Common response 2 from two perspectives:

(1) Following the suggestion from reviewer FPAJ, we added results on more compositions of modules and losses.
In addition to the previously mentioned $G+R+D_{\text{adv}}$ and $G+D_{\text{adv}}+D_{\text{diff}}+L_{\text{mu}}$, we also included $G+R+D_{\text{diff}}$, $G+R+D_{\text{adv}}+D_{\text{diff}}$, and $G+R+D_{\text{adv}}+D_{\text{diff}}+L_{\text{mu}}$.
A new observation from these compositions is that adding $R$ to support $G$, without using $D_{\text{rep}}$ to discriminate from the SC representation perspective, does not significantly improve performance.

(2) Additionally, we followed your suggestion to conduct an ablation study on DFG inputs.
The results indicate that all inputs are necessary, particularly the diffusion time $t$.

The completed ablation study has been updated in the revised PDF.

W4 (Detailed explanation for Fig. 4):

We find that our SC-based representations exhibit better class discriminability compared to pre-defined semantics and the VADS method, which dynamically updates class-level semantics to instance-level semantics.
Although VADS can also mine instance-level semantics, it sometimes sacrifices class discriminability.

Specifically, as shown in Fig. 5 in the revised paper (moved from Fig. 4 due to page limitations), we randomly select 8 classes from the CUB dataset and visualize the heatmap of class prototype similarities. For the three class similarities marked by the red dash, the similarities from pre-defined class-level semantics are (0.84, 0.82, 0.85), while the similarities produced by VADS remain very close: (0.85, 0.85, 0.86).
In contrast, our SC-based representations make these hard classes more distinctive, with class similarities of (0.78, 0.89, 0.83).

Due to page limitations and to improve readability, the figure has been moved to the Appendix, while the key findings are retained in the main paper.

Dear Reviewer Kqoz, We appreciate your constructive feedback provided in your earlier positive review. It indeed inspired us to reflect and improve our paper. As the deadline draws close, could you please kindly let us know if our rebuttal may properly address your concerns? We eagerly look forward to any further thoughts you may wish to share.

Common consideration2: Ablation study

Our ablation study is also a common consideration. We have supplemented additional ablation studies from two perspectives. For convenience, we also provide the results and analysis with the openreview fashion here.

(1) Following the suggestion from reviewer FPAJ, we have supplemented results concerning additional compositions of modules and losses, including $G+R+D_{\text{adv}}$, $G+D_{\text{adv}}+D_{\text{diff}}+L_{mu}$, $G+R+D_{\text{diff}}$, $G+R+D_{\text{adv}}+D_{\text{diff}}$, and $G+R+D_{\text{adv}}+D_{\text{diff}}+L_{\text{mu}}$.

From the results, we can draw three key observations:

(i) Comparing pairs (b,c), (g,h), and (i,j), we observe that $D_{\text{diff}}$ and $L_{\text{mu}}$ consistently improve performance across the three datasets.

(ii) (additional) Comparing the pairs (a,e) and (b,g), we find that adding $R$ to support $G$, without using $D_{\text{rep}}$ to discriminate from the SC representation view, does not significantly improve performance.

(iii) However, once both $R$ and $D_{\text{rep}}$ are used, the performance improvements become substantial, as seen in pairs (g,i) and (h,j). This indicates that $G$ benefits significantly from the inclusion of both $R$ and $D_{\text{rep}}$ in many cases. It also supports our motivation that SC-based representations can capture instance-level semantics to aid feature generation.