GenZSL: Generative Zero-Shot Learning Via Inductive Variational Autoencoder

Response: Thank you for the comprehensive reviews and detailed comments! We are very happy to help to address your concerns!

Q1: If there are not similar classes for the unseen classses in the seen class set, the method may out of work?

A1: Thank you for this constructive comment. In response to the Q6 of Reviewer eZyG, if we randomly sample the seen class samples to synthesize unseen class samples, the performances of GenZSL may drop slightly, i.e., CUB (acc: 63.3%$\rightarrow$62.5%; H: 57.4%$\rightarrow$55.9%) and AWA2 (acc: 92.2%$\rightarrow$91.1%; H: 87.4%$\rightarrow$85.3%). These results show that GenZSL may not heavily rely on the similar samples from seen classes for synthesizing unseen class samples, and the similar classes for unseen classes in the seen class set can further improve GenZSL.

Q2: In Sec 3, the authors state that the refinement of text embeddings preserves the semantic relationships between classes. Is there any theoretical evidence or empirical analysis to validate this statement?

A2: As shown in Fig. 3 and Fig. 8, the qualitative results show that the high similarities of two classes, they still keep relatively high similarities after refinement using CDP. We statistic the similarities between class semantic vectors of pre/post-CDP, and we find that they keep similar class relationships. We will further highlight this in Sec. 3.1.

Q3: Provide more discussions on inconsistent performances on Tab. 2.

A3: As shown in Tab. 2, GenZSL achieves better improvements on SUN and AWA2 over CUB. This is because CUB is a fine-grained dataset consisting of 200 bird classes, which are very similar to each other for all classes, GenZSL inevitably synthesizes similar classes for various unseen classes. That is, the diversity of the synthesized samples is limited, and thus the performance gains are not significant. Although SUN is also a fine-grained dataset, it consists of more classes (717 classes) and has higher diversities between various classes. Meanwhile, AWA2 is a coarse-grained dataset, all classes also have higher diversities. Accordingly, GenZSL obtains better performance gains on SUN and AWA2.

Q4: In Tale 4, the unseen class performance of GenZSL falls short of standalone CLIP results. The improved seen class performance, while valuable, somewhat diverges from the primary objectives of zero-shot learning.

A4: In the GZSL setting, the goal of ZSL methods is to achieve good performance both in seen and unseen classes. That is, we mainly evaluate the harmonic mean in GZSL. The unseen accuracy of GenZSL slightly drops over CLIP on CUB, but it significantly improves the performance on seen classes (i.e., 7.1% improvement). This means that GenZSL well address the seen-unseen bias issue for GZSL. We will add these discussions to the final version.

Response: Thank you for the comprehensive reviews and detailed comments! We are very happy to help to address your concerns!

Q1: The claim for CDP lacks quantitative validation (e.g., semantic similarity metrics pre/post-CDP)

A1: The quantitative validation for CDP is presented below, we take these results in ablation study section. Results show that the CDP effectively improves the performance of GenZSL.

Here are the results of pre-CDP.

Here are the results of post-CDP.

Q2: Speed comparisons (Fig. 6) lack details on hardware parity or implementation optimizations.

A2: We implement the results in Fig. 6 on a single NVIDIA RTX 3090 graphic card with 24-GB memory without any further implementation optimization following the official codes. Furthoremore, our GenZSL inductives the unseen samples from the similar seen classess with the guidance of target class semantic vectors, which is more easy to learn the target distribution than the imagination-based generative models that learns from the scratch (e.g., Gaussian distribution). As such, GenZSL learns target distribution of unseen classes efficiently. We will further highlight these discussions in the final version.

Q3: Sensitivity to top-k is untested on SUN.

A3: Due to the page limitation, the hyperparater analysis on SUN and AWA2 are presented in Appendix E. Specicfically, the evaluation of top-k on SUN is presented on Fig. 10(b).

Q4: Are gains due to induction or CLIP’s inherent cross-modal alignment?.

A4: As shown in Tab. 4, GenZSL achieves improvements over CLIP-based models (e.g., CoOp, CoOp+SHIP). Meanwhile, GenZSL also outperforms the imagination-based generative model with identical class semantic vectors (e.g., CLIP embeddings). Due to the different dimensions between CLIP visual features and human-annotated attributes, GenZSL cannot be implemented based on them. As such, we cannot provide the results of GenZSL (strong) similar to f-VAEGAN in Tab. 5. However, f-VAEGAN (strong) obtains better performances than f-VAEGAN (weak), which demonstrates human-annotated attributes could be a better semantic condition in generative ZSL than CLIP embeddings. Furthermore, GenZSL (weak) achieves performance gains over f-VAEGAN with various semantic vectors. These results demonstrate the effectiveness of the induction mechanism in generative ZSL.

We will further highlight these discussions in the final version.

Q5: Comparisons between embedding-based and generative methods.

A5: In Tab. 1, we mainly categorize the compared methods based on the visual features extracted from ViT or CNN structure, and CLIP visual features are also extracted from the ViT network structure. Furthermore, we also compared GenZSL with VADS (Hou et al., 2024) which uses CLIP visual features. Results show the superior performances of GenZSL.

Q6: Comparison between description based ZSL vs. class-name based ZSL.

A6: Thank you for this helpful comment. Indeed, we take description based ZSL (e.g., I2MVFormer-Wiki (Naeem et al., 2023) and I2MVFormer+ (Naeem et al., 2024) into comparison under the CZSL setting in Tab. 1. Recently, TPR (Chen et al., 2024) provided comprehensive results for description based ZSL under the GZSL setting. We will take these description based GZSL methods (especially prompt based methods, e.g., TPR, SHIP, PromptSRC, MaPLe) into discussions in the final version.

Q7: Fig.1 caption line 4: strong should be limited.

A7: Thank you for this helpful comment, we will update it.

Response: Thank you for reviewing our submission and the comments! Here are our responses to your concerns:

Q1: Can the authors provide evidence that these vectors are also more diverse?

A1: In Fig. 3 and Fig. 8, we present the class semantic vectors’ similarity heatmaps that are extracted by the CLIP text encoder and CLIP with our class diversity promotion (CDP). Results show that similarities between various class semantic vectors are smaller with CDP, such as the mean similarity between various classes drops from 0.5726 to 1.825$e^{-5}$ on CUB. That is, CDP makes the refined class semantic vectors nearly perpendicular to each other. As such, the refined class semantic vectors are more diverse.

Q2: Can the authors discuss why the performance does not positively correlate with the amount of augmented data in Fig. 5(c)?

A2: When the number of augmented data for unseen classes is set to small (e.g., smaller than 1600 on CUB), GenZSL can synthesize high-quality samples of unseen classes to effectively train a supervised classifier. However, if the number is set to too large, the model may fail to synthesize diverse unseen samples, resulting in overfitting to seen classes. This is because there exists an upper bound on synthetic diversity. We will add this discussion into Sec. 4.5 in the final version.

Q3: Using weak class semantic vectors for feature generation has been explored in previous ZSL/FSL studies [1]. The sample selection module also aligns with prior works that transfer information from base classes to novel classes for data generation [2][3].

A3: We should emphasize that our GenZSL is different from existing FSL studies. The reasons are illustrated below:

First, [1] is an imagination-based generative model to synthesize data for data augmentation in FSL tasks, it selects representive samples to learn an imagination-based generative model (e.g., standard VAE) for data augmentation. On the contrary, our GenZSL is an induction-based ZSL model, which is an novel induction-based generative model mimicking human-level concept learning. It is effective in synthesizing high-quality samples for unseen classes. The imagination-based generative model synthesizes samples from scratch data, in which the generator learns from scratch without sufficient data to capture the high-dimensional data distribution.

Secondly, [2][3] samples the top-k samples based on the data distribution of visual features, which requires the samples of novel classes. However, there are no samples of unseen classes in the ZSL task, and thus they can be used for FSL but not for ZSL. On the contrary, GenZSL is based on the class semantic vectors to select similar classes for synthesizing unseen samples. Accordingly, GenZSL can applied in the ZSL task, which is focused in this manuscript.

To avoid reader’s confusion, we will add these discussions into the final version.

[1]Xu & Le, Generating representative samples for few-shot classification, CVPR 2022.

[2]Yang et al, Free Lunch for Few-shot Learning: Distribution Calibration, ICLR 2021.

[3]Schwartz et al, ∆-encoder: an effective sample synthesis method for few-shot object recognition, NeurIPS 2018.

Q4: Adding GG (Cavazza et al., 2023) and VADS (Hou et al., 2024) into Tab. 2 for comparison?

A4: Yes, we will add these two works for comparison in Tab. 2.

Q5: Can the authors provide some analysis for choosing temperature parameter in equation (4)?

A5: Initially, we set the temperature parameter $\tau$ to 0.007 following [a] default. Following [4], we set $\tau$ to 0.007/0.2/0.3/1.0 respectively on CUB for analysis. Results show that GenZSL achieves better performance when $\tau$=0.007.

[4] "Understanding the Behaviour of Contrastive Loss." In CVPR, 2021.

Q6: Can the authors also show the effectiveness of the semantically similar sample selection module?

A6: We conduct additional experiments on GenZSL without a semantically similar sample selection module, results show that the performances will decrease compared to GenZSL (full). We will add these results to the Tab. 3.

Results on CUB.

Results on AWA2.