Hierarchical divide-and-conquer grouping for Zero shot learning

1. Pointing out a mistake of this paper below:

• Line 324 & 370 : GZSC -> 'GZSL'?

2. While the combination of proposed ideas is interesting and practical, the individual ideas have been used extensively in literature and are mildly novel.
3. While the authors present ablation studies on the various components they have proposed in Table 2 and Figure 4(a), they do not provide a clear baseline. For instance, it is much easier to classify in the CLIP space, but what about the performance of simply using real training images plus generated images for unseen classes to learn a classifier ?
4. The results are shown only on the three GZSL dataset. It is difficult to evaluate the model's performance in these limited experiments. I request the author use the other datasets like Flower and APY.
5. The main contribution of this paper is the hierarchical divide-and-conquer grouping paradigm. Can you show me its performance in other spaces? For example, using Resnet/ViT as the vision backbone and expert-annotated attributes to demonstrate its efficacy. If it can still offer such a significant improvement, I would be more than happy to raise my rating.

Major problem:

1. The definition of Zero-shot Learning (ZSL) is not correctly used. The authors mentioned ZSL requires the model to recognize unseen classes without additional training data, but in your method, you use pretrained CLIP and Stable Diffusion trained by large-scale dataset. For example, CLIP is trained from millions text-image pairs from web. How can you ensure the test data is not never seen?

2. Besides, in the original paper of CLIP, it has clearly claimed they changed the definition of ZSL from class-level to dataset-level. "In computer vision, zero-shot learning usually refers to the study of generalizing to unseen object categories in image classification (Lampert et al., 2009). We instead use the term in a broader sense and study generalization to unseen datasets."  This excerpt is from Section 3.2 of the CLIP paper. There is a significant gap between CLIP’s definition of zero-shot learning and the task you are addressing, which still focuses on classifying unseen categories. This difference is substantial and cannot be overlooked.

3. In a short, I suggest the proposed method is a transductive classification method insteat of ZSL method actually since the pre-trained Stable Diffusion and CLIP models have been exposed to numerous unseen class samples. Thus, the experiments are extramely unfair for other SOTAs.

4. The generated image and description examples about SUN are missing.

Minor problems:

5. The method types at Table1 are not correct. The Calibration-based methods still are belong to embedding methods.

6. The method I2MVFormer is cited two times at Table1. And at the second time, the authors mis-spell it as I2VMFormer.

7. The method ICCE misses its publish time at Table1.

8. Why only divide two subspaces for seen/unseen spaces?

• The proposed method primarily consists of two strategies for Zero-Shot Learning (ZSL): generating novel class images using a diffusion model and GPT-4, and proposing a divide-and-conquer classification strategy. However, the experimental results may be misleading, as it's unclear how the divide-and-conquer classification strategy would perform when applied to traditional generative methods, without the use of a diffusion model and GPT-4 for novel class generation. In other words, it is unclear where the good performance come from, the powerful diffusion model and GPT-4 model, or the proposed divide-and-conquer strategy.
• Based on the experimental results in Table 1, the performance gain appears to stem primarily from model fine-tuning. Without fine-tuning, the performance on CUB and SUN datasets is notably poor. Further explanations are needed to address this discrepancy.
• It is unclear why setting the depth of the segmented label space to 2 yields the best performance. Intuitively, the optimal depth value should depend on the number of classes in both base and novel categories. For datasets with a larger number of classes, more segmentations might be beneficial, while fewer may suffice for smaller datasets. Therefore, applying the same depth value across different datasets may not yield optimal results.
• It is hard to clearly distinguish the difference between Figure 4(b) and Figure 4(c).

1. Given the use of large models on relatively small ZSL datasets, it would be beneficial for the authors to discuss GPU utilization and the time required for generation and inference.

2. The connection between Section 3.1 and Section 3.2 appears weak. It seems that Section 3.1 primarily addresses data generation for unseen classes. A concern is whether lighter-weight generative models could achieve similar results. Comparing this approach to other image and language models could strengthen the evaluation.

3. Most importantly, how can data leakage be prevented? It is crucial to ensure that the diffusion model and the CLIP model have never been exposed to unseen test images. This issue needs to be explicitly addressed to confirm the integrity of the ZSL task.

4. To enhance clarity, it would be helpful to provide a direct comparison between real unseen images and the generated unseen images, highlighting the differences.

5. More detail is needed on the use of the Normalized Mutual Information (NMI) metric, particularly regarding how it is applied to determine thresholds in this context.

6. Regarding comparison fairness, since many existing methods do not use CLIP as the image encoder, it would be useful to conduct additional experiments with other visual backbones. This would provide a more comprehensive comparison across methods.