Vision Foundation Models Bridge the Geometric Knowledge Across Domains for Long-Tailed Recognition

• The selection of the external knowledge base is crucial for the proposed method, making it a significant limitation.
• The method struggles with complex datasets and has not been tested on the widely-used long-tailed dataset iNaturalist 2018. The reviewer anticipates that the method would perform poorly on iNaturalist, which has over 8,000 categories—far more than ImageNet-1k's 1,000 classes. Additionally, iNaturalist is a fine-grained dataset, leading to many categories being mapped to the same class, thereby confusing head and tail classes and exacerbating the long-tail problem.
• The direct use of ImageNet-1k as the external knowledge base suits the three smaller datasets used in this study, particularly since the classes of CIFAR-10-LT and CIFAR-100-LT are subsets of ImageNet-1k, meaning its rich features sufficiently cover the target datasets. The reviewer believes that the positive results on these datasets are more related to the comprehensive features of ImageNet-1k itself rather than the proposed method. Moreover, the paper does not provide results of fine-tuning the networks pre-trained on ImageNet-1k on the corresponding datasets.
• For ImageNet-LT, the authors use head class knowledge as the external knowledge base to enhance the tail classes, seemingly to avoid using the full ImageNet-1k dataset. If this is the approach, why not use head class data to enhance tail classes across all datasets?

• Potential Data Leakage (L137): The use of the full dataset rather than specifically long-tailed data could introduce label leakage, especially since CIFAR-LT is sampled from CIFAR. How do the authors accurately obtain the true distribution of tail classes in naturally imbalanced datasets, such as iNaturalist, where such knowledge might not be available?

• Section 2 Organization: Section 2 lacks clarity, as it includes a number of prerequisite trials that could be streamlined. I suggest focusing on key observations in the main paper and moving intermediate findings to the supplementary material to improve readability.

• Fairness of Experimental Comparisons: The comparisons with methods that train from scratch seem unfair due to the strong prior used from the VLM. It might be more appropriate to compare with methods that also incorporate strong priors or clarify the differences in setup.

1. My main concern is a lack of theoretical justification for the proposed method. The authors claim that latent-feature distribution of similar classes have similar shape and size, however, the proposition is not theoretically proved. The only evidence provided by the authors is the similarity plots shown in Figure 3, 4 and 5. As mentioned in Section 2.2.1 that small models do not associate geometric shapes across datasets, I am concerned if the results shown in Figure 3, 4, and 5 only apply to the selected datasets (CIFAR, Caltech-101 and ImageNet) and models (DINOv2 and CLIP).

2. The performance of head classes drop significantly on ImageNet-LT and Places-LT (Table 2)

3. The definition of size (Eqn 2) is weird. Let say we have two pairs of $\lambda_1, \lambda_2$: $0.5, 0.5$ and $0.9, 0.1$. The former should have larger area then the later. But, they instead have the same size according to Eqn 2.

4. Figure 3 doesn't support the correlation between shape similarity and categorical similarity. Nearly 20% of the rows have high similarity of shapes to the top-48 nearest-neighbor classes.

5. Figure 5 doesn't support the correlation between size similarity and categorical similarity. For each query class, the author should plot the size ratio (y-axis) w.r.t the feature similarity (x-axis). A query class may have a size ratio of 1 w.r.t the least-similar common class.

6. Figure 6 is confusion. The definition of "IF" is missing.

1. The paper utilizes embedding geometry from an external dataset (ImageNet) to support the claim that the foundation model can bridge geometric structures across domains. While I understand the rationale behind this choice. I am curious about the method's performance if the geometry were derived from the head classes of the same dataset, as in the experiment on ImageNet-LT. This approach would avoid the need for external data, lead to a fair comparison with others.

2. The novelty of the proposed method feels somewhat limited, as the core concept closely resembles that of FUR [1]. The primary difference here is the transfer of geometry from an external dataset and the use of foundation models to facilitate this.

[1] Ma, Yanbiao, et al. "Geometric Prior Guided Feature Representation Learning for Long-Tailed Classification." International Journal of Computer Vision (2024): 1-18.