BLG: BALANCED LANGUAGE DISTRIBUTION AS GUIDANCE FOR ROBUST LONG-TAILED VISION CLASSIFICATION

Connection with contrastive learning.

We sincerely appreciate the thoughtful reviews and the valuable insights provided by each reviewer. We have carefully considered your comments regarding the connections and differences between our method and previous works based on contrastive learning for imbalanced image classification. We acknowledge the similarities but would like to emphasize the essential differences that set our approach apart.

(1) Distinguishing from Contrastive Learning:

Our method is not solely based on contrastive learning. While we use CLIP and propose the use of a supervised contrastive learning loss function, leveraging the optimal transport matrix to enhance model learning and improve visual features, this aspect represents only a supplementary component within our broader method. Furthermore, our experiments demonstrate that even without the proposed contrastive supervision loss function, our method achieves state-of-the-art performance.

(2) Optimal Transport Matching vs. Contrastive Learning:

When addressing the challenge of improving biased visual features to achieve better balance and separability, we uniquely formulate this problem as an optimal transport matching problem. This approach stands out differently from traditional contrastive learning. Our objective is to minimize the optimal transport distance, ensuring a closer alignment between visual features of the same type and their prototypes while creating a more significant separation between different types.

(3) Leveraging Complementarity of Multi-Modal Information:

In the context of imbalanced image classification, effectively leveraging the complementarity between multi-modal information for robust visual feature learning remains an open challenge. Our experiments in rebuttal reveal that naively employing text features as traditional positive samples or anchor points in contrastive learning methods does not directly contribute to effective model learning. Therefore, we find it imperative to design and implement more nuanced methods to address imbalanced image classification under multi-modal conditions.

We believe these distinctions underscore the uniqueness of our method and its contributions to addressing the complexities of imbalanced image classification. We appreciate your thorough evaluations and welcome any further suggestions or inquiries.

Thanks for your kind suggestion and for providing related work to help refine our work.

1. Reply to #w1 and #w2:

Thank you for sharing these papers. We noticed with them before, as they share a similar motivation to ours: learning balanced and discriminative visual features for long-tailed recognition. Despite these similarities, there are significant differences between our approaches.

The first paper maximizes mutual information between visual latent features and ground truth labels, employing a contrastive learning method based on image modality. In contrast, our method leverages complementary textual information, to facilitate the learning of robust visual features. Notably, our approach does not explicitly model or utilize label frequency, as seen in the first paper, where it is used to compute logit adjustment terms and construct class-wise queues. Instead, our focus is on achieving high separability between classes based on textual features. Moreover, our method does not rely on the teacher-student network structure. Furthermore, our proposed supervised contrastive learning loss is only part of our contribution, which serves as a complementary for the OT-based prototype-guided matching. As shown in Table 3, our method can still perform best even without the proposed $\mathcal{L}_{\text{SCT}}$.

The second paper employs k-positive samples as anchors for contrastive loss, deviating from using augmented instances. This approach, dependent on the number of samples in the minority class, cannot guarantee the distance between k-positive samples from different classes, potentially causing tail classes to be pushed closer together. In contrast, our method is independent of the number of samples and consistently identifies prototypes with strong discriminability based on textual features. Additionally, our approach is multimodal, utilizing optimal transport to align visual features and prototypes in an unsupervised manner. We introduce class-level supervision signals based on the transport matrix to further enhance feature learning.

In summary, while these referenced papers effectively aim to learn more discriminative visual distributions, our work distinguishes itself through a focus on multimodal data, the incorporation of optimal transport for balanced prototype-guided visual representation learning and class-level supervision signals based on transport plan. We will discuss more detailed differences in the related works .

2. Reply to #Q1:

Firstly, we argue that text features are a more high-level abstract semantic representation, which inherently has better discriminability. Therefore, they can serve as queries in the Vision Language Model to perform tasks such as classifying the input image features. This discriminability, in our work, is manifested as balance. That is, the language encoder is less affected by the tail class. As shown in the visualization in Appendix A, textual features always maintain good separability and balance, regardless of whether the language encoder of CLIP has been fine-tuned or not. Secondly, when we construct the text prompt for the input image, we use the class name information. This kind of information already provides high separability for textual features. From the text features, we can often directly infer which object a sentence is describing by the classname. Therefore, they can serve as prototypes with good classification properties. We aim to make visual features better cluster around the corresponding prototypes. In addition, when we use balanced textual features to build prototypes, each class uses the same number of text prompts. From this perspective, the prompts between different classes are balanced. The head class does not have more prompts than the tail class. Finally, when we fine-tune CLIP, we only use the InfoNCE loss in the image-retrieval direction, because our goal is to use text to classify images, which can also reduce the negative impact on textual features during Phase A to some extent.

3. Reply to #Q2.

Thanks for your kind reminder. We will update our tables for a clear comparison in the revision.

4. Reply to #Q3.

Yes. The introduction of OT loss brings some lightweight additional computational burden. The added computational burden mainly comes from the need to calculate the optimal transport (OT) distance when aligning visual features and prototypes. We have comprehensively compared our method with BALLAD and VL-LTR in Appendix G. As shown in Table 14, our method is comparable to other methods in terms of the training cost. Compared with BALLAD, we added 10 epochs in the training phase and required more time, but we needed less inference time and had a significant improvement in performance (51.9% vs 49.5%). Compared with VL-LTR, we require fewer epochs for training, less training time, and no additional data. Still, we achieve better performance (51.9% vs 50.1%) than VL-LTR.

Thanks for your kind suggestion and for providing related work to help refine our work.

1. Reply to #W1, #Q1 AND #Q2:

It involves a trade-off in performance among three different shot divisions in long-tail. Despite this, our method boasts several key advantages:

(1) We view imbalanced recognition as a task primarily concerned with enhancing the classification performance of tail classes, minimizing the performance gap with head classes, and achieving superior overall classification performance. While BALLAD and VL-LTR excel in overall performance by mainly improving head class results, they fall short in addressing the tail. For instance, when based on RN50 and additional texts for training, VL-LTR achieves 77.8% accuracy in the Many-shot in ImageNet-LT but only 50.8% accuracy in the Few-shot, resulting in a substantial 27.0% performance gap. In contrast, our method exhibits a significantly lower performance gap of 8.7%.

(2) Our approach outperforms in medium-shot, few-shot, and overall performance, effectively narrowing the performance gap between these divisions and many-shot. This aligns with our motivation to attain a more balanced visual feature distribution, resulting in a more balanced classification performance. Notably, we achieve a more substantial performance improvement in few-shot scenarios compared to previous methods. On ImageNet-LT, our method showcases a 10% increase in the few-shot category. Similarly, on Places-LT, the improvement exceeds 12%.

Furthermore, we wish to draw a distinction between the few-shot division in long-tailed recognition and few-shot learning in the general sense. The former serves as a means and perspective for performance evaluation after data division under the long-tail problem, indicating performance on minority class samples. In contrast, the latter focuses more on the disparity between base classes and novel classes, involving learning with limited samples after the introduction of new classes to achieve improved performance. In the former, each arrival constitutes a completely imbalanced dataset, requiring the model to learn and classify the entire dataset. Ultimately, our evaluation considers performance under three different divisions and overall performance.

2. Reply to #W2:

Fully fine-tuning of the CLIP encoders indeed introduces a large computation cost. We notice there are some parameter-efficient methods developed for tuning CLIP encoders. We will focus on this issue and try more efficient ways to improve our method.

3. Reply to #W3:

Thanks for your kind suggestion. We will try our method on more scenarios, such as few-shot learning and noisy data. Due to the rebuttal time limit, we may not give comprehensive results now.

4. Reply to #Q3:

We employ a random sampling strategy when fine-tuning CLIP encoders in the first stage, driven by three key considerations.

Firstly, as indicated in related works [1,2], the feature extractor's impact on classification performance may not be significant. The pivotal factor in imbalanced classification accuracy lies in the learning of the classifier. Consequently, utilizing random sampling during encoder training or fine-tuning, even with skewed visual features, remains a viable approach. Following this, methods often choose to fix encoders and employ new sampling strategies to train the classifier. Based on this concept, we adopt a random sampling strategy during encoder fine-tuning in the first stage. In the second stage, we shift to a balanced sampling strategy to refine visual features and train a balanced classifier. This decision aligns with our motivation.

Secondly, experiment results indicate that using a balanced sampling strategy to fine-tune encoders in the first stage is not the optimal approach. Updated experiment results on sampling strategy in Appendix H support this observation. Notably, employing random sampling in the first stage and balanced sampling in the second stage yields the best results, outperforming the strategy of using balanced sampling in both stages. This finding is consistent with conclusions drawn in previous related work, such as BALLAD.

Finally, a new t-SNE visualization result in Appendix H compares visual features extracted from the same class under two different sampling strategies. Even when a balanced sampling strategy is used in Phase A, it does not significantly enhance tail classes' feature distribution distinctiveness. The visual feature distribution obtained by balanced sampling does not markedly differ from that obtained by random sampling on tail samples. However, for head samples, visual features of the head class are less densely clustered than those obtained by random sampling. This underscores the necessity of employing different sampling strategies in distinct stages from an alternative perspective.

[1] Decoupling representation and classifier for long-tailed recognition. [2] Improving calibration for long-tailed recognition.

Dear Reviewer nBTe,

Given the constrained timeframe for further discussion, we genuinely appreciate any feedback you may have on our revised submission. If there are additional questions or points of clarification needed, we are more than willing to promptly address them.

Again, we express our gratitude for your time, extensive efforts, and valuable insights. Thank you for the opportunity to engage in this discussion.

Best regards, The Authors

Thank you to the authors for your diligent efforts and feedback, which have resolved some of my queries. However, I still hold reservations regarding the following points:

1. Same as what I mentioned in weakness. As a paper to solve the LTR problem, the proposed method does not appear to have a specific design tailored for handling long-tail data distributions.

2. The authors have only compared the results of combining their method with [1], without comparing it with the results of using [1] in isolation. Additionally, the results indicate that the method in [1] only slightly decreases the performance for 'many' and 'medium' split sets while enhancing the 'few' set. However, the complexity of the method in [1] is significantly lower than that of the proposed method. Moreover, it employs a CLIP linear probe without the need to fine-tune the entire CLIP model, making it a more efficient and direct approach to leveraging textual features.

[1] https://arxiv.org/abs/2301.06267

3. There is a substantial domain gap between textual and image features. Is it reasonable to use textual features as initial centroids to guide the clustering of image features? Specifically, while textual features possess strong discriminative power, the distance between textual features and image features (in the test set) remains considerably larger than that between image features in the training and test sets. Using textual features to guide the training set image features might yield more discriminative features and decision boundaries. However, this approach could increase the distance between the original (training set) image features and the (test set) image features, potentially leading to overall poorer performance.

Dear Reviewer nBTe,

Thanks for your insightful question and informative feedback to help refine the paper.

1. Reply to #W1

Allow us to elaborate on how our approach aligns with the challenges posed by long-tailed recognition (LTR), both in motivation, method design, and experimental outcomes.

(1) Motivation:

Our motivation is grounded in the persistent balance and separability observed in text feature distributions even after processing imbalanced datasets through CLIP encoders. Recognizing the inherent challenges of long-tailed data, we aim to harness this balance in text features to guide the learning of highly discriminative visual representations, thereby enhancing the performance of long-tailed vision recognition.

(2) Method Design:

In the design of our method, we initiate the classifier (prototypes) with balanced text features and employ a balanced data sampling strategy during model training. By utilizing optimal transport to match visual features with their prototypes, our approach aims at achieving a delicate balance between learning discriminative visual features and addressing the imbalances inherent in long-tailed datasets. The incorporation of a residual structure refines biased visual features, promoting their balance and separability in the long-tailed context.

(3) Experimental Insights:

The experimental outcomes corroborate the effectiveness of our method in the long-tailed recognition scenario, showcasing state-of-the-art performance. In response to the reviewer's inquiry about the applicability to limited data scenarios, our additional experiments demonstrate the adaptability of our method to few-shot learning tasks. While the versatility of our approach is evident, it is essential to underscore that the primary strength of our method lies in its substantial improvement in addressing the challenges posed by long-tailed datasets in visual recognition.

In summary, our method exhibits a nuanced approach tailored to the intricacies of long-tailed recognition. By leveraging the balance in text features, our method not only specifically excels in imbalanced vision recognition but also extends its effectiveness to the nuanced domain of few-shot learning, reinforcing its applicability across various challenges in visual recognition tasks.

2. Reply to #W2

We appreciate the opportunity to provide further clarification and additional insights regarding the comparison with the referenced work [1], and we acknowledge the need for a more comprehensive and accurate depiction of our method's nuances.

Firstly, we would like to revisit the essence of [1]. The primary objective of [1] is to enhance CLIP's performance in few-shot learning tasks by incorporating paired visual and textual features as input. This method utilizes cross-modal information to train a robust classifier, ultimately improving CLIP's classification performance on few-shot tasks. The core of [1] lies in the introduction of textual information as input, a strategy we also explored in our experiments.

In our previous response, we conducted experiments by incorporating the idea of introducing text as input into our existing method, which involves the full fine-tuning of CLIP encoders, the vision adapter, and the utilization of prototype-guided matching loss. Therefore, the outcome of these experiments revealed a marginal decrease in performance compared to our original method.

To provide more clarity on the differences, we provide additional experiments. We utilized the released code from [1] and evaluated its performance on the Long-Tailed Recognition (LTR) task. Without fine-tuning CLIP encoders and without employing any of our proposed losses, [1] achieved an overall performance of only 42.9% on Places-LT based on ViT-B/16. This result underscores that while [1] is effective and insightful for few-shot learning, it is not inherently tailored for the challenges posed by LTR tasks, and its performance does not significantly contribute to improving classification accuracy.

Furthermore, when fine-tuning CLIP encoders and exclusively using cross-modal input for training the classifier, [1] achieved an overall performance of 48.20%. This outcome, although indicative of the effectiveness of [1] in few-shot learning, falls short in comparison to the performance achieved by our method in LTR tasks.

In summary, while [1] offers a valuable approach for few-shot learning, its adaptability to LTR tasks is limited, as evidenced by the comparative experiments. Our method, designed specifically for LTR tasks, demonstrates superior performance and effectiveness in addressing the challenges inherent in long-tailed datasets.

Thanks for your kind suggestion and for providing related work to help refine our work.

1. Reply to #W1.

This paper introduces a targeted contrastive learning loss (TSC) aimed at aligning class features with target features on the vertices of a regular simplex. We identify four key distinctions between our approach and TSC:

(1) Learning Targets and Complexity:

TSC employs SGD to learn targets, posing challenges in finding the optimal solution on a hypersphere [1,2]. This method imposes a demanding requirement that is difficult to fulfill perfectly. In contrast, our approach is more straightforward, avoiding complex assumptions and calculations to obtain balanced prototypes. Our prototypes, derived from textual features, carry rich semantic information, enhancing the discriminative power of image features. Unlike TSC, which constructs targets without class semantics, we leverage a more abstract and balanced text modality. Therefore, our methods differ fundamentally, despite sharing a general motivation. Additionally, TSC requires prior knowledge of the number of classes, whereas our method is more flexible.

(2) Implementation Complexity:

TSC faces the challenge of assigning class indices to each unlabeled anchor point, adding complexity to the method. In contrast, our approach constructs prototypes in a more natural and intuitive manner.

(3) Incorporation of Components:

TSC relies solely on supervised contrastive learning loss, while our method incorporates various components. TSC updates the model by minimizing the distance between visual features and assigned targets. In contrast, our method transforms balanced textual prototypes into a linear classifier, employing optimal transport for unsupervised alignment of visual features and prototypes. Furthermore, we introduce a supervised contrastive learning loss based on the transport matrix to enhance supervision signals and intra-class information. Therefore, our contribution extends beyond a supervised contrastive learning loss. From the ablation study, our method can still perform best even without the proposed supervised contrastive loss.

(4) Modality and Multimodality:

TSC operates as an image-only method, while our approach is multimodal. To assess TSC's impact, we integrated its method into our work and conducted a performance comparison. Indicating our proposed three losses as $L_{CE}$, $L_{OT}$ and $L_{SCT}$, we compare the performance replacing $L_{SCT}$ of $L_{TSC}$ based on ViT-B/16 and Places-LT dataset. As shown in the table below, using textual prototypes to initialize anchor points and applying TSC in Phase B led to a significant performance drop, demonstrating that our method is not a mere substitution of anchor points in TSC with textual features.

In summary, these four aspects highlight the substantial differences between our method and TSC, reinforcing the unique contributions of our approach.

[1] https://arxiv.org/pdf/physics/0609231.pdf [2] https://arxiv.org/abs/cond-mat/0506616

2. Reply to #W2.

The linear classifier plays a pivotal role in this context, serving as a crucial bridge. Its utilization is essential for maintaining coherence between visual and textual features. This necessity arises because, in the second phase, an additional residual module is introduced to refine biased visual features. Subsequently, the linear classifier serves a dual purpose in the final classification and is ultimately employed to guide the refinement of biased visual features using balanced textual initialized prototypes.

When we use textual features to initialize the classifier, our classification process no longer depends on the text prompt and text encoder. In the learning process of the linear classifier, we initialize with textual features and use one-hot labels and cross-entropy loss to optimize it. This approach means that the decision boundary constantly adapts to the input mini-batch. As performance ascends, the alignment between the textual features initialized classification boundary and visual features achieves a more nuanced congruence. Consequently, the task can be comprehended as the reduction of the gap between textual and visual features. Finally, the parameters of the classifier are harnessed for prototype-guided matching to enhance visual features.

In instances where the linear classifier is not employed, our model structure aligns with BALLAD. The results of the ablation study in Table 4 reveal that without the use of a trainable linear classifier, our approach outperforms BALLAD on ImageNet-LT and Places-LT by 1.1% and 0.3%, respectively. Nevertheless, the incorporation of the classifier, accompanied by continuous refinement of the classification boundary to strengthen the coherence between textual and visual features, leads to further performance enhancements.

Furthermore, we supplement our findings by presenting experimental results when the classifier is fixed, and only the optimization of residual links to improve visual features is pursued. As indicated in the table below, when the classifier is frozen, a certain performance impact is observed, resulting in a 1.5% decrease. Therefore, a learnable classifier can help continually and adaptively keep the consistency between textual features (prototypes) and the input visual features, from the perspective of both motivation and experiment results.

Besides, we update a visualization result in Appendix E.3 to demonstrate how the learnable linear classifier benefits our method and keeps the consistency of visual features and the corresponding textual prototypes (the weight of the classifier).

3. Reply to #W3.

A cost matrix serves as a representation of the cost function in a discrete optimal transport problem, where goods or resources are distributed among a finite number of sources and destinations. This matrix is structured with a row for each source and a column for each destination, where each entry denotes the cost of transporting one unit of goods or resources from the corresponding source to the respective destination. In practical terms, various measurements capable of computing the distance between two distributions can function as a cost matrix. Examples include cosine similarity, Euclidean distance, or a matrix derived through learning in a specific manner (reverse OT problem).

It is worth noting that minimum entropy might be unsuitable as a measurement for the cost matrix in this context. For our purposes, we opt for cosine similarity due to its universality and superior overall performance compared to other distance metrics. The comparison results of cosine similarity and Euclidean distance, employed as a cost matrix based on Places-LT and ViT/B-16, are presented in the following table:

Dear Reviewer pMKo,

Given the constrained timeframe for further discussion, we genuinely appreciate any feedback you may have on our revised submission. If there are additional questions or points of clarification needed, we are more than willing to promptly address them.

Again, we express our gratitude for your time, extensive efforts, and valuable insights. Thank you for the opportunity to engage in this discussion.

Best regards, The Authors

Connection with contrastive learning.

We sincerely appreciate the thoughtful reviews and the valuable insights provided by each of you. We have carefully considered your comments regarding the connections and differences between our method and previous works based on contrastive learning for imbalanced image classification. We acknowledge the similarities but would like to emphasize the essential differences that set our approach apart.

(1) Distinguishing from Contrastive Learning:

Our method is not solely based on contrastive learning. While we use CLIP and propose the use of a supervised contrastive learning loss function, leveraging the optimal transport matrix to enhance model learning and improve visual features, this aspect represents only a supplementary component within our broader method. Furthermore, our experiments demonstrate that even without the proposed contrastive supervision loss function, our method achieves state-of-the-art performance.

(2) Optimal Transport Matching vs. Contrastive Learning:

When addressing the challenge of improving biased visual features to achieve better balance and separability, we uniquely formulate this problem as an optimal transport matching problem. This approach stands out differently from traditional contrastive learning. Our objective is to minimize the optimal transport distance, ensuring a closer alignment between visual features of the same type and their prototypes while creating a more significant separation between different types.

(3) Leveraging Complementarity of Multi-Modal Information:

In the context of imbalanced image classification, effectively leveraging the complementarity between multi-modal information for robust visual feature learning remains an open challenge. Our experiments in rebuttal reveal that naively employing text features as traditional positive samples or anchor points in contrastive learning methods does not directly contribute to effective model learning. Therefore, we find it imperative to design and implement more nuanced methods to address imbalanced image classification under multi-modal conditions.

We believe these distinctions underscore the uniqueness of our method and its contributions to addressing the complexities of imbalanced image classification. We appreciate your thorough evaluations and welcome any further suggestions or inquiries.