Supervised Dimension Contrastive Learning

Motivated by Barlow Twins, this paper proposes a method to constrain the feature diversity in supervised contrastive learning. Experimental results show the effectiveness of the proposed method.

This paper enhances Barlow Twins by using the class-correlation loss and orthogonal regularization loss. The class correlation loss uses supervision information, and the orthogonal regularization loss enforces the aggregate function to be full-rank. These additional losses positively affect the overall performance. Through numerical experiments on in-domain classification and out-domain transfer learning tasks, the proposed SupDCL demonstrated superior performance.

The paper introduces Supervised Dimension Contrastive Learning (SupDCL), which combines supervision with dimension-wise contrastive learning. It targets limitations in supervised contrastive learning (high cross-correlation and limited feature diversity) by reducing redundancy in embedding dimensions while enhancing class discriminability.

This paper argues that representations learned by supervised contrastive learning methods have limited feature diversity and high cross-correlation between dimensions. The authors propose an approach to learn representations that are both augmentation-invariant and dimension-wise decorrelated, while also being discriminative for classification. They achieve this by extending Barlow Twins, a dimension contrastive learning model, to supervised dimension contrastive learning. Specifically, they introduce a discriminativeness loss instead of the cross-entropy loss, consisting of a class correlation loss and an orthogonal regularization loss. Experimental results show that the proposed SupDCL performs well on both in-domain supervised classification tasks and out-of-domain transfer learning tasks.

Dear Reviewers,

We thank the reviewers for their valuable feedback, which guided us to strengthen the theoretical and empirical justification of our proposed dimension-based framework. Initially, we focused on the motivations behind our approach. Now, we theoretically and empirically demonstrate that all three components addresses specific challenges. Specifically, we decompose the mutual information between class labels and learned representations into two components (refer to Revised G2): diversity and discriminativeness, and together, they form a comprehensive framework for robust generalization. All justifications have been revised and reflected in the updated version of the paper.

$I(C;Y)$

$=\underbrace{H(Y)}_{\text{diversity}}$ “the total information capacity of the learned representation $Y$”

$-\underbrace{(H(\hat{C}|C)+H(Y_\text{null}))}_{\text{discriminativeness}}$ “the uncertainty and information loss related to the aggregation process”

1. Dimension Decorrelation - Maximizing $H(Y)$:

   • Reduces inter-dimension correlation using Barlow Twins (BT), theoretically shown to increase information capacity of representation (Revised G2 of the global response and Appendix A.2).
   • Empirically, it improves generalization performance, as evidenced by the relationship between dimension correlation and whole-domain accuracy (Figure 1) and the effectiveness of BT+CE over CE alone (Table B).

2. Class Correlation - Minimizing $H(\hat{C}|C)$:

   • A novel method for robust distributional alignment, theoretically shown to be invariant to covariate shift, and minimize distance between predicted and true class distributions (Revised G2 of the global response and Appendix A.3.1).
   • Empirically, class correlation improves in-domain and out-domain performance, consistently outperforming CE across all settings (Table B).

3. Full-rank Aggregation - Minimizing $H(Y_\text{null})$:

   • Preserves class-relevant information of representation, theoretically shown to minimize information capacity of null space (Revised G2 of the global response and Appendix A.3.2).
   • Empirically, comparisons such as BT+CE vs. BT+CE+Orthogonal and SupDCL w/o Orthogonal vs. SupDCL confirm its significant contribution to both in-domain and out-domain performance (Table B).

Combining these components, our method achieves state-of-the-art whole-domain generalization, outperforming all supervised and self-supervised baselines across in-domain and out-domain tasks (Table C).

Table C. In-domain classification and out-domain transfer learning. Linear evaluation performance comparison on 10 downstream datasets, for ResNet-50 pre-trained on ImageNet-1K.

Table B. Comparison of SupDCL and straightforward extensions of Barlow Twins with cross-entropy loss

I have read and agree with the venue's withdrawal policy on behalf of myself and my co-authors.