I-Con: A Unifying Framework for Representation Learning

We thank the reviewer for their thoughtful feedback and have worked to address each of the key concerns raised below. In addition we also added several other key improvements that we detail in the global response. If the reviewer finds these additions helpful we kindly request that they raise their score to help us share this work with the community.

Adding experiments beyond the domain of clustering

In a new series of experiments we show that I-Con not only improves clusterers, but can improve contrastive feature learners. As requested by reviewers this experiment also demonstrates that I-Con works on an additional backbone and datasets. We follow the contrastive feature learning experiment put forth by Hu et al. at ICLR 2023. In particular we train a ResNet34 network contrastively on CIFAR10 and evaluate its linear probe and KNN performance on both CIFAR10 and CIFAR100. We include a detailed description of this experimental setting and its results in the Supplement. In summary, we show that our distribution improvement methods like debiasing and replacing gaussian kernels with cauchy kernels dramatically improves feature quality both in terms of linear probe accuracy and KNN accuracy.

Comparing I-Con to Related Survey Papers, Such as Tschannen et al.

In the revised version of our paper, we included a short discusses on Tschannen et al.'s paper. Their work critically examines the role of mutual information (MI) maximization in representation learning, highlighting that the success of MI-based methods often arises from architectural choices and parameterization of MI estimators rather than from MI maximization itself.

Our I-Con framework differs from Tschannen et al.'s focus in several key ways:

1. Scope and Unification:
   While Tschannen et al. critique the effectiveness of MI maximization, our framework provides a unifying equation that encompasses a wide range of methods across clustering, spectral methods, dimensionality reduction, contrastive learning, and supervised learning.

2. Avoiding MI Estimation Challenges:
   We sidestep the difficulties associated with estimating MI in high-dimensional spaces by formulating our objective as minimizing the KL divergence between conditional distributions. This approach avoids the estimation issues highlighted by Tschannen et al.

By including this comparison, we aim to highlight the contributions of our work in advancing the understanding and practice of representation learning beyond the limitations identified in prior studies.

We thank the reviewer for their thoughtful feedback and have worked to address each of the key concerns raised below. In addition we also added several other key improvements that we detail in the global response. If the reviewer finds these additions helpful we kindly request that they raise their score to help us share this work with the community.

Correct naming of methods in main table and using a single version of the method

This is a good point and we have adjusted the naming in the table to InfoNCE Clustering to make it clear that this is a derived clustering method from the broader I-Con framework. We also now use the same version of the method (with EMA, Walks, and Debiasing) across the entries of Table 2 as requested.

Report multiple seeds and error bars for clustering results to strengthen empirical evidence.

We have added error bars corresponding to the standard error of the mean in table 2. We calculate these error bars from 5 different seeds. The main conclusions of the work still hold, and we maintained the SOTA across all seeds.
Evaluate whether the I-Con framework can reproduce or improve original methods like InfoNCE on ImageNet. Yes, not only is InfoNCE a special case of the I-Con framework, it can also be improved by I-Con. In particular, similar to how T-SNE improves SNE by using cauchy distributions, we see a similar improvement in contrastive learning. Secondly, we find that our debiasing operator also improves classic vanilla InfoNCE. We add a variety quantitative experiments verifying these claims in the supplement.

On replacing MNIST qualitative Figures with Imagenet

We chose MNIST because of the small number of classes which made it easy to visualize, we have since refined our visualization to apply to imagenet and have updated the figure accordingly.

Thank you for your review and thoughtful comments. We have made several updates to clarify and improve the paper.

---

I agree that compared to previous works, I-Con can self-balance the certainty in representation learning without additional methods (e.g., entropy penalties, activation normalization, activation sharpening, or EMA stabilization). However, I still think this conflicts with your claims in the ablation study in Section 4.3, which highlights the sensitivity to propagation size and other hyper-parameters that still require tuning. Is it possible to claim that I-Con doesn’t require additional regularizations rather than hyper-parameter tuning?

Yes, not requiring regularizers is a correct way of saying it, we updated the text in that section 4.2 to better reflect that view and make it less confusing!

---

In Table 2, why not report error bars for baselines and scores in Table 3?

Due to time constraints during the initial submission, we prioritized reporting the most significant row. However, in this revision, we have updated all entries in Table 3 to include error bars and have added error bars to Figure 4 as well. Thank you for your patience as we worked on this!

---

Is I-Con (Ours) stated in Table 2 the result of the integrated method discussed in Section 3.3, or is it the best among them? Table 3 shows results of each transferred intuition, and results of I-Con (Ours) stated in Table 2 appear to be the best among them. Referring to the best among them as I-Con (Ours) in Table 2 may not be a fair comparison.

Thank you for raising this point. We have updated the numbers in Table 2 to clarify that "(Ours)" refers to the model with EMA, walks of length $2$, and $\alpha = 0.4$.

---

DiNO [1] also showed results of k-NN classifiers [2] in the paper. Could results in Table 3 maintain its gains when using k-NN classifiers as stated in [1]?

Table 3 presents the Hungarian clustering accuracy, the standard method for evaluating unsupervised classification systems. Since we are not learning a continuous vector specifically designed for retrieval in these experiments, we believe that such a metric would not be meaningful. However, in our new feature learning experiments on CIFAR-10, CIFAR-100, STL-10, and Oxford-IIIT Pet (the last two were added in this revision), we have included k-NN accuracy results for both in-distribution and out-of-distribution evaluations.

---

Table 2 in the first line in Section 4.2 does not have a link to it.

Thank you this got lost in the rebuttal whirlwind, we have since added it!

---

Thanks for your lovely round of feedback and clarifications! We hope this next revision clears up the remaining points.

Providing Guidance on Choosing $p$ and $q$ Distributions According to I-Con

Thank you for raising this point. We added a section to the supplement discussing this intuition and tried to make it a bit clearer in the main text.

In summary, I-Con provides a single unified way to view a variety of different methods in the literature as particular choices of $p$ and $q$ in Equation 1. The I-Con framework helps reveal some of the difficult-to-see implicit distributions assumed by various methods. For example, when studying K-means, it's not easy to see that it preserves neighborhoods of uniform Gaussian variance. We elaborate on some of the key design decisions to consider when choosing $p$ and $q$.

---

1- Parameterization of Learning Signal

• Parametric:
  Learn a network to transform data points to representations.

  • Use a parametric method to quickly represent new datapoints without retraining.
  • Use this option when there are enough features in the underlying data to properly learn a representation.
  • Ideal for datasets with sparse supervisory signals to share learning signals through network parameters.

• Nonparametric:
  Learn one representation per data point.

  • Use a nonparametric method if data points are abstract and don’t contain natural features useful for mapping.
  • Better for optimizing the loss of each individual data point.
  • Avoid in sparse supervisory signal regimes (e.g., augmentation-based contrastive learning), as there aren’t enough links to resolve each embedding.

---

2- Choice of Supervisory Signal

• Gaussians on distances in the input space:
  Though common and underlying methods like K-means, with enough data, it is almost always better to use k-neighbor distributions, as they better capture the local topology of data.
  This is the same intuition used to justify spectral clustering over K-means.

• K-neighbor graph distributions:
  If your data can naturally be put into a graph, we suggest using this instead of Gaussians in the input space.

  • This allows the algorithm to adapt local neighborhoods to the data, rather than considering all point neighborhoods equally shaped and sized.
  • Better aligns with the manifold hypothesis.

• Contrastive augmentations:
  When possible, add contrastive augmentations to your graph.

  • Improves performance when quantities of interest (e.g., image class) are guaranteed to be shared between augmentations.

• General kernel smoothing techniques (e.g., random walks):
  Generally improves optimization quality.

  • Connects more points together.
  • Sometimes mirrors geodesic distance on the manifold [Geodesics in Heat, Crane 2016].

• Debiasing:
  Use debiasing if you think negative pairs have a small chance of aligning positively.

  • For a small number of classes, this parameter scales like 1/K where K is the number of classes.
  • Improves optimization stability.

---

3- Choice of Representation

Any conditional distribution on representations can be used. Consider the desired structure, such as a tree, vector, or cluster, and choose a simple, meaningful distribution for that representation.

• Discrete:
  Use discrete cluster-based representations if interpretability and discrete structure are important.

• Continuous Vector:
  Use a vector representation if generic downstream performance is a concern.

  • Easier to optimize than discrete variants.

We thank the reviewer for their thoughtful feedback and have worked to address each of the key concerns raised below. In addition we also added several other key improvements that we detail in the global response. If the reviewer finds these additions helpful we kindly request that they raise their score to help us share this work with the community.

New Series of Experiments Demonstrating that I-Con improves Feature Learning and works on different backbones and datasets.

In a new series of experiments we show that I-Con not only improves clusterers, but can improve contrastive feature learners. As requested by reviewers this experiment also demonstrates that I-Con works on an additional backbone and datasets. We follow the contrastive feature learning experiment put forth by Hu et al. at ICLR 2023. In particular we train a ResNet34 network contrastively on CIFAR10 and evaluate its linear probe and KNN performance on both CIFAR10 and CIFAR100. We include a detailed description of this experimental setting and its results in the Supplement. In summary, we show that our distribution improvement methods like debiasing and replacing gaussian kernels with cauchy kernels dramatically improves feature quality both in terms of linear probe accuracy and KNN accuracy. [update: we added more datasets (STL-10 and Oxford-IIT Pets) in the latest comment as well!]

Proving that MDS, PCA (Linear Autoencoders), Masked Language Modeling, Spectral Clustering, and Triplet Loss are unified by I-Con

We introduce new theorems that unify 6 additional methods with the I-Con framework. In particular we show that MDS and SNE can be unified in the limit of large gaussian scales, which is a novel theoretical result first proved in this work. As requested by reviewer iHot, this result allows us to integrate linear autoencoders (PCA) into I-Con. As requested by reviewer iHot we also show masked language modeling can be considered in the I-Con framework. Intuitively this is a simple corollary of our existing proof connecting Cross Entropy with I-Con, except replacing class embeddings with embeddings for masked tokens. We also show that Spectral Clustering emerges naturally from I-Con. Finally, we show that the commonly used triplet loss for contrastive learning emerges as in the limit of small width gaussians in I-Cons InfoNCE connection. These additional proofs can be found in the supplement.

Elaborating on the difference between representation learning, clustering, and contrastive learning

We want to take the opportunity to clear up some questions and misunderstandings about I-Con and representation learning more broadly. In particular

1. I-Con does already unify several representation learning methods. And is not just a unification of clustering methods. In particular I-Con unifies many methods in Dimensionality Reduction (t-SNE, SNE, In this revision MDS, PCA (Linear AutoEncoding), Contrastive Learning (SimCLR, MocoV3), and in this revision we added masked pretraining as well.
2. We don’t claim to unify every representation learning method, but we do show a diverse collection of popular algorithms are unified by I-Con. For example DinoV2 is not an I-Con method and suffers from collapse that needs to be rectified through activation sharpening.
3. When we say “representation learning” we usually refer to the broader context where one can learn any kind of representation, discrete, continuous, low-dim etc. The difference between contrastive feature learning and contrastive cluster learning is whether the method learns vectors in R^n or cluster assignments. These two methods need to be evaluated differently.
4. In our original submission we did not use linear probe evaluation strategies because they apply to continuous representations.
5. For our clustering experiments we use the commonly accepted hungarian accuracy. Nevertheless we added additional experiments with linear probe and KNN analysis to show I-Con can improve contrastive feature learners ad well as clusterers.

Clarifying that data augmentation-based contrastive learning methods are already unified within the I-Con framework.

I-Con already unifies data-augmentation based contrastive methods and we point the reviewer to our analysis of the InfoNCE loss in the appendix. We also tried to make these connections a bit more obvious in the main text. To briefly recap here, augmentations are a particular type of supervisory distribution, p, where p(i|j) = 1 iff i and j are different augmentations of the same image or object. In this setting I-Con reproduces standard InfoNCE style learning over augmentation pairs popularized in SimCLR.

Improving motivation of Unifying Framework

We add a section further motivating the I-Con framework and detailing some of the things a unified framework provides. As noted by reviewer xJUE, this unification “not only provides a deeper understanding of these methods but also opens up the possibility of creating new methods by mixing and matching components.” We explicitly used this property to discover new improvements to both clustering and representation learners. In short, I-Con acts like a periodic table of machine learning losses. With this periodic table we can more clearly see the implicit assumptions of each method by breaking down modern ML losses into more simple components: pairwise conditional distributions $p$ and $q$.

One particular example of how this opens new possibilities is with our generalized debiasing operation. THrough our experiments we show adding a slight constant linkage between datapoints improves both stability and performance across clustering and feature learning. Unlike prior art, which only applies to specific feature learners, our debiasers can improve clusterers, feature learners, spectral graph methods, and dimensionality reducers.

FInally it allows us to discover novel theoretical connections by compositionally exploring the space, and considering limiting conditions. We use I-Con to help derive a novel theoretical equivalences between K-Means and constrastive learning (initial work) and between MDS, PCA, and SNE (In this rebuttal and in the new supplement). Reviewer xJUE mentioned that transferring ideas between methods is standard in research, but in our view it becomes much simpler to do this if you know methods are equivalent. Previously, it might not be clear how exactly to translate an insight like changing gaussian distributions to Cauchy distributions in the upgrade from SNE to T-SNE has any effect on clustering or representation learning. In I-Con it becomes clear to see that similarly softening clustering and representation learning distributions can improve performance and debias representations.

Fixing Figure 1

Thank you for raising this issue, we have corrected the label in Figure 1. If you would like to see the main figure modified in any other way please let us know.

We thank the reviewer for their thoughtful feedback and have worked to address each of the key concerns raised below. In addition we also added several other key improvements that we detail in the global response. If the reviewer finds these additions helpful we kindly request that they raise their score to help us share this work with the community.

1- New Series of Experiments Demonstrating that I-Con improves Feature Learning and works on different backbones and datasets.

In a new series of experiments we show that I-Con not only improves clusterers, but can improve contrastive feature learners. As requested by reviewers this experiment also demonstrates that I-Con works on an additional backbone and datasets. We follow the contrastive feature learning experiment put forth by Hu et al. at ICLR 2023. In particular we train a ResNet34 network contrastively on CIFAR10 and evaluate its linear probe and KNN performance on both CIFAR10 and CIFAR100. We include a detailed description of this experimental setting and its results in the Supplement. In summary, we show that our distribution improvement methods like debiasing and replacing gaussian kernels with cauchy kernels dramatically improves feature quality both in terms of linear probe accuracy and KNN accuracy.

2- Proving that MDS, PCA (Linear Autoencoders), Masked Language Modeling, Spectral Clustering, Mocov3, and Triplet Loss are unified by I-Con

We introduce new theorems that unify 6 additional methods with the I-Con framework. In particular we show that MDS and SNE can be unified in the limit of large gaussian scales, which is a novel theoretical result first proved in this work. As requested by reviewer iHot, this result allows us to integrate linear autoencoders (PCA) into I-Con. As requested by reviewer iHot we also show masked language modeling can be considered in the I-Con framework. Intuitively this is a simple corollary of our existing proof connecting Cross Entropy with I-Con, except replacing class embeddings with embeddings for masked tokens. Finally, we show that the commonly used triplet loss for contrastive learning emerges as in the limit of small width gaussians in I-Cons InfoNCE connection. These additional proofs can be found in the supplement.

3- Connecting I-Con with Variational Bayes and Mutual Information Maximization

We add sections relating I-Con to both Mutual Information maximization and Variational Bayesian methods, as requested by reviewer AnKo. Intuitively speaking, I-Con is linked to Mutual Information Maximization through a variational lower bound. I-Con is theoretically related to Variational Bayesian Methods as one view can represent hidden variables $h$ and the other observed variables $x$. We include detailed analysis and discussion of these intuitions in 2 new supplemental sections.

4- Providing guidance on choosing conditional distributions $p$ and $q$ and hyperparameters

As requested, we introduce a new section of the paper that details the different intuitions we have discovered using the I-Con framework. In particular many of these intuitions are based around choosing neighborhood distributions $p$ and $q$. These distributions should be carefully considered depending on the problem at hand, available information, and the desired output of the unsupervised learning method. However we highlight several generic probability distribution modification techniques that have generic interpretations as debiasers, or distribution smoothers. We show these general transformations yield both state of the art clusterers, and improved contrastive representation learners.

5-Improving motivation of Unifying Framework

We add a section further motivating the I-Con framework and detailing some of the things a unified framework provides. As noted by reviewer xJUE, this unification “not only provides a deeper understanding of these methods but also opens up the possibility of creating new methods by mixing and matching components.” We explicitly used this property to discover new improvements to both clustering and representation learners. In short, I-Con acts like a periodic table of machine learning losses. With this periodic table we can more clearly see the implicit assumptions of each method by breaking down modern ML losses into more simple components: pairwise conditional distributions $p$ and $q$.

One particular example of how this opens new possibilities is with our generalized debiasing operation. Through our experiments we show adding a slight constant linkage between datapoints improves both stability and performance across clustering and feature learning. Unlike prior art, which only applies to specific feature learners, our debiasers can improve clusterers, feature learners, spectral graph methods, and dimensionality reducers.

Finally it allows us to discover novel theoretical connections by compositionally exploring the space, and considering limiting conditions. We use I-Con to help derive a novel theoretical equivalences between K-Means and constrastive learning (initial work) and between MDS, PCA, and SNE (In this rebuttal and in the new supplement). Reviewer xJUE mentioned that transferring ideas between methods is standard in research, but in our view it becomes much simpler to do this if you know methods are equivalent. Previously, it might not be clear how exactly to translate an insight like changing gaussian distributions to Cauchy distributions in the upgrade from SNE to T-SNE has any effect on clustering or representation learning. In I-Con it becomes clear to see that similarly softening clustering and representation learning distributions can improve performance and debias representations.

6- Improving experimental robustness with multiple seeds

As requested by reviewer FvTp we report standard errors on our I-Con result in Table 2 to demonstrate that the improvements are indeed statistically observable.

7- Fixes, Correcting Typos, and other Clarifications

We have corrected several typos raised by reviewers, and tried to provide more elaboration on tpoints requested by reviewers. Finally we add a variety of sections into the supplemental material that provide detailed explanations and answers to many questions raised by reviewers including

1. Detailing the motivation of I-Con
2. Connecting I-Con with other literature reviews
3. Explaining of how to select distributions $p$ and $q$.
4. Explaining the relation between I-Con and Variational Bayes
5. Explaining the relation between I-Con and classical MLE inference
6. Clarifying the role of hyperparamter tuning and clarifying our claim that I-Con is self-balancing

Additional Experiments: Demonstrating I-Con's Effectiveness Across Backbones and Datasets

We are happy to share results from additional experiments conducted to evaluate I-Con's performance on feature learning. Beyond our new feature learning experiments on CIFAR-10 and CIFAR-100, we included evaluations on STL-10 (in-domain) and Oxford-IIT Pets (out-of-domain). Detailed descriptions of the experimental setup, along with result analyses and visualizations, are provided in the Supplement.

In summary, the results support that methods focused on distribution improvements—such as debiasing and replacing Gaussian kernels with Cauchy kernels—significantly enhance feature quality across datasets. This improvement is evident both in linear probe accuracy and KNN accuracy.

3. Questions on PCA: It's unclear to me what the PCA objective is taken to be - the last line of Theorem 2, $\mathcal{L} = -2 Var(X)$ seems to not depend on the projection matrix? Say that we try to find a low dimensional representation $\mathbf{Z} \in \mathbb{R}^{n, l}$, which is related to the high-dimensional data $\mathbf{X} \in \mathbb{R}^{n, d}$ in a linear way: $\mathbf{X} = \mathbf{ZW}$. The objective typically involves $tr(\mathbf{X}^T\mathbf{X} (\mathbf{W}^T\mathbf{W})^{-1})$ - this can be seen through a least squares approach, or through a probabilistic perspective as in [3]. Inference for $\mathbf{Z}$ intead of $\mathbf{W}$ can be done instead in a very similar way [4].

   Maximum likelihood/MAP interpretations have a KL minimisation interpretation, but note here that both distributions p and q are either Gaussian (if we consider the data directly [4, Sec 2.6.1]) or both Wishart (if we consider the covariance matrix $\mathbf{XX^T}$ or $X^TX$) [1].

   If I'm understanding correctly, there is a discrete random variable considered and not the data here directly, and you show that KL minimisation here recovers PCA? That's super cool but I feel like there are some details missing, e.g. in Theorem 2, regarding the objective.

4. On SimCLR: I'm not very familiar with this work, but in [5], SimCLR was shown to be a KL minimisation algorithm with a von-Mises-Fisher distribution on the latents (distribution "p"), and the variational constraint ("q") is degenerate. I think there's again a difference in the random variable on which a distribution is placed, but it'd be amazing if the authors could verify consistency between these interpretations (as it's suggested in this work that the distribution is Gaussian instead of vMF? These are similar, in the latter case, the embeddings just have unit norm).

References

[1] Dimensionality Reduction as Probabilistic Inference, Ravuri et al. (2023). Accessible here.

[2] Towards One Model for Dimensionality Reduction, Ravuri et al. (2025). Accessible here.

[3] Probabilistic Principal Component Analysis, Tipping and Bishop (1999).

[4] Probabilistic Non-linear Principal Component Analysis with Gaussian Process Latent Variable Models, Lawrence (2005).

[5] Representation Uncertainty in Self-Supervised Learning as Variational Inference, Nakamura et al. (2023)

[6] A Probabilistic Graph Coupling View of Dimension Reduction, Van Assel et al. (2022)

[7] On UMAP's True Loss Function, Damrich et al. (2021)

[8] From t-SNE to UMAP with contrastive learning, Damrich et al. (2022)