Cross-Entropy Is All You Need To Invert the Data Generating Process

We thank Reviewer wFAY for praising our identifiability result for being “both new and interesting” and for the constructive feedback regarding phrasing and experimental evaluation.

W1: Claim strength

Thank you for pointing out the underspecification of our claim. We changed our framing in all text instances to include that besides a cross entropy-based objective, it is also crucial to assume a certain data generating process. We revised the manuscript accordingly.

W2: Uniformly distributed instance labels

Thank you for pointing this out. We agree that, for the supervised case, this assumption can often be unrealistic. We moved out the assumptions used for Thm. 2. into Assms. 3, where we clearly state and discuss this limitation.

In the self-supervised instance discrimination case, the assumption that the instance label is uniformly chosen from the set of instances is a very natural one and doesn’t restrict the scope of Thm. 1. The instance label only refers to index of the sample within the training dataset and not to the actual class/cluster. In fact, the set of instances belonging to the individual clusters may be highly imbalanced and our theorem still applies.

W3 and Q1: Explaining lower R2 scores for higher latent dimensionsionalities and conditional concentrations

Thank you for pointing out the need to better explain why identifiability performance can suffer, even though our assumptions are satisfied.

The increasing difficulty of recovering higher dimensional ground truth latents is especially often encountered in prior works on identifiability under SSL methods [1,2,3]. In a DGP-agnostic setup, the issue of extracting large dimensional representations from practical, real-world datasets is also an open question (see e.g. [4, 5]).

We speculate that a major reason is the low signal-to-noise ratio of the gradient when computed on a finite batch size. The intuition is that most SSL objectives (including DIET) saturate, i.e. get close to optimum as the underlying pretext problem (in our cases the instance discrimination) is nearly solved. However, relatively low-dimensional features are sufficient to achieve this near-optimum discrimination. Thus, the population gradient has a very low norm. Nevertheless, as the empirical loss is calculated based on a finite batch size, the variance of the gradient overtakes the norm, and the training effectively stalls.

For example, with the higher concentration values intra-cluster samples become more indistinguishable. This implies that the uncertainty in predicting the correct instance label increases, i.e. the loss function has a very high optimum value. Achieving close to optimum instance discrimination loss is easy with relatively coarse-grain features that are sufficient for discerning samples coming from different clusters. In the revised manuscript, we ablated the effect of high concentration under increasing batch size in Appx. D and Fig. 5 Left. The results suggest that this issue can, at least theoretically, be mitigated. Appx. D also contains this discussion.

For the increased dimensionality $d=20$, we also ran an ablation under increasing batch size. With substantially increased training times, we saw surprising gains in recovery score across all batch sizes. In the unnormalized $w_i$’s case, for dimensionality $d=20$ and batch size $1024$ we improved the identifiability score from the previous 81.9% (Table 2) up to 85.7%. Nevertheless, we could not yet achieve complete identifiability. Settings with much larger batch sizes (2048-65K) seem to benefit from the increased training time even later than the settings with smaller batch size. Due to this reason, we cannot yet confirm a scaling law of the $R^2$ score wrt batch size and we postpone the full ablation until post-rebuttal.

It is important to mention that none of these observations contradict our theory, as in the latter we made statements about the global optimizer of the DIET population loss. In practice, the additional challenges of estimation error (due to finite sample size and finite batch size) and algorithmic error (using GD-based methods to solve a likely non-convex problem) may impose adverse effects on the evaluation.

We note that Reviewer vcau had very similar questions. Thus, our updated submission provides an extensive empirical evaluation of different kinds of assumption violations in Appendix D, along the same line of reasoning.

Bibliography

[1] Zimmermann et al. (2021): “Contrastive Learning Inverts the Data Generating Process”

[2] Von Kügelgen et al. (2021): “Self-Supervised Learning with Data Augmentations Provably Isolates Content from Style”

[3] Rusak et al. (2024): “InfoNCE: Identifying the Gap Between Theory and Practice”

[4] Simon et al. (2020): “On the Stepwise Nature of Self-Supervised Learning”

[5] Jing et al. (2021): "Understanding Dimensional Collapse in Contrastive Self-supervised Learning"

We thank Reviewer vcau for praising our experimental evaluation, including cases of assumption violations, and our submission’s clear writing. We are also grateful for pointing out unclear parts and for the constructive suggestions, which we answer in the following.

W1 & Q2 & Q3: Declining identifiability with increasing dimensionality and concetration/Potential violation of our assumptions

As the Reviewer noted, increasing the concentration parameter or the latent dimensionality is challenge in the empirical demonstration of the identifiability result.

On one hand, the increasing difficulty of recovering higher dimensional ground truth latents is especially often encountered in prior works on identifiability under SSL methods [1,2,3]. On the other hand, in a DGP-agnostic setup, the issue of extracting large dimensional representations from practical, real-world datasets is also an open question (see e.g. [4, 5]).

We speculate that a major reason is the low signal-to-noise ratio of the gradient when computed on a finite batch size. The intuition is that most SSL objectives (including DIET) saturate, i.e. get close to optimum as the underlying pretext problem (in our cases the instance discrimination) is nearly solved. However, relatively low-dimensional features are sufficient to achieve this near-optimum discrimination. Thus, the population gradient has a very low norm. Nevertheless, as the empirical loss is calculated based on a finite batch size, the variance of the gradient overtakes the norm, and the training effectively stalls.

For example, with higher concentration values intra-cluster samples become more indistinguishable. This implies that the uncertainty in predicting the correct instance label increases, i.e. the loss function has a very high optimum value. Achieving close to optimum instance discrimination loss is easy with relatively coarse-grain features that are sufficient for discerning samples coming from different clusters. In the revised manuscript, we ablated the effect of high concentration under increasing batch size in Appx. D and Fig. 5 Left. The results suggest that this issue can, at least theoretically, be mitigated. Appx. D also contains this discussion.

For the increased dimensionality $d=20$, we also ran an ablation under increasing batch size. With substantially increased training times, we saw surprising gains in recovery score across all batch sizes. For instance, in the unnormalized $w_i$’s case, for dimensionality $d=20$ and batch size $1024$ we managed to improve the identifiability score from the previous 81.9% (Table 2) up to 85.7%. Nevertheless, we could not yet achieve complete identifiability. Besides, settings with much larger batch sizes (2048-65K) seem to benefit from the increased training time even later than the settings with smaller batch size. Due to this reason, we cannot yet confirm a scaling law of the $R^2$ score wrt batch size and we postpone the full ablation until post-rebuttal.

It is important to mention that none of these observations contradict our theory, as in the latter we made statements about the global optimizer of the DIET population loss. In practice, the additional challenges of estimation error (due to finite sample size and finite batch size) and algorithmic error (using GD-based methods to solve a likely non-convex problem) may impose adverse effects on the evaluation.

Bibliography

[1] Zimmermann et al. (2021): “Contrastive Learning Inverts the Data Generating Process”

[2] Von Kügelgen et al. (2021): “Self-Supervised Learning with Data Augmentations Provably Isolates Content from Style”

[3] Rusak et al. (2024): “InfoNCE: Identifying the Gap Between Theory and Practice”

[4] Simon et al. (2020): “On the Stepwise Nature of Self-Supervised Learning”

[5] Jing et al. (2021): "Understanding Dimensional Collapse in Contrastive Self-supervised Learning"

We thank Reviewer 8oBq for praising our submission’s “significant progress towards better theoretical understanding on… classification tasks”, its potentially “substantial influence on theories of adjacent research areas as well as many applications”, and our “efforts to make the writing approachable” We are also grateful for the many constructive suggestions, for which we provide answers below.

Label Swapping Experiments

We appreciate the reviewer’s experimental suggestion, which provides an approach to evaluate the robustness of our method under increasing violations of the assumptions. In the revised manuscript, we have incorporated a new set of experiments to assess robustness to label noise, now detailed in Appx. D and Fig. 6. Appx. D also contains the following discussion.

For the DIET case (Figure 6 Left), the experiment was conducted as follows: after generating a sample, the instance label was perturbed with a probability equal to the label noise ratio (x-axis in the figures). Perturbed samples received a new label drawn uniformly from the set of all instance labels. For the supervised case (Figure 6 Right), the cluster label was perturbed instead. In both cases, the y-axis represents the identifiability score ($R^2$ score).

This experiment addresses points 1 and 2 of the Reviewer in a general manner, as the mislabeling involves all clusters rather than just two. Furthermore, it is a concrete interpretation of suggestion 3 (“swapping labels from multiple clusters”). We believe this setup reflects realistic scenarios, where label noise is equally likely to affect any data point, as commonly assumed in the literature [1,2]. Despite this increasingly challenging setup, the results demonstrate remarkable robustness to label noise. Up to an 80% label noise ratio, latent recovery shows minimal degradation, and the cluster recovery performance of DIET remains perfect. We attribute this robustness to the symmetry of label noise across all instances and labels. While increased uncertainty does reduce the accuracy of individual label predictions, the optimal logit values predicted by the encoder shouldn’t change under symmetrical label noise. The stability of deep learning models to label noise has also been shown by [3].

Experimental details on DisLib and ImangetNet-X

Thank you for pointing out the missing experimental details. We added the missing hyperparameters, the references to the implementation we used, the evaluation metrics and our evaluation protocol in Appendix C. We also provide our code as supplementary material and will release the repository in GitHub upon acceptance.

Bibliography

​​[1] Nettleton et al. (2010): "A study of the effect of different types of noise on the precision of supervised learning techniques."
[2] Frénay & Verleysen (2014): "Classification in the presence of label noise: a survey."
[3] Rolnick et al. (2017): "Deep learning is robust to massive label noise."

Dear Reviewer 8oBq,

Thank you again for your constructive feedback! We have updated our submission with your proposed experiments, described the requested experimental details, and attached our code as supplementary material. Please let us know if you have any further questions.