Exploring Structural Degradation in Dense Representations for Self-supervised Learning

Thank you for your time and effort in reviewing our paper! Please find our detailed responses below.

Q1: Broader validation on other tasks.

R1: We have conducted additional experiments on the depth estimation task. Following our linear probing setting, we freeze the model and train a linear layer to predict depth values on the NYU-depth v2 dataset. The results are presented in the table below:

Table 1: The SDD phenomenon on the depth estimation task, reporting RMSE (lower is better) on the NYU-depth v2 dataset.

As shown in Table 1, the SDD phenomenon also appears in the depth estimation task. The performance trend is similar to that observed in the segmentation task. These results confirm that SDD is a general phenomenon and is not limited to segmentation. While we cannot present the full performance curves due to NeurIPS 2025 policy, we will include them in the final version.

To further evaluate the effectiveness of the proposed DQE metric for depth estimation, we also compute Kendall's $\tau$ coefficient between the DQE metric and depth estimation performance. The results are shown below:

Table 2: The Kendall's $\tau$ coefficient between DQE and RMSE metric.

As shown in Table 2, the DQE metric is positively correlated with depth estimation performance. Although the DQE metric works across different downstream tasks, the Kendall's $\tau$ coefficient is lower compared to the segmentation task. There are two main reasons: 1) The DQE metric is derived from segmentation performance, and class separability may not fully reflect the needs of depth estimation. 2) As a regression task, the RMSE curve is less stable, making prediction more difficult. We plan to further improve the DQE metric to enhance its effectiveness for depth estimation.

Table 3: The DQE-based model selection results on the depth estimation task.

Table 3 shows that DQE-based model selection consistently improves model performance. This demonstrates that DQE is generalizable to different downstream tasks.

Q2: Lack of deep insight into causes of degradation.

R2: Intuitively, SSL aims to balance semantic alignment and the effective dimensionality of representations. These two objectives often involve a trade-off. For instance, fully uniform representations have high effective dimensionality but may lack separability; conversely, focusing too much on semantic alignment can lead to dimension collapse. We argue that the degradation seen in different methods is due to the failure to maintain this trade-off during training, which causes a bias toward one objective.

Q3: Issue with the evaluation setup.

R3: Thank you for your detailed review. In fact, we use only the last layer's embeddings for evaluation as is standard in SSL [1,2,3,4]. For ResNet50 and Swin Transformer, the fourth layer is exactly the last layer, as both have only four layers (stages). We apologize for any confusion caused by our previous description and have revised the experimental setup section for clarity.

Beyond this clarification, we also find it interesting and important to explore the SDD phenomenon at different representation layers. We evaluated the dense performance of various layers of DINO on the COCO-Stuff dataset. The results are shown below:

Table 4: SDD phenomenon in different layers.

While the SDD phenomenon is weaker in shallower layers, the overall performance trend remains similar. At the 6th layer, the SDD phenomenon is almost negligible, but the performance is much worse, and it is not suitable for downstream tasks.

Q4: Analysis of failure case in DINO.

R4: Thank you for your insightful comment. The main reason is that DINO uses an EMA update strategy, where the momentum coefficient $\beta$ increases from 0.996 to 1 during training. In the later training stages, the teacher model stops updating, making the student more likely to experience dimension collapse when aligning to a fixed target. As shown in Fig. 5, this leads to a significant increase in the dimensionality metric. While our metric captures this trend, the large value of $\lambda$ gives too much weight to the dimensionality metric, causing a mismatch between the final DQE score and actual performance. Therefore, for DINO, it may be necessary to dynamically adjust $\lambda$ together with the EMA momentum coefficient to better account for different training phases.

Q5: More discussion about clustering-based SSL method ReSA [5]

R5: Thank you for pointing out this related study. We find the idea of clustering-based optimization in ReSA [5] interesting and novel, and it achieves strong experimental results. To investigate whether clustering-based optimization affects SDD and the DQE metric, we reproduced ReSA and tested its dense performance. The results are shown below:

Table 5: The SDD phenomenon in ReSA.

Table 6: The Kendall's $\tau$ coefficient between the DQE metric and dense performance.

Table 7: The DQE-based model selection results on ReSA.

As shown, the SDD phenomenon also occurs in ReSA. This may be because clustering optimization is not directly designed for dense representations in ReSA. The DQE metric still aligns well with ReSA’s performance. Therefore, the main contributions of our paper remain valid for ReSA. While we have tried to reduce the SDD phenomenon by directly optimizing the DQE metric, we believe further work exploring clustering-based optimization for dense representations, or studying how SDD behaves on encodings and embeddings, would be valuable future directions. We will add the ReSA results to the final version as an important baseline and provide more discussions in our paper.

[1] Caron, Mathilde, et al. "Emerging properties in self-supervised vision transformers." Proceedings of the IEEE/CVF international conference on computer vision. 2021.

[2] Oquab, Maxime, et al. "Dinov2: Learning robust visual features without supervision." arXiv preprint arXiv:2304.07193 (2023).

[3] Zhou, Jinghao, et al. "ibot: Image bert pre-training with online tokenizer." arXiv preprint arXiv:2111.07832 (2021).

[4] Chen, Xinlei, Saining Xie, and Kaiming He. "An empirical study of training self-supervised vision transformers." Proceedings of the IEEE/CVF international conference on computer vision. 2021.

[5] Weng, Xi, et al. "Clustering Properties of Self-Supervised Learning." arXiv preprint arXiv:2501.18452 (2025).

Thank you very much for your valuable feedback and suggestions. We appreciate your positive comments on the SDD phenomenon and our proposed DQE metric. Below are our detailed responses:

Q1: Limitations discussion

R1: Thank you for your valuable comment. As discussed in Appendix G, our DQE metric primarily focuses on off-the-shelf performance estimation. Therefore, when the backbone is fine-tuned for an extended period or with a large prediction head, the estimated performance may become inaccurate. In addition, the DQE is designed as a general performance estimator. Explaining the model-specific causes of the SDD phenomenon, such as the degradation in MoCo v3 or DINO, is an interesting and important future direction. We will move the limitations discussion into the main paper in our revision.

Q2: Regarding the class separability factor in DQE, is it measured at the image level or the pixel level?

R2: We apologize for the confusion. Both the class separability and dimensionality measures are calculated on patch/pixel-level representations. We will explicitly clarify this at the beginning of the method section.

Q3: Why do different methods experience the SDD phenomenon for various reasons? Does increasing the dimensionality of MoCo v3 lead to better performance?

R3: Thank you for your insightful comment. In general, self-supervised learning aims to achieve both semantic alignment and high effective dimensionality in representations. These two objectives often require a trade-off. For example, a fully uniform representation ensures high effective dimensionality but may reduce class separability, while emphasizing semantic alignment can lead to dimensionality collapse. We argue that the degradation seen in different methods results from an inability to maintain this trade-off throughout training, which causes the model to become biased toward one side.

Addressing your question about MoCo v3, we conduct an experiment by incorporating our $M_{dim}$ term into the loss function of MoCo v3. We resume MoCo v3 from epoch 40 (when it starts to collapse) and train it for another 10 epochs. The results are shown in the following table:

Table 1: The performance of MoCo v3 with increased dimensionality.

As shown in this table, the effective dimensionality increases from 0.744 to 0.892. Consequently, the downstream performance is significantly improved by 3.2% mIoU. The results demonstrate that mitigating the spotted cause of SDD could effectively improve the performance.

Q4: The sub-Gaussian assumptions might not hold for real-world cases.

R4: Thank you for your constructive comment. We would like to restate the assumption:

Assumption: For all $j \in [K]$, given the representation set $Z^j$ of the $j$-th category, the representations $\\{z_i^j\\}_{i=1}^{N_j}$ in $Z^j$ are i.i.d. $R$-sub-Gaussian distributed, or formally: $\forall \lambda \in \mathbb{R}, \mathbb{E}[\exp(\lambda Z)] \leq \exp(\lambda^2R^2 / 2)$.

Intuitively, this means the tail of $Z$ decays at least as fast as that of a Gaussian distribution. Since "sub-Gaussian" does not refer to a specific distribution family, standard tests like the Kolmogorov–Smirnov or Shapiro–Wilk test are not applicable. Therefore, we use the Orlicz norm to assess whether the tail decay of $Z$ is sufficiently fast, following the theorem below:

Theorem (Equivalence of a sub-Gaussian Variable, Proposition 2.5.2 in [1]): Let $\psi_2(x) = \exp(x^2) - 1$ be the Orlicz function of sub-Gaussian. The $\psi_2$ norm of random variable $Z$ is defined as: $\|Z\|\_{\psi\_2} = \inf\\{t > 0\~|\~\mathbb{E}[\psi_2 (|Z|/t)]\leq 1\\}$. We have $Z$ follows a sub-Gaussian distribution if $\|Z\|_{\psi_2} < \infty$.

Based on this theorem, we assess conformity to the sub-Gaussian assumption by computing the corresponding $\psi_2$ norm as a quantitative measure. Specifically, we select models trained with EsViT, iBOT, and SwAV, extract the dense representations on COCO, and compute the $\psi_2$ norm for each dimension, reporting the mean values as follows:

Table 2: The $\psi_2$ norm of dense representations on COCO.

As a reference, the norm of the standard Gaussian distribution is $\sqrt{8/3} \approx 1.633$, while the norm for the actual representations is around 1.61 in most cases. This indicates a faster tail decay than the standard Gaussian distribution, thus satisfying the sub-Gaussian assumption.

Q5: The performance for other dense tasks like depth estimation is under-explored.

R5: Thank you for your constructive comment. We conducted additional experiments on depth estimation. Following our linear probing setup, we freeze the model and train a linear layer to predict depth values on the NYU-depth v2 dataset. The results are reported in the table below:

Table 3: The SDD phenomenon in depth estimation. We report the RMSE metric (lower is better) on the NYU-depth v2 dataset.

As shown in Table 3, the SDD phenomenon also occurs in the depth estimation task. The performance trend is similar to that observed in segmentation tasks. These results validate that SDD is a general phenomenon and not specific to segmentation. While we cannot present the full performance curves here due to NeurIPS 2025 policy, we assure you that these results will be included in the final version.

To further evaluate the effectiveness of the DQE metric on depth estimation, we also compute Kendall's $\tau$ coefficient between the DQE metric and depth estimation performance. The results are shown below:

Table 4: Kendall's $\tau$ coefficient between DQE and RMSE.

As shown in Table 4, the DQE metric positively correlates with depth estimation performance. Although these results show the effectiveness of DQE across different downstream tasks, we note that Kendall's $\tau$ coefficient is relatively lower compared with segmentation. There are two main reasons: 1) Our DQE metric is derived from segmentation performance, so class separability may not fully benefit the depth estimation task. 2) As a regression task, the RMSE curve fluctuates, making prediction more difficult. We plan to further improve the DQE metric to enhance its capability for depth estimation.

Table 5: DQE-based model selection results on depth estimation.

In Table 5, we further examine whether the selected models perform better on the depth estimation task. As shown, DQE-based model selection consistently improves model performance, demonstrating that DQE is generalizable to different downstream tasks.

[1] Vershynin, R., 2018. High-dimensional probability: An introduction with applications in data science (Vol. 47). Cambridge University Press.

We sincerely appreciate your time reviewing our paper! Following your detailed comments, we have conducted additional experiments to further support our claims. Our detailed responses are provided below:

Q1: Incomplete and Biased Empirical Coverage (major concern)

R1: Thank you for your constructive feedback. We acknowledge the importance of including these baselines to strengthen our claims. However, for each baseline, we need to reproduce the results and evaluate the per-epoch dense performance and the DQE metric, which is computationally intensive (requiring thousands of GPU hours per baseline). As a first step, we have included key baselines such as VICReg, VICRegL, and Mugs.

Table 1: The SDD phenomenon on additional baselines.

Table 2: The Kendall's $\tau$ coefficient between the DQE metric and dense performance.

Table 3: The DQE-based model selection results on additional baselines.

Based on the results in Tables 1 to 3, we conclude:

1. The SDD phenomenon appears in all additional baselines. In particular, the volume-maximization design, especially in VICRegL, can effectively reduce the SDD phenomenon. While Mugs achieves the strongest dense performance among all baselines, its last performance is still much lower than its best performance. Thus, we could expect an improved baseline if the SDD phenomenon is effectively addressed.
2. The proposed DQE metric is effective for these baselines, showing strong generalization to models trained with different paradigms. Consequently, model selection effectively improves the downstream performance.

With these new results, our empirical analysis is more complete. We also agree that it is necessary to present the results for BYOL and Barlow Twins. Although we cannot provide these results now due to limited computational resources, we promise to include them in the camera-ready version.

Q2: Misrepresentation of Prior Work

R2: Thank you for your valuable feedback. We have carefully checked the manuscript but it appears that we did not make any statements like "SDD is largely unaddressed." Nonetheless, we acknowledge that previous dense SSL approaches have significantly improved the dense performances. We will add a new subsection in the related work to discuss the idea of these dense SSL approaches. In addition, we agree that it is important to clarify the differences and contributions of our work compared to previous studies. The main distinctions are as follows:

1. Previous works such as DenseCL, PixPro, DetCon, CroC, CrIBo, and FLSL mainly focus on addressing misalignment within specific methods, often overlooking the generality of the SDD phenomenon. In contrast, we introduce a general metric (DQE) to quantify the severity of SDD.
2. Despite the advancements in methods like iBOT and Mugs, noticeable SDD still exists, indicating that the problem remains unsolved and requires further investigation.
3. Dense representation degradation can take various forms, such as those caused by domain shift or suboptimal training objectives. These types of degradation may occur throughout training. In contrast, we specifically target the degradation induced by the training dynamics of separability and dimensionality, which typically emerge in the later stages. This also explains why SDD still appears in iBOT and Mugs, i.e., the degradation we focus on is of a different nature.

We will revise the Related Work section to more clearly describe the connections with prior work and discuss the contributions of the aforementioned methods.

Q3: Mismatch Between SSL Paradigm and the DQE Assumptions (major concern)

R3: Thank you for your insightful comment. In this paper, we propose the DQE metric to address the negative impact of the SDD phenomenon, focusing on comparisons between checkpoints trained with the same method. We acknowledge that as you pointed out, the current version of DQE is incapable of cross-paradigm comparison, such as between MAE and DINO. In the current design, cross-model comparison can be achieved by first selecting the best checkpoints with DQE, then comparing the downstream performances of these checkpoints. This would also significantly speed up the evaluation compared to simply testing all the checkpoints.

To further test the robustness of DQE to the choice of $k$, we conducted additional experiments:

Table 4: The Kendall's $\tau$ coefficient between DQE metric and dense performance with different $k$.

From the results in Table 4, we see that the Kendall's $\tau$ coefficient is insensitive to the change of $k$, regardless of whether the pretraining paradigm is clustering-based, contrastive, or generative. This demonstrates that our fixed $k$ strategy is sufficient for our needs.

Q4: Ambiguity in DQE's Scope of Comparison

R4: We apologize for the confusion. You are correct that DQE is used for intra-method comparison. We will add an explicit explanation in the methodology section and incorporate the discussions in R3 to our paper.

Q5: No Visual Validation of Clustering Geometry (major concern)

R5: Thank you for your valuable comment. To visualize dimensional collapse, we first project the representations onto principal components. We then divide the dimensions into eight segments based on the eigenvalues and perform t-SNE visualization for each segment. We observe that the structure of the top-k principal components matches the class separability, while the bottom components quickly collapse to a single point as the dimensional index decreases, confirming consistency between the two. We also compare the t-SNE plot between the last model and our selected model. The selected models have more compact intra-class representations and larger inter-class distances, indicating a better class-separability compared with the last models. Although we cannot present figures here due to NeurIPS 2025 policy, we promise to include these qualitative results in the final version.

Q6: Lack of Efficiency and Overhead Analysis (major concern)

R6: Thank you for your constructive comment. Indeed, efficiency analysis is important to determine whether the proposed method is practical at scale. We present two experiments below:

1. DQE-based model selection balances efficiency and effectiveness.

Table 5: Effectiveness and efficiency analysis of different model selection methods.

Table 6: Runtime analysis of different components of DQE (measured in seconds on a single 4090 GPU).

Loss-based model selection or early stopping does not reliably reflect dense performance due to the SDD phenomenon, while supervised model selection is too expensive in practice. In contrast, the proposed DQE-based model selection achieves strong performance with minimal overhead. We will also include these results in Table 3 of our paper.

2. DQE regularization improves the best performance with an acceptable cost.

Table 7: Runtime analysis of DQE regularization with 50 epochs of training on DINO.

From Table 7, DQE regularization introduces about 22% additional training time, which we believe is acceptable. Importantly, DQE regularization improves the best performance achieved by DINO. This demonstrates the potential of integrating DQE into training, and we will further explore ways to speed up DQE regularization by improving the clustering and metric calculation.

Q7: Lack of Scaling Study

R7: Thank you for your constructive comment. To study the effect of model size on the SDD phenomenon and the DQE metric, we conducted an additional experiment on iBOT with ViT-Base-16. The results are shown below:

Table 8: The SDD phenomenon on the larger backbone.

Table 9: The Kendall's $\tau$ coefficient between DQE metric and dense performance.

Table 10: The DQE-based model selection results.

The main conclusions still hold at this larger scale: The SDD phenomenon persists and the DQE metric can still accurately predict the trend of dense performance. Due to limited computational resources and the timeline for the rebuttal, we will conduct more experiments on larger models in the updated version.

We sincerely appreciate your detailed and constructive comments, which significantly help us improve the quality of our manuscript. Below are our detailed responses:

Q1: The paper lacks ablation studies showing the effect of each DQE component when used as a regularizer.

R1: Thank you for your insightful suggestion. Below are the ablation results for the DQE regularization:

Table 1: Ablation studies of different components of DQE regularization.

We draw three main conclusions from these results:

1. Different components of DQE separately contribute to improving class separability and effective dimensionality. Consequently, this improves downstream performance, which aligns with our theoretical analysis of the DQE metric.
2. The performance improvement mainly comes from addressing model-specific degradation. For example, DINO's performance improves primarily due to enhanced class separability, whereas resolving dimensional collapse notably boosts dense-task performance in MoCo v3. These observations align with our model-specific analysis presented in Sec. 5.2.
3. When both factors are used together, class separability often dominates optimization. Thus, introducing a hyperparameter to balance these factors may further enhance performance.

Q2: Evaluate on other dense tasks.

R2: We conducted additional experiments on the depth estimation task. Similar to our linear probing setup, we froze the model and trained a linear layer to predict depth values on the NYU-depth v2 dataset. Results are shown in Table 2.

Table 2: The SDD phenomenon in the depth estimation task, we report the RMSE metric (lower is better) on the NYU-depth v2 dataset.

As shown in Table 2, the SDD phenomenon persists in depth estimation tasks. The performance trends are similar to segmentation tasks, validating that SDD is a general issue, not specific to segmentation. Although we cannot show the performance curves due to NeurIPS 2025 policy, we promise that these results will be included in the final version.

Next, to evaluate if the DQE metric is effective for depth estimation, we computed Kendall's $\tau$ coefficient between the DQE metric and depth estimation performance. Results are shown in Table 3:

Table 3: Kendall's $\tau$ coefficient between DQE and RMSE metrics, since RMSE is lower the better, we multiply the RMSE by -1 before computing Kendall's $\tau$.

As shown in Table 3, the DQE metric positively correlates with depth estimation performance. Although the results confirm DQE's effectiveness across various downstream tasks, the Kendall's $\tau$ coefficient is lower compared to segmentation tasks. We attribute this to two reasons:

1. The DQE metric is derived from segmentation performance, the class separability measure may be less relevant for depth estimation.
2. As a regression task, the RMSE curve fluctuates, making predictions challenging. We aim to further refine the DQE metric for better performance on depth estimation.

We also study the effectiveness of DQE-based model selection on the depth estimation task:

Table 4: DQE-based model selection results for the depth estimation task.

As shown in Table 4, DQE-based model selection consistently enhances model performance, demonstrating DQE's generalizability across different tasks.

W3, Q3: Qualitative results.

R3: Thank you for this valuable suggestion. To visualize the dimensional collapse, we project the representations to the principal components. Then, we divide the dimensions into 8 segments based on the eigenvalues and perform t-SNE visualization for each segment separately. We observed that the top principal components align well with class separability, while lower components rapidly collapse into single points as the dimensionality decreases. These results validate the consistency between class-separability and dimensionality. Comparing the last and selected models, the selected models showed more compact intra-class and larger inter-class distances, indicating better class separability. Due to NeurIPS 2025 policy restrictions, figures cannot be presented here, but we will include them in the final version.

W4, Q4: How does DQE behave with image-level representations, such as classification?

R4: We appreciate your constructive comment! Technically, the analysis should hold for both dense and image-level settings, as image-level learning can be viewed as a special case where an image is only split into one patch. To investigate how DQE performs at the image level, we conducted the following experiments. DQE originally accepts input tensors shaped as (Num_images, Num_patch_tokens, Dimensionality). For image-level tasks, we first extract the class token (or averaged dense tokens), obtaining a tensor of shape (Num_images, Dimensionality). Then, we resize this tensor to (Num_images // 200, 200, Dimensionality) and calculate the DQE metric. For evaluation, we computed Kendall's $\tau$ coefficient between the adapted DQE and ImageNet $k$-NN performance, comparing it with the state-of-the-art unsupervised transferability estimation method RankMe. The results are shown in the following table:

Table 5: Kendall's $\tau$ coefficient between DQE and ImageNet $k$-NN performance.

As demonstrated, the adapted DQE metric is a precise estimator for classification, achieving an average Kendall's $\tau$ coefficient of 0.86, significantly surpassing RankMe. Additionally, DQE consistently performs well across methods (with lower variance), confirming its generalizability and suitability for predicting downstream task performance.

Q5: Why does SDD occur in dense tasks but not image-level tasks under SSL?

R5: Thank you for your constructive comment! The primary reason is likely the misalignment between SSL objectives and dense tasks. In image-level SSL, by aligning different parts of objects, the different objects within a class are pulled together [1]. While alignment between dense representations primarily happens in the same position [2,3,4], e.g., parts of the same object might not be sufficiently aligned. Hence, compared with image-level SSL, the dense representations of different objects within the same class are more likely to be dispersed in the representation space.

Q6: Causes of remaining uncertainty in DQE correlation?

R6: We appreciate the valuable comment. While the DQE metric could precisely estimate the trend of downstream performance with an average Kendall's $\tau$ coefficient of 0.54, we believe the remaining uncertainty arises from:

1. In our experiments, the downstream performance is primarily evaluated by training a classification head on dense representations. In the derivation of the DQE metric, the head is simplified to a nearest neighbor classifier for the sake of analysis. Simplifying the classification head to a nearest neighbor classifier may introduce bias compared to more complex heads in practice.
2. Since we do not have the downstream data or labels, we estimate the performance on the training set by k-means clustering. Estimating performance with k-means clustering on training data might introduce additional noise. For a more comprehensive analysis, please refer to Appendix E.

In future work, we aim to address these issues through refined theoretical analysis for more specific classification heads and more precise pseudo-labeling techniques.

[1] Huang, Weiran, et al. "Towards the generalization of contrastive self-supervised learning." arXiv preprint arXiv:2111.00743 (2021).

[2] Ziegler, Adrian, and Yuki M. Asano. "Self-supervised learning of object parts for semantic segmentation." Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2022.

[3] Zhou, Jinghao, et al. "ibot: Image bert pre-training with online tokenizer." arXiv preprint arXiv:2111.07832 (2021).

[4] Li, Zhong-Yu, Shanghua Gao, and Ming-Ming Cheng. "SERE: Exploring feature self-relation for self-supervised transformer." IEEE Transactions on Pattern Analysis and Machine Intelligence 45.12 (2023): 15619-15631.