Simplifying DINO via Coding Rate Regularization

Dear Reviewer ubYD,

We deeply appreciate your insightful review and am pleased to see that you find our work intuitive to follow, and well-justified by extensive empirical studies. Here we attempt to address the concerns you have raised in the review.

3-1: This is a great suggestion. It is possible to apply coding rate to other representation learning methods, as it is a principled measure of non-collapse (we provide a sketch of theoretical justification in our response for Reviewer 5gcy). As you rightfully point out, we focus on simplifying DINO and DINOv2 in this work as they are widely used and attain SOTA performance. The study of extending the coding rate to other learning frameworks is interesting, but out of scope for this paper and we leave it for future work.

5-1: Thanks for your comment. As discussed in the previous point, this work focuses on the DINO family of models as they are very widely used and typically best-performing across various downstream tasks. In ImageNet classification results, we also have included additional baselines (e.g. MoCov3) for reference. With that said, we are happy to incorporate more baselines in the final version of the paper.

5-2: We would like to respectfully note that few-shot learning is not really necessary here, since we already obtain linearly separable features from DINO/SimDINO training (demonstrated by the superior $k$-NN performance). Due to this property, few-shot learning was not evaluated in the original DINOv2 paper. We hope this helps clarify and resolve your concern.

10-W1: Thank you for your question. We can attach training plots in the final version of the paper. As long as the balancing coefficient $\gamma$ is selected appropriately, the coding rate value will steadily increase, meaning that the features are spread out without collapse. If it’s too small, then feature alignment loss will dominate and lead to collapse (coding rate approaches 0). If it’s too large, the opposite happens, where the coding rate dominates. In this case, the features don’t collapse but feature alignment between global and local views is no longer enforced. We will follow your suggestion and put these analyses in the paper.

10-W2: We appreciate your suggestion. In our preliminary test, using regularization such as direct contrastive loss leads to worse performance than coding rate regularization. We will run and include additional experiments with other regularization methods in the revised version of the paper.

We are grateful to your suggestions and comments, which will no doubt improve the quality of our work. Please let us know if you have further questions or concerns, and we are happy to address them.

Dear Reviewer 5gcy,

We deeply appreciate your insightful review and am pleased to see that you find our work backed by solid empirical studies, with potential to improve practice of visual representation learning. Here we attempt to address the concerns you posed.

Theoretical justification for coding rate regularization

Thanks for your suggestion. It actually can be shown that when coding rate is maximized, it will induce the feature $Z$ to be full-rank. This can be proved by a spectral analysis as $\log\det(I+ Z^\top Z) = \sum_{i=1}^{\min(n,d)}\log(1+\sigma_i^2)$, where $\sigma_i$ is the $i$-th singular value of $Z$. Maximizing coding rate then leads to uniform non-zero singular values (as columns of $Z$ are $\ell^2$ normalized), making $Z$ full-rank. This means the learned features are spread on the unit sphere, avoiding a collapsed solution. On the other hand, the $\ell^2$ distance encourages the student features to be close to the teacher features on the same image, ensuring that we do not arbitrarily project different crops of the same image to large distances in the feature space. This is the extent of our added theoretical analysis for now; note that a full optimization theory-esque landscape analysis is very challenging and/or unreflective of practice due to the self-distillation (teacher/student) aspect, and we leave such analysis for future work. We hope these clarifications alleviate your concerns and we will put the relevant theoretical analysis in the final version of the paper.

The connection between MSE and dot product similarity

Thank you for pointing out this minor algebraic error. Note that in in eq.(7) we have included a factor of $\frac{1}{2}$, so the correct additive constant is +1. We will amend this in the final version of the paper.

We greatly appreciate your time in evaluating our work and your suggestions will certainly help improve the quality of our work. Please let us know if you have more questions or concerns and we are happy to address them.

Dear Reviewer nrJd,

Thank you for your insightful comments and suggestions. We are encouraged by your compliments on the simplification of the proposed method as well as the empirical evidence and presentation quality of our work. Here we attempt to address the concerns raised in your review.

Including more variants of regularizers and coding rate terms

We appreciate your suggestion. First, we use coding rate as the anti-collapse loss here due to its simplicity and theoretical grounding (also refer to our response to Reviewer 5gcy for further theoretical justification). In terms of its computation, we would like to clarify that there do not exist many variants (i.e. all the cited works compute the exact same quantity). Other than coding rate, we agree it is possible to prevent collapse with other choices of regularizers and is an interesting direction to explore. We have experimented with directly using contrastive loss, but the results lag behind our choice. We will include these ablations in the camera-ready version of the paper.

Other variants of DINO and DINOv2 pipelines

To the best of our knowledge, there aren’t any major variants of DINO and DINOv2 pipelines. Generally people either directly use the pretrained checkpoints or rely on the official implementations. We believe this can also be largely attributed to the complexity and fragility of the DINO pipeline, as we have demonstrated in Table 5 in the appendix. To compare our methods with the official pipeline, Figure 1 directly compares and summarizes the relevant changes.

Some issues on presentation

Thank you for your comments. Regarding the use of "pareto improvement", we respectfully assert that a "Pareto improvement" is when the whole Pareto curve is moved forward (i.e., when improvements are gained without any cost). You may have been thinking of moving along the Pareto frontier, which involves tradeoffs. For "vision SSL" (page 2, line 88), we want to suggest that simplification can be very valuable to the larger field of vision SSL, and our work pushes this broader envelope. Rather than adding more tricks and complexities, we show that identifying the necessary components (i.e. feature alignment while avoiding collapse) naturally leads to a greatly simplified design and improved performance. We believe such practices are useful for the larger field of vision SSL. If this causes confusion, we are happy to revise it. We hope these clarifications can resolve your concerns.

We again thank you for your time and effort in evaluating our work, which will no doubt improve its quality. We welcome your feedback and will be happy to engage in further discussion.

Dear Reviewer S9wV,

Thank you for the detailed review. Many comments, especially about presentation or typos, are helpful and will be incorporated in the camera-ready version; we do not address them here. Others require clarification or even stem from misunderstandings on your end, and we reply to these as follows.

Claims and Evidence

Training duration

Longer training usually leads to better performance, which is demonstrated in Appendix F.3. SimDINO still outperforms DINO with more training epochs. We are limited by compute and cannot adopt the exact same configurations as the original DINO(v2). With that said, we have since trained a ViT-B SimDINO model for 400 epochs with better performance in $k$-NN accuracy (76.6) than the official DINO model (76.1). This is still unfair comparison since the official setting uses full fp32 precision while ours only uses fp16 due to memory constraints. We will include more such results in our final version.

Additional baselines

Please see our response to Reviewer ubYD (5-1).

Training stability

It may seem contradictory but is actually not. For example, checkpoints at 100-th epoch in a 100-epoch run are very different from those in an 800-epoch run, due to difference in learning rate, teacher momentum, etc., and are incomparable. We also notice a similar performance dip in DINO trained for 200 epochs.

Computational efficiency

The coding rate computation is efficient since covariance matrices are PSD. We will include quantitative analysis in the final version.

Methods and Evaluation Criteria

Definition of $\mathbb{R}^{D \times *}$

$\bigcup_{t = 1}^{\infty}\mathbb{R}^{d \times t} = \{x \colon \exists t\ \text{s.t.}\ x \in \mathbb{R}^{d \times t}\}$ is the set of all finite-size matrices with $d$ rows. This is a standard definition for the symbol $\bigcup_{t = 1}^{\infty}$.

Definition of $\mathbb{S}_{d - 1}$

• The sphere in $\mathbb{R}^{d}$ is a $(d - 1)$-dimensional submanifold of $\mathbb{R}^{d}$. The definition including "$\mathbb{S}_{d - 1} \subseteq \mathbb{R}^{d}$" disambiguates this notation.

Definition of $\Theta$

Here $\Theta$ is the weight space. We will make this more explicit.

Multiple definitions of view

As is relatively common in SSL, we identify the function $v$ and the object $v(X)$ with each other for conceptual ease. We will make this more explicit.

Expectations over $X$

The expectation is over $X$; the variables $z, p$, etc., are functions of $X$. We plan to streamline this notation to show dependencies.

Typo in the loss definition

Thank you very much for pointing it out. We emphasize that this is only a typographical error, and we use the official DINO(v2) losses in experiments. We will fix this in the paper. However, it makes very little difference in the main body and Appendix B, as the symmetry is not important for our argument.

Parameter $\epsilon$

In this work, we only care that a smaller $\epsilon$ makes the regularizer larger. However, for completeness we will add a short description of the information-theoretic role of $\epsilon$.

MSE and similarity

Since the features are normalized on the sphere, an MSE term is a similarity term.

Covariance approximation

The provided argument applies only when the MSE term is small. Since we choose $\gamma$ to balance the terms’ relative sizes, a well-trained SimDINO(v2) model will have a small MSE term (induced by the feature alignment loss term), so the argument will apply.

Supplementary Material

Section name choice

We use this section label precisely because there is no self-distillation in this setting; one just tries to minimize the distance between two views' features, instead of taking one model as a privileged teacher from which the other network is distilled.

Experimental Designs and Analyses

Performance at scale

Our experiments (up to ViT-L and on ImageNet-1k) use very popular/standard settings in vision SSL. They are by-no-means considered to be limited scale. We are bound by our compute resources, and leave further scaling-up of our approach as future work.

Teacher/student weights

Your assertion is false; both DINO and our work use the teacher weights for evaluation. This is shown in the original DINO paper (Fig.6), where the teacher model consistently outperforms the student.

CNN backbones

As you point out, DINO(v2) perform strongly with ViT models, which are our focus as well. In particular, DINOv2 does not provide training on CNN backbones either.

Overall

Thanks again for your valuable feedback. We will fix many raised issues about prose and typos in the camera-ready version. We hope that our responses clarify the strength and correctness of our work. Given that we have attempted to address all of your main concerns and clarified several critical misconceptions of yours, we humbly request that you reconsider your recommendation for this work, as we believe it has good value to the community.