Hard View Selection for Contrastive Learning

Lack of Theoretical Analysis: The paper seems to focus on empirical validation but doesn't provide a theoretical foundation for why the "hard view selection" approach should work, which could limit its scientific rigor.

Thank you for your feedback. We also thought about ways to incorporate a theoretical foundation and validation to our manuscript. Given the nature of our study however, we found it challenging to find an easy enough framework where math/theory could be applied, especially given the dynamics and assumptions of the various baselines HVS is applied to in our manuscript. Note also that, to our best knowledge, little to no theory exists for the considered baselines in this paper. We appreciate your point on enhancing scientific rigor and would be grateful for any suggestions or insights you might have on how to integrate theoretical analysis more effectively into our work.

Unclear Impact on Convergence: Introducing harder samples could potentially slow down the convergence of the training process

Across all our experiments, we fortunately observed no slow-down in convergence during training. More broadly speaking, the training loss pattern remained identical to the baseline training. The only difference to the baseline trainings we were able to observe is that the training loss is slightly higher (see Fig. 7 in the supplementary material for an example of the training loss with/without HVS). This might encourage better generalization.

Could you elaborate on the computational overhead introduced by the Hard View Selection (HVS) method, especially when dealing with large-scale datasets or complex models?

Thank you for communicating this concern. We have addressed this point in the common comment to all reviewers, and we hope that the additional information provided adequately addressed your inquiry. If you have any further questions regarding the computational overhead, please feel free to let us know.

Could you provide more details on the complexities involved in computing sample-wise losses, particularly in the context of different neural network architectures?

Since most contrastive/non-contrastive learning approaches preserve the sample-wise losses within a batch in their objective function, applying HVS works out of the box. In the rare cases where approaches (or architectures) collapse the batch (or sample) dimension, HVS cannot be applied out of the box. One example where this is the case is Barlow Twins [1]. Here, the authors choose an objective function where the cross-correlation matrix between embedding outputs is computed to minimize the distance to the identity cross-correlation matrix. During the computation of the cross-correlation, the batch-dimension is collapsed by matrix multiplication that results in loss of any sample-wise information. To still apply HVS here, one potential approach is to explore the construction of sub-matrices from the full cross-correlation matrix. These sub-matrices could be designed to correspond to individual samples, allowing for the recovery of sample-specific information. It's worth noting that, as of now, we haven't conducted tests to validate the effectiveness of this approach.

[1] https://arxiv.org/abs/2103.03230

Does the introduction of "harder" samples through HVS have any noticeable impact on the convergence speed or stability of the training process?

When designing the HVS approach, we also thought of the potential risk of negative impacts with regard to convergence speed. However, empirically, we realized that HVS works extremely stably and has not impacted convergence speed across any of our experiments. Moreover, and as mentioned in the paper, it is also extremely robust to the baseline hyperparameters and does not require adaptation of these.

We express our gratitude once more for the insightful feedback provided.

[...] Ablation experiments regarding the initial number of view pairs that determine the hardness of the selected view pair is also provided.(higher initial pool of view pairs would lead to more adversarial view selection for CL training). The paper is well-written and easy to follow. The paper provides code for reproducibility.

We thank the reviewer for their constructive feedback and for acknowledging our efforts to make our work transparent and reproducible.

A detailed analysis of the HVS overhead in terms of running time/ wall clock/ FLOPs/ throughput along with memory requirement for each the baselines DINO (2x overhead), SimSiam(1.55x overhead), SimCLR. Preferably in a format of a table. [...] Extending on the above point, it seems since the overhead is added due to HSV, a more fair choice of x-axis in table 1 (and other tables) should be training time instead of epochs, as we see in table 1, that algorithm's performance increases with longer epoch training. For example assuming DINO + HSV is twice slower than DINO, it would be great to see a performance comparison between DINO+HSV at epoch 100 with DINO at epoch 200 (similarity with other baselines).

We have addressed the concern on computation overhead in the common comment to all reviewers. Please let us know if your concerns persist so that we can further discuss them with you.

It might be good to see some qualitative results on how good the learned features are with HSV strategy. For example, adding some attention maps as done in DINO paper, to get an idea if the attention maps also improves using HVS.

This is an interesting idea, which indeed may lead to insights as to what type of features HVS learns. We are working on this and will report back to the reviewer here once we have these figures.

As HSV is proposed as a general view sampling strategy it might be helpful to see some results on other relevant baselines like SimCLR_v2, DINO_v2, Swav. If some of the recent methods require huge amounts of compute or dataset, showing some results on ImageNet with lower training epochs would give the reader a better idea about the efficacy of the hard view selection strategy.

We thought carefully and long about the selection of baselines for our paper. Indeed, there exist other baselines as well. For DINOv2, we kindly note that it was not pretrained on ImageNet-1k but on the unreleased LVD-142M dataset and so there would be no way to add HVS and compare against it fairly. For SimCLRv2, we note that only a simplified version has been open-sourced by the authors [1]. For example, the public code of v2 does incorporate contrastive learning which is used in v1. In order for us to show that our method works with the original (contrastive) v1 loss, we decided against using v2. In the case of Swav, we notice that it comes from the same authors as DINO which is the more recent and better-performing method that, similar to Swav, also incorporates the multi-crop technique.

Regarding the reviewer’s comment on including a more recent baseline, we hope we were able to address it by adding the popular iBOT baseline (see general comment to all reviewers above). We believe this result again underpins the general applicability of HVS since it also works well with a MIM objective.

[1] https://github.com/google-research/simclr/issues/115

It could be a good idea to relate and mention in the sub-section of “Taking a Closer Look at the Intersection over Union” with the idea of “local-to-global” correspondences (as introduced in SimCLR [...]

We thank the reviewer once again for this valuable feedback point. Indeed, this is a rich addition to the existing description and builds a useful analogy to SimCLR as well as DINO through the local-to-global correspondences and multi-crop settings. We have added this to the end of the mentioned subsection in the paper.

How many seeds are used for Table 1 average computation. “Ref: averaged over multiple seeds unless otherwise mentioned”

Thank you for pointing this out. Our results are averages of 3 seeds (except for iBOT where we currently average over 2 seeds). We have updated the mentioned sentence in the paper to be more specific.

What is the time taken for doing 1 DINO forward call and 1 backward call as the reviewer was expecting the DINO+HSV to be slower than 2x.

Did we answer your question with the common comment to all reviewers and with the additional method-specific speed information provided there? Please do let us know if your concerns persist so that we can provide the missing information.

Dear reviewer, We wanted to follow up on your suggestion to create attention maps as a way to qualitatively study the potential effects and differences of the HVS learning method and their resulting features when compared to the baseline. We now added an overview of the generated attention maps under Section “J Attention Maps” in our supplementary material (last 3 pages), where we contrast 9 attention maps from a DINO ViT-S/16 100-epoch model for each HVS and the baseline.

All input images are from the ImageNet-1k validation split. The color code used depicts strong attention in yellow and weak attention in green and attentions from the HVS model are shown in the top row and attentions from the baseline in the bottom row, respectively. To summarize our take-away from this study, we do not see apparent strong differences between HVS and the baseline. What we occasionally notice is that the HVS models seem to capture the shape of some subtle/indistinct objects better (e.g. the creek/river in the valley in Fig. 13, the lizard in Fig. 14, the speaker in Fig. 15) or focus slightly more on the context and background (Fig. 17, Fig. 18, and Fig. 19). However, we note that these characteristics also often seem subtle. Would the reviewer agree?

The computational overhead introduced by the proposed method ("about a factor of 1.55× for SimSiam") is a bit unfair for the baselines. I wonder if this extra compute time can be used in favor of other models too, e.g. by training models longer. Cause longer training schedules often bring substantial gains for self-supervised methods. Also, it should be noted that the proposed method has seen more samples (due to encoding multiple pairs) which already impacted for instance batch-norm statistics (although loss is not backpropagated over unused pairs). Then it would be nice to see baselines processed 4x more samples.

Thank you for communicating this critique with us. We have addressed the concern on computation overhead in the common comment to all reviewers. Please let us know if your concerns persist so that we can further discuss them with you.

Regarding the concern about the models having seen more samples, we kindly disagree with the reviewer. First, the DINO and iBOT ViT-S models do not use batch norm but layer norm layers which do not keep track of running statistics and those experiments still show the same improvements. Moreover, even when batch norm layers are used, the batch norm learnable parameters are not affected by the HVS forward passes since we disable gradients for the selection step and hence, the BN parameters are not updated during the selection. Note also, that the running mean and variance buffers are only used for inference, see the pytorch documentation [1] which states:

“Also by default, during training this layer keeps running estimates of its computed mean and variance, which are then used for normalization during evaluation.”

During training, we keep our model in model.train() state so the running mean and variance are not used. Consequently, the HVS models do not have an advantage over the baselines in terms of having seen more samples.

[1] https://pytorch.org/docs/master/generated/torch.nn.BatchNorm2d.html?highlight=batchnorm2d#torch.nn.BatchNorm2d

We express our gratitude once more for the constructive feedback provided. Aiming to have effectively addressed your concerns, we kindly inquire whether there is an opportunity to increase your score. Alternatively, we welcome any additional points for improvement that would allow for a score increase.

As shown in Figure-3, positive pairs used for computing loss overlap less thanks to the proposed method, and this facilitates better representations. Tian 2020b already shows this phenomenon, i.e. there is a sweat spot in the IoU ratio which leads to the "optimal" performance. It is not clear what more this paper offers.

We thank the reviewer for pointing out the relevance of Tian2020b. Indeed, we agree with the reviewer that in hindsight we should have contrasted our paper in greater detail with Tian2020b and regret to not have done that in the initial version of the paper. Since also Reviewer #a1az has brought up this point, we are copy-pasting our response here, which we believe discusses the similarities and differences to Tian2020b comprehensively:

First, Tian2020b leverages the information of views to characterize good views for a given task based on the mutual information principle and use this framework to discover sweet spots for data augmentation hyperparameters. These data augmentation hyperparameters are widely adopted by SSL approaches and our baselines. Additionally, Tian2020b use their insights about good views to then adversarially learn flow-based models that generate novel color spaces for the small STL-10 dataset. After the views have been learned offline, they then perform standard contrastive learning on these generated views.

This approach is different to HVS on multiple fronts: with HVS we propose a method that can be integrated online into CL and non-CL pipelines without the use of a potentially instable adversarial learning objective. Our method is dependent on the model state and on individual samples since it is integrated online into the learning procedure, which is different to Tian2020b since their approach is not entangled with the model state.

While Tian2020b ablates also based on distance metrics in their study, they do not include the IoU metric. Moreover, another central difference is that we use the distance metric as a tool to better understand the selection of HVS and to discover that model-state and image-dependence is essential, where Tian2020b uses it as a motivation for their framework. Furthermore, our ablation studies include experiments that quantify the difficulty of predicting good/hard pairs, which Tian2020b does not have. In summary, we view Tian2020b as complementary to our work, particularly in the context of their derived data augmentation policies and believe that HVS could even be combined with the findings of Tian2020b. However, it's important to underscore the limited overlap in methodology and experimental design between the two approaches. HVS in the end could be seen as an early work that shows that for good views, we need to derive better sampling methods that can dynamically adapt to the actual state of training.

We have now added the expanded discussion on Tian2020b from above to the Introduction and Related Work sections of our paper.

[...] One distinct angle in this paper is the fact that multiple positive pairs are utilized during training (although loss is computed over only 1 pair in the end). I wonder what would happen if loss was computed over all crops, by simple averaging or weighted averaging (depending on the difficulty of a pair).

Thank you for this idea on averaging the losses. Please allow us to make sure that the method was correctly understood by the reviewer. We emphasize that the loss is computed for all pairs individually but only backpropagated for one pair (namely the one with the highest loss). Let us make sure we understood your suggestion correctly: instead of forwarding all view pairs and discarding all except the pair with the highest loss, you are suggesting to keep all pair losses and compute an average across all pair losses or smart averaging (by weighing the pairs with some difficulty score)? Assuming we understood your suggestion, we believe it would significantly ramp up the computation cost of the method since it would effectively multiply the batch size (since all activations for all pairs would have to be kept in memory for the backward pass). While applying HVS with a weighted loss that amplifies the gradient updates proportional to some difficulty measure sounds very exciting, but since we have no experience in this regard we lack the intuition about the consequences of such a modification with regard to training stability. Please let us know if we misunderstood your suggestion so that we can engage in a follow-up discussion.

The title says “contrastive” but most of the experiments use non-contrastive methods like simsiam and dino. This is confusing.

Thank you for pointing this out. We agree with you that the title is indeed misleading and may suggest that HVS is applicable to only contrastive learning. We changed the title to “Hard View Selection for Self-Supervised Learning”. Do you agree that title is better suitable?

In addition to the point above, it actually seems that the proposed method is better defined for non-contrastive losses. In fact, for contrastive losses (which contrast the attraction of the positives with the repulsion of the negatives) the method is not guaranteed to find the hardest possible set of positives and negatives. This would require exploring all the possible combinations of negatives, which is intractable. A better idea would be to use all the negatives from all the views at the denominator. In that case, since the denominator works as a max at low temperatures, you would be “guaranteed” to have the hardest negative. This is not described and discussed in the paper except one sentence at the end of 3.3.

We agree with the reviewer that this point is described vaguely and requires more clarity in the paper. We describe this in greater detail in the Appendix under “I.1 SimCLR”. If we understood the reviewer right, yes, that is correct and we updated I.1 with the following description for more clarity:

“For each iteration, we evaluate all possible view pairs and contrast each view against every other example in the mini-batch. Intuitively, the pair that yields the highest loss is selected, which is the pair that at the same time minimizes the numerator and maximizes the denominator in the above equation. In other words, the hardest pair is the one, that has the lowest similarity with another augmented view of itself and the lowest dissimilarity with all other examples.”

Did this address your concern and did we understand you correctly? We hope we were able to give some clarification and are happy to discuss further suggestions.

In algorithm 1, the authors report that they first organize the views into pairs and then forward each pair. This entails forwarding every view multiple times (one view can be in more than one pair). This is confusing and does not make sense at all. It would be better to first forward the views separately and then compare them 2 by 2. Can the authors please explain why they first create the pairs and then forward?

Thanks, you are right. In practice, we indeed only forward each view once to generate its embedding before we create the pairs over the embeddings. We fully agree with the reviewer that Algorithm 1 does not reflect this properly. For didactic reasons, we thought that it would be more understandable to a reader if the view pairs were first generated. However, after considering the reviewer’s feedback, we agree it would be better to stay closer to the actual (more efficient) implementation and updated Algorithm 1 accordingly. Thank you for pointing this out.

Why not just make all comparisons and backprop all of them? Why is this not ablated?

While we agree with the reviewer that the result of this experiment would certainly be interesting and valuable, its execution would require significantly larger compute resources because instead of selecting one pair from many and only computing the gradients for this pair, one would now have to compute and hold all gradients for all pairs which would drive up the memory and speed costs.

Our central arguments can be summarized as follows:

1. There are little to no diminishing returns when training longer with HVS (seen for 300 epoch trainings on ImageNet and for 4000 epochs on CIFAR100 in Fig. 5).
2. When normalizing Table 1 with respect to training time, HVS still yields slightly better performances, even without any of the many possible efficiency improvements.

Given these arguments, we see our current investigation as an early-stage exploration that highlights the novelty and efficacy of selecting more challenging views based on the model’s learning state for improving contrastive and non-contrastive learning performance. Our work may lay the groundwork for future explorations that could devise more efficient sampling methods to generate hard views and which in turn could benefit various SSL approaches.

We have revised the PDF to integrate the valuable feedback from the reviews, enhancing the clarity of our points (with ongoing refinements as the rebuttal progresses). To facilitate a quick turn-around, we have not fine-tuned the main paper to still fit into 9 pages during this discussion period, but would of course do so upon acceptance.

[1] https://arxiv.org/abs/2111.07832

[2] https://arxiv.org/abs/2006.09882

[3] https://arxiv.org/abs/2304.12210