Thank you for your feedback! Below, we address your questions and concerns. We remain available to answer any additional questions you may have.

Relying on extra metadata and a pretrained model limits the applicability compared to the contrastive baselines. I also think this is a simple solution.

We agree that relying on extra metadata is limiting compared to traditional SSL methods. However, in this paper we focus on the best way to introduce the similarity information into training when it's available, and propose an objective for doing so. We find that this helps learn better representations compared to SupCon on ImageNet and CLIP on Conceptual Captions.

The idea of soft-targets for cross-entropy based contrastive loss is very general. It also lack some novelty to the paper's related work on soft-targets

We agree that related work has used soft targets, but that was done mostly for distillation. In our work, we focus on introducing the similarities without emphasis on where they come from. We experiment with different sources of similarities, including WordNet hierarchy distance, which doesn't require running another model at all. This approach, to the best of our knowledge, is novel, and tackles a different problem from distillation methods.

Suggestion: Finding a soft target graph without extra metadata or pretrained model with self-supervised techniques would give better applicability.

Thank you for the suggestion! We agree that methods that do not require metadata are less restricted in their applications, but in this work we explicitly focus on introducing the extra metadata by means of cross-sample similarities. The goal of this work is to show that modeling cross-sample similarities leads to better representations. This of course assumes having a source of those similarities. We agree that deducing similarities without needing metadata is an interesting direction for future work.

Thank you for your feedback! We are glad you find this paper well written and enjoyable to read. Below, we address your questions and concerns. If you have any additional questions, we remain available to answer them.

Notes regarding presentation.

It may be useful to shortly mention what $f_\theta$ and $K$ are after equation (1) in line 202 if space is not a concern.

Thank you for pointing out this omission. We added this to the main text in the revision.

In the data scarcity setting (figure 3a), SupCon appears to be better than X-CLR, while the text mentions that X-CLR improves over SupCon.

Thank you for pointing this out, we have corrected the wording in the main text in the revision. SupCon in this case indeed is better than X-CLR when trained with a subset of the data.

**The coloring of SupCon and SimCLR in figures 3a and 3b is flipped, which is a bit confusing at first glance. **

Thank you for pointing this out. We have fixed the coloring to make it consistent across the two plots in the revision.

The paragraph related to table 4 does not mention training on CC3M, which might be useful to mention for clarity.

Thank you for noticing this, we fixed this omission in the revision.

Questions

What happens in imbalanced settings? Do you think this would have a strong effect on the method? Have you tried experimenting with for example ImageNet-LT?

We haven't experimented with explicitly imbalanced settings, but we can see our experiments with conceptual captions as imbalanced data. In that dataset, we see that the similarities are much more skewed towards 0, as shown in figure 5a. When looking at the caption data in CC3M, we see that for example the word 'actor' occurs in ~125K captions, while the word 'athlete' occurs only in 18K samples, meaning that actor images will get more signal on average. Therefore, the data is already quite imbalanced, with athlete images coming up a lot less than actor images. In table 6, we see that although our method outperforms CLIP when trained on CC12M, better balanced ImageNet labels yield better representations. This leads us to the hypothesis that imbalanced data could be an issue when applying our method, although we believe that this issue can be addressed with custom data sampling. For example, to have more learning signal, we can construct the batch in a way that there are always some samples that are similar between each other, addressing the imbalance. This would be akin to hard negative sampling in contrastive learning (https://arxiv.org/pdf/2010.04592). We haven't investigated this in detail, but this could be an interesting direction for future work.

An annoying problem of contrastive learning is its sensitivity to the batch size (B)... I imagine that X-CLR partially solves this problem, especially for larger batch sizes... Is this something you looked into? If yes, how sensitive is X-CLR to changes in the batch size?

In my understanding, contrastive learning usually requires a large batch size to sample enough diverse negatives to train a good representations (e.g. Figure 9 in SimCLR paper) . We hypothesize that X-CLR can extract more signal per batch due to soft similarities, and therefore doesn't require batch sizes to be that large. On the other hand, when the batch size is large, as you correctly noted, the weights of individual samples similarities will go down. However, like with contrastive learning, it's possible that having a bigger batch will help due to having more samples' pairs in each batch. We do note that change in batch size may require adjusting the parameter $\tau_s$, as we show in Figure 5b.

We didn't extensively experiment with various batch sizes, and used the fixed batch size dictated by the hardware capacity. It would be interesting to see how X-CLR performs with different batch sizes, but we leave this study to future work.

Do the similarities for ImageNet (shown in figure 5a) come from the way the text encoder input is constructed (so from the fixed template part)? If this is the case, do the similarities not give a false view of how strongly related images are?

Yes, your understanding is right. The histograms show actual values of $s_{i,j}$, i.e. values after the softmax with temperature $\tau_s$. The non-zero similarities are due to the fact that the template is shared, and therefore each caption has a shared part "a photo of a". We agree that this is not ideal, but our results show that although the similarities for ImageNet are almost never zero, the signal from relative similarities still helps the model learn better representations. We tried randomizing the templates used to generate captions and calculating similarity between samples using the average over a set of templates, but didn't see any improvement over just using a fixed "a photo of a _ " template.

Thank you for your feedback! Below, we address your questions and concerns. We remain available to answer any additional questions you may have. If we addressed your questions well, we kindly ask you to take that into account when reconsidering the evaluation of this paper.

Responses:

The paper should clearly separate the SSL methods on images from image-text methods such as CLIP and clearly define X-CLR modification for both setups.

We apologize for the confusion. We compare to other SSL methods on images because our method is based on SimCLR objective and introduces additional information to the loss. This makes SimCLR a natural comparison to analyze the effect of introducing extra similarities to the objective. In the section where we use Conceptual Captions data, we compare to CLIP because in our case we use text captions to build the similarity matrix. CLIP is then a natural comparison because it also uses both images and text during training.

We believe the difference from SimCLR and SupCon is explained in Section 3 and appendix 9. We additionally add a section in the appendix to discuss the difference from CLIP, see appendix 10 in the revision. We hope this clarifies this issue enough.

Figure 1.b presents a pseudocode for the X-CLR loss that is valid only when paired image-text data is available.

Thank you for pointing this out. We chose to show this pseudocode because that's the method we ended up using the most throughout the paper. That said, using a text encoder to calculate similarities is only one way to obtain the target similarities. To address your concern, we additionally introduce pseudocode for general similarities in the Appendix 11 in the revision. We also copy it here:

def training_step(images, sims):
    # encode images with the model we are training
    img_emb = img_enc(images)  # N x D_img
    # convert targets to distributions
    tgt_sims = softmax(sims / tao_s, dim=1)
    # calculate similarities
    img_sims = 
        softmax(img_emb @ img_emb.T / tao, dim=1)
    # the loss is the cross-entropy
    return CE(tgt_sims, img_sims, dim=1).mean()

sims here comes from the the target similarities graph. As you can see, neither the text encoder nor captions come up here at all.

The tables and comparisons should specify which models have access to additional data and/or additional pretrained models.

Thank you for pointing this out. We apologize for the confusion. To clarify the distinction, we modified Table 1 in the revision to clearly show which methods use labels. Please let us know if this addresses your concerns.

The analysis in section 4.5 that compares the overhead of X-CLR to SimCLR needs to account for the training time of the text encoder as well.

We agree that training the text encoder has a hidden computational cost. However, we choose not to include training time of the text encoder in this table because the similarities may be computed in a variety of ways depending on the data available, and using the text encoder is only one of them. We acknowledge that obtaining the similarities may be computationally expensive. However, in this table, we only aim to highlight that incorporating additional information into the loss function doesn't make it significantly slower.

Are the authors claiming X-CLR can be better than CLIP distillation methods?.

We apologize for the confusion, we are not claiming that our method is better than CLIP distillation methods. Our message is that our method is not focusing on distillation, but on introducing similarities coming from an external source, whatever that source may be. We experimented with a variety of similarities sources, including WordNet hierarchy distance which doesn't involve running another model at all. Having those similarities come from an already trained model (distillation) is just one approach. In that sense, our approach encompasses distillation.

I did not find a clear definition of X-CLR for image-text loss that allows for training both image/text towers even if it uses pretrained image and/or text pretrained models.

As you correctly pointed out, X-CLR doesn't introduce a cross-modality loss. In this work, we focus on training the image encoder by introducing cross-sample similarities. Using a text encoder is just one way of computing those similarities. As we show in Table 2 in the paper, similarities can come purely from the data, e.g. WordNet hierarchy distance. This is an interesting direction of research, but we leave the extension of X-CLR to training both image and text towers to future work.

Are both Tables 1,3,4 using pretrained frozen sentence transformers as the text encoder?

Yes, that understanding is correct. Unless otherwise specified, the experiments throughout the paper use frozen sentence transformer.

Thank you for taking the time to read our response!

We have updated the manuscript and replaced the claim that "X-CLR is more general that distillation methods" with "X-CLR is not a model distillation method". We apologize for this overreaching claim, and hope that this addresses your concern. We do not claim to be better than model distillation, we only aim to emphasize that we use similarities that can come from any source, not necessarily another model.

Thank you for the feedback regarding novelty and experiments. We did investigate how X-CLR affects training on different datasets (Tables 1, 3, and 4), and how the choice of the metadata affects the performance (Table 2), and, according to reviewers HZRT and R31V, these experiments are comprehensive. We will consider extending these experiments. In the meantime, please don't hesitate to share with us if you have any ideas regarding what experiments could strengthen this paper.

Thank you for your feedback! Below, we address your questions and concerns. We remain available to answer any additional questions you may have. If we addressed your questions well, we kindly ask you to take that into account when reconsidering the evaluation of this paper.

Responses:

There is limited novelty to the proposed contrastive learning loss function...

In this work, we focus on introducing similarities into self-supervised learning. We show that introducing similarities during pre-training improves representation quality. We see the loss function itself as a technical detail. While we agree that the loss-function by itself has limited novelty, the introduction of external similarities to the objective is, to the best of our knowledge, novel.

There not much detailed analysis of how this method related to existing ones beyond a comparison of the binary vs. soft label graph.

Thank you for pointing this out. We present an analysis comparing it to SupCon in Appendix A9. Additionally, we add a section to compare to CLIP in Appendix A10. Let us know if you have any other questions or concerns, or have concerns about comparisons to any other methods.

There is only one network architecture tested

We agree this is a limitation, and it would be valuable to have results for e.g. transformer based encoders. We chose to only use ResNet-50 as it's the most studied architecture in SSL community, and allows for faster iteration due to its size. We unfortunately do not currently have the computational resources to conduct experiments with bigger models, and leave this extension to future work.

Concerns regarding table captions and writing

Thank you for pointing this out. We have revised the table captions to include information about the conducted experiments. Please let us know if this addresses your concerns, or if you have other feedback on writing.

Some of the results you get for competing methods are far below what those methods report in their papers.

We apologize for this confusion. This difference you noted is due to the fact that we train all methods for 100 epochs. We acknowledge that SupCon benefits from significantly longer training (500 epochs), but train all methods for 100 epochs to due to computational constraints. We also mention this in the paper in appendix A7. We agree that this is a major detail, so it should be in the main text. To make this more clear, we added this to the main text in the revision.

In your CLIP baseline, are you actually training a vision encoder and language encoder jointly with a CLIP loss, or are you taking a pretrained CLIP model and performing training with X-CLR loss?

CLIP baseline is just a CLIP model trained with the CLIP loss throughout, we do not fine-tune it in any way. To compare to our method, we only take the clip encoder, and perform linear probing on top of the frozen encoder.

The only place where we used a pre-trained CLIP model is in our experiment with different similarity sources in table 2. There, we use a pre-trained CLIP text encoder to calculate target similarities for X-CLR.

Is X-CLR equivalent to CLIP training with a frozen / pre-trained text encoder in this case?

X-CLR with CLIP encoder similarities is still not equivalent to CLIP training with a frozen encoder in this case. CLIP loss uses contrastive learning, and pushes together positive pairs of text and image representations, while pushing the rest apart. X-CLR loss differs in two key aspects:

• During training, unlike CLIP, X-CLR does not push image representations to be similar to text representations. In our method, we do not assume we have access to text representations, we only assume access to sample similarity values. We work with similarities between image representations, while clip works with image-text similarities.
• CLIP loss is not 'soft', i.e. the similarities in the contrastive loss are 0 or 1.

We include a more detailed analysis in the appendix in the revision, see Appendix 10.

Does the similarity matrix end up being the same as SupCon?

The similarity matrices of SupCon and X-CLR are indeed similar, but are not identical. We show a more detailed analysis of this in appendix A.9. For ImageNet, as you have correctly pointed out, similarities between the samples of the same class are going to be 1 like in SupCon, but for samples of other classes, the similarities can be between 0 and 1. The exact amount of weight put on the positives coming from the same class is controlled by the softmax temperature parameter $\tau_s$, with the X-CLR loss becoming SupCon as $\tau_s \rightarrow \infty$. With lower temperature values, more weight will be put on the samples other than the true positive, see Figure 5b in the paper.

Thank you for your response! We respond to the points in your response below:

Novelty

We agree that the idea of the loss function is intuitive and straightforward, but we consider this a strength rather than a weakness. Although a similar objective has been used for distillation, using it to inject additional similarity information to the learning objective is, to the best of our knowledge, novel, and we don't think the simplicity of the idea detracts from the novelty.

Experiments

We agree that running ImageNet training for more epochs would be a great addition to the paper, we will do our best to add these results, but we don't have time to do this before the end of the discussion period unfortunately. That said, we disagree that running on ImageNet for more epochs is more important than other experiments we ran. In this work, we focused on extending the method to bigger datasets as opposed to focusing solely on ImageNet as was done in SupCon and SimCLR papers. We believe that experiments scaling our method to Conceptual Captions dataset with 12M samples in table 3 highlight scalability of our method to a more practical setting with longer training and a bigger dataset better than training for more epochs on ImageNet would. Running experiments on the full Conceptual Captions dataset with 12 million samples (CC12M) took 9 days for a single run, and took significant computational resources. We kept the number of epochs fixed, and compared to the ImageNet dataset with 1M samples, our experiments on CC12M amount to 12 times more gradient updates than experiments on ImageNet with 100 epochs.

CLIP loss similarity

This is a good point, we thought about this connection when building this method. If we use text similarity as is done in most of the experiments, we do not push image representations to be similar to text representations as is done in CLIP. As a result, the only thing carried from the text representations to image representations when training with X-CLR is the relationship between samples. The representations themselves can be different in terms of at least rotation and reflection. For example, let's define representations learned by X-CLR as $X \in \mathbb{R}^{N \times D}$, where N is the number of data samples, and D is the representation dimension. Then the pairwise similarities are: $S = XX^\top$ If we perfectly optimize the X-CLR objective we learn representations with similarities that are the same as those of a frozen text encoder. For the frozen text encoder representations $Y \in \mathbb{R}^{N \times D}$, we'd get $S_\mathrm{text} = YY^\top$. So, for the perfectly optimized objective, we'd get $S = S_\mathrm{text}$. But that doesn't mean that $X=Y$ as would be the case for using the CLIP objective. To show this, let us introduce an orthonormal rotation/reflection matrix $T \in \mathbb{R}^{D \times D}$, such that $T^{-1}=T^\top$. Then, if we rotate representations $X'=XT$, we get the similarity: $S' = X'X'^\top=XT(XT)^\top=XTT^\top X^\top = XX^\top = S$

This means that rotating and/or reflecting the representations doesn't change the similarities. Therefore minimizing X-CLR objective learns image representations that could be a rotation/reflection of the text representations.

We will add this explanation to appendix A.10. Unfortunately, we are no longer able to update the revision.

Thank you again for the insightful feedback! We remain available to answer any other questions you may have.