• Another point is that the experimental setting is not practical and sufficient to demonstrate the effectiveness of GPS-SSL when evaluating self-supervised contrastive learn is that they only consider pretraining with 200 epochs, which is very few epochs required by SSL models to fully converge. As shown in SimSiam or many SSL (MoCo, BYOL, Barlow Twins, VICREG,... ) the performance is best achieved with long enough self-supervised pretraining (800-1000 epochs). As a result, the comparison in long training should be considered for both methods.

Response: We thank the reviewer for raising this limitation, we entirely agree with the need to provide a longer training schedule. We had previously reported results where most of the models were trained for 400 epochs, and some were trained for 200 epochs. The first change we have conducted following your suggestion is to make all the models in the reported tables in the submission with 400 training epochs. (Table 2 and 6).

Second, we performed a few additional experiments training for 1000 epochs with a ResNet18 using SimCLR (with and without GPS) which we report below (and in Table 3)

Where we employ strong augmentations (the standard SSL setting). We observe that GPS consistently improves performances and arrives on par with SimCLR on Cifar10 after 1000 epochs. Interestingly (and we thank the reviewer again for the suggestion that led to quantifying that benefit) we observe that GPS converges in fewer epochs than SSL. This is again another possible benefit of opening new avenues for informed positive pair sampling: faster SSL training. We hope that those novel experiments answered the reviewer’s empirical validation concerns.

• It is not clear what is the metric they have shown in Table 1. Reading its caption, it is challenging to capture what metric they are comparing, top-1 ACC or error or something else.

Response: We apologize for the confusion, the reported metrics are top-1 accuracy, we have made sure to state that clearly in the caption, thank you for raising that discrepancy.

• SSL contains another branch that is also very promising with the fine-tuning accuracy on downstream tasks such as MAE [1], this approach also depends very little on data augmentation (only cropping or without any augmentation already made the very good performance). This example (MAE method) will challenge the proposed method in terms of dependency on augmentation because the proposed method could not work without augmentation. I believe that modern SSLs should include this metric (fine-tune accuracy) and compare both contrastive learning and MAE approaches.

Response: We thank the reviewer for this suggestion. Indeed, once allowing for fine-tuning, we believe that most of those variants should produce comparable performances. However, we focused on a frozen backbone as a means to precisely assess the quality of the learned representations, as guided by the employed training criterion. In fact, as part of our motivation is to argue on the current sub-optimal positive pair sampling strategy, we resorted to assessing such a performance gap without fine-tuning. We will, however, clearly state this limitation and also mention in our contributions that our goal is to improve frozen backbone SSL.

• It should also include the linear evaluation of the only CLIP RN50 or supervised RN50 model when they have been used as the feature extractor for the downstream tasks on each considered dataset. It is to see without any training, how well these pre-trained model can perform, and based on that we can assess their contribution to the GPS-SSL (which is a combination of existing SSL + pre-trained CLIP/RN50).

Response: That is an excellent point and we thank you for mentioning it. We added a table that depicts the linear probe evaluation of a pretrained RN50 with CLIP on Cifar10 and FGVCAircraft dataset:

As it can be seen, GPS-SSL indeed improves the classification performance on both datasets when trained on a RN50 with strong augmentations, showing that GPS-SSL indeed contributes. Furthermore, we show that even when using a CLIP pretrained RN50, GPS-SimCLR outperforms SimCLR in all states, showing that the contribution is due to the better “positive sampling”, rather than using a pre-trained RN50.

• The proposed method needs a heavier component (such as a neural network ResNet-50) to generate the positive data sample, which is significantly computational compared to a simple calculation of data augmentation even for strong augmentations with a series of cropping, color jittering, distortion, hue, etc…

Response: Very good point; thank you for raising it. First of all, we will make sure to mention that explicitly in the introduction and when we derive our method. That being said, while the addition of another DN is clearly a more intense computational burden, we show that our method converges faster (thus needs fewer epochs, compensating for the extra cost). For example, we now provide a new experiment with a longer training schedule and obtain matching or improved results on both datasets with GPS-SimCLR and GPS-VICReg.

Also, it is possible that the NN sampling, relying only on the latent space representations, can be made on a saved bank of representations obtained from the NN architecture, i.e., no need to forward each iteration. We will mention that in the conclusion.

• With the aid of a strong knowledge (and heavy) model trained on millions or hundred million of data (CLIP, ImageNet) the performance of the proposed method brings minimal advantage even worse than the existing SSL method such as VICReg in Table 2 with strong augmentation. In the weak augmentation setting, GPS-SSL may give better performance but still lag significantly behind the optimal setting (strong augmentation) of both streams, making it questionable about the contribution of the proposed method.

Response: We thank the reviewer for raising that concern. First, we would like to mention that one important contribution of our submission is in demonstrating a new path for SSL methods: improving the positive pair sampling strategy. To that end, we did employ, in a few places, some pretrained embedding that had access to more samples than the SSL baseline (although this was already clearly stated in our manuscript when such results were reported). However, we completely understand that this may limit the applicability and reach of our claims. As such, we have proposed two new sets of experiments to address that concern. The first is to actually let the SSL baseline benefit from that same additional information by replacing the randomly initialized backbone with the same one being used by GPS’s embedding. In short, all methods now have access to the exact same extra information from the get-go:

We observe that even in this setting, GPS is able to consistently improve the final SSL representations compared to the baseline (more details in our general answer). The second experiment we propose is to employ GPS with an embedding model that is trained on the same dataset being considered, i.e., no additional knowledge is present in GPS training:

Where we observe that GPS is again able to match or improve the SSL baseline.

Dear Reviewer sJCd,

We hope that you have had a chance to consider our rebuttal. Your comments greatly helped us improve our manuscript, we sincerely thank you for that. We would like to kindly remind you that ICLR won't have a second round of discussion. As such, we would deeply appreciate it if you could raise any remaining concern or comment that you may have. This way, we will have a chance to clarify any last issue before the end of the discussion period which is now very near. If you consider your concerns addressed, then we hope that you will find time to revise your score and review accordingly. Thank you again for participating in the peer-review process.

Best regards, The authors

Thanks authors for their response. The rebuttal partly addressed my concerns. However, the additional results remain that the proposed method still lags behind the traditional SSL with the aid of the strong knowledge of the prior CLIP model (as mentioned in the weakness before).

• For example, in CIFAR-10 with optimal training (1000 ep), it clearly shows that SimCLR and VIGReg are still better than the proposed method (meaning that GPS failed to improve the SSL baselines in the optimal setting).
• For the FGVCAircraft dataset, although it can show some improvements, I think it is a very weak baseline, for example, paper [1] (Table 2) shows that SSL baselines such as MoCo-2 already achieved 52.54% and [1] even achieves 55.87% which is by far better than the results provided by GPS.

[1] Learning Common Rationale to Improve Self-Supervised Representation for Fine-Grained Visual Recognition Problems, CVPR 2023

I tend to keep my initial rating.

• The difference between GSP-SSL and NNCLR lies primarily in their respective positive sampling strategies. However, the novelty of the proposed strategy is limited.

Response: Our method is indeed inspired by NNCLR. However, GPS-SSL is a more general form of NNCLR. As mentioned in our general answer and in the theoretical part of our submission, GPS should be thought of as a general formulation of positive pair sampling using guidance that takes the form of prescribing the space in which nearest neighbors are sampled. That space, however, need not be given from another deep neural network but can be given from known data augmentations (thereby recovering baseline SSL), or from external knowledge such as querying an oracle. Last but not least, our formulation is identical for any SSL method employed (as opposed to NNCLR, which has been derived and tested uniquely on SimCLR). Our hope is that GPS can be seen as a general add-on akin to data augmentation that can be employed with any SSL loss.

• It would be better to make prior knowledge in an unsupervised manner, except using pretrained visual model and target dataset

Response: We entirely agree with that concern and have now provided numerous experiments to address it, including in the above answer to concern 1., and the general comment.

• The supervised results are supposed to be shown in Table 2

Response: We are sorry for the confusion, we have updated the caption for Tables 1 and 2 to make sure that we clearly state which method and which metric is being presented (classification accuracy).

• The employment of prior knowledge, specifically in the form of a pretrained visual model and the target dataset, diverges from the fundamental principles of Self-Supervised Learning (SSL)

Response: We thank the reviewer for raising that concern. While we agree with the fact that using a pretrained model as part of SSL training is a shift, we ought to emphasize that one of our core contributions is in bringing forward the importance of positive sample selection (which we often illustrated employing pretrained models). We believe that this point carries a lot of weight to guide future research direction in SSL: improving the sampling and selection of positive pairs has the potential to greatly outpace progress made around data augmentation and architecture search.

Additionally, we emphasize that even when we employed, for example, CLIP pretrained embeddings for GPS, such models did not have access to labels during training. That is, the empirical result remains in the realm of SSL. We entirely agree with the reviewer’s point for the cases where we used a supervised embedding, but again, our main motivation in doing so was to demonstrate how much richer SSL representations can be with a label-informed embedding. We made sure to clarify that point and use purely SSL embeddings, i.e., CLIP, in our manuscript in the main experiments (Table 2 and 6) and removed the supervised embeddings.

Furthermore, we hope to further address the reviewer’s concern with a new set of experiments (presented in Tables 8 and 9):

Which we hope demonstrate that even when no pre-trained embedding is accessible to GPS, and a new one needs to be re-trained on the given dataset (e.g., using MAE as in the above results), its use in GPS can yield final performances at least as good as the SSL baseline (SimCLR) or even greater (VICreg).

• The incorporation of such prior knowledge raises concerns about the fairness of comparisons with existing SSL methods. There is a potential risk that the pretrained visual model and target dataset might leak additional information into the model, thereby skewing results and leading to issues of unfairness.

Response: This is a great observation that needs to be addressed. First (as briefly mentioned in the above point), we have conducted new experiments where none of the DNN is pretrained, which showed great benefits of the proposed strategy. Another direction that we have also explored is to see what would happen if all the methods (GPS and the SSL baselines) had access to that extra information. To that end, we have conducted an experiment where we employed SSL and GPS training on CIFAR10 and FGVCAircraft starting from a backbone that is either random (realistic setting), Imagenet pretraining, or CLIP pretrained. That is, all methods have access to the same “leaked” information. We observe that even in that setting, GPS manages to produce greater results. This added experiment also further reinforces our main point: being able to correctly select positive samples is crucial to ensure that SSL training is successful, even when starting from a near-optimal initialization.

Dear Reviewer aLRr,

We hope that you have had a chance to consider our rebuttal. Your comments greatly helped us improve our manuscript, we sincerely thank you for that. We would like to kindly remind you that ICLR won't have a second round of discussion. As such, we would deeply appreciate it if you could raise any remaining concern or comment that you may have. This way, we will have a chance to clarify any last issue before the end of the discussion period which is now very near. If you consider your concerns addressed, then we hope that you will find time to revise your score and review accordingly. Thank you again for participating in the peer-review process.

Best regards, The authors

• With prior knowledge from another encoder, GPS has an advantage. Hence, comparison with methods that don't have prior knowledge might not be fair. Could the regular SSL (with augmentation) also use prior knowledge? For example, the encoder is initialized by prior knowledge and then regular SSL is performed.

Response: We thank the reviewer for raising that concern. Thanks to your suggestion, we have provided that exact experiment in response 1 above (and in the general comment). We believe that this experiment has been extremely insightful to use as it clearly highlights that the current SSL positive pair sampling strategy (eq (2)) is indeed suboptimal, as it moves away from a potentially better-initialized model (as opposed to GPS).

• Sec 4.1, how do you predict if the classes do not overlap in the training and test sets (unseen classes branches/chains)?

Response: Thank you for raising this concern. The paper that introduces the dataset mentions how they divide the total data into the train and the multiple test splits. More specifically, they explain that first, a set of chains (along with all their branches) is reserved for the D_{UU} to make sure the chains (super-classes) and branches (classes) are not seen during training. Next, out of the remaining chains, a set of branches is chosen to add all of their images to the D_{SU} test split (since the training set will have other images from other branches from the same chain, but not the same branch images). Finally, out of the remaining branches, the images in each are split between D_{SS} and train, creating the final test split that has a subset of the branches seen during training. With this procedure, they ensure the table of overlapping below:

• With prior knowledge, GPS seems to have an advantage over regular SSL, which generally does not use prior knowledge. According to Figure 1, data augmentation is used in GPS-SimCLR. So GPS seems to differ only in the use of prior knowledge to find positive pairs.

Response: We thank the reviewer for raising that concern (which was shared with other reviewers). We have provided two novel sets of experiments to demonstrate that GPS is able to help SSL learn richer representations by alleviating the sub-optimal positive pair generation employed by SSL that goes far beyond the use of additional knowledge that exists in the pretrained embedding. The first is to actually let the SSL baseline benefit from that same additional information by replacing the randomly initialized backbone with the same one being used by GPS’s embedding. In short, all methods now have access to the exact same extra information from the get-go:

We observe that even in this setting, GPS is able to consistently improve the final SSL representations compared to the baseline (more details in our general answer). The second experiment we propose is to employ GPS with an embedding model that is trained on the same dataset being considered, i.e., no additional knowledge is present in GPS training:

Where we observe that GPS is again able to match or improve the SSL baseline.

• Theorem 1: GPS-SSL: employing eq (2) or (3) into eq (1)?

Response: Eq (2) is the standard SSL solution to take a single sample, apply two different data-augmentations to it, and form a positive pair this way. Eq (3) is the proposed strategy obtained by applying the data-augmentation on two samples that are nearest neighbors in some given embedding space. Note that (3) can recover (2) in the limit case where we pick the sample itself as its nearest neighbor, we have put ``GPS’’ as part of eq (3) to make sure that the distinction was as clear as possible.

• Table 2: why are two different kinds of prior knowledge is used?

Response: We thank the reviewer for raising this concern. We have updated Table 2 accordingly to only use one prior knowledge (RN50_{CLIP} embeddings) for consistency and staying purely in the realm of SSL methods. In short, all employ GPS-SimCLR, the left table shows comparison using GPS with a few different embedding models, while on the right we use GPS-SimCLR with a few different hyper-parameters (to show its stability). All cases are on the same FGVCAircraft dataset.

• How is \tau set in Equation 3?

Response: That is a very important point, and we thank the reviewer for raising it. Based on the quality of the metric space g_{\gamma}, we can set \tau to be larger or smaller. For example, if we know g_{\gamma} is reliable, we can set \tau to larger values. However, if we are unsure about the reliability of g_{\gamma}, we can set \tau to smaller values to make sure that the positive pairs convey useful information rather than noise. In fact (as mentioned in our original submission) as tau gets close tk 0 as GPS recovers vanilla SSL positive pair sampling (we made sure to emphasize that observation which enables a qualitative guess for the value of \tau.

Dear Reviewer uQnq,

We hope that you have had a chance to consider our rebuttal. Your comments greatly helped us improve our manuscript, we sincerely thank you for that. We would like to kindly remind you that ICLR won't have a second round of discussion. As such, we would deeply appreciate it if you could raise any remaining concern or comment that you may have. This way, we will have a chance to clarify any last issue before the end of the discussion period which is now very near. If you consider your concerns addressed, then we hope that you will find time to revise your score and review accordingly. Thank you again for participating in the peer-review process.

Best regards, The authors