Is self-supervision enough for training sentence embeddings?

We would like to thank the reviewer for their helpful feedback.

[...] although the experimental results about the superior performance of cropping are somehow different from the conclusion in the SimCSE paper [Gao et al. 2021], there is no discussion or investigation about the reason. Gao et al. [2021] show that SimCSE is better than cropping and the next sentence pairing [Logeswaran and Lee, 2018].

In the SimCSE paper the authors indeed claim that dropout augmentations are better than crop augmentations; however, they only evaluate on STS tasks. Later benchmarks (e.g. MTEB benchmark paper) have found that performance on STS tasks does not correlate well with performance on other tasks, and that SimCSE model does not perform very well across the tasks. The main point in our work was to compare these two augmentation strategies under the exact same training setting, and we observed that dropout augmentations (as well as the out-of-the-box SimCSE model) perform substantially worse on all tasks except for the STS (in fact, our dropout fine-tuning did not lead to a good performance on STS either; it is unclear why the out-of-the-box SimCSE model performs better on STS). Apart from that, the SimCSE paper does not report any details on their "crop augmentation" setup, making it difficult to investigate this difference further.

Given the popularity and the fame of SimCSE paper, we consider our dropout-vs-crops comparison an important contribution.

Did the self-supervised training use test-split data too? If yes, the setting is a little tricky

Yes it did. The same concern was raised by another reviewer as well, so to demonstrate that this is not a problem, we repeated our experiments, this time excluding the test set from the SSL training data. The results almost did not change at all. In this table, the "Dropout" and "Crops" columns show results for SSL training on the full data (our original results), and the columns with "(train)" show results for SSL training only on the train set.

Given that this was raised as a repeated concern, we would be happy to switch to this kind of evaluation.

"We show that STS performance does not correlate with nearest-neighbor quality". Where is the result?

We evaluated kNN accuracy of the out-of-the-box SimCSE model and the results were poor on all tasks apart from the STS tasks (see Tables 1 and 2). We will reformulate this sentence to clarify.

Apart from that, we will add a Limitations section (as an example, our SSL training does not surpass SBERT, and it remains for future work to understand why exactly). We will also consider alternative titles, however we think that having "sentence embeddings" in the title already makes it unambiguous.

Minor comments

Why train using cosine but evaluate using Euclidean?

The Euclidean and cosine evluations gave very similar results (see Table S2, l. 843). We would be happy to switch to cosine evaluation in future revisions.

How was the learning rate chosen? [...] How do I know that your dropout results are not poor simply because you selected your learning rate based on what worked best for cropping?

See supplementary Section A.1 on hyperparameter choices. We have now assessed the performance of dropout augmentation with various learning rates, and found that it varied very little and was always substantially worse than with crops. We will add it to the paper.

Why was MPNet chosen as the model to do the experiments with? No justification is given. [...] I’d think it would make the paper stronger if you can show the same results for different models

As written on line 118, we chose MPNet because it is the base model used by SBERT. Please note that MPNet is not fundamentally different from any other BERT-based model. We applied our fine-tuning to many other models as shown in Figure 2 and described in Section 3.4.

Apart from that, we are happy to include suggested citations and adjust the introduction and discussion formulations, as suggested.

We would like to thank the reviewer for very helpful feedback.

The title is honestly just completely wrong — the work is not about whether SSL is “enough” for training sentence embeddings. The work, if I put it bluntly, proposes a tried and trusted method in machine learning: tuning on the test set. Please correct me if I’m wrong, but the approach fine-tunes on the same dataset that it is evaluated on, using essentially the same objective, so it is no surprise that the method outperforms its (weak) baseline.

There may be some confusion here. The SSL objective that we used for training and the evaluation metric are very different: the objective brings sentences from the same text close together in the representation space, without any use of class labels, while the evaluation metric quantifies how close different texts of the same class are.

To demonstrate that there was no overfitting, we repeated our SSL training process using only the training set, subsequently trained the kNN classifier again only on the training set, and finally evaluated the classifier on the test set. This way, the test set was never used for the SSL training. As you can see in the table below, this made almost no difference. In this table, the "Dropout" and "Crops" columns show results for SSL training on the full data (our original results), and the columns with "(train)" show results for SSL training only on the train set.

Does this address your concern? The same point was raised by another review, and so to avoid the confusion, we would be happy to switch to this kind of training/evaluation setup.

[...] the approach does not outperform SBERT. What is the value of applying this method if I could simply apply SBERT?

The goal of our work was not to propose a new approach or model outperforming others, but to study self-supervised training process for sentence embeddings and to systematically compare the two most popular augmentation approaches (which to the extent of our knowledge have not been systematically compared to date and have been reported in the literature as yielding conflicting results).

If the approach really works, couldn’t we apply it to SBERT to get SOTA?

This we have already done in the paper, at least for the ICLR dataset: shown by the green arrow in Figure 2 (change in kNN accuracy when finetuning SBERT with crop-based SSL). We have now repeated this experiment for all other MTEB datasets from Table 1 and the results are similar: SBERT representation quality does not improve after fine-tuning with crop augmentations.

But again, our goal was not to improve on SBERT, but rather to study how close and how fast can one get to its performance using only self-supervised learning (and what changes occur in the model during this fine-tuning).

In l.375-77 “We conclude that the majority of the improvement was due to generic sentence-level adaptation, while domain adaptation had a smaller effect” — it’s still a similar domain, scientific literature clustering? This claim is much too strong.

This is a fair point, and we agree that the PubMed dataset is not entirely "out of domain". We have now repeated this experiment using the Reddit dataset which is arguably more "out of domain". As the reviewer expected, this yielded slightly worse performance on ICLR compared to the training on PubMed dataset: with PubMed, we obtained 56.8% accuracy, while with Reddit we obtained 53.5%. However, this is still a large improvement compared to the starting MPNet performance (37.4%), so we still believe that the majority of the improvement is due to sentence adaptation (~75% of the improvement in this new experiment). We will include this experiment into the paper and rephrase this section accordingly.

We thank the reviewer for their comments.

The paper lacks originality, as most of its observations are already well-known in the research community [...]

The point of the paper is not to introduce a new technique, but to systematically compare the existing augmentation techniques under the exact same training setup (model, data, hyperparams, etc.). Among other things, we find that the cropping augmentation outperforms dropout augmentation (SimCSE) by a very large margin. We believe this is not a well-known result. If the reviewer believes otherwise, we would very much appreciate a reference.

In fact, another reviewer asked why our results contradict the claims given in the SimCSE paper. Our results cannot possibly be both "well-known" and contradictory or surprising given the literature! In fact, given the popularity and the fame of the SimCSE paper, we consider our dropout-vs-crops comparison an important contribution.

The study does not compare with current state-of-the-art models, such as BAAI's BGE models or commercial models (such as OpenAI [...]

Actually we did compare with state-of-the-art models, including commercial models such as embed-english-v3.0 by Cohere and text-embedding-3-large by OpenAI (see line 309). We found that on the ICLR dataset they performed similar to SBERT. Following the reviewer's suggestion, we now also evaluated BAAI's FlagEmbedding model (BAAI/llm-embedder) on the ICLR dataset and it yielded an accuracy of 62.7%, making it not much different from other commercial models and SBERT.

Its conclusions would be far more convincing if the proposed model was trained on the same or similar data as state-of-the-art models.

Most commercial models do not disclose which data they used for training. Furthermore, most SOTA models (including SBERT) are trained with supervised positive pairs, whereas our paper is explicitly about self-supervised learning only. Doing a comparison of the self-supervised and supervised contrastive training would be interesting, but this is out of the scope of our work; we discussed this idea in our discussion (line 510) and it is an interesting future research direction.

the evaluation is limited, omitting several popular benchmarks; for instance, many datasets from the MTEB benchmark are not included

We did not include all MTEB tasks in our paper for simplicity, but instead, chose multiple tasks from each evaluation modality of MTEB (clustering, retrieval, etc.). In addition to that, we evaluated kNN classification accuracy which is not even included into MTEB. We would be happy to add any specific evaluation, if the reviewer has concrete suggestions.