Enhancing Unsupervised Sentence Embeddings via Knowledge-Driven Data Augmentation and Gaussian-Decayed Contrastive Learning

Thanks very much for taking your time to review the manuscript so seriously and responsibly. We sincerely apologize for our careless syntactic errors, and we will fix them in the revision. Please see the following for our response to your questions:  

Q1: Why the Gaussian-decayed function G() in the denominator? It seems that the Gaussian-decayed function has no connection with the denominator. Can we just add the Gaussian-decayed function to the contrastive learning loss?

A1: Thank you for your question. In terms of the loss function, the Gaussian-decayed function G() could, in principle, be separated and directly incorporated into the contrastive learning loss. However, including it in the denominator simplifies the overall expression, preventing the formula from becoming unnecessarily complex.

Q2: The main results are based on BERT and Roberta. However, LLMs are effective in sentence embedding [1,2]. Please use LLMs for evaluation. Using only ChatGLM-3 for evaluation is not convincing. Therefore, the experimental evaluation in this paper should be redone.  

A2: Thank you for your valuable feedback. We have expanded our experimental evaluation to include multiple LLMs for unsupervised sentence embedding tasks. Specifically, in Appendix F, we have added results for Llama3.2-3B, Llama3-8B, GLM4-9B, and Qwen2.5-14B, alongside ChatGLM3-6B, to assess the effectiveness of our data augmentation strategy across various models. The results are presented in Table 10.

From these additional experiments, we can confirm that our data augmentation method is effective across different LLMs. However, it is important to note that while some LLM-based methods have achieved impressive results on supervised datasets, LLMs do not show a significant advantage over BERT-base in the unsupervised scenario.

Moreover, our target scenario focuses on leveraging LLM knowledge to enhance the sentence representation learning capability of smaller encoder-based models. These models offer advantages in computational efficiency and resource requirements, enabling them to assist LLMs in tasks such as RAG (retrieval-augmented generation) for semantic querying. We hope these additional results and clarifications could address your concerns.

Q3: Analysis of \alpha and \beta.  

A3: Thank you for your question. We have conducted an ablation analysis of the hyperparameters α and β in Equations 10 and 11. The impact of varying these hyperparameters on model performance is detailed in Appendix H, with the corresponding results presented in Figure 10.  

Q4: Two missing references. [3] is about data noise and [4] is about data augmentation. ** **

A4: Thank you for pointing that out. We have added the missing references in the Introduction and Related Work sections to properly cite the relevant literature.

Thanks very much for taking your time to review the manuscript so seriously and responsibly. We have added the missing references, and please see the following for our response to your questions:  

Q1: 1、Why was a Gaussian-decayed function chosen?

A1: Thank you for your question. We chose the Gaussian-decayed function to balance sample diversity while minimizing the impact of noise from false negatives during training. Specifically, in the initial training steps, the Gaussian-decayed function helps align all hard negatives with the distribution of the evaluation model. As training progresses, it allows gradient weights for hard negatives that deviate more significantly from the evaluation model's distribution to recover gradually.  

Q2: 2、Why train a frozen encoder first, followed by training a new encoder with LLM-generated data? What's the benefit?

A2: Thank you for your question. The reason we first train a frozen encoder and then train a new encoder with LLM-generated data is to address the noise present in the synthetic data, particularly false positive and false negative samples. Due to the generative ability and potential biases of LLMs, it is difficult to eliminate this noise during the data synthesis process.

To mitigate this issue, we utilize an unsupervised evaluation model trained on data from the same distribution, which helps the GCSE filter out some of the noise present in the generated samples. Furthermore, for hard negative samples that are more challenging to classify, we use the Gaussian-decayed function to align with the evaluation model’s probability distribution during the initial training phase. It reduces the loss for all hard negative samples and ensures that the model is not influenced by noisy hard negatives in the early stages of training. This two-step process helps the model achieve better performance by first creating a robust evaluation model that filters out noise, and then training the new encoder with more reliable guidance.

Dear Reviewer,

We thank again for your contributions to the reviewing process. The responses to your concerns and the corresponding paper revision have been posted. Please let us know whether we have properly addressed your concerns. We look forward to your reply and welcome any further questions.

Best regards,

Authors of Paper: "Enhancing Unsupervised Sentence Embeddings via Knowledge-Driven Data Augmentation and Gaussian-Decayed Contrastive Learning"

Thanks very much for taking your time to review the manuscript so carefully. Please see the following for our response to your questions:  

Answer 1: Thank you for your insightful suggestion! We acknowledge that the choice of the threshold 0.5 may not be universally optimal for distinguishing positive and negative samples. We have carefully considered your proposed approach of selecting positives and negatives based on the top-k similarity scores. While this method is effective for binary classification tasks, we believe it might not adequately capture the fine-grained distribution of sentence embedding similarity scores, which is critical for sentence representation learning tasks.

To address this limitation, we introduced a new score normalization method to better align predicted scores with the target labels. Details of the normalization formula and its implementation have been provided in Appendix I. Following the normalization, we defined false negative samples as those with a root mean square error (RMSE) greater than 0.2 between predicted and normalized label scores. Positive samples were selected as those where the predicted score exceeds the corresponding label. We have updated Figure 1 with the recalibrated predictions based on this improved approach.    

Answer 2: Thank you for your question. Could you please clarify which specific results you are referring to for statistical significance testing? In our main experiments, we followed prior work and used the SentEval for performance evaluation. We validated our approach on multiple benchmark datasets, and some experiments were conducted with k-fold cross-validation. This methodology is designed to mitigate the influence of randomness in the results.

Answer 3: Thank you for pointing this out. We have updated Figure 4 to include a clearer annotation within the Gaussian-Decay function module, explicitly indicating the role of E′.  

Answer 4: Thank you for your question. We chose the Gaussian-decayed function to balance sample diversity while minimizing the impact of noise from false negatives during training. Specifically, in the initial training steps, the Gaussian-decayed function helps align all hard negatives with the distribution of the evaluation model. As training progresses, it allows gradient weights for hard negatives that deviate more significantly from the evaluation model's distribution to recover gradually. This smooth adjustment is a key advantage over alternative loss functions like MSE or Cross-Entropy.

We observed that using KL divergence loss exhibits instability in certain situations. This instability likely arises from the sparsity of the label distribution. For example, when probabilities for certain classes are close to zero or exactly zero, and the predicted probabilities approach one, the logarithmic term in KL divergence can become excessively large, leading to highly negative loss values. By contrast, the Gaussian-decayed function provides a more stable and effective way to achieve our training objectives.  

Answer 5, 6: Thank you for these questions. The alignment of hard negatives between the training encoder and the frozen evaluation encoder is motivated by the potential noise in LLM-generated data, which can include false positive and negative samples. Due to the generative ability and potential biases of LLMs, it is challenging to eliminate such noise during the data synthesis process.

The evaluation encoder, trained on data from the same distribution, serves as a noise-filtering mechanism to initially reduce the impact of noisy samples in the generated data. Secondly, for the false negative noise in the hard negatives, which is more difficult to distinguish. We aim to align the training encoder with the evaluation encoder’s probability distribution during the initial training step.

As for the two-stage training approach, it is necessary to first train the evaluation encoder in an unsupervised manner before using it to assist in training the new encoder with LLM-generated data. Combining these steps into a single stage would bypass the noise-filtering benefits provided by the evaluation encoder and could lead to suboptimal performance.

Answer 7: Thank you for your question. Initially, the gradients for hard negatives are relatively small due to the Gaussian-decayed loss. However, as training progresses, the model leverages other in-batch negative pairs to refine the semantic space. This process gradually alters the predicted scores for hard negatives.

For those hard negatives whose predicted scores increasingly diverge from the evaluation model's scores, the gradient decay applied to these samples progressively diminishes. We have further visualized and analyzed the changes in the spatial distribution of negatives in Appendix G, providing additional descriptions into this process.

Dear Reviewer,

We thank again for your contributions to the reviewing process. The responses to your concerns and the corresponding paper revision have been posted. Please let us know whether we have properly addressed your concerns. We look forward to your reply and welcome any further questions.

Best regards,

Authors of Paper: "Enhancing Unsupervised Sentence Embeddings via Knowledge-Driven Data Augmentation and Gaussian-Decayed Contrastive Learning"

Thanks very much for taking your time to review the manuscript so seriously and responsibly. Please see the following for our response to your questions:  

Q1: 1. Is it possible to visually demonstrate the improvement in data diversity and reduction in data noise brought about by the proposed method through some quantitative metrics (same as Weakness) ?

A1: Thank you for your suggestion. We have supplemented the analysis with additional visualizations to demonstrate the improvements in data diversity and the reduction of data noise brought about by our proposed method. In Appendix B, we provide t-SNE visualizations of the synthesized samples. The results indicate that incorporating entity knowledge significantly enhances the diversity of samples in the semantic space. To address potential noise in the synthesized samples and the role of the Gaussian-decayed function in mitigating its impact, we have included further analyses in Appendix G. This section features visualizations and examines the changes in the spatial distribution of negative samples, highlighting the effectiveness of our approach in handling noisy data. We hope these additions provide the quantitative and visual clarity you are seeking.  

Q2: 2. Ablation analysis of the hyperparameters α and β in equations 10 and 11.  

A2: Thank you for your question. We have conducted an ablation analysis of the hyperparameters α and β in Equations 10 and 11. The impact of varying these hyperparameters on model performance is detailed in Appendix H, with the corresponding results presented in Figure 10.

Q3: 3. The impact of different LLMs on performance, and whether replacing with a more powerful LLM could raise the method's upper limit.  

A3: Thank you for raising this important question. The quality of synthetic samples indeed has a significant impact on the final model performance, and the parameter size of the LLM can influence both the adherence to prompts and the quality of the generated samples. Theoretically, using a larger LLM should result in better performance due to higher-quality sample synthesis. To explore this, we conducted additional experiments using the same experimental settings but synthesized samples with GLM4-9B and ChatGPT. The results of these experiments have been added to Tables 2 and 7. Notably, our model demonstrates substantial improvements over prior best methods. For semantic textual similarity (STS) tasks, our approach outperforms the previous state-of-the-art by:

• 1.05% with BERT-base,
• 1.62% with BERT-large,
• 0.49% with RoBERTa-base,
• 1.50% with RoBERTa-large.

Thanks very much for taking your time to review this manuscript! Please see our response to your comments below:  

Q1: If SynCSE and GCSE use the same LLM, would the improvement be greater? Could you provide these results?

A1: Thank you for raising this important question. The quality of synthetic samples indeed has a significant impact on the final model performance, and the parameter size of the LLM can influence both the adherence to prompts and the quality of the generated samples. Theoretically, using a larger LLM should result in better performance due to higher-quality sample synthesis. To explore this, we conducted additional experiments using the same experimental settings but synthesized samples with GLM4-9B and ChatGPT. The results of these experiments have been added to Tables 2 and 7.

Notably, our model demonstrates substantial improvements over prior best methods. For semantic textual similarity (STS) tasks, our approach outperforms the previous state-of-the-art by:

• 1.05% with BERT-base,
• 1.62% with BERT-large,
• 0.49% with RoBERTa-base,
• 1.50% with RoBERTa-large.

These results underscore the robustness of our method across different LLMs and model architectures.

Dear Reviewer,

We thank again for your contributions to the reviewing process. The responses to your concerns and the corresponding paper revision have been posted. Please let us know whether we have properly addressed your concerns. We look forward to your reply and welcome any further questions.

Best regards,

Authors of Paper: "Enhancing Unsupervised Sentence Embeddings via Knowledge-Driven Data Augmentation and Gaussian-Decayed Contrastive Learning"