Contradiction Retrieval Via Sparse-Aware Sentence Embedding

We thank Reviewer wWVg for providing valuable feedback!

Append instructions to the input:

Thanks for proposing this baseline. First, adding a task-specific prompt in front of the input task is already used by models on the MTEB leaderboard. For example, SFR-Mistral-Embedding appends 'Given a claim, find documents that refute the claim' in front of the input when testing Arguana. Their reported result is 67.17, while the models we chose, 'uae-large-v1' and 'gte-large-en-v1.5,' have zero-shot scores of 68.3 and 72.5, respectively. This indicates that appending a specific prompt does not improve performance for the counter-argument retrieval task. Readers can refer to the results on MTEB for other models with similar prompts, such as Instructor-xl, which has a score of 55.65. Second, using language models to generate a contradiction and then applying similarity search is indeed a baseline method, though it requires significantly higher computational resources to generate the contradiction before performing the search. We have already experimented with this in our ablation study. Please refer to the paragraph 'Different Scoring Function for Contradiction Retrieval' and Figure 5 at the end of page 9.

Stable choice of alpha

Thanks for proposing this interesting question and example! First, we want to clarify that contradiction is indeed a somewhat subjective opinion that varies across different minds. As mentioned in your example, what about “A huge asteroid hit the earth 65 million years ago and killed many dinosaurs.” Or “Some scientists hypothesize that a huge asteroid hit the earth 65 million years ago and caused the extinction of dinosaurs, but this hypothesis hasn’t been fully verified yet.” We believe that different people make different judgments about whether these are contradictions or not. Second, we want to note that we use two different models to provide the cosine similarity score and the sparsity score. Since the two models are trained independently and potentially with different backbones and temperatures, it is not meaningful to directly sum them without proper scaling. For example, when we trained 'bge-base-v1.5' on Arguana with temperatures of 0.02 and 0.05, the average cosine similarity between ground truth pairs was 0.81 and 0.69, respectively. Therefore, cosine similarity scores are unstable given different temperatures, but they are effective at providing relative ranking for retrieval tasks. This motivates us to use a weighted average to combine the cosine similarity score and sparsity score.

Why do you think semantic similarity should play a role in contradiction retrieval

Please refer to Appendix E for our case study on how the similarity and sparsity scores are used in a typical contradiction retrieval example.

Thanks for your new results. I think the result will support the conclusion of this paper with more references to validated resources. However, there is a critical concern that restrains me from accepting this paper - the necessity of contradiction retrieval itself as a task requiring special methods. As mentioned in our previous discussion, contradiction retrieval can be formalized as regular retrieval with context modification (such as appending instructions). It is not sure whether contraction retrieval can be unified with regular retrieval in fine-tuning. I appreciate the efforts the authors devote to the discussion period, I will directly raise my rating to accept once the two following modifications have been done:

• Running an experiment with fine-tuning text embeddings (such as BGE) on contradiction pairs formalized as regular retrieval pairs. e.g. Contradict(Statement1, Statement2) -> Similar("Not true: {Statement 1}", "{Statement 2}"). Then test the retriever with "Not true: {Statement}".

• Add the SNLI benchmark result to Table 1. At least one other pair-matching benchmark should be considered. I suggest the Winograd (https://huggingface.co/datasets/ErnestSDavis/winograd_wsc) challenge, which includes challenging NLI for contradiction.

We thank Reviewer 83EJ for providing valuable feedback!

The scalar alpha requires tuning on target dataset:

Thanks for proposing this interesting question! First, we want to state that contradiction is indeed a somewhat subjective opinion that varies across different individuals. As mentioned by another reviewer, consider the query text, 'Dinosaurs live on Earth today,' and two other texts: 'A huge asteroid hit the Earth 65 million years ago and killed many dinosaurs.' and 'Some scientists hypothesize that a huge asteroid hit the Earth 65 million years ago and caused the extinction of dinosaurs, but this hypothesis hasn’t been fully verified yet.' These texts may seem to contradict the query, depending on individual perspectives. We believe that different people make different judgments about whether these are contradictions.

Second: We want to note that we use two different models to provide the cosine similarity score and the sparsity score. Since the two models are trained independently and potentially with different backbones and temperatures, it is not meaningful to directly sum them up without proper scaling. For example, when we trained 'bge-base-v1.5' on Arguana with temperatures of 0.02 and 0.05, the average cosine similarity between ground truth pairs was 0.81 and 0.69, respectively. Therefore, cosine similarity scores are unstable given different temperatures, but they are effective at providing relative ranking for retrieval tasks. This motivates us to use a weighted average to combine the cosine similarity score and sparsity score.

The Arguana dataset is small: real-world dataset

Arguana (Retrieval of the Best Counterargument without Prior Topic Knowledge, ACL 2018) is a widely used argument/counter-argument dataset with over 100 citations. As far as we know, Arguana is the only public dataset suitable for our test (both positive and negative documents are included in the corpus). For this reason, we created synthetic datasets to perform additional tests. Finally, we would like to note that many popular retrieval datasets are on a small scale. For example, among the popular BEIR benchmark datasets, NFCorpus (corpus size: 3.6K) and SciFact (corpus size: 5K).

Potential data contamination:

Thank you for raising this concern. We have verified that there is no overlap between the MSMARCO and HotpotQA datasets we have created. Therefore, we believe the zero-shot experiment we conducted is sound. In the following, we provide the number of unique passages for MSMARCO and HotpotQA:

Contradiction detection model:

Thanks for the comment. We agree that our scoring function can be used to classify whether a pair of passages contradicts each other. For example, we evaluate the classification accuracy on the Arguana dataset, where we trained our model. We select half of the test data to be true argument/counter-argument pairs and the other half to be arguments paired with a random argument from its top 10 most similar arguments, ensuring that the labels are balanced. The accuracy is listed below:

Thanks for pointing out the format issue. We have revised that reference format in the uploaded version.

Thank you for your suggestion. According to your proposal, we have performed an experiment on the SNLI dataset to prove the validity of our method there.

As suggested by reviewer wWVg, we only extract those entailment and contradiction pairs from the SNLI dataset. We finetune our BGE model using standard contrastive learning and our SparseCL method.

Similar to what we did in our main experiment, we construct a training tuple like the following: (Original text, Contradiction (positive example), Entailment (negative example))

We first measure the average cosine similarity among entailment, contradiction, and random pairs before training.

After performing contrastive learning, we can get the following statistics:

After performing SparseCL, we can get the following statistics:

By applying a threshold on hoyer sparsity to do entailment/contradiction classification, we can get accuracy 0.647, where the threshold is selected on the validation set. However, the standard contrastive learning doesn’t work here, whose binary classification accuracy is 0.507. That’s why we don’t think standard contrastive learning makes sense for contradiction retrieval task, but we still perform it as requested.

We admit that for the simple entailment/contradiction classification task, relying only on cosine similarity is good enough, as entailment are always closer to the original sentence than contradictions, while the point here is to show that our method based on hoyer sparsity also works for NLI datasets.

Additionally, we want to point out that relying on cosine similarity to identify contradictions from a large corpus is hard because we can observe that the cosine similarities within contradictions are between entailments and random or unrelated pairs. However, by applying SparseCL, we manage to let contradiction pairs stand out with the highest hoyer sparsity score. Our results here are consistent with what we reported in our paper.

Thanks again for your insightful feedback on our work! We've carefully worked to address your comments/questions. Are there any further questions or concerns we should discuss?

We thank Reviewer 7xxh for providing valuable feedback!

Statistical significance tests are not conducted:

We thank the reviewers to point out that the current relative gain on the Arguana dataset might not be significant. So we choose different random seeds to re-run the experiments.

We computed the p-value of the difference between our method LINE2 and LINE1, and LINE4 and LINE3, and the p-value is 0.003 and 0.01, respectively, indicating that the relative gain is significant.

Only one real dataset is examined in the experiments:

As far as we know, Arguana is the only direct task where we can verify our method and compare it with results from the literature. We admit that there are some other similar fact verification datasets like FEVER, SciFact, while they are not ideal for testing our method because we need BOTH supporting and contradicting documents to exist in the corpus. If only either supporting or contradicting evidence exists in the corpus, standard similarity retrieval can already extract the related one. Experiment on such datasets requires manually injecting more conflicting documents to the corpus, which is exactly what we have done to HotpotQA and MSMARCO datasets.

Thank you for your suggestion in applying our SparseCL method to other datasets. We have performed more experiments on SNLI and MNLI datasets. Please see the updated results in Table 7 of Appendix A.

As an application of our SPARSECL method, we demonstrate that our method can be useful for distinguishing entailments and contradictions in natural language inference datasets. For SNLI Bowman et al. (2015) and MNLI Williams et al. (2018) datasets, we extract entailment and contradiction pairs, fine-tune using standard contrastive learning and our SPARSECL, and then report the average cosine similarity / Hoyer sparsity score between entailments, contradictions, and random pairs.

We can observe from the table that, in the zeroshot setting, the average cosine similarity of contradiction pairs lies between the ranges of random and entailment pairs. For the fine-tuned model using standard contrastive learning (CL), the average cosine similarity of contradiction pairs is almost indistinguishable from that of entailment pairs. Finally, after being fine-tuned using SPARSECL, the model exhibits higher average Hoyer sparsity scores for contradiction pairs compared to other two types of relationships.

We thank Reviewer MNad for providing valuable feedback!

Regarding transitivity: Thanks for pointing out this! We admit that cosine similarity is not transitive. We want to comment that for normalized embeddings, ranking by cosine similarity is equivalent to ranking by decreasing l2 distance, where l2 distance is transitive. We will make this claim rigorous in the final version of our paper.

More quantitative analysis or qualitative examples: Thanks for the comment. We actually have some qualitative examples as case studies. Please see appendix E for a case study, where we give an example to show how sparsity score helps retrieving contradicting documents.

Does each train/dev/test split stem from a different set of original passages: Thanks for bringing out this concern. We noticed the overlap issue when creating the dataset and have filtered them out, so there is no overlap between the train/dev/test set now.

We briefly summarize the reviewer’s concerns and our response:

• Statistical Significance Testing

  • Feedback: Statistical significance tests were not conducted to validate performance improvements.
  • Response: We have conducted two-tailed t-tests to confirm whether the observed improvements are statistically significant and added the p-values in the experimental results section.

• Dataset Diversity

  • Feedback: The experiments are limited to one real dataset (Arguana) and two synthetic datasets. Additional benchmarks like NLI datasets are recommended.
  • Response: We have included experiments on the NLI benchmark dataset to validate the generalizability of SPARSECL to other domains.

• Task Justification

  • Feedback: The importance of contradiction retrieval as a standalone task needs better justification.
  • Response: We demonstrated how contradiction retrieval can enhance downstream tasks, such as claim verification and dataset cleaning, by providing concrete examples and results. (See section 4.4 and Appendix A)

• Baseline Comparisons

  • Feedback: The lack of strong baselines, including models adapted for contradiction retrieval, weakens the claims.
  • Response: We added baseline models with context prefixes to compare against SPARSECL and demonstrate the necessity of the sparsity-based metric. (See Table 5)

• Generalization and Scalability

  • Feedback: The tuning of hyperparameters, such as the scalar in the scoring function, may limit generalization.
  • Response: We explored testing with fixed hyperparameters and reported results on unseen datasets to validate robustness.

• Data Contamination Concerns

  • Feedback: The potential overlap between MSMARCO and HotpotQA may compromise generalization claims.
  • Response: We conducted data-cleaning steps to ensure no contamination and ran the experiments with strict train/test splits.

• Qualitative Analysis

  • Feedback: The role of sparsity loss lacks detailed analysis and examples.
  • Response: We added qualitative examples and visualizations showing how sparsity-based embeddings capture contradictions effectively. (See Appendix F)

• Synthesis Bias

  • Feedback: The synthetic datasets may favor the proposed method due to oversimplified contradictions.
  • Response: We provide two reasons: First, we inject more paraphrases into the corpus to confuse the similarity search, which causes the standard similarity search to perform poorly in this case. Please refer to our dataset construction and result analysis (lines 359-364) in Section 4.2. Second, our synthetic HotpotQA and MSMARCO datasets contain more balanced paraphrases and contradiction pairs, providing 88k (original sentence, positive example, negative example) tuples for fine-tuning. For Arguana, we were only able to gather 14k such tuples for fine-tuning. Also, we have conducted experiments on the NLI benchmark dataset to validate the generalizability of SPARSECL to other domains.