Description-Based Text Similarity

Thank you for the positive feedback and recognition of the novelty of our work.

Dataset Size and Practical Implications: We acknowledge that the current dataset may not be large enough to fully assess the practical implications of our approach in broader document retrieval scenarios, however, we put emphasis on the quality of the queries used in the evaluation, as we manually verified each of them, and edited them when it was necessary. While the dataset, consisting of 200 queries, is not very long, it is still long enough to showcase the limitations of existing approaches to the problem.

Thank you for the detailed and thoughtful feedback.

Specialization and Fairness: While our model is indeed specialized for the task of retrieving text by abstract description, this specialization underscores a critical point: existing models, trained on broader and often less specific definitions of similarity (I.e., many different similarity datasets), fail to capture this notion, as demonstrated in our experimental evaluation. The in-distribution test data was necessary to fairly evaluate the capability of our model in this specific context. Out main contribution is not the training of the specific model (indeed, it is very likely that better models can be found), but rather in advocating for this specific notion of similarity and showing that current models fall short in acquiring it. Please note that the test set was sourced from GPT-generated queries, but was further manually evaluated and refined by the authors.

Tying encoders weights: We have tried that in preliminary experiments, but this resulted in somewhat decreased performance. Please note that in our setting, the queries and the results are very different in their surface form: queries are short and highly abstract, while results are concrete and detailed

Color Scheme in Figure 2: We apologize for the oversight in our color selection. We will revise Figure 2 to use a more distinct and accessible color scheme.

Thank you for the feedback and the opportunity to address the concerns raised. Our task of retrieving text based on abstract descriptions introduces a new notion of similarity. It has no relation to TREC's long query tasks.

Unlike TREC's long query tasks, which generally focus on detailed and verbose queries directly related to the content, our work addresses the challenge of aligning abstract, high-level descriptions with specific instances. This requires a different approach and demonstrates significant novelty in the context of information retrieval. We do not understand in what sense this is “incremental”.

Thank you for the insightful feedback.

Significance of Description-Based Similarity Task: We envision the task of retrieving texts by their description to be directly useful on its own for many end-uses, in particular information seeking by experts in large document collections. This is true regardless of any relevance for an existing NLP end-task. A second-order user-study can be useful, but is not in the scope of the current work.

In the introduction section, we discuss the use cases of our proposed approach in information-seeking scenarios. Its Usefulness is particularly highlighted in scenarios where experts aim to locate domain-specific relevant text in large corpora. For instance, a legal researcher can inquire about "precedents for intellectual property disputes". Natural language descriptions provide an effective way for domain experts to search over large corpora of natural language texts.

Fair Comparison with Fine-Tuned Baselines: We focus on drawing attention to the need to retrieve by description, and showcasing the limitations of existing encoders in this task. We find it essential to test models on our data to showcase the limitations of SOTA general-purpose encoders in capturing the notion of similarity we argue is useful. Their failure is a nontrivial finding, as these models were trained on orders of magnitude more data. It is not possible to highlight this inherent limitation of existing models without evaluating them on data tailored for our notion of similarity.

InfoNCE Temperature Setting: The choice of the temperature parameter in InfoNCE was based on preliminary experiments where a value of 0.1 yielded better performance for our specific task. We acknowledge the need for an ablation study to support this choice and will include such a study in the revised manuscript to provide a detailed justification.

Choice of GPT-3 over ChatGPT: Please note that we used text-davinci-003, an instruction-tuned version. We found that it worked better than ChatGPT, which generated highly verbose responses. However, the LM choice is pretty much orthogonal to our approach.

Dataset Statistics: We will present detailed statistics of our dataset, including train/dev/test set sizes and the number of positive and negative examples, in a table for better clarity.

Missing References: We appreciate the pointers to relevant literature. We will include the suggested references on LLM-based synthetic data generation.