A General Framework for Producing Interpretable Semantic Text Embeddings

2. Removing Probing Stage for CQG

To isolate the impact of the probing mechanism, we removed it from the CQG method and evaluated performance using the original LLM-generated question order. The results are presented in Tables 3 and 4 below.

From this comparison, we observe that CQG-MBQA without probing still outperforms QAEmb-MBQA, though the probing mechanism provides additional performance improvements.

In our next revision (to be submitted by the end of November 27), we will:

• Clearly describe how the QAEmb baseline was adapted for general text embedding.
• Highlight the fundamental differences in task settings between QAEmb and CQG.
• Include the results of these additional experiments to ensure transparency and fairness in the comparisons in the appendix.

Table 3: Spearman Correlation with/without Probing

Table 4: Cognitive Load with/without Probing

---

We hope our responses and additional experiments address your concerns and clarify our contributions. Thank you again for your thoughtful feedback, which has greatly helped improve our work. We look forward to any further suggestions you may have.

To ensure consistency, we carefully designed the following prompt, which includes a detailed task description and selection criteria:

You are an expert in natural language processing and text embeddings. From the following list of questions, select the **{num_to_keep} best questions** that would be most effective for text embedding tasks.

**Task Description:**

Text embedding is a process where we convert text into numerical vectors that capture semantic meaning. Good questions for text embedding should help in:

1. **Capturing the main topics and themes in texts**
2. **Understanding the semantic relationships between different pieces of text**
3. **Identifying key concepts and ideas**
4. **Distinguishing between different contexts and meanings**
5. **Enabling effective text similarity comparisons and search**

**Selection Criteria:**

The selected questions should:

1. **Be clear and well-formed**
2. **Cover diverse semantic aspects**
3. **Be general enough to apply to various texts**
4. **Avoid redundancy and similar phrasings**
5. **Focus on meaningful content rather than superficial details**
6. **Help in extracting semantic features useful for embedding generation**
7. **Exclude any questions that are unclear or ambiguous**

**Instructions:**

- From the list below, select **EXACTLY {num_to_keep} questions** that best meet the above criteria.
- **Aim for diversity** by choosing questions that cover a wide range of semantic features.
- **List only the numbers** of the **selected questions**, separated by commas. For example: "1, 5, 8, 12".
- **Do not include any explanations or additional text** in your response.
- **Your response should strictly follow the format specified.**

**Here are the questions:**

{questions}

Using this sparsity penalty approach, we evaluated the QAEmb-MBQA model with different percentages of # questions as output dimensions.

The results are presented in Tables 1 and 2 below. We observe that while the embedding quality remains comparable or slightly lower than QAEmb-MBQA (Full), the cognitive load is notably reduced.

Table 1: Spearman Correlation vs. the Percentage of # Questions Retained

Table 2: Cognitive Load vs. the Percentage of # Questions Retained

We sincerely thank the reviewer for their insightful feedback and valuable suggestions. Based on these recommendations, we have made several revisions and conducted additional experiments. Below is a detailed response outlining the changes and improvements we have implemented.

---

The authors should discuss this 2023 style embeddings paper, which was an early precursor to the work here.

Thank you for bringing this paper to our attention. It employs the generative and instruction-following capabilities of LLMs to analyze and summarize text styles, using LLM outputs to train a smaller language model for enhanced efficiency. Additionally, post-processing techniques are applied to identify the most effective style attributes.

This work is indeed highly relevant and pioneering for our research. We will ensure it is appropriately cited and thoroughly discussed in the next revision, which we plan to complete by the end of November 27.

---

The authors should clarify whether distilling the yes/no answers into a single model is a novel contribution — the style embeddings paper & QA-Emb paper both seem to do this as well.

Thank you for your feedback. We would like to clarify that distilling LLM-generated answers into a single model is a practice employed in both the style embeddings paper and the QAEmb paper. Rather than positioning this as a novel contribution, our goal is to highlight its role in creating a practical and complete pipeline.

Specifically, while the QAEmb paper conducted distillation experiments for the fMRI task, it did not extend this approach to other tasks. Our work focuses on developing a general and practical framework for text embeddings, prioritizing cost-effectiveness and scalability. We will ensure this point is clearly emphasized in the revised version to prevent any potential misleading.

---

The main issue seems to be the authors’ treatment of related work: the CGQ method generates questions through prompting and filters them based on their discriminative ability. The baseline QA-Emb also generates questions through prompting but filters them with a sparsity penalty. From their description, the authors don’t seem to implement the sparsity penalty, which likely skews the comparisons. Can the authors provide a comparison with the baseline including the sparsity penalty? Maybe showing the performance as a function of the number of questions kept?

Thank you for raising this concern. The original QAEmb paper focuses on task-specific text embeddings with task labels, whereas our framework aims to create general text embeddings, similar to pre-trained text encoders.

In our work, we adapted the QAEmb method for general text embedding by incorporating LLM-based question generation and applying semantic deduplication as post-processing. In contrast, we developed a contrastive question generation (CQG) method, which includes a probing mechanism (to filter questions based on their discriminative ability) and semantic deduplication as post-processing.

We acknowledge that the absence of a sparsity penalty equivalent in the QAEmb baseline for filtering question discriminative ability might leave uncertainty about whether the observed improvements stem from the probing stage or higher-quality question generation. To address this, we conducted two additional sets of experiments:

1. Adding Sparsity Penalty for QAEmb

The paragraph of "Post-hoc pruning of $Q$" in the original QAEmb paper proposes two approaches for filtering:

• (1) Using feature selection models (e.g., elastic net) with task labels.
• (2) Using an LLM to select questions relevant to a task description.

Since our work focuses on a general text embedding task without task labels, we followed approach (2) and developed a LLM-based method to filter low-quality questions using a sparsity penalty approach. Specifically, we clustered the questions generated by QAEmb and used the LLM to select subsets of questions within each cluster, varying the percentage of questions retained (from 10% to 90% of the cluster size).

We sincerely thank the reviewer for their thoughtful and constructive feedback. Your comments have provided valuable insights that have helped us refine our work and identify areas for improvement. In response, we have conducted additional experiments, addressed specific concerns, and provided detailed explanations to clarify our design decisions and their impact. Below, we outline our responses and highlight the changes made based on your feedback.

---

Cognitive load is simply the number of overlapping "yes" answers (or 1s in the embeddings) between the representations of a pair from an STS dataset. It is highly dependent on dimensionality and sparsity (Figure 4 & 5). It also doesn't really make sense because the interpretability of an embedding should depend on how many yes's there are for a pair compared to other pairs; embeddings cannot be understood simply by looking at the inner product of a pair of embeddings.

Thank you for your valuable suggestion. We have provided Table 1 below, which shows the normalized cognitive load (in percentage) for the STS task. These normalized results indicate that our CQG-MBQA framework consistently achieves a lower cognitive load compared to the QAEmb-MBQA method across different datasets. Furthermore, the observed trend aligns closely with the results presented in Table 5 of our original paper.

In the revised version of the paper (to be submitted by the end of November 27), we will update Table 5 to include this additional information.

Table 1: The Normalized Cognitive Load (in percentage) for the STS task

---

Many of the important design decisions in the framework are not ablated. Is filtering important? How much does choosing positive and negative samples matter, or clustering? How much does training a surrogate model affect performance? Due to the complicated system and lack of ablations, it is not easy to understand why these embeddings outperform other interpretable embeddings such as QAEmb

We conducted extensive ablation studies to address these questions:

1. Comparison of CQG with and without the probing mechanism

In our CQG-MBQA framework, we employ the probing mechanism for filtering the low-quality questions. To study the effectiveness of filtering, we conducted ablation studies to evaluate the effect of the probing mechanism.

For this experiment, we removed the probing mechanism and used the original LLM-generated order. Results in Table 2 below indicate a performance drop to 76.01, compared to 77.60 when the probing mechanism is included.

Table 2: Spearman Correlation on STS Datasets with/without the Probing Mechanism

---

Unclear cost analysis of running this method on a downstream dataset How much inference time does it take to run on the retrieval datasets?

Thank you for your suggestion. We measured inference time (in seconds) on 9 MTEB benchmark retrieval and clustering datasets using a single H100 GPU. The results are presented in Table 8 below.

Our model is highly efficient as the MBQA framework requires only a single Transformer forward pass, with minimal additional overhead from small classification heads.

Table 8: Inference time (in seconds) on 9 MTEB benchmark retrieval and clustering datasets

---

I think this citation could be relevant: Learning Interpretable Style Embeddings via Prompting LLMs (Patel et al., EMNLP Findings 2023)

Thank you for bringing this paper to our attention. It employs the generative and instruction-following capabilities of LLMs to analyze and summarize text styles, using LLM outputs to train a smaller language model for enhanced efficiency. Additionally, post-processing techniques are applied to identify the most effective style attributes.

This work is indeed highly relevant and pioneering for our research. We will ensure it is appropriately cited and thoroughly discussed in the next revision, which we plan to complete by the end of November 27.

---

What inspired this measurement of cognitive load?

Thank you for your question. The cognitive load measurement in our study is inspired by the approach introduced in the COGAM paper [1], which uses the number of visual chunks as a proxy for cognitive load in the context of model explanations.

Similarly, in our work, we use the number of questions a user needs to read as a proxy for cognitive load. This approach aligns with the idea of measuring the cognitive effort required to interpret or interact with the system.

[1] COGAM: Measuring and Moderating Cognitive Load in Machine Learning Model Explanations (CHI 2020)

---

How were the retrieval datasets chosen?

We conducted experiments on retrieval datasets from the MTEB benchmark, which includes datasets from the BEIR benchmark. To ensure a robust evaluation, we selected a diverse range of datasets of reasonable size from MTEB.

Additionally, to further enhance diversity, we incorporated an extra dataset for news retrieval, as the news datasets included in BEIR are private and not publicly accessible.

---

How much does the QA model quality affect embedding quality?

Thank you for raising this point. The backbone encoder of the MBQA model plays a crucial role in determining the quality of the QA model. To explore this further, in addition to UAE-Large-V1, we evaluated two alternative encoders: stella_en_400M_v5 and gte-large-en-v1.5. Both of these models are highly ranked on the MTEB benchmark and have a comparable parameter size (approximately 400M–500M).

A summary of the results on the STS task is provided in Table 9 below. The results indicate that while these alternative encoders perform reasonably well, UAE-Large-V1 consistently achieves the best performance.

We acknowledge that the impact of different encoders remains an important area for further exploration. However, due to time constraints, we focused on using a recently published, high-performing model (UAE-Large-V1) for our experiments to ensure robust and timely evaluations.

Table 9: Spearman Correlation on STS Datasets for Different Encoder Models

---

Thank you once again for your thoughtful and constructive feedback. Your insights have been invaluable in helping us refine our work and address key areas of improvement. We look forward to any further suggestions or comments you may have.

Table 4: the Classification Task with 12 Datasets

Table 5: the Pair Classification Task with 3 Datasets

Table 7: the Summarization Task with 1 Dataset

2. Comparison between vanilla CQG with different types of samples:

We conducted ablation studies to evaluate the impact of different types of samples on the CQG method's performance. Specifically, we tested three scenarios, with results summarized in Table 3 below:

• (a) Only positive samples:

  • We modified the question generation prompt to exclude explicit negative samples:
    • Original Prompt:
      ...where for all questions, the answer will be "yes" for ALL the positive articles and "no" for ALL the negative articles.
    • Modified Prompt:
      ...where for all questions, the answer will be "yes" for ALL the positive articles and "no" for general articles.
  • Results show that explicit negative samples improve performance from 76.57 to 77.60 (average across STS datasets).

• (b) No hard negatives:

  • We set the hard negative samples per cluster ($n_h$) and the hard negative probe samples per question ($p_h$) to 0.
  • Results show performance drops to 76.26, compared to 77.60 with both hard and easy negatives included.

• (c) No easy negatives:

  • We set the easy negative samples per cluster ($n_e$) and the easy negative probe samples per question ($p_e$) to 0.
  • Results show performance drops to 75.26, compared to 77.60 with both hard and easy negatives included.

From Table 3, it is evident that the full CQG-MBQA method (including positive samples, hard negatives, and easy negatives) yields the highest question quality and achieves the best downstream performance on the STS datasets.

Table 3: Spearman Correlation on STS Datasets for Different Types of Samples

3. Comparison between the MBQA surrogate model and directly using the LLM’s outputs

Unfortunately, conducting experiments using LLM output on the STS datasets poses significant computational and financial challenges. Specifically, it would require approximately 47.7 million API calls, even if we group 20 questions into a single prompt. This would entail thousands of dollars in API credits (assuming 300 tokens per API call) or at least several months of computation time using a local LLM (assuming each API call takes 0.1 seconds).

As an alternative, we have evaluated the accuracy of our MBQA model in replicating LLM outputs. In Table 6 of the appendix in our original paper, we demonstrate that the classification accuracy is 96%, indicating that the MBQA model is sufficiently accurate when compared to the LLM output.

In addition to the STS, retrieval, and clustering tasks, we have also tested our framework on four additional downstream tasks from the MTEB benchmark: classification, pair classification, reranking, and summarization evaluation. The results are provided in Tables 4-7 below.

As shown in Tables 4-7, our framework consistently outperforms existing interpretable text embedding models and achieves performance comparable to many advanced black-box models across all tested downstream tasks.

The strong performance of our model on seven downstream tasks (three reported in our original paper and these four additional tasks) provides indirect evidence of the MBQA model's competitiveness and reliability. We hope this addresses your concern and clarifies the robustness of our approach.

---

Did you evaluate your model on the fMRI task presented in the QAEmb paper for a direct performance comparison?

Thank you for your suggestion. We did not evaluate the fMRI task for the following reasons:

1. The fMRI task is highly specialized within the neuroscience domain and requires a task-specific design with a relatively small number of questions (e.g., 29 questions as reported in the QAEmb paper).
2. None of the authors possess the domain expertise necessary to conduct a thorough evaluation for this task.

Moreover, there are fundamental differences in the experimental settings: QAEmb focuses on task-specific text embeddings using task labels, whereas our framework is designed to produce general-purpose text embeddings, similar to pre-trained text encoders. Inspired by QAEmb, we recreated the QAEmb baseline (QAEmb-MBQA) under a general text embedding setting for fair comparison.

Instead, we demonstrate the generalizability of our framework through evaluations on 7 diverse downstream tasks from the MTEB benchmark, as presented in our original paper and summarized in Tables 1–4 in this response.

---

How do variations in the number of initial clusters (k) or the use of different clustering methods affect the performance of CQG?

We selected $k=5000$ based on the elbow point observed in the MSE vs. number of clusters plot. In the revised version of the paper (to be submitted by November 27), we will include this figure in the appendix to provide further clarity.

We agree that the number of clusters ($k$) significantly impacts performance, as it determines the granularity of the positive and negative sample boundaries. However, evaluating the framework with different values of $k$ and exploring alternative clustering methods requires rerunning the entire set of experiments multiple times, which would take several weeks.

Due to time constraints, we were unable to conduct comprehensive studies on $k$ or alternative clustering methods. Nevertheless, we agree that exploring this aspect further, such as testing other clustering algorithms (e.g., DBSCAN or hierarchical clustering), is an excellent direction for future research.

---

We hope our detailed responses and additional analyses address your concerns and provide clarity on our contributions and methodology. Your feedback has been invaluable in improving the robustness and generalizability of our work. Thank you again for your time and effort in reviewing our paper, and we look forward to your reply.

---

Having cost analysis would be beneficial as MBQA requires significant LLM inferences (or API calls) during training time and even more may be required during Post-processing (probing).

Thank you for your insightful suggestion. We have summarized the incurred costs in Table 12 below.

In comparison to the cost of generating question-answer pairs for training, the probing process incurs negligible expenses. Additionally, when compared to the costs of generating question-answer pairs for inference directly using GPT-4o-mini (as shown in Table 1 of our original paper), the cost of generating question-answer pairs for MBQA training is significantly lower.

Table 12: The Cost Analysis for Different Steps in CQG-MBQA (using GPT-4o-mini)

---

Including a case study on bad examples would also be beneficial—for instance, displaying cases where two texts result in similar embeddings even when those two texts do not necessarily have similar semantics. Are they completely off? Or how could one improve your approach?

Thank you for this valuable suggestion. We have analyzed a challenging case from the STS Benchmark dataset, detailed below:

• Text Pair:
  • Text 1: And that is happening in GOP-controlled states.
  • Text 2: Michigan IS a GOP-controlled state.
• Ground Truth Similarity Score (0-5): 0.8
• CQG-MBQA Prediction (0-1): 0.851

This pair is labeled as having low similarity (0.8 on a [0, 5] scale) in the dataset; however, our model outputs a relatively high similarity score (0.851 on a [0, 1] scale). The discrepancy can be attributed to the following factors:

1. Incorrect Prediction of QA Answers:
   For instance, in dimension 94, the question Is there an emphasis on the Democratic Party's dynamics? was incorrectly answered as "yes" for both texts. The correct answer should be "no" for both, as the GOP refers to the Grand Old Party (Republican Party), not the Democratic Party.

2. Overly General Questions:
   In dimension 209, the question Is the article set in the United States or Canada? was correctly answered as "yes" for both texts. However, this question is too general to capture the nuanced differences between the two texts. Specifically:

   • Text 1 discusses an unspecified event occurring in GOP-controlled states.
   • Text 2 states a factual assertion about Michigan being a GOP-controlled state.

   Such general questions dilute the embedding’s ability to capture subtle distinctions, leading to an inflated similarity score.

We will continue to investigate this issue and explore potential improvements to enhance the robustness of our approach in future work.

3. Comparison between vanilla CQG with different types of samples:

Thank you for your suggestion. We conducted ablation studies to evaluate the impact of different types of samples on the CQG method's performance. Specifically, we tested three scenarios, with results summarized in Table 9 below:

• (a) Only positive samples:

  • We modified the question generation prompt to exclude explicit negative samples:
    • Original Prompt:
      ...where for all questions, the answer will be "yes" for ALL the positive articles and "no" for ALL the negative articles.
    • Modified Prompt:
      ...where for all questions, the answer will be "yes" for ALL the positive articles and "no" for general articles.
  • Results show that explicit negative samples improve performance from 76.57 to 77.60 (average across STS datasets).

• (b) No hard negatives:

  • We set the hard negative samples per cluster ($n_h$) and the hard negative probe samples per question ($p_h$) to 0.
  • Results show performance drops to 76.26, compared to 77.60 with both hard and easy negatives included.

• (c) No easy negatives:

  • We set the easy negative samples per cluster ($n_e$) and the easy negative probe samples per question ($p_e$) to 0.
  • Results show performance drops to 75.26, compared to 77.60 with both hard and easy negatives included.

From Table 9, it is evident that the full CQG-MBQA method (including positive samples, hard negatives, and easy negatives) yields the highest question quality and achieves the best downstream performance on the STS datasets.

Table 9: Spearman Correlation on STS Datasets for Different Types of Samples

4. Comparison of CQG with and without the probing mechanism

As suggested, we conducted ablation studies to evaluate the effect of the probing mechanism on performance. For this experiment, we removed the probing mechanism and used the original LLM-generated order. Results in Table 10 below indicate a performance drop to 76.01, compared to 77.60 when the probing mechanism is included.

Table 10: Spearman Correlation on STS Datasets with/without the Probing Mechanism

---

Also, because the cognitive load is defined using the dot product, this measure would be directly influenced by the total number of questions. A normalized version (e.g., dot product divided by number of questions) would provide a fairer comparison across different interpretable models in Table 5.

Thank you for your valuable suggestion. We have provided Table 11 below, which shows the normalized cognitive load (in percentage) for the STS task. These normalized results indicate that our CQG-MBQA framework consistently achieves a lower cognitive load compared to the QAEmb-MBQA method across different datasets. Furthermore, the observed trend aligns closely with the results presented in Table 5 of our original paper.

In the revised version of the paper (to be submitted by the end of November 27), we will update Table 5 to include this additional information.

Table 11: The Normalized Cognitive Load (in percentage) for the STS task

Table 7: Spearman Correlation vs. the Binary Classification Threshold

Table 8: Cognitive Load vs. the Binary Classification Threshold

2. Comparison between MBQA and directly using the LLM’s outputs

Thank you for your suggestion. Conducting experiments using LLM output on the STS datasets poses significant computational and financial challenges. Specifically, it would require approximately 47.7 million API calls, even if we group 20 questions into a single prompt. This would entail thousands of dollars in API credits (assuming 300 tokens per API call) or at least several months of computation time using a local LLM (assuming each API call takes 0.1 seconds).

As an alternative, we have evaluated the accuracy of our MBQA model in replicating LLM outputs. In Table 6 of the appendix in our original paper, we demonstrate that the classification accuracy is 96%, indicating that the MBQA model is sufficiently accurate when compared to the LLM output. Furthermore, the strong performance of our model on seven downstream tasks (three reported in our original paper and four additional tasks shown in Tables 1–4) provides indirect evidence of the MBQA model's competitiveness and reliability.

We hope this addresses your concern and clarifies the robustness of our approach.

---

Lack of ablation studies to assess the efficacy of the proposed approach

• lack of comparison between different models in Figure 4 and 5, and lack of comparison between the MBQA method and directly using the LLM’s outputs.
• comparison between vanilla CQG with positive and hard/easy negative, and CQG with positive and negative samples
• comparison between having and not having the probing mechanism to refine the generated questions

We conducted extensive ablation studies to address the specific sub-points raised:

1. Comparison between QAEmb-MBQA and CQG-MBQA under varying dimensions and thresholds

In Tables 5-8, we present a tabular version of the results shown in Figures 4 and 5. Our results reveal that the performance of both CQG-MBQA and QAEmb-MBQA improves as the number of dimensions increases from 1000 to 3000, stabilizing beyond 3000 dimensions. Notably, CQG-MBQA consistently outperforms QAEmb-MBQA across all dimensionalities and binary classification thresholds while maintaining a lower cognitive load.

In the revised version of the paper (to be submitted by the end of November 27), we will incorporate QAEmb-MBQA into Figures 4 and 5 using data from Tables 5-8. These additional results will provide a more comprehensive comparison with QAEmb-MBQA and further enhance the clarity and depth of our analysis.

Table 5: Spearman Correlation vs. the Number of Dimensions

Table 6: Cognitive Load vs. the Number of Dimensions

Table 1: the Classification Task with 12 Datasets

Table 2: the Pair Classification Task with 3 Datasets

Table 3: the Reranking Task with 3 Datasets

Table 4: the Summarization Task with 1 Dataset

We would like to sincerely thank the reviewer for their thorough and thoughtful feedback. Your comments have been instrumental in helping us improve the quality of our work, both by addressing specific concerns and by identifying areas that required further exploration. In response, we have conducted additional experiments and analyses to provide more comprehensive evidence and insights. Below, we detail our replies to each of your concerns and highlight the changes made.

---

In retrieval tasks, there is a significant performance gap compared to black-box models, and the performance is also lower than BM25. Therefore, additional performance comparisons are needed when applying them to various downstream tasks such as sentiment classification and retrieval.

Thank you for your valuable suggestions. We agree that evaluating additional downstream tasks is crucial for thoroughly demonstrating the performance of our framework.

In addition to the STS, retrieval, and clustering tasks, we have also tested our framework on four additional downstream tasks from the MTEB benchmark: classification, pair classification, reranking, and summarization evaluation. The results are provided in Tables 1–4 below.

As shown in Tables 1–4, our framework consistently outperforms existing interpretable text embedding models and achieves performance comparable to many advanced black-box models across all tested downstream tasks. These results further highlight the generalizability and robustness of our framework for diverse text embedding tasks.

---

Question 2: Figure 4 shows that with around 3,000 questions, CQG-MBQA can achieve high-quality text embeddings on STS tasks, and using additional questions does not improve embedding quality; instead, it decreases interpretability. Does this imply that the yes/no questions generated by CQG have semantic overlap or inclusion relationships? In other words, is there a large number of semantically similar questions within the set of yes/no questions?

Yes, despite our use of the CQG method to generate diverse and discriminative questions, some semantic overlap naturally exists within the generated set. To address this, our CQG-MBQA framework incorporates two postprocessing steps, probing and deduplication, to identify the most effective and non-duplicated questions. Specifically, during the deduplication step, we ensure that the semantic similarity (measured by cosine similarity) between the question embeddings does not exceed 0.8.

However, even after deduplication, certain semantic relationships remain. For instance, the questions "Is the enhancement of community well-being through environmental policies discussed?" and "Is there a connection made between health and environmental factors?" are not duplicates but are semantically related with differing focal points.

To provide additional insights, Tables 5 and 6 below present the performance and cognitive load results on STS tasks for both CQG-MBQA and QAEmb-MBQA models across different dimensional settings.

Our results reveal that the performance of both CQG-MBQA and QAEmb-MBQA improves as the number of dimensions increases from 1000 to 3000, stabilizing beyond 3000 dimensions. Despite we have some semantic overlaping between questions, CQG-MBQA consistently outperforms QAEmb-MBQA across all dimensionalities while maintaining a lower cognitive load.

Table 5: Spearman Correlation vs. the Number of Dimensions

Table 6: Cognitive Load vs. the Number of Dimensions

---

We hope these responses address your concerns and provide clarity on the flexibility, generalizability, and design choices of our framework. Thank you once again for your invaluable feedback, and we look forward to your thoughts on our clarifications and updates.

Table 1: the Classification Task with 12 Datasets

Table 2: the Pair Classification Task with 3 Datasets

Table 3: the Reranking Task with 3 Datasets

Table 4: the Summarization Task with 1 Dataset

We sincerely thank the reviewer for these insightful and thought-provoking questions. They address key aspects of our framework's generalizability, design choices, and practical implications. Below, we provide detailed responses to each question, supported by additional experiments and clarifications to address the raised concerns comprehensively.

---

Question 1: Does the CQG-MBQA framework need to generate a new set of yes/no questions every time it is applied to a different dataset? If so, is there a way to enhance the generalizability of the CQG-MBQA model? In other words, could a more universal set of yes/no questions be designed to handle multiple tasks/datasets, rather than creating a separate set tailored to each specific task/dataset?

Thank you for raising this insightful question. We want to clarify that the CQG-MBQA framework does not require generating a new set of questions for each dataset. In our experiments, we generated a universal set of questions using the CQG method on the MEDI2 dataset, trained the MBQA model on these questions, and tested the same model across multiple datasets from various downstream tasks without additional training or fine-tuning. The superior results across diverse downstream tasks demonstrate the model's generalizability.

However, our framework can allow users to generate task-specific questions using their own text corpus. This optional customization requires no labeled training data and can better suit specific needs. For example, questions generated from hotel customer reviews could help uncover customer preferences across different hotel types.

Our experiments show that the pre-trained model, using questions derived from the MEDI2 dataset, performs robustly on three downstream tasks (STS, retrieval, and clustering) as reported in our original paper, without requiring further question generation. To further validate this generalizability, we conducted experiments on four additional downstream tasks from the MTEB benchmark. Tables 1–4 below highlight how our framework consistently outperforms existing interpretable text embedding models and achieves comparable performance to many advanced black-box models across all tested downstream tasks.

In the revised version of the paper (to be submitted by the end of November 27), we will clarify this point in the main paper and include the results from Tables 1–4 in the supplementary materials.

---

Implementation details can be moved more to the main paper as they are mostly in appendices.

Thank you for your kind suggestion. We will include additional implementation details in the main paper in the next revision, which will be submitted by the end of November 27.

---

In order to achieve performance of regular dense representation models, which aspects of the framework and the implementation details do the authors think worth scaling up? Is there any interesting evidence? e.g., better datasets for generating questions; more questions to form the embedding dimensions, etc..

We carefully selected the training corpus and the number of dimensions to strike a balance between effectiveness and efficiency. However, there is room for improvement, particularly through the use of a more diverse and higher-quality training corpus. For domain-specific applications, users can extend the framework by training it on a corpus tailored to their specific domain (no labels required), enabling the generation of questions that are more relevant and discriminative for that context. While this extension is a promising direction, it lies beyond the primary focus of this work. We plan to explore this avenue in future research.

---

From my understanding in the appendix, the paper uses UAE-large-v1 as the encoding model, why? What happens if the encoding model is some better models - does it help with the performance of the final interpretable embeddings?

We selected UAE-Large-V1 because it is a state-of-the-art model published in ACL 2024, open-source, highly ranked on the MTEB benchmark, and designed with generalizability in mind, as it does not heavily rely on training data from the MTEB benchmark. This choice aligns with our goal of developing a general text embedding framework.

In response to this feedback, we conducted additional experiments, and the results in Table 4 demonstrate that UAE-Large-V1 outperformed the other two models, stella_en_400M_v5 and gte-large-en-v1.5.

However, we acknowledge that using an even better encoder could potentially improve final embedding performance by enhancing clustering quality and representation as the backbone of the MBQA model. Due to time constraints, we opted to focus on a recently published, high-performing model rather than exhaustively testing all available encoders.

---

We hope our responses and additional experiments have clarified the points raised and demonstrated the improvements made. Thank you for your valuable feedback, which has significantly strengthened our work. We appreciate your time and thoughtful review and look forward to any further suggestions you may have.

2. Question Difficulties

The parameters within the Contrastive Question Generation (CQG) method influence the difficulty level of the generated questions by providing varying amounts of information to the LLM during question generation.

As suggested, we have conducted ablation studies to examine how modifications to the CQG method affect performance, enabling us to better understand the impact of question difficulty on the overall results. Specifically, we tested four cases:

• (a) Only positive samples:

  • We modified the question generation prompt to exclude explicit negative samples:
    • Original Prompt:
      ...where for all questions, the answer will be "yes" for ALL the positive articles and "no" for ALL the negative articles.
    • Modified Prompt:
      ...where for all questions, the answer will be "yes" for ALL the positive articles and "no" for general articles.
  • Results show that explicit negative samples improve performance from 76.57 to 77.60 (average across STS datasets).

• (b) No hard negatives:

  • We set the hard negative samples per cluster ($n_h$) and the hard negative probe samples per question ($p_h$) to 0.
  • Results show performance drops to 76.26, compared to 77.60 with both hard and easy negatives included.

• (c) No easy negatives:

  • We set the easy negative samples per cluster ($n_e$) and the easy negative probe samples per question ($p_e$) to 0.
  • Results show performance drops to 75.26, compared to 77.60 with both hard and easy negatives included.

• (d) Without the probing mechanism:

  • We removed the probing mechanism and used the original LLM-generated order.
  • Results show performance drops to 76.01, compared to 77.60 with the probing mechanism.

From these ablation studies, we demonstrate that the CQG-MBQA (full) method (including positive samples, hard negatives, easy negatives, and the probing mechanism) yields the highest question quality and achieves the best downstream performance on the STS datasets. Please refer to Table 3 below for the complete results.

Table 3: Spearman Correlation on STS Datasets for Different Ablated Models

3.Different Encoders

Beyond UAE-Large-V1, an advanced encoder that ranks highly on the MTEB benchmark, we further evaluated two alternative encoders: stella_en_400M_v5 and gte-large-en-v1.5. Both of these models also rank highly on the MTEB benchmark and have a comparable parameter size (approximately 400M–500M).

A summary of the results on the STS task is provided in Table 4 below. The results indicate that while these alternative encoders perform reasonably well, UAE-Large-V1 consistently achieves the best performance.

Table 4: Spearman Correlation on STS Datasets for Different Encoder Models

In the revised version of the paper (to be submitted by the end of November 27), we will incorporate QAEmb-MBQA into Figure 4 using data from Tables 1 and 2. Additionally, we will include the ablation study results from Tables 3 and 4 in the supplementary materials. These additions will provide a more comprehensive comparison with the baseline model and further enhance the clarity and depth of our analysis.

We appreciate the reviewer’s thoughtful feedback and valuable suggestions, which have significantly helped us improve the quality and robustness of our work. In response, we conducted extensive additional experiments and revisions to address the points raised, focusing on performance ablations, implementation details, and encoder model selection. Below, we provide detailed responses to each comment, outlining the changes and insights gained.

---

Performance ablations about setups in the framework can be very interesting although currently missing (e.g., performance across different dimensionality, question difficulties, different encoding models, etc..).

Thank you for highlighting this point. We have conducted several ablation studies based on your suggestions:

1. Dimensionality

Figure 4 in our paper (Spearman correlation and cognitive load vs. number of dimensions) demonstrates the performance of our model across varying dimensionalities. To provide additional insights, Tables 1 and 2 below present the performance and cognitive load results on STS tasks for both CQG-MBQA and QAEmb-MBQA models across different dimensional settings.

Our results reveal that the performance of both CQG-MBQA and QAEmb-MBQA improves as the number of dimensions increases from 1000 to 3000, stabilizing beyond 3000 dimensions. Notably, CQG-MBQA consistently outperforms QAEmb-MBQA across all dimensionalities while maintaining a lower cognitive load.

Table 1: Spearman Correlation vs. the Number of Dimensions

Table 2: Cognitive Load vs. the Number of Dimensions