Retrieval-based Disentangled Representation Learning with Natural Language Supervision

Thank you for your detailed review and insightful feedback. We address your comments and questions below.

W1: Not convincingly demonstrated to enhance explainability.

We appreciate the feedback and recognize the critical importance of assessing the explainability. Indeed, evaluating the explainability of any system is an continuous challenge, which often necessitates human assessment. Especially in the disentanglement field, where a universally accepted evaluation methodology is yet to be established [1]. Despite these challenges, we are fully committed to demonstrating the explainability of our framework through a comprehensive approach that encompasses human evaluation (Section 6.2, Appendix H), real-world case studies (Section 6.2 and Appendix F), and retrieval benchmarking.

In addition, it is worth noting that our framework's explainability can be partially assessed through non-parametric inference. This involves comparing the similarity between the BoW representations of queries and representations of their corresponding target data. The similarity score indicates how well the dimensional values within $p$ representation reflects the tokens presented in $q$. Although this method only utilizes half of the model's parameters and being stringent, VDR framework still shows effective performance in text-to-text and fair performance in cross-modal retrieval. This indirectly supports its explainability.

W2: Lack of analysis on non-retrieval tasks.

Thanks for the feedback! Our current research primarily focuses on retrieval benchmarks, where well-established datasets and comparable baselines allow for a robust evaluation to validate our approach. However, we acknowledge the importance of extending this research to a wider range of applications beyond retrieval. We consider this a foundational step towards their broader application and eagerly anticipate future advancements in this area.

Q1: The authors use elu1p activation as a replacement for ReLU. Do you have experimental results that motivate this decision?

We have conducted a comprehensive ablation study for text-to-text retrieval, detailed in Appendix I. The result demonstrates that using elu1p activation alongside top-k sparsification resulted in an average 3% improvement in NDCG@10 on the BEIR benchmark compared to using ReLU activation.

Q2: Do the authors have any intuition as to why VDR works better than DPR?

The reviewer's question highlights an important aspect of our work! We share two key insights from representation learning perspective:

1. VDR capitalizes on the strengths of masked language models (MLM) more effectively than DPR.
   • MLM pre-training projects $d$-dimensional latent representations to |V|-dimensional vocabulary space. These projected |V|-dimensional representations are optimized to predict masked tokens, leading to reasonable vocabulary distributions where high dimensional values reflect great contextual relevance to the corresponding tokens. VDR harnesses this by employing an elu1p activation and max-pooling over these |V|-dimensional representations, aligning closely with the nature of text matching tasks.
   • DPR relies on the [CLS] token representation, which are originally optimized for next sentence prediction in MLM pre-training, creating a gap in suitability for information retrieval tasks.
   • We provide empirical evidence supporting this in below table where VDR outperforms DPR significantly when using a BERT-based encoder without any fine-tuning.

2. VDR utilizes a significantly larger representation space than DPR. This expanded space allows for a greater capacity to capture nuanced features in data. This concept aligns with the principles in mixture-of-experts (MoE) research [2]. Just as MoE increases network capacity through a larger number of selectively activated parameters, VDR expands its representational capability through increased dimensionality and introduces sparsity for supervision and fast downstream inference.

[1] Carbonneau, Marc-André, et al. "Measuring disentanglement: A review of metrics."

[2] Shazeer, Noam, et al. "Outrageously large neural networks: The sparsely-gated mixture-of-experts layer."

Dear Reviewer LxNZ,

We sincerely appreciate the time and effort you have dedicated to reviewing our paper. Your insightful questions and valuable feedback have greatly contributed to the enhancement of our research.

In response to your valuable feedback, we have diligently worked on a rebuttal that addresses concerns you raised. As the discussion period is approaching its conclusion in two days, we would greatly appreciate your review of our rebuttal at your earliest convenience. Should there be any further points that require clarification or improvement, please know that we are fully committed to addressing them promptly.

Thank you once again for your invaluable contribution to our work.

Warm Regards,

The Authors

Q1: What the authors consider to be the difference between disentanglement and grounding in this case?

Thank you for the insightful question.

• Disentangled representations, originating from Bengio's early work[5,6], is defined as a property of "good" representations for machine learning, which focuses on the representations and the representation learning approach.
• Grouding[7] generally refer to linking abstract knowledge (often in textual form) to tangible real-world examples, enabling AI systems to comprehend and interact with their environments. Grounding emphasizes practical applications and task-oriented solutions.
• Disentangled representations benefit grounding tasks and a variety of other applications by improving interpretability, controllability, debuggability. For instance, both CLIP and VDR systems can identify relevance between images and text. While VDR employs disentangled representations, it offers a more nuanced perspective by pinpointing specific elements (tokens) that facilitate the matching process.

Q2: The authors claim that VDR outperforms the baselines with a significant margin, can the authors give the results of a significance test that backs up this claim?

We understand the importance of substantiating claims with empirical evidence and appreciate the opportunity to further clarify our findings.

In our submission, we have been careful to specify conditions such as "with comparable model size and training costs" when referring to the primary baselines. When comparing VDR with advanced baselines, we have used terms like "comparable" or "fair" performance to avoid overstatements.

Our evaluation adheres to well-established retrieval evaluation standards common in both academic and industrial research. Specifically, VDR exhibits an average improvement of +8.7 over DPR and +2.2 on SPLADE on NDCG@10 across 12 datasets. This margin of improvement is noteworthy, especially considering the VDR and DPR have similar parameter counts and identical training pipelines.

To add depth to our analysis, we have responded to relevant questions from other reviewers that further elucidate our method's robustness. For instance, in our response to Reviewer fuH7 Q1 (Link), we have shown the stability of VDR with increasing vocabulary size. Additionally, in our reply to Reviewer LxNZ Q2 (Link), we provided insights into the underlying reasons for VDR's superior performance compared to DPR.

We hope these togather can give your comprehensive view and address your concern. We are open to further discussion and engage into more dicussion to address your concerns.

[5] Bengio, Yoshua. "Learning deep architectures for AI."

[6] Bengio, Yoshua, Aaron Courville, and Pascal Vincent. "Representation learning: A review and new perspectives."

[7] Chandu, Khyathi Raghavi, Yonatan Bisk, and Alan W. Black. "Grounding'Grounding'in NLP."

Thank you for your detailed review and insightful feedback. We address your comments and questions below.

W1(1): How 'advanced' baseline is defined and whether this is a a fair distinction?

The "advanced baselines" refer to baselines that employ sophisticated techniques known for significantly improving the performance number but orthogonal to the retrieval design. These techniques often introduce substantial computational cost and tuning effort.

While we have detailed these techniques in cross-modal scenario in Section 4.4, below we offer additional details about these techniques in text-to-text scenario:

• Retrieval-oriented pre-training utilizes large-scale pre-training tasks[1] specifically designed to boost retriever.
• Specialized negative sampling is well known as crucial in contrastive learning and training retriever.[2]
• Knowledge distillation[3] is a process where knowledge from a larger model (usually cross-encoder) is transferred to a simpler one (bi-encoder). While this improves performance, it also increases complexity and computational demands.
• Access to Wikipedia data during training potentially benefit the performance on datasets constructed from Wikipedia, such as the NQ dataset. [4]

W1(2): Issues related to evaluations.

• The examples for case study and human evaluation were randomly selected.
• We provide more details about the setup and analysis of human evaluation in Appendix H.
• In addition to case studies and human evaluations, our framework's explainability can be also demonstrated through a retrieval benchmark. This is significant for the following reasons. Our framework utilizes disentangled representations, which are notably sparse (< 2% activation). This sparse nature ensures that the similarity between query-target pairs ($q_i$ and $p_i$) is partially dependent on the number of dimensions they both activate. Achieving high performance in retrieval tasks require the representations of each pair ($q_i$, $p_i$) assgin large values to the dimensions that reflect their data characteristics in common, while assigning lower values to dimensions representing other potential targets $p_j$ in the large corpus. Such a mechanism indirectly validates the effectiveness and reliability of our disentangled representations.

W2: A couple of unsubstantiated claims and citation error for AdamW.

Thank you for your valuable feedback. We apologize for any confusion caused by our previous statement. We have revised the statement to better reflect the intended meaning:

1. "In practice, individuals commonly use natural language to convey the distinctive features of objects, thereby aiding in the differentiation of the discussed object from others."
2. We have revised the definition of IR to more closely align with the broader perspective in Manning, 2009.
3. We have corrected the citation of AdamW in the revised version.

[1] Chang, Wei-Cheng, et al. "Pre-training tasks for embedding-based large-scale retrieval."

[2] Lin, Sheng-Chieh, Jheng-Hong Yang, and Jimmy Lin. "In-batch negatives for knowledge distillation with tightly-coupled teachers for dense retrieval."

[3] Gou, Jianping, et al. "Knowledge distillation: A survey."

[4] Ren, Ruiyang, et al. "A thorough examination on zero-shot dense retrieval."

Thanks for your reply and additional explanations. A couple of follow up questions:

Re W1(1): I am happy with this explanation, and I think it would be good to add a paragraph to the paper that explains in more detail how 'advanced' is defined. Maybe you can add parameter counts / computational cost to the results table.

Re W1(2): I indeed read the additional explanation in Appendix H, but this still leaves me with a number of questions that are unaddressed, for example: (1) What platform was used? (2) How were the human raters selected? (3) participants are asked which set of tokens described the image better -- what was done to avoid position bias in this question? (4) In the second part of the human eval, did participants asses the same 20 images, or did everyone rate a different image? (5) What was done to compute interrupter agreement? Etc.

Re Q2: My question was specifically targeted towards the claim that the scores were significant. Significance testing actually is a well-established practice within the IR community. Just some examples from SIGIR 2023 that report NDCG@k and report the results of significance testing:

• https://arxiv.org/pdf/2305.01522.pdf
• https://dl.acm.org/doi/pdf/10.1145/3539618.3591683
• https://dl.acm.org/doi/pdf/10.1145/3539618.3591702
• https://arxiv.org/pdf/2305.00319.pdf etc.

We appreciate the reviewer's feedback, as it has been instrumental in enhancing the quality of our research. We are pleased to offer further clarifications below.

It would be good to add a paragraph to the paper that explains in more detail how 'advanced' is defined.

Thank you for the suggestion! We have elaborated on the definition of advanced baselines in Appendix J and referenced it in Section 4.4 for clarity. Additionally, we've included details about parameter amount and training specifics. While computational costs are challenging to standardize due to varying infrastructures and preprocessing pipeline, we've provided our training time and device information to support future research comparison and replication efforts.

W1(2): Issues related to Evaluations.

This still leaves me with a number of questions that are unaddressed, for example...

(1) Our evaluation did not rely on a specific, pre-existing platform. Instead, we shared the evaluation samples and forms online directly with the raters.

(2) Our raters were university students recruited with diverse backgrounds, which was to minimize bias and to reflect a broad spectrum of perspectives in the evaluation.

(3) To mitigate potential position bias, we have randomized the order of the manually-selected top-5 tokens from VDR and also the presentations order of the two methods.

(4) The participants evaluated the same 20 images in both phases (satisfaction and comparison). This is designed to ensure their familiarity for effective comparison, enhancing the reliability of their comparative assessments.

(5) Regarding inter-rater (interrupter) agreement. Due to our budget limitations and the goal to cover a wide array of images, each image was assessed by a single rater. While this approach limits the application of standard inter-rater reliability metrics, we believe that the two-phase evaluation process, combined with the diversity of rater backgrounds, offers a multi-faceted perspective on the evaluations. We acknowledge this as a limitation in our study and suggest it as an area for future research with increased budget.

These details are further elaborated in Appendix H of our submission. We hope this additional information addresses your concerns.

Q2: Can the authors give the results of a significance test that backs up this claim?

Thank you for your valuable feedback and for kindly sharing the examples to us. In response to your concerns, we have conducted comprehensive significance testing to reinforce the robustness of our work. We have included detailed analyses in Appendix K of our paper.

Briefly, we conducted two-sided t-tests comparing VDR against baselines with identical parameter counts and training pipelines. Below tables are the results of significance tests where $\triangle$ means the improvement is significant with p<0.01. Our results demonstrated significant improvements (p<0.01) in the majority of text-to-text retrieval datasets and across all recall measures for cross-modal retrieval.

Thank you for your insightful feedback and an excellent summarization of our work! We address your comments and questions below.

W1: Unit itself may not represent a meaningful entity.

While our analysis show most tokens in BERT vocabulary are interpretable, we acknowledge that this may not be universally applicable to all tokenizers. In cases where most tokens are not individually meaningful, we suggest pre-defining a set of textual concepts and measuring the data-concept relevance at a coarse-level, as conventional retrieval does.

W2: Human evaluation with complete BLIP caption.

Thank you for your valuable suggestion. We conducted an assessment on complete BLIP captions on the same sample set, and have detailed it in Appendix H. The evaluation revealed a 76% satisfaction rate, compared to 85% for the top-5 tokens manually selected from these captions. This discrepancy likely arises because BLIP captions, typically 2-10 words long, are succinct, making the top-5 tokens effective for summarizing core ideas. Furthermore, complete captions, while thorough, can risk in introducing specific but inaccurate details (like location and quantity), leading to a reduction in overall satisfaction.

Q1: How stable is the proposed approach to increasing the vocabulary size?

We experiments on mBERT (bert-base-multilingual-uncased) encoder with vocabulary size of 110k, as detailed in Appendix E. We observed a minor performance drop compared to BERT encoder (42.6 vs. 44.5). This slight decline is likely due to the gap between multilingual encoder and English-focused tasks but not the increased vocabulary size itself, as VDR consistently outperforms DPR by about 23% in both type of encoders. This finding align with our analysis in Section 3.3. The "silver bullet" here is that pretrained masked language models can provide a well-established gating distribution for subsequent dimension-wise supervision, which remains effective irrespective of vocabulary size changes.

Q2: Would it be possible to have an XLM-Roberta encoder with the latent vectors being represented by a BERT vocabulary?

The reviewer raises an interesting point! Integrating an XLM-Roberta encoder with BERT vocabulary involves training a new linear projection in the DST head. This setup parallels the challenges encountered in cross-modal scenarios (Section 3.3). The feasibility varies, depending on whether we have a viable gating to start with:

1. Both Q and P side have well-established gating distribution (e.g., text-to-text senario). Effective supervision for downstream task optimization are achievable.
2. Only text-side has well-established gating distribution (e.g., cross-modal senario). Substantial training (more data, larger batches) and enhancement (contrastive mask) is necessary for valid supervision.
3. Neither side has well-established gating distribution (e.g., Appendix C). The bi-encoder struggle to conduct valid supervision and likely fail to learn useful representations.

Therefore, substituting only one BERT-based encoder with XLM-Roberta is feasible. However, this may forfeit the advantages of a pre-trained MLM model, and a substantial training cost would be required to train this projection from scratch.

Q3: Does the non-parametric inference still work without using the non-parametric entry loss?

Our ablation study (Appendix I) reveals that while the non-parametric inference remains functional without this loss (denoted as "w/o bow" below), there is a noticeable decline in performance in NDCG@10 from 44.6 to 40.3.

Q4: Would it be possible to elaborate more on how the analysis for Figure 4 (b) was done?

Your understanding is correct! We only aggregate the selected patch representations for the max pooling rather than pooling across all patches in DST head.

Q5: Typos issues.

Thank you for catching these! We have revised all the typos pointed out by the reviewers, and will continue to proof-read to further improve the manuscript.

Dear Reviewer fuH7,

We sincerely appreciate the time and effort you have dedicated to reviewing our paper. Your insightful questions and valuable feedback have greatly contributed to the enhancement of our research.

In response to your valuable feedback, we have diligently worked on a rebuttal that addresses concerns you raised. As the discussion period is approaching its conclusion in two days, we would greatly appreciate your review of our rebuttal at your earliest convenience. Should there be any further points that require clarification or improvement, please know that we are fully committed to addressing them promptly.

Thank you once again for your invaluable contribution to our work.

Warm Regards,

The Authors