Perplexity Trap: PLM-Based Retrievers Overrate Low Perplexity Documents

Thanks for acknowledging our work and the constructive comments. Please kindly find point-to-point responses below.

W1: Limited to PLM-based retrievers. One may argue that a well behaved retriever, or existing retrievers do not possess this issue. Or there're more standard ways to address this such as by standard training. One may argue that this topic is not of general interest to the ICLR community as it is more SIGIR focused, as the setting is specific and not widely acknowledged.

Response: Thanks for the comments from the ML perspective. Below are our point-by-point clarification:

1. Previous studies[1, 2] have found that neural retrievers prevalently exhibit bias towards LLM-generated documents. One contribution of this paper lies in providing a more rigorous experimental validation and theoretical analysis of such empirical conclusions, narrowing the statements that PLM-based retrievers exhibit bias towards documents with low PPL, thus offering a clearer definition of the source bias. Except frequency-based models perfering human-written documents, it remains unclear whether non-pretrained models exhibit similar biases, which is a question for future research.
2. There may be numerous approaches to address this issue such as adding a debiased training constraint [1]. We presents a tractable method based on our causal analysis, its effectiveness strengthens our explanations. We believe that these explanations can also inspire model training techniques such as synthesizing training data and designing training objectives, leading to emergement of more elegant solutions in the future.
3. In this paper, we verify that the text embeddings (representations) produced by fine-tuned PLMs may have inherent biases in downstream retrieval task, further explanation on which inductive bias leads to this phenomenon are provided in our theoretical analysis. Thus, we believe this analysis aligns with the theme of ICLR. Historically, many papers featured in causal retrieval or recommendation[3, 4, 5] are accepted in ICLR, so we believe our topic is suitable to ICLR based on past outcomes.

W2: Analysis is limited to MLM, while MLM is becoming less and less relevant. Indeed, the retrievers used in the paper are relatively old (most recent 2022). The proposed method also shows more effect on older PLMs than more recent ones. This again raises the concern about how relevant the paper is to the broader ML community. How about models trained with CLM?

Response: Thanks for the inspiring comments. we will explain this from two aspects.

1. We indeed initially focused on analysis on MLMs for the following reasons: BERT-style MLM architectures are generally considered more suitable and widely used in textual representation tasks such as infomation retrieval, as we have discussed in the Appendix C1. In fact, latest SOTA retrievers from MTEB benchmark are MLM-based [6, 7].
2. Following your feedback, we have expanded our analysis to include other architectures like CLM and find that our theoretical framework remains applicable. We find that although different architectures utilize diverse contextual information in token prediction, this variation does not alter the fundamental methods for calculating perplexity and relevance scores. More specifically, in Corollary 1, the variation between architectures only affects the matrix of partial derivatives of the embedding vectors with respect to the input text $\frac{\partial \boldsymbol{d}_2^{\mathrm{emb}}}{\partial \boldsymbol{d}_2}$, but does not impact the first-order gradient term of the loss function with respect to the embedding vectors $\frac{\partial \mathcal{L}}{\partial \boldsymbol{d}^{\mathrm{emb}}}$. Therefore, Corollary 1 still holds.

We will include these expanded findings and present a more universally applicable conclusion in the final version. Thanks for your valuable comments again.

For more responses to the Weaknesses and Questions, please see the next part response. We hope the above response can address your concerns and look forward to further discussions. Thanks again!

[1] Neural Retrievers are Biased Towards LLM-Generated Content, KDD2024

[2] Cocktail: A Comprehensive Information Retrieval Benchmark with LLM-Generated Documents Integration, ACL2024

[3] Causal Fairness under Unobserved Confounding: A Neural Sensitivity Framework, ICLR2024

[4] Calibration Matters: Tackling Maximization Bias in Large-scale Advertising Recommendation Systems, ICLR2023

[5] From Intervention to Domain Transportation: A Novel Perspective to Optimize Recommendation, ICLR2022

[6] jina-embeddings-v3: Multilingual Embeddings With Task LoRA, arXiv: 2409.10173

[7] mGTE: Generalized Long-Context Text Representation and Reranking Models for Multilingual Text Retrieval, arXiv: 2407.19669

W3: The causal analysis and approach leveraged in the paper is conventional. This is not necessarily a "weakness" but it does not make the paper a stronger case.

Response: Thanks for the discussion about the causal analysis approach we used. 2SLS based on instrumental variables is a conventional yet well-developed and widely-used tool for causal inference. The application of this approach to the novel domain of source bias supports the credible verification of our conclusion in the paper, also demonstrating the effectiveness of this statistical method. Appendix D provides a comprehensive exposition on how this method fits our analysis of source bias, indicating our conclusion is appropriate and persuasive.

Q1: Would not "document semantic" be explained away?

Response: Thanks for the question on our causal graph-based explanation, where document semantic is assumed to indirectly affect estimated relevance scores. Some explanation on rationales behind each edge in the causal graph lies in Section 3.2. We will explain further analysis from two aspects.

1. In Section 3.1, we discuss that LLM-rewritten documents possess almost identical semantic as their origin. This is supported by the human evaluations in Appendix E2.1 and embedding similarity comparasion presented in [1]. Therefore, source bias cannot be directly attributed to document semantics alone.
2. Meanwhile, to mitigate the impact of subtle semantic differences, we conducted a causal analysis based on 2SLS, a method that is able to tolerate changes in document semantics. So considering changes in document semantics does not affect the reliability of our conclusions.

Q2: Please list the statistics of each dataset used.

Response: We provide the detailed statistics of experimental datasets in the follow table:

Thanks again for the comments and questions. We hope the response can address your concerns and look forward to further discussions.

[1] Neural Retrievers are Biased Towards LLM-Generated Content, KDD2024

Dear Reviewer,

First of all, we would like to sincerely thank you for taking the time to review our paper and provide valuable feedback. Your comments have been incredibly helpful in improving the content of the paper. As we are currently in the rebuttal stage, we would kindly like to remind you that if you have any further suggestions or feedback on our response, we would greatly appreciate it if you could share them by the 27th. After this date, we will no longer be able to make modifications to the PDF based on the reviewers' comments. Your continued guidance is crucial for us to refine the paper. Once again, thank you for your hard work and support, and we look forward to your valuable response.

Best regards,

Authors

Thanks for the reply. After reviewing the reply and other reviews, I decide to keep the score. I'm still not fully convinced about the timeliness, broad interest, and significance of this work.

Thanks for acknowledging our work and the constructive comments. Please kindly find point-to-point responses below.

W2: The rewriting approach appears to be sound but lacks experimental support to show that the semantic content was not changed. Q1:“Specifically, we manipulate the sampling temperatures during generation to obtain LLM-generated documents with different PPLs but similar semantic content.” and “Since document semantics remain unchanged during rewriting,”. -> This is a strong assumption, that is, in my opinion, not verified. Using high temperature could lead to texts with different semantic content. The article lacks an experimental verification using LLM based eval, human eval, LSA/LDA based approaches or Non LLM embedding similarity. The question asked to the human evaluator in E.2.1 only evaluate relevance (author should provide the complete question asked to the annotator) and not semantic equivalence, and evaluates only 20 triplets. Additionally, increasing temperature might lead to hallucination, mechanically decreasing the relevance of the document. These should be evaluated properly. Could you elaborate on this?

Response: Thanks for the careful concern on the data. In our opinion, the impacts of LLM rewriting on semantics indeed lack more consideration although they have been designed possibly credible. Since simulated environment construction is not our main contribution, we have adopted the datasets provided by previous works [1, 2] or follow their methodology to evaluate the source bias of retrievers. In [1], the document embeddings are compared using cosine similarity, and a more detailed human evaluation was conducted to assess the various impacts of LLM rewriting, which indicated no significant changes in document semantics. We will conduct more meticulous semantic checks to pursue more rigorous conclusions in the future.

At the same time, the instrumental variables method we employed eliminates the interference of document semantic changes on the estimation of causal effects. In our causal graph explanation framework, document semantics is a confounder, whose changes do not affect the causal effect estimation of perplexity on the estimated relevant scores. Therefore, in spite of the changes of the documents semantics, the conclusions of perplexity trap remain credible.

W1: My main concern is the theoretical development in section 4.2. It lacks formalism, some notations are badly introduced or not at all. This leads to some aspect (either details or important things) either difficult to understand and follow, or that seem false.

Response: Thanks for the comment on our theoretical analysis. Please allow us to briefly introduce our analytical approach once again. First, in Theorem 1, we found that if several specific conditions are met, there is a special linear relationship between the gradients of the MLM loss function and the IR loss with respect to document embeddings. Then, in Corollary 1, we found that if LLM-generated documents have lower perplexity at the token level, their estimated relevance score will be higher than that of their corresponding human-written counterparts, thus producing Source Bias. To illustrate the practicality and rationality of Theorem 1, we obtained Corollary 2 using a similar method, which indicates that the stronger a model's MLM capability, the stronger its IR capability. The phenomenon found in previous paper [1] and our empirical experiments both validate the conclusion in Corollary 2, further proving the rationality of the hypothesis we proposed. We have revised the analysis in the latest version of PDF with clearer expressions and we explain the notations with more details in the response to Q2.

Q3: ANCE is used before being introduced line 264 (e.g. line 145 ) Shouldn’t MLM task loss divided by L only rather than L \times D ?

Response: Thanks for the careful notice. We will adjust the order in the final version to avoid declarations used before they are made. Additionally, the MLM loss in the experimental section is divided by $L$ only. As for the theoretical analysis, we standardize it to be divided only by $L$ in the updated PDF, which doesn't affect the conclusions.

For more responses to the Weaknesses and Questions, please see the next comment. We hope the above response can address your concerns and look forward to further discussions. Thanks again!

[1] Neural Retrievers are Biased Towards LLM-Generated Content, KDD2024

[2] Cocktail: A Comprehensive Information Retrieval Benchmark with LLM-Generated Documents Integration, ACL2024

[3] Probability and computing: Randomization and probabilistic techniques in algorithms and data analysis, Cambridge university press2017.

Q2: detailed question on (1) d_{emb} assumed to have a L2 norm. (2) weight matrix W is orthogonal assumption. (3) fine tuning conserve the inversion property assumption. (4) the explanation of k = sum_i(d^{emb}W)_i and related QA-AM.

Response: Thanks for the patient concern on our theoretical analysis again. These questions is due to our imprecise use of notations. Here, we select some key points to provide detailed explanation, the rest of the content has also been revised line by line in the latest version of PDF with clearer expressions.

1. $\boldsymbol{d}^{\mathrm{emb}}\in\mathcal{R}^{L\times N}$. The L2 norm of $d^{emb}$ is actually $\|(\boldsymbol{d}^{emb})_l\|=1, \dots, L$, which means the embedding of each token is normalized.
2. $W \in \mathbb{R}^{N\times D}$ satisfies the Semi-Orthogonal Weight Matrix assumption $\boldsymbol{WW}^T = \boldsymbol{I}_N$, which is necessary to achieve mathematical tractability. Since practical PLMs uses 2-layer MLPs rather than the weight matrix $W$, this can't be verified directly. If we ignore the activation function in the MLPs of BERT, let $\boldsymbol{W}=\boldsymbol{W}_1\boldsymbol{W}_2$，then $\frac{1}{N^2}\|\boldsymbol{WW}^T\|_F \approx 50\cdot\frac{1}{N^2}\|\mathrm{diag}(\boldsymbol{WW}^T)\|_F$, which suggests that the diagonal elements are much larger than the others. One reasonable intuition is a conclusion in high-dimension probabilities which states "for any $\epsilon>0$, there are $m=\Omega(e^N)$ vectors in $\mathcal{R}^N$ such that any pair of them are nearly orthogonal."[3] This part is now added in the Appendix C.1. Since $N \geq 768$ for commonly-used retrievers, the hypothesis holds with high probability.
3. Encoder-decoder cooperation assumption has a certain practical background. The experiment in Section 4.2.2 can be viewed as a verification of this assumption, where finetuned encoder is used with unfinetuned MLPs to do MLM task. In this setting, the hybrid model recieve relatively low perplexity as PLMs. In practice, the beginning learning rate of finetuning retrievers is usually set at $1\cdot e^{-5}$, which makes retrievers more likely to the conserve of inversion property. This part is aslo added in the Appendix C.1.
4. $\boldsymbol{d}\_{L}\in\mathbb{R}^L$ satisfies $k_l = \sum_d^D (\boldsymbol{d}^{\mathrm{emb}} \boldsymbol{W})_{ld}$ we claim $k_l > 0$ as an assumption in the latest version.
5. QM-AM states quadratic mean(the L2 norm of $(\boldsymbol{d}^\mathrm{emb}W)\_l$) is greater than the arithmetic mean ($k_{l} = \sum_d^D (\boldsymbol{d}^{\mathrm{emb}}\boldsymbol{W})_{ld}$).

Hope our explanations are helpful in addressing your concerns . We sincerely apologize for any non-standard use of notations that may have caused confusion, and further in-depth discussions are welcomed to make theoretical analysis more clear.

Dear authors, I increased soundness contribution and score by 1 Regards,

Thanks for acknowledging our work and the constructive comments. Please kindly find point-to-point responses below.

W1: The focus on small and older models like BERT and RoBerta is slightly exclusive of more recent LLMs and SLMs.

Response: Thanks for the inspiring comments. we will explain this from two aspects. (1) We indeed initially focused on analysis on MLMs for the following reasons: BERT-style MLM architectures are generally considered more suitable and widely used in textual representation tasks such as infomation retrieval, as we have discussed in the Appendix C1. In fact, latest SOTA retrievers from MTEB benchmark are MLM-based [1, 2]. (2) Following your feedback, we have expanded our analysis to include other architectures like CLM and find that our theoretical framework remains applicable. We find that although different architectures utilize diverse contextual information in token prediction, this variation does not alter the fundamental methods for calculating perplexity and relevance scores. More specifically, in Corollary 1, the variation between architectures only affects the matrix of partial derivatives of the embedding vectors with respect to the input text $\frac{\partial \boldsymbol{d}_2^{\mathrm{emb}}}{\partial \boldsymbol{d}_2}$, but does not impact the first-order gradient term of the loss function with respect to the embedding vectors $\frac{\partial \mathcal{L}}{\partial \boldsymbol{d}^{\mathrm{emb}}}$. Therefore, Corollary 1 still holds.
We will include these expanded findings and present a more universally applicable conclusion in the final version. Thanks for your valuable comments again.

W2: Similarly, there is some lack of cross-model experiments for models used to generate data and for the retrieval task, popular closed- and open-source models could demonstrate very different results (A more inclusive experimental setup would be more convincing that your argument generalizes and isn’t overfit to your experiments).

Response: Thanks for the advice. Considering that the web content maybe generated by diverse LLMs, we expand our evaluations to assess the generalizability of the CDC method across datasets generated by different LLMs,including Llama2, GPT-4, GPT-3.5, and Mistral. The results displayed in Table 3 in the PDF confirm that CDC is capable to generalize across various LLMs and maintain high retrieval performance while effectively mitigating bias. We will place greater emphasis on the cross-model generalization capabilities of CDC in the final version.

W3: There might be a limitation with closed models where training data is unknown for the CDC methodology.

Response: Thanks for the insightful comments. Given that CDC method can generalize across different domains and different LLM-generated dataset, it is feasible to create training pairs on any human retrieval datasets through LLM rewriting. Usually, 128 training instances are sufficient to effectively mitigate source bias, making the cost of constructing the training data acceptable. We will provide more details and explanation on constructing the training data for CDC in the final version.

Q1: How relevant do you think your study will remain given the rise of synthetically trained models, given the effects of low ppl and high relevance on LLM-generated data by majorly human-data trained retrieval models in your experiments?

Response: Thanks for the enlightening question. Currently, there is a lack of in-depth exploration regarding the impact of training data on Source Bias in PLM-based retrievers. An intuitive analysis suggests that the Perplexity Trap is not caused by differences in PPL between positive and negative samples in the training data. This is because a document may be relevant(positive) to a query and irrelevant(negative) to another. The rise of synthetic training data may offer the following insights. On one hand, the main conclusions drawn so far are empirical findings from collected data, hence there are potential variables that may affect Source Bias. Using synthetic data to conduct controlled experiments is a promising direction for exploration, of which our experiments in Section 3.1 can be viewed as a practice. On the other hand, our research can provide more inspiration for the training of retrieval models, such as how to synthesize training data to correct potential bias or train unbias model from scratch, similar to the use of counterfactual samples to enhance LLM training.

We hope the above response can address your concerns and look forward to further discussions. Thanks again!

[1] jina-embeddings-v3: Multilingual Embeddings With Task LoRA, arXiv: 2409.10173

[2] mGTE: Generalized Long-Context Text Representation and Reranking Models for Multilingual Text Retrieval, arXiv: 2407.19669