Time-Reversal Provides Unsupervised Feedback to LLMs

We sincerely thank the reviewer for their valuable feedback on our work. It is indeed encouraging that the reviewer finds our work to be of great significance to the NeurIPS community and worthy of a strong accept.

• Clarity in some parts: We thank the reviewer for this suggestion. We acknowledge that the explanation around Table 1 needs better clarity. We will add a Problem Statement section outlining all tasks we address in this work using the additional page in the camera ready version.
• Section-6 as a subsection of Section-5: We agree with the reviewer, and we will update Section-6 to be a subsection of Section-5 in the final version.
• Improvement percentage: We thank the reviewer catching this error. We will change the same to "improvement by 44.19 points" in the final camera ready version.
• Significance of Section-4: We thank the reviewer for this suggestion. We would like to clarify that the theory presented in the paper is targeted at explaining the intuition behind the proposed approach, and is a merely intended to be a proof of concept that it works in some stylized settings. We thus acknowledge this in the Limitations section as well, as noted by the reviewer. We would like to emphasize that in the theory section, we also show that if the reverse score (P(Q|A)) is used as a reward function in an RLHF framework (which can be approximated by best of N reranking as shown in a recent work [1]), it leads to sampling Answer such that it is proportional to P(Question|Answer)*P(Answer|Question). This is significantly different from the conventional forward scoring, which we show to be merely a temperature scaling of the usual P(Answer|Question) distribution. This observation, while easy to derive, makes the motivation for reverse scoring very clear and conveys the key intuition that it considers both conditionals of P(A|Q) and P(Q|A) without the need for additional hyperparameter tuning during reranking. We will clarify this takeaway in the final version. We hope to be able to add the additional explanations related to experiments in the extra page that we are allowed in the camera ready version. In case of lack of space, we will certainly move parts of the theory as well to the Appendix.

[1] J. Q. Yang, S. Salamatian, Z. Sun, A. T. Suresh, and A. Beirami. Asymptotics of language model alignment.

We thank the reviewer again for their feedback and recognition of our work. We will be happy to address any further concerns as well.

We sincerely thank the reviewer for their valuable feedback and relevant references. We are happy that the reviewer finds this direction to be relatively unique, the toxicity filter to be an interesting idea, and the theoretical motivation to be helpful. We hope to address the reviewer's concerns in this rebuttal.

• Novelty of the core idea and related works: We thank the reviewer for suggesting references that are relevant to our work. We will certainly add a discussion regarding these works in our revision. We briefly outline the discussion below:

  • Li et al. [1] use mutual information between source and target for decoding. TRLM-Ba, trained in reverse token order, scores in reverse the forward generations and it results in outputs that satisfy a non-trivial objective for decoding. To see this, we argue in the theory section (Lemma 2), that doing RLHF using only the reverse score effectively samples answers proportional to $P(A|Q) P(Q|A) = \frac{P(A,Q)^2}{P(A) P(Q)}$. We note that this expression when summed over all answers and questions would be the $\chi^2$ divergence between dependent and independent distribution. This shows that forward generation’s contribution is the $P(A|Q)$ term while reverse scoring’s contribution is $P(Q|A)$ and it actually achieves the objective (in a qualitative $\chi^2$ sense instead of the KL sense) as in [1].

  • DialoGPT proposed by [2] pretrained a "backward" model to score sources given targets -- this is specific to conversational understanding. This is different from results on general instruction following benchmarks we present, which are significantly more general and open ended. For very large language models and on public, challenging benchmarks like AlpacaEval that test their instruction following capabilities, our work shows that full reverse token pre-training and reverse scoring gets the best set of results in re-ranking which is novel.

  • Zhang et al., [3] propose adversarial mutual information maximization, by using a "backward proposal network" during training of the model to maximize mutual information to improve the informativeness of generated responses. Further the proposed method involves GAN style training which could be unstable. We sidestep this issue by either prompting a forward model in the reverse direction (TRLM-Fo) to score or pre-training a model to learn in reverse that naturally scores in reverse (TRLM-Ba). We propose a method that can be used with any LLM and can be applied to any query.

  • In summary, all previous works seem to motivate the need for better decoding based on scores in both directions. We show that reverse scoring alone, when used with forward generations, will achieve this naturally using a formal RLHF based argument and strong empirical results to back it.

  • As the reviewer has noted, we explore several other novel applications such as citations, retrieval and defense strategy to amplify input filters. This demonstrates the generalization of TRLM across different tasks.

• Projecting output responses to input space: We would like to clarify that the projection of output into the input space is in fact a key ingredient of the proposed defense.

  • We claim that given a toxic output, there is a non-zero probability of generating a relevant question that is classified as toxic (although non-toxic questions may also be generated as the reviewer pointed out). This probability is significantly higher than the probability of generating a toxic question given a safe response. We specifically rely on this property to design our defense, which also empirically holds from the results. Thus, the proposed method augments input filters effectively while maintaining a low False Positive Rate. Thus we think the projection of output responses to the input space is indeed a principled approach.

  • We would like to further clarify the intuition behind the proposed defense. Output filters get confounded by the presence of artifacts in a long response, while input filters can focus only on higher level reasoning of whether a question can elicit toxic content. This is evident from the significantly better performance of input filters empirically when compared to output filters in recent works (Table-1 in [4]).

  • We also perform a small ablation to highlight the better reasoning capabilities of input filters by taking toxic questions and jailbreak responses from Jailbreakbench for Llama-2, and checking the accuracy of predicting this as toxic by input and output filters. We additionally encode the prompts and responses by inserting a space between each character to check the understanding of the filter. We note that in both cases, the LLM is the same, and the input and output filters differ only with a prompt/ response indication in the system prompt. It can be noted that the true positive rate drops by 51% in case of input filter, and by 88% in the output filter - highlighting the better reasoning of input filters.

• Our strategy thus allows the model to reason, where the projection of response to inputs acts as a chain-of-thought step while filtering output content. Thus, when we project a toxic response back to the input space, we combine the benefits of input and output filters while overcoming their individual shortcomings.

• Minor Points: We will correct the notational issues and typos pointed out by the reviewer.

[4] ShieldGemma: Generative AI Content Moderation Based on Gemma

We will be happy to address any further concerns as well.

We sincerely thank the reviewer for their feedback. We are happy that they find our work to be novel and versatile, our theory valuable, our experiments comprehensive and our improvements significant. We clarify the concerns in this rebuttal.

• [Limitations section] Disclaimer on assumptions in theory: We clarify that the theory presented in the paper is targeted at explaining the intuition behind the proposed approach, and is intended to be a proof of concept that it works in some stylized settings. We thus include a disclaimer in the Limitations section to caution the reader about this. We will rephrase this is in the final version and clearly state that the method is not theoretically grounded, and requires empirical justification, which we present in Sections 5 and 6.
• [Broader Impact section] Disclaimer on defense evaluations: We clarify that we indeed report results on the popular Jailbreakbench benchmark, that contains a diverse set of questions with toxic answers from several models families such as Llama, Vicuna and GPT. However, since the defense relates to the sensitive application of safety, we include a disclaimer to caution the reader to verify it on domain-specific applications before deploying the defense in practice. We will rephrase the same to clarify this in the paper.
• Computational costs: The cost for reranking is negligible compared to the overall inference cost which we explain below. Consider a setting where an LLM is deployed and served to end users. To answer the user's query, the LLM generates 16 responses simultaneously in a batch for $t$ decode steps ($t$ is the length of the response). Since the inference is autoregressive, this process introduces a factor of $t$ to the total inference time. Subsequently, TRLM is used for scoring all 16 responses in a single pass, while using nearly 10x lesser parameters. Assuming that compute scales linearly with the number of parameters, we can compare the computational time of:
  • (a) Multiple response generation which is $t$, and
  • (b) TRLM re-ranking which is $1/10$, since scoring is done in a single pass by a model of roughly 10x lower capacity Thus, (a) takes nearly $10\cdot t$ times more time than (b). Considering that $t$ (response length) is generally between 20 and 500, the additional cost of reranking is negligible.
• Generalization of TRLM to other languages: We perform experiments to check this, and find that TRLM based reranking is indeed valuable in other languages as well. We present the experiment and results below.
  • For a direct comparison with the Alpaca Eval results presented in the paper, we perform reranking for Alpaca eval in German. For a fair comparison, we use the same set of English questions and 16 English responses (from Gemini-Pro-1.0) that were used for presenting results in Table-2 of the paper. Only for reranking, we translate the question and responses to German using the Google Translate API and ensure that the TRLM models rerank German answers. Since the judge is designed and verified only for English answer evaluation, we use the corresponding English answer of the best response as input to the judge.
  • We present the results in the below table. LCWR refers to Length controlled win rate while WR refers to normal win rate. We observe significant gains of 2.69% and 6.82% for German reranking using TRLM-Fo and TRLM-Ba respectively when compared to reranking all 16 responses using the standard forward direction (Forward Baseline). We note that these gains are similar to the reported gains. Thus, our TRLM-PaLM models are valuable for reranking in other languages as well.

• Potential risks of safety filters: The key risk involved in any defense is to incorrectly classify benign inputs as toxic, or a higher false positive rate (FPR). We show in Table-6 and Fig.4 that the proposed approach effectively amplifies input safety filters, i.e. reduces False Negative Rate (FNR) while not increasing FPR.
• Sensitivity to prompts: We use simple prompts in all cases to ensure that the observed results are not a result of excessive prompt engineering. We note that it may be possible to perform prompt engineering to improve scores in all cases. TRLM-Ba however does not require any prompts other than the "Question:" and "Answer:" tags, since Answer -> Question is its natural scoring direction. This is a key advantage of this model over TRLM-Fo. We further present results with two extreme cases of prompts for TRLM-Fo (Forward model prompted in reverse):

We note that predicting $P(Q|A)$ without any prompting (Simple) works best. This is very close to what we used for reporting, and thus represents a fair comparison with TRLM-Ba, which also does not use prompting.

• Potential biases: We note that bias/ fairness issues do exist in LLMs, and apply to our work as well. The proposed approaches merely score the quality of the response, and thus do not present any additional bias/ fairness concerns to the best of our knowledge. We will include this as a caution to the reader.
• Privacy implications of defense: Our defense works on LLM responses and not on user data. Since LLM responses already protect user privacy, the proposed defense does not pose additional threats.

We will be happy to answer any further questions as well.