Enhancing LLM Faithfulness in Rationale Generation via Dual-Reward Probabilistic Inference

Thank you very much for spending your valuable time in reviewing our work and we address your concerns as follows.

Q1: Missing references

R1: The paper “Evaluating Human Alignment and Model Faithfulness of LLM Rationale" was released after our paper submission, and we appreciate you bringing it to our attention. This paper focuses on faithfulness explanation from human perspectives and provide important insights for future faithful evaluation. We will add to the our related work discussion. We have read the paper “On Measuring Faithfulness or Self-consistency of Natural Language Explanations” when preparing our paper submission. It introduced a method of calculating faithfulness as a continuous value rather than a binary measure. We didn’t cite it as we followed more traditional and widely-used methods for faithful evaluation, as outlined in[1,2,3]. We will add a discussion of this faithfulness metric in the related work section.

Q2: Explanation of Figure2

R2: For faithfulness, we firstly observed that domain-specific words are important in enhancing context-adherence in Table1, which inspired us to increase the generation probability of domain-specific words to enhance faithfulness of the rationales generated. This motivated the design of our approach using global rewards (expert model). To the best of our knowledge, this is the first time faithfulness has been studied by encouraging domain-specific tokens. In the experiment, we show the distribution of domain-specific words for both the backbone model and our method, highlighting that our model successfully generates more domain-specific words (indicated by the blue line above the yellow line).

Q3: Experiment on other backbone

R3: We have updated the evaluation results on Mistral 7B, shown in Appendix B1. The evaluation results shows that our method can also achieves better results in accuracy and better faithfulness in rationale.

Q4: Experiment on other datasets

R4: Thanks for your suggestions. We have tested our method across three distinct tasks, i.e., student essay assessment, natural language inference and question answering, using 7 different datasets. The consistent improvements across these tasks validate the effectiveness of our methods, with an absolute accuracy improvement of 33% across the seven datasets, along with a 10% improvement in faithfulness evaluation. Please kindly let us know if you have any particular datasets you recommend for further evaluation.

Please let us know if you have further concrete questions or concerns that we can address. Thank you for your engagement with our work.

The discussion that we can participate in will end soon. Could you kindly confirm whether our responses and the revision in the newly uploaded pdf have appropriately addressed your concerns? If you find that we have properly addressed your concerns, we kindly request that you consider adjusting your initial score accordingly. Please let us know if you have further comments.

Thank you for your valuable time and effort in reviewing our work.

Thank you for your valuable time and your detailed feedback! We address each point as follows.

Explanation of the proposed method

The equations referenced are primarily derived from the probabilistic framework of the Feynman-Kac model. We have significantly revised Sections 3.3 and 3.4 in the manuscript, along with an updated Algorithm 1 that introduces our method in a step-by-step manner. Please refer to the revised PDF for detailed explanations. Below is a pointwise response to your proposed questions:

• Q1: Motivation of applying Feynman-Kac Formulae in the faithfulness-seaking search framework?

• R1: We refer to the Feynman-Kac framework, which is a general framework of incorporating a potential function to adjust the original conditional probability. For faithfulness, we observed that the expert model tends to generate more domain-specific tokens, which contribute to context-coherent and faithful rationales. This insight motivated us to employ the expert model’s conditional generation probability as the potential function to adjust the backbone model’s conditional generation probability accordingly.

• Q2: Explanation of Eq1.

• R2: $\mathbb{P}_t​(s_t​)$ represents the probability of reaching $s_t$ from the $\mathbb{P}_t$. Specially, $[S_t=s_t]$ is an indicator function that is equal to 1 if the state at $t$ is $s_t$, and 0 otherwise. The numerator inside the expectation represents the product of rewards and the probability of reaching state $s_t$, ensuring that paths leading to high rewards over time are given more weight. For better understanding, we connect with concepts in MCTS. Rollout in MCTS is used to estimate the value of future actions, helping navigate and expand the search tree effectively. The Eq.(1) represents a probabilistic reward distribution of state $s_t$. $G_t$ is analogous to reward function in MCTS, with superiority in estimate the reward via lookahead. In our framework, the global expert model $U^g$ performs reward estimation to the generated policy from backbone model (see GlobalReward function).

• Q3: Explanation of line 179 about motivation of local constraint.

• R3: Firstly, we would like to emphasise that attributes in language, such as toxicity and faithfulness, can manifest across longer spans of text rather than being confined to explicit attribute-bearing words. This is why we opt not to utilize token-level constraints like logit fusion [1] for addressing the faithful rationale generation problem. Such methods may fall short in capturing the nuanced and distributed nature of these attributes over extended contexts. Instead, we align with existing literature highlighting the effectiveness of domain-specific experts in achieving higher accuracy on tasks rich in domain knowledge. For example, task-specific classifiers are known to excel in tasks requiring specialised expertise. This observation inspires our approach of incorporating domain experts for label prediction to enhance both accuracy and faithfulness in rationale generation.

• Q4: why not just use the expert to predict the scores.

• R4: Thanks for raising this interesting point! (a) The primary goal of our framework is to maintain a trade-off between task performance (accuracy) and rationale faithfulness. Incorporating predicted results directly as prompts can sometimes lead to issues like hallucination and irrelevant context generation, as observed in models like ChatGPT [2]. (b) From the results in Table 5 and Table 6, it is evident that relying solely on local or global expert guidance fails to strike the required balance. This underscores the importance of our dual-reward framework, which combines both local (answer accuracy) and global (faithful rationale) rewards. Our approach ensures the LLM generates outputs that are not only accurate but also aligned with domain-specific knowledge. (c) Our framework integrates expert predictions in a probabilistic manner, i.e., treating the prediction as a conditional input within the generation process. This will fundamentally guide the rationale generation process and is likely to produce a rationale that is more consistent and faithful to the prediction.

• Q5: local contraint

• R5: For better clarity, we remove the Eq2 and Eq3. The calculation of Local reward is as follows: We introduce a set of classification label words $\mathcal{C}$ and we remove the label words which are not included in the expert model’s prediction $c_0$. We then renormalise the output probability based on the new vocabulary (see function LocalMask for details).

Missing comparison with expert model

Thank you for pointing out this critical point. The comparison results are updated in Table 5 and Table 6, where CLS is the expert model used to predict the answer and the expert model refers to the global expert model used to generate rationale. As CLS is a classifier that can't generate rationale, we only apply it in acc evaluation, while the expert model is applied in both metrics calculation. Below is our analysis:

(a) comparing with fine-tuned backbone model: We have updated the ablation results in Table 5 and Table 6. For the ASAP dataset, the expert model uses the same backbone as the primary model (i.e., Llama3-8B) and has been fine-tuned on the ASAP training set. This aligns well with your intended comparison with Vanilla Fine-Tuning of Backbone Model

• Accuracy Results: In Table 5, by comparing the expert model with our full framework, we observe overwhelming advantages of our full model across all four subsets, especially on Q2, with results showing 68% vs. 48% (ours vs. expert). And the average results are 74% and 69% for our (full) and expert model.
• Faithfulness Results: The expert-only method achieves better faithfulness on three subsets (except for Q4). Interestingly, our full framework (including CLS & expert) behaves better than our+expert model, showing the synergised effects of CLS and expert.
• Overall Results: Although our method fails to exhibit better faithfulness compared with the fine-tuned model expert model, it instead strikes a challenging trade-off between accuracy (CLS) and faithfulness (expert model), which has been discussed in [1]. This is also one of the core motivations: combining the strengths of classification-specialised and rationale-specialised models.

(b) compare with smaller fine-tuned models:
For other datasets where the expert models are Llama2-7B (smaller than our backbone): (i) The comparison between our full (i.e., our+cls+expert) and our+expert also shows that the synergistic effects of combining the two experts. (ii) the faithfulness for our (full) is better than the expert model on SNLI and MNLI, with 15% vs 13% and 19% vs 9%, respectively. This suggests that our approach not only achieves faithfulness improvements over smaller expert models but also has the potential to leverage weak supervision to unlock the capabilities of larger backbone models.

(c) Generalised to Out-of-Task Expert Model:
Note that it is not strictly necessary for the expert model to be trained on the exact task dataset. For instance, we experimented with Expert Model 2 (/Weyaxi/Einstein-v2-7B in huggingface), which is trained on general science question-answering instead of ASAP-specific data. Results in Table 12 show that incorporating this out-of-task expert model leads to improved faithfulness on 11 out of 12 metrics. These results validate the generalisability of our method.

Overall, our framework demonstrates (i) superior performance in balancing accuracy and faithfulness when the expert model is of the same size, (ii) improved faithfulness compared to smaller, in-task trained expert models, and (iii) robust generalizability, effectively adapting to scenarios where the expert model has not been specifically trained for the inference task.

Evaluation details: Thanks for asking the question about detailed evaluation, which makes our method clarified. We have updated the experiment setup section in Appendix A, with prompt used in different tasks, and the hyperparameter configuration.

Q1: The exact prompt and task format used
Yes, we use zero-shot evaluation. Please refer to the prompt template in Appendix A.

Q2: How to extract answers from the outputs to calculate the accuracy?
The answer/predicted labels are extracted using specifically designed regular expressions corresponding to our format requirements within our prompt instruction.

Q3: Explanation of the undesirable performance of backbone models
The backbone model performs lower since those models are not directly trained on our selected datasets. This is also evident in the LLaMA paper [2] as well, which shows that zero-shot inference on TruthfulQA with the LLaMA 7B model has an accuracy of 33%.

Q4: number of rollouts
We use a beam of 3 and 10 particles for decoding. Therefore, 30 different rollouts are calculated during decoding.

References:
[1] Question Decomposition Improves the Faithfulness of Model-Generated Reasoning. ICML23
[2] LLaMA: Open and Efficient Foundation Language Models. 2023

We've taken your initial feedback into careful consideration and incorporated them into our revised Pdf as indicated in the Summary of Revision. Could you kindly confirm whether our responses have appropriately addressed your concerns? If you find that we have properly addressed your concerns, we kindly request that you consider adjusting your initial score accordingly. Please let us know if you have further comments.

Thank you for your time and effort in reviewing our work.

We sincerely appreciate the reviewers' valuable time and thoughtful feedback, which have provided us with an opportunity to further clarify our contributions and strengthen our work.

Inconsistant notation, $f_\theta$, $\pi_t$

We totally agree that consistency and clarity are very important, so we have updated the PDF to keep the complete form. Actually, We use $f_\theta$ refers to a parameterised function, explicitly highlighting that $\theta$ are the parameters. While in Algorithm 1, the parameters are not the focus, so we use $f$ throughout the algorithm. For $\pi_t$ and $\pi$ in the localmask function, we omit the $t$ as we have the input within the function $\pi$, such as $\pi(x_t=v’|x_{t-1})$.

How Algorithm connected to Feynman-Kac and search algorithm

Q1: No Adjustment to $\pi$?

R1 The Feynman-Kac framework is a classical framework proposed in 2024 and serves as a general foundation for probabilistic inference. Its key idea is to introduce the potential function $G_t(s_{t-1}, s_t, f_\theta)$, which acts as a reward function to score the current state. Some methods modify the policy $\pi_t$​ to derive a new policy $\pi’_t$. An example is our localmask method. In contrast, our global reward mechanism reweights the output probabilities using an expert model, allowing tokens with relatively low probabilities to be sampled while eliminating high-probability tokens. In that way, we could change the selected topk tokens according to the score from expert model. Based on the changes in the output sequence at step $t$, the backbone model will generate differently at timestep $t+1$, as the conditional generation depends on the updated conditioned input. It’s worth noting that there are numerous practical approaches to modifying output distributions. The core idea of the Feynman-Kac framework is to employ a function with lookahead characteristics, which helps avoid local optima.

Q2: Why search algorithm?

R2: The existing Monte Carlo Tree Search (MCTS) algorithm, utilizes a policy model to generate multiple candidate nodes. These nodes are then evaluated by a reward model, and the state (node) with the highest reward is selected for tree expansion. Similarly, in our Algorithm 1, at each timestep, the backbone model generates K tokens (where K is the beam search size). These tokens are then evaluated by the expert model. Among the $K \times N$ generated trajectories, only the trajectory with the highest reward is selected. While our search process does not exactly replicate the step-by-step node selection and expansion in MCTS, it generates multiple trajectories in parallel to reduce computational cost and it should still be categorised as a MCTS algorithm, such as the paper [1][2].

I still don't understand why Localmasking would benefit faithfulness. To me it look like a way to improve the accuracy.

Yes, your understanding is totally correct. Local masking (CLS) benefits the accuracy, while global reward (expert) benefits the faithfulness.

So seems like the whole algorithm is to use domain-expert to predict the correct label, then use the large backbone model to generate some candidate explanations, and use another faithful expert to score them.

In general, yes, we use a domain-expert model, a classifier (CLS), to ensure prediction accuracy, and a generative expert model to enhance faithfulness. However, we would like to clarify two key points:

• The predicted label from the CLS is not directly used as the final label.
• The expert model is applied step-wise rather than after the sequence generation is complete. These differences can lead to distinct outcomes. For example, we generate 3 tokens at each timestep $t-1$, the step-wise approach ensures that the top-K tokens are selected based on the expert model’s score, denoted as $w_t^{e}$​ (e.g., apple, orange, peach). This selection may differ from the top-K tokens determined solely by the backbone model’s output distribution, $w_{t-1}^{b}$​ (e.g., this, the, that). Each token in $w^{e}_{t-1}$ is then incorporated into the conditional context to guide the backbone model’s generation at $t$. For instance, the generation at $t$ would be conditioned on that is $\pi_t(x_t|\text{apple})$, rather than is $\pi_t(x_t|\text{this})$. Thus, the expert model provides process-level, fine-grained controllability over the generation, rather than a post-hoc adjustment.

how the class label words $\mathcal{C}$ are chosen still remains.

Our label words $\mathcal{C}$ are selected based on the datasets’ label space, see the details in Line 238-239. Please let us know if you have further concrete question.

CLS The definition of the term "CLS" is removed from the revised paper, where it used to be the first sentence of section 5.1.

Thank you for noticing the change! We apologise for this mistake and have corrected accordingly in the newly updated revised version.

Let the backbone model answer directly can also potentially hallucinate. I don't understand why localmasking will hallucinate less than directly use domain-expert to predict the label. As for irrelevant context generation one can simply do as in local masking to restrict the domain-expert's prediction only on the label words for extracting the answer.

R1: According to the response above. Our method is not equal to let the backbone answer directly, the expert model does affect step-wise generation.

R2: Localmask is essentially classifier (not for generation task, so that they can't generate label words), and they are more likely to generate highly accurate prediction than generative models, i..e, expert model, as shown in Table 5. Therefore, we do not use the generative model for label prediction.

I don't understand your point. From Table 5 apparently CLS (Expert model) perform better than your method on all datasets? It is also mentioned in the paper "Notably, when incorporating the CLS, our method does not necessarily perform as well as the classifier alone." So why not just use CLS to predict the label word? Anyway your method uses domain experts, so why not choose the most accurate model for generating the answer? The faithfulness of the rationale can be addressed separately.

Sorry for the confusion.

• First, we would like to clarify that our framework utilizes two types of expert models for local and global rewards, respectively. The local reward model, referred to as CLS in our experiments, is a classifier (not a generative model) and serves as the primary source of accuracy by scoring prediction consistency. The global reward model, referred to as the expert model, is a generative model and primarily contributes to enhancing faithfulness in the generated outputs.
• Second, if I understand correctly, the idea is to use two models—one aimed at accurate answer generation and the other at maintaining faithfulness. We acknowledge that improving both accuracy and faithfulness within a single model remains a key challenge, as emphasized in recent works (e.g., [1]). Moreover, even if we had access to accurate labels and directly fed them into the generative model, this approach would not necessarily guarantee improvements. On the contrary, incorporating predicted results directly as prompts can introduce issues such as ungrounded hallucinations, as highlighted in recent studies [2, 3]. Thus, effectively balancing accuracy and faithfulness continues to be a non-trivial and active area of research.

Please let us know if you have any concerns about our motivation behind or any other concrete questions.

Thanks again for your patience and valuable time in enaging in our work!

References

[1] Question Decomposition Improves the Faithfulness of Model-Generated Reasoning.

[2] Siren's Song in the AI Ocean: A Survey on Hallucination in Large Language Models.

[3] Chain of Natural Language Inference for Reducing Large Language Model Ungrounded Hallucinations

Thanks for adding the expert model results. In this case, I don't see a clear benefit of the proposed method over the expert models. As for the argument that the proposed method achieves a trade-off between accuracy and faithfulness, one can easily combine CLS and Expert by first predicting label words using CLS then explaining the answer using the global expert model. I feel this should be a strong baseline and would at least perform as well as the proposed method in achieving the balance between accuracy and faithfulness.

Thanks for raising this critical point! Your question about the performance about backbone model inspired us to rethink the evaluation results. As the model are differently sensitive to the text, so the faithfulness metrics vary lot across different datasets, for example, from 0.01 to 0.05 on ASAP, and around 0.1 for NLI task. Consequently, we have provided a normalised faithfulness evaluation table below. The faithfulness score is scaled by the maximum and minimum scores in each task.

Table1: Normalised faithfulness evaluation

Our full method achieves the highest faithfulness score compared to using the backbone or the expert model alone.

Then, we provide an average faithfulness-accuracy comparison in Table2.

Table2: Averaged normalised faithfulness-accuracy

Our(full) achieved the highest in a combination of accuracy and faithfulness evaluation, with clear improvements.

Furthermore, we are in the process of implementing the baseline you mentioned—using the label predicted by CLS as a prompt for the rationale generation model. We will update our results accordingly once this implementation is completed.

Thank you again for your valuable suggestions. We are truly grateful for your feedback, which has significantly improved the clarity of our paper. We sincerely hope that the adjustments and clarifications we have provided address at least part of your concerns. If so, we kindly request that you consider adjusting your scores to reflect our efforts in responding to your feedback as a positive signal. Also, please let us know if you have further more suggestions!

Thank you once again for your detailed feedback and thoughtful comments!

We would like to emphasise that our expert model can select different $x_{t-1}$, based on the $x_{t-1}$, the backbone model which is responsible for trajectory generation will generate/sample differently as the generation probability is $p(|x_{t-1})$. Details in the response to why search algorithm above.

We greatly appreciate your valuable feedback, which has been instrumental in refining our work. We will continue improve our paper!

Thank you!

Q3: Eq.1-3 are doing in plain words.

R3: The equations referenced are primarily derived from the probabilistic framework of the Feynman-Kac model. We have significantly revised Sections 3.3 and 3.4 in the manuscript, along with an updated Algorithm 1 that introduces our method in a step-by-step manner. Please refer to the revised PDF for detailed explanations.

Additionally, we provided a general response to all reviewers, detailing how the two rewards—local and global—are calculated (Eq. 2 and Eq. 3). Below is a pointwise response to your proposed questions:

• Eq1: In the context of generating tokens $s_t$ using model $f_\theta$, The potential function $G_t$ maps $(s_t, s_{t+1})$ to a non-negative score, analogous to the reward function. The adjusted probability of $f_\theta$ generates $s_t$ is calculated Eq1. $[S_t=s_t]$ is an indicator function that is equal to 1 if the state at $t$ is $s_t$, and 0 otherwise. The numerator inside the expectation represents the product of rewards and the probability of reaching state $s_t$, ensuring that paths leading to high rewards over time are given more weight.} Generation continues until a terminal token or the maximum length of the sequence $T$, i.e., $t \wedge T=\text{min}(t,T)$
• Local Reward (Eq2): We introduce a set of classification label words and we remove the label words which are not included in the expert model’s prediction. We then renormalise the output probability based on the new vocabulary.
• Global Reward (Eq3): The expert model generates a lookahead reward by evaluating the plausibility of the tuple $(x_t, x_{t+1})$ generated from the backbone model.

References

[1] Faithfulness tests for natural language explanations. ACL2023

[2] Faithful explanations of black-box nlp models using llm-generated counterfactuals. ICLR24

[3] Can llms produce faithful explanations for fact-checking? towards faithful explainable fact-checking via multi-agent debate. 2024

[4] Confidence-aware learning for deep neural networks. ICML2020

Please let us know if you have further questions or any feedback. Thanks!

Thanks for reviewing our work and we address your concern in the generalisability of our framework.

Q1: Generalise to situations where the rationale is generated before the answer.

R1: It is certain that LLMs can generate answers either before or after the rationale.

• (i) To prevent scenarios where an overly long rationale causes the answer to exceed the output length limit, we prioritize generating the answer first. This is achieved by providing explicit instructions and demonstrations where the answer precedes the rationale, ensuring the backbone LLM generates the answer at the beginning.
• (ii) In our current setting, this approach is motivated by the observation that specialised smaller models often perform better at classification tasks. By leveraging these models for accurate rating, we establish a prior of a likely correct rating, which in turn helps ensure that the rationale generated afterward is more faithful to the truth.
• (iii) Through Algorithm 1 updated in Section 3.3, our incorporation of the function localmask for answer prediction is implemented at the first timestep. Meanwhile, the rationale reward is applied throughout the sequence. This design allows for the localmask to also be applied at the final timestep, guided either by a length limit or a terminal token indicator. Additionally, we need to evaluate whether feeding the generated rationale back into the classifier via the localmask function would degrade or enhance classification accuracy.

Q2: Limited tasks to reasoning.

R2: Thank you for raising this critical point.

Firstly, we would like to clarify that existing faithfulness evaluations are traditionally based on the assumption that answers can be straightforwardly evaluated as either identical or not. Faithfulness evaluation primarily focuses on determining whether changes in the input lead to corresponding changes in the answer. Evaluating open-ended questions introduces the additional challenge of assessing semantic equivalence, which is not the primary focus of most existing studies on faithful rationales. For example, research in this area often evaluates tasks with clear answer spaces, such as Natural Language Inference (NLI) and multiple-choice QA [1] (both of which are included in our evaluation), as well as sentiment classification [2] and fact-checking tasks framed as binary classification [3].

Secondly, our method is extendable to scenarios with an infinite label space $( |\mathcal{C}| = \infty )$, even though the current evaluations are conducted on tasks with a constrained label space $( |\mathcal{C}| = N \in \mathbb{N} )$. For instance, in mathematical problem-solving tasks, the answer could be any arbitrary number. In such cases, the expert model provides a prediction $M$, with its confidence expressed as the probabilities $w_1$ for $M$ and $w_2$ (for the second most probable prediction). The ratio $\frac{w_1}{w_2}$ serves as an indicator of the expert's confidence in $M$ [4]. This confidence is then used as a multiplier to enhance the backbone model's prediction for $M$. Finally, the backbone model's transition distribution is renormalised to ensure a valid probability distribution.

We've taken your initial feedback about (i) "the generalisability of method" (ii) writing about method into careful consideration and incorporated them into the revised pdf. Could you kindly confirm whether our responses have appropriately addressed your concerns? If you find that we have properly addressed your concerns, we kindly request that you consider adjusting your initial score accordingly. Please let us know if you have further comments.

Thank you for your time and effort in reviewing our work.

Many thanks for recognising the improvements in our updated version and for your valuable feedback! We also appreciate your constructive suggestion to further clarify the distinction between our method and ILQL.

Firstly, at a high level, both methods share the similarity of leveraging logit perturbation to enhance the output of the backbone model. For clarity, we denote the backbone model as $M_b$ and the expert model as $M_p$.

Then, we provide a detailed comparison in the following aspects:

Model Types

• ILQL: Both the backbone model (referred to as the standard language model) and the perturbed model are GPT-2 small.
• Ours: Our backbone models are LLaMA3-8B and Mistral-7B, while perturbed models include LLaMA3-8B, LLaMA2-7B, and Mixtral-7B. Importantly, our method supports backbones and perturbed models of differing sizes and tokenisations.

Does it require extra training before deployment?

• ILQL: Yes, there are three types of $M_{p}$: policy generation $\pi_B$, value function model for $Q$ and $V$ generation, and a target value network. They are trained via implicit Q-learning objectives on the exact dataset same as inference tasks.
• Ours: No, we can use any $M_{p}$ pretrained on in-task datasets, or even out-of-task datasets (see response above Fair Comparison with Vanilla Fine-Tuning of Backbone Model.)

Perturbation strategy:

• ILQL: calibrate the original policy using trained $Q$ and $V$, i.e., $\pi(a \mid h) \propto \pi_\beta(a \mid h) e^{\beta(Q(h, a) - V(h))} = \exp(\log(\pi_\beta(a \mid h)) + \beta(Q(h, a) - V(h))).$
• Ours: the step-wise reward is given by $M_p$, and then select the sequence with highest sentence-level reward from generated $K$ sequences (see in Algorithm 1). This reward strategy is inspired by our preliminary study in Figure 1 that $M_p$ prefers in-domain text and could contribute to faithfulness.

Overall analysis

• Our method can fit the situation where $M_b$ and $M_p$ have different but larger backbones. In contrast, ILQL's reliance on training three models for $M_p$ can limit its scalability to larger backbone models, such as fine-tuning a LLaMA3-8B. Our approach, however, can leverage any expert model, even if pretrained on out-of-task datasets (see the response above, Fair Comparison with Vanilla Fine-Tuning of Backbone Model).
• Despite the simplicity of our method, our global reward inherently captures lookahead characteristics, which are crucial in offline RL (including MCTS) for avoiding local optima. This aligns with the main contribution of ILQL, which highlights that “our offline RL method can lead to significant improvements in final performance as compared to such ‘single step’ approaches, particularly when the training data is highly suboptimal for the desired task.” Additionally, we provide a comparison with the local-optimal (single-step) method, i.e., logitfusion [1], in Table 7.

Reference:

[1] Tuning language models by proxy

We hope this pointwise comparison with ILQL addresses your concerns.

Please let us know if you would like a more detailed comparison on other aspects.

Dear Reviewer msmv,

The discussion that we can participate in will end soon. Could you kindly confirm whether our responses have appropriately addressed your concerns? If you find that we have properly addressed your concerns, we kindly request that you consider adjusting your initial score accordingly. Please let us know if you have further comments.

Thank you for your time and effort in reviewing our work.

Best, Authors

Thank you for your valuable time and your thoughtful feedback! We address each point as follows.

Writing about method

Thanks for your thoughtful questions regarding the method details. We have updated the method section (Section 3.3 and Section 3.4) with a newly updated Algorithm1 that elaborated on the pipeline, addressing your suggestion that it would be better to describe the method in more detail.

• The notation $t \wedge T$ represents the minimum of the two values $t$ and $T$. This mathematical convention is common in contexts like stochastic processes and optimisation. Here, it indicates that the product runs from $i = 1$ up to the lesser of $t$ and $T$. This effectively caps the sequence or product based on the minimum value.
• $\mathbb{P}_t​(s_t​)$ represents the probability of reaching $s_t$ from the $\mathbb{P}_t$. Specially, $[S_t=s_t]$ is an indicator function that is equal to 1 if the state at $t$ is $s_t$, and 0 otherwise. The numerator inside the expectation represents the product of rewards and the probability of reaching state $s_t$, ensuring that paths leading to high rewards over time are given more weight.
• Rollout in MCTS is used to estimate the value of future actions, helping navigate and expand the search tree effectively. The Eq.(1) represents a probabilistic reward distribution of state $s_t$. $G(s_t,s_{t+1})$ is analogous to reward function in MCTS, with superiority in estimate the reward via lookahead. In our framework, the global expert model $U^g$ performs reward estimation to the generated policy from backbone model (see GlobalReward function).
• Inference efficiency: we totally agree that inference-time cost is an important consideration in MCTS-like methods. Unlike existing explicit MCTS requiring expensive rollouts or simulations to evaluate potential actions, we compute expected rewards in a more integrated way, as shown in the numerator of Eq.(1), streamlining the decoding process. Our cost is similar to beam-search, while detailed comparison results is available in Table 14 (Table13 in original submission).
• Equation about reward calculation: to ease understand, we remove the Eq.2 and Eq.3, instead, we update function LocalMask and GlobalReward in Algorithm 1 in Section 3.3.
  • Local Reward: We introduce a set of classification label words and we remove the label words which are not included in the expert model’s prediction. We then renormalise the output probability based on the new vocabulary (See LocalMask function in Algorithm 1).
  • Global Reward: The expert model generate a lookahead reward by evaluating the plausibility of the tuple $(x_t, x_{t+1})$ generated from the backbone model (See GlobalReward function in Algorithm 1).

Fair Comparison with Vanilla Fine-Tuning of Backbone Model

Thank you for pointing out this critical point. The comparison results are updated in Table 5 and Table 5, where CLS is the expert model used to predict the answer and the expert model refers to the global expert model is used to generate rationale. As CLS is a classifier that can't generate rationale, we only apply it in acc evaluation, while expert model is applied in both metrics calculation. Below is our analysis:

(a) comparing with fine-tuned backbone model: We have updated the ablation results in Table 5 and Table 6. For the ASAP dataset, the expert model uses the same backbone as the primary model (i.e., Llama3-8B) and has been fine-tuned on the ASAP training set. This aligns well with your intended comparison with Vanilla Fine-Tuning of Backbone Model

• Accuracy Results: In Table 5, by comparing the expert model with our full framework, we observe overwhelming advantages of our full model across all four subsets, especially on Q2, with results showing 68% vs. 48% (ours vs. expert). And the average results are 74% and 69% for our (full) and expert model.
• Faithfulness Results: The expert-only method achieves better faithfulness on three subsets (except for Q4). Interestingly, our full framework (including CLS & expert) behaves better than our+expert model, showing that the synergised effects from CLS and expert.
• Overall Results: Although our method fails to exhibits better faithfulness compared with the fine-tuned model expert model, it instead strikes a challenging trade-off between accuracy (CLS) and faithfulness (expert model), which has been discussed in [2]. This is also one of the core motivation: combining the strengths of classification-specialised and rationale-specialised models.

(b) compare with smaller fine-tuned models:
For other datasets where the expert models are Llama2-7B (smaller than our backbone): (i) The comparison between our full (i.e., our+cls+expert) and our+expert also show that the synergistic effects of combining the two experts. (ii) the faithfulness for our (full) is better than expert model model on SNLI and MNLI, with 15% vs 13% and 19% vs 9%, respectively. This suggests that our approach not only achieves faithfulness improvements over smaller expert models but also has the potential to leverage weak supervision to unlock the capabilities of larger backbone models.

(c) Generalised to Out-of-Task Expert Model:
Noted that it is not strictly necessary for the expert model to be trained on the exact task dataset. For instance, we experimented with Expert Model 2 (/Weyaxi/Einstein-v2-7B in huggingface), which is trained on general science question-answering instead of ASAP-specific data. Results in Table 12 show that incorporating this out-of-task expert model leads to improved faithfulness on 11 out of 12 metrics. These results validate the generalisability of our method.

Overall, our framework demonstrates: (i) superior performance in balancing accuracy and faithfulness when the expert model is of the same size, (ii) improved faithfulness compared to smaller, in-task trained expert models, and (iii) robust generalizability, effectively adapting to scenarios where the expert model has not been specifically trained for the inference task.

Case study

We appreciate your observation that “submergible” does not necessarily mean the individual is submerged. Our key point here is to emphasise that our method can faithfully respond to meaningful changes in the input (even if the response is imperfect), whereas the backbone model completely ignores the perturbation.

For better clarity, we replace an example shown below (also updated in the revised pdf):

Hypothesis: Does not require free registration.
Backbone: entailment; Requires free registration is a necessary condition for only if Requires free registration.
Ours: Contradiction; The premise states that the website [frugally] requires free registration, which implies that a user must provide some information or sign.

References:

[1] The unlocking spell on base LLMs: Rethinking alignment via in-context learning. ICLR2024

[2] Question Decomposition Improves the Faithfulness of Model-Generated Reasoning. ICML23

[3] On Measuring Faithfulness or Self-consistency of Natural Language Explanations. ACL24

[4] Fine-tuning Large Language Models for Domain-specific Machine Translation

[5] SciBERT: A Pretrained Language Model for Scientific Text. ACL2019

[6] BioBERT: A Pretrained Biomedical Language Representation Model for Biomedical Text Mining. Bioinformatics 2020.

[7] Don’t Stop Pretraining: Adapt Language Models to Domains and Tasks. ACL 2020

Please let us know if you have further concrete questions or concerns that we can address. Thank you for your engagement with our work.

We've taken your initial feedback into careful consideration and incorporated them into our manuscript as indicated in our response. Could you kindly confirm whether our responses have appropriately addressed your concerns? If you find that we have properly addressed your concerns, we kindly request that you consider adjusting your initial score accordingly. Please let us know if you have further comments.

Thank you for your time and effort in reviewing our work.

As the discussion period is nearing its end, we kindly request your feedback on the comparison with ILQL provided below. We hope it addresses your concerns adequately.

We followed the thread of your feedback and noticed it was mistakenly placed in the "Reviewer msmv" block. To ensure clarity, we have moved our response here. We apologise for any inconvenience this may have caused and sincerely appreciate your attention to this matter.

Firstly, at a high level, both methods share the similarity of leveraging logit perturbation to enhance the output of the backbone model. For clarity, we denote the backbone model as $M_b$ and the expert model as $M_p$.

Then, we provide a detailed comparison in the following aspects:

Model Types

• ILQL: Both the backbone model (referred to as the standard language model) and the perturbed model are GPT-2 small.
• Ours: Our backbone models are LLaMA3-8B and Mistral-7B, while perturbed models include LLaMA3-8B, LLaMA2-7B, and Mixtral-7B. Importantly, our method supports backbones and perturbed models of differing sizes and tokenisations.

Does it require extra training before deployment?

• ILQL: Yes, there are three types of $M_{p}$: policy generation $\pi_B$, value function model for $Q$ and $V$ generation, and a target value network. They are trained via implicit Q-learning objectives on the exact dataset same as inference tasks.
• Ours: No, we can use any $M_{p}$ pretrained on in-task datasets, or even out-of-task datasets (see response above Fair Comparison with Vanilla Fine-Tuning of Backbone Model.)

Perturbation strategy:

• ILQL: calibrate the original policy using trained $Q$ and $V$, i.e., $\pi(a \mid h) \propto \pi_\beta(a \mid h) e^{\beta(Q(h, a) - V(h))} = \exp(\log(\pi_\beta(a \mid h)) + \beta(Q(h, a) - V(h))).$
• Ours: the step-wise reward is given by $M_p$, and then select the sequence with highest sentence-level reward from generated $K$ sequences (see in Algorithm 1). This reward strategy is inspired by our preliminary study in Figure 1 that $M_p$ prefers in-domain text and could contribute to faithfulness.

Overall analysis

• Our method can fit the situation where $M_b$ and $M_p$ have different but larger backbones. In contrast, ILQL's reliance on training three models for $M_p$ can limit its scalability to larger backbone models, such as fine-tuning a LLaMA3-8B. Our approach, however, can leverage any expert model, even if pretrained on out-of-task datasets (see the response above, Fair Comparison with Vanilla Fine-Tuning of Backbone Model).
• Despite the simplicity of our method, our global reward inherently captures lookahead characteristics, which are crucial in offline RL (including MCTS) for avoiding local optima. This aligns with the main contribution of ILQL, which highlights that “our offline RL method can lead to significant improvements in final performance as compared to such ‘single step’ approaches, particularly when the training data is highly suboptimal for the desired task.” Additionally, we provide a comparison with the local-optimal (single-step) method, i.e., logitfusion [1], in Table 7.

Reference:

[1] Tuning language models by proxy

We hope this pointwise comparison with ILQL addresses your concerns.

Please let us know if you would like a more detailed comparison on other aspects.