SCOPE: A Self-supervised Framework for Improving Faithfulness in Conditional Text Generation

Thank for your review and feedback. You will find below our answers to your questions. We have added new experimental results following each of your suggestions.

W1 & Q2. Evaluation on domain-specific datasets

Thanks for the suggestion. Benchmarking SCOPE on a domain where hallucinations are problematic could better demonstrate the robustness of the approach.

We followed the experimental setup of the recent article [1] on medical evidence summarization, applied to a subset of the PubMed dataset [2]. In this setting, we have evaluated SCOPE and the baselines on this additional dataset (see Tables 3 and 5 in the paper). SCOPE’s training method demonstrates strong faithfulness gains, compared to baselines and vanilla fine-tuning. We updated the paper to include this new dataset, hoping this will strengthen the robustness of the approach.

W2 & Q3. SCOPE on larger models

We followed your suggestion (shared with reviewer eSUQ), we extended our experiments to include SCOPE and the baseline methods on a larger Llama-2-13B model. The results are conclusive regarding scalability: SCOPE achieves a significant improvement in faithfulness compared to the baseline, demonstrating its consistent effectiveness across model scales. We have included these findings in the paper (see Table 3), as they further validate the robustness and scalability of the method.

W3 & Q5. Using other metrics for summarization

We also followed your suggestion and re-evaluated the models using two additional widely used metrics for assessing faithfulness in text summarization:

• QuestEval [3], a QA-based evaluation method.
• FactCC [4], a model trained to identify conflicts between the summary and the source document.

The results, now included in the paper (see Table 3), show that models trained with SCOPE achieve substantial improvements on these metrics compared to the baselines. We believe these findings further reinforce the reported effectiveness of SCOPE.

Q1. Rationale behind splitting the datasets

Our initial idea was to modify the training process by leveraging the training samples differently while maintaining the same amount of annotated data. We developed the SCOPE pipeline and tested it using a default 50/50 split, which we retained for the following experiment reasons:

• Fine-tuning on the first half of the dataset yields performance comparable to fine-tuning on the full training set (see Appendix A.3).
• Across all datasets, SCOPE with a 50/50 split consistently delivers strong results.

To further justify this split, we conducted an ablation study on ToTTo. In this study, we fine-tuned a model on 25% (resp. 75%) of the dataset and preference-tuned on the remaining 75% (resp. 25%) with noisy samples. Results on the validation set are shown in the table below. On automatic faithfulness metrics (NLI and PARENT), all splits yield comparable results, though a bit higher with a split of 50/50. Based on these results, we recommend sticking with a base value of 50/50 when using SCOPE. We have updated the paper to include this experiment in Appendix A.4.

[1] Tang, L. et al. Evaluating large language models on medical evidence summarization. npj Digit. Med. 6, 158 (2023). https://doi.org/10.1038/s41746-023-00896-7

[2] Cohan, A. et al. (2018). A discourse-aware attention model for abstractive summarization of long documents. Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers). Association for Computational Linguistics. https://doi.org/10.18653/v1/n18-2097

[3] Thomas Scialom et al 2021. QuestEval: Summarization Asks for Fact-based Evaluation. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 6594–6604, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.

[4] Wojciech Kryscinski et al . 2020. Evaluating the Factual Consistency of Abstractive Text Summarization. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 9332–9346, Online. Association for Computational Linguistics.

Thanks for your review and relevant feedback. Please find below our answers to your concerns and questions:

W1. Presentation of the baselines

We updated the paper to better present the baselines. CAD/PMI are decoding methods while Critic and CLIFF require training. We hope this enhances the clarity of the paper.

W2. Ablation on negative sampling creation

We followed your suggestion and applied our same training framework using as an alternative to our method a NER replacement method: the noisy samples are generated by running a NER tool and replacing entities by other random entities of the same type. We experimented on ToTTo and XSum (results on the table below). This method however, did not report gains as high than SCOPE. We believe this experiment strengthens our initial intuitions regarding the importance of leveraging hallucination patterns that align closely with the fine-tuned model’s behaviors.

W3-Q2. Adding metrics regarding negative examples

Following your suggestion, we scored the negative examples using our faithfulness metrics (PARENT for data-to-text and AlignScore for summarization). We updated the paper to include these analysis in Appendix C. Overall, we observe a clear decrease of faithfulness in the generated samples. Combined with a qualitative (human) analysis of the samples directly presented (see Table 18 in Appendix C), we believe we can conclude that negative sampling significantly introduces unfaithfulness issues in the generations.

Q1. Fluency of noisy generation step

You raise an important point: when combining the log probabilities of the fine-tuned model $p_\theta(y_t \mid y_{<t}, c)$ and the pre-trained model $p_{LM}(y_t \mid y_{<t})$, it is possible for a token with relatively low likelihood in both distributions to rank highly after the weighted combination. This could indeed lead to selecting less suitable tokens.

This concern directly motivated our proposed noisy generation method. As described in Section 3.2 and Algorithm 1, we actually do not use a weighted mixture of distributions. Instead, at each step, we sample exclusively from either the fine-tuned or the base model ($\alpha_t$ is either 0 or 1). This ensures tokens are drawn from fluent distributions, avoiding the issue you describe.

Q2. Rate the quality of disprefered samples.

Thank you for your suggestion. One way to measure this effect is to explicitely generate samples with an increasing degree of noise. We have included both quantitative and qualitative analyses of the noisy samples in the appendix. Specifically, we plotted PARENT (for data-to-text generation) and AlignScore (for summarization) for the negative samples as a function of the noise parameter $\alpha$. As anticipated, both scores decrease as $\alpha$ increases. Qualitative observations indicate that this decline results from the introduction of extrinsic and intrinsic errors relative to the input context. For further details, please refer to Appendix C.

W1. Lack of human analysis and quality of dispreferred samples

We followed your suggestion and we conducted a human assessment of the negative samples. We summarize below the main findings, please refer to Appendix C for more details and to the examples in Table 18. These qualitative evaluations indicate that SCOPE dispreferred samples contains extrinsic and intrinsic errors, rather than exhibiting a shift in the language distribution.

However, rating the quality of these samples systematically is not straightforward, and while this observation supports the importance of sample quality, our current findings remain qualitative. A more thorough study examining various quality dimensions of these samples could provide deeper insights into their role and impact, but we leave this as an avenue for future work.

W2. Comparison to instruction-tuned and RLHF baselines

Our method is indeed inspired by RLHF with human feedback but operates in a fully unsupervised setup. While our initial focus was on specialized models, we followed your suggestion and trained instruction-tuned models on the Alpaca dataset. One model was standardly tuned, while the other was trained using the SCOPE method, with noisy samples generated from the Alpaca dataset.

We evaluated both models on our initial tasks (data-to-text and summarization), where SCOPE consistently demonstrated improvements in faithfulness (albeit with smaller gains compared to its performance on specialist models) (Appendix A.7, Table 16) and table below.

Additionally, we verified that SCOPE does not impair reasoning capabilities, as evidenced by its performance on tasks from the OpenLLM benchmark (Appendix A.7, Table 17) and table below. We believe these results further underscore the robustness and versatility of the method.

As a comparison to a RLHF baseline, we also report the performance of Llama-2-7b chat models on our tasks. While it performs better on summarization tasks (likely due to the prevalence and alignment of summarization with instruction-tuned models) it falls short compared to specialist models on data-to-text generation. This highlights the limitations of general-purpose models on highly specific tasks and underscores the continued need for task-specific models in certain use cases.

Q1. Impact of the training set size

Across all datasets whose sizes span from 10K (for FeTaQA) to 120K samples (for ToTTo), we found that SCOPE consistently improves over traditional fine-tuning (see Tables 2 and 3) for Llama2-(7b and 13b) and Mistral-7b.

We then did not observe a direct correlation between training set size and the effectiveness of SCOPE. It is true that the largest datasets, ToTTo and XSum, show the highest gains in terms of automatic metrics. However, for the rebuttal, we also performed additional experiments on a selected subset of the PubMed dataset (following the experimental setup of [1]) which also demonstrates substantial improvement despite the dataset being relatively small (around 10K examples) (see Table 3 in the paper). Based on these observations, we believe SCOPE's effectiveness is more closely tied to the intrinsic difficulty of the task than to the size of the training set: XSum involves extreme summarization, ToTTo requires advanced table understanding, and PubMed focuses on summarizing technical medical articles.

[1] Tang, L. et al. Evaluating large language models on medical evidence summarization. npj Digit. Med. 6, 158 (2023). https://doi.org/10.1038/s41746-023-00896-7

Thank you for responding, based on your response, I have changed my score of contribution to 3: good.

Thank you for your feedback and relevant suggestions. Please find below our answers to your concerns and questions:

W1. Ablation on $\beta$

We initially used a default value of 0.1 for $\beta$, a common choice in DPO experiments. We acknowledge the lack of ablation analysis for this parameter. To address this, following your suggestion, we conducted an ablation study (see the table below). Overall, we found that $\beta=0.1$ performs consistently well for both tasks. We have included this ablation in the paper.

In the original DPO derivation, $\beta$ controls the strength of KL-regularization relative to the reference model, which in our case is the SFT model trained on half the dataset. As shown in the table, setting $\beta$ too low results in slightly worse performance, as the model may diverge excessively from the reference. Conversely, as $\beta$ increases, the gains diminish because the model is overly constrained and cannot sufficiently improve upon the reference SFT model.

Q1-W2. SCOPE on larger models

Indeed, our initial experiments focused only on 7B models, and we agree that evaluating SCOPE on larger models is indeed an insightful suggestion. In response, and in alignment also with feedback from reviewer aJYu, we conducted additional experiments with Llama2-13B on our tasks within the limited time available, and we updated the paper accordingly, see Tables 2 and 3. However, we were unable to extend our evaluation to 70B models due to computational constraints.

For the 13B models, we observed very similar trends to those seen with 7B models, indicating that SCOPE demonstrates consistent effectiveness across different model scales.

We provide below an excerpt of the results. More complete results are provided in Tables 2 and 3 in the paper.

Q2. Value of $\alpha$

We acknowledge that the choice of $\alpha$ is empirically driven. However, across all datasets and models, we consistently found that $\alpha$ values in the range [0.4, 0.6] yield effective results for SCOPE, with 0.5 often being the optimal choice (see Appendix A.2, Table 10). This consistency across tasks and models gives us confidence in recommending this range as a reliable starting point for most setups.

I thank the authors for explaining the parameter details.

My original score still reflect my judgement about this paper. Nevertheless, I have improved my assessment regarding soundness and contribution.

Thank you for sharing your feedback on our work; we truly appreciate it. Below, you'll find our responses to your concerns and questions:

W1. Novelty of the method

• We acknowledge that generating artificially bad samples is not a novel idea, and we appreciate the additional references you pointed out, which we will address in the related work. A key distinction between our method and some of the cited references is that many of them focus on improving post-fine-tuning generations by bootstrapping over model-generated sample pairs (e.g., BRIO, SLIC, SLIC-NLI). In contrast, our work introduces a novel fine-tuning approach that leverages training samples differently from standard fine-tuning, making SCOPE potentially complementary to these methods.
• Another point we would like to highlight is the simplicity of our method. Implementing the noisy decoding process requires only a few lines of code, whereas existing training approaches often rely on significantly more complex setups involving additional models or tools. Our approach explicitly leverages the base pre-trained language model directly, avoiding external dependencies. This design aligns with the intuition that it introduces hallucination patterns more consistent with those the fine-tuned model, derived from the same pre-trained base, would naturally generate.

W2-Q3. Detailed analysis of the samples

We have revised the paper to include both quantitative and qualitative analyses of the noisy samples in the Appendix. Specifically, we plotted PARENT (for data-to-text generation) and AlignScore (for summarization) for the negative samples as a function of the noise parameter $\alpha$. As expected, both scores decline as $\alpha$ increases. Qualitative (human) observations indicate that this decline is caused by the introduction of extrinsic and intrinsic errors relative to the input context. Further details can be found in the updated Appendix C.

W3. Faithfulness pointwise evaluation

Thank you for the suggestion. We initially considered pairwise evaluation, considering that it was more suitable for human assessments. Following your suggestion, we also conducted a pointwise faithfulness evaluation of both SCOPE and SFT on XSum using GPT-4 (instead of human evaluation in the limited amount of time), utilizing the following prompt:

Your task is to determine whether the provided summary accurately reflects the information in a given article, ensuring that every detail in the summary can be directly inferred from the article without adding any external information. The summary can have one or more of the following errors:

• Extrinsic Information: the summary contains new information not present in the source material.
• Intrinsic Error: the summary contradicts the source material.

Rate the summary on a scale of 1 to 5 based on faithfulness:

(5) Completely faithful: the summary is completely faithful to the article.

(4) Insignificant faithfulness errors: the summary is mostly faithful, with slight inconsistencies not affecting main points.

(3) Partially faithful: overall faithful, with a few inconsistencies with the article.

(2) Severe faithfulness errors: nearly half of the summary is faithful, with severe deviation from main points.

(1) Completely unfaithful: the entire summary is unfaithful to the article.

First, identify the list of errors that the summary makes, ensuring each error is clearly stated and specific.

Conclude the response with an evaluation using the following format: "Therefore, the score is: [insert score]."

For data-to-text generation, we ask two human annotators to evaluate the faithfulness of 50 outputs from SCOPE and SFT using the same protocol as XSum described above.

Please find below the average results (/5) of our assessments:

On XSum, both models exhibit faithfulness issues, but SCOPE shows a significant improvement over the SFT baseline. Overall, the pointwise evaluation confirms the trends found with pairwise assessment.

W4. Comparison with alternative noisy samples generation methods.

In response to similar feedback from other reviewers, we conducted an additional comparison using an alternative method for generating noisy samples, which involved randomly swapping entities from the reference samples. However, this approach did not yield gains as significant as those achieved with SCOPE. We believe this experiment reinforces our initial intuition about the importance of leveraging hallucination patterns that closely align with the behaviors of the fine-tuned model.

Dear reviewer,

Thank you once again for your thoughtful and detailed feedback on our paper. As the rebuttal period concludes, we wanted to let you know that we have carefully reviewed and addressed each of your comments in our revisions. We hope the changes we made effectively address your concerns and would be grateful if you could take them into consideration during your evaluation.

Thank you again for your time and effort.