OLAPH: Improving Factuality in Biomedical Long-form Question Answering

We would like to kindly appreciate all the constructive and beneficial feedback and questions provided by the reviewers. We will provide comments addressing the weaknesses and questions raised by the reviewers.

The method itself (OLAPH) is not really "novel" as the authors claim - the general frameworks has been used in many post-training of LLMs such as Llama and Claude models. I suggest softer contribution claims here.

The expressions have been toned down to some extent and reflected in the paper.

For the analysis of or RQ 3, what's exactly experimental setup? I am not getting why you will need to supply domain knowledge here, and since you train multiple models in RQ 1 and 2, which models did you end up using for this?

As shown in lines 483-485 and Figure 5, the experiment measures the FactScore of Mistral, Self-BioRAG, and BioMistral trained with our OLAPH framework and also we set the GPT-4 and human expert scores from the K-QA golden dataset as upper bounds. The additional construction of domain knowledge (in here biomedical knowledge) to evaluate FactScore is due to the method's approach of dividing the model's response into atomic facts and comparing them with knowledge sources to measure entailment. We considered expanding the knowledge sources beyond the Wikipedia dump to include rich and diverse medical knowledge from resources such as PubMed, PMC full-texts, Clinical Practice Guidelines (CPG), and medical textbooks.

As seen in Figure 5, when comparing the two lines representing the upper bound, it is evident that the biomedical knowledge sources slightly outperform the Wikipedia dump in FactScore. This indicates that even when incorporating domain-specific knowledge sources for additional comparison, our OLAPH framework still demonstrates meaningful improvements in factuality.

In Figure 5, if using SFT leads to initial drop of performances, have you try to remove it? And what would the impact be?

Thank you for asking the ablation studies of our OLAPH framework. We've also explore the removal of SFT which sometimes results in an initial drop in performance. Since it shows significantly better performance compared to only applying alignment tuning, it is challenging to eliminate the SFT component. Additionally, achieving quantitatively high performance without SFT proved to be extremely difficult. Despite extensive hyper-parameter searches, we struggle to find an experimental setup that could reach peak performance, leading us to conclude that scalability across different experimental setups is hard to achieve. Furthermore, after repeated alignment-tuning, we observe an increase in qualitatively odd responses, such as repetitive phrasing and excessive response length, as well as a notable reduction in response diversity. We add this context into our appendix part.

Line 235 - 236, this may simply suggest that answers in K-QA is terrible, not necessarily that GPT-4 answers are good. Do you have more analysis here?

According to the K-QA paper, since the K-QA answers were manually annotated by domain experts, it was difficult to question the quality of these answers. Therefore, we proceeded with the assumption that we only needed to verify with a medical expert whether the GPT-4 answers were better or not.

Line 248 - 252, I am not 100% convinced by this claim. Could it be that annotators are biased to believe whatever the model generates?

As the reviewer mentioned, I agree that annotators could introduce bias when evaluating the model-generated statements. However, as authors with relatively limited domain knowledge in comparison to the six panel medical experts who validated the process, we did not perceive it as necessary to further re-validate the results on our own.

Line 282 + 283, do you use statements as SFT targets too? If so, what's the data format?

For SFT, only the response with the highest automatic evaluation score among the generated sample answers for each model was used for training. Therefore, MH and NH statements were not utilized in the training process.

We truly appreciate your thoughtful consideration and the time you've taken to engage with our work.

First, thank you for recognizing the value of our paper. We deeply appreciate your understanding of the unique strengths we aimed to highlight, and we are grateful for your thoughtful feedback. Below, we provide answers to the points you raised.

As we know, medical knowledge advances over time. So, I am curious to understand how you plan to deal with the challenges of maintaining factual relevance if your framework/dataset is deployed over a longer period?

This is an excellent question. Medical knowledge is continuously evolving and updating, and it has become a significant area of growth. As part of our future research, we are exploring the creation of chronological knowledge specifically for the medical domain to assess the extent of knowledge elicitation in existing open-source models. To maintain factual relevance, we are working on establishing a dataset construction pipeline that can continuously update changing knowledge. This would allow us to build a robust answer update pipeline, which could ultimately evolve into a comprehensive framework. While this idea is still in its early, somewhat naive stage with initial experiments underway, we believe it could become a crucial source of information in the medical field as it develops further.

it does not cover the framework's behavior for a domain-specific case like disease diagnosis or recommendation of treatment.

The authors are currently conducting further research on a domain-specific case in history taking, aiming to develop a conversational agent that provides personalized medical diagnoses based on patient characteristics. The importance of factuality in this process is recognized, and exploring how the OLAPH framework can be applied to disease diagnosis would make for an excellent research direction.

Were there specific medical domains or types of questions where synthetic responses diverged from expert standards? I think it would be helpful if you could discuss any plans to validate the quality of synthetic data using a diverse set of generative models to assess robustness.

I fully agree with the point raised. To address this, we plan to conduct a qualitative analysis using randomly sampled data instances from the LLaMA-3.1-70B model to compare it with the decomposed must-have and nice-to-have statements provided by medical experts. We will examine the differences in detail and update the paper accordingly. This will be done as soon as possible, but since it requires significant time and resources, we aim to provide a thorough analysis rather than a simple observation. We hope for your understanding in this regard.

Once again, thank you sincerely for reviewing our paper and providing such valuable feedback. We truly appreciate your thoughtful consideration and the time you've taken to engage with our work.

First and foremost, we would like to appreciate all the constructive and beneficial feedback and questions provided by the reviewers. We will provide comments addressing the weaknesses and questions raised by the reviewers.

The novelty of the proposed OLAPH framework is limited, as it mostly follows the standard preference optimization process (SFT and DPO)

We understand that the OLAPH approach may seem somewhat limited in novelty, given the presence of existing studies that iteratively apply SFT (Supervised Fine-Tuning) and DPO (Direct Preference Optimization) with rejection sampling or other sampling methods. However, we hope the novelty of our work is recognized in terms of how we design a meaningful synthetic dataset—whether for supervised learning or alignment tuning—and the direction of exploiting automatic evaluation, training details, and effectiveness of the expected outcomes when employing iterative training with under 7B size of models in the medical domain.

The paper should address the convergence properties of OLAPH: Is convergence guaranteed? How do the number and types of evaluation metrics influence convergence rates?

By comparing Figure 4 and 6, it becomes evident that the guarantee of convergence varies depending on the evaluation metric. Specifically, this variation is particularly noticeable across different metrics. For instance, metrics like word composition and semantic similarity tend to show relatively minor differences between generated responses across samples. These differences are somewhat normalized, maintaining a consistent range or converging over iterations.

However, for metrics evaluating factuality, such as hallucination and comprehensiveness, the variation between responses is significantly larger across samples. While these variations generally decrease as iterations progress, the extent of this reduction varies across models (particularly for LLaMA2 and Meditron). Thus, it cannot be confidently concluded that the results always converge. We plan to explore future research directions aimed at confidently ensuring convergence under these conditions.

Can the evaluation criteria for the qualification of generated data in section 3.2 be used as preference criteria for preference optimization process in section 4.2?

This is a very intuitive question! The nine evaluation criteria utilized in Section 3.2 are assessed based on the definitions provided in Table 4, which serve as the foundation for evaluating QA pairs. To effectively apply these evaluations in Section 4.2, two prerequisites seem necessary:

1. A method for numerically representing each evaluation criterion, such as using rubrics for scoring.
2. A model capable of learning this process in a manner that minimizes error propagation.

If the model’s response to a given query incorporates scores for each criterion—beyond our proposed cost-effective and multifaceted automatic evaluation—and normalizes them within the same range, it could significantly enhance the sophistication of the automatic evaluation process.

Why does the paper follow cross-validation approach for experiments instead of the traditional train/test split approach?

As mentioned in the evaluation of the MedLFQA benchmark paragraph in Section 5, data is extremely scarce due to the unique nature of the medical domain and there is no dedicated training dataset for the entirety of MedLFQA. Therefore, we adopted a cross-validation approach, where one subset of the data was treated as the test dataset while the remaining subsets were used as the training dataset for learning and evaluation.

We truly appreciate your thoughtful consideration and the time you've taken to engage with our work.

First and foremost, we would like to appreciate all the constructive and beneficial feedback and questions provided by the reviewers. We will provide comments addressing the weaknesses and questions raised by the reviewers.

it is not persuasive that the current setting can effectively serve as the hallucination metric. This setting only evaluates a subset of hallucination; the LLM generated incorrect facts that outside of the MH and NH will not be taken into account in the metrics

The authors truly agree that our suggested OLAPH framework evaluates the subset of hallucinations. By categorizing hallucinations into intrinsic and extrinsic types, our study focuses more on handling extrinsic hallucinations. Extrinsic hallucinations occur when the model generates information that is factually incorrect or contradicts external, verifiable facts. However, we believe that MH and NH can effectively measure against external and verifiable facts. This is because our approach trains the model to generate preferred answers which are decomposed into MH and NH statements and has shown improvements in experiments involving such facts.

Additionally, when applying FactScore to retrieve relevant documents from resources like the Wikipedia dump and biomedical dump, and comparing them at the atomic fact level, we observed that the application of our OLAPH framework resulted in an increase in the FactScore. This suggests that measuring factuality by considering MH and NH positively influences external and verifiable facts.

The calculation of contradicts and entails are based on a fine-tuned BioBERT model, which brings uncertainty of the evaluation, especially on the medical domain. The metrics are highly affected by the performance of the fine-tuned BioBERT model, so a sensitivity analysis (with different models other than fine-tuned BioBERT) needs to be tested.

The reviewer raises a very insightful point. I agree with this perspective, as using a single classification model to evaluate entailment can be incomplete (and potentially not robust), which may also introduce additional error propagation at the evaluation stage. However, as shown in Table 11, we proved that the fine-tuned BioBERT is sufficiently capable of measuring entailment as an NLI model. Still, we agree that reducing the uncertainty that may arise during the evaluation stage is a highly desirable direction. If the reviewer could provide an alternative NLI models that can measure entailment, we would be happy to conduct additional experiments and evaluate it using that model.

It is no surprise that the proposed OLAPH framework can improve the three metrics (Word composition, Semantic Similarity, and Factuality), as these three metrics are directly optimized during the training (similar to the comparison that fine-tuned model is better than non fine-tuned).

That's an excellent point. The three evaluation categories reviewer mentioned should indeed be directly optimized to improve performance. However, since our experimental setup ensures a clear separation between the training and test sets—where the model is trained solely on the training set without exposure to the test set—it seems that this does not pose a significant issue.

In table 5, when set the alpha_3 as 1.0, the performance seems better that other smaller alpha_3 scores. Why not to enlarge the alpha_3 over 1 (2,3,5 or more) to test the model performance?

In lines 831-857, alpha was utilized as a means to normalize the range of the multifaceted automatic evaluation from the perspective of convex optimization. Therefore, we did not consider experimenting with allocating a higher value to alpha_3. However, based on the results in Table 6, it seems possible to predict the outcomes of experiments where the values of alpha_1 and alpha_2 are set to 0. In Table 6, experiments were conducted with alpha_3 set to a value between 0 and 1. Although alpha_1 and alpha_2 did not show as significant performance changes as alpha_3, we observed that learning to decompose MH and NH for entailment led to improvements in word composition and semantic similarity. Therefore, it can be anticipated that if we set alpha_1 and alpha_2 to 0 and gradually increase the value of alpha_3, the performance will align and show a slight improvement.

The primary goal of the OLAPH framework is to integrate an automatic multifaceted evaluation into the learning process. The core idea is that this allows the model to generate synthetic data using self-generated responses for training. While the current metrics, which are highly correlated with each other, show an aligned performance trend as discussed above, there is a high likelihood that new evaluation metrics—such as fluency and sympathy—will yield different patterns when automatically incorporated. Keeping this in mind will provide a clearer perspective on the results.

We truly appreciate your thoughtful consideration and the time you've taken to engage with our work.