Thank you for your constructive comments. We will address your concerns in the following parts.

Limited data size

Thank you for your valuable feedback. The current version of MedicineQA focuses on commonly used medicines to address a broad range of prevalent needs. Although current MedicineQA contains only 300 samples, it covers most of the medicines commonly used in real-world scenarios, spanning 10 aspects of medicine application. We appreciate your suggestion to increase the dataset size by expanding the set of topics for various specific conditions, which is crucial for improving the scope of our benchmark. We sincerely plan to include more specialized medications in the future. However, the construction process is time-consuming.

The development of MedicineQA involved significant input from medical experts, with each step being guided by real-world data and supervised by physicians to ensure the benchmark’s quality. Additionally, expanding the dataset requires incorporating more medicines from the original, large-scale Medicine Database into our knowledge base. This process involves rigorous review and verification. We are currently in the process of reviewing additional medicines and expanding the dataset. We regret any delays in providing a larger benchmark and sincerely appreciate your understanding and patience.

Dear Reviewer 5Hq1,

We would like to sincerely thank you for your time and effort in engaging during the discussion phase. We greatly appreciate the constructive feedback and thoughtful comments you provided, which have been invaluable in improving the quality of our work.

Best regards,

Authors

Thank you for your constructive comments. We will address your concerns in the following parts.

W1: Limited Comparison

We apologize for any confusion caused by the unclear motivation. Our primary focus is to address the unique challenges of medication consultation scenarios. Medication consultation is a knowledge-intensive task that requires an authoritative medicine database to provide relevant evidence for generating correct and comprehensive responses without hallucinations. In real-world scenarios, dialogues are often ambiguous and verbose. For example, users may use layman's terms rather than medical terminology and provide more information than what is medically relevant. As demonstrated in Table 3 of our submitted paper, directly utilizing the user's query or the entire dialogue history yields poor results. Thus, effectively summarizing user inputs and eliminating extraneous information to retrieve appropriate evidence from the medicine database is crucial. To address this, we propose a novel approach for generating a simple yet robust search query, which we refer to as the "Distill" step in our Distill-Retrieve-Read framework. Specifically, we prompt the LLM to generate search keywords for several predefined search engines. When a user query comes, the LLM mimics how a human would use search engines, generating appropriate search keywords based on the current dialogue context. This approach differs from the current mainstream methods, which typically prompt LLMs to summarize the previous context into search keywords. Recent studies have demonstrated the impressive capabilities of LLMs in summarizing contexts into keywords. Following the previous premise, we conduct experiments by comparing our proposed "Distill" method with methods that prompt LLMs to summarize. As shown in Table 2 in our paper, our "Distill" method demonstrates superior performance over directly prompting LLMs to summarize.

Thank you again for your valuable feedback. Your recommended approaches have truly inspired us. Previous works in RAG have typically focused on improving retrieval accuracy through specially designed frameworks or extracting useful information from dummy documents. We are regretful that few have addressed the critical challenge of generating more accurate search keywords. To the best of our knowledge, MedicineQA, with our medicine database, is the first benchmark in the medical domain that evaluates the robustness of generated search keywords. Since the medicine database is constructed in Chinese, it would be time-consuming to translate this huge database into English while ensuring its correctness. It is unfair to compare models in English with a raw medicine database. Thus, we also compare our RagPULSE with GPT-4, which is one of the most powerful LLMs. As shown in the following table. We can observe that RagPULSE achieves comparable results in generating search keywords with GPT-4 and performs better in precise retrieval (i.e., 65.67 vs. 62.33 in document retrieval and 28.33 vs. 26.67 in attribute retrieval).

W2: Details of Experiments

We apologize for any confusion caused by the omission of certain details. We hope the following clarifications will provide a clearer understanding of our methodology.

Q1: Illustration of the workflow via the example in Figure 3: Each sample in MedicineQA consists of three parts: (Dialogue History $H$, Current Question $Q_{T+1}$, Relevant Evidence {$K_c$, $K_f$}). During the construction process, physicians summarized selected records into a question ($Q_{T+1}$) about a specific medicine. The Dialogue History ($H$) is generated by GPT-4 and further revised by physicians. The Relevant Evidence ($K_c$, $K_f$) is annotated by experts, with more details available in the Appendix. We apologize again for any confusion caused by the ambiguous figure.

Specifically, Dialogue History includes the previous user input, such as "I am overweight..." and the previous model response, "Note that the following information...". Current Question is, for example, "How many pills...". The Prompt for Distillation is the System Prompt for RagPULSE, designed to instruct the LLM that multiple search engines can be used to retrieve information. To search for relevant information from the Medicine Database, RagPULSE uses both Dialogue History and Current Question as inputs, summarizing them into a search query (e.g., "Orlistat Capsules: Dosage, Usage, and Adverse Reactions"). In practice, the output of RagPULSE is search engine(Orlistat Capsules: Dosage, Usage, and Adverse Reactions), where the model mimics the human process of using search engines.

Q2: Illustration of settings for baseline models: For the baseline models, we first prompt them to summarize the Dialogue History $H$ and Current Question $Q_{T+1}$ into a search query using the instruction (Based on the above conversation about medical inquiries and medication queries, please summarize the search keywords for the user's final question using the dialogue record. Retrieve relevant medication information and return it in JSON format as follows:{"query'': $\dots$}) . The generated search query is then used to query the database to retrieve evidence. Based on the retrieved evidence, we measure the HR@K for the baseline models according to the Relevant Evidence.

Q3: Elaboration of the results: We are very grateful for your careful reading. As shown in Table 2, some baseline models exhibit poor performance in generating accurate search queries for retrieving evidence from the medicine database. To evaluate the quality of the generated responses based on the relevant evidence, we immediately adopt the corresponding ground-truth medicine information (i.e., the annotations from MedicineQA, as you mentioned) as the context to guide the response generation. We apologize again for any confusion in line 389. In this section, we aimed to investigate whether fine-tuning on the synthetic dataset could improve the model's ability to summarize context. It is important to note that both PULSE and RagPULSE are domain-specific models. Apart from the synthetic dataset, all other fine-tuning data are the same as those used for PULSE. RagPULSE† represents the version where we use the same instruction for baseline models to prompt RagPULSE to generate the search query rather than using our proposed "Distill" step. The comparisons between PULSE and RagPULSE† in Table 3 indicate that finetuning on the synthetic dataset contributes to enhancing the model's capability to summarize context effectively.

In Table 2, RagPULSE leverages the retrieved evidence to generate the answer. While the retrieval process in RagPULSE is not optimal, it still achieves notable performance in combining the evidence to generate the final response. This can be attributed to PULSE’s specialized proficiency in the medical domain. To address your concerns, we present additional results showing PULSE without evidence and PULSE with the ground-truth medicine annotations from MedicineQA in the following table (More results can be seen in Table 1 of the revised paper). PULSE attains higher performance than other publicly available models. However, without utilizing retrieved evidence, the performance is not optimal. PULSE*, referring to ground truth evidence of medicine, distinguishes itself from other models in the domain of medication consultation responses.

W3: Details of the Medicine Database

We purchased a comprehensive Chinese medicine database from an authoritative institution as a foundation for our RAG model. It includes detailed descriptions of hundreds of thousands of medications, encompassing both traditional Chinese and Western medicines. Each medication document has been subjected to a rigorous triple review and verification process to ensure compliance with established medical standards. We are currently in the process of translating and validating subsets of the database. We regret any delays in making this resource available as an open-source tool and appreciate your understanding.

Dear Reviewer gvnx,

We sincerely appreciate your time and effort in reviewing our work. We thank you for your deep comprehension and suggestions, which have strengthened our manuscript. We would appreciate the opportunity to continue dialogue with you to fully meet your expectations. If you still find some aspects of our work unclear or have other additional concerns, we are always prepared to discuss or provide more information. Our commitment is to address any concerns you may have and enhance the quality of our paper.

Best regards, Authors

Thank you once again for your time and valuable feedback. Specifically, with the exception of Baichuan2, Qwen2, and the GPT series, the other models are domain-specific for the medical field. The suboptimal performance of these models, even with ground-truth retrieved evidence, further highlights the challenges present in medication consultation scenarios.

In fact, domain-specific fine-tuning can indeed enhance an LLM's ability to understand medical contexts and generate responses. However, it contributes little to the summarization and keyword generation necessary for effective evidence retrieval. As shown in the table below, InternLM, the base model of PULSE, actually outperforms PULSE in generating search queries for retrieval. When further augmented with the Distill approach, RagPULSE shows a significant improvement. The results validate the effectiveness of our proposed framework. The primary reason for selecting PULSE is that it is open-source and available for fine-tuning with the existing dataset.

We hope our response could address your concerns.

Dear authors,

thank you for your answers. I still think the paper is confusing to read and would benefit from a rewrite - I also don't think your answers did a good job addressing my questions. Therefore, I would like to maintain my rating. However, if the other two reviewers would like to champion this paper, I am not opposed to acceptance either.

On an off note, and this is supposed to be a well-meant advice and has no influence on my assessment of your submission: When addressing reviewer questions, it's best to be precise and to the point, to help the reviewer find the answers they're looking for. A wall of text is not very helpful in this case. With ChatGPT, everyone can write a long-winded answer. The tricky part is to present the necessary information in a concise manner. Sorry to be blunt, but when I'm looking for answers to my questions, I really don't care whether my "recommended approaches have truly inspired" you.

Thanks for your constructive comments and suggestions for our manuscript. Below is our detailed response to address your concerns.

Limited data size

Thank you for your valuable feedback. We hope the following clarifications address your concerns regarding the limited size of MedicineQA:

1. MedicineQA focuses on commonly used medicines to meet a wide range of prevalent needs. We aim for models performing well on MedicineQA to provide basic healthcare support in regions with relatively scarce medical resources. We began by collecting records from five major online consultation websites, which are rich sources of real patient-doctor dialogues. From such a large volume of data, we extracted conversations involving predefined commonly mentioned medicines. These dialogues were then utilized for evidence retrieval and dialogue revision, as detailed in the Appendix. Although MedicineQA contains only 300 samples, it covers most medicines commonly used in real-world scenarios across ten aspects of medicine application.

2. The construction of MedicineQA involved significant input from medical experts. Each step of the process was guided by real-world data and supervised by physicians to ensure the benchmark's quality. Our dataset construction was carried out by a panel of five board-certified physicians, with a senior advisor overseeing the entire process. Each sample in the benchmark, including the questions, search results, and answers, was carefully reviewed and revised by experts to ensure that the progression of questions was logical and the dialogue content accurately reflected real-world scenarios. Thus, we made a deliberate trade-off between scale and quality.

3. MedicineQA was specifically developed to address gaps in existing medical QA datasets by solving medication consultation scenarios with retrieval-augmented generation. Specifically, it relies on an authoritative database to provide relevant medicinal knowledge. The knowledge source for our RAG model is a subset of a private Chinese medicine database from an authoritative institution. We plan to expand the benchmark in the future to include more specialized medications. This expansion will require integrating additional medicines into our knowledge source. We are currently reviewing more medicines. The rigorous review and verification process is time-consuming, and we regret any delays in providing a larger benchmark. We appreciate your understanding and patience.

Details of Construction and Benchmark

We apologize for any confusion caused by the omission of certain details regarding the construction process and the resulting benchmark. Specifically, after the physicians summarized the selected records into a question about a specific medicine, we prompted GPT-4 with the context of the relevant record to generate multi-round dialogues. This approach ensured that the generated dialogue content accurately reflected real-world scenarios. To prevent GPT-4 from hallucinating inappropriate content, physicians manually revised the dialogue to ensure it was realistic within the medical context and consistent with medical knowledge and logic. As a result, MedicineQA consists of 300 samples covering over 150 medicines, spanning 10 aspects (from Recommendation to Storage). In total, MedicineQA contains 989 rounds of dialogue.

Dear Reviewer rJMn,

We would like to sincerely thank you for your time and effort in engaging during the discussion phase. We greatly appreciate the constructive feedback and thoughtful comments you provided, which have been invaluable in improving the quality of our work.

Best regards,

Authors