Magical: Medical Lay Language Generation via Semantic Invariance and Layperson-tailored Adaptation

We sincerely appreciate your recognition of the comprehensiveness of our ablation studies, the clarity of our figures and visualizations, and the importance of the addressed task. We have carefully considered the shortcomings you pointed out in our manuscript and provide detailed responses below.

Response to Generalization

We appreciate your suggestion regarding the issue of generalizability. We believe that evaluating the generalization capability of Magical is a crucial step toward improving the quality of our manuscript. Specifically, we investigate the generalizability of Magical in the following two important scenarios:

1. Generalization to unseen datasets within the MLLG task domain

To explore this critical issue, we introduce a new MLLG dataset named EASI [1]. We selected LLaMA3.1-8B-Instruct as backbone LLM, the experimental settings and result are presented in the table below.

Experimental results show that both prompt-based and LoRA-based methods (Rows 1–4) exhibit poor generalization, performing significantly worse than end-to-end fine-tuning with a newly initialized LoRA on the target dataset (Row 5). The prompt-based approach suffers from limited performance due to its inability to comprehensively capture the target lay-style. LoRA, on the other hand, is optimized toward a specific lay-style on the training set, and thus demonstrates a marked performance drop when there is a distribution shift between the training and test sets.

We evaluate Magical by reusing matrix A trained on Cochrane, eLife, and PLOS Genetics, and introducing a newly initialized matrix B, which is fine-tuned on EASI while keeping matrix A frozen (see Row 6). The results show that Magical outperforms even the fully fine-tuned LoRA on EASI (Row 5) across all evaluation metrics.

Our key insight is as follows: each heterogeneous MLLG dataset corresponds to a unique lay-style. Current approaches—including prompt-based methods, LoRA, and Magical—are not yet capable of robust generalization under truly unseen lay-style conditions.

Nevertheless, we argue that Magical’s matrix A constitutes a highly generalizable component. This is because Magical explicitly decouples abstractive summarization from lay-style generation. Matrix A, which is responsible for summarization, is independent of the lay-style and thus enjoys broad generalizability.

This claim is supported by the above empirical results, where Magical achieves superior performance and lower parameter overhead compared to LoRA under equivalent fine-tuning settings.

2. Generalization to new domains

Following your suggestion, we further introduce a legal summarization task using a widely adopted dataset [2] in the community (IN-Abs, IN-Ext, and UK-Abs). We employ LLaMA3.1-8B-Instruct as the backbone LLM and follow exactly the same experimental settings as used for the MLLG task in the manuscript. The experimental results are presented in the table below.

The results show that Magical outperforms all baseline methods across most metrics, with the only exception being a slightly lower R-1 score than rsLoRA on the UK-Abs dataset. This demonstrates that Magical is also effective in the legal summarization domain.

Response to Human Evaluation

We fully understand your concern regarding the need for layperson evaluation to assess whether the generated texts meet the expectations of target end-users. However, we would like to clarify that evaluating the quality of generated text also crucially involves verifying factual accuracy and content correctness. For lay users, detecting subtle or domain-specific inaccuracies in the generated lay text can be extremely challenging. Therefore, we chose to conduct expert evaluations instead of layperson assessments.

We sincerely apologize for the space limitations. We warmly invite you to refer to our discussion with Reviewer TVd2 under the section "Response to External Validations for Automated Metrics," where we provide two evaluation methods for assessing real-world quality, including human evaluation.

Response to the Analysis of the Number of B Matrices and the Choice of Top-K Semantic Layers

Thank you for highlighting this point. We agree that analyzing the sensitivity to the choice of top-K layers is important, as it supports our core claim that different layers of LLMs encode distinct functional representations (see Line 174 of the manuscript).

Following your suggestion, we have added a sensitivity analysis of the top-K selection based on LLaMA3.1-8B-Instruct. The experimental results, shown in the table below, reveal that both overly small values of K (e.g., K = 8) and overly large values (e.g., K = 24 or 32) result in degraded performance. This is because selecting too many layers introduces semantic constraints to layers that are not semantically relevant, leading to interference with the LLM’s functional integrity. Conversely, too few layers may fail to impose constraints on all semantically relevant layers, resulting in semantic shift.

Regarding the number of B matrices, we assign one matrix B per heterogeneous dataset. This is not treated as a hyperparameter in our experiments but rather as a direct reflection of the number of heterogeneous sources involved.

Response to the Clarity of Figure.2

Thank you for your valuable suggestion. We are highly attentive to the readability and clarity of our manuscript. In Figure 2, we project the representations of expert text samples and the corresponding generated lay text samples onto the top-2 singular directions of the semantic subspace, which form the x- and y-axes of the KDE plot. In response, we have revised the caption of Figure 2 to improve clarity and added axis labels to the plot.

Response to Novelty

We sincerely appreciate your valuable feedback and understand your concerns regarding the novelty of the manuscript. We sincerely apologize for the space limitations. We warmly invite you to refer to our discussion with Reviewer gwM8 under the section "Response to Novelty," where we clarify our perspective on the novelty of Magical.

Response to Clinical Application and Limitation Discussion

Thank you for your valuable suggestion. We believe that expanding the discussion on how Magical can be applied to real-world clinical settings, as well as elaborating on its more limitations, is important. This would greatly enhance the clarity of our work’s practical applicability.

To this end, we offer some preliminary insights on how Magical might be translated into practical clinical use. We propose that its deployment follow four key periods:

1. Expert-Guided Validation: Initially, the system should be open to medical experts to ensure the accuracy and reliability of the generated content.

2. Lay-Style Generation: Leveraging Magical’s strengths, it can then produce diverse lay-style text, allowing users to freely select the most suitable ones based on their latent prefenence.

3. User-Centered Feedback and Learning: User preferences can be collected in this phase to construct a Recommendation Agent.

4. Personalized Lay Text Recommendation: Based on user feedback, the system can ultimately provide precise recommendations of preferred lay-style outputs, facilitating efficient and personalized communication.

In response to your concerns, we have added a new Discussion section in the revised manuscript, where we further elaborate on the plans for clinical application, as well as additional limitations of Magical, including those you mentioned such as scalability, generalizability, ethical risks.

Thank you again for your constructive suggestions.

References

[1] Med-EASi: Finely Annotated Dataset and Models for Controllable Simplification of Medical Texts, AAAI, 2023

[2] Legal Case Document Summarization: Extractive and Abstractive Methods and their Evaluation, AACL | IJCNLP，2022

---

We sincerely thank you once again for your thoughtful guidance. Your feedback has been immensely valuable to us, and incorporating your suggestions has significantly improved the quality of our manuscript. If you have any further questions or concerns, please feel free to contact us at any time. We are always available and look forward to further discussions with you.

Best regards,

All Authors

Dear Reviewer 1PZy,

We hope this message finds you well. We completely understand that your schedule may be demanding, and we sincerely appreciate the time and effort you have devoted to reviewing our manuscript.

In response to your valuable suggestions, we have made the following clarifications and enhancements to our manuscript:

1. We investigated two aspects of the generalization ability of our method, including its performance on unseen datasets and its transferability to new domains such as the legal summarization task.

2. We expanded our evaluation to include both Large Language Models as Evaluators (Full-scale Evaluation) and Human Expert Evaluation (Focused-scale Evaluation) to assess the actual perceived quality of the generated lay texts.

3. We further included an analysis of our method with respect to the Top-K parameter.

4. We improved the clarity of Figure 2.

5. We clarified the novelty of our proposed approach.

6. We have added further discussion on clinical applications and additional limitations of our work.

We are eager to understand whether our efforts have effectively addressed the limitation in the original manuscript. Given your expertise, we are eager to engage in further discussion with you. However, as the discussion phase draws to a close, we have yet to receive your response. If possible, we would be deeply grateful if you could spare a moment to share your thoughts, which would be of immense value to us.

Thank you again for your time, thoughtful feedback, and invaluable guidance.

Warmest regards,

All Authors of Submission #22904

Dear Reviewer 1PZy,

Thank you for your careful reading of our manuscript and rebuttal, and for your thoughtful and positive response. Your suggestions are invaluable to us, and we truly appreciate every opportunity to engage in this exchange, which is essential for improving the quality of our work.

Once again, we are sincerely grateful for your kind guidance.

Best wishes,

All Authors of Submission #22904

We sincerely thank you for acknowledging the importance of the task, the sufficiency of our preliminary investigation, and the comprehensiveness of our experiments. We deeply value the constructive criticisms you raised and respond to each concern in detail below.

Response to Novelty (Weakness#1 & Weakness#2)

We sincerely appreciate your valuable feedback and understand your concerns regarding the novelty of the manuscript. We would like to clarify that novelty in academic research should not be limited to proposing disruptive algorithms, but should also encompass the ability to inspire interest, provoke community discussion, and open up new directions for future work. In this regard, we argue that Magical achieves meaningful novelty through the following contributions:

1. Magical provides a comprehensive review of existing approaches and proposes a new perspective backed by empirical evidence: “Standard LoRA fails to meet the requirement for semantic fidelity and diverse lay-style generation in the MLLG task.”

2. Magical performs timely adaptations and responds to the unique challenges faced by LoRA in the MLLG setting by offering a tailored solution that directly addresses these shortcomings.

3. Most importantly, Magical serves as a catalyst for rethinking LoRA-based approaches in MLLG, encouraging the community to explore future directions for semantic preservation and adaptive lay-style generation.

While we acknowledge that Magical is inspired by HydraLoRA, we highlight that the two methods differ in their applicability. In fact, Magical identifies and remedies a limitation of HydraLoRA, which we elaborate on in the next section.

Response to Comparison Between HydraLoRA and Magical (Question#1), and on Not Using a Basic Learning-Based Selector(Question#2)

Thank you for your valuable suggestion. In response, we revisited the tasks addressed in HydraLoRA, such as Summarization, Closed QA, and Information Extraction. It is evident that these tasks are clearly distinguishable, and simple prompts are sufficient for an LLM to differentiate among them and select the appropriate matrix B. However, when it comes to heterogeneous MLLG tasks, it becomes challenging to accurately convey the target lay-style using simple prompts (as discussed in line 227 of our manuscript). This limitation hinders the effective automatic selection of matrix B.

To validate this point, we conducted a detailed experiment in Appendix E.1: Dilemma of Router-Controlled Selection, where we adopted HydraLoRA’s original design using a learnable router (i.e., a basic learning-based selector, as you suggested). The results demonstrate that the router fails to learn meaningful selections—our Figure 5 shows a confusion matrix where the learned routing behavior is indistinguishable from random selection.

Further evidence is provided in Table 3 (“Switch → Router” ablation), where replacing our manual switching with routing leads to a significant drop in performance.

These findings show some limitations of HydraLoRA: router-based selection collapses when the task cannot be explicitly described via simple prompts, which is often the case in MLLG. Although Magical uses a simpler, manually defined selection strategy (our optimal setting assumes that decisions are made based on a Recommendation Agent), it addresses this core failure effectively.

Response to Sensitivity of Recommendation Agent Performance (Question#2)

We share your concern about whether the overall performance of Magical is sensitive to the quality of the recommendation agent. In Appendix E.2: Sensitivity of Recommendation Performance, we present experiments that simulate reduced accuracy in recommendation. Notably, even when the agent's hit rate drops to 60%, Magical still outperforms standard LoRA.

Response to Alpha in Equation (2) and Generalization to Unseen Data (Question#3)

In Equation (2), the vector $\alpha$ is a binary indicator: the entry corresponding to the selected matrix B is set to 1, while all others are set to 0.

Same as you, we are also deeply concerned with Magical's generalization ability on unseen datasets. To explore this critical issue, we introduce a new MLLG dataset named EASI [1]. We selected LLaMA3.1-8B-Instruct as backbone LLM, the experimental settings and result are presented in the table below.

Experimental results show that both prompt-based and LoRA-based methods (Rows 1–4) exhibit poor generalization, performing significantly worse than end-to-end fine-tuning with a newly initialized LoRA on the target dataset (Row 5). The prompt-based approach suffers from limited performance due to its inability to comprehensively capture the target lay-style. LoRA, on the other hand, is optimized toward a specific lay-style on the training set, and thus demonstrates a marked performance drop when there is a distribution shift between the training and test sets.

We evaluate Magical by reusing matrix A trained on Cochrane, eLife, and PLOS Genetics, and introducing a newly initialized matrix B, which is fine-tuned on EASI while keeping matrix A frozen (see Row 6). The results show that Magical outperforms even the fully fine-tuned LoRA on EASI (Row 5) across all evaluation metrics.

Our key insight is as follows: each heterogeneous MLLG dataset corresponds to a unique lay-style. Current approaches—including prompt-based methods, LoRA, and Magical—are not yet capable of robust generalization under truly unseen lay-style conditions.

Nevertheless, we argue that Magical’s matrix A constitutes a highly generalizable component. This is because Magical explicitly decouples abstractive summarization from lay-style generation. Matrix A, which is responsible for summarization, is independent of the lay-style and thus enjoys broad generalizability.

This claim is supported by the above empirical results, where Magical achieves superior performance and lower parameter overhead compared to LoRA under equivalent fine-tuning settings.

Response to Writing Issues (Weakness#3)

We are grateful for your close reading and for pointing out the writing issues in our manuscript. We sincerely apologize for these oversights. In response, we have thoroughly reviewed and revised the manuscript to correct all writing and formatting issues. Thank you again for your careful attention.

Response to the Use of a Quantitative Semantic Similarity Metric (Weakness#4)

We appreciate your valuable suggestion regarding the evaluation of semantic similarity. In response, we have incorporated BERTScore as a complementary evaluation metric. We select LLaMA3.1-8B-Instruct as the backbone LLM and conduct our experiments accordingly. The new results are presented in the Table below, and they align well with the trends we observed using KDE, thereby further confirming the semantic fidelity of Magical.

Response to Generalization to Other Domains (Question#4)

We fully acknowledge your point that evaluating Magical on datasets from other domains is essential for assessing its generalizability.

Following your suggestion, we further introduce a legal summarization task using a widely adopted dataset [2] in the community (IN-Abs, IN-Ext, and UK-Abs). We employ LLaMA3.1-8B-Instruct as the backbone LLM and follow exactly the same experimental settings as used for the MLLG task in the manuscript. The experimental results are presented in the table below.

The results show that Magical outperforms all baseline methods across most metrics, with the only exception being a slightly lower R-1 score than rsLoRA on the UK-Abs dataset. This demonstrates that Magical is also effective in the legal summarization domain.

References

[1] Med-EASi: Finely Annotated Dataset and Models for Controllable Simplification of Medical Texts, AAAI, 2023

[2] Legal Case Document Summarization: Extractive and Abstractive Methods and their Evaluation, AACL | IJCNLP，2022

---

We sincerely thank you once again for your thoughtful guidance. Your feedback has been immensely valuable to us, and incorporating your suggestions has significantly improved the quality of our manuscript. If you have any further questions or concerns, please feel free to contact us at any time. We are always available and look forward to further discussions with you.

Best regards,

All Authors

Dear Reviewer gwM8,

We hope this message finds you well. We completely understand that your schedule may be demanding, and we sincerely appreciate the time and effort you have devoted to reviewing our manuscript.

In response to your valuable suggestions, we have made the following clarifications and enhancements to our manuscript:

1. We clarified the novelty of our proposed approach.

2. We distinguished the applicability of our method from that of HydraLoRA.

3. We reiterated the sensitivity analysis of our method with respect to the recommendation agents.

4. We introduced quantitative metrics to evaluate semantic similarity.

5. We investigated two aspects of the generalization ability of our method, including its performance on unseen datasets and its transferability to new domains such as the legal summarization task.

We are eager to understand whether our efforts have effectively addressed the limitation in the original manuscript. Given your expertise, we are eager to engage in further discussion with you. However, as the discussion phase draws to a close, we have yet to receive your response. If possible, we would be deeply grateful if you could spare a moment to share your thoughts, which would be of immense value to us.

Thank you again for your time, thoughtful feedback, and invaluable guidance.

Warmest regards,

All Authors of Submission #22904

Response to the Clarity of alpha

We thank you for your valuable feedback. We take this comment very seriously and have revised the manuscript to clarify this point in the main manuscript.

Response to Semantic Similarity Metrics

Thank you very much for your suggestion. We have revised our manuscript to incorporate this feedback.

Response to Qualitative Analysis

We share your concern on this matter. In the original manuscript, we presented a case study in Appendix E.4 to illustrate the semantic inconsistencies caused by LoRA, whereas Magical ensures semantic fidelity. We would like to further share a similar observation we made in the legal summarization task.

• Original Legal Case

  Appeal No. 801 of 1963.

  September 2, 1959 of the MadAppeal by special leave from the judgment and decree dated ras High Court in Second Appeal No. 774 of 1957.

  ...

  On the findings recorded by the High Court, it is clear that the properties belonged to the temple; that they have been transferred by persons who must be deemed to be the previous managers of the temple; and that they have been transferred for valuable consideration. The present suit has been brought against respondents 1 to 3 who are appointed trustees of the temple by respondent No. 4; and so, all the ingredients prescribed by the first column of article 134 B are satisfied. That is why we must reject the ingenious argument urged before us by Mr. Tatachari that article 134 B does not apply to the present case.

  ...

• Summary Generated Text by LoRA

  The appellants filed a suit under section 87 of the Madras Hindu Religious and Charitable Endowments Act, 1951

  ...

  The High Court held that article 134 B does not apply to the present case

  ...

• Summary Generated Text by Magical

  The appellants filed a suit against the respondents for a declaration of their title to certain properties

  ...

  The High Court held that all the ingredients prescribed by the first column of article 134 B were satisfied

  ...

Due to the length of the text (original text: 4572 words; LoRA-generated text: 809 words; Magical-generated text: 664 words), we are presenting only the most salient portions here. The complete cases will be included in the appendix of our revised manuscript.

In this case, the assertion that "article 134 B does not apply" is the argument of Mr. Tatachari, rather than the conclusion of the High Court. In fact, the High Court concluded that “all the ingredients prescribed by the first column of article 134 B are satisfied.” LoRA misinterpreted this distinction, which resulted in a semantic inconsistency. Magical, in contrast, successfully accomplished the crucial task of preserving semantic fidelity in MLLG.

---

We would like to express our sincere gratitude to the reviewers for their time and effort in providing a thorough review of our rebuttal. We truly value the insightful comments and suggestions, which have been instrumental in enhancing the quality of our work. Please do not hesitate to contact us if you have any further questions or concerns. We are readily available and look forward to discussing this further with you.

Warmest regards,

All Authors of Submission #22904

Dear Reviewer gwM8,

We sincerely thank the reviewer for their careful reading of our previous rebuttal and for their positive reception of our responses. We are very pleased that the shortcomings in our original manuscript have been fully clarified and amended. We are also very grateful that the reviewer has provided further questions. This indicates to us that our manuscript still has areas for improvement, a point which we take very seriously, and we are eager to engage in further constructive discussion.

Response to Novelty

We completely understand and respect the reviewer’s decision. We understand that the definition of novelty can be nuanced and acknowledge that this is a matter of seeking common ground while respecting diverse perspectives.

Response to Manually-Specified Recommender

Thank you very much for your suggestion. We would like to offer the following three clarifications:

1. We would like to clarify that our core proposal is the adoption of a "divide-and-conquer" strategy, which utilizes an external agent to perform the recommendation, thereby avoiding the coupling of lay-style recommendation and lay-style generation within the LoRA. The current manually-specified recommender is intended as a specific implementation of this strategy, rather than the ultimate goal of our methodology. We highly value this distinction. To prevent any misunderstanding, we describe the conceptual "recommendation agent" in the methodology section (Section 3.3) and then detail this specific implementation in the experimental setup (Section 4.1).
2. We wish to clarify that the use of a manual selection process, in lieu of proposed recommendation agent, stems from a lack of data to model layperson profiles (as discussed in Section 5 Limitation). We believe this is a limitation not just for our work, but for the MLLG community. Ideally, Magical would involve the collection and curation of user data to realize its complete design, including the data-driven recommendation agent. In that state, Magical would undoubtedly be more compelling. However, we must acknowledge the consensus that dataset construction is a long-term endeavor, which contradicts Magical’s stated purpose as a timely adjustment. We think that without a framework like Magical, MLLG research might continue to be constrained by the limitations of navie LoRA, potentially leading to a waste of resources in the MLLG community.
3. We acknowledge what you mentioned, that the current version of the Recommendation-guided Switch is merely a conceptual placeholder. However, we would like to clarify that making conceptual declarations in methodology while using alternative validation in experiments, particularly when limited by contextual factors such as technology, data, and ethics, is a common and reasonable scientific practice path in academic development. Magical follows the path: declaring the "Recommendation-guided Switch" to replace traditional Router, implementing it using a manual recommender as a proxy for the envisioned recommendation agent, and validating the correctness of this declaration via ablation studies and sensitivity analysis of the recommender's performance.

Response to Practical Utility of Magical

We thank the reviewer for this important suggestion. We would like to clarify the practical utility of Magical, particularly within the MLLG community where user data on lay-style preferences is currently lacking.

Specifically, we offer some preliminary insights on how Magical might be translated into practical clinical use. We propose that its deployment follow four key periods:

1. Expert-Guided Validation: Initially, the system should be open to medical experts to ensure the accuracy and reliability of the generated content.

2. Lay-Style Generation: Leveraging Magical’s strengths, it can then produce diverse lay-style text, allowing users to freely select the most suitable ones based on their latent prefenence.

3. User-Centered Feedback and Learning: User preferences can be collected in this phase to construct a Recommendation Agent.

4. Personalized Lay Text Recommendation: Based on user feedback, the system can ultimately provide precise recommendations of preferred lay-style outputs, facilitating efficient and personalized communication.

We posit that Magical can be instrumental in facilitating the collection and construction of this much-needed user lay-style preference data. In turn, the data-driven recommendation agent, which Magical currently lacks, can be developed and integrated through this process. These two aspects are mutually reinforcing, collectively promoting the practical application and advancement of the MLLG community.

Thank you for your acknowledgment on the importance of our task and the quality of our manuscript. We are especially grateful for your critical comments, which prompted deep reflection on the limitations of our original submission and motivated us to substantially improve our work. Below, we address your concerns point by point.

Response to Evaluation of the Actual Quality of Generated Lay Texts (Weakness#1)

Thank you for your valuable suggestion. We would like to clarify that our initial choice of BLEU and ROUGE metrics was based on established practices in prior MLLG literature [1], where these metrics have been widely adopted. However, we agree with you that such metrics do not fully reflect the actual perceived quality of generated lay texts.

To address this important limitation, we incorporated two complementary evaluation strategies:

1. Large Language Models as Evaluators (Full-scale Evaluation):

Following your suggestion, we used o3-mini and DeepSeek-R1 to rate outputs from Magical and all baseline systems on a 1–10 scale. These models acted as automatic evaluators simulating expert judgment across the full test sets. In this experiment, we selected LLaMA3.1-8B-Instruct as backbone LLM. As shown in the Table below, Magical consistently outperformed all baselines across all datasets, indicating its robust ability to generate high-quality lay texts.

2. Human Expert Evaluation (Focused-scale Evaluation):

To further validate Magical’s quality in a real-world setting, we randomly sampled 20 examples from each MLLG dataset (60 samples total), and invited two medical data analysis experts to finish the human evaluation. Each expert was shown the expert-level source text, and two lay versions generated by LoRA and Magical—with the generation sources anonymized to ensure fairness. The results, presented in the Table below, show that both experts consistently rated Magical higher than LoRA, validating that Magical substantially improves the clarity and quality of generated lay texts.

We acknowledge that the current Human Expert Evaluation—based on only 60 samples—has significant limitations. We sincerely hope for your understanding that conducting a comprehensive human evaluation is inherently a long-term effort. This challenge is further compounded by the length of the data samples used in our study (as shown in Figure 1.a of the original manuscript, each instance averages approximately 300 words), which substantially increases the burden on human experts. Given the constraints of the rebuttal period (one week), it is challenging for us to complete a thorough human evaluation within such a limited timeframe.

We have completed the revision of our manuscript and have incorporated the details of the human evaluation guidelines, the prompts used for LLM-based assessment, and the full results into the updated version.

Response to Full-Parameter Fine-Tuning (FFT) Baseline (Weakness#2)

Thank you for your suggestion. We agree that FFT is an important baseline, as it helps readers better understand why existing MLLG studies tend to choose LoRA over FFT.

In response to your suggestion, we added FFT as a new baseline in our experiments. As shown in the Table below, Magical consistently outperforms the FFT across all datasets and three different backbone LLMs. Moreover, FFT introduces over 150× more trainable parameters compared to Magical. These results highlight Magical’s dual advantages in performance and parameter efficiency.

Our perspective on FFT vs. LoRA is as follows: We actually considered using FFT for SFT, yet LoRA are a more economic and effective choice for MLLG task. The reason is that for MLLG, we often do not have a large amount of training data, and the FFT strategy may lead to overfitting (e.g., learning to respond with some specific tokens instead of learning what the style really depicts). Research works like [2,3,4] also demonstrate that FFT suffers from overfitting and cannot perform as good as other PEFT methods. Moreover, the relatively high training cost of FFT may pose a bottleneck in terms of training efficiency. Therefore, taking both training efficiency and effectiveness into consideration, we choose to apply LoRA as the optimization algorithm for SFT.

Response to Large-Scale LLMs as Baselines (Question#1)

Thank you for your suggestion. We believe that adding a large-scale LLM such as o3-mini as a new baseline would help readers better appreciate the performance advantages of Magical.

Following your suggestion, we added o3-mini as a new baseline and applied the prompt-based method described in Section 4.1 of the original manuscript. As shown in the Table below, the performance of o3-mini with prompting was suboptimal, consistent with the conclusions of our original submission. We argue that simple prompt strategies are insufficient to comprehensively capture the target lay-style, making non-learning-based approaches poorly adaptable to the target dataset. This observation is also reported in recent literature. For example, [5] presents results showing that LoRA-based fine-tuned LLaMA-2 outperforms GPT-4.

Response to Formatting Issues

We greatly appreciate your attention to detail. We sincerely apologize for the formatting errors caused by our oversight. In response, we carefully re-checked our full manuscript against the official NeurIPS formatting guidelines and have corrected all inconsistencies. We take this issue seriously and thank you again for bringing it to our attention.

References

[1] A dataset for plain language adaptation of biomedical abstracts. Nature Scientific Data, 2023

[2] Low-rank adaptation for multilingual summarization: An empirical study, NAACL, 2023

[3] Parameter-efficient orthogonal finetuning via butterfly factorization. ICLR, 2024

[4] AdaLoRA: Adaptive budget allocation for parameter-efficient fine-tuning. ICLR, 2023

[5] Retrieval augmentation of large language models for lay language generation. Journal of Biomedical Informatics, 2024

---

We sincerely thank you once again for your thoughtful guidance. Your feedback has been immensely valuable to us, and incorporating your suggestions has significantly improved the quality of our manuscript. If you have any further questions or concerns, please feel free to contact us at any time. We are always available and look forward to further discussions with you.

Best regards,

All Authors

Dear Reviewer DTBh,

We hope this message finds you well. We completely understand that your schedule may be demanding, and we sincerely appreciate the time and effort you have devoted to reviewing our manuscript.

In response to your thoughtful suggestions and questions, we have carefully prepared a detailed rebuttal that we hope addresses your concerns.

Specifically, we have made the following improvements:

1. We expanded our evaluation to include both Large Language Models as Evaluators (Full-scale Evaluation) and Human Expert Evaluation (Focused-scale Evaluation) to assess the actual perceived quality of the generated lay texts. Results consistently demonstrate the superiority of our method under both evaluation protocols.

2. Following your suggestion, we introduced new baselines including Full-Parameter Fine-Tuning and Large-Scale LLMs (e.g., o3-mini). Experimental results show that our approach outperforms these strong baselines, and we further provide insightful analysis into their limitations.

We are eager to understand whether our efforts have effectively addressed the limitation in the original manuscript. Given your expertise, we are eager to engage in further discussion with you. However, as the discussion phase draws to a close, we have yet to receive your response. If possible, we would be deeply grateful if you could spare a moment to share your thoughts, which would be of immense value to us.

Thank you again for your time, thoughtful feedback, and invaluable guidance.

Warmest regards,

All Authors of Submission #22904

Dear Reviewer DTBh,

We hope this message finds you well.

We are writing to kindly remind you that the deadline for the discussion phase is drawing close, with only one day remaining. We greatly appreciate the time and effort you have already dedicated to reviewing our work, and we have provided point-by-point responses to your constructive suggestions and questions.

However, it appears that we have not yet received your response to our rebuttal, and we are wondering if our responses have addressed your concerns.

Your insights and feedback are invaluable to us, and we sincerely hope you can participate in the discussion phase so that we can together promote a review-author discussion environment characterized by fairness, politeness, and calmness, as encouraged by the NeurIPS Program Chairs, and foster a more positive NeurIPS community.

Thank you once again for your time and consideration. We are looking forward to hearing from you soon.

Warmest regards,

All Authors of Submission #22904

Thank you for your acknowledgment on the usefulness of our method, the importance of the task, the thoroughness of our experiments, and the potential to engage a wider audience. We are particularly grateful for your clarification that the limitations of evaluation due to static and narrow datasets are a well-known and general challenge in the field of medical LLMs. Your perspective is highly encouraging and has motivated us to further advance the task of MLLG.

Response to Evaluation Realism Concerns (Weaknesses#1)

We appreciate your insightful observation. We also acknowledge the inherent limitations posed by the static nature and limited scope of the datasets, which affect the realism and generalizability of LLM evaluation. As you noted, this remains a general problem across the medical NLP community. Despite these challenges, we adhered to standard academic protocols by employing benchmark datasets [1,2,3] and evaluation metrics [4] that are widely used in prior research. In the manuscript, we aims to present our observations and insights to inspire further discussion within the community. We would like to emphasize that, within such a constrained setting, Magical makes a modest yet meaningful step forward.

Response to External Validations for Automated Metrics (Question#1)

We appreciate your raising the critical question: “How do we know the proposed methods do not just overfit to the measured quantity vs. actual perceived quality?”—a fundamental concern in evaluating MLLG systems. To address this, We conducted a human evaluation study and invited two experts specializing in medical data analysis. We randomly selected 20 samples from each MLLG dataset (totaling 60 samples) and asked the experts to rate the quality of generated lay summaries on a scale from 1 to 10. Each expert was presented with the original expert-level text and two lay versions generated by LoRA and Magical, respectively, in a blind setting (i.e., the sources of the lay summaries were not disclosed).

We select LLaMA3.1-8B-Instruct as backbone LLM and the evaluation results are summarized in the table below. Both experts consistently rated Magical significantly higher, confirming its superiority in producing high-quality, understandable lay summaries.

We acknowledge that the current evaluation—based on only 60 samples—has significant limitations. We sincerely hope for your understanding that conducting a comprehensive human evaluation is inherently a long-term effort. This challenge is further compounded by the length of the data samples used in our study (as shown in Figure 1.a of the original manuscript, each instance averages approximately 300 words), which substantially increases the burden on human experts. Given the constraints of the rebuttal period (one week), it is challenging for us to complete a thorough human evaluation within such a limited timeframe.

To complement this, we additionally employed o3-mini and DeepSeek-R1 as LLM-based evaluators to simulate large-scale expert-like judgments. These results, presented in the table below, consistently show that Magical outperforms all baselines across all datasets, further supporting that its improvements are not merely due to overfitting to specific metrics, but reflect genuine quality gains.

We have completed the revision of our manuscript and have incorporated the details of the human evaluation guidelines, the prompts used for LLM-based assessment, and the full results into the updated version.

References

[1] Paragraph-level Simplification of Medical Texts, NAACL, 2021

[2] Making Science Simple: Corpora for the Lay Summarisation of Scientific Literature, EMNLP, 2022

[3] Retrieval augmentation of large language models for lay language generation, Journal of Biomedical Informatics, 2023

[4] A dataset for plain language adaptation of biomedical abstracts. Nature Scientific Data, 2023

---

Once again, we deeply appreciate your constructive feedback. Your emphasis on perceived quality evaluation is crucial for MLLG and has inspired us to push further toward building methods with real-world impact and user value. If you have any further questions or concerns, please feel free to contact us at any time. We are always available and look forward to further discussions with you.

Best regards,

All Authors

Dear Reviewer TVd2,

We hope this message finds you well. We completely understand that your schedule may be demanding, and we sincerely appreciate the time and effort you have devoted to reviewing our manuscript.

In response to your thoughtful suggestions and questions, we have carefully prepared a detailed rebuttal that we hope addresses your concerns.

Specifically, following your valuable feedback, we conducted two evaluations to assess the perceived quality of our method:

1. Human Expert Evaluation (Focused-scale Evaluation): As suggested, we invited two domain experts to manually assess a randomly selected subset of our dataset, enabling a precise and qualitative examination.

2. Large Language Models as Evaluators (Full-scale Evaluation): To compensate for the limited scale of human evaluation, we employed two large language models as judges to conduct a comprehensive assessment over the entire dataset.

Experimental results indicate that our method enhances actual perceived quality rather than merely optimizing for superficial metric gains.

We are eager to understand whether our efforts have effectively addressed the lack of evaluation on actual perceived quality in the original manuscript. Given your expertise, we are eager to engage in further discussion with you. However, as the discussion phase draws to a close, we have yet to receive your response. If possible, we would be deeply grateful if you could spare a moment to share your thoughts, which would be of immense value to us.

Thank you again for your time, thoughtful feedback, and invaluable guidance.

Warmest regards,

All Authors of Submission #22904

Dear Reviewer TVd2,

Thank you for your careful reading of our manuscript and rebuttal, and for your thoughtful and positive response. Your suggestions are invaluable to us, and we truly appreciate every opportunity to engage in this exchange, which is essential for improving the quality of our work.

Once again, we are sincerely grateful for your kind guidance.

Best wishes,

All Authors of Submission #22904