Enhancing Parameter Efficiency in Summarization via Expertise Separation

Thank you for the valuable comments that help us improve the work. Below we address the concerns mentioned in the review:

---

Q1: Limited baselines

A1: we would like to mention that our compared baseline is not limited to BART. We also include prompt-tuning approach Prefix [1] and adapter-based baseline Lightweight [2], which all underperform our model significantly. For the Summareranker, we’d like to highlight the difference between these two models: Summareranker reranks the generated summaries from multiple models, while in our model, we have one unified model for multiple datasets and domains. Hence, we include it in our revision in related work.

[1] Prefix-Tuning- Optimizing Continuous Prompts for Generation, ACL 2021

[2] Lightweight Meta-Learning for Low-Resource Abstractive Summarization, SIGIR 2022

---

Q2: Using the method on other architecture such as PEGASUS

Thanks for your thoughtful consideration. In our Appendix A.4, we include the comparison with not only BART-large, but also BART-base and PEGASUS. The results show the generalization ability of our framework that does not depend on specific model scales or architectures. We choose the BART-large structure in our main experiment due to its better performance.

---

Thank you again for the insightful questions. We hope that our answers and revised paper have addressed your questions. If you have future suggestions or questions about our paper, we will feel very happy to share more responses.

Thank you for the valuable comments that help us improve the work. Below we address the concerns mentioned in the review:

---

Q1: it would be very informative to have a table with actual parameter count for each setting, especially during training (frequently the most important bottleneck).

A1: Thanks for your suggestion, below we show the parameter table and we also include it in the revision in Appendix 9:

It can be seen that further scaling up the training data does not significantly increase the number of parameters. We have included this table in the revised version.

---

Q2: FactorSum also uses a BART backbone to capture general importance and uses a separate optimization procedure for domain-specific factors such as summary conciseness. It achieves superior performance (measured by ROUGE) than MoeSumm on arXiv and PubMed using less parameters.

Thanks for bringing this great work to our attention, and we’ve included it in the revision! In the meantime, we would like to highlight the essential differences between our work:

In FactorSum, the BART backbone model is trained on a single target dataset to acquire general summarization skills, where ‘general’ refers to summarization without specific consideration for their length. Consequently, a separate model is trained for each dataset to accommodate the different summarization requirements.

In contrast, in our setting, the general summarization ability is not limited to one dataset and its length information, but it is learned from different datasets. Take Figure 5b in the paper for example, our specialized ability includes the understanding of the professional terms in a specific domain.

---

Q3: Analysis of the weight of each feature with respect to summary length

The summary length is indeed an important factor when evaluating the summary performance. As shown in Figure 5a, we can see that MoeSumm w/o deputy experts lacks information regarding the target length, while MoeSumm efficiently halts the generation process to produce an optimized summary length. The use of deputy experts is also linked to the length of summaries. For instance, our findings show that MoeSumm when paired with Deputy Expert #1 typically produces shorter sentences, averaging around 15 words, whereas MoeSumm working with Deputy Expert #3 tends to create longer sentences, averaging about 37 words.

---

Q4: Why the zero-shot performance of MoeSumm is significantly better than BART-mix on some datasets (arXiv, MultiNews) but not others (Gigaword, Billsum)?

Thank you for your insightful question. We believe that MoeSumm's effectiveness is attributed to reusing summarization skills from in-domain training datasets. For arXiv and MultiNews, which feature long, specialized content, skills learned from similar datasets like PubMed and CNN/DM are applicable. Conversely, Gigaword and Billsum present unique challenges due to their brevity and legislative content, not covered in our training. Following this direction, we draw the performance on Gigaword as training datasets get larger. As shown in Appendix A.5, MoeSumm's performance on Gigaword notably improves (by 7% in ROUGE-1 scores) when trained with similar datasets such as AESLC and Reddit.

---

Q5: Are differences in human evaluation scores (Table 2) statistically significant?

Yes, the improvements to the second-best performance are statistically significant for $\alpha=0.01$.

---

Thank you again for the insightful questions. We hope that our answers and revised paper have addressed your questions. If you have future suggestions or questions about our paper, we will feel very happy to share more responses.

Thank you for the valuable comments that help us improve the work. Below we address the concerns mentioned in the review:

---

Q1: Allocating each deputy to a single dataset doesn’t sound too practical and scalable as the number of datasets increases. If many datasets are similar, why are we not sharing similar deputies?

We apologize for any confusion caused. Indeed, you are absolutely correct in noting that similar applications should share similar deputy experts, which aligns perfectly with our motivation for our MoeSumm structure. In practice, we do not allocate a specific deputy to a single dataset. Instead, we utilize shared multiple deputy experts which are dynamically selected based on the context and dataset category. We have revised and emphasized this aspect in Section 4 for clearer understanding.

---

Q2: The design of the nonlinear function in Equation 9.

We choose a Quintic function as it offers more stability according to previous study [1]. We now include a new ablation study in the revision:

It can be seen that our MoeSumm keep relatively good performance on different nonlinear function design, particularly excelling with the Quintic function.

[1] Prevent the Language Model from being Overconfident in Neural Machine Translation

---

Q3: Max-margin loss to solve the imbalance problem

A3: Thanks for listing the interesting references about the imbalance issue in MoE. However, we would like to clarify that our max-margin loss is not to balance between deputy experts. Instead, it is an adapted loss for hierarchical MoE structure, to differentiate between general ability in main expert and specialized ability in deputy experts. Hence, the listed works on imbalance loss can not replace our max-margin loss. Still, we are happy to include these references in our revision.

---

Q4: Ablation study on the imbalance problem (max-margin loss)

A4: As explained in the previous answer, our max-margin loss is not designed for imbalance problem. In fact, we design a simple and intuitive way for the imbalance problem, which incorporates the dataset category information when selecting deputy experts. Our method is mostly related to [1. Task-Based MoE for Multitask Multilingual Machine Translation], where we both incorporate task(dataset) information into the selection process.

Thanks for your suggestion, and we conduct new experiments to see if other imbalance-related techniques can be used to solve the imbalance problem in our scenario. Specifically, we choose the embedding method proposed in [1], which introduces task embedding, additional task gate, and MoE gate to selector the adaptors. The results show that the baseline and our methods achieve similar balanced distributions but with fewer parameters.

---

Q5: Human evaluation details

We ensure high-quality annotations by hiring well-educated PhD students, specifically selecting those with backgrounds in computer science, chemistry, and biomedicine to guarantee diversity. Each case was compensated at $ 0.01$, leading to a total payment of $6 per student.

---

Q6: Evaluation by LLM

Thanks to your advice, we conduct an evaluation with the latest GPT4. The instruction is:

Score summary1, summary2, and summary3 given the article in respect to fluency, with one (*) to three stars (***), where one star means bad and three stars means good. Only return the stars.

The ranking aligns with our human assessment, affirming the reliability and effectiveness of both evaluation methods and showing our model's advantage.

---

Q7: MoESumm w/o DI details

By removing the dataset information in the selector, the dataset-aware selector degrades into a naive selector. That is, the $W_e$ matrix for $e$ dataset is changed to $W$, which is the same for all the datasets. Though the dataset information is removed from the selector, we still have the main and deputy experts, thus, the max-margin loss still works in differentiating these experts.

---

We hope that our answers and revised paper have addressed your questions. If you have future suggestions or questions about our paper, we will feel very happy to share more responses.

Thank you for the valuable comments that help us improve the work. Below we address the concerns mentioned in the review:

---

Q1: The compared BART baseline is weak. Should include [Adaptsum] and [SPEC].

A1: Thank you very much for bringing these papers to our attention. First, we would like to mention that our compared baseline is not limited to BART. We also include strong baselines including prompt-tuning approach Prefix [1] and adapter-based baseline Lightweight [2], which all underperform our model significantly. This is introduced in the paper and details can be found in Appendix. For [Adaptsum], it utilizes more training datasets (300 vs 10) and its performance is lower than our model and baseline Lightweight in different datasets such as on Reddit and AESLC, as shown below. For [SPEC], it proposes several criteria for choosing meta training data, while in our work, we do not specifically choose the training data but use a unified one. Our main target is not to improve performance by data selection or data augmentation. Instead, we aim to separate general and specific summarization abilities. Hence, the result is not directly comparable. Following your suggestion, we now include the two works and the discussion in our revision.

[1] Prefix-Tuning- Optimizing Continuous Prompts for Generation, ACL 2021

[2] Lightweight Meta-Learning for Low-Resource Abstractive Summarization, SIGIR 2022

---

Q2: MoE models are worse in efficiency in comparison with existing few-shot baselines that directly finetune the model.

A2: Firstly, we want to clarify that our model has fewer parameters compared with a number of existing few-shot baselines. As shown in the table, our model has 10% less parameters compared with the recent baseline Prefix and 24% less parameters with Lightweight.

Secondly, our model is different from existing MoE models. Our MoeSumm is a hierarchical structure with main and deputy experts. The general summarization ability for various domains is shared to achieve parameter efficiency.

---

Q3: Contributions of the paper

A3: The contribution of the paper can be summarized into three aspects: (1) We propose MoeSumm, a parameter-efficient summarization model that is applicable to a variety of in-domain summarization tasks and is well-suited for out-of-domain few-shot and zero-shot summarization scenarios. The model's parameter efficiency is ensured by the shared general summarization ability. (2) To achieve the above goal, we design MoeSumm with a mixture-of-expert structure and a max-margin loss to distinguish the roles of the different experts. (3) Experimental results demonstrate that MoeSumm brings substantial improvements over strong baselines in both in-domain and out-of-domain scenarios across benchmark datasets.

---

Q4: The title in the system and the title in the paper are not consistent.

A4: Thank you for the reminder. We’ll keep the title in the paper ‘Flexible and Adaptable Summarization via Expertise Separation’.

---

Thank you again for the insightful questions. We hope that our answers and revised paper have addressed your questions. If you have future suggestions or questions about our paper, we will feel very happy to share more responses.