Q1: My first concern is that this paper mainly compares the proposed method with Diffuseq, SeqDiffuSeq, and Dinoiser. There are many other diffusion models tailored for general seq2seq tasks [1-6]. I would suggest the authors carefully review the existing literature and select more recent baselines to compare.

Response:: Thank you for the insightful feedback. We have included additional baselines in the following Table, specifically those that are evaluated on the same dataset as ours and have open-source implementations (Diffuseq-v2, BG-DiffuSeq and TESS). We demonstrate that our Meta-Diffu$B$ framework can integrate various baselines as exploiter models, achieving superior performance compared to the original models. This is consistent with the experiments and results presented in Appendix F.

More recent baselines compared with our Meta-Diffu$B$ on QG and QQP datasets

Q2: As the paper mentioned, parts of the methodology are motivated by Meta-Exploration. I would suggest the authors add a section in the preliminary to provide more necessary details about the meta-exploration techniques, which will benefit readers who are not familiar with Meta-Exploration.

Response: To assist readers in understanding the foundational concepts, we will move the discussion of Meta-Exploration from the Related Work section to the Preliminary section in our final version.

Q3: The authors claim that the proposed method results in a contextualized noise schedule, which makes sense to me in terms of the scheduler generating Meta-Instructions conditioned on. However, from Figure 3, even though the total amount of noise indeed fluctuates across different training epochs, it always keeps at the same level and the differences seem insignificant. I'm wondering why this makes the proposed methods beat existing baselines.

Response: In Figure 3, our method differs from approaches like SeqDiffuSeq or Dinoiser, which rely on predefined rules. Instead, our approach is data-driven, with the model learning the underlying patterns. As a result, the degree of variation—whether large or small—is not the primary focus. Additionally, the learned noise can be flexibly integrated into other systems in a plug-and-play manner.

Q4: Can you provide a deeper interpretation of the noise scheduling visualizations and their implications for different types (easy or hard as mentioned in the paper) of sentences? How do different noise schedules affect the generation quality and diversity for various types of sentences? Any patterns or insights observed from these visualizations?

Response: From the visualizations, we observe that the contextualized noise scheduling mechanism helps in maintaining a delicate balance between exploration (diversity) and exploitation (quality). The patterns indicate that our model has learned different strategies for different types of sentences. These insights demonstrate the effectiveness of our Meta-Diffu$B$ framework in handling various sentence complexities, resulting in superior performance across multiple datasets.

Q5: Could you provide an intuitive explanation of why the scheduler can function as a plug-and-play model? From the analysis in section 4.7, I'm suspecting it is because the learned schedule is not quite different from the pre-defined noise schedule.

Our scheduler model learns directly from sentence semantics, allowing it to provide learnable noise across different text datasets. Figure 3 shows that Dinoiser and SeqDiffuSeq rely on rule-based noise variations, which do not necessarily lead to better performance.

Q6: Did you run the empirical evaluation for several runs by setting different random seeds? If so, could you also highlight which results are statistically significant than the baseline method?

Response: Thank you for your question regarding the empirical evaluation with different random seeds. We followed the evaluation protocols of DiffuSeq and Diffusion-LM, running multiple experiments with different random seeds. In Tables 1, 2, and 3, we used MBR, so the results in bold are all statistically significant compared to the baseline methods.

Q1: Incremental Innovation: In terms of the scheduler, the use of the Meta-Exploration method for scheduling contextualized noise is noted. However, I concern that the exploiters fully follow the DiffuSeq work, resulting in a lack of significant innovation.

Response: Thank you for your feedback. As noted in Appendix F, our framework can utilize different diffusion models as exploiters, demonstrating its versatility. The key innovation is integrating these models with our scheduler, significantly enhancing performance. We propose a general method that strengthens other S2S-diffusion models, providing more than incremental improvements. In Sections 4.5 to 4.7, we compare our Meta-Diffu$B$, using DiffuSeq as the exploiter model, with other S2S-Diffusion models that incorporate adaptive noise to highlight its effectiveness.

Q2: Missing Experimental Baselines: The cited baselines are all from before February 2023. It would be beneficial to include more recent baseline results [1] [2] [3] and take both generation efficiency and training speed into evaluation metrics.

Response: Thank you for the insightful feedback. We have included additional baselines in the following Table, specifically those that are evaluated on the same dataset as ours and have open-source implementations (Diffuseq-v2, BG-DiffuSeq and TESS). Results where Meta-Diffu$B$ combined with different models show improved performance are indicated in bold. Our Meta-Diffu$B$ framework can integrate various baselines as exploiter models, achieving superior performance compared to the original models. This is consistent with the experiments and results presented in Appendix F.

More recent baselines compared with our Meta-Diffu$B$ on QG and QQP datasets

Q3: Dataset Limitations: The selected dataset completely follows DiffuSeq and does not take into account machine translation tasks and related datasets from SeqDiffuSeq and DINOISER. It would be advantageous to include test results on IWSLT14 and WMT14.

Response: Thank you for the valuable feedback. We supplement additional machine translation datasets in the following table and employed the same evaluation metrics used by SeqDiffuSeq and Dinoiser for these tasks. Results where Meta-Diffu$B$ combined with different models show improved performance are indicated in bold. Furthermore, we demonstrate that our Meta-Diffu$B$ framework, when combined with different S2S-Diffusion models, achieves superior performance in machine translation tasks as well.

Experiment of Meta-Diffu$B$ on Machine Translation dataset

Q4: Plug-and-Play Capability: The study only demonstrates the plug-and-play model and its corresponding effects on DiffuSeq. It's recommended to show the plug-and-play capability in other S2S Diffusion models like SeqDiffuSeq, DINOISER, and other baselines mentioned to prove the claim in the conclusion.

Response: Thank you for your suggestion. We supplement our experiments with plug-and-play capabilities on Dinoiser and SeqDiffuSeq in the following tables. Our scheduler, trained on different datasets, improves the performance of both Dinoiser and SeqDiffuSeq. The fields "Dinoiser" and "SeqDiffuSeq" indicate which dataset these two models are trained on. When the scheduler field is null, it indicates the use of the model's own noise scheduling. Results where the model performs better with its own noise are indicated in bold.

Plug-and-play experiments on SeqDiffuSeq integrated with our scheduler

Plug-and-play experiments on Dinoiser integrated with our scheduler

Q1: L120, 123: the quote for 'skipping' is wrongly formatted

Response: Thank you for pointing out the formatting issue. We will correct the quotation marks for 'skipping' in lines 120 and 123 to ensure proper formatting.

Q2: L202-203 it should be either "The maximum Minimum Bayes Risk (MBR)..." or "The Minimum Bayes Risk (MBR)....", also it cites paper [20] (the Rouge package) which I think is incorrect for MBR.

Response: Thank you for your reminder. We will correct this in the final version by citing the Diffusion-LM and DiffuSeq papers in this reference.

Q2: Besides the increased training time compared to DiffuSeq (L217-218), it would be beneficial to know the increased inference time too.

Response: Thank you for this suggestion. We will supplement the increased inference time compared to DiffuSeq in the following Table.

Meta-DiffuB's computational complexity compared to DiffuSeq

Q3: How well does the Meta-Diffu$B$ model generalize to other types of text generation tasks beyond the Seq2Seq framework? Have experiments been conducted to evaluate its performance in different text generation scenarios?

Response: We conducted extensive experiments to evaluate the generalization capabilities of the Meta-Diffu$B$ model across various text generation task. These tasks include generating informative dialogue responses (CC dataset), question generation (QT dataset), text simplification (WA dataset), and paraphrase generation (QQP dataset), as discussed in Section 4.1 and Appendix A. Thanks to the reviewers' suggestion, we supplemented additional machine translation datasets, as shown in the following table, and employed the same evaluation metrics used by SeqDiffuSeq and Dinoiser. Results where Meta-Diffu$B$ combined with different models show improved performance are indicated in bold. Our results show that Meta-Diffu$B$, when combined with different S2S-Diffusion models, achieves superior performance on machine translation tasks as well.

Experiment of Meta-Diffu$B$ on Machine Translation datasets

Q4: How does the model scale with larger datasets or more complex text generation tasks? Are there any specific optimizations or modifications required to handle real-world data volumes?

Response: Meta-Diffu$B$ scales effectively with larger datasets, as demonstrated by our experiments on the Commonsense Conversation (CC) dataset with 3 million data points. We maintained consistent model parameters, architecture, and optimization methods across all datasets, including CC. Our framework's exceptional performance, even on large datasets, highlights its robustness and scalability. Additionally, as shown in the table from Q3, our Meta-Diffu$B$ performs well on machine translation tasks with the IWSLT14 dataset (160k data points) and the WMT14 dataset (4475k data points), without requiring any adjustments.

Q1: For example, Figure 3, the noise level is quite stable for the proposed method but in the text below, it claims 'Meta-Diffu$B$ applies adaptive noise at varying training epchos.

Response: In Figure 3, our method differs from approaches like SeqDiffuSeq or Dinoiser, which rely on predefined rules. Instead, our approach is data-driven, with the model learning the underlying patterns. As a result, the degree of variation—whether large or small—is not the primary focus. Additionally, the learned noise can be flexibly integrated into other systems in a plug-and-play manner.

Q2: Another example is in Figure 4 at least for QQP dataset - Meta-Diffu$B$ applies less noise than the other methods as its curves lay below the curves of the other methods. thus, the claim 'This noise-scheduling approach—applying more noise to the harder sentences (H) and less to the easier sentences' looks conflict with such observation from the figure.

Response: We appreciate the opportunity to correct our previous statement. We will correct this error in the final version.

Q3: Besides, as algorithm 1 alternates the training of scheduler and exploiter, It would be guideful if some insights about the convergence rate are provided.

Response: We supplement the Training Curve Image. Our Meta-Diffu$B$ shows Our model converges significantly faster than DiffuSeq, achieving convergence within just 10k epochs.