We sincerely thank Reviewer e1Ke for reviewing our submission and for recognizing the clarity, creativity, and practical value of our work. Below, we address your comments in detail.

Reply to Q1

In some formulas, it's unclear whether the pairs (xi, yi) are drawn from the real-world distribution P or the synthetic data distribution Q. For instance, in Appendix C, it is clearly noted that (yp', xp') is from distribution P. Could you use similar notation consistently throughout the main body to help readers differentiate between real and synthetic data sources?

Thank you for pointing out this potential issue regarding the notation. In Appx C of this version, $(\mathbf{y}_p', \mathbf{x}_p')$ represents a dataset, where $\mathbf{y}_p'$ is a set of ground truth labels and $\mathbf{x}_p'$ is a set of input data. We found this notation may have caused some confusion, since $(\mathbf{y}_p', \mathbf{x}_p')$ looks like a data point. Therefore, to improve clarity, Eq(21) should be interpreted as the prodcut of each point's conditional probability (due to i.i.d):

$$\arg\max_{(\mathbf{x}, y) \in D_Q} \prod_{(\mathbf{x}', y') \in D_{P'}} \hat{P}(y'|\mathbf{x}'; \theta_{t}, \{(\mathbf{x}, y)\})$$

We are revising the terms throughout the manuscript. For instance, the confusing $(\mathbf{y}_p', \mathbf{x}_p')$ could be updated to $(\mathbf{y}_p', \mathbf{X}_p')$ to clearly indicate this refers to a dataset, while variables such as $(\mathbf{x}_i, y_i)$ from $D_Q$ could be $(\mathbf{x}_i^{Q}, y_i^{Q})$, and similarly for other distributions. We think this update would help differentiate between real and synthetic data sources more consistently and intuitively.

We appreciate your suggestion, and we will refine our notations in the next revision as soon as possilbe to ensure the notation is unambiguous.

Reply to Q2

To improve clarity, consider starting a new paragraph with the sentence: "Thus, inspired by the objective of Hu et al. (2019), we introduce two novel, efficient, automatic weighted-loss approaches: Importance Loss (IMP-Loss) and Dynamic Importance Loss (DIMP-Loss)..." This will make the contributions of this paper stand out more clearly, preventing them from being mixed with discussions of prior work.

Thank you for carefully reviewing our work and providing this valuable suggestion. We have incorporated your feedback and made the necessary edits in the revised version of the manuscript.

Reply to Weaknesse1

The core idea of weighting synthetic data samples is not entirely novel. A similar concept was explored in the paper "Self-Guided Noise-Free Data Generation for Efficient Zero-Shot Learning" published last year. Although the formulation and implementation in this paper differ, the underlying principle of prioritizing certain synthetic data points is conceptually similar, which somewhat limits the novelty of the contribution.

Thank you for highlighting this impressive work. In the revised manuscript, we have extended our experiments to include a comparison with SunGen [1], which is closely related to the idea of weighting synthetic data samples. The results demonstrate in Table 1 and Table 3, shows that while SunGen employs a meta-learning-based approach that requires approximately seven times the computational cost of our DIMP-Loss method, our approach still consistently outperforms SunGen in overall performance. These findings suggest that our method offers a promising and more efficient alternative for advancing the field of synthetic data weighting. We believe this highlights the potential of our approach as a practical and impactful contribution to this area of research.

Reply to Weaknesse2

Another concern lies in the strong reliance on the assumption that a model trained on a small real-world dataset P can approximate the performance of a model trained on a larger, complete dataset. The paper would benefit from a more in-depth discussion or experimentation addressing this assumption. Specifically, it would be helpful to explore the minimum number of real-world examples required for this assumption to hold and to provide guidelines on the amount of data practitioners should aim to collect.

We appreciate your concern regarding the reliance on a small real-world dataset to approximate the larger distribution. In response to your feedback, we have included CE-Loss in Figure 2 in the revised manuscript to facilitate easier comparisons. Empirically, we provided a comparison of different amounts of training data used for the quality checker, as illustrated in Figure 2. The results indicate that increasing the size of the real-world training dataset improves performance. However, even a small amount of real-world data proved sufficient to significantly enhance the performance of both IMP-Loss and DIMP-Loss compared to directly using CE-Loss and the Quality Checker alone.

Reference

[1] Gao et al., Self-Guided Noise-Free Data Generation for Efficient Zero-Shot Learning, ICLR 2023.

We sincerely thank Reviewer eiLk for taking the time to review our submission and providing thoughtful feedback. We are grateful for your recognition of our work's potential and the strong positive results it demonstrates. Below, we provide detailed responses to each of your comments.

Reply to Q1

I am still uncertain whether the proposed diversity checker functions as intended. What happens if unrelated data is fed into the diversity checker?

Thank you for your insightful suggestion. To address this concern, we have extended our noise data experiments to this revision in Appendix G by incorporating unrelated input data from a financial benchmark. The average weights derived from the Quality Checker, Diversity Checker, and overall weight adjustments (Figures 4, 6, and 7) provide a comprehensive analysis. Our findings indicate that while the Diversity Checker assigns slightly lower scores to unrelated data (lower is better), the Quality Checker scores drop significantly (higher is better). This discrepancy results in a considerable reduction in the overall weight of unrelated data compared to both the original data and duplicate data. These findings align with our derivations, emphasizing that the model should be trained on data points that exhibit both quality and diversity properties.

Reply to Q2

Some comments regarding the paper's formatting

Thank you for the suggestions. We have double checked the formatting of capitalization, spacing and citations.

Reply to Weakness 1

The authors primarily build their argument by citing studies focused on generating synthetic data for instruction tuning of LLMs, such as Alpaca. However, the method proposed in this paper is aimed at mitigating noise in synthetic data for text classification, which might create some confusion.

Thank you for pointing out these potential issues in our paper. We appreciate your suggestion regarding the relevance of related studies. In response, we have revised the Introduction and Related Work sections to highlight the studies you mentioned, which are indeed more pertinent to our proposed methods. These updates are clearly marked in red in the latest version of the manuscript to ensure they are easily identifiable. Your feedback has helped us improve the clarity and relevance of our target task.

Reply to Weakness 2

I believe discussing these studies would be more relevant compared to the current organization of the manuscript. Accordingly, the authors should have discussed these works and used them as baselines.

Thank you for your valuable feedback and suggestions. To address your concerns and improve the manuscript, we have extended our experiments to include comparisons with the impressive work SunGen [1], as presented in Table 1 in latest version. Indeed, SunGen, with its meta-learning-based approach, demonstrates greater robustness compared to Hu et al.'s method, making it a more suitable baseline for our study. However, our results show that both IMP-Loss and DIMP-Loss consistently outperform SunGen in overall performance. Additionally, we have included a computational time comparison in Appendix E, which highlights that SunGen's approach requires approximately seven times more computation time than our DIMP-Loss method. This reinforces the efficiency of our proposed approach.

Furthermore, we have added UniGen [2] and FuseGen [3] in the latest version of the Related Work section, with updates highlighted in red for clarity. UniGen employs contrastive learning with the generator's confidence scores, which are not applicable in our setting due to our reliance on the closed-source LLM GPT-3.5; therefore, we cannot conduct the pseudo-relabelling mentioned in UniGen. FuseGen, on the other hand, generates synthetic data from multiple LLMs and uses a multi-stage regeneration process, culminating in a boosting technique to adjust data weights. This approach is interesting and inspiring; however, for single LLM, it seems unable to show its potentials on the classification task. We did some pilot study, using their implementation on financial data generated from a single source (GPT-3.5), and revealed limitations in FuseGen's effectiveness in this setting. The optimal testing accuracy achieved was 67.99%, significantly lower than the 77.39% accuracy achieved with CE-Loss. Additionally, the computational cost of boosting is still considerable, as it requires inference over the entire training dataset during each epoch to compute errors for weight adjustment. In contrast, our method calculates weights once before training, drastically reducing computational overhead.

In summary, we believe our approach offers a more efficient, robust, and versatile solution for tasks of this nature, with the extra advantage of compatibility with any loss function using common gradient optimization, such as SGD. Therefore, our approaches can also be used on UniGen or FuseGen, and potentially complement and enhance these frameworks.

We sincerely thank Reviewer R1bf for reviewing our submission and for recognizing the clarity of our theoretical derivations, the robustness of our results, and the structure of our paper. Below, we address your comments in detail.

Reply to Q1

For the metrics "diversity" and "quality", would it be possible to verify that they actually correspond to what they refer to? I.e., does higher quality data actually get higher quality metrics? Currently, it seems like a term for the mathematical expression. While I understand it intuitively, it would also be nice to verify them empirically.

For quality, we consider data points that are more relevant, and for diversity, we consider those data appear less frequently. In Appendix G.3 of new revision, depicts three scenarios characterized by lower quality (random labels, unrelated inputs) and reduced diversity (repeated data). Furthermore, Figure 4 presents the average scores of the Diversity Checker and Quality Checker, as well as the weight assigned to each of the three scenarios in comparison to real-world data. We appreciate your attention to these details and hope this additional clarification addresses any potential concerns.

Reply to Q2

In the tweet irony case in Figure 1, DIMP loss start out with worse performance than other baselines for the first and second epoch. Could you explain why such a scenario would occur? Does this mean that these methods introduce instability to the training process?

Thank you for your observation, in DIMP-Loss, we use a random initialized diversity checker, so the variance in the begining stages is expected. However, from the result of experiments, we can see the variance of DIMP-Loss converges in later epochs.

Reply to Weakness 1

The results are derived using a BERT-Based model, which is quite small compared to contemporary LLMs. How would the method scale to larger LLMs? How would the costs grow as the model size grows?

We appreciate your observation regarding the potential of our method to be applied to larger LLMs. In this work, our focus is on fine-tuning text classifiers on LLM-generated data. To explore scalability, we conducted an empirical study using BERT-Large as the fine-tuning model while employing a smaller model as the quality checker, as shown in Table 2. The results demonstrate that DIMP-Loss achieves significant improvements compared to CE-Loss, indicating the method's potential to scale effectively to larger models without requiring an equally large model for the quality checker. This makes our approach more practical and cost-efficient. We believe it would be exciting to further explore the application of our method in training larger models.

We sincerely thank Reviewer gCSy for thoroughly reviewing our submission and appreciating the clarity, robustness, and structure of our work. Below, we provide detailed responses to your comments.

Reply to Q1

Paper assumes Q(x) ~ P(x) on line 171, but in line 154-164, the synthetic data has high diversity and entropy. Wondering how to interpret the two together?

Thank you for pointing out the potential confusion, the high conditional entropy in $Q(y|x)$ reflects the diversity of the synthetic data and may include label noise, as noted in Section 3.2. This does not contradict the assumption $P(x) \approx Q(x)$ since the assumption pertains solely to the marginal input distributions.

Additionally, this assumption is required solely for IMP-Loss and does not impact the derivation of DIMP-Loss.

Reply to Q2

Synthetic data includes the label for the data. For Q(x) ~ P(x) on line 171, does that means Q(x, y) ~ P(x, y)? Seems the paper mix Q(x, y), Q(y|x) eg at line 942 too. Line 999 missing ', so it should be x_{p'}, right?

Thank you for pointing out the potential confusion in our notation. The assumption $Q(x) \approx. P(x)$ pertains solely to the input distribution, not the joint distribution. It is possible that $Q(x, y) = Q(x)Q(y|x) \neq P(x)P(y|x) = P(x, y)$ due to $P(y|x) \neq Q(y|x)$. In line 942, $Q(x, y)$ refers to the joint distribution, and the subsequent reference to $Q(x)$ isolates the input distribution since $Q(x, y) = Q(y|x)Q(x)$, and $Q(y|x)$ is eliminated in the denominator of the weight. The expectation remains over the joint distribution $Q(x, y)$. We will refine our notation in the next version to make this clearer, ensuring $Q$ and $P$ consistently denote joint distributions, and revising $H(P)$ to $H_P(Y|X)$ in Section 3.2 to explicitly indicate conditional entropy.

Lastly, you are correct about the typographical issue in line 999. The term should indeed be $x_{p'}$, and we have already updated this in our edits. Thank you for your careful reading and valuable feedback.

Reply to Q3

The table 1 shows Focal-loss and Hu et al.'s method did not work better than CE-Loss, esp for the "large real world" case, could you comment on the reason?

Focal Loss prioritizes more diverse or difficult data points, which can be beneficial in certain out-of-distribution scenarios, such as handling imbalanced labels. However, it does not explicitly account for the quality of the data, which may result in inadvertently prioritizing noisy or mislabelled data points during training, reducing its effectiveness compared to CE-Loss in such cases.

Additionally, in our opinion, Hu et al.'s method can be viewed as a bilevel optimization approach [1] that adjusts data weights while training the model. However, in practice, it often exhibits unstable behaviour and is highly sensitive to hyperparameters [2]. In our experiments, we used the code provided by the authors of the paper and tested several hyperparameter configurations, including the default settings. The best results from these experiments are reported in the table. To provide a more comprehensive comparison, we conducted additional experiments on another meta-learning-based method, SunGen [3], as shown in Table 1 in this revision. The results demonstrate that while SunGen is able to surpass CE-Loss in most cases, its performance remains slightly lower than that of IMP-Loss and DIMP-Loss. Furthermore, as shown in Table 3, SunGen requires approximately seven times the computational time compared to our methods.

Reference

[1] Can (Sam) Chen, et al., Gradient-based Bi-level Optimization for Deep Learning: A Survey, 2023
[2] Minyoung Kim, et al., A Stochastic Approach to Bi-Level Optimization for Hyperparameter Optimization and Meta Learning, 2024
[3] Gao et al., Self-Guided Noise-Free Data Generation for Efficient Zero-Shot Learning, ICLR 2023.

Reply to Question 4

For the DIMP-Loss at line 274, if we apply it to large real data, where P' is also trained on the same large real data. In this case, the weighting will converge to 1, is that correct? In that case, the final performance at convergence should be the same as CE-Loss, right? How does this case affect table 1?

Indeed, in large real-world scenarios, if the training dataset and the dataset used for the quality checker are sufficiently large, the weights from both DIMP-Loss and IMP-Loss will converge to 1. In this case, the performance at convergence would align with CE-Loss. However, in our experiments, the training dataset size was around 3000 samples, which may have a distribution gap compared to the real-world distribution. In our view, the quality checker and diversity checker play crucial roles in mitigating overfitting on the training data. By focusing on minimizing real-world loss rather than just training loss.

Reply to Weakness 1

The three benchmarks can be expanded to more NLP tasks. Eg nn the ZEROGEN paper cited here there are several benchmarks: sQuAD, QNLI and AdversarialQA, which are particularly challenging for synthetic data generated from LLM. The performance was far below supervised results. It will be interesting to apply the weighting method to see if the gap can be reduced for these challenging benchmarks.

We appreciate you noticed our methods had the potentials to apply on other NLP tasks, but we need to collect more synthetic data from LLMs for these challenging benchmarks; therefore, we focused on text-classification task due to limited budget.

Reply to Weakness 2

The paper used 5 epochs for training the downstream models by Line 1101. It is unclear whether the models have converged. It is interesting to not only see weighting proposed here can speed up learning, but what happens at convergence.

Thank you for your insightful observation. In Appendix E, we tried to compare the computational time, so the numbers of epoch were fixed to 5. We also tested the performance in more epochs. For example, in our experimental setup of Table 1 (Appendix F), we initially tested 5 and 7 epochs, reporting the best accuracy run in Table 1.

To address concerns about convergence, we conducted additional experiments with 10 epochs for the tweet benchmark and present the training dynamics. As shown in the plot from anonymous link, the results align with those in paper's Figure 1. Both IMP-Loss and DIMP-Loss exhibit low variations by the end of training, with testing accuracy converging to higher values than the baselines, even when the total numbers of epoch are doubled.

Additionally:

• It is worth noting that convergence occurs relatively late due to the use of a learning rate scheduler and warm-up.
• In our experiments, we consistently used a small real-world dataset as a validation set to select the best checkpoint. This demonstrates that our method aligns well with standard model selection practices and remains practical for common scenarios.