Less is More: Adaptive Coverage for Synthetic Training Data

Thank you for your comprehensive review of our work. We greatly appreciate all the feedback, most of which we have addressed and elaborated on here or in the general comment to all reviewers + revised pdf. Let us know if we have addressed your concerns and, if so, we kindly encourage you to raise your score accordingly.

Comparison to Baselines:

As noted in the general comment to reviewers, we additionally implemented the downsampling method based on an LLM judge as used in the AlpaGasus through their open-source codes. We demonstrate that our method outperforms this method. Further, note that this prior work only compares against a random downsampling baseline. Several of the other baselines suggested gave comparable results and, for the case of active learning in particular, we ran a rather small-scale experiment on the MNIST dataset. Per your suggestion, we include an experiment in which we compare the performance of a neural net model with two hidden layers trained on a subset of the data selected by ACS, Margin Sampling [1], and at random. For margin sampling, we run the model after each step of training on the entire training datasets, calculate the difference between the two top predicted labels, sort the data according to the margin values, and select the top k data points with the highest margin. Our experiments show a clear advantage of ACS over active learning with Margin Sampling, even though ACS is model agnostic and does not require running inference on the training data at all. Further plots depicting these results are included in the appendix of the updated submission and, if the reviewer feels they are insightful, then we are happy to incorporate them into the final version of our paper.

Table 1. Model Accuracy for Different Sampling Algorithms

Issues with writing:

We appreciate the feedback on the writing of our results. We encourage the reader to see the attached pdf which incorporates the noted changes and furthermore improves upon the language (especially pertaining to the experimental section) to hopefully better emphasize the importance and novelty of our results. In particular, we have included a discussion of the noted additional literature–we thank you for pointing out these papers.

Questions:

The synthetic samples were drawn from the prior work as noted for best comparison to this work. This was done to ensure that our prompting generation does not give any advantage to the ACS technique. Quality checking of these samples is inherently conducted by ACS and is fundamentally the problem at hand with selecting “good samples”. The idea is that you can generate a corpus of N data samples freely using an LLM, and subsequently do a quality check to obtain a much smaller subset of training samples. This is well captured by the diversity metric experiments included in the appendix of the updated pdf (and discussed in the general comment to all reviewers).

We emphasize that our downsampling method can be used to filter any such dataset to remove highly redundant or detracting samples (as demonstrated by our experiments on the MNIST dataset, included to showcase the generalizability of our findings across modalities). We here motivate the problem on synthetic data as this quality assurance to select meaningful samples is a highly pressing issue due to the noted bias and repetitive nature of this generation strategy.

On the point of randomization being taken over 25 or 5 trials, we reiterate that at the end of Section 3 the discussion is on randomization for the BERT model seeds and ordering of data. So, N = 5 different random seeds for each results in 25 random pairings total.

Regarding the minimum similarity threshold, we intentionally fix this value because a low-similarity edge between two data points (e.g., a similarity of 0.2) does not meaningfully contribute to coverage and risks misleading results. Our fixed threshold of 0.7 (corresponding to a 45-degree angle) ensures that only edges with significant similarity are considered, improving the quality of coverage.

[1] Zhou, Jin, and Shiliang Sun. "Improved margin sampling for active learning." Pattern Recognition: 6th Chinese Conference, CCPR 2014, Changsha, China, November 17-19, 2014. Proceedings, Part I 6. Springer Berlin Heidelberg, 2014.

We thank the reviewer for their feedback and noting the importance of the research problem at hand. For discussion of the AlpaGasus baseline and improvements on the writing, see the general comment to reviewers and updated pdf of our paper. Additionally, we thank the reviewer for noting the need for experiments quantifying the diversity of our subsampling–these are now provided in the appendix of our updated pdf (and discussed in the general comment to all reviewers). We here address the other noted concerns. Please let us know whether we have addressed your concerns. We kindly encourage you to raise your score if your concerns were sufficiently addressed.

Ablation Study:

To address this point, we refer you to the plots illustrating the effect of coverage on model accuracy for SST and MNIST in Section 4 of the paper, which demonstrate the robustness of our approach. While further exploration of the effect of the maximum outdegree parameter could be insightful, we note that this parameter is set deterministically and automatically based on the desired k and the dataset size, ensuring consistency. This choice aligns with principles like the pigeonhole argument, which supports its validity. Similarly, regarding the minimum similarity threshold, we intentionally fix this value because a low-similarity edge between two data points (e.g., a similarity of 0.2) does not meaningfully contribute to coverage and risks misleading results. Our fixed threshold of 0.7 (corresponding to a 45-degree angle) ensures that only edges with significant similarity are considered, improving the quality of coverage. We hope this clarifies our design choices and provides sufficient context for their justification.

Tuning Hyperparameters for Downstream Tasks:

The similarity threshold is automatically selected by ACS for the given coverage level and via a binary search procedure as noted in the paper. The maximum degree is bounded by the dataset size and desired value, k, while coverage values can be optimized for based on the task. We show that a coverage level below 1.0 (in this case, 0.9) is sufficient to see improved results. Moreover, in our experimentation while we see strong performance without the degree cap, enforcing it further enhances the overall performance. Although this value could be optimized for each dataset, we set a simple rule based on the number of requested samples and data size to deterministically calculate and enforce the maximum degree of the nodes and avoid using it as a hyperparameter in ACS.

Complexity Scaling:

As noted in our response to Reviewer NUow, “locality sensitive hashing” (LSH) allows the graph construction to remain efficient at scale. Since this is the rate limiting step of our algorithmic procedure, the ACS approach scales to large datasets with ease (assuming proper implementation of SOTA graph building like LSH). Furthermore, there are several other time consuming steps, such as generation for the synthetic data and fine-tuning the model, that could hide the extra computation needed to build the graph.

We thank you for your comments. We have included small scale ablation tests of the two constraints in the appendix of our revised pdf in the hopes that we can further emphasize the interplay of these constraints on the overall model performance. As you will see, the cap on node degree and similarity threshold around 0.707 similarity (as measured by cosine similarity) are highly useful for ensuring strong model performance across $k$ values.

With respect to the computational cost, we emphasize that the optimal threshold for ACS is determined via binary search, significantly reducing computational overhead. Importantly, the graph construction is performed only once for the minimum similarity threshold. In subsequent binary search steps, the graph is trimmed—a relatively inexpensive operation with a time complexity of O(|E|). Additionally, we refer to established works such as Distributed Coverage Maximization via Sketching and Parallel Set Cover and Hypergraph Matching via Uniform Random Sampling, which illustrate scalable and fast implementations of set cover procedures. These insights demonstrate that our approach remains computationally efficient, even for large datasets.

If these points + revisions help address your concerns, then we encourage you to increase your score accordingly!

We sincerely thank the reviewer for their careful reading of our submission and for highlighting that our method is “of great originality” for resolving this important problem. We here address the noted weaknesses and questions and believe that these revisions have significantly strengthened the paper. Moreover, we refer the reviewer to our general comment and updated pdf for more revisions. We kindly request that the reviewer reconsider their scores in light of these improvements and let us know whether we have addressed their concerns or not.

The first noted weakness on how to determine the value of k is an excellent point but somewhat removed from the research question we aimed to address here. Specifically, we aimed to show that small subsamples of the data can be isolated which provide meaningful improvements beyond using the full corpus of synthetic data which is known to be biased and redundant. Additionally, though we do not address the question of what an optimal k value is for downstream tasks, our method can be readily applied to find such a value. For example, one could run ACS at increasing values of k and monitor the model's accuracy on a hold-out validation dataset to identify the optimal sized subsampling.

With respect to the issues with the figures, we thank you for pointing this out and refer you to the updated pdf attached to our general response to reviewers where we have rectified the issues.

On your question: as noted in Section 4.2 we discover that a coverage value less than 1 yields optimal performance, and as such we proceed through the experiments in Section 5 with a coverage level of 0.9. We note that this coverage value could be further optimized for each specific task, but we here make the design choice of using the parameter at a fixed value across the experiments.

We thank the reviewer for their careful reading of our paper and for noting the importance of selecting LLM-generated samples. We refer you to our general comment to all reviewers to address some of the concerns and here focus on the more specific points raised. Let us know if we have addressed your concerns and if so, we kindly encourage you to raise your score accordingly.

Computational Cost:

We highlight with modern graph building methods such as “locality sensitive hashing” (LSH), the construction of the similarity graph is significantly more efficient than a quadratic complexity [1,2,3]. In practice, graphs of the size used in our paper (and much larger with billions of nodes) are frequently constructed in this manner efficiently [1,2]. Moreover, we emphasize that we do not claim to make training faster. Rather, we demonstrate a “less is more” approach wherein just generating samples freely will not give a good set of synthetic training data. Using our methods, we can distill a large (noisy / redundant) set of data into a more manageable subset of diverse samples.

“Optimal” Subset:

On this point, we argue that one of the core advantages of the ACS procedure is its model-agnostic nature. This makes ACS particularly useful in addressing the “cold-start” problem, a common limitation in other techniques. By not requiring this model feedback, ACS ensures flexibility and applicability across different settings without reliance on an initial model or precomputed embeddings. Moreover, ACS is a submodular sampling technique that can order the data in diminishing order of value for any such model.

Furthermore, ACS can complement the model-feedback-based approaches. For example, users can employ ACS as a first-pass sampling technique and then refine their training datasets using active learning techniques. To address these comments, we additionally present results on the MNIST dataset contrasting ACS with a well-established active learning technique: margin sampling [4]. Results demonstrating that ACS continues to outperform are presented in the table below, with plots added to the appendix of the updated pdf for reference as well.

Specific Implications for Synthetic Data:

On this point, we first refer the reviewer to our updated pdf containing an appendix on the diversity of our sampling technique as measured by the SelfBLEU metric. Our approach mitigates overfitting to overrepresented data by selecting a (diverse) representative subset. While this does not fully resolve the issue and close the gap to human data performance, it highlights the potential of our method to address this gap in data distributions.

Moreover, our experimentation shows that ACS indeed mitigates the imbalanced data problem. Per your suggestions, we conducted a simple experiment on the MNIST dataset where we removed 75% of images for one digit (discriminated digit, 5) to artificially create an imbalanced dataset. Then, we trained a simple neural net classifier with two hidden layers. The table below depicts the results of training such a model on a subset of data with different sizes and selected either randomly or by ACS. We further included a section in the appendix of the updated pdf which includes plots for more easy visualization of the results. The table shows the accuracy of the trained model on classifying the discriminated digit (digit 5). As shown by the results, ACS achieves an accuracy of over 50% by sampling 150 images, while the model trained on random samples completely fails to classify the discriminated digit. ACS deterministically sampled 6 of the most most representative examples of the discriminated digit, while random sampling sampled only one, with an expected number of 3-4 samples in general. Note that the labels were hidden from both sampling strategies. We included the experimental results below, and further plots in the updated pdf. If the reviewer thinks these results are relevant, we are happy to incorporate them into the final version of our submission.

[1] Chen et al. "Scalable Graph Representation Learning via Locality-Sensitive Hashing."

[2] Dasgupta et al. "Fast locality-sensitive hashing."

[3] Jafar et al. "A survey on locality sensitive hashing algorithms and their applications."

[4] Zhou et al. "Improved margin sampling for active learning."

We sincerely thank the reviewers for their valuable feedback and engaging discussion, which greatly improved the quality of our paper. In response, we conducted several additional experiments, such as adding additional baselines and diversity quantifying experiments, and provided clarifications to better highlight the novelty and broader applicability of our approach. These updates demonstrate the robustness and importance of our contributions. We hope the revisions address the concerns raised and encourage reviewers to consider raising their scores. Thank you again for your time and thoughtful engagement.