Cost-Effective Synthetic Data Generation for Post-Training using QWICK

• This paper may be challenging for readers unfamiliar with reinforcement learning, including multi-armed bandits. The authors could consider presenting the methods in a more straightforward manner for clarity.
• The authors only test each dataset on models from the same series. Since models from different series could have comparable costs, how would selecting models from different origins impact the results?
• The prompt to generate synthetic data is missing, whether baselines are the same with QWICK.

• Dependency on Initial Model Pool: The algorithm's performance is tied to the quality of the initial pool of models, and it does not account for model improvement over time, which may limit its effectiveness in environments with evolving model capabilities.
• Constrained scenario: I assume the authors consider the scenario where the constructer of the dataset uses only API calls. This scenario is kind of narrow, since most of the companies and researchers may have their own GPU resources, I wonder this approach is still applicable under the assumption that some GPU is accessible, so that inference can be done locally.
• What happens to the data generated in previous iterations: I wonder are the samples generated in iterations prior to the termination still kept in the generated dataset? Or are they simply abandoned? If it is the latter, wouldn't it cause waste of computation?

• There are some missing comparisons including comparison to always using the best and worst models as well as comparisons with real data. it was unclear to me if this is the red dotted line in the figures as this line was not included in the legend. Having the results for each model included the figure would also help show how much better the selection is at various amounts of compute versus only using the 2B, 9B, or 27B models respectively.
• It is unclear how strong the models have to be as even the Gemma-2-2B models are already quite powerful, and the effectiveness of the algorithm is contingent on there being a large diversity in the models. Can the authors include some ablation with a larger variance of models? This also goes in hand with the previous comment on showing the performance of each model individually.
• One limitation of the proposed approach is that the proposed approach is applied to tasks where the answer is easy to verify. For other tasks (summarization or more complex instruction following), it may be hard to extend the proposed approach as knowing whether a model produces the correct answer is more difficult.
• The experiments only evaluate on train/test with the same data. It would be interesting to know if the synthetic data has other impact on the evaluation particularly downstream robustness (e.g. train on MATH and test on GSM).

The key weaknesses of the paper can be grouped into the following broad themes:

Methodological Limitations

1. Sparse Data and Inefficiency: QWICK tracks raw rewards and costs for each (question, LLM) pair individually, using average reward and cost estimates per pair for model selection. This approach has limitations, as empirical data required for reward and cost estimation can become sparse, particularly as the number of questions grows, the LLM pool expands, or the budget decreases. Furthermore, the algorithm is inefficient, as it estimates reward and cost for each question in isolation without leveraging similarities across questions. A more effective approach might involve using dense representations of questions and training reward and cost models for each LLM.

Inadequate Experimental Validation

The paper's experimental design lacks critical baseline comparisons, raising questions about the benefits of the proposed approach:

2. Lack of Individual Model Baselines: Current comparisons focus only on model selection approaches. Benchmarking against baselines that generate synthetic data by sampling from individual models in the pool (e.g., the weakest, strongest, etc.) within budget constraints would clarify whether model selection truly adds value or if a single model could achieve comparable performance.

3. Question-Wise UCB1 Comparison: The paper only benchmarks against a dataset-level UCB1 approach, where UCB1 selects a single model for the entire dataset. However, UCB1 could also be adapted for query-level model selection by using the cost term from Algorithm 1 solely to determine when to stop generating rather than to select between models. A comparison of QWICK with this question-level UCB1 algorithm would provide a fairer assessment of the benefits of incorporating cost into the model selection process.

4. Baselines Incorporating Model Performance Priors: LLMs often come with prior information about their general or task-specific performance, which can inform model selection by using this information as a reward prior. To evaluate QWICK more thoroughly, it should be compared against simple baselines that leverage this performance data. For instance, rather than sampling models uniformly at random (as done in the Random baseline), models could be sampled from a weighted distribution that reflects their relative performance, whether general or task-specific.

5. Missing details: Key experimental details are missing in the paper, such as:

   • Diversity and Coverage Metrics: Although these metrics are referenced throughout the paper, they are not formally defined, leaving their specific calculation and interpretation unclear.
   • Budget Interpretation: The paper employs normalized budgets to compare methods but lacks a clear, interpretable context for these values. For instance, it is unclear whether a normalized budget of 1 represents the resources needed for a single pass over the entire dataset by the weakest LLM, the strongest LLM, all LLMs combined, or multiple passes with these models. Without this grounding, it’s challenging to gauge the flexibility or limitations of a given budget setting, and to determine which additional baselines would be relevant for comparison.

Unclear Broader Impact

The broader impact and applicability of the paper’s approach are unclear, as it is evaluated within a narrow context: generating Chain-of-Thought (CoT) responses from a limited pool of LLMs for fine-tuning. This limited scope raises questions about the approach's generalizability and whether QWICK is the optimal method for synthetic data generation under resource constraints.

6. Specifically, the paper lacks comparison to alternative methodologies that could also offer an efficient use of the data generation budget, such as:

• Advanced Inference Techniques: More sophisticated (and costlier) methods, like Monte Carlo Tree Search (MCTS) or self-refinement, could be used to generate higher-quality responses rather than relying on standard CoT sampling.
• Focusing on Challenging Tasks: Instead of generating numerous correct answers for simpler tasks, prioritizing accurate solutions for more challenging tasks could make better use of resources, especially when combined with advanced inference techniques.
• Training with Other Preference Learning Approaches: Preference learning methods like Direct Preference Optimization (DPO) can utilize incorrect samples, which QWICK currently discards. Comparing QWICK’s performance when training with DPO rather than Supervised Fine-tuning (SFT) on its generated data, alongside appropriate DPO baselines, would provide a more thorough evaluation of its cost-effectiveness.

Comparison with these alternative approaches would offer a clearer view of QWICK’s relative strengths and limitations, contributing more substantially to understanding best cost-effective synthetic data generation practices for fine-tuning LLMs.

Ablation and Scaling Experiments

The paper lacks ablation experiments on various components of the algorithm, and its scalability remains uncertain.

7. Question-Level Rewards Ablation: It would be insightful to ablate the question-level rewards in QWICK, which can get sparse, and rely solely on dataset-level rewards for model selection. This comparison would clarify the necessity of question-level reward tracking.
8. Dataset-Level Rewards Ablation: Similarly, removing dataset-level rewards and evaluating its impact on overall performance would provide useful insights.
9. Scaling the Number of Models: The current experiments limit QWICK to selecting among a maximum of three models from the same family. Testing QWICK’s scalability with a larger and more diverse model pool could reveal how it handles sparser rewards and increased costs, and may also help identify an optimal range for the number of candidate models.
10. Scaling the Dataset Size: Evaluating QWICK’s performance on larger datasets could yield valuable insights as well, as this would also lead to sparser rewards and increased costs.