Balancing Label Quantity and Quality for Scalable Elicitation

• The paper appears to have been submitted in an unfinished state
  • Different microeconomic assumptions are mentioned as part of the contributions and discussion, but I did not find any discussion of these in the main text (and very little in the appendix).
  • There is a range of minor issues that are unproblematic on their own, but add up:
    • The paper uses the wrong template (ICLR 2024 rather than 2025)
    • The paper containe multiple instances of questions in odd places that appear to be todos by the authors. For example, "All datasets?" at the end of the references, and "What about scenarios where the cost you invested into training up your labelers or understanding a problem has externalities for other training runs etc?" in Appendix B.
    • A bunch of citations are missing details like the arxiv identifier
    • Figures 1 and 2 are presentented in double column without any apparent reason, making things more difficult to read.
  • The logical flow of the writing could be improved at times. For example: "the role of pretraining in weak-to-strong generalization" is mentiond before weak-to-strong generalization is introduced, making the remark difficult to understand.
  • There is also room for improvement in terms of "local" writing clarity. For example:
    • "Results broken down by each of the three datasets can be found in Appendix figures 6, 7, and 8, suggesting that the results hold across tasks and weak label qualities."
      • What result is supposed to "hold" here? I can see that the pareto-fronts have somewhat similar shapes, but what does that imply?
    • The description of the quantity-dominant setting is difficult to parse (perhaps in part due to the dual-column structure).
• Apart from the paragraph on scalable oversight, the related work section is relatively sparse and could benefit from incorporating existing work on sample values [1] and quality/quantity tradeoffs in machine learning [2,3,4].
• While this might be hindsight bias speaking, I do not find the main result, that a mixture of higher quality and lower quality labels is optimal, particularly surprising. For example, while [3] is not fully comparable, in part because high quality labels are selected actively, that work also finds a mixture of higher and lower quality labels to be optimal for training.

[1] Torralba, Antonio, and Alexei A. Efros. "Unbiased look at dataset bias." CVPR 2011. IEEE, 2011.

[2] Sheng, Victor S., Foster Provost, and Panagiotis G. Ipeirotis. "Get another label? improving data quality and data mining using multiple, noisy labelers." Proceedings of the 14th ACM SIGKDD international conference on Knowledge discovery and data mining. 2008.

[3] Chen, D., Yu, Z., and Bowman, S. R. Clean or annotate: How to spend a limited data collection budget. arXiv preprint arXiv:2110.08355, 2021.

[4] Crammer, Koby, Michael Kearns, and Jennifer Wortman. "Learning from data of variable quality." Advances in Neural Information Processing Systems 18 (2005).

My biggest concernis that the insights of the paper is not very clear. As it is true that there is a trade-off between quantity and quality, it is unclear how the specific finding in this paper can be generalizable to tasks beyond NLP binary classifications showcased in the paper. In this case, it might make sense to come up with a scaling law that captures this tradeoff.

While I appreciate the efforts and empirical results by the paper, I feel that this paper looks more like an engineering technical results for the following reasons:

1. Scientific rigor:

(a) Please include error bars on all reported plots. If such errors are not available, please provide discussions on the statistical significance of the reported observations.

(b) The experimental observations can be made more precise. For example, while the paper claims that "$256-$512" is the mixed regime in Figure 1, this appears very task dependent in Figure 2.

(c) The discussion on limitations can be more in depth. For example, a distinction can be made on low/high-quality data provided by human vs. smaller LMs (as in the experiments). Does the paper specifically study training data provided by LMs? Does this generalize to low/high-quality data provided by people or not? Also in the experiments, the smaller LMs are trained on other data ("we generate weak labels using small LMs that have been finetuned on the task"), which confounds the conclusions made in the paper about low/high-quality data in the experiments.

2. Unsubstantiated contributions:

(a) The second claimed contribution includes "microeconomic assumptions", but as far as I can tell, this is not investigated in the paper. Appendix B appears unfinished.

(b) I also think the third claimed contribution is very limited. While research framing can be a valid contribution, in this case, the problem of data usage and training methods is a well-known research topic and extensively studied in the past. The Pareto frontier is essentially a piece of empirical result that provides some practical evaluation and guidance on what training methods work better in what settings.

2. Lack of context:

I find the background knowledge introduced in this paper insufficient for audience beyond LLM practitioners. For example, what are the tasks "BoolQ, HellaSwag, SciQ, Cola, ..."? What's "LoRA" for supervised finetuning? What does it mean to "early-stop based on validation AUROC"? Why are "Qwen 1.5-0.5B and Qwen1.5-4B" chosen for the initial set of experiments? What's a new "head" for training? What's the procedure for "few-shot prompting" and what are "in-context examples"?

3. Insufficient literature review:

The literature review only provides citations on very recent advances specifically tied to LLM development. I personally find it insufficient without giving credits or contextualizing the research question with respect to other lines and domains of work. Two of such domains are:

(a) The field of crowdsourcing primarily concerns how to make use of noisy human data, which is closely relevant to the paper.

(b) The field of economics computation literature concerns data valuation and pricing, which is also closely relevant to the paper.

4. Clarity (minor):

(a) In the result section, the paper mentions "quality of weak labels". What does this mean? I thought weak labels mean low quality in this paper.

(b) I suggest the paper using different terms than "quantity-dominant" and "mixed". These terms are confusing because the quantity-dominant regime cannot afford enough weak labels, and the mixed regime can afford enough weak labels.

(c) The wording "scaling results" in Section 4.1 is vague. What results does this refer to?