$\textbf{Only-IF}$: Revealing the Decisive Effect of Instruction Diversity on Generalization

1. The authors argue that using diverse instructions to train models can enhance performance. However, as illustrated in Table 1, the performance of the configuration with a Base Budget of 60, +OSS(14), +Alpaca(0.5), and +CoT(0.5) is lower than that of the configuration with a Base Budget of 60, +OSS(14), +Alpaca(1), and +CoT(0). Similarly, the performance of the configuration with a Base Budget of 60, +OSS(0), +Alpaca(7.5), and +CoT(7.5) is lower than that of the configuration with a Base Budget of 60, +OSS(0), +Alpaca(15), and +CoT(0). These findings do not align with the authors' argument.

2. The results of combining all three different domains (e.g., +UI(6), +OO(6), +AL(4)) are missing. Exploring this setting would provide interesting insights.

As an empirical study, the authors tries to make a point about the importance of data diversification. I have no doubt about the point, but would like to point out the following:

1. The viewpoint is not new - if we check foundation models's tech reports, we can already observe that their instruction tuning data is very diverse. Then, what's the additional insight?
2. Given that, what I expect is not to prove the correctness of the claim, but a more systematic and clear recipe to collect and curate such diverse data. For example, for generalist models, how much data points we need in the instruction tuning set, what will be the recommended (if not the optimal) distribution of the instruction tuning data, including domain, length, use case, persona, etc. Unfortunately, I still have no idea after reading the paper.
3. Although the string-based toy experiments are interesting, can we really make any conclusion out of them? For example, in 2.2, the author train GPT-2 models, then how to make sure that any observation scales with larger model sizes and newer architecture family?
4. Even for more realistic settings, I would expect a more comprehensive experiments with larger LLMs beyond 7B - a natural questions is, for larger models trained on (possibly) larger and more diverse corpus, does the diversity of instruction tuning data matter as much?

Finally, I would like to justify my points a bit: as an empirical study paper with a strong claim, the reviewer naturally wants to see clear and strong evidence to support the claim. I am not asking for experimenting with all types of LLMs, which is apparently not possible. But at least, more model sizes and settings should be considered before making such a general conclusion. However, the story will be very different if there were any theoretical backups.

• The setting is weird here: Why should we isolate the IF from the reasoning? On my understanding, these are two joint capabilites, and instruction following can be part of the reasoning. Moreover, IF is a general capability, which is supposed to include many other abilities such as coding, math, etc.
• I have a strong concerns on the section 2.2, where they train a GPT-2 on some easy tasks for real LLMs, e.g., LLaMA-3.1-70B-Instruct. GPT-2 is a pre-LLM-era model and sft it does not reflect anything. I suggest the author using the LLaMA-70B models as the baseline and finetune these models to see if it can have the similar results/conclusions, which make more sense since the ultimate goal of this paper is to have a clear guideline for improving data quality/highlighting the critical role of diversity for future LLM sft/designs.
• For the section 4.2, they conduct some experiments on coding, which yields very similar findings with the previous works [1, 2]. I don't think the contributions they claim is strong here. I am expecting a more interesting/impressive findings.
• Overall speaking, the paper does not meet the bar of ICLR.

[1] Bukharin, A. and Zhao, T., 2023. Data diversity matters for robust instruction tuning. arXiv preprint arXiv:2311.14736.

[2] Zhang, X., Chen, Z.Z., Ye, X., Yang, X., Chen, L., Wang, W.Y. and Petzold, L.R., 2024. Unveiling the Impact of Coding Data Instruction Fine-Tuning on Large Language Models Reasoning. arXiv preprint arXiv:2405.20535.

• The link between the research gap and the experimental findings is unclear. The authors discuss the “decomposition of reasoning and knowledge retrieval in LLMs” in Line 51, but then shift to emphasizing data diversity for specialists and generalists in Lines 80-88. How do these concepts directly connect? The introduction would benefit from reorganization to clarify this relationship.
• The paper’s structure is difficult to follow. Providing a brief overview of the experimental setup at the end of the introduction could improve readability and coherence.
• The layout of content and figures is confusing. For instance, Figure 1 appears on Page 1 but isn’t referenced until Page 8. It would help to either highlight the main insights from Figure 1 in the introduction or improve figure placement.
• While the results on specialists and generalists are insightful, more quantitative analysis would strengthen the paper. For example, how much general data is needed to enhance specialist models effectively for a given task?
• In Section 5, the authors claim that greater domain diversity enhances generalists. However, it’s unclear how they controlled for the impact of domain scaling versus data size. How can these two factors be disentangled?