Mosaic-IT: Free Compositional Data Augmentation Improves Instruction Tuning

Weakness:

Q1: The core problem of the proposed method is the lack of a detailed explanation of the core reasons for the proposed method. It does not provide justifiable and experimental explanations for the effectiveness of the proposed method. The primary motivation of the proposed method should be further cleared here.

Please kindly refer to the Q1 of the General response.

Q2: From the experiments, it seems that the proposed method does not improve the multi-turn data for MT-Bench. Would there be any explanations for this?

1. Mosaic-IT does not necessarily improve the second-round dialogue of LLMs if the instruction data are single-round conversations. Our method composites several instructions into one but it is still under the setting of single round conversations.
2. The second-round performances are affected by both the instruction data and the LLMs to be trained. As shown in Table 3, when more advanced data and LLMs are used, the second-round performances can be further improved.

More discussion will be included in the later version.

Q3.1: Many instruction-following methods and literatures focus on data augmentation. There are no comparisons with those baselines.

Most existing instruction-following methods and literature focus on data augmentation utilizing other LLMs to generate new data samples, which is a data synthesis process relying on other LLMs. However, our method is a model-free method that aims to exploit the potential of existing data, which is largely different from existing methods. To our best knowledge, our method is the first of this kind. Please let us know if you find any works of this kind. In more detail:

1. Our method avoids the entire cost of data synthesis by any language models or neural networks, which still produce carbon footprints.
2. Our method aims to exploit the potential of existing data so the quality of augmented data is 100% guaranteed. In contrast, the newly generated data still require careful and complex quality checks in practice to reduce the risk of degrading the original LLM (i.e., negative finetuning).
3. Mosaic-IT is orthogonal and complementary to existing data synthesis approaches. It can be applied together with other methods to further improve LLMs.
4. As a model-free method, Mosaic-IT can avoid the potential violation of licenses limiting the usage of existing LLMs.

Q3.2: In addition, how would mask methods be different from the other dropout, etc. methods?

There might be some misunderstanding of the maskout strategy, which is applied to the training data to create data augmentations with different outputs. This is different from dropout-typed approaches that are applied to neurons or parameters of a model, whose goal is to perturb and regularize the neural network training. But it does not change the output in the training data.

By maskout in Mosaic-IT, we instruct LLMs to ignore some of the instructions and when generating the responses and train LLMs to follow such meta-instructions. We will consider changing the name of this strategy to avoid any misunderstanding.

Dear Reviewer 7WNG,

As we are approaching the deadline of the discussion period, we would like to cordially inquire about the extent to which we have successfully addressed the concerns outlined in your review. Should there be any lingering points that require further attention, please rest assured that we are enthusiastic about the opportunity to provide comprehensive responses to any subsequent queries or comments you may have.

Your constructive input remains invaluable to us, and we appreciate your dedication to enhancing the quality of our manuscript. Thank you for your time and consideration.

Best,

Authors

Thanks for the prompt response! We are still confused about how to apply image augmentations MixMatch or MixUp for representation learning to text data for autoregressive language modeling (in particular, finetuning of modern LLMs such as Llama, which is our focused problem). To the best of our knowledge, whether and how these methods can be applied to LLM finetuning is still unknown as both models and the training objectives are very different. We are eager to hear from you how these methods can be leveraged for the LLM finetuning. Thanks!

Weakness:

Q1: The paper lacks a theoretical basis for why random concatenation should improve instruction-following abilities; structured or semantically grouped concatenations could offer further insights.

Please kindly refer to the Q1 of the General response.

Q2: Randomly concatenated instructions may introduce noise, potentially impacting training stability. An analysis of this effect on model perplexity would strengthen the work.

Thank you for your insightful suggestions!

We examined the training stability by checking the training curves of loss on different LLMs. Similar to the curves in Figure 4, the starting loss of our method is typically higher compared to the baseline training curves due to the difficulty of the compositional data. This indicates the weakness of existing LLMs in following these compositional instructions, possibly due to the noises or the interference among multiple instructions. However, during the training phase, the loss declines consistently and stably, which indicates that the LLM makes progress on generating each response selectively or in a predefined order according to the corresponding instruction and meta-instruction, without being affected by the interference or noise from other instructions. More discussion will be included in our later version.

Dear Reviewer cPom,

As we are approaching the deadline of the discussion period, we would like to cordially inquire about the extent to which we have successfully addressed the concerns outlined in your review. Should there be any lingering points that require further attention, please rest assured that we are enthusiastic about the opportunity to provide comprehensive responses to any subsequent queries or comments you may have.

Your constructive input remains invaluable to us, and we appreciate your dedication to enhancing the quality of our manuscript. Thank you for your time and consideration.

Best,

Authors

Weakness:

Q1: The techniques to prevent over-fitting and memorization by preparing various formatted templates and order randomization has been well studied and widely known approach; such as pioneering work of Instruction Tuning (Wei et al., 2021, Flan-T5 paper).

This paper is indeed an excellent pioneering work of Instruction Tuning, which greatly influences us. However, with all due respect, we disagree with your viewpoints on our paper and the Flan-T5 paper.

1. There is a misunderstanding of the novelty and contribution of our work. Our main contribution is a cost-free compositional data augmentation method, which concatenates existing instruction-tuning samples into complex ones. We did not utilize different templates for training. To our best knowledge, we are the first to propose this type of method for improving the instruction-following capability of LLMs, please let us know if you find any reference applying similar ideas to LLMs.
2. The Flan-T5 work does not utilize or mention any idea of concatenating multiple instructions into one to make it more complex, which is our main contribution and motivation.
3. The instruction tuning setting of modern LLM in this paper is different from the one used in the Flan-T5 era. Flan-T5 needs to cover various tasks and curate a lot of samples per task, thus requiring different formats of templates to prevent overfitting. However, in the modern instruction tuning settings, every instruction is regarded as a different task and it does not require many samples per task, so the diverse templates are no longer required. Our work is based on the modern LLM instruction tuning setting.

Please kindly let us know if you have any feedback.

Q2.1: Figure 3 is unclear to me. Could you clarify what is a “mixture count”?

The mixture count is the number of original samples/instructions to be composited in Mosaic-IT augmentations. For example, in Figure 2, the mixture counts for the examples are all 3. We will further clarify it for more times in the paper.

Q2.2: While “Fix” strategy is adopted, the number of “mixture count” seems to be distributed among 1-10 (not fixed?).

When “Fix” is applied, we fix the number of samples-to-be-composited to 10. However, some composited samples might exceed the maximum token limit for the instruction tuning. In this case, we decrease the number of samples to reduce the length below the limit. Otherwise, these augmentations need to be abandoned and some original samples are ruled out from the training. We mentioned this process in line 249, we will further modify the narratives to make it clearer.

Q2.3: Why do you use Uniform as a default despite its not the best performance?

1. Uniform distribution is not the best but it does not require search or tuning of the distribution and it already brings non-trivial improvement to the LLM. Trying more advanced distributions is optional but requires extra cost. We present the performance of other distributions to show the potential additional improvement that can be achieved by further tuning of distributions.
2. Uniform distribution by default keeps the method simple and the ablation studies clearer to understand, e.g., the ablation for the max number of instructions in Table 4 and the time reduction in Table 7.

Q3: In Figure 4, we can see that Mosaic-iT accelerates its training, but the performance at the convergence seems to be the same or even worse than the baselines, which is contradictory to your main results that improve the performance against baselines. Could you clarify the relationship between the convergence performance and the logic of performance improvement.

Thank you for your comment, but we disagree on using losses (especially training losses) to evaluate the performance of instruction tuning.

1. The evaluation of instruction-following capability of LLMs is an open challenge in the community. Perplexity-based loss is not a reliable metric for evaluating the capability achieved by instruction tuning.
2. The data in the baseline method are entirely different from ours: our data is the concatenation of the baseline’s data, which is much more challenging for LLMs to learn. Thus their loss values are not comparable.

Q4.1: In Table 4 (a) (ablation of Mosaic-IT), you tried Format, Permute, Maskout, and Permute/Maskedout. Why didn’t you try all the combinations?

We evaluated the mix of Permute/Maskout strategies by applying them to different samples, as illustrated in line 381. We did not apply all strategies together to one sample in order to avoid potential ambiguities of the meta-instructions and misinterpretations of LLMs.

For example, for a Mosaic-IT instruction “Ins1, Ins2, Ins3, Ins4, Ins5”, if we combine Permute and Maskout together, the meta-instruction can be “Respond to the instructions with the order of [3,2,1,5,4]. Ignore the 2nd and 3rd instructions.” Then there might be a misunderstanding on which instructions to ignore (the 2nd and 3rd of the original order or the permuted order?)

More discussions will be included in our paper.

Q4.2 Also, your best performance came from Maskout, but the adopted variant for Table 1 seems Permute/Maskedout. Why didn’t you use Maskout only?

Training LLMs to follow instructions in a predefined order is more challenging than ignoring some instructions. However, this skill might be more important than maskout as it is common to instruct LLMs to execute codes/instructions in a predefined order in practical applications.

Dear Reviewer mjBt,

As we are approaching the deadline of the discussion period, we would like to cordially inquire about the extent to which we have successfully addressed the concerns outlined in your review. Should there be any lingering points that require further attention, please rest assured that we are enthusiastic about the opportunity to provide comprehensive responses to any subsequent queries or comments you may have.

Your constructive input remains invaluable to us, and we appreciate your dedication to enhancing the quality of our manuscript. Thank you for your time and consideration.

Best,

Authors

Thank you authors for the detailed response and additional experiments. Also, I really sorry for the late reply due to my personal matter.

I realized that my misunderstanding of the main focus of this paper. This paper aims to improve the capability of instruction-following under multiple rules, rather than general LLM's capability. I agree with the effectiveness of compositional combination of instructions to improve the instruction-following performance, and based on it I raised the score.

My remaining concern is the evaluation. I don't think Alpaca Eval 2, MT-Bench, and Huggingface Open LLM Leaderboard are the suitable test bed to measure the capability of instruction-following, such as "Respond in reverse of the original order", "Ignore the longest one/several task(s) according to the word count.", "Enclose each reply with [START] and [END].", etc. While this method trains LLMs with those instructions, the capability of instruction-following with such instructions is not directly measured. I think the strong experimental performance in the relevant benchmarks does not appropriately support the claim on instruction-following capability.

In addition, for the Figure 4, the author said,

Thank you for your comment, but we disagree on using losses (especially training losses) to evaluate the performance of instruction tuning.

The evaluation of instruction-following capability of LLMs is an open challenge in the community. Perplexity-based loss is not a reliable metric for evaluating the capability achieved by instruction tuning.
The data in the baseline method are entirely different from ours: our data is the concatenation of the baseline’s data, which is much more challenging for LLMs to learn. Thus their loss values are not comparable.

If this is true, why Figure 4 is presented in the paper? or it might better to remove it from the paper? The faster loss decrease does not related to the performance improvement. I think because we cannot compare the loss, the statement on the training efficiency does not make sense. The y-axis should be the performance on the instruction-following tasks strictly evaluated with whether the given instruction is satisfied or not.

Q2: The absolute performance on Alpaca2-LC seems too low.

Thank you for reading our paper in such detail!

For finetuning on Vicuna 1M data, we randomly selected 300K data for the training due to the computation budgets, considering the diverse data quality of Vicuna 1M, (which contains conversations of other languages and some dummy conversations like “Hello!” “Hello!”), thus we think the main cause of the low performance is the random selection. Thus, we further finetuned Llama-3-8B with all the 1M data to see how the performance goes, the results are shown below, which is much better:

For finetuning on Magepie data, we also used the filtered 300K data. All the training settings are kept the same as reported in the paper, except for the maximum sequence length, which we set to 4096 compared with their 8192 for the computation budget. We further finetuned Llama-3-8B with the 8192 length to see how the performance goes, the results are shown below, which is slightly better:

Despite the absolute performance variances between models finetuned by us and the reported performances, the comparison between the baseline and our method is totally under the same setting to ensure a fair comparison, which we believe is solid for verifying the effectiveness of our methods.

Q3: Table 2 suggests that baseline methods have better 2-round MT-Bench scores than Mosaic-IT. Shouldn’t the second round MT-Bench scores improve with Mosaic-IT augmentation?

1. Mosaic-IT does not necessarily improve the second-round dialogue of LLMs if the instruction data are single-round conversations. Our method composites several instructions into one but it is still under the setting of single round conversations.
2. The second-round performances are affected by both the instruction data and the LLMs to be trained. As shown in Table 3, when more advanced data and LLMs are used, the second-round performances can be further improved.

More discussion will be included in the later version.

[1] LIMA: Less Is More for Alignment. (NeurIPS’23) [2] AlpaGasus: Training a Better Alpaca with Fewer Data. (ICLR’24) [3] From quantity to quality: Boosting llm performance with self-guided data selection for instruction tuning. (NAACL’24) [4] Rethinking Data Selection at Scale: Random Selection is Almost All You Need.

Q1.1: Motivation: how much of instruction tuning data acquisition is a bottleneck? There are several papers that show that a small number of instruction tuning data is enough to enable strong instruction-following capabilities.

This is still an open problem in the community. There are papers supporting that “a small number of instruction tuning data is enough for instruction-following capabilities”, including LIMA [1], Alpagasus [2], Cherry LLM [3], etc. However, there is also a very recent paper [4] mentioning data filtering is not effective for large-scale data. We will include this discussion in our later version.

However, the discussion of this problem is out of the main scope of our paper, as we do not filter or discard any given data. Our motivation and method are orthogonal to data filtering and data synthetic methods. We aim to achieve cost-free data augmentations to further exploit existing instruction datasets without data filtering. The main difference and advantage of our method is that our data augmentation does not require any other models and is cost-free.

Q1.2 With the rise of powerful small language models (e.g., 4o-mini, Gemini-Flash, Haiku), getting a lot of instruction tuning data is not a bottleneck in terms of resources.

Powerful small language models can reduce the generation cost of instruction tuning data more easily, however:

1. Our method avoids the entire cost of data synthesis by any language models or neural networks. In contrast, small language models (SLMs) still produce concerning carbon footprints. Their lower cost per sample may come with a price of more trials. For example, to achieve superalignment (SLM-generated data improves LLMs), strategies like best-of-n are often needed but significantly increase the cost.
2. Our method aims to exploit the potential of existing data so the quality of augmented data is 100% guaranteed. In contrast, the newly generated data by SLMs still require careful and complex quality checks in practice to reduce the risk of degrading the original LLM (i.e., negative finetuning).
3. Mosaic-IT is orthogonal and complementary to existing data filtering and data synthesis approaches. It can be applied together with other methods to further improve LLMs.
4. As a model-free method, Mosaic-IT can avoid the potential violation of licenses limiting the usage of existing LLMs.

Q1.3: I do not understand the connection between instruction tuning and Dense and Aligned Captions paper from the VL literature. The authors should rethink the motivation in the introduction.

Thank you for your insightful suggestion! We used the idea of “dense alignment” from the “Dense and Aligned Captions” paper to motivate the importance and challenges of generating multiple responses aligning with the corresponding instructions in the input according to the meta-instruction in Mosaic-IT. In the cited paper, learning with dense captions helps the model to achieve the capability of aligning each part of the image with corresponding descriptive captions. This is similar to the dense alignment between multiple instructions and responses in the compositional data by Mosaic-IT. We will modify the motivation to make it clearer.

Q1.4: It is unclear whether this strategy scales with data i.e., having more Mosaic-IT data beneficial or not.

Our experiments include datasets with various data sizes including 50k, 70k, 300k, and 1M (results shown in Q2). Training with Mosaic-IT data achieves consistently better performances across these datasets.

Dear Reviewer w54h,

As we are approaching the deadline of the discussion period, we would like to cordially inquire about the extent to which we have successfully addressed the concerns outlined in your review. Should there be any lingering points that require further attention, please rest assured that we are enthusiastic about the opportunity to provide comprehensive responses to any subsequent queries or comments you may have.

Your constructive input remains invaluable to us, and we appreciate your dedication to enhancing the quality of our manuscript. Thank you for your time and consideration.

Best,

Authors