Adapt-$\infty$: Scalable Continual Multimodal Instruction Tuning via Dynamic Data Selection

Dear Reviewer eQ3c,

Thank you for reviewing our work, please see our response to the weaknesses and questions below:

Novelty: We politely emphasize that our paper introduces three novel components in the research field.

1. Introducing the practical and realistic continual learning scenarios for MLLM instructional tuning: Lifelong Instruction Tuning (LiIT) (Lines 52-53 and Lines 72-75) with clarification of four critical challenges.
2. A new multi-way data selection approach for LiIT that can aggregate the benefits of various scoring functions based on the task.
3. A new scoring function i.e., image grounding score, that can distinguish visually-grounded data samples from the not-grounded samples.

The main novelty of our work lies in our proposed lifelong instruction tuning scenario where the model is trained on a multi-task instruction tuning at each time step. This is in stark contrast to the conventional continual learning setup explored in most existing works and is a more practical variation of lifelong instruction tuning based on frequent data releases as seen in the research community today. This scenario brings on unique challenges that have not been tackled in previous continual learning works such as overlapping tasks, redundant data, rare tasks, etc. Our experiments range from frequently used baselines such as multi-task learning, random selection, and scoring-based selection to more sophisticated methods based on high-dimensional representations, yielding insightful results on the nature of lifelong learning in LLMs. Our proposed method is based on systematic insights from these experiments; further, the two-step process of task-based selection and semantic deduplication has not been explored in previous works. Most importantly, we show significant improvements with our method and establish a strong baseline for the scenario of lifelong instruction tuning.

At the same time, we respectfully note that other reviewers (9s1q, bvXZ, VSGx) agree with the novelty and significance of our proposed scenario. For instance, Reviewer 9s1q stated, “...the paper reflects the authors' deep understanding of the field and direction, providing readers with extensive knowledge and unique insights, ..., opening up new avenues for future research.” Additionally, these reviewers recognized that the proposed method appropriately and logically addresses well-structured challenges in lifelong multimodal instruction tuning scenarios, emphasizing the significance and uniqueness of the proposed approach compared to prior methods.

The authors hope that the reviewer eQ3c understands our significance and practicality (often considered another ‘novelty’ from a scientific perspective) in formulating and addressing lifelong multimodal instruction tuning for the real world.

Evaluation on Visual Chat: Thank you for the great suggestion, we have included examples of visual chat in our paper for our models (Figure 6, 7, and 8). Since there is no formal evaluation benchmark or quantitative metrics for multi-turn dialogue with MLLMs, we used a single representative example for qualitative analysis of visual chat results in the updated draft.

Figure 6 shows the multi-chat skills of the LLaVA model at $t$=0. In Figure 7, we note that the models trained using sequential learning (left) and random pruning (right) have poor retention of the multi-chat skill and reply with few-word and/or inaccurate answers. However, our method (results in Figure 8) retains this skill because it pools the multi-chat training data samples into a distinct cluster during the pseudo-task clustering step and selects sufficient samples from this cluster for training at each time step.

Efficiency Analysis: Thank you for the great suggestion, we have added this to the paper. We present a comparison of the time taken by various data selection methods for our experimental setting (for training with 25k samples at each time step) in Table 6 in the Appendix (see revised pdf). Results are presented for 8 A100 GPUs. The total time taken to train the model without any methodical data selection (i.e., random pruning) is approximately 21 hours. Scoring-based selection methods generally require a forward pass of the model to extract the score (e.g., EL2N, entropy), which takes nearly 48 hours on 8A100 GPUs for all datasets in our experiments. Methods that use feature embeddings (e.g., COINCIDE) takes a similar amount of time since they uses a forward pass to extract hidden layer outputs from the models as data representations. Our proposed method requires longer time i.e., 92 hours, to perform a backward pass over the model and extract gradients, similar to LESS - which is a state-of-the-art method for data selection for targeted instruction tuning. However, with Lite-Lamp, we are able to significantly reduce this time due to systematic compression of the dataset at each timestep.

Dear Reviewer bvXZ,

Thank you for reviewing our work and appreciating the rigorous thoughts and experimental analysis that went into the paper. Please see our response to the weaknesses below:

LAMP for LLMs: Thank you for the suggestion, we agree that the data selection method designed in our paper can be extended to LLMs as well, and will be exciting future work. During the rebuttal period, we have decided to add experiments using LAMP on LLMs only. One thing to note is that LLMs generally undergo multiple post-training phases after instruction tuning, such as alignment and other fine-tuning stages. On the other hand, our method and the proposed lifelong instruction tuning scenario focus on the instruction tuning phase, and it might be important to understand how data selection interacts with these subsequent post-training phases, which will be highly practical and exciting future research direction in lifelong learning of LLMs with data selection.

Our experimental setup is as follows:

Model: LLaMA3-8B

Datasets in the order of training:

• Natural Instructions [~1600 NLP tasks]
• Alpaca CoT [geared towards Chain-of-Thought ability]
• xP3 [Multilingual]
• UltraChat [Multi-turn dialogues]
• Firefly [Chinese only]

Since training and evaluation takes substantial time, we plan to report our results on this experimental setup by this Friday.

t-SNE Seed: We have used the t-SNE visualizations to only understand the spatial distribution of feature vectors from different sources. We repeated the t-SNE visualization experiment across 10 seeds and did not find a significant difference in the quality of clusters across these seeds for any of the feature vector sources; the results reported in our paper are stable across seeds. The figures reported in the paper are generated from the same seed for different datasets for fair comparison.

We have results from our runs with LAMP and LLaMA3-8B with the datasets enumerated in our earlier response i.e., Natural Instructions, Alpaca CoT, xP3, UltraChat, and Firefly. We use the set of evaluation benchmarks used in [1] to evaluate our models (MMLU, GSM, BBH, TydiQA, CodexEval, AlpacaEval) and compute the following metrics based on performance across these benchmarks: Average accuracy, Relative gain, and Forgetting rate (see Sec. 5.1 in our paper). The results at each timestep are as follows (numbers indicate average accuracy):

Based on these results, the relative gain and forgetting rate are as follows:

The model is trained on 25k samples at each time step, similar to the MLLM experiments in our paper. We implemented the Lite-Lamp version of our method for these experiments. We see up to 18% forgetting rate in the sequential setting, which comes down to 5.6% and 2.3% with random pruning and LAMP pruning. The average accuracy is highest with our method at the final time step. These results suggest that:

(a) lifelong learning from multi-task instruction tuning datasets is a significant problem in the language-only domain as well and

(b) our proposed method can significantly alleviate forgetting of skills over time, as well as preserve the maximum accuracy that can be achieved from each dataset.

[1] How Far Can Camels Go? Exploring the State of Instruction Tuning on Open Resources

Dear Reviewer bvXZ,

Thank you for your effort in reviewing our paper. We kindly notify you that the end of the discussion stage is approaching. Could you please read our responses to check if your concerns are clearly addressed? During the rebuttal period, we made every effort to address your concerns faithfully:

• We have additionally demonstrated strong generalizability of the proposed LAMP in the language domain.
• We have clarified the reviewer's question regarding the relevance between t-SNE stochasticity and model performance.

Thank you for your time and effort in reviewing our paper and for your constructive feedback, which has significantly contributed to improving our work. We hope the added clarifications and the revised submission address your concerns and kindly request to further reconsider the rating/scoring. We are happy to provide further details or results if needed.

Warm Regards,
Authors

Dear Reviewer VSGx,

Thank you for reviewing our paper and recognizing the merits of our work. Please see our response to the weaknesses below:

Sequential Learning: We wholeheartedly agree that sequential learning is not a practical approach since most LLMs and MLLMs rely on multi-task instruction tuning from a massive dataset. Our work is motivated by this very fact, and hence, in contrast to most continual learning works that train the model on a single task at each time step, we train our model on multi-task datasets at each timestep. We consider the scenario where multiple multi-task instruction tuning datasets are available for training and there is significant overlap between tasks as well as some rare tasks that appear in one or more datasets only. This scenario is more practical because we see frequent releases of new multi-task instruction tuning datasets in the research community. Our experimental setup and proposed method enable the learning of new tasks and reinforcing as well as forward transfer of old tasks without having to retrain the model from scratch. We have emphasized and expanded these points in the revised draft (see the blue text in the Introduction).

Computational Complexity of Data Selection: Random pruning is a rather effective method when it comes to data selection, as reported in many prominent works in the data selection literature. For instance, [4] reports that selecting 5% instruction tuning data via random pruning results in performance that is very close to that with 100% data; their proposed method i.e., gradient-based similarity results in a few points improvement over random. [5] show that it is hard to beat random pruning in very high pruning settings for the ImageNet dataset. When the availability of computational resources is low, random pruning is indeed a promising tradeoff for gain vs. compute. Nevertheless, data selection methods, including LAMP, are worth investigating for several reasons:

• Selecting representative and important data samples can accelerate training [6, 7]. Besides, it is important from the perspective of scientific research as to why one data subset works better than another subset, selected randomly or otherwise. Developing data selection methods that are guided by concrete and intuitive hypotheses is the best way to understand this science.

• Data selection methods either require importance scores or high-dimensional representations of data samples in order to represent the data landscape correctly and incur computational costs in this process. Similarly, LAMP uses high-dimensional representations for selecting data correctly and as we show in Table 2, exceeds the performance of random pruning by >5%, yielding important insights about data selection for lifelong learning. More importantly, LAMP promotes the forward transfer of skills during lifelong learning (109% relative gain) whereas random selection falls short (95.3% relative gain). This suggests that random selection is sub-optimal in practical settings where tasks can be unbalanced, redundant, or rare. Such scenarios need more sophisticated techniques for effective learning.

• We adopt several techniques that significantly reduce the computational complexity of our method such as

  (i) random projections for reducing memory footprint of high-dimensional vectors,

  (ii) MiniBatch k-Means for clustering of large datasets, and

  (iii) systematic compression of datasets at each time step using deduplication (as outlined in Lite-Lamp, Section 4.4).

LAMP can be made further optimized using libraries like faiss that quantize the high-dimensional space for fast nearest neighbor computations. Moreover, methods like LAMP will continue to reap benefits from faster inference algorithms. Thus, we think that LAMP is a computationally efficient approach to practical data selection for lifelong learning.

LAMP vs. Lite-LAMP: The relative performance drops due to the reduction in budget size. Larger data budget enables more diversity in the data that is selected for training the model in the next step whereas a reduction in data size leads to lesser diversity and smaller improvements. It is notable that in spite of the significantly lower data budget in Lite-LAMP, our method mitigates catastrophic forgetting at all timesteps and preserves ~100% relative gain.

Dear Reviewer VSGx,

We sincerely appreciate your efforts in reviewing our paper and your constructive comments. Since there are less than two days left in the discussion period, could you please read our responses to check if your concerns are clearly addressed? We believe that our responses resolved all your concerns.

We understand that the criteria for rating a paper can sometimes be subjective; however, if you agree that our work does not have remaining major concerns, we would like to kindly suggest your re-evaluation of the initial rating of this submission.

Please let us know if you have any remaining questions, and we will be more than happy to address them.

Best,
Authors

Dear Reviewer 9s1q,

Thank you for recognizing the merits of our work. Please see our response to the weaknesses below:

Figures: Thank you for the suggestions for making the figures more understandable. We have updated the explanations for subfigures A and C in Figure 2 and improved the text font in all of our figures for clarity in the updated version of our paper (see revision).

Computational costs of clustering in LAMP: We note that our clustering approach incurs only marginal additional computational cost and does not require significant resources. Naive k-means clustering has a computational complexity of $O(tknd)$ where $t$ = iterations of the algorithm, $k$ = number of clusters, $n$=number of samples, $d$=dimension of vectors, implying that the computation costs rise linearly with each of these factors.

• The value $t$ is constant in our experiments.
• The variables $k$ and $d$ are small in our experiments (see Lines 354-358). The optimal values of $k$ are between 5 and 50, and the value of $d$ is capped at 8192 by virtue of using random projections to compress high-dimensional vectors (Lines 877-881 in Appendix). The value of $k$ can increase with the size of the datasets, however, there exist many efficient methods for calculating pairwise cosine similarities effectively. We use the efficient implementation of MiniBatch-KMeans in scikit-learn to compute clusters for our datasets, which significantly reduces the compute time.
• The maximum time taken for a single k-means clustering run in our LAMP experiments using this implementation is 48 minutes on a 48-core CPU for approximately 1.8 million samples. In addition, the faiss library can also be used for efficient computation of pairwise cosine similarities using quantization.
• Further, our proposed approach Lite-Lamp seeks to reduce the number of samples $n$ at each step, effectively reducing the time taken for k-means clustering. The maximum time taken for a single k-means clustering run in Lite-LAMP experiments is 15 minutes on the 48-core CPU for approximately 660K samples, which makes it as efficient as other data selection methods like LESS [1] and COINCIDE [2].

Since there are many ways to control each of the variables that contribute to the complexity of k-means clustering ($k$, $n$ and $d$), we think that LAMP is not a prohibitively computationally expensive method.

[1] Xia, Mengzhou, et al. "Less: Selecting influential data for targeted instruction tuning." arXiv preprint arXiv:2402.04333 (2024).

[2] Lee, Jaewoo, Boyang Li, and Sung Ju Hwang. "Concept-skill Transferability-based Data Selection for Large Vision-Language Models." arXiv preprint arXiv:2406.10995 (2024).

Combinatorial Prediction of Function Tools for Sample Selection: Each function tool assigns a different value to every sample in our dataset. Since each functional tool has its unique range and distribution, we normalize these values to enable comparison of entropies across these tools. We appreciate the suggestion for combining these distributions to enable further gains for our method. We believe this idea presents an exciting direction for future work, potentially improving the robustness and efficiency of data selection in future implementations, but also introduces additional challenges that need to be addressed:

1. determining top-$k$ or dynamic number of beneficial function tools for each data pair, and
2. controlling the relative influence of predictions.

The second point would be crucial as different tools may have varying degrees of relevancy and reliability for different samples, and an imbalance could affect the robustness of the selection process.

In summary, the authors appreciate the reviewer for raising this constructive discussion and believe that exploring these challenges, as well as developing methods to effectively combine the outputs of multiple functional tools, will be a meaningful direction for future work.

Dear Reviewer 9s1q,

Thank you for your effort in reviewing our paper. We kindly notify you that the end of the discussion stage is approaching. Could you please read our responses to check if your concerns are clearly addressed? During the rebuttal period, we made every effort to address your concerns faithfully:

• We have updated the explanations for subfigures A and C in Figure 2 and improved the text font in all of our figures
• We provide the details regarding the computational costs of clustering in LAMP. We would like to note that the proposed approach is as efficient as other data selection methods like LESS [1] and COINCIDE [2].
• We further provide a discussion regarding the Combinatorial Prediction of Function Tools for Sample Selection, which should be a promising research direction that entails two distinct and meaningful research challenges.

Thank you for your time and effort in reviewing our paper and for your constructive feedback, which has significantly contributed to improving our work. We hope the added clarifications and the revised submission address your concerns and kindly request to further reconsider the rating/scoring. We are happy to provide further details or results if needed.

Warm Regards,
Authors

Dear Reviewer 9s1q,

We sincerely appreciate your efforts in reviewing our paper and your constructive comments. Since there are less than two days left in the discussion period, could you please read our responses to check if your concerns are clearly addressed? We believe that our responses resolved all your concerns.

We understand that the criteria for rating a paper can sometimes be subjective; however, if you agree that our work does not have remaining major concerns, we would like to kindly suggest your re-evaluation of the initial rating of this submission.

Please let us know if you have any remaining questions, and we will be more than happy to address them.

Best,
Authors