Dear Reviewer zLAH,

Thank you for your valuable review. We respond to each comment as follows and sincerely hope that our response can properly address your concerns.

Figures and Tables can be found in zLAH.md in https://anonymous.4open.science/r/ICML25-EVIC-D5E8

Methods And Evaluation Criteria

M1: Do you find that the interactions eventually stabilize and evolve less? How does the time to stabilization compare with the model convergence time?

Res: We do not observe the stability of the interaction matrix. Taking Llama-3.1-8B and Mistral-7B-v0.3 as examples, the Frobenius norm of the interaction matrix after the warm-up phase, 1 iteration, 2 iterations, and 3 iterations is as follows.

• Llama-3.1-8B: 9e12 -> 5e12 -> 6e12 -> 4e12
• Mistral-7B-v0.3: 9e12 -> 2e12 -> 4e12 -> 2e12

M2: How asymmetric are the interactions? Is there any chance that the asymmetry is just due to random noise? Do you have any other hypothesis as to why these interactions are asymmetric?

Res: Taking Llama-3.1-8B and Mistral-7B-v0.3 as examples, their interaction matrices consistently have more than 99.7% of sample pairs $(i,j)$ satisfying $Int(i,j) \neq Int(j,i)$. We further analyze the proportion of pairs $(i,j)$ with opposite influence directions during training, i.e., those satisfying $sign(Int(i,j)) \neq sign(Int(j,i))$. The percentages after the warm-up phase, after 1 iteration, 2 iterations, and 3 iterations are as follows.

• Llama-3.1-8B: 19.3% -> 17.9% -> 16.7% -> 17.4%
• Mistral-7B-v0.3: 22.1% -> 21.5% -> 21.8% -> 21.1%

The asymmetry of the interaction matrix arises from the difference between each sample's original gradient $Grad$ and Adam gradient $Adam$. Since $Int(j,i) = \langle Adam(j), Grad(i) \rangle$, $Int(i,j) = \langle Adam(i), Grad(j)\rangle$, we naturally have $Int(j,i) \neq Int(i,j)$.

M3: (1) Have you tried EVIC with initial portions other than 5%? (2) Is EVIC's performance highly dependent on which samples are initially selected?

Res: (1) Yes, please see Figure 3 in Section 4.3.

(2) The standard EVIC randomly samples 5% of the dataset for warm-up. To investigate the impact of warm-up samples, we have added additional experiments only using samples from code and general domains for warm-up, denoted as CodeWarmUp and GeneralWarmUp. The results are shown in Tables zLAH-1 and zLAH-2 in the anonymous link. As shown, unbalanced warm-up sample distribution will reduce model performance.

M4: It would be interesting to see a histogram of how often samples are chosen as helpful during training. For instance, a bar for samples being never chosen, a bar for samples being chosen once/twice, etc.

Res: We have added figures in the anonymous link following your suggestion.

M5: Will EVIC negatively impact the performance of model on the rarer domain when training on domains with datasets of varing sizes?

Res: Imbalanced datasets can indeed hinder the learning of rarer domains. Fine-tuning on such imbalanced datasets is another important challenge in multi-domain finetuning [1,2,3]. To enhance the learning of rare domains, we can weight the interaction matrix computation in EVIC based on the sample size of each domain, increasing the probability of selecting samples from rare domains.

Extending EVIC to imbalanced datasets is a new direction, and we will continue to explore this in future work. If you are interested, we would be glad to discuss this topic further.

Experimental Designs Or Analyses

E1 There are a strong number of baselines and related works included in the evaluation and the evaluation appears to be well-designed by including tasks of varying sizes. However, the evaluation could be expanded and strengthened by including more models beyond GPT-4 and additional domains/datasets.

Res: Thanks for the positive feedback on our experimental design.

• Regarding the models: Our experiments use three of the most popular and widely-used base models---Mistral-7B-v0.3, Llama-3.1-8B, and Qwen2.5-14B.
• Regarding the domains/tasks: Since DMT is experience-baesd and cannot be directly applied to other domains (as the authors of DMT do not provide relevant experience), we follow DMT's setting using mixed dataset containing code, math, and general domains to ensure fairness. If you have any other domains or datasets of interest, we would be happy to hear from you through a rebuttal response. While our computational resources are limited, we will make every effort to conduct additional experiments to follow your suggestions.

Other Strengths And Weaknesses

W1 The "firness" in Line 252 should be "fairness".

Res: Thanks for pointing this out. We will correct this typo.

---

[1] Mixture-of-Skills: Learning to Optimize Data Usage for Fine-Tuning Large Language Models

[2] Massively Multilingual Neural Machine Translation in the Wild: Findings and Challenges

[3] Unsupervised Cross-Lingual Representation Learning at Scale

Dear Reviewer JcLd,

Thank you for your valuable review. We respond to each comment as follows and sincerely hope that our response can properly address your concerns.

Figures and Tables can be found in JcLd.md in https://anonymous.4open.science/r/ICML25-EVIC-D5E8

Other Strengths And Weaknesses

W1: It would be poignant to clarify the contribution of this paper is not to derive the influence computation using Adam gradients.

Res: We will accordingly modify our presentation.

W2: Often times they observe marginal improvement and sometimes no improvement (within the std dev). For example, in the Qwen and Mistral experiments.

Res: We confidently yet humbly believe EVIC significantly outperforms existing methods for the following reasons:

1. Our focus is on the overall (AVG) performance improvement in multi-domain fine-tuning. EVIC outperforms the second-best method in terms of AVG by 4.1, 0.8, and 1.0 on Mistral, Llama, and Qwen, respectively. We believe this improvement is significant, as even a 0.5 gain on a multi-domain datasets with inter-sample conflicts is challenging.

2. We respectfully speculate that you may find EVIC's improvement in some single domains marginal. However, please note that EVIC consistently ranks first or second across all domains, which is also a remarkable and challenging achievement.

W3: Table 3 does not provide a holistic picture of how the evaluation metrics are evolving over training across different methods of multi-domain finetuning. Could you actually provide a more informative version of Table 3 where we can visualize the evaluation metrics over the training?

Res: Due to the deletion of intermediate results and checkpoints for some baselines, we need to retrain them. With limited computational resources, the experiment is still in progress. We will complete it before the discussion deadline and provide more informative tables or figure as suggested. We would greatly appreciate your understanding.

Other Comments Or Suggestions

S1: Can this method be adopted to calculate the influence between test samples and training samples, and use the scores to select training samples?

Res: The approach of calculating the influence between test and training samples is orthogonal to and compatible with our method. However, it faces challenges in multi-domain fine-tuning of LLMs. When test samples conflict (e.g., the evaluation is across different domains), even positively influencing training samples may still conflict with each other, hindering the performance. This brings us back to the core challenge of "how to conduct multi-domain fine-tuning."

S2: How are the standard deviations in Table 2 calculated? How many runs of the experiments were done?

Res: We provide relevant information in Lines 254-264 (left column) of our initial submission but acknowledge it lacks detail. We will revise it as follows.

Specifically, all training is conducted once using the same random seed due to limited resources. For inference:

• HumanEval: We run model inference with a temperature of 0.3, using random seeds from 1 to 10. Then, we report the mean and standard deviation (std dev) of Pass@1 and Pass@10 using Numpy.
• GSM8K-test: We use greedy decoding with a temperature of 0.0 (so there is no std dev) and report accuracies.
• AlpacaEval 2.0: We run inference with a temperature of 0.7, using random seeds from 1 to 10. We use the AlpacaEval library, with GPT-4 as the judge, to compare model outputs against GPT-4 Turbo outputs, and report the mean and std dev of (length-controlled) win rate using Numpy.

S3: How expensive is computing the influence at each iteration of EVIC?

Res: The cost of calculating influence (interactions) in each iteration is mainly dominated by the gradient computation for all samples, which is rougthly equivalent to performing one epoch of MTL. We humbly believe the additional computational cost of the interaction matrix is acceptable and worthwhile. For more details, please see our response to C2 of Reviewer q9CU due to the rebuttal length constraints.

Questions For Authors

Q1: Please see W3.

Q2: How does the frequency or sample selection size of each iteration change the dynamics of EVIC?

Res: The higher the frequency and the fewer samples learned per iteration, the more accurate the gradient computation and interaction matrix estimation become, leading to better EVIC performance. However, excessively high frequency would result in numerous full-dataset gradient computations and extremely high costs. Therefore, in our initial submission, we simply set the frequency and sample selection size as follows.

• Sample selection size: All samples with a non-negative row sum in the interaction matrix are selected, with no limit on the number.
• Iteration frequency: After all samples selected in the previous iteration have been learned, the process moves on to the next iteration.

Dear Reviewer q9CU,

Thank you for your valuable review. We respond to each comment as follows and sincerely hope that our response can properly address your concerns.

Tables can be found in q9CU.md in https://anonymous.4open.science/r/ICML25-EVIC-D5E8

Claims And Evidence

C1: There is no curriculum learning (CL) baseline in the comparison. It is imporatant to compare EVIC with some basic CL methods.

Res: Apologies for the lack of clarity in our paper. The baselines in our initial submission (DMT and MoS) belong to progressive CL and balanced CL methods as categorized in [1,2]. We will refine this in the paper.

Additionally, we have added vanilla CL and self-paced CL as baselines. Thus, our experiments have covered four CL method categories. Tables q9CU-1 and q9CU-2 in the anonymous link show that EVIC, DMT, and MoS significantly outperform other CL baselines, underscoring the importance of CL methods tailored for multi-domain fine-tuning.

C2: EVIC seems to be computationally expensive due to the gradient computation for every sample when computing the interaction matrix.

Res: We humbly believe the additional computational cost of the interaction matrix is acceptable and worthwhile. Additionally, to reduce costs and improve efficiency, we provide some ideas on implementations and methodological improvements.

1. EVIC’s gradient computation and total cost are less than twice that of MTL, which is accpetable and worthwhile for real-world applications. In multi-domain LLM fine-tuning, performance bottlenecks are more challenging than efficiency bottlenecks, as extending training time often fails to overcome performance limits. Our proposed EVIC effectively improves the performance via iterative interaction estimation, thereby making a significant contribution.

2. However, the efficiency and cost concerns you mentioned are also important. For full dataset gradient computation, we parallelize computations across eight GPUs. We would like to point out that there is a trade-off between efficiency and performance. To reduce costs, we could use historical gradients of some samples without recomputing them, but this would result in inaccurate interaction estimates and thus decrease training performance. In our initial submission, we prioritize performance over efficiency, but in efficiency-critical scenarios, using historical gradients could be explored.

Methods And Evaluation

M1: Why computing interactions with "Adam" gradients? Does the momentum heavily influence the calculation in Sec. 3.1? How much of the evolution of interactions is due to the momentum term?

Res: Because the interactions are modeled as influences between samples during "training" and LLMs are usually trained with Adam optimizer, the "Adam" gradients naturally appears in the computation of interactions. In a nutshell, our interactions are defined based on Adam gradients the corresponding momentum.

We further add experiments to demonstrate the rationale behind computing interactions based on Adam gradients, where we compare EVIC with its variant that computes interactions using the inner product of the original gradients. Please see Table q9CU-3 in the anonymous link for details due to rebuttal length constraints.

Experimental Designs Or Analyses

M2: I can't find how $M$ and the total length of each iteration is determined in this paper.

Res: We provide the relevant information in Lines 248-252 (right column) in our initial submission but acknowledge it lacks detail. We will revise it as follows.

Specifically, we set the number of iterations $M$ to be as large as possible while ensuring that the total training steps of EVIC do not exceed those of baselines to maintain fairness. Thus, $M=4,3,2$ for Mistral, Llama, and Qwen experiments, respectively. As for the length of each iteration, it refers to the number of training steps required to learn all selected samples in this iteration, i.e., $|D_m|/{\rm batch size}$.

Other Comments Or Suggestions

S1: Adding some qualitative examples of how the models select examples will make it more intuitive to understand how EVIC works.

Res: Thanks for your valuable suggestion. We have added some qualitative examples in Figure q9CU-1 in the anonymous link, but we are not sure if this is exactly what you want. Could you please clarify your suggestion in more details so that we can better follow it?

S2: Definitions of LC-WR and WR should be added when first introduced.

Res: We will add the definitions based on [3]. Due to the rebuttal length constraints, we cannot provide the definitions here. We would greatly appreciate your understanding.

S3: Two repetitive references.

Res: We will make modifications accordingly.

---

[1] Curriculum Learning: A Survey.

[2] A Survey on Curriculum Learning.

[3] Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluators.