CLUES: Collaborative Private-domain High-quality Data Selection for LLMs via Training Dynamics

We thank the reviewer for the insightful and positive feedback!

W1 ans Q1:

Although the methodology is thoroughly described, the authors could improve the clarity of Figure 3 by providing a more intuitive explanation of how training dynamics enhance collaborative training processes.

Figure 3 demonstrates an optimized training approach for collaborative learning of multiple models. By selecting high-quality training data for each local model, we are selecting gradients that positively impact loss trajectories. These trimmed gradients are accumulated, leading to an improved position in the weight space. By considering the interference during our data selection (gradient selection) of $\Delta\theta'{1}$ and $\Delta\theta'{2}$, we reduce the interference of weight updates from different models. After parameter aggregation, the merged model $\Delta\theta_{merged}$ can be improved to an enhanced position in the weight space represented by $\Delta\theta_{targeted}$.

W2 and Q2:

The paper contains some grammatical errors and typos. For example, in Table 3, "Performance of Data quality" should be "Performance of data quality", and on line 290, "the data influenced calculated via the last layer weights prone to a cancellation effect" should be corrected to "the data influence calculated via the last layer weights is prone to a cancellation effect".

We appreciate your detailed suggestions on the typos! We have fixed them all and will update them in our final version.

Thank the author for providing further information and contributing great work to the community! I have read other reviewers' comments and the rebuttal information. I would like to remain my rating as 'Accept.'

W1 & Q1:

Please explain how the proposed method performs on datasets from different domains beyond the medical field.

As mentioned in s1, our method by nature can be adapted to various models ‘without the need for task-specific adjustments’. We conduct comprehensive evaluations on different medical and healthcare dataset which covers different domains in medicine. Our assumptions are not limited to any specific domain, and our selection of anchor data is not domain-specific. We chose medical data because of the availability of high-quality public datasets in this field.

We are running the experiments on the financial dataset: FiQA. We randomly sampled 2000 data samples for each of the 4 clients from FiQA dataset, and polluted each of them with a low-quality data ratio of 80%, 20%, 10%, and 50% respectively. Once we get the result, we’ll update our response.

W2 & Q2

Experiment with varying proportions of low-quality data to understand the method's effectiveness across different levels of data quality. We have been running the experiments on different proportions of low-quality data: 80% and 100%. Once we get the result, we’ll update our response.

We reported the estimated running time of our collaborative fine-tuning pipeline (for each proportion of low-quality data on a certain dataset on A40). Due to the time limits, we are still running the experiments and expected to have additional experimental results within the discussion period.

W3 & Q3

Impact of using different numbers of anchor samples on the performance and reliability of the method

We show the ablation study of the anchor data selection and validation set selection in Figure 2 in our PDF, which demonstrates the robustness and unbiased selection in our experiments. Fig 2 shows that the threshold keeps stable with the increase of the number of anchor data, thus demonstrating the robustness of the anchor data selection.

I would like to also highlight that the number of anchor data will not affect the order of the selected high-quality data, the only impact is the number of data selected.

W4 & Q4

Why our method performs better than Oracle?

In our experimental setup, the global threshold is $0.48$, and the total number of selected data is slightly more than the number of oracle data, and considering more data diversity potentially lead to better performance. We also discussed it in our future work part in our paper.

Why the other three data evaluation methods perform so poorly in Table 1?

IFD assumes that all data in the dataset is of good quality where it selects informative data samples. It cannot effectively handle scenarios with poor-quality data, where IFD tends to select complex, difficult-to-learn noise patterns. PPL tends to select longer data samples. Both IFD and PPL are not well-equipped to deal with situations where malicious or intentionally corrupted data is present in the dataset. The dataInf method is only related to the optimal theta point and does not consider whether the validation loss actually decreases. This may result in selecting bad data that could increase the loss. In contrast, our approach can better handle settings with bad data.

Additionally, DataInf makes several strong assumptions, which lead to more limitations such as:

1. the total approximation error is bounded by $O\left(\sum_{l=1}^L d_l^2\right)$, and the approximation error is tolerable only when $d$ is small.
2. The effectiveness of adapting DataInf to LoRA is limited to the number of learnable parameters is small, such as 1, 2, 4. The correlation coefficient of DataInf generally decreases as the rank increases.

These limitations highlight the challenges in developing influence measurement functions and data selection, especially when dealing with diverse, heterogeneous data, whereas our proposed methods are good at broader scenarios.

Experiment II: Varying Levels of Low-Quality Data

To evaluate the robustness of our data selection method under different data quality conditions, we conducted a series of experiments with varying proportions of low-quality data. We maintained a consistent proportion of low-quality data across all clients for each experiment, ranging from 0% to 100%, including 20%, 50%, and 80% pollution levels.

$\textcolor{blue}{Table 4}$ presents the performance of models trained with and without our data selection method across these different proportions. The results demonstrate that our method effectively enhances data quality across all scenarios with GPT-4 scoring. And in terms of accuracy of the data selection, our method consistently selected over 99% of the high-quality data across different proportions of low-quality data.

$\textcolor{blue}{Table 4:}$

To understand the adaptability of our global threshold, we analyzed how the global threshold changes with different proportions of low-quality data. $\textcolor{blue}{Table 4}$ illustrates that our global threshold adjusts across varying levels of data quality.

Limitation: The author's work is similar to federated learning. Should it be compared with the recent federated learning?

Our work has covered federated learning as one of the collaborative training paradigms, and our evaluation was conducted in a federated learning setting. For a detailed discussion of empirical results, please refer to our response to Reviewer ngAe regarding W2 & L1. If you're referring to specific recent developments in federated learning that warrant additional comparison, we'd appreciate more details and elaboration.

To summarize, these results collectively demonstrate the robustness and effectiveness of our method across a wide spectrum of data quality scenarios, reinforcing its potential for real-world applications.

We have completed all the requested experiments and are pleased to share the results.

Regarding generalizability beyond the medical domain, we conducted experiments using the FiQA dataset, which focuses on financial question answering.

Dataset: We have four clients, each with 2000 training samples. We split the data into a training set (8000 samples in total), a validation set (also called the anchor set, 100 samples), and a test set (500 samples).

Evaluation metric: Responses from the model fine-tuned on our dataset are rated by GPT-4 on a scale from 1 to 10, reflecting criteria including relevance, accuracy, and fluency. To address potential positional bias, we send our response along with the benchmark output to GPT-4 twice, with different orders. We then calculate the average of these scores as the final performance score.

Experiment I: Quality Heterogeneity

To evaluate our data selection method's effectiveness in scenarios with heterogeneous data quality, we simulated varying levels of data pollution across clients. Specifically, we randomly polluted 80%, 20%, 10%, and 50% of the training set for each of the four clients, respectively.

$\textcolor{blue}{Table 1}$ presents the results of our model merging experiments. These findings demonstrate that our method significantly enhances data quality even when clients have differing proportions of low-quality data:

$\textcolor{blue}{Table 1:}$

We conducted an ablation study on our global threshold to further validate our approach. $\textcolor{blue}{Table 2}$ illustrates the advantage of using a global threshold determined by our anchor data for data selection in this heterogeneous setting, compared to selection based on average ratio or pre-determined scores. These results demonstrate that our approach successfully balances the identification of positive cases with the minimization of false positives, offering a robust and superior solution.

$\textcolor{blue}{Table 2:}$

Additionally, we performed a qualitative analysis by manually comparing the outputs generated by models fine-tuned on our selected high-quality data versus the original low-quality data. This comparison ($\textcolor{blue}{Table 3}$) provided insights into the tangible improvements in model performance and output quality.

$\textcolor{blue}{Table 3:}$

Why the other three data evaluation methods perform so poorly in Table 1?

We have compared different data scoring methods for their accuracy in selecting high-quality data out of the mixed-quality datasets in the FiQA Experiment I setting mentioned above. Our method demonstrates significant advantages as follows.

Thanks so much for your valuable comments and feedback.

W1:

Detailed algorithm definition

Thank you for your valuable suggestions! Following the reviewer's constructive feedback, we have included our pseudo-code algorithm in the PDF.

W2 & L1:

Evaluation We appreciate your questions on the bolded numbers! We have fixed them all in our uploaded PDF.

To further clarify our evaluation setup and clear up misunderstandings: Our preliminary results (Table 1 and Table 2 in the uploaded PDF) show severe performance degradation when mixing our constructed bad data into the raw data. This demonstrates the large impact of low-quality data in collaborative settings, which motivates the importance and pressing challenges for data selection methods in such settings. Our objective is to achieve better performance compared to the low-quality baseline, not the raw data performance. Oracle serves as the theoretical upper bound.

To further highlight our empirical results:

1. For both pre-trained models and tasks, with other settings remaining the same, model merging performs better than federated learning. This indicates that loose communication between the local model and the server, compared to frequent communication, might lead to better generalization.
2. The performance boost with selected data in the federated setting is larger than in the model merging setting. This might be because during federated learning, we calculate the data score based on the global model (instead of the local model in the model merging setting) at each timestamp, which can better trace and regularize the training trajectory to the optimal location.
3. In both federated and model merging settings, our data selection can achieve over 96% and over 91% of the theoretical upper bound performance, respectively.
4. Our method outperforms the other centralized data selection baselines under the GPT4 Scoring metrics. Compared to the other methods which cause severe forgetting during instruction tuning, the performance of our method on the Knowledge-based benchmark remains within an acceptable range. This shows that our methods are able to improve domain-specific tasks without forgetting knowledge injected during pretraining.

Q1: We appreciate your detailed suggestions on the table and figure reference numbering! We have fixed them all and will update in our final version.

Q2: We split the whole dataset into the training set, the validation set (and anchor data), and the test set. They are all from the same data sources. The anchor data/validation set is in the same distribution as the training data, which is essential to ensure that the measurement of high-quality data aligns both during the selection (validation set and anchor set) and in the evaluation (test set).

Dataset construction process:

For the Medical QA task: We use 16k samples in total, with 8k samples randomly sampled from PMC-LLama and Medalpaca-flashcards each. We uniformly partition the total samples into 20 clients in this task. For the Multilingual QA task: We use 6312 samples randomly sampled from MMedBench and 1052 samples per language for each of 6 clients. (Domain Heterogeneity setting) For our newly added Financial QA task: We randomly sampled 2000 data samples for each of 4 clients from the FiQA dataset, and polluted each of them with low-quality data ratios of 80%, 20%, 10%, and 50% respectively. (Quality Heterogeneity setting)

We provided examples of raw data and low-quality data with their scores calculated by our proposed method in our uploaded PDF.

Test set:

Test data are sampled from the same data sources. For open-ended GPT4 evaluation, the test samples are randomly sampled from Medalpaca-flashcards and MMedBench for the medical QA task and multilingual QA task, respectively. For knowledge-based evaluation, the test samples are multiple-choice tasks, which use accuracy as the main metric. This additional evaluation is to see if fine-tuning will degrade or forget the knowledge obtained from pretraining. Following convention, we adopt three prominent medical benchmarks for evaluation.

Anchor data and validation data:

We followed the convention of existing data selection/valuation methods, which requires a clean validation set drawn from the target distribution. This setup has been adopted and proven to work well in many previous data selection methods, including Data Shapley and Influence Function, etc.[1] It doesn't limit the real-world applicability of the method, because there are always a few public high-quality data available; otherwise, we couldn't clearly define what high-quality data looks like.

[1] LAVA: Data Valuation without Pre-Specified Learning Algorithms, ICLR 2023.

(contd)

We express the relationship and differences between federated learning and model merging using mathematical formulas as follows:

Model merging:

let $f_\theta \in \mathcal{F}$ denote the language model and $D\_k \in \mathcal{D}$ denote the training dataset on client $k$. Given the training datasets $D_k$, we can define a model merging operator $\mathcal{M}\_K( \cdot ; D_k , k \in K=\{1, \cdots, n\}): \mathcal{F} \rightarrow \mathcal{F}$.

The model merging process can be expressed as

Federated Learning:

Based on the notation of model merging, the federated averaging process can be expressed as

where $T$ is the round number. $S_t(K)$ is the index set of selected datasets to train on each client at round $t$.

Ablation Studies:

Thank you for giving us the opportunity to clarify our approach!

1. Merging Techniques (Section 4.3): Figure 4(a) in our paper demonstrates that different weighted merging or aggregation techniques lead to varying performance. Notably, the performance of our data selection method with the Linear Merging technique doesn't even reach the performance of low-quality data with TIES merging technique, highlighting the significant impact of weighted merging techniques on overall performance. To make the comparison more clear, we presented it in the following $\textcolor{blue}{Table 1}$:

$\textcolor{blue}{Table 1:}$

2. Hit Rate and Low-Quality Data Selection (Sections 4.4 and 4.5): We apologize for the confusion. The "hit rate" in 4.4 and the "number of selected low-quality data" in 4.5 refer to the same metric: the proportion of bad data being filtered out. We presented this in two forms (percentage and data size) to provide different perspectives. A higher value indicates more effective filtering of bad data by our algorithm.

3. Additional Experiments: (Since the start of the rebuttal period, we have been conducting additional experiments to perform ablation studies. We now have comprehensive results that we believe are both convincing and robust. We are pleased to report these findings as follows)

To provide a more comprehensive ablation study, in addition to the experiments in the "domain heterogeneity" setting shown above, we conducted additional experiments in a "quality heterogeneity" setting using the FiQA dataset, which focuses on financial question answering. We randomly polluted the training sets of four clients with varying degrees of low-quality data: 80%, 20%, 10%, and 50%, respectively. The following $\textcolor{blue}{Table 2}$ illustrates the advantage of using a global threshold determined by our anchor data for data selection:

$\textcolor{blue}{Table 2:}$

These results demonstrate that our approach successfully balances the identification of positive cases with the minimization of false positives, offering a robust and superior solution. We have also conducted additional ablation studies on the global threshold, which can be found in Figure 2 of the updated manuscript.

Furthermore, we conducted the ablation study of different merging techniques on the FiQA dataset, demonstrating the importance of weighted merging, shown in $\textcolor{blue}{Table 3}$. We use pairwise evaluation here, where responses from the model fine-tuned on our dataset are rated by GPT-4 on a scale from 1 to 10, reflecting criteria including relevance, accuracy, and fluency.

$\textcolor{blue}{Table 3:}$

Evaluation:

Thank you for your suggestions on presenting the experimental results. Since we cannot directly modify the original PDF at this stage, we will implement the following changes in the updated version of the paper:

• We will bold the best-performing selection method in each column of Table 1 (Table 4 in the supplement), as you suggested. This includes bolding "Ours" in the Mistral GPT-4 and Llama GPT-4 columns, and "PPL" in the Mistral Knowledge and Llama Knowledge columns.

• We will update the table caption to read: "The best performing selection method (row) in the lower subtable is bolded for each benchmark evaluation (column)."

If you have any further questions or concerns about the experimental setup or results, we would be more than glad to continue the discussion on OpenReview.

Threshold:

We do not use dynamic thresholds, nor is there any description related to dynamic thresholds in our submitted paper. We appreciate the opportunity to clarify this misunderstanding and address the confusion: The threshold is calculated only once throughout the entire pipeline. For details and proof on determining the global threshold, please refer to our response to R1.

To clarify our method, here's a detailed breakdown of the data selection pipeline in collaborative fine-tuning, summerizing from Section 3.1 in our original paper and the pseudocode in our manuscript:

• Stage 1 (On each client): Local fine-tuning with low-quality data; save model checkpoints.
• Stage 2 (On each client): Calculate gradients, compute scores for each training sample, send scores to the server.
• Stage 3 (On server): Calculate gradients, compute scores of anchor data, determine the global threshold using anchor data scores and client scores.
• Stage 4 (On each client): Select data with scores not lower than the global threshold.
• Stage 5 (On each client): Local fine-tuning with high-quality data, then send model parameters to the server.
• Stage 6 (On server): Merge client model parameters to obtain the final global model.

Your understanding that "in expectation, grad x grad products between local training samples and local validation samples will be large for clean samples and small for corrupted samples" is correct! However, we want to clarify that we do not use "a contextual threshold based on the current loss." As mentioned earlier, the threshold is calculated only once.

Federated versus Model Merging

We integrate both federated learning and model merging into a unified algorithmic framework because they fundamentally involve local training followed by model aggregation in the parameter space. Since weights in neural networks are updated through optimization algorithms based on accumulated gradients, our gradient-based method influences the model's weight space through selective gradient use. This underlying intuition allows our method to be applicable to both collaborative learning scenarios.

The only difference between federated and model merging algorithms is: In the federated setting, there's periodic communication between the server (global model, or $\theta\_{merge}$ in our paper) and clients (local models). Clients perform local training between communication rounds. At each round, clients send their current model parameters to the server, which aggregates them to form the current global model, then sends it back to all clients. After each communication round, server and client models are aligned. In contrast, in the model merging setting, there's no periodic communication during local training; local models train independently throughout the entire process. Parameters are sent to the server only once, at the end of training, to form the final global model.

To clarify Point 2 further: In the federated setting, the checkpoints we save for each local client are based on the global model updated from the last model aggregation on the server, which typically occurs after a fixed number of local training epochs or iterations. This process incorporates information from other clients' models. Crucially, at the end of each communication round, the model parameters on the server and all participated clients are synchronized. This ensures that all participating clients start the next round from the same point, incorporating collective knowledge. This synchronization can help our data selection method better trace and regularize the training trajectory toward the optimal location.

We don't calculate gradients or scores during the FL process itself; instead, we compute the scores only after the entire training is complete.

W1

The assumption of homogeneity for model architecture

We clarify that we follow the commonly agreed assumption of previous work on collaborative learning (federated learning, model merging) [1, 2, 3]: in order to aggregate the model in the parameter space, we should have the same model architecture or the same adapter. It's impossible to perform model aggregation if the models have different architectures. Exploring different model architectures may be beneficial for other applications but is not the focus of our paper; for example, multimodality.

Usage of low-rank adaptation

We wanted to clarify that the setup and motivation for using collaborative fine-tuning is to gather knowledge in an efficient manner. We focus on situations where individual clients don't have enough data or compute resources to pre-train a large language model or even perform full parameter fine-tuning. Low-rank adaptation is a commonly used parameter-efficient fine-tuning method for such scenarios. Therefore, the usage of low-rank adaptation is not a limitation but a way to achieve broader applicability. It is also regarded as a strength from other reviewers' perspectives: ngAe and igLM.

W2

The experimental settings/details and the ablation study are limited.

We have provided more ablation studies and experimental details in the uploaded PDF. We would really appreciate hearing your feedback regarding the limitations in more detail!

Q1

Global threshold as the unified standard, versus local thresholds:

It is necessary to have a global threshold to deal with the heterogeneity of data quality. Since the ratio of local low-quality data is unknown, and the ratios for each client are different from each other, local thresholds are not able to adapt to different and diverse low-quality data ratio scenarios. We show the comparison between global and local thresholds in Figure 1 of our PDF. There, the local optimal threshold represents the optimal thresholds for each client and is unknown in practice. The global threshold can be close to the local optimal, while local thresholds are far from the optimal thresholds.

Anchor data selection and biases

The anchor data is selected as a held-out high-quality dataset from the same data source as the test data. We show the ablation study of the anchor data selection and validation set selection in Figure 2 in our uploaded PDF, which demonstrates the robustness and unbiased selection in our experiments. In Figure 2 (Left), the global threshold remains stable and robust with different numbers of anchor data selected. In Figure 2 (Right), the box plot shows the data scores for 20 training samples randomly sampled from our training dataset over 5 different validation sets of equal size. Although the scores vary depending on the validation set used, the variance is within an acceptable range, ensuring that the order of their scores remains stable and robust. Thus, bias is not a concerning factor in our paper. We acknowledge that, in other cases, validation set selection will significantly affect the data selection process, and this is a common issue for all data selection work using validation sets.

Q2

Per-sample gradients are calculated for each training sample from the checkpoint saved during the model training.

For SGD, $\boldsymbol{\theta}^{t+1}-\boldsymbol{\theta}^t=-\eta_t \nabla \ell\left(\boldsymbol{z} ; \boldsymbol{\theta}^t\right)$

For Adam, $\boldsymbol{\theta}^{t+1}-\boldsymbol{\theta}^t=-\eta_t \mathcal{L}\left(\boldsymbol{z}, \boldsymbol{\theta}^t\right)$, $\mathcal{L}\left(\boldsymbol{z}, \boldsymbol{\theta}^t\right) \triangleq \frac{\boldsymbol{m}^{t+1}}{\sqrt{\boldsymbol{v}^{t+1}}+\epsilon}$

For AdamW, $\boldsymbol{\theta}^{t+1}-\boldsymbol{\theta}^t=-\eta_t \mathcal{L} \left(\boldsymbol{z}, \boldsymbol{\theta}^t\right)$, $\mathcal{L}\left(\boldsymbol{z}, \boldsymbol{\theta}^t\right) \triangleq \frac{\boldsymbol{m}^{t+1}}{\sqrt{\boldsymbol{v}^{t+1}}+\epsilon}{+\lambda \boldsymbol{\theta}^t}$

• overall compute complexity, where $N$ is number of checkpoints, and $d$ is gradient dimension.

• overall storage complexity:

Q3

We use LoRA to reduce the number of trainable parameters, which freezes the pre-trained weights and adds a low-rank adaptor to linear layers throughout the network. This means that, by nature, the trainable matrices are low rank. While our method is based on LoRA, it would also be very easy to adapt our proposed method to training where per-sample gradients are sparse and low-rank, for example, GaLore [4]. Our method is orthogonal to different training methods, and we believe this is a very promising direction for another future work.

Q4

ablation study of directly applying the proposed methods to raw data (supposed high-quality data).

As we clarified in our global response, raw data is not one of the baselines we are comparing our data selection method with, since the scope of our paper is to conduct data selection to filter out the low-quality data mixed in with the high-quality data. However, we agree it would be interesting to see how our methods perform on the dataset without low-quality data. We are currently running these experiments and will update the results once we have them.

Q5

Examples of low-quality data samples

Sure, we have provided the low-quality and high-quality data samples with their scores in our PDF for your kind review.

[1] TIES-Merging: Resolving Interference When Merging Models, NeurIPS 2023.

[2] Editing Models with Task Arithmetic, ICLR 2023.

[3] Communication-Efficient Learning of Deep Networks From Decentralized Data, Artificial Intelligence and Statistics, 2017.

[4] GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projection, ICML 2024.

2. Global Threshold:

I suggest that the authors incorporate the discussion of the necessity and justification for global thresholds and their implications

According to Definition 1.2 and Remarks, global thresholds are essential in our framework to effectively handle domain heterogeneity mentioned in our paper. As we highlight in our paper, local models can interfere with each other when merged in the weight space. Since our objective is to reduce the validation loss of the global model rather than local models, data selected by local thresholds don't necessarily translate to an optimal global model when merged. Global thresholds are necessary to coordinate different local distributions, thereby avoiding potential conflicts or interference during model merging or aggregation.

The current conclusion is based on limited observation from the selected datasets.

We have also conducted experiments on the FiQA dataset with different settings (see more details in our response to Reviewer V7za). They all demonstrate the same pattern, showing that the global threshold minimizes the sum of the distances to each local optimal threshold, while local thresholds cannot lead to global optimal solution. According to the instructions for the discussion period, we are not allowed to include new figures. However, we will incorporate figures showing the score distribution with different thresholds in the final version of our paper.

In addition, we provide details and proof on how we determine the global threshold as follows:

Suppose the score distribution of all data (including both good and bad data) is denoted by $F(x)$. The score distributions for good data and bad data are represented by $f(x)$ and $g(x)$ respectively. The relationship between these distributions can be expressed as:

where $\alpha$ is the proportion of good data in the dataset.

We can estimate $F(x)$ using the scores from all clients:

Similarly, we estimate $f(x)$ using the scores from anchor data:

Assuming $\alpha$ is known, which can be estimated from the quality of the global data, we can estimate the bad data distribution as follows:

The optimal global threshold $\tau^*$ can be obtained by : % $$ % This implies that the proportion of good data above the threshold $\tau^*$ is maximized.

where $\lambda$ is a hyper-parameter that balance the data number and good data ratio.

For a specific submodel $k$, the algorithm can be further refined. We denote the estimated good data distribution for a specific data source as $\hat{f}^{(k)}(x)$:

The threshold for the specific submodel can then be determined by: % $$

3. Per-sample gradient calculation:

We appreciate the opportunity to clarify our approach to calculating sample gradients from a checkpoint: For a given checkpoint with parameters $\theta_t$, we compute the gradient of a single sample $z_i$ as follows:

where $x_i$ and $y_i$ are the input and label of $z_i$, respectively. To enhance computational efficiency, we can focus on specific parts of $\theta$, such as individual layers or LoRA modules (e.g., q, k, v, o). For instance, if we concentrate solely on the gradients of parameters in layer 0, we adjust our gradient calculation as:

It's important to note that this gradient computation process is based on a single checkpoint, with no parameter updates occurring throughout. This allows us to process each training data point in parallel, significantly improving computational efficiency.

Regarding your concern about scalability, did you mean "micro-batching" here? In our scoring calculation step, we use a micro batch size of 1. Since there are no model updates during this process, scalability is not an issue.

We'd be glad to open-source our code implementation for more details.

First of all, we would like to clarify the following definition and notations:

Definition 1.1 (Data Quality on Specific Domain $k$). Given a model architecture $\theta$, a training configuration (optimizer, etc.), and a validation set $D_{val}$ in a specific domain $k$, the quality of training data $z$ is defined as follows: for $z\_1, z\_2 \in \mathcal{D}\_{train}$, if $\mathcal{L}\_{val}(\theta(z\_1), D\_{val}) < \mathcal{L}\_{val}(\theta(z\_2), D\_{val})$, then the quality of $z_1$ is considered higher than that of $z_2$. Here, $\mathcal{L}_{val}$ denotes the validation loss. In other words, the lower the validation loss, the higher the data quality.

Definition 1.2 (Data Quality on Collaborative Private Domains). Given a model architecture $\theta$, a training configuration (optimizer, etc.), and a validation set $D\_{val}=\{\mathcal{D}^{(1)}\_{val}, \mathcal{D}^{(2)}\_{val}, \ldots, \mathcal{D}^{(K)}\_{val}\}$ for all $K$ tasks, the quality of training data $z$ is defined based on the validation loss of the global model $\theta_{merged}$ on $D_{val}$. Specifically, for $z\_1, z\_2 \in \\mathcal{D}^{(k)}\_{train}$, if $\mathcal{L}\_{val}(\theta\_{merged}(z\_1), D\_{val}) < \mathcal{L}{val}(\theta_{merged}(z\_2), D\_{val})$, then the quality of $z_1$ is considered higher than that of $z_2$. As in the single-domain case, lower validation loss indicates higher data quality.

Remarks (Enhancing Data Quality on Collaborative Private Domains). In the collaborative learning framework, the ratio and distribution of low-quality data are unknown a priori. Only the server has access to the global distribution of both high-quality and low-quality data, while individual users cannot infer the global distribution from their local distributions due to statistical heterogeneity. The server can infer the distribution of high-quality data from public anchor data. Our objective is to select data points that most significantly reduce the validation loss of the global model, rather than optimizing for each local model independently. It is important to note that the scope of this study is not considering new models joining during training or continual learning paradigms.

Below are our responses to each follow-up question. We appreciate your insightful comments and the time and effort you've engaged in helping us clarify our proposed method. We look forward to continuing discussion with you on the OpenReview system if you have any remaining concerns, questions, or if you feel there's any misunderstanding on our part regarding the questions!

1. Mixed Model Architectures:

As per Definition 1.1 above, data quality is specific to each model architecture. Different architectures naturally lead to the selection of different high-quality data points. Under the current definition of data quality, it is not a reasonable assumption to uniform data selection across models with varying configurations. For example, the data quality standard would differ significantly between LoRA models with rank $r=16$ and those with $r=4$.

When it comes to individual models or groups of models from different domains with identical configurations, our proposed method is orthogonal and can be broadly applied. It is compatible with any model that allows gradient computation and can seamlessly integrate with state-of-the-art PEFT techniques such as AdaLoRA or Prompt Tuning.

Given that current collaborative learning and distributed training scenarios predominantly focus on models with homogeneous architectures, we believe our algorithm has broad applicability. While we acknowledge the potential for mixed fine-tuning with various PEFT methods, our current focus is on providing a SOTA solution for the most common use cases in the field.

We would like to thank you once again for your time and effort in providing insightful feedback and comments. As there are only a few hours remaining for further discussion, could you kindly let us know if you have any remaining concerns or points of confusion? We would be more than happy to discuss, address, and resolve them with you! If our responses have addressed your follow-up concerns, we would greatly appreciate your considering an increase in your rating to reflect these new improvements and clarifications, which we'll incorporate in our updated paper.

Thank you for your continued engagement and support in strengthening our research!

Q4

ablation study of directly applying the proposed methods to raw data (supposed high-quality data).

We have completed all the requested experiments and are pleased to share the results.

Regarding generalizability beyond the medical domain, we conducted experiments using the FiQA dataset, which focuses on financial question answering. For other results, please refer to our discussion with Reviewer V7za.

Dataset: We have four clients, each with 2000 training samples. We split the data into a training set (8000 samples in total), a validation set (also called the anchor set, 100 samples), and a test set (500 samples).

Evaluation metric: Responses from the model fine-tuned on our dataset are rated by GPT-4 on a scale from 1 to 10, reflecting criteria including relevance, accuracy, and fluency. To address potential positional bias, we send our response along with the benchmark output to GPT-4 twice, with different orders. We then calculate the average of these scores as the final performance score.

The table presents the performance of models trained with and without our data selection method across these different proportions. The results demonstrate that our method effectively enhances data quality across all scenarios with GPT-4 scoring. And in terms of accuracy of the data selection, our method consistently selected over 99% of the high-quality data across different proportions of low-quality data.

To understand the adaptability of our global threshold, we analyzed how the global threshold changes with different proportions of low-quality data. It illustrates that our global threshold adjusts across varying levels of data quality.

We appreciate your insightful question regarding the scalability of our method!

From an implementation perspective, per-sample gradient calculation is a widely encountered quantity in differential privacy and meta-learning. There are many tools we can use to compute per-sample gradients efficiently if needed, such as: (1) the Opacus library, which focuses on extending PyTorch for differential privacy, or (2) functorch, where composing vmap and grad provide significant speedup. In general, vectorization with vmap should be faster than running a function in a for-loop and competitive with manual batching.

Moreover, it's important to clarify the context of our work: our focus is on collaborative fine-tuning scenarios where clients have limited resources and data, naturally constraining the scale of per-client data selection. This setting often doesn't involve extremely large datasets on individual clients. In multi-GPU setups, we can leverage parallel computation across GPUs to enhance efficiency.

For cases involving clients with extremely large datasets or single GPU, in addition to using tools orthogonal to our proposed method like the Opacus library or functorch, we can also adapt our algorithm to address scalability requirements: We can compute a coarse-grained per-batch score, select all samples in batches exceeding our per-batch threshold, and apply per-sample gradient computation only to lower-scoring batches. This extended approach allows us to balance efficiency and precision in data selection, addressing potential scalability issues in large-scale scenarios from the algorithm perspective.

We would like to thank all the reviewers for their comments and discussions, and provide a summary of our responses. We will add all corresponding discussions and experimental results to our updated paper.

Background and Motivation:

• We have clarified our assumptions on collaborative settings, definition of data quality, and goals of data selection in the collaborative setting. (for Reviewer Xfnf)
• We have validated the reasonableness of our low-quality data construction methods: a larger portion of low-quality data has resulted in lower validation loss. (Please refer to the $\textcolor{blue}{Table}$ showing validation loss at the bottom of this response.)
• We have clarified the data sources of validation data and anchor data. (for Reviewer Xfnf and ngAe)
• We have discussed the same nature of federated learning and model merging. We have expressed the relationship and differences between federated learning and model merging using mathematical formulas. (for Reviewer ngAe)

Methods:

• We have refined the notations for improved clarity and have provided more detailed pseudo-code for our method in the PDF file. (for Reviewers igLM, ngAe)
• We have elaborated on the order of the whole procedure. (for Reviewer ngAe)
• We have elaborated on how we determine the global threshold. (for Reviewer Xfnf)
• We have discussed how we calculate the per-sample gradient (for SGD, Adam, and AdamW) and its complexity and scalability. (for Reviewer Xfnf)
• We have offered additional explanation on our method's training dynamics. (for Reviewer igLM)
• We have provided low-quality and high-quality data samples with their scores in our PDF. (for Reviewer Xfnf)

Experiments: We have spent considerable time and effort preparing these new results. We have more results that all support our claims and conclusions and we plan to add all of them to the paper or to the appendix.

• We have added experiments on a new dataset from a data-sensitive private domain - financial question answering. (for Reviewer V7za)
• We have clarified the baselines we are comparing to in our PDF. (for Reviewer ngAe)
• We have conducted evaluations on two challenging scenarios mentioned in the introduction of our paper: domain heterogeneity and quality heterogeneity.
• We have elaborated on our ablation study on weighted merging techniques. (for Reviewer ngAe)
• We have presented a comparison study on the global threshold determined by anchor data, comparing with other thresholds. (for Reviewers Xfnf, ngAe, and V7za)
• We have provided further discussions of the experimental results, including a comparison of data selection between the federated setting (frequent communication) and model merging (loose communication) setting and the potential reasons for the results. (for Reviewer ngAe)
• We have experimented with varying proportions of low-quality data to understand the method's effectiveness across different levels of data quality. (for Reviewer V7za)
• We have compared our gradient-based method with other centralized data selection methods including PPL, IFD, and DataInf, and have shown our advantages. (for Reviewer V7za)
• We have conducted an ablation study on the impact of using different numbers of anchor samples on the performance and reliability of the method. (for Reviewer V7za)

We once again express our gratitude to all reviewers for their time and effort devoted to evaluating our work. We hope our discussion and summary have addressed your concerns, misunderstandings, or questions.

$\textcolor{blue}{Table}$: A larger portion of low-quality data results in lower validation loss. After data selection, the validation loss becomes the same as that of the raw data, demonstrating the effectiveness of our low-quality data construction and data selection methods.