We thank the reviewer for their detailed review, insightful comments, and positive evaluation of our paper. We appreciate that the reviewer recognized our novel mathematical analysis, computational efficiency and empirical results.

W1 & Q3: Why do we need model-agnostic techniques? Why would legal and commercial decisions be made with respect to model-agnostic techniques?

Thank you for this question. We clarify the commercial and legal motivations for model-agnostic data valuation.

Commercial incentives: When training large models, two important reasons why data valuation would be important:

• Large models are often trained on noisy web-crawled data containing duplicates [10] and data poisoning attacks [11]. Therefore, it is essential that the data is cleaned before training.
• At the same time, companies also pay substantial costs for high-quality datasets (OpenAI-Time Magazine, Shutterstock deals [1][2]). In this case, it is crucial to be assess the value of data being bought.

Why model-agnostic? Model training takes months and costs millions of dollars [3], so it is reasonable to assume the largest models are trained only once (or there is a large cost to train a model again). This means data valuation must happen before training begins. Traditional methods like Data Shapley are impractical because they require training multiple models. Other model-based approaches too require at least one training run before they perform data valuation. Therefore, companies need model-agnostic approaches for data valuation.

Legal incentives: EU AI Act Article 10 [4] requires that training data be "relevant, sufficiently representative, and to the best extent possible, free of errors and complete" and the Act mandates assessment of "the availability, quantity and suitability of the data sets that are needed". Importantly, these requirements apply to the data itself, not to any specific model trained on it. This data-centric approach makes sense because datasets are commonly shared and reused across different AI applications and organizations. Model-agnostic methods are therefore essential for regulatory compliance.

W2: Why is KAIROS approximating MMD LOO better than LAVA approximating Wasserstein LOO a good thing?

Thanks for this thoughtful question. Model-agnostic methods define influence based on distributional distances, and both Wasserstein and MMD are valid choices. Our key contribution is providing a method that faithfully approximates its stated objective. LAVA faces inherent challenges with Wasserstein approximation due to the non-unique dual potentials and regularization bias, leading to deviations from true LOO rankings. KAIROS offers exact computation of MMD-based influence, providing users with transparent and reliable valuations. This transparency is particularly important in applications like data markets where stakeholders need to understand exactly what the valuations represent and how they were computed. We acknowledge that if someone specifically requires Wasserstein distance for their application, LAVA would be the appropriate choice. However, our empirical evaluations show that MMD-based data valuation generally works well across diverse tasks, and KAIROS consistently achieves superior performance compared to existing methods.

W3 & Q1: If we use validation data to estimate P(y|x), what is the point of data valuation when the goal is often to estimate P(y|x)?

Thank you for this important question. The validation set is generally small (300 samples in our experiments), so P(y|x) estimates are noisy. However, this empirically seems to provide sufficient signal to identify corrupted training points. After removing these points, we train on the cleaned, much larger training set to get better P(y|x) estimates and higher accuracy.

We demonstrate this on CIFAR-10 feature noise: A model trained only on validation data achieves 72% test accuracy, while training on the full training set after removing the bottom 20% (identified by KAIROS) achieves 88.5% test accuracy. Even selecting just the top 300 training samples (matching validation size) gives 87.3% accuracy, demonstrating that KAIROS improves sample quality beyond what the small validation set can provide. We will add this discussion and results to the final paper.

W4 & Q2: The equation for influence of E-MCMD (Eq 8) does not seem to depend on training distribution Q. Is the dependence hidden?

Thank you for this insightful question. The dependence on Q appears in two ways:

First, in our derivation of this influence (Appendix B.4, Line 634), there is a Q-dependent constant term. We omit this term in practice because it only affects absolute magnitudes, not relative rankings. However, the absolute value of the influence (that determines whether a small perturbation increases or decreases the E-MCMD) does depend on Q directly.

Second, the expected E-MCMD influence at any point x, $E_{y \sim Q(y|x)}[IF_{EMCMD}(x,y;P,Q)]$, depends on Q's conditional distribution. This means that when Q is noisier (more corrupted labels), more points will have high negative influence values. This allows KAIROS to detect label corruption as shown in our experiments.

W5: Description of Theorem 1

Thank you for pointing this out. We will change the wording to: "Under mild regularity assumptions, the expected train–validation loss gap is bounded above by the sum of the marginal MMD and conditional E-MCMD. Consequently, removing a point which decreases this distance provably tightens the out-of-distribution error bound for any learning algorithm. The influence is a first-order approximation of the effect of removing a point, suggesting that removing points with large influence scores could decrease the out-of-distribution error.”

W6: Symmetry property (Proposition 2)

Thank you for this question. We will clarify that points making equal contributions to $\hat{\text{MMD}}$ receive identical $\hat{\text{IF}}$ scores. Since the true MMD is unknown in practice and $\hat{\text{MMD}}$ has established convergence bounds to MMD [5], this consistency property is meaningful. For the true MMD and IF, defining "equal contribution" would require circular reference to the influence function itself, making such a statement trivial. We will revise our wording to reflect our claim correctly.

W7: Unclear Notation

We will make changes in the final revision to make the notation clear - defining $n$, $N$, the kernel $k$, and clarifying that the rescaling is up to additive and positive multiplicative constants.

W8: References

Thank you for pointing this out. We will cite specific pages and results in the final version.

1. For Line 105, the O($\frac{1}{n²}$) error follows from Taylor expansion of d($P$,$Q_ε$) around ε=0. The second-order term is O(ε²). For leave-one-out with n samples, ε = $\frac{1}{n-1}$, giving O($\frac{1}{n²}$) error. We will cite Chapter 20, Page 291 of the textbook [6] which shows a similar expansion.

2. For the non-uniqueness of Kantorovich potentials, we will cite Page 256, Remark 10.30 of the book [7]. We will also cite [8] and Remark 2.3 in [12] which discuss this in detail. Further, we will add this concrete example to illustrate non-uniqueness:

Consider P = $\frac{\text{Uniform(0,1)}}{2}$ + $\frac{\text{Uniform(10,11)}}{2}$ and Q = $\frac{\text{Uniform(2,3)}}{2}$ + $\frac{\text{Uniform(12,13)}}{2}$, where Uniform(a,b) denotes the uniform distribution on interval [a,b]. Both of these functions are optimal dual solutions for Wasserstein-1:

• f₁(x) = x
• f₂(x) = x if x < 5, f₂(x) = 10-x if 5 ≤ x < 7, f₂(x) = x-4 if x ≥ 7

This demonstrates that optimal Kantorovich potentials are not unique.

3. We will make two clarifications in lines 122-126:

• Change "Crucially, this benign decay does not hold for most IPMs" to "Crucially, this benign decay need not hold for most IPMs."
• Add a reference to Page 39-40 [9] which provides an example where the rate $O(f^{\*} - f^{\*}_{ε})$ is less than 1. In our case, this corresponds to a uniform distribution P with Q and Q_ε being the pushforward measures under potentials |x| and max(|x|,$ε^{\frac{1}{d}}$). The analysis shows that $O(f^\* - f^\*_ε)$ = $ε^{\frac{d+2}{2d}}$ < $ε^1$ when d > 2.
• For the claim "In the Wasserstein-1 case, $f^\*$ corresponds to a Kantorovich potential, which is non-unique" we will provide the specific citations discussed in point 2 above to establish that Kantorovich potentials need not be unique, which may lead to non-deterministic influence values for LAVA.

[1]Reuters. Time, openai sign multi-year content deal, June 2024. Published 27 Jun 2024.

[2]Sherwood News. Ai deals to make up a third of shutterstock’s revenue by 2027, June 2024

[3]Cottier, Ben, et al. "The rising costs of training frontier AI models." arXiv preprint arXiv:2405.21015 (2024).

[4]Ortigosa, Adrián Palma. "Data and Data Governance (Article 10)." The EU regulation on Artificial Intelligence: A commentary. Wolters Kluwers Italia, 2025.

[5]Gretton, Arthur, et al. "A kernel two-sample test." The journal of machine learning research 13.1 (2012).

[6] Van der Vaart, A.W. Asymptotic statistics, volume 3. Cambridge University Press, 2000.

[7]Cédric Villani et al. Optimal transport: old and new, volume 338. Springer, 2008.5

[8] Staudt, Thomas, Shayan Hundrieser, and Axel Munk. "On the uniqueness of Kantorovich potentials." SIAM Journal on Mathematical Analysis 57.2 (2025).

[9] Letrouit, Cyril. "Lectures on quantitative stability of optimal transport." (2025).

[10] Lee, Katherine, et al. "Deduplicating training data makes language models better." arXiv preprint arXiv:2107.06499 (2021).

[11] Carlini, Nicholas, et al. "Poisoning web-scale training datasets is practical." 2024 IEEE Symposium on Security and Privacy (SP). IEEE, 2024.

[12] Del Barrio, Eustasio, and Jean-Michel Loubes. "Central limit theorems for empirical transportation cost in general dimension." (2019).

We thank the reviewer for their valuable review and positive assessment of our contributions. We are glad the reviewer appreciated the novelty of our closed-form MMD-based influence function, its theoretical properties, and empirical effectiveness, particularly under noisy and adversarial conditions.

W1: It would have been better if the authors have covered a bit large-scale datasets

Thank you for your suggestion! The current baselines in the paper are challenging to run on larger datasets, due to runtime (model-based methods) or memory consumption issues (LAVA). However, we conduct additional experiments comparing KAIROS with SAVA (a scalable version of LAVA) on the ImageNet dataset, which contains 1.28M training examples and 1000 classes. We use a ResNet50 encoder to extract feature representations from ImageNet, and we use an A100 GPU with 40GB VRAM for computing data values.

Our results demonstrate that KAIROS achieves better performance and efficiency on this large-scale dataset. In particular, when facing a large number of classes, the pair-wise conditional Wasserstein distance computation in SAVA becomes expensive, leading to a 15x runtime compared to KAIROS. Please refer to our response to W1 & Lim1 of Reviewer UAd2 for the detailed results and analysis.

W2: On differences and comparison on sample reweighting methods like FSR and MAPLE

Thank you for suggesting this comparison. FSR and MAPLE are model-based sample reweighting methods that differ fundamentally from KAIROS. Unlike KAIROS, they do not compute data values but rather learn reweighting functions for specific training objectives. While they can be applied to corrupted label detection as you mentioned, KAIROS offers broader applicability across tasks such as detecting harmful fine-tuning data and data poisoning attacks. These methods are also not model-agnostic and inherit the limitations of model-based approaches that we discussed earlier. To provide a comprehensive evaluation, we conducted experiments comparing KAIROS with MAPLE on CIFAR-10 feature and label noise detection tasks. We use the target labels as group labels for MAPLE since no explicit group labels are available for our datasets. We report the AUC of the fraction of covered corrupted data vs the fraction of inspected data (we will include full plots in the final paper, but images are not allowed in rebuttals):

Table 6: AUC for CIFAR-10 feature noise and label noise tasks

The results agree with the reviewer’s intuition. MAPLE performs the best on label noise detection but has very poor performance on feature noise detection. This illustrates that while reweighting methods can be effective for specific corruption types like label noise, KAIROS provides consistent performance across diverse tasks including feature noise, label noise, adversarial attacks, and harmful fine-tuning detection. We will include this comparison in the revised paper.

We thank the reviewer for their constructive and insightful feedback, and appreciate their positive evaluation of our theoretical analysis and computational efficiency.

W1 & Lim1: Experimental performance is good, although the evaluation is fairly basic and could be improved. There needs to be a comparison with more recent and competitive baselines such as SAVA.

Thank you for the feedback! SAVA was primarily built to address the memory consumption issue of LAVA, making it scalable to larger datasets. Therefore, we compare KAIROS with SAVA on ImageNet, which has 1.28M training samples and 1000 classes. We use a ResNet50 encoder to extract feature representations from ImageNet, and we use an A100 GPU with 40GB VRAM for computing data values. We report the AUC of the fraction of covered corrupted data vs the fraction of inspected data (we will include full plots in the final paper, but images are not allowed in rebuttals):

Table 2: KAIROS and SAVA on Imagenet

Feature Noise Detection

Label Noise Detection

Our results show KAIROS outperforms SAVA in both efficiency and effectiveness. Since SAVA is built based on LAVA, similar to the results in Figures 2 and 3, KAIROS outperforms SAVA in both experiment settings, especially in label noise detection.

The efficiency improvement stems from KAIROS's closed-form solution, making it more suitable for batch-based GPU acceleration compared to Sinkhorn computations. In contrast, SAVA (and LAVA) needs to compute pairwise conditional Wasserstein distances between P(x|y) for every pair of classes, requiring $\frac{1000\times 999}{2}$ computations, thus leading to a high runtime. Specifically, the computation of pairwise conditional Wasserstein distances takes more than 80% of the overall runtime.

W2: The method still depends on training a classifier for label conditional MMD so is not technically model-free.

Our method is model-agnostic with respect to the training data being valued. We never train models on the training set. We only train a single classifier on the validation set (3% of training size in our experiments) to estimate P(y|x). This preserves the core benefits of model-agnostic approaches: 1) avoiding multiple model retrainings (unlike Data Shapley or Data-OOB) and 2) data preprocessing before training the downstream model. Further, this makes our approach significantly faster than model-based baselines (Figure 7, page 9).

W3a: The method is limited to classification tasks.

The conditional component (E-MCMD) currently targets classification, but our framework applies broadly to many other tasks which do not have a target (y), such as image poisoning attacks and harmful LLM fine-tuning data detection (Figure 4). The marginal MMD component works for any data type. We acknowledge the classification limitation for labeled tasks and identify regression extension as future work (Section 5).

W3b & Q2a: This method assumes a clean reference set. Distributional measures may have issues in this scenario where there is no clean reference available. How robust is the method to a noisy or biased reference set?

Thank you for this important question. For data valuation, one needs to measure utility with respect to something. In our case, we assume access to a small sample from the target distribution, which we call the validation (reference) dataset. This assumption is shared with LAVA. Further, model-based methods also value data based on performance on the validation set which is generally assumed to be representative of the test distribution.

We test the robustness of KAIROS and other baselines to a noisy reference set. We conduct additional experiments by adding noise to the validation set for the feature noise CIFAR-10 task. We consider two settings where we randomly corrupt 3% and 7% of the validation samples respectively.

Table 3: Robustness to Noisy Reference Set - AUC Performance

Our results show that KAIROS is robust and maintains original performance in the presence of noisy reference sets. Moreover, KAIROS outperforms both model-based and model-agnostic baselines across different noise levels.

Q1: How does this method compare against more recent data attribution methods for large models like Trak, Logra?

Thank you for this question. TRAK and LOGRA are model-based attribution methods that require training models (TRAK often trains multiple models, while LOGRA provides optimizations to make influence functions more efficient for large models). In contrast, KAIROS is model-agnostic and requires no training. We test KAIROS against these methods on feature-noise detection and label-noise detection tasks using CIFAR-10 with the same experimental setup described in the paper. Since we cannot share plots in the rebuttal, we report the AUC of the curves below. We will include the full plots in the final version.

Table 4: Brief Description of Experiments - AUC Performance Comparison

For both experiments, KAIROS achieves the highest AUC compared to baselines, including TRAK and LOGRA, particularly in the feature noise detection setting.

Q2b: How robust is the method to distribution shift between training and validation set?.

Thank you for this question. KAIROS is designed for distribution shift between training and validation distributions. The method measures how training points from noisy distribution Q contribute to distance from clean validation distribution P. All our experiments involve such shifts where training sets and validation sets come from different distributions.

Q3: How to choose the kernel and bandwidth in practice for different datasets? How to balance λ? Is the median heuristic always a sensible choice?

We agree that kernel and bandwidth selection is an important practical consideration. The median heuristic has proven effective across all our experimental tasks and is widely adopted in MMD literature [2-4]. We also experiment with a polynomial kernel of degree 2, which demonstrates excellent performance. Please refer to the response to W1 of reviewer g4SV for detailed experimental results. For the balancing parameter λ, we set it to align the variance of the feature and label components in Equation 12 (see Appendix D). While the median heuristic works well in practice, we acknowledge that adaptive kernel selection could be beneficial for some applications and identify it as future work (Section 5). We will add results comparing different kernel choices to the revised paper.

Q4: How would the method perform on more nuanced issues like selection bias, concept drift, or data redundancy?

Thank you for suggesting this direction. We explore selection bias where subgroups are under-represented in training data. We use the ACS Income dataset from WhyShift [1], which predicts income from demographic attributes. This dataset exhibits geographic bias where some states like Puerto Rico (PR) are severely under-represented while others like California (CA) are over-represented. Liu et al. [1] also show that models trained predominantly on CA data fail to generalize to PR. We simulate selection bias by creating a training set with 80% CA and 20% PR samples (1000 total), while our reference validation set contains balanced 50% CA and 50% PR samples (300 total). We evaluate how well KAIROS and baselines identify points from the under-represented group (PR). Data attribution methods should assign high influence to under-represented samples. We report the AUC of the fraction of under-represented samples detected vs top k fraction of data chosen.

Table 5: Selection Bias Detection Performance

The results show that KAIROS considerably outperforms both model-agnostic and model-based baselines. We will add these results to the final version of our paper.

For data redundancy, KAIROS rankings remain stable when datasets contain duplicates. We empirically verified this by duplicating the training dataset up to 10 times and confirming that relative rankings are preserved, demonstrating robustness to redundant data.

Q5: How does this work differ from and related to Data Distribution Valuation?

Data Distribution Valuation values entire distributions (or datasets) by computing MMD between each dataset and a reference (or uniform average of all datasets). Our work addresses a fundamentally different problem: quantifying individual datapoints' contributions to the MMD between training and validation distributions via influence functions. This measures how removing a single point changes the distributional distance, not the MMD between that point and the reference distribution. We will include a detailed comparison in the appendix of the final version.

[1] Liu et al. "On the need for a language describing distribution shifts: Illustrations on tabular datasets." NeurIPS 2023

[2] Shekhar et al. "A permutation-free kernel two-sample test." NeurIPS 2022

[3] Gretton et al. "A kernel two-sample test." JMLR 2012

[4] Doran et al. "A Permutation-Based Kernel Conditional Independence Test." UAI 2014

We thank the reviewer for their thorough review and strong support of our work. We appreciate your recognition of our theoretical analysis, empirical validation, and computational efficiency.

W1: The performance of KAIROS might depend on kernel choice.

Thank you for this question. Any MMD-based method depends on kernel choice, and we acknowledge this consideration for practitioners. To address concerns of sensitivity to the kernel, we conducted additional experiments using a polynomial kernel of degree 2 on the feature-noise CIFAR-10 task. We report the AUC of the fraction of covered corrupted data vs the fraction of inspected data (we will include full plots in the final paper, but images are not allowed in rebuttals):

Table 1: AUC for detecting feature noise in CIFAR-10

The results show that KAIROS with polynomial kernels achieves nearly identical performance to Gaussian kernels (0.856 vs 0.857) and outperforms all baselines. Further, as mentioned in Section 5, adopting learned kernels that adapt to specific tasks is a promising direction for future research.

W2: Although KAIROS is highly efficient in the online setting, its computational complexity in the offline setting is comparable to that of LAVA.

Thank you for recognizing KAIROS's superior efficiency in online settings, where we achieve O(BN) complexity compared to LAVA's O(N²) (where B is batch size and N is dataset size). Regarding the offline setting, while both methods have O(N²) theoretical complexity, there are important distinctions: LAVA achieves O(N²) only through Sinkhorn approximation, which introduces bias and influence that may be non-deterministic (Section 3.1). In contrast, KAIROS provides exact MMD influence computation without approximation while maintaining the same complexity. Furthermore, our empirical results (Figure 7) demonstrate that KAIROS achieves faster actual runtime than LAVA even in the offline setting.