An Instrumental Value for Data Production and its Application to Data Pricing

We thank the reviewer for the detailed and constructive comments. We will answer your questions according to logical order.

We respectfully disagree with the reviewer’s comment on relevance to the machine learning community. Below, we provide our detailed explanations:

1. In many scenarios, such as experimental design and clinical data, sellers can effectively control the subjects and collect data. Even in the field of deep learning, data annotation companies like Scale AI can provide labeling for data with specific features. These are examples of scenarios where perfect customization can be applied.

2. Similarly, the Bayesian setting is also widely used in machine learning. For example, please refer to the Bayesian Clinical Trials article published in Nature (https://www.nature.com/articles/nrd1927). Also, Bayesian updating is also very common in deep learning literature—recent work (https://arxiv.org/pdf/2503.04722) even shows that LLMs exhibit Bayesian behavior.

3. In Section 2, we consider the general Bayesian setting of valuation, which does not rely on a known data-generating distribution. Then, in Section 3, we examine a special linear case and Gaussian noise. The reasons are as follows: Gaussian noise is the most common type of noise in nature. Regarding linearity, some studies show that data valuation is transferable—the value ranking under linear models often holds under nonlinear ones as well (see right side, lines 253–260). Moreover, when the true model is unknown, linear models are the most robust (lines 65–70). Finally, linearity allows for tractable theoretical analysis, which helps us better understand the underlying economic intuition. Extending our framework and tailoring it to specific scenarios is out of the scope of this paper but a promising direction for future work.

4. For potential machine learning applications, especially in industries such as clinical healthcare, our instrumental value can effectively evaluate the value of sequential data, whereas the Shapley value tends to overestimate it. Moreover, as shown in Section 2.3, despite certain assumptions possibly being violated in large-scale datasets, instrumental value reduces the computational complexity from exponential to polynomial. Appendix B.3 further demonstrates through numerical experiments that instrumental value exhibits a certain degree of correlation with the widely used Shapley value. Using instrumental value as an empirical approximation for data valuation—and potentially for pricing—in large-scale datasets could be a promising direction for future applications.

5. ICML features many fundamental machine learning papers, such as those in statistical learning, and is not solely focused on deep learning or empirical work. This is also why the theory track—including areas like game theory—exists. These theoretical contributions provide the foundation and guidance for future large-scale applications. The mutual reinforcement between theory and application is what makes machine learning a broad and inclusive field. Therefore, we believe that the problem we study, categorized in Theory-> Game Theory, is not only of interest to the ML community but also meaningful and impactful.

We hope our rebuttal has clarified the reviewer’s confusion and respectfully hope that the reviewer would consider re-evaluating the merit of our work accordingly.

Thank you for your kind remarks and questions, and we would now like to answer your questions in logic order. Below, we first address the reviewer's major questions, and then clarify a few minor comments. We are happy to engage with any further questions.

Re Corresponding Utility Function in Thm 3.1: In Theorem 3.1, the corresponding utility function reflects the confidence in the estimation, which is captured by the entropy. As our confidence in the estimation increases, the corresponding entropy decreases.

Re Design Matrix in the Limited Customization: In the limited customization setting, we assume that the design matrix $X$ is public knowledge, but the buyer does not observe the corresponding responses $Y$. In practice, for example, in clinical trials, information about the subject group is usually known, but the evaluation of the drug’s effectiveness needs to be purchased.

Re Realized Data and Noise: By "realized data," we refer to the realized response $Y$, while the variance function $\sigma$ is assumed to be known. In real-world scenarios, sellers typically disclose the measurement error or the specifications of the experimental instruments, such as their precision. When the buyer has no information about data quality, pricing in our setting becomes infeasible.

Re Other Comments Or Suggestions:

1. Thanks for your suggestion. Our idea is to first introduce the characteristics of valid valuations (Sections 2 and 3), and then present a concrete application from game theory, say, mechanism design (Section 4). We'll include a roadmap in the final version to outline the content of each section.
2. Due to the page limit, we have deferred the calculation of the posterior distribution and variance to the appendix. Please refer to Lines 667–677.
3. Thank you for pointing out the typo. We will correct it in the camera-ready version.
4. We introduced the definition of $g^n[x]$ in Lines 334–343, which produces $n$ responses for the buyer’s type $x$. Thank you for your suggestion — we will highlight its definition again around Theorem 4.1.

We hope our rebuttal has clarified your concerns and would greatly appreciate your re-evaluation of our work’s merits if these clarifications address your doubts.

We thank the reviewer for the detailed and constructive comments. We will answer your questions according to logical order.

Comparison with Other Data Valuation Methods: Compared to the fairness-based Shapley value, our instrumental value accounts for sequential data settings where Shapley can underestimate value (cf. Example 4). Theoretically, instrumental value avoids the exponential complexity of Shapley, making it more scalable for large ML datasets. The acceleration methods you mentioned just approximate the Shapley value, and similar ideas may apply to instrumental value. Our experiments show the conceptually significant advantages of instrumental value in terms of computation and sequential settings. Defining a universally better metric is unrealistic and not our intention. Meanwhile, Shapley value in fact also requires a suitable choice of valuation function, while we, under the Bayesian framework, rigorously characterize what valuations are valid for decision making (Sec 2).

As for https://arxiv.org/abs/2405.13954, it also stems from Shapley's inefficiency but takes a different angle—focusing on gradient-based LLMs. In contrast, we take an economics perspective (hence the game theory track), aiming for decision-oriented valuation (Sec 2.2) with theoretical guarantees, which they do not pursue. Thanks for your suggestions—we’ll discuss these works in the final version.

Discussing Our Assumptions: Regarding the Bayesian assumption, we can use either strong or uninformative beliefs (see right, lines 120–130), just to cover the general case. Bayesian updating is also very common—recent work (https://arxiv.org/pdf/2503.04722) even shows that LLMs exhibit Bayesian behavior. The paper you mentioned (https://pubsonline.informs.org/doi/10.1287/msom.2020.0915) also adopts a standard Bayesian setting, but we focus on what constitutes a valid valuation under this framework (Sec 2), while it assumes a specific utility function. We then explore the mechanism design problem under valid valuations, which is also entirely new.

Regarding linearity, some studies show that data valuation is transferable—value order under linear models often holds under non-linear ones as well (see right side, lines 253–260). Moreover, when the true model is unknown, linear models are the most robust (lines 65–70). Finally, linearity enables tractable theoretical results, which help us analyze the underlying economic intuition.

As for the DPP assumption, one example is mechanism design in clinical trials, where experimenters actively select samples and often update beliefs in a Bayesian manner—see Bayesian Clinical Trials published in Nature (https://www.nature.com/articles/nrd1927). Building an experimental platform based on instrumental value is a promising direction with high scientific and economic potential.

Utility vs Valuation: Our data valuation measures how much utility the posterior distribution (after observing data) brings over the prior—for example, more precise estimates (with lower variance or entropy) yield higher monetary revenue. This mapping is common in practice; in clinical trials, for instance, pharmaceutical companies often have clear estimates of the value of improved precision.

Difference from Active Learning: While data sequence has effects on both our instrumental data value and active learning, however these two areas are completely different. Active learning studies how to select sequential data and train models to maximize accuracy, whereas our paper studies how to define a valid data valuation function that captures downstream users’ utilities. We approach this study from a utilitarian perspective, and our value corresponds to information gain only at a very canonic special case. We will clarify this point in the camera-ready version.

We hope our rebuttal has clarified the reviewer’s confusion and respectfully hope that the reviewer would consider re-evaluating the merit of our work accordingly.