Incentivized Black-Box Model Sharing

We thank Reviewer jHme for reviewing our paper and for the positive and encouraging feedback!

We hope to answer your questions as follows.

---

Q1:

Could you provide one specific example that motivates why individual rationality is chosen as one of your key metrics?

Let's consider a scenario where three private hospitals (A, B, and C) want to share predictions based on their patient data to improve the predictive performance of their model. However, Hospital C has already invested significant resources into collecting a high-quality dataset (with a high data diversity), while Hospitals A and B have datasets that are noisy or of lower quality in comparison. If the ensemble predictions are naively shared among all three hospitals, Hospitals A and B would benefit more because they can significantly improve their models, while it may not benefit Hospital C at all (i.e., no collaborative gain in the form of either monetary or improved model) and its competitors are better off. In other words, hospital C did not improve its utility by collaborating (i.e., the individual rationality is not satisfied) and would be disincentivized to collaborate. Therefore, enforcing individual rationality can ensure Hospital C's collaboration by ensuring its utility will improve (or positive collaborative gain).

Additionally, a similar example is used to motivate individual rationality in the case of autonomous driving to achieve a win-win outcome (in the form of legislation that enforces individual rationality) for both consumers and car companies [1].

Therefore, individual rationality is one key incentive that we need to consider in collaboration.

[1] Karimireddy, S. P., Guo, W., & Jordan, M. I. (2022). Mechanisms that incentivize data sharing in federated learning. arXiv preprint arXiv:2207.04557.

---

Q2:

Why do you consider Shapley fairness as your main fairness notion?

Intuitively, the more one contributes, the more it should receive from collaboration. To formalize this intuition, there are many ways to do so. In our problem, we adopt the Shapley value as an interesting and meaningful way to achieve this because it is the unique solution that satisfies the four fairness properties (efficiency, symmetry, dummy party, and linearity) as in App. B.1.

• [symmetry] To ensure two equally contributing parties are equally recognized, symmetry ensures that two parties with equal marginal contributions to any coalition in the collaboration receive the same collaborative gain;

• [dummy party] To prevent free-riders, dummy party ensures that parties with zero marginal contributions to any coalition receive no collaborative gain.

  Additionally, it is shown by Sim et al. (2020) that Shapley value also satisfies some other fairness properties: strict desirability and strict monotonicity, which we have newly added to App. B.1. Hence, the Shapley value is a careful design choice in our work.

As additional justification, the Shapley value is a very widely used solution in previous data sharing works (Ghorbani & Zou, 2019; Jia et al., 2019a; Sim et al., 2020; Tay et al., 2022; Karimireddy et al., 2022;) as the notion of fairness.

Any other fairness notions that might fit into your framework?

One other common fairness notion, known as egalitarian, aims at achieving equitable outcomes for all the parties (e.g., by rewarding all the parties equally). This does not satisfy the dummy party property and can cause the free-rider problem since even the non-contributing party is rewarded equally with the high-contributing parties.

Another common fairness notion (in the allocation of goods) is envy-freeness, which is not applicable here. Because it is typically used for rivalrous goods, but the rewards here (i.e., ensemble predictions) are non-rivalrous (since they can be replicated), and it is the reason for our proposed definition of fair replication game.

For discussions of different fairness notions, we will incorporate them in the revision.

---

We answer the other question in the subsequent comment.

Q3:

In Sec 5 you suggested that "We will later empirically show that the virtual regret $\epsilon$ is not needed and the strict IR is satisfied". Is this a purely empirical observation or do you believe stronger theoretical results can be established here?

It is mainly an empirical observation, but we think stronger theoretical results may be possible. The reason is the derivations for our theoretical result on the virtual regret $\epsilon$ involve several inequalities (e.g., Hoeffding's inequality, the triangle inequality, the constraint of the growth function by the VC dimension, and the supremum inequality of the domain divergence measure, as in App. C.3.). The improvement of the tightness of one or more of these inequalities can potentially lead to a stronger theoretical result. We will include this comment in our revision.

---

We hope our responses have addressed your concerns and helped you to understand our work better. We are happy to answer any additional questions.

We thank Reviewer 1Wgz for taking the time to review our paper and for recognizing that our studied problem is important and practical, our model achieves desirable properties, and our analysis is thorough. We wish to provide the following clarifications and have incorporated some changes in our revision (with highlighted text in the updated pdf).

---

The fact that there is an FAQ section on the model speaks to how the model is not completely clear.

Our provided Q&A section in App. A.2 is meant to further supplement the main paper with more nuanced and specific details. In our revision, we will improve the exposition of our work to specify the problem formulation more clearly.

W1:

How should we compare prediction error to monetary payments to "rewards" (samples of ensemble predictions)? (Do they use the same unit of measurement?)

The monetary payments (i.e., $p_i$) and the value $r_i$ of the rewards (of size $T_i$) can be viewed to use the same unit of measurement (i.e., in the same domain). This is by design so that the allocation mechanism (Theorem 1) enables a translation between ensemble predictions (i.e., the realized rewards $T_i$) and monetary payment. We note that this has not been previously achieved by existing works. Please refer to Q4 in App. A where we illustrate how the monetary currency is projected into the positive real domain $\mathbb{R}_+$ to align with the reward domain.

When you say "compare prediction error to monetary payments", we assume you mean the relationship between the two. To elaborate more on the relationship: An increase in the monetary payment $p_i$ leads to a higher reward $r_i$, and thus a larger size for $T_i$, which can decrease the upper bound on the prediction error $L_{\mathcal{D}}(h_i')$:

• From Proposition 2, the upper bound of prediction error (i.e., $L_{\mathcal{D}}(h_i')$) depends on payment $p_i$, because $L_{\mathcal{D}}(h_i') \leq L_{\mathcal{D}}(h_i)+\epsilon_i$ and $\epsilon_i$ depends on $T_i$. To compare the prediction error to monetary payment with approximation, we investigate "the change in $\epsilon_i$" as a function of $p_i$ in a newly added empirical result (i.e., Fig. 9 in App. D.2.1), which provides an empirical quantification between the accuracy gain w.r.t. the increase in (monetary) payment.

---

W2:

Relatedly, what is the formula for the utility of party i?

By definition, $u_i$ is the Shapley value to the linearly combined game of $G$ and $G^p$. By derivation, the explicit formula for the utility $u_i$ of party $i$ is given in Theorem 1: $u_i = r_i + r_i^+ + p_i^+ - p_i$, where

• $r_i=\phi_i$,
• $r_i^+ = \frac{\mathcal{V}_N\times p_i}{\mathcal{V}_N - \phi_i}$,
• $p_i^+=\sum_{j \in N \setminus i }\frac{\phi_i \times p_j}{\mathcal{V}_N - \phi_j}$, and
• $p_i\in \mathbb{R}_+$ is determined by the party itself.

The utility $u_i$ represents the overall gain of party $i$ from the collaboration, as a sum of $i$'s total reward $(r_i + r_i^+)$ and payoff/net compensation $(p_i^+-p_i)$.

• $(r_i + r_i^+)$ represents the value of ensemble predictions that party $i$ receives from the collaboration,
• $p_i^+$ represents the value of the received monetary compensation from all other parties, and
• $p_i$ is the payment that party $i$ made to the host in exchange for additional rewards of the value $r_i^+$.

Our formula of the utility fully specifies all the terms. Note that these interpretations (of the utility) are provided in the main paper (above Theorem 1 and below Remark 1).

---

W3:

Payments can be made from one party to another. Does each party, decide on their own, how much to pay to each other party, or is this transfer also specified as part of the mechanism?

Each party decides on its own how much to pay in total (i.e., $p_i$) to the central host; the exact distribution of $p_i$ to each other party is specified by our mechanism (and managed by the central host) to achieve fairness: The proposed formula of $p_i^+=\sum_{j\in N\setminus \{i\}}\frac{\phi_i \times p_j}{\mathcal{V}_N - \phi_j}$ in Theorem 1 specifies this payment distribution from every party $j\in N\setminus \{i\}$ to party $i$.

Does each party have a budget?

Our mechanism can achieve fairness whether or not a party has a budget constraint:

• [With budget constraint.] If party $i$ has a budget of $B_i$, it can freely choose payment on its own as long as $p_i \leq B_i$, and fairness is always achieved by the reward design and realization, as shown in App. C.6.

• [W/o budget constraint.] If party $i$ does not have any budget constraint, then it can make the maximal payment $p_i^\ast = (\phi^\ast -\phi_i)(\mathcal{V}_N-\phi_i)/\phi^\ast$ to receive all the reward of value $\mathcal{V}_N$, which is mentioned in the paragraph below Remark 1. In this case, fairness is achieved by fairly distributing the payments.

---

We address the other concerns in the subsequent comment.

We are grateful to the reviewers for their constructive and detailed feedback. We will incorporate this feedback into our revised work. We respond below to their concerns:

---

W1:

I’m a little confused about how the weights $\beta_{i,x}$’s are set if the true label for point x is unknown.

Please kindly refer to our response below in Q1 and App. B.2 for how $\beta_{i,x}$’s are set if the true label for point x is unknown.

add more prose throughout the paper.

To avoid significant changes (listed below) in the rebuttal as instructed by the guideline in ICLR 2024 "Area chairs and reviewers reserve the right to ignore changes that are significantly different from the original paper", we will make the following changes in the revision:

(1) move the valuation part in section 6 to Appendix

(2) move some remarks in Appendix to the main paper to add more elaborations

(3) include more experiments with non-optimal (practical) ensemble methods in the main paper

---

Q1:

So, is it just that in the very beginning ... the host asks the clients to participate and find these weights in the very beginning and use these weights going forward?

No. In our setting, no matter what ensemble method is used, the host will query each party for each data point and continually determine the ensemble weights.

But more realistically, the host would want to query each party to provide predictions for points for which the true label is unknown.

Your realistic consideration is correct. In fact, in some of our experiments, the host queries each party for each data point and continually determines the ensemble weights, where the true label is unknown.

...for which the true label is unknown. In those cases, how would the host want find these weights?

We assume the host knows the ground truth only when we use the optimal ensemble method. As we have described in App. B.2, ensemble methods from previous works such as average ensemble (AVG), majority vote (MV), knowledge vote (KV), and multiplicative weight update (MWU) do not need the ground truth. Those ensemble methods can set the weight $\beta_{i,x}$ both differently for each party and each data $x$.

On the other hand, if the ground truth is available, we can observe a much stronger correlation (i.e., -0.72) between the average ensemble weight $\mathcal{V}\_i$ and the generalization error $L\_{\mathcal{D}}(h\_i)$ on MNIST in Table 2, compared with the correlation of MV (i.e., -0.24) and MWU (i.e., -0.24) on MNIST in table 1.

We conduct an additional experiment (update in App. D.1) and highlight that the requirement on the ground truth is not very restrictive, where the labeled data (i.e., ground truth) of size as small as 100 is sufficient to identify high-quality models, as shown in Fig. 6 in App. D.1. The negative correlation of size 100 is as good as that of size 4000. In practice, those weights estimated on the small labeled dataset can be potentially used for other unlabeled data.

---

We address the other concern in the subsequent comment.

I'm aware that the results still hold true regardless of what the chosen weights are, but my point is that the results hold only with respect to the given weights.

So, if I choose arbitrary weights, the contribution determined by these weights would still be arbitrary and the theoretical guarantees would be arbitrary as it's with respect to those arbitrary weights. Therefore, in order to fully capture the contribution of each agent faithfully, we better choose "good" weights that truly captures the contribution of each agent because the good theoretical guarantees hinge upon the fact that the weights are faithfully measuring the contribution. Simply said, the quality of the theoretical guarantees hinge upon the quality of the weights.

And my concern is that coming up with good weights and verifying that those are good weights is hard to do when there are no ground-truth; the paper is only empirically showing the previous ensemble methods seem to do well, but these are just practical heuristics, and there's no theoretical guarantee that these are good weights. Note that even in the experiments, a hold-out dataset is being used to show that the ensemble methods are performing well. But in the absence of any hold-out dataset with ground truths, how can one even tell whichever chosen ensemble method is actually obtaining good weights or not?

Or am I still misunderstanding the results?

We thank Reviewer gZai for taking the time to review our paper and for providing a very detailed summary and questions and positive feedback that our problem is well-motivated, our design approach is interesting, and for appreciating our efforts in supporting key assumptions and empirical evaluations.

We would like to address the comments as follows.

---

W1:

Limitations of the proposed method are not discussed clearly.

We have added a section (App. A.1) discussing the limitations and will make this clearer in our revision.

For example, one limitation of our method is that it is developed for ensemble methods that take the form of weighted sum formulation (in Section 4.1), which does appear in several common ensemble methods (mentioned in Section 4.1).

---

W2:

Unclear applicability for practical ensemble methods

We indeed have experimental results for non-ideal ensemble methods (which are more practical) and have also included additional experiments.

• [Results for non-ideal ensemble methods.] The results are in Tables 4 and 5 (in App. D.1). We observe a stronger correlation between the average ensemble weight $\mathcal{V}\_{i}$ and the generalization error $L\_{\mathcal{D}}(h\_{i})$ with the practical ensemble methods (i.e., MV, KV, and MWU) as shown in Table 5 under the non-i.i.d. data setting, which we believe is more important because the setting is more realistic. If the ensemble is bad (e.g., AVG shown in Table 5), our method of valuation and allocation still works, but parties with better models may not be identified as such in the collaboration.

• [Additional experiments.] We included additional results in our paper, in Fig. 8 (in App. D.2.1), to demonstrate the accuracy gains w.r.t. different non-ideal (practical) ensemble methods used. In particular, we observe that strict IR is always satisfied.

  For the additional experiment on CIFAR-10, we will include it if it is complete before the end of the rebuttal; otherwise, we will include it in the revision.

As these different (non-ideal) ensemble methods produce predictions of varying qualities, we use the ideal (i.e., optimal ensemble) method to illustrate the effect of an ideal case of collaboration. The non-ideal ensemble methods might not be as effective in achieving fairness or IR as the ideal ensemble, but they can be applied nevertheless.

---

W3:

Many remarks which were very helpful to my understanding only appeared in Appendix A.

Thank you for taking the time to carefully read our prepared appendix. It would be very helpful if the reviewer would let us know the specific remarks (currently in appendix) that would aid the understanding of the reader, if moved to the main paper. We will improve the presentation accordingly.

---

We address the other concerns in the subsequent comment.