Incentivizing Time-Aware Fairness in Data Sharing

We sincerely thank the reviewer for taking the time to review our work and for recognizing our interesting problem and complete analysis. We address your questions below:

...whether non-IID or IID data is considered?

How is the data modeled? How does the data distribution affect model quality?

The data distribution can be either IID or non-IID. Our work adopts the perspective that the value of the data depends on the learned model. That is, both data quantity and data quality matter. Hence, we consider valuing the data based on the conditional information gain on the model parameters or the dual of the accuracy function, which inherently captures the dependency on models and are applicable to all data distributions. Note that this perspective is shared many data valuation works [13, 25] that also value data by the model performance when trained on specific subsets. In our experiments, we explicitly include both IID data with different data quantities (Friedman and CaliH datasets, lines 312--314) and non-IID data distributions (MNIST and CIFAR-100 datasets, lines 317--318 and lines 910--914). The experiment results in Section 7 demonstrate that our methods consistently capture the relative value of each party under both settings.

Fairness is not modeled in this work.

How is fairness modeled in this work? Fairness should be a formal mathematical metric rather than an intuition.

In our work, fairness is modeled through satisfying formal incentive-based criteria (F1--F5 in Sec. 4). As noted in line 128, our work adopts a well-established view in the literature that fairness can be defined in terms of satisfying certain axiomatic properties. For example, in [47], fairness is defined as satisfying incentives such as symmetry, desirability, uselessness which are inspired by intuition. We explicitly include these incentives or generalizations thereof in our formulation (Sec. 4), and our proposed methods are designed to satisfy them, hence achieving fairness.

In addition, we introduce the novel time-aware incentives (F6--F8) to capture time-aware fairness, which mathematically formalize the perspective that earlier parties should be rewarded accordingly. It is a widely recognized perspective in psychology, economics (lines 54--57) and motivated by practical scenarios (lines 45--53). These incentives are formal mathematical/logical constriants that are inspired by intuition like individual rationality and desirability in the literature [37, 47, 52].

F7 and F8 are human-invented properties.

Why is it reasonable to set higher rewards for earlier participation? I understand the intuition, while a formal mathematical formulation is needed to motivate this setting. That is, there should be an objective with physical meaning, solving which we can obtain that "higher rewards for earlier participation".

We respectfully suggest that a formal mathematical formulation or objective with physical meaning may not be strictly necessary to motivate the incentives and setting considered in our work. Incentives and other properties desirable to humans may not be naturally occurring (i.e., solution to objective with physical meaning) but should be artificially guaranteed to realize benefits through well-designed solutions, as often done in economics and game theory. For example, in the seminal work on Shapley value [46], the author proposed the axioms of symmetry, efficiency (total payoff must be the value of the game) and additivity (when two independent games are combined, their values must be added party by party) and showed that the Shapley value satisfies them. These axioms turn out to be useful in many applications such as in data valuation [13] where we want to attribute model accuracy to individual data points. As another example, fairness (equal rewards for equal contribution; higher reward for contributing more valuable data) [39, 47] does not naturally occur but satisfying it would encourage parties to contribute more.

It is "reasonable" to set higher rewards for earlier participation because it is desirable in ML applications (as we have explained in Sec. 1 and App. A). Incentives for earlier participation must be designed and provided precisely because parties will not naturally do so. We do define the incentives formally as math equations in Sec. 4.

F1 and F2 are basically the same concept. People usually use individual rationality without mentioning non-negativity.

We wish to clarify that F1 (non-negativity) and F2 (individual rationality) are not the same concept as F1 ensures all parties receive non-negative rewards while F2 ensures the rewards are at least as valuable as each party's own data. However, if we assume non-negativity of the valuation function (line 205), then F2 implies F1. We still include F1 as one of the incentives as it is only implied under this extra assumption in a later section (Sec. 5), and it is convention in the literature as seen in [47, 52].

In F5, "improving model quality" is non-rigorous. As mentioned earlier, the data together with the model was not characterized in the formulation.

We wish to clarify that in lines 118--120, we define $v_C$ as the model performance/quality achieved by training on the aggregated dataset $D_C$. Accordingly, in F5 (uselessness), the condition $v_{C \cup \{i\}} = v_C$ implies that $i$'s data fail to improve the model quality.

How is gamma tuned?

We demonstrate the effects of different hyperparameters ($\beta$ and $\gamma$) in our experiments (Sec. 7). In practice, the trusted mediators (e.g., a data sharing platform) can test out different values of $\beta$ and $\gamma$ to guide future decisions when the valuation function is the same (e.g., dual of accuracy).

Concretely, the mediator can start by setting $\beta=\gamma=1$, which is shown to provide a reasonable trade-off between incentivizing early participation and encouraging high-quality data in our experiments (Sec. 7). For future collaboration, the mediator can then adjust these values based on observed behavior of parties: decrease $\beta$/increase $\gamma$ if early participation is sparse; increase $\beta$/decrease $\gamma$ if the majority of data received seems rushed and of low-quality. In addition, mediators can share their empirical experiences with parameter tuning (without exposing underlying data) to form best practices over time.

The approach should be compared with existing works. Although there may not be works considered similar settings, some incentive works should be compared with to show the enhancement for addressing heterogeneous arrivals.

We appreciate the reviewer's understanding that there may not be works tackling similar settings. We would like to clarify here and in our revised paper that comparisons with classical non-time-aware reward schemes are already included in our experiments (lines 318--322). Specifically, existing methods that do not account for joining time [41, 47] can be seen as special cases of our methods when $\beta \to \infty$ and $\gamma=0$. The baselines correspond to the horizontal lines in our experiment results (e.g., Fig. 3b and Fig. 7a-b), which suggest that they assign constant rewards regardless of joining time, thereby fail to satisfy incentive F8 (time-based strict monotonicity).

The approach proposed in this work is relatively standard. Again, it is not actually related to data sharing, because those key features were not characterized in the formulation.

We have addressed the concern regarding IID/non-IID data and improving model quality by clarifying that data is valued relative to the model in our formulation. If there are remaining concerns, we would appreciate if the reviewer could clarify which specific "key features" of data sharing they believe are missing from our formulation, and how these differ from the key incentives in data sharing outlined in Sec. 4, or the various data distributions (both IID and non-IID) considered in our experiments (Sec.7), so that we can respond more precisely.

We would like to emphasize that our approach is related to data sharing as it addresses a key underexplored practical problem in data sharing: When parties share their data at heterogeneous times, how should we fairly reward each party based on both their time of arrival and value of data? A key contribution of our work lies in formulating and posing this problem itself.

We would also like to clarify that we propose two reward schemes that incorporate the joining time information to address this problem. These schemes generalize the standard coalition game approaches (e.g., the Shapley value) by embedding the time information in the reward cumulation stage (Sec. 6.1) and the valuation stage (Sec. 6.2). Importantly, both schemes provably satisfy all the proposed incentives, this also poses additional technical challenges.

Thank you for your comments! We hope our clarifications above can improve your opinion of our work.

We sincerely thank the reviewer for taking the time to review our work and for recognizing that we are tackling a realistic setting with clear motivation. We address your questions below:

Q1. I am curious about how to set the parameters in practice to control the emphasis on joining times and early participation. A user may want to know these parameters...

We agree that the parameters should be broadcasted before the collaboration starts. In practice, the trusted mediators (e.g., a data sharing platform) can test out different values of $\beta$ and $\gamma$ to guide future decisions when the valuation function is the same (e.g., dual of accuracy).

Concretely, the mediator can start by setting $\beta=\gamma=1$, which is shown to provide a reasonable trade-off between incentivizing early participation and encouraging high-quality data in our experiments (Sec. 7). For future collaboration, the mediator can then adjust these values based on observed behavior of parties: decrease $\beta$/increase $\gamma$ if early participation is sparse; increase $\beta$/decrease $\gamma$ if the majority of data received seems rushed and of low-quality. In addition, mediators can share their empirical experiences with parameter tuning (without exposing underlying data) to form best practices over time.

Q2. I understand that it may not be possible to guarantee the strong theoretical properties of the reward values under the online FL setting. However, it would be helpful to see a simple extension of your method to this setting along with empirical results.

We appreciate the reviewer's understanding that extending to the FL setting may violate the theoretical properties of our methods (lines 560--563). Notably, in an online FL setting, client contributions must be evaluated dynamically, and reward values must be updated and distributed during each communication round. This introduces a key practical issue: once rewards are distributed, it becomes difficult to claw back or revise them to ensure overall fairness if the reward values change when new clients join (lines 580--582).

Nevertheless, we show how to adapt our time-aware reward cumulation method (Sec. 6.1) to dynamically update the reward values of each client during each communication round of the online FL setting. We wish to clarify that this adaptation experiment is primarily intended to showcase the feasibility of extending our method to other settings, and the empirical results may not fully satisfy the theoretical properties established in the main paper.

Adapting Time-Aware Reward Cumulation to online FL

Denote $r_i^{(t)}$ as the reward value of client $i$ at communication round $t$. We update $r_i^{(t)}$ via a running average of Shapley values weighted by the penalizing factor $\beta$ on joining time: $r_i^{(t)} = \frac{r_i^{(t-1)} \cdot t + \phi_i^{(t)} \cdot \beta^t}{t + 1}$ where $\phi_i^{(t)}$ is the modified Shapley value defined in line 246. For computing $\phi_i^{(t)}$ in the FL setting, the valuation function we use is the dual of $v_{FL}^{(t)}$, the validation accuracy of the weighted averaged model at round $t$, analogous to FedAvg [38]: $v_{FL}^{(t)}(C) = acc \left( \sum_{i \in C} \theta_i^{(t)} \cdot \frac{n_i}{\sum_{j \in C} n_j} \right).$

Experiment Setup

We conduct preliminary experiments on CIFAR-10 using FedAvg with ResNet-18 for 8 communication rounds. The training data is assigned to 3 clients as follows:

• Client 1 receives a random half of the dataset;
• Client 2 receives all data from 7 randomly chosen labels in the remaining half;
• Client 3 receives the rest.

Clients 1 and 2 always join at round $t=0$. We vary the joining time of client 3 and assume, following [60], that clients remain in the FL process once joined. Client 3 will train on its own data until it participates in the FL process. We set $\beta=0.98$ for a moderate penalization on late participation.

Experiment Results

We first compare the reward values of each client at each communication round when all clients join at $t=0$:

We observe that $r_1^{(t)}$ > $r_2^{(t)}$ > $r_3^{(t)}$, which is consistent with how we assign each client with their data: Client 1 has the most valuable data and client 3 the least. This reflects fairness and equal-time desirability (F4). As we are working with non-IID setting, a client's gradient update may degrade the global model performance in early communication rounds. Hence, we can see some negative entries early on which may violate non-negativity (F1) and individual rationality (F2), but this is expected as the adapation no longer enjoys the full theoretical guarantees, and the results improve in later communication rounds/near convergence.

We then investigate how delaying client 3' participation affects its reward values:

We observe that as client 3 joins later, its reward value generally decreases, reflecting time-based strict monotonicity (F8) and demonstrates that our method disincentize clients from delaying their participation. Some deviations (e.g., when client 3 joins at $t=2$) are expected, as the adaptation no longer enjoys the full theoretical guarantees. However, the overall trend remains consistent.

While our adapation method demonstrates promising empirical results, we view this as a first step toward broader applicability. Future directions include (i) developing a theoretically sound method for the online FL setting, and (ii) designing mechanisms to retroactively adjust rewards to address the aforementioned "claw back rewards" challenge.

Thank you for your comments! We hope our clarifications above can improve your opinion of our work.

We sincerely thank the reviewer for taking the time to review our work and for recognizing that our problem is motivated by practical concerns, and the claims are backed by both theoretical and empirical results. We address your questions below:

there is no comparison between the proposed time-aware reward scheme and classical reward schemes that do not take the time-awareness into account

We would like to clarify here and in our revised paper that comparisons with classical non-time-aware reward schemes are already included in our experiments (lines 318--322). Specifically, existing methods that do not account for joining time [41, 47] can be seen as special cases of our methods when $\beta \to \infty$ and $\gamma=0$. The baselines correspond to the horizontal lines in our experiment results (e.g., Fig. 3b and Fig. 7a-b), which suggest that they assign constant rewards regardless of joining time, thereby fail to satisfy incentive F8 (time-based strict monotonicity).

Is there any redundancy among the proposed eight time-aware reward incentives? Does complying with a subset of them imply certain compliance on others?

Not with additional assumptions. If we assume non-negativity of the valuation function (line 205), then F2 (individual rationality) implies F1 (non-negativity of rewards). We still include F1 as one of the incentives as it is only implied under this extra assumption in a later section (Sec. 5), and it is convention in the literature as seen in [47, 52].

Additionally, in Remark 4.1, we show that F3 (equal-time symmetry) and F7 (time-based monotonicity) together imply time-based equal-value desirability, a generalization of the classical desirability incentive [47] to the time-aware setting. We did not include this as a separate incentive to avoid redundancy.

Thank you for your comments! We hope our clarifications above can improve your opinion of our work.

We sincerely thank the reviewer for taking the time to review our work and for recognizing that our problem is new and highly relevant to the real-world. We address your questions below:

Q1. ...it suffices for either i or j to join the collaboration to get a non-zero value model...

We believe that there may be a misinterpretation of F6 (necessity). The precondition in F6 requires both i and j to be in coalition C for its value $v_C$ to be non-zero. This means that i and j are necessary to one another, and neither can achieve a non-zero value collaboration without the other. Therefore, it is appropriate to assign equal rewards to party i and j, as it ensures that both parties are incentivized to participate, regardless of who joins earlier.

Q2. ...this will depend (in a complicated fashion) on how much "better" the data set is and how the parameters \beta (or \gamma) are tuned. Is this correct?

Yes, this is correct, and we appreciate the reviewer's clarification. We will revise the sentence by adding conditions under which it is true after the original bolded sentence, as follows:

A party may get a higher reward from taking time to curate a higher quality dataset instead of sharing a less valuable dataset as early as possible. This holds in two key scenarios: (i) When a greater emphasis is placed on data quality relative to joining time, e.g., by setting a large $\beta$ or small $\gamma$. (ii) When the parties are necessary parties, i.e., their data is essential for achieving non-zero value collaboration.

Q3. Can you elaborate more on the aspect of rewards...

We first decide a target reward value and seek to realize a model with the target reward value (Sec. 6.3). Note that the reward value is not always measured using the accuracy function.

For example, in Fig. 3,6 and Fig. 5, we seek to control the conditional information gain. This is possible to achieve exactly and efficiently based on the likelihood tempering approach proposed in [49], which we outline in App. G.

In Fig. 7, we seek to control the model validation accuracy. This is slightly harder to control and we propose to use the subset selection method (details in App. G) to achieve this approximately. [39] has also proposed early stopping to achieve different accuracies when training a neural network.

We will add the clarification in our revision.

From the technical side, the paper seems pretty direct.

We would like to clarify that while the use of Shapley value may appear straightforward in isolation, mathematically formulating all the proposed incentives in our time-aware setting (Sec. 4) is non-trivial. This includes both generalizing existing fairness notions to our time-aware setting and formalizing a new time-aware fairness perspective. Furthermore, we propose two schemes for deciding reward values (Sec. 6) that provably satisfy all the proposed incentives, this also poses additional technical challenges. We also wish to reiterate that a key contribution of our work lies in posing the problem itself. The time-aware data sharing and collaborative learning problem is underexplored despite its strong relevance to many real-world applications.

However, given that the model is new, highly relevant to the real-world setting, I would say that overall I think this is a decent submission to NeurIPS.

We are encouraged that the reviewer also recognizes the importance of this problem, and we sincerely appreciate the acknowledgment.

Thank you for your comments! We hope our clarifications above can improve your opinion of our work.