Performance Adjustment for Federated Learning Marketplace

Please see above

1. The denomination between monetary quotation and model improvement is not designed and analyzed carefully. As a high-level question, how much money should correspond to the model performance increment from 90% to 91% for a specific client’s validation data?
2. From Eq. (7), I can see that monetary quotations from all clients are distributed back to clients depending on the contributions $B$. However, this monetary reward is not linked to performance guarantees and may make clients lose incentive. To elaborate, even if the monetary quotation is high for a client (i.e., the client is willing to spend much money for an improved performance), but it is entirely possible that the model for the client cannot be further improved (e.g., one possibility is that other clients’ data is not helpful). In this case, the money quotation from the specific client still gets distributed to others, when there is no model improvement. How do you justify this case under your design?
3. Firstly, I believe there is a typo for the green legend in Figure 4 (left): The green label should be for “10:0” right? From this figure, we observe that client 1 gets a lower F1 score when the quotation is 10 (green circle) as compared to when the quotation is 5 (blue circle). I.e., client 1 gives more money but gets a worse model. This reinforces my Question 2 about having no guarantee for model performance with respect to monetary quotation. Could you justify this case? Could you explain the client’s incentives and behaviors, too?
4. In the paper, the training aggregation weights $\beta_i$ are treated as hyper-parameters. However, they are closely linked to the final model trained as well. Why can’t they be considered as designs to favor specific clients, just like what $\alpha$ does?
5. There is no convergence result for the algorithm proposed. In fact, the authors wrote in section 3.3 that “$\alpha_i^{(t)}$ can be changed in different iterations”. How can you assure convergence when the loss (or, objective) is constantly changing? Does the objective converge as well? If no theoretical convergence can be given, then the paper becomes a lot less convincing.
6. The design choices for SHA and FedEx need to be better justified. Could you elaborate on why these methods fit your specific incentive mechanism? Or, any other FedHPO method could possibly work too?
7. Wouldn’t a client benefit a lot more by giving a very difficult, or potentially adversarial (e.g., wrong label) validation points? In this case, the other clients are unlikely to improve on this client’s validation dataset. Therefore, this client would not need to pay other clients (or pay very little) and obtain more money back. Even worse, when every client is now incentivized to state adversarial validation datasets, the federated training process can be adversely affected.
8. The assumptions 2) and 3) in Property 2 sound very stringent to me, it might make the property much less useful in practice.
9. The empirical experiments are not comprehensive enough. The results are not convincing with only 3 clients, could you produce results for at least 50 federated clients with data splits following standard heterogenous FL baselines/benchmarks? The dataset used (CIFAR-10) is also very well-behaved. Can you show experiments with larger-scale datasets like ImageNet and Sentiment-140?
10. I see in Table 3 of Appendix B.2 that the aggregation weights candidates are a discrete set of potential values. Could you optimize a continuous vector for it instead?
11. There are existing works that have looked into model incentives (instead of monetary incentives), but none of them are cited or discussed in the paper. One example would be [1].

[Minor]

12. It should be made clear that (3) is not a direct reduction of (2). Please correct me if I am wrong.
13. In the fourth line of the “Computation of MBS.” paragraph, there is a typo. It should be $w_{\mathcal{C} \setminus i}^{(t)}$ instead of $w_{\mathcal{C} / i}^{(t)}$.
14. What is the reason for creating an influence score matrix $\mathbf{B}$ after Eq. (4)? You are going to add them over the clients anyway, i.e., in Eq. (5).

References

[1] Gradient-Driven Rewards to Guarantee Fairness in Collaborative Machine Learning. NeurIPS, 2021.

See weaknesses above.

Please refer to "Weaknesses".