Mycroft: Towards Effective and Efficient External Data Augmentation

• RetrieveTopK function in Algorithm 3 is confusing. The textual explanation suggests that this function is a sorting "greedy heuristic." However, it is hard to understand how this function is "ensuring coverage of $D^{hard}$."
• Algorithms 4 and 5 are hard to understand without textual description.
• The inequality below Equation (5) in the proof of Theorem 3.1 does not match the definition of $e_\lambda$ (Equation (1)). From what I see, the proof suggests that the first gradient-matching term in Equation (1) is squared.
• I do not clearly understand how sparsity implies the inequality below Equation (5).
• The definition of $f_{\lambda}$ in the proof of Theorem 3.1 does not match the definition of $f_{\lambda}$ in Section 3.
• The description of OMP by Elenberg et al. (2016, Algorithm 3) does not result in Algorithm 2 because Algorithm 2 does not account for the gradient of the feature-regularization term, $\lVert \Psi w \rVert^2$.
• Section 4 does not specify the size of $D^{hard}$, which makes the interpretation of the results hard.
• Sections 4 and 5 evaluate the final model only on the subset of $D^{hard}$. While I see how this metric reflects data relevance, it is also important to know how the additional data affects the performance on the whole dataset $D^{val}$ or $D^{test}$, especially for experiments in Section 5.
• Similarly, it is unclear in which experiments the final evaluation happens on the held-out portion of $D^{hard}$ and in which experiments the evaluation happens on the whole $D^{hard}$. Thus, it is impossible to know whether the method overfits to $D^{hard}$.
• The paper does not conduct experiments in settings where data privacy is protected (e.g., with any technique from Appendix G). Since privacy is a key concern in data sharing, judging the method's efficacy without such experiments is hard.

• It is not clear about the motivation of this paper, privacy protection, or others, it also lacks some real scenarios, it makes me feel like dataset condensation or distillation.
• One of the challenges in 2.1 is the reluctance to share data due to privacy, so why can the MySoft framework confirm that they are willing to share a small amount of data? There should still be privacy in there. These challenges are also related to FL.
• The problem this paper investigated do not seem to be very real needs
• What’s the performance of MySoft and some other dataset distillation methods, that DOs can distill their dataset into a subset that best represents the local data?
• This hypothesis assumes that MT has learned the information it needs to improve before, but has no good performance yet. If MT needs to test categories it has never seen before, is MySoft still useful. According to my understanding, when MT and DO data are similar, reasonable proxy can help select useful data. What will happen when new task data is needed or the domain gap between reasonable proxy and the original model is large?

• There is not much in this paper that has not already been presented in other papers. I see this paper as more of a formalizaton of a problem and then a survey of the existing techniques for that problem. While that is useful it is not substantial enough for conference publication.
• It would be helpful for the authors to lay out exact real life applications where this setup is happening / being suggested
• Overall, I think the results are interesting and in line with what prior works have demonstrated in terms of using these algorithms for the outlined problem.

1.Limited Novelty in Data Selection Techniques: The core techniques (functional similarity via gradient matching and feature similarity) are adaptations of existing methods, which limits the novelty. While combining them is beneficial, this integration may not be sufficient to stand out as a fundamentally new approach. 2.Scalability Concerns: Although the authors provide complexity analyses, the scalability of Mycroft, especially with high-dimensional datasets, remains a concern. For larger datasets, the computational requirements for gradient-based data selection could become prohibitive. 3.Potential Vulnerability to Strategic Data Owners: The paper briefly mentions the possibility of strategic or malicious data owners, but it does not explore how Mycroft would handle cases where data owners intentionally provide misleading subsets.