Choose Before You Label: Efficient Node and Data Selection in Distributed Learning

pls see weaknesses

• Why is sharing the label fractions of nodes infringing privacy and why does loneliness as a metric reveal less about the data than label fractions?
• What if a local dataset is not selected for its loneliness score but it would be crucial for the performance on the overall distribution to train on it? Don't we run the risk of overfitting on a subset of "easy" datasets?
• Which norm is used in Eq. 1? Isn't it problematic to use a simple norm, such as $L_2$, for high-dimensional inputs, since norms become similar in high dimensions? That is, for large image data, it is likely that all images are equally lonely. (do they perform a good evaluation of their metric?)
• Why use the minimum over the entire dataset? This means I can set the loneliness of any dataset to 0 by simply cloning one datapoint.
• Why is it a good thing that loneliness does not depend on a label?
• Why is loneliness better than things like local outlier factor (cf. [1,2,3] f, or metrics like QI^2 [4] or ECS [5]?
• MNIST is probably unsuitable for a realistic evaluation of loneliness, since nearest neighbors suffice to explain the label in that case (k-nearest neighbour achieves SOTA performance on this dataset). It is likely that loneliness performs worse if the structures in input space are more complex.
• Why would you measure loneliness in input space? The goal of representation learning is to transform the potentially complex input space into a more benign feature space. Thus, high loneliness in input space might be meaningless if we can train a neural network to map into a feature space with more benign properties (cf. [6]).
• In your experiments, you use a Dirichlet distribution to simulate label heterogeneity with $\alpha\in [1,5]$, which corresponds to fairly mild heterogeneity. It would be more interesting to see results for $\alpha << 1$, e.g., the typically used $\alpha = 0.01$ in most heterogeneous FL papers.
• There is no systematic evaluation of loneliness as a metric, i.e., analyzing how the metric behaves for different datasets, how it changes with noise added to features, or other perturbations that might affect federated learning.
• The theoretical result you rely on basically says that the more outliers a dataset has, the harder it is to fit (since the KL divergence is $-\log(Q_W(W^*))$ which is simply model complexity at the optimum). This is fairly unsurprising. Also, it seems that for larger datasets, both KL divergence and loneliness become less useful as predictors of model performance.
• The Goldilocks procedure assumes that we already know an ideal value of the metric we evaluate. This, however, is the hard part. Selecting nodes or datapoints with as close an ideal value as possible is trivial. So without a method to infer upon the optimal value $\tau$, the approach only works if we apriori know the optimal value of the metric. Why is this a realistic assumption? How sensitive is the approach to getting that value right? And how can we estimate it in practice in a realistic application scenario?
• If small datasets are of concern, the paper should compare node and dataset selection with baselines for this scenario, such as FedDC [7].

References:

[1] Alghushairy, Omar, et al. "A review of local outlier factor algorithms for outlier detection in big data streams." Big Data and Cognitive Computing 5.1 (2020): 1.

[2] Wang, Gang, and Yufei Chen. "Robust feature matching using guided local outlier factor." Pattern Recognition 117 (2021): 107986.

[3] Xu, He, et al. "Outlier detection algorithm based on k-nearest neighbors-local outlier factor." Journal of Algorithms & Computational Technology 16 (2022): 17483026221078111.

[4] Geerkens, Simon, et al. "qi 2: an interactive tool for data quality assurance." AI and Ethics 4.1 (2024): 141-149.

[5] Sieberichs, Christian, et al. "ECS: an interactive tool for data quality assurance." AI and Ethics 4.1 (2024): 131-139.

[6] Petzka, Henning, et al. "Relative flatness and generalization." Advances in neural information processing systems 34 (2021): 18420-18432.

[7] Kamp, Michael, Jonas Fischer, and Jilles Vreeken. "Federated Learning from Small Datasets." Eleventh International Conference on Learning Representations. OpenReview. net, 2023.

• What is the rationale behind for defining the loneliness level $\lambda(X^k)$, as shown in Eq. (3)?

• Why not directly use the KNN distance to calculate simple-wise loneliness, $\ell(i, k)$? From my understanding, $\ell(i, k)$ could be interpreted as a 1-NN distance. In general, using a larger K could provide more stability than relying on just a single neighbor.

• The dataset-level loneliness is defined as the smallest sample-wise loneliness, which may be less meaningful in practice. In a relatively large dataset, the majority of samples tend to have similar feature distributions, making the minimum value an unreliable measure. Using the average loneliness value would likely provide a more reasonable and representative metric.

• What is the exact definition of metric $\hat{\mu}$? The explanations provided in line 321 is unclear.

See weaknesses