Spectral Self-supervised Feature Selection

1. The paper focuses on a traditional unsupervised feature selection task and tries to improve a classical feature selection method Laplacian score. I'm not sure whether this paper can attract wide interest in learning representations community. Maybe this paper is more approriate to the ICML community.
2. The compared methods are out of date. Since feature selection is a very classical task in machine learning, there are a lot of feature selection methods proposed every year. However, most of the compared methods are proposed about 10 years ago, which cannot demonstrate the effectiveness of the proposed method. The paper should compare with plenty of most recently state-of-the-art methods.
3. It is simple to give an example to show that use the k leading eigenvectors is suboptimal. But could you provide some more theoretical analysis about the selected k eigenvectors of the proposed method?

1. Absence of Theoretical Justification: The paper lacks rigorous theoretical analysis supporting its central claim, specifically that "certain features might not significantly affect supervised learning, they can heavily impact unsupervised tasks." In the absence of a theoretical framework, the claim rests primarily on empirical observations. To further the contribution, the authors should delve into a theoretical discussion, offering insights into why and under what conditions their claim holds.

2. Ambiguity in Real-World Data Analysis: While the paper demonstrates superior performance against other models by potentially extracting lower-ranked but more informative features, it does not provide direct evidence of this assertion in its real-world-data experiments. A more thorough investigation or a controlled experiment showing how these "lower-ranked but more useful features" contribute to better performance would substantially strengthen the paper's claims.

3. Inconsistencies in Benchmark Results: The discrepancies between the accuracies reported in this paper and those in prior works like "Nonnegative Discriminative Feature Selection (NDFS)," specifically for datasets like GISETTE, raise concerns about the reproducibility and fairness of the comparisons. The significant difference in accuracy (66.4% in this paper vs. 74.4% in the NDFS paper) cannot be dismissed as a minor discrepancy. It's crucial to clarify if the datasets used in both papers were identical, if any pre-processing steps differed, or if there were variations in experimental settings. The authors must either provide an explicit justification for these discrepancies or conduct their experiments under the same conditions as prior works to ensure a fair comparison.

The presentation is not clear at times. For example, it takes a while to understand the motivation of the approach, which is done a bit by "reverse-engineering": once it is clear that a methodology to evaluate eigenvectors independent of the eigenvalues is established, the motivation for such a choice is presented via experiments. This could be more clearly stated in the introduction by contrasting what is being proposed here (at a high level) with the approaches from the literature.

The proposed approach can be judged as ad-hoc; it is not clear when is it necessary to search eigenvectors beyond the top choices.

The flexibility of the proposed approach leaves some questions that are not fully addressed, such as how to select proxy models to use in each one of the two steps where they are involved.

Minor comments:

In Section 2.1 ISOMPAS should be ISOMAP. Algorithm 1 line 1: should "pseudo pseudo-labels" be "pseudo-labels"? And "features variance" be "feature variances"?

The proposed method builds on the Multi-Cluster Feature Selection (MCFS) framework, where feature selection relies on predicting pseudo-labels from spectral clustering. However, its distinct contributions compared to existing MCFS methods remain ambiguous.

On Page 5, the mathematical formulations, including the definition and indexing of $s_m$ are unclear. I am also sure the distinctions between $h_{i, b}$ and $h_i$ and the differences between $s_m$ and $\bar{s}_m$. Additionally, in Section 3.3, I do not understand how the $k$ models are trained how they relate to each other, and their relevance to feature selection.It is also challenging to understand the rationale behind the definition of $score(m)$.

In the experiments, image and speech data are tested. I wonder whether a feature extraction method is applied to the data in the pre-processing. I doubt whether a feature selection method would work on the raw image and speech data.

I have several concerns about the evaluation method. The paper claim to follow Cai et. al, but to me there seem to be important differences between their methodology and the reference, though their method seems closer to Li 2012, which I also think uses somewhat questionable evaluation metrics.

Evaluating unsupervised methods is quite difficult, and the paper makes a good effort for a fair and thorough evaluation, but parts of the method are unclear and other parts seem questionably. In particular, I would prefer the use of adjusted rand index or possibly normalized mutual information instead of clustering accuracy, as more standard metrics. Second, because of the randomness involved in many aspects of the algorithm, I would appreciate it if there were repetitions of the results, and standard deviations reported. Using the maximum performance over several different numbers of features increases the potential variability of results. I did a simple experiment to understand the nature of the numbers, on the TOX-171 dataset. Randomly sampling an ordering of features, I used the same evaluation methodology used in the paper, and obtained results between an accuracy of 40% and 48% depending on the run. This means that completely random selection is likely indistinguishable from half of the methods considered on this dataset (though the proposed method is performing much better). If the algorithms are re-run for each number of features, which is not clear to me from the paper, the baseline would be picking a new random order for each of the number of features. This produces relatively robustly a score of 45% on this dataset, beating two of the competing algorithms. I'm uncertain whether the protocol of picking the maximum over different numbers of features is a good practice, but given that, the paper should make clear if the algorithm is re-run for each number of features, or if the order of features is fixed. Also, the paper should ideally include the scores of a random baseline, and an oracle result, i.e. what is the best possible result that can be achieved (say using the real labels as pseudo labels - another more expensive and potentially better scoring oracle would be to go over all subsets and run k-Means but that would be prohibitive). Interestingly, SSFS is close to the oracle performance on TOX-171 based on my analysis, which is quite impressive and somewhat surprising to me.

Finally, the paper does not seem to explain the selection of the kernel hyper-parameters. This is quite critical, in particular the gamma parameter when using the rbf kernel. It's not clear to me what practice I would propose for that, but whatever methods was used should be described in the paper.

minor points: In the second paragraph of the introduction, "lack of labels" should probably be "absence of labels". I think section 3.1 could be shortened since the notation is not used afterwards, and in essence you are stating that in a disconnected graph, the eigenvectors to eigenvalue 0 of the laplacian are indicator functions of the connected components.