Unsupervised Feature Selection using a Basis of Feature Space and Self-Representation Learning

Response to Questions:

[Question1]. In the section of methodology, the equation (6) is confusing and not so clear. It seems impossible to subtract the matrix XUV of shape mm from the matrix X with the shape mn?

[Response]. With respect to the second comment of the previous part, we have adjusted the way we introduce the matrices $U$ and $V$ after formula (6) by incorporating their respective sizes. This subtraction is feasible because the matrix $X$ is $m \times n$, matrix $U$ is $n \times k$, and matrix $V$ is of size $k \times n$. Therefore, the confusion has been resolved by including the dimensional information of the matrices.

[Question2]. As the feature matrix B is fixed by the VBE method proposed in section 3.3, it is unclear why the basis coefficient matrix G in equation (7) is a parameter to be optimized. Why the matrix G can not be determined by equation (4) directly and reduce the number of parameters.

[Response]. This is an insightful question, and initially, we believed it could aid in reducing computations by fixing the basis matrix $\mathbf{B}$ and computing $\mathbf{G}$ from Equation (4). However, it's essential to note that Equation (4) constitutes the first term of our objective function in Equation (8). To be more precise, in addition to relation (4), the matrix $\mathbf{G}$ simultaneously plays a crucial role in preserving the geometrical information of the data in the final equation of subsection 3.1 and is actively involved in the subspace learning process in the concluding equation of Subsection 3.2. Therefore, to ensure that all parts of the objective function in (8) decrease during the updating process, we must consider $\mathbf{G}$ as a part of the whole and update it within the general framework, as we did. If we only compute $\mathbf{G}$ based on equation (4), we can only guarantee the decrease of the first term of the objective function.

[Question3]. In section 3.1, subspace learning that introduces graph regularization seems to be existing methods. Should this part of the content be moved to related work?

[Response]. The reviewer highlighted the graph regularization term within the framework of manifold learning, which is an established method that has been widely used in dimensionality reduction techniques. However, the novelty of our regularization term lies in establishing a connection between the concept of the basis matrix and the geometrical structure of the data. Specifically, we introduce a new term, $\mathrm{Tr}\left(\mathbf{B}\mathbf{G}\mathbf{L}\mathbf{G}^T\mathbf{B}^T\right)$, where the basis matrix $\mathbf{B}$ plays a crucial role, facilitated by the coefficient matrix $\mathbf{G}$. Therefore, as a consequence of our self-representation framework in Section 3.1, we used the basis matrix $\mathbf{B}$ for preserving the geometry of data.

Response to Weaknesses:

[Weaknesses1]. In the end of Introduction section, the second and third contribution points is not sufficient, as these constraints of regularization are not proposed in this article.

[Response]. We thank the reviewer for raising this point. In response to the reviewer's comment, we would like to mention that we have revised the contribution points of our proposed method. The main contributions of the paper are summarized below:

1- To the best of our knowledge, GRSSLFS is the first feature selection method that applies a basis of linearly independent features into a unified framework of subspace learning and self-representation.

2- GRSSLFS characterizes the challenges of subspace learning and self-representation by utilizing a basis matrix derived from the feature space. The objective is to eliminate unnecessary features and identify informative ones within high-dimensional data.

3- GRSSLFS constructs a basis for the feature space by incorporating the variance information of features into the Basis Extension method. The goal is to create a feature basis with the highest variance score. This ensures that the basis for the feature space comprises elements with the most dispersion in space, thereby minimizing the accumulation of features.

[Weaknesses2]. In the methodology section, some formula calculations are confusing and not very convincing. Such as the multiplication in formula (6) and the optimization target in the optimization goal (formula 7). These issues will be described in detail in subsequent questions 1 and 2.

[Response]. We greatly appreciate the reviewer for providing valuable comments that helped clarify certain aspects. To address the concerns raised, we thoroughly reviewed all relevant matrices involved in the formulations (6) and (7) and included their sizes which were omitted in the initial submission. Specifically, we appended the sizes of matrices $\mathbf{U}$ and $\mathbf{V}$ after equation (6) and the size of matrix $\mathbf{B}$ before equation (7).

[Weaknesses3]. In the methodology section, the description of the algorithm is not complete enough. The specific process of selecting features according to the matrix U in Algorithm 1 has not been described in detail.

[Response]. We thank the reviewer for raising this issue. Regarding this comment, we have added an explanation about choosing the features according to the matrix $\mathbf{U}$, in the revised manuscript, just after the main problem in equation $(9)$ on page 6. Please note that, as we explain in Section 3.2 on subspace learning and based on the term $\|\mathbf{X}-\mathbf{XUV}\|$, which is reflected in $\|\mathbf{B}-\mathbf{BGUV}\|$ in our basis-based model, we can interpret the matrix $\mathbf{X}\_I=\mathbf{XU}$ as the approximation of selected sub-matrix. We have $\mathbf{X}\_{I}=\mathbf{X} \mathbf{U}=[\mathbf{f}\_{1},\ldots,\mathbf{f}\_{n}]\mathbf{U}=\mathbf{f}\_{1}\mathbf{u}^{(1)} + \cdots + \mathbf{f}\_{n}\mathbf{u}^{(n)},$ where $\mathbf{u}^{(i)}, i=1,\ldots,n$ are the rows of the matrix $\mathbf{U}$. The above representation indicates that each row of the matrix $\mathbf{U}$ can be interpreted as a weight in displaying the sub-matrix of selected features. So, we use the norms of the rows of the matrix $\mathbf{U}$ as the importance measure of each feature (column) of the input data $\mathbf{X}$.

Dear Reviewer, We kindly request your prompt review of our responses as the deadline is in a few hours.

Thank you for your timely assistance

We thank the reviewer for raising this point. According to the self-representation formulation (4),
$\min\_{\mathbf{G}\in\mathbb{R}\_+^{m\times n}}\Vert\mathbf{X}-\mathbf{BG}\Vert\_F,$ and letting $\mathbf{G}=[\mathbf{g}\_1,\ldots,\mathbf{g}\_n]$, we get

$

\mathbf{X}\simeq\mathbf{BG}\Rightarrow[\mathbf{f}_1,\mathbf{f}_2,\ldots,\mathbf{f}_n]\simeq[\mathbf{B}\mathbf{g}_1,\ldots,\mathbf{B}\mathbf{g}_n].

$

Therefore, each feature vector $\mathbf{f}\_r$ is approximated with the vector $\mathbf{Bg}\_r$.

To address the reviewer's comments, we have clarified the meaning of similar structure by adding a description in the main paper that our metric for measuring the similarity between features relies on their geometric distance in space. Specifically, to incorporate the geometric structure of the data, the elements of the similarity matrix $\mathbf{A}=[a\_{qr}]\in\mathbb{R}^{n\times n}$ are constructed as follows:

$

\mathbf{a}_{ij} = \begin{cases} e^{\frac{-\Vert \mathbf{f}_{i} - \mathbf{f}_{j} \Vert_2^2}{t^2}}, & \text{if}\quad \mathbf{f}_{j} \in N_{k}(\mathbf{f}_{i})\quad \text{or} \quad \mathbf{f}_{i} \in N_{k}(\mathbf{f}_{j}) \ 0, & \text{otherwise,} \end{cases},

$

where $N\_{k}(\mathbf{f}\_{i})$ refers to the set of $k$ feature vectors that are closest to $\mathbf{f}\_{i}$, and $t$ denotes the scale parameter of the Gaussian distribution. Please note that, by this definition, if the vectors $\mathbf{f}{q}$ and $\mathbf{f}{r}$ are close, then the value of $a\_{rq}$ is large, and vice versa. One advantage of this formulation, based on the k-nearest neighborhood approach, is that it makes the matrix $A$ sparse, and the computational complexity of the manifold learning process is considerably reduced.

In the theory of manifold learning, it is commonly assumed that if two vectors are similar to each other in the original space, then their representations in the transformed space should also be close. Based on this observation, if the two features $\mathbf{f}\_{q}$ and $\mathbf{f}\_{r}$ are similar, it is expected that their mapped vectors $\mathbf{Bg}\_q$ and $\mathbf{Bg}\_{r}$ will be also similar. To this end, the problem of manifold learning for transformation can be defined as:

$

\min\_G \frac{1}{2}\sum\_{q=1}^n\sum\_{r=1}^n\left\Vert  \mathbf{B}\mathbf{g}\_{q}-\mathbf{B}\mathbf{g}\_{r} \right\Vert\_2^2a\_{qr}

$

implying that $\left\Vert \mathbf{B}\mathbf{g}\_{q}-\mathbf{B}\mathbf{g}\_{r}\right\Vert$ must be small whenever $a\_{qr}$ is large (that means $\mathbf{f}\_{q}$ and $\mathbf{f}\_{r}$ are close). Computations show that

$

\frac{1}{2}\sum\_{q=1}^n\sum\_{r=1}^n\left\Vert  \mathbf{B}\mathbf{g}\_{q}-\mathbf{B}\mathbf{g}\_{r} \right\Vert\_2^2a\_{qr}
=&\frac{1}{2}\sum\_{q=1}^n\sum\_{r=1}^n\left(\mathbf{B}\mathbf{g}\_{q}-\mathbf{B}\mathbf{g}\_{r}\right)^T
\left(\mathbf{B}\mathbf{g}\_{q}-\mathbf{B}\mathbf{g}\_{r}\right)a\_{qr}\nonumber\\
=&\sum\_{r=1}^n\left(\ \mathbf{g}\_{r}^T\mathbf{B}^T\mathbf{B} \mathbf{g}\_{r}\right){p}\_{rr}-

\sum_{q=1}^n\sum_{r=1}^n\left(\mathbf{g}_{r}^T\mathbf{B}^T\mathbf{B} \mathbf{g}_{q}\right)a_{qr}\nonumber\ =&\mathrm{Tr}\left(\mathbf{B}\mathbf{G}\mathbf{L}\mathbf{G}^T\mathbf{B}^T\right)\nonumber\

$

in which $\mathbf{L}=\mathbf{P}-\mathbf{A}$ is the Laplacian matrix, where $\mathbf{P}=\mathrm{diag}(\mathbf{p}\_{11},\mathbf{p}\_{22},\ldots,\mathbf{p}\_{nn})$ with $\mathbf{p}\_{rr}=\sum\_{q=1}^n \mathbf{a}\_{rq}$, for $r=1,\ldots,n$.

We thank the reviewer for raising this point. In order to address the reviewer’s question, we would like to confirm that the variance of a feature measures its dispersion or range. When a feature exhibits high variance, it suggests that its values are widely spread over a broader range. In the context of the variance basis extension, altering the selection order of feature vectors based on their variance scores implies a focus on features with significant variability or dispersion.

However, it is important to note that while high variance indicates a wide spread of values for an individual feature, it does not necessarily mean that this feature contributes the most to the overall dispersion or variability in the dataset. The idea of the "largest amount of data dispersion" is better captured by considering the overall variance or variability of the entire dataset, which takes into account the combined effects of all features. In summary, high variance for individual features reflects their own level of dispersion, but identifying the features contributing the most to the overall data dispersion requires considering the collective impact of all features. For more information, we would like to mention that the theoretical investigations into this issue can be found in some related studies, such as ``Unsupervised feature selection based on variance-covariance subspace distance, Neural Networks, 2023."

We thank the reviewer for raising this point. In response, we should mention that considering the explanations provided to respond the Weaknesses section, in our algorithm for generating the basis (in our proposed Variance Basis Extension (VBE) method in subsection 3.4), we select the basis vectors $\mathbf{f}\_{i\_1}, \cdots, \mathbf{f}\_{i\_m}$ from the columns of the input data matrix $\mathbf{X}$, denoted as ${\mathbf{f}\_{1}, \mathbf{f}\_{2}, \cdots, \mathbf{f}\_{n}}$. Therefore, in our subspace learning formulation in subsection 3.2, the $0,1$ matrix $\mathbf{U}$ exists such that $\mathbf{XU}=\mathbf{B}$.

Because of this issue, in the proof of Theorem 3.1, we explained how to construct this $0,1$ matrix $\mathbf{U},$ and then, as the reviewer mentioned, the proof is straightforward.

We thank the reviewer for raising this point. In response to the reviewer's comment, we would like to mention that in general, a basis for a vector space, generated by a set of vectors such as $\\{\mathbf{f}\_1,\ldots,\mathbf{f}\_n\\}$, can be formed in two ways:

First case: the basis members are chosen directly from the vectors $\\{\mathbf{f}\_1,\ldots,\mathbf{f}\_n\\}$. A well-known method for this selection is the Basis Extension (BE) method, detailed in the manuscript. The BE method is capable of constructing a set of basis vectors for any vector space, including the feature space in our problem. In this scenario, as addressed in the initial question from the reviewer, there exists a $0,1$-matrix $\mathbf{U}$ such that $\mathbf{X}\mathbf{U}=[\mathbf{f}\_{1},\mathbf{f}\_{2},\cdots,\mathbf{f}\_{n}] =[\mathbf{f}\_{i\_1},\ldots,\mathbf{f}\_{i\_m}] =\mathbf{B}.$ This implies that the columns of $\mathbf{B}$ are derived from the input feature space.

Second case: the basis members are extracted using the vectors $\\{\mathbf{f}_1,\ldots,\mathbf{f}_n\\}$. For instance, as noted by the reviewer, the basis vectors of PCA can be utilized to ensure that the constructed basis vectors are mutually perpendicular. However, this is not the case for our proposed feature selection framework.

In response to the other reviewer's comment, we would like to mention that as stated in Subsection 3.1 (just after Equation (2)), our manuscript assumes $\mathrm{rank}(\mathbf{X}) = m$ and $m \leq n$ throughout. Consequently, the matrix $\mathbf{B}$ is consistently a square matrix of size $m$.

It is worth emphasizing that, even in scenarios where $\mathrm{rank}(\mathbf{X}) < m$ or $m > n$, the creation of a basis matrix for the feature space is still viable and does not pose any challenges in the development of our feature selection framework. To elaborate further, for any matrix $\mathbf{X}$ with $\mathrm{rank}(\mathbf{X})=r$ (where it is proved that $r$ is always smaller than $m$), the BE method can be applied to generate the matrix basis $\mathbf{B}\in\mathbb{R}^{m\times r}$. This basis matrix can then be utilized in our proposed feature selection method. In such cases, it is sufficient to assume that the matrix $\mathbf{G}\in\mathbb{R}^{r\times n}$, and the matrix product $\mathbf{BG}$ will be in $\mathbb{R}^{m\times n}$ and will have meaningful implications. Moreover, it can be easily seen that the expression $\mathbf{X}-\mathbf{BG}\in\mathbb{R}^{m\times n}$ will be well-defined. To address the reviewer's comment, we have incorporated a brief explanation of this issue in a remark on Page 4 of the revised manuscript.

We thank the reviewer for raising this point. In response to the reviewer's comment, we would like to mention that the assumption $\mathbf{X}\mathbf{U}=\mathbf{B}$ holds and there always exists a matrix like $\mathbf{U}$. The reason is that in our proposed Variance Basis Extension (VBE) method which is explained in Subsection 3.4 (Subsection 3.3 in the old version), we generate the basis matrix as $\mathbf{B}=[\mathbf{f}\_{i\_1},\ldots,\mathbf{f}\_{i\_m}] \in \mathbb{R}^{m \times m}$ in which the basis vectors $\mathbf{f}\_{i\_j}, j=1,..,m$ are selected from the input features set $\\{\mathbf{f}\_{1}, \mathbf{f}\_{2}, \cdots, \mathbf{f}\_{n} \\}$. These are the columns of the data matrix $\mathbf{X}$. Therefore, we have $\\{\mathbf{f}\_{i\_1}, \mathbf{f}\_{i\_2}, ,\cdots, \mathbf{f}\_{i\_m} \\} \subseteq \\{\mathbf{f}\_{1}, \mathbf{f}\_{2}, ,\cdots, \mathbf{f}\_{n} \\}$. Now, as in the proof of Theorem 3.1, we can define the matrix $\mathbf{U} \in \mathbb{R}^{n \times m}$ as $\mathbf{U}=[\mathbf{e}\_{i\_1},\ldots,\mathbf{e}\_{i\_m}]$, where $\mathbf{e}\_{i\_j} \in \mathbb{R}^{n}$ (for $j=1,\ldots,m$) is a vector whose $i\_j$-element is 1 and other elements are 0. Then it can be seen that

0 & 0 & & & 0 \newline \vdots & \vdots & & & \newline 0 & 0 & & & \newline 1 & 0 & & & \newline 0 & \vdots & \cdots & \cdots & \vdots \newline \vdots & 1 & & & \newline & 0 & & & 1 \newline & \vdots & & & \vdots \newline 0 & 0 & & & 0 \end{bmatrix} =[\mathbf{f}\_{i\_1},\ldots,\mathbf{f}\_{i\_m}] =\mathbf{B}.$$ Thus, the $0,1$-matrix $\mathbf{U}$ always exists in our method of generating the basis matrix, and the equality $\mathbf{X}\mathbf{U}=\mathbf{B}$ is satisfied. On the other hand, considering that the matrix $\mathbf{B}$ serves as a basis for the feature space, it follows from a fundamental theorem in linear algebra (namely, each vector in a vector space can be expressed as a linear combination of basis members) that a matrix $\mathbf{G}\in\mathbb{R}^{m\times n}$ exists such that $\mathbf{X}$ can be represented in its basis form, denoted as $\mathbf{X} = \mathbf{BG}$. According to the $0,1$-matrix $\mathbf{U}$ mentioned above, it can be seen that $\mathbf{X}=\mathbf{B}\mathbf{G}=\mathbf{X}\mathbf{U}\mathbf{G}$. Hence, $\mathbf{G}$ as the coefficient matrix can be used to recover the data matrix $\mathbf{X}$ from the basis matrix $\mathbf{B}$. This explanation is provided to show how the matrices $\mathbf{G}$ and $\mathbf{B}$ are used in the framework of subspace learning.

In response to the reviewer's comment, we would like to mention that as mentioned on Page 6 of the paper, the extraction of the update rules introduced for $\mathbf{G}$, $\mathbf{U}$, and $\mathbf{V}$ involves three steps:

(1) Calculating the gradient of the objective function in regards to one of the variables with the assumption of keeping the other variables constant;

(2) Setting the obtained gradient equal to zero;

(3) Applying the Karush-Kuhn-Tucker conditions (for example, in our problem, we have $\mathbf{M}\_{ij}\mathbf{G}\_{ij}=\mathbf{N}\_{ij}\mathbf{U}\_{ij}=\mathbf{W}\_{ij}\mathbf{V}\_{ij}=0$).

To ensure that the denominator of the fraction does not become zero, two common workarounds are employed in the problems based on matrix factorization:

First option: Utilizing the assumption of non-negativity for the input matrices of the problem. Notably, one of the most famous dimension reduction methods, Non-negative Matrix Factorization (NMF), assumes the non-negativity condition for the data matrix $\mathbf{X}$ and the unknown matrices used in the optimization problem, such as the matrices $\mathbf{G}\geq 0$, $\mathbf{U}\geq 0$, and $\mathbf{V}\geq 0$ in our proposed framework. This guarantees that the updated matrices $\mathbf{G}$, $\mathbf{U}$, and $\mathbf{V}$ remain non-negative, avoiding the zero denominator problem. Theoretical investigations into this issue can be found in some studies related to NMF, such as Lemma 1 on Page 3 of the following reference:

$\bullet$ ``A convergent algorithm for orthogonal nonnegative matrix factorization. Journal of Computational and Applied Mathematics, 260, 149-166."

Second option: This method, which is applicable even when assuming non-negativity, has a more practical aspect. To prevent the denominator from reaching zero and causing numerical calculation errors, a small non-negative number, such as $\epsilon = 10^{-10}$, is used. Specifically, we set

$denominator \leftarrow \max(denominator,\epsilon).$

Employing this strategy ensures that fraction calculations are performed without encountering zero-related problems.

In the framework designed for our problem, the first option (assuming non-negativity of inputs) is used. Additionally, when implementing the method, the second option (comparing the denominator with a very small non-negative number) is also considered.

It is worth mentioning that even if the assumption of non-negativity on the input variables is not considered, there is a crucial principle in matrix analysis stating that any matrix $\mathbf{X}$ with real elements can be expressed as the difference of the two non-negative matrices. This is demonstrated as follows:

$\mathbf{X} = \mathbf{X}^+ - \mathbf{X}^{-},$

where $\mathbf{X}\_{ij}^+ = \frac{|\mathbf{X}\_{ij}| + \mathbf{X}\_{ij}}{2}$, and $\mathbf{X}\_{ij}^- = \frac{|\mathbf{X}\_{ij}| - \mathbf{X}\_{ij}}{2}$. By utilizing this representation and the two methods mentioned above, the problem of encountering zero in dimension reduction problems based on matrix factorization is effectively prevented.

To respond to the reviewer's comment, we would like to provide the list of feature selection methods, along with their corresponding computational complexities. These methods are utilized as comparative benchmarks in our experiments. In the following list, $m$ represents the number of samples, $n$ is the number of features, $k$ signifies the number of iterations, $c$ denotes the number of classes, and $B$ indicates the batch size:

1- LS: Structure Preserving-based method, with $\mathcal{O}(nm^2)$,

2- CNAFS: Matrix factorization and Structure Preserving-based method, with $\mathcal{O}(mn^2 +nm^2 + m^3)$,

3- OCLSP: Structure Preserving-based method, with $\mathcal{O}(mn^2 + nm^2 + m^3)$,

4- RMFFS: Matrix factorization-based method, with $\mathcal{O}(n^2)$,

5- RNE, Structure Preserving-based method, with $\mathcal{O}(n^2)$,

6- VCSDFS: Matrix factorization-based method, with $\mathcal{O}(n^2)$,

7- CAE: Encoder-decoder-based Method, with $\mathcal{O}(Bnk)$,

8- CD-LSR: Least Square Regression-based method, $\mathcal{O}(k^3 + k^2c + k^2d + kdc)$,

9- Our proposed method, GRSSLFS: Matrix factorization and Structure Preserving-based method, $\mathcal{O}(m^2n^2)$.

In order to address the reviewer's comment, it is noteworthy that in the literature on matrix factorization-based feature selection methods, it is evident that the computational complexity of such approaches is a function of $m^2$, $m^3$, $n^2$, or $n^3$. This pattern is observed in various matrix factorization-based methods, including CNAFS, RMFFS, VCSDFS, and our proposed method, GRSSLFS. Notably, our proposed method, GRSSLFS, introduces an additional step involving three unknown variables, in contrast to many unsupervised feature selection methods like CNAFS, RMFFS, and VCSDFS, which typically use two unknown variables. Due to this and the inclusion of an extra step considering the basis matrix, the computational complexity of our proposed method is quadratic in both the number of samples and the number of features. However, in case the number of features far exceeds the number of samples, as in our study, the complexity can be approximated as $\mathcal{O}(n^2)$.

The reviewer's great attention to this issue is mostly appreciated. In order to address the reviewer's question, we would like to highlight that the primary objective of this paper is to further investigate the impact of a basis for the feature space on tasks related to feature learning. As discussed in the paper, we first defined the self-representation problem based on the basis matrix as follows:

$\min\_{\mathbf{G}\in\mathbb{R}\_+^{m\times n}}\Vert\mathbf{X}-\mathbf{BG}\Vert\_F^2,$

where $\mathbf{B}=[\mathbf{f}\_{i\_1},\ldots,\mathbf{f}\_{i\_m}]\in\mathbb{R}^{m\times m}$ is a basis matrix. Moreover, in order to learn a low-dimensional representation of high-dimensional data, we established the subspace learning problem based on the basis matrix as:

$\min\_{\mathbf{G}, \mathbf{U}, \mathbf{V}}\Arrowvert \mathbf{B}-\mathbf{BGUV}\Arrowvert\_F^2,$ where $\mathbf{G}\in\mathbb{R}\_+^{m\times n}, \mathbf{U}\in\mathbb{R}\_+^{n\times k}$, and $\mathbf{V}\in\mathbb{R}\_+^{k\times m}$, such that $k\leq n$ is the dimension of the selected feature space. And, we finally defined our proposed feature selection method based on the above-mentioned problems designed via the basis for the feature space. In summary, our proposed feature selection method employed the following structure:

$

\small Dataset,,\mathbf{X}\rightarrowConstruct the Basis,,\mathbf{B},,for the Feature Space\rightarrowDefine the Feature Selection Problem Using,,\mathbf{B}.

$

To provide further clarification, in contrast to classical feature selection methods that select a subset of features by searching the original data feature set, our proposed feature selection method conducts the selection process by searching through a basis of features rather than all features. As discussed in Subsection 3.3 of the manuscript, the selected feature matrix $\mathbf{X}\_{\text{selected}}$ can be written as $\mathbf{X}\_{\text{selected}}=\mathbf{BGU}.$

On the other hand, considering that the matrix $\mathbf{BG}$ serves as a basis representation for the data matrix $\mathbf{X}$, it follows from a fundamental theorem in linear algebra (namely, each vector in a vector space can be expressed as a linear combination of basis members) that $\mathbf{X}$ can be represented in its basis form, denoted as $\mathbf{X} = \mathbf{BG}$. Consequently, it can be inferred that $\mathbf{X}\_{\text{selected}} = \mathbf{XU}$.

In order to address the reviewer's question, it is worthwhile to note that

1- The representation $\mathbf{BGU}$ for the selected feature matrix $\mathbf{X}\_{\text{selected}}$ is based on the aforementioned theorem in linear algebra, stating that each vector in a vector space can be written as a linear combination of basis members (for instance, in our case, $\mathbf{X} = \mathbf{BG}$).

2- The selected feature matrix $\mathbf{X}{\text{selected}}$ and the matrix $\mathbf{U}$ are explicitly connected through the relationship $\mathbf{X}\_{\text{selected}} = \mathbf{BGU}$. In essence, our proposed feature selection method conducts the selection process by exploring a basis of features rather than considering all features.

3- The selected feature matrix $\mathbf{X}\_{\text{selected}}$ and the matrix $\mathbf{U}$ can be implicitly linked through the relationship $\mathbf{X}\_{\text{selected}} = \mathbf{XU}$.

In both scenarios, the matrix $\mathbf{U}$ functions as the feature weight matrix, and the feature selection process is carried out based on $\mathbf{U}$.

Our responses are as follows:

1- What is the interested region?

For patients diagnosed with pneumonia, the chest X-rays reveal various diagnostic features, particularly in the cardiac section. For more information, please refer to ``Chest Radiology: Patterns and Differential Diagnoses, Elsevier Health Sciences, 2017." Notably among these characteristics are an augmented cardiac shadow, alterations in cardiac positioning, the presence of effusion, pericardial effusion, and instances of cardiomegaly. In this study, the application of our proposed feature selection method to the PneumoniaMNIST dataset is assessed with the goal of precisely identifying and analyzing the cardiac silhouette in chest X-ray images, which is located in the central part of the chest.

2- How many selected features are in the interested region?

For our proposed method when the number of selected features is 50 and 100 (as depicted in Figure 1 on Page 9), all the selected features are consolidated within our area of interest.

3- The validation is not quantified. How many radiologists are involved in the evaluation? What is the performance measured?

The validation process incorporated the expertise of two board-certified radiologists. Both professionals thoroughly reviewed the results of chest X-ray images, offering valuable insights to ensure the accuracy and reliability of our proposed method. Their assessments played a pivotal role in validating the model's performance within a clinical context. Additionally, given the study's exclusive focus on detecting the cardiac silhouette without making any diagnoses, both radiologists concurred that assessing specific performance criteria is unnecessary. For this reason, our analysis is focused solely on identifying and examining the cardiac silhouette in chest X-ray images.

4- How do other compared methods perform?

In agreement with the reviewer, we have included results related to the selection of 100 features using comparative methods in Section J on Page 23 (refer Figure 8 in the Appendix). Examining this figure, it is evident that some methods, including our proposed method, RMFFS, and VCSDFS, exhibit a tendency to identify the central area of chest X-ray images. Conversely, some other methods, while successfully identifying portions of the central areas, may overlook additional regions that could provide valuable information. These methods fail to capture such information in radiological analysis.

Finally, in response to the reviewer's questions, we have incorporated additional explanations regarding ``the region of interest" and clarified our purpose for conducting the test on the PneumoniaMNIST dataset in the revised manuscript. Furthermore, the results concerning the selection of 100 features by other comparative methods are presented in Figure 8 of the revised manuscript.

Dear Reviewer, We kindly request your prompt review of our responses as the deadline is in a few hours.

Thank you for your timely assistance

We thank the reviewer for raising this point. We agree that this was not noticed previously, and we have considered three state-of-the-art feature selection methods in the revised version of the manuscript in Table 1. Our proposed approach can outperform these recent SOTA methods as well. The considered SOTA methods are as follows:

1- CNAFS, Convex Non-Negative Matrix Factorization With Adaptive Graph for Unsupervised Feature Selection, IEEE Transactions on Cybernetics, vol. 52, Pages 5522-5534, 2022.

2- OCLSP, Unsupervised Feature Selection via Orthogonal Basis Clustering and Local Structure Preserving, IEEE Transactions on Neural Networks and Learning Systems, vol. 33, Pages 6881-6892, 2022.

3- VCSDFS, Unsupervised Feature Selection Based on Variance-Covariance Subspace Distance, Neural Networks, vol. 166, Pages 188-203, 2023.

The primary rationale for using these methods is their foundation in matrix factorization or subspace learning. These methods have been recently introduced and showed promising performance in the field of feature selection.

The results for the three previously employed methods namely MFFS, MPMR and SCFS which could be considered old methods, are moved to the Appendix (Section I.5). Instead, the results of these recent methods, CNAFS, OCLSP, and VCSDFS, are added to Table 1 on Page 8 of the revised manuscript. Along with these numerical results, all other tables and plots, have been updated to reflect these changes.

Table 1: Comparing the best values of the clustering ACC of nine algorithms on all datasets. Here, Baseline refers to the case where all features are considered. The highest value in each row is in bold.

Table 1: Comparing the best values of the clustering NMI of nine algorithms on all datasets. Here, Baseline refers to the case where all features are considered. The highest value in each row is in bold.

We appreciate the reviewer's thorough attention to these issues. To respond to the reviewer's comment regarding the motivation, we would like to mention that the main goal of the feature selection process is to opt for a subset of features with the purpose of diminishing noise and improving the efficiency and effectiveness of a learning task, such as classification and clustering. In essence, this process of selection aims to identify features that are important for the learning task while ensuring that the chosen set of features is non-redundant, thereby strengthening robustness. Within this framework, a crucial factor in feature selection is redundancy, which involves examining the similarity among features and assessing how effective it is to introduce a new feature into a designated feature set for data analysis.

In the field of linear algebra, a direct connection exists between redundancy and the concept of linear independence. Specifically, a set of vectors in a vector space is deemed linearly dependent if non-zero scalars can be identified in a way that their linear combination equals zero. This suggests that it is possible to express one of these vectors as a combination of the others. In contrast, a collection of vectors is deemed linearly independent if there is no linear dependence among them. Consequently, we can deduce from this principle that a set of vectors exhibiting linear dependence includes redundancy, enabling the removal of one vector from the initial set without altering the span of the remaining vectors. A significant benefit of eliminating linearly dependent features from the initial feature set is that it aids in fulfilling a crucial aim in feature selection, which is the reduction of redundancy. In the pursuit of this goal, the concept of a basis, a fundamental element in the theory of linear vector spaces, becomes crucial.

Each basis for the feature space exhibits two essential traits. Firstly, it covers the entire feature space, and secondly, all features within a basis are linearly independent. These attributes positively and constructively influence the discriminative feature selection process. The initial property guarantees that a basis can represent the complete feature space through its elements, implying that the fundamental characteristics of the original features are present in the basis with a significantly reduced number of elements compared to the entire feature space. The second characteristic of a basis, which involves its elements being linearly independent, leads to a significant reduction in data redundancy. Exploiting the advantages of a basis in a vector space, the primary objective of this paper is to further investigate the impact of a basis for the feature space on tasks related to feature learning.

In order to reply and address the other comment of the reviewer, we follow the matrix structure of $\mathbf{B}=[\mathbf{f}\_{i\_1},\ldots,\mathbf{f}\_{i\_m}]$ and $\mathbf{V}=[\mathbf{v}\_{1},\ldots,\mathbf{v}\_{m}]$, as stated in the manuscript. According to the subspace learning problem mentioned in Problem (7) of the manuscript:

$$\min\_{\mathbf{G}, \mathbf{U}, \mathbf{V}\geq 0}\Arrowvert \mathbf{B}-\mathbf{BGUV}\Arrowvert\_F^2,$$

we concluded $\mathbf{B}\simeq\mathbf{BGUV}$. This means that $[\mathbf{f}\_{i\_1},\ldots,\mathbf{f}\_{i\_m}]\simeq[\mathbf{BGU}\mathbf{v}\_{1},\ldots,\mathbf{BGU}\mathbf{v}\_{m}]$ and implies that each feature $\mathbf{f}\_{i\_l}$ of the basis are represented by $\mathbf{BGU}\mathbf{v}\_{l}$, for $l=1,\dots,m$. To be more precise, the features are expressed as:

$$\begin{cases} \mathbf{f}\_{i\_1}\simeq (\mathbf{BGU})\mathbf{v}\_1=\mathbf{v}\_{11}(\mathbf{BGU})\_{1}+\mathbf{v}\_{21}(\mathbf{BGU})\_{2}+\cdots+\mathbf{v}\_{k1}(\mathbf{BGU})\_{k}, \newline \mathbf{f}\_{i\_2}\simeq (\mathbf{BGU})\mathbf{v}\_2=\mathbf{v}\_{12}(\mathbf{BGU})\_{1}+\mathbf{v}\_{22}(\mathbf{BGU})\_{2}+\cdots+\mathbf{v}\_{k2}(\mathbf{BGU})\_{k},\newline \vdots \newline \mathbf{f}\_{i\_m}\simeq (\mathbf{BGU})\mathbf{v}\_1=\mathbf{v}\_{1m}(\mathbf{BGU})\_{1}+\mathbf{v}\_{2m}(\mathbf{BGU})\_{2}+\cdots+\mathbf{v}\_{km}(\mathbf{BGU})\_{k}. \end{cases}$$

Hence, the sparser the columns of $\mathbf{V}$ are, the fewer members of $\mathbf{BGU}$ are used in the representation of $\mathbf{f}\_{i\_l}$ of the basis matrix $\mathbf{B}$. As a result of this fact, a reduction in redundancy is more likely to occur. In order to reflect the importance of the representation matrix $\mathbf{V}$ in our proposed feature selection process, the idea of inner product regularization is used. Let us consider the representation matrix $\mathbf{V}$. The Gram matrix of $\mathbf{V}$ is written as

$$\mathbf{V}^T\mathbf{V}= \left[\begin{array}{cccc} \langle\mathbf{v}\_1,\mathbf{v}\_1\rangle & \langle\mathbf{v}\_1,\mathbf{v}\_2\rangle& \cdots& \langle\mathbf{v}\_1,\mathbf{v}\_n\rangle \newline \langle\mathbf{v}\_2,\mathbf{v}\_1\rangle & \langle\mathbf{v}\_2,\mathbf{v}\_2\rangle& \cdots& \langle\mathbf{v}\_2,\mathbf{v}\_n\rangle\newline \vdots & \vdots & \ddots & \vdots \newline \langle\mathbf{v}\_n,\mathbf{v}\_1\rangle & \langle\mathbf{v}\_n,\mathbf{v}\_2\rangle& \cdots& \langle\mathbf{v}\_n,\mathbf{v}\_n\rangle \end{array}\right],$$

where $\mathbf{v}\_l$ is the $l$-th column of $\mathbf{V}$, for $l=1,\ldots,m$. Here onwards, the expression $\mathrm{Tr}(\mathbf{1}\_{m \times m} \mathbf{V}^T\mathbf{V})-\mathrm{Tr}(\mathbf{V}^T\mathbf{V})$ readily corresponds to the summation of the off-diagonal elements within the Gram matrix $\mathbf{V}^T\mathbf{V}$, that is to say $\mathrm{Tr}(\mathbf{1}\_{m \times m} \mathbf{V}^T\mathbf{V})-\mathrm{Tr}(\mathbf{V}^T\mathbf{V}) = \sum\_{l,p=1, l\neq p}^m \langle\mathbf{v}\_l,\mathbf{v}\_p\rangle$. Therefore, by the subsequent problem formulated as follows:

$$\min\_{\mathbf{V} \in\mathbb{R}\_+^{k\times m}}\mathrm{Tr}(\mathbf{1}\_{m \times m} \mathbf{V}^T\mathbf{V})-\mathrm{Tr}(\mathbf{V}^T\mathbf{V}) = \min\_{\mathbf{V} \in\mathbb{R}\_+^{k\times m}}\sum\_{l,p=1, l\neq p}^m \langle\mathbf{v}\_l,\mathbf{v}\_p\rangle,$$

and under the assumption that $\mathbf{V}\in\mathbb{R}\_+^{k\times m}$, it can be deduced that the optimal solution to the above problem is attained when the inner product terms $\langle\mathbf{v}\_l,\mathbf{v}\_p\rangle$ tend toward zero for all pairs of $l$ and $p$ except when $l=p$. On the other hand, since $\langle\mathbf{v}\_l,\mathbf{v}\_p\rangle = (\mathbf{v}\_l)^T\mathbf{v}\_p$ and considering the non-negativity assumption for $\mathbf{v}\_l$ and $\mathbf{v}\_p$, the value of $\langle\mathbf{v}\_l,\mathbf{v}\_p\rangle$ tends towards zero when either $\mathbf{v}\_l$ or $\mathbf{v}\_p$ become zero or extremely sparse vectors. Consequently, the optimization problem defined based on the inner product regularization can result in a significant degree of sparsity within the coefficient matrix $\mathbf{V}$.

In order to address the reviewer's comment, we have included new explanations following the above-mentioned details. For more information, we kindly ask the reviewer to refer to Subsection 3.3 on Page 5 and Section A on Page 13 of the revised manuscript.

Dear Reviewer, We kindly request your prompt review of our responses as the deadline is in a few hours.

Thank you for your timely assistance