Stray Intrusive Outliers-Based Feature Selection on Intra-Class Asymmetric Instance Distribution or Multiple High-Density Clusters

We sincerely thank Reviewer 5k2j for the constructive and valuable comments. The concerns are addressed as follows.

Q1: In Fig. 1b, class "2" has two high-density clusters, but there is only class "1" and "3" in the figure.

Many thanks for the comment. We revise this typo from class "2" to class "3".

Q2: What is the definition of 3σ principle? Can authors explain its insight?

Sorry for the unclear description. We add the classical literature (Harris & Stocker, 1998) and [1] to support the 3$\sigma$ principle. For $\xi\sim N(\mu,\sigma^2)$, the probability (denoted as $Pr$) that $Pr(\xi\in(\mu-3\sigma,\mu+3\sigma))>0.997$. This provides a robust framework for analysing variability in normally distributed data and guides thresholds for outlier detection in the context of ADMHC.

[1] Groeneveld, R. A., & Meeden, G. (1984). Measuring Skewness and Kurtosis. Journal of the Royal Statistical Society: Series D (The Statistician), 33(4), 391-399.

Q3: There are different types of outliers and low density assumption can not guarantee that all outliers are not included.

We clarify that due to random errors in the data, especially in synthetic datasets, it is impossible to correctly identify all of them. In this paper, only low-density cases are identified and used as outliers. We will include the above details of the outliers we have identified in the final version.

Q4: Why using l1 distance in Eq.(1)?

As presented in the original paper (see line 79, right column), the $\ell_1$ norm can treat each component of the feature vector equally. $\ell_1$ treats all $\vert x_{if}^{(k)}-x_{jf}^{(k)} \vert$ linearly while $\ell_2$ penalizes large $\vert x_{if}^{(k)}-x_{jf}^{(k)}\vert$ quadratically. This makes it easier to identify outliers in $\ell_1$ than in $\ell_2$.

As suggested, we add comparisons with $\ell_1$ distance and the typical $\ell_2$ distance. Note that we only replace $\ell_1$ distance with $\ell_2$ distance. As shown in Rebuttal Table D, SIOFS with $\ell_1$ distance outperforms that with $\ell_2$ distance on all datasets. We will include these details in the final version.

Rebuttal Table D. Comparative results of SIOFS with $\ell_1$ distance (denoted as "SIOFS") and $\ell_2$ distances (denoted as "w/ $\ell_2$") on some datasets. "w/ $(\cdot)^1$" means that the $(\cdot)^3$ terms are directly replaced by $(\cdot)^1$ in Eq.(4). The average ACC over all 11 datasets is also reported for a comprehensive comparison.

Q5: Explain the modified SC in Eq.(4). Why $(d_i^{(l)}-\mathrm{u}^{(l)} )^3$ is used? How about other terms like $(d_i^{(l)}-\mathrm{u}^{(l)} )^1$.

According to the literature (Linton, 2017), the original formula of the SC is $SC=\frac{\frac{1}{n}\sum_{i=1}^n(X_i-\bar{X})^3}{\left(\frac{1}{n}\sum_{i=1}^n(X_i-\bar{X})^2\right)^{3/2}}$, where $X_i$ are the individual data points, $\bar{X}$ is the mean and $n$ is the number of instances. As mentioned in lines 142-150 (right column), the modified SC can be obtained by substituting $\mathrm{u}^{(l)}$ and $\hat{\sigma}^{(l)}$ into the formula $SC$. That is, the term $(d_i^{(l)}-\mathrm{u}^{(l)})^3$ is directly derived from the term $(X_i-\bar{X})^3$ and has the same statistical meaning. The SC is a statistical measure that quantifies the asymmetry of a probability distribution. It indicates the degree to which data deviate from a symmetric, bell-shaped normal distribution [1]. Since we address the highest-density subclusters in the context of ADMHC, it is reasonable to use and modify the SC via Eq.(4).

In addition, we appreciate the kind suggestion to discuss other terms such as $(d_i^{(l)}-\mathrm{u}^{(l)})^1$. We add the comparison between $(d_i^{(l)}-\mathrm{u}^{(l)})^3$ and $(d_i^{(l)}-\mathrm{u}^{(l)})^1$. To make the comparison reasonable, two terms with $(\cdot)^3$ in Eq.(4) are directly replaced by $(\cdot)^1$ (denoted as "w/ $(\cdot)^1$"). As demonstrated in Rebuttal Table D, our SIOFS using the term $(\cdot)^3$ achieves higher ACCs compared to $(\cdot)^1$. We will incorporate the above discussions into the final version.

Q6: In Line 192, why normalize $\hat{s}^{(l)}$ by $\hat{s}^{(l)}/3$?

Please see our response to Reviewer aEFk, Q5.

We sincerely appreciate the reviewer’s feedback. Below, we address the concerns in detail.

Q1: Some ambiguous descriptions about the theorems. Inconsistency between theory and experiment about α.

As addressed in our response to Reviewer 52hZ, Q1, we clarify that the condition for in Theorem 2 does not contradict the experimental results. When $\alpha$ is small, the conclusion in lines 184-187 still holds. $\alpha$ has a significant impact on ACC because of the scattered distribution of clusters and multiple local high-density subclusters (see Fig. 1b). This problem can be solved by selecting more features.

In addition, we regret the unclear descriptions. We delete "larger" and "the most of", and revise Theorem 2 as follows: $d_1^{(l)},d_2^{(l)},\dots,d_{n_l}^{(l)}$, $\dots$. When $\alpha\in(0,1)$ and $\sigma^{(l)}>0$, $u^{(l)}+2\sigma^{(l)}>mode^{(l)}$ holds with probability 1 if $\hat{s}^{(l)}>0$, and $u^{(l)}+2\sigma^{(l)}<mode^{(l)}$ holds with probability 1 if $\hat{s}^{(l)}<0$. We will include the above details in the final version.

Q2: Lack evidence or practical examples to validate the advantage on deep learning-based FS.

We appreciate the constructive feedback provided. As mentioned in lines 417-418, deep FS has extensive applications, such as Remote Sensing (RS) scene classification and 3D object recognition. We add the ACC results by selecting all features (denoted as "AllFeat") in Rebuttal Table B and the CMs with the top 5% of features (see https://i.postimg.cc/HLw5K8z6/CMs-AID.png) on AID dataset. These results demonstrate that the SIOFS is superior to AllFeat while some comparative methods (such as ILFS, S^2DFS) is inferior. As shown in CMs, our SIOFS can further improve the classification performance on deep features against other methods. Combined with Fig.1a, for the "School" with intra-class ADMHC, the number of instances correctly classified by SIOFS is 28, while it is 24 by TRC and 27 by Fisher and ReliefF. SIOFS gains better performance in predicting the confusing classes, being able to identify key factors influencing RS monitoring. On UCM, SIOFS (94.95) also performs better than AllFeat (94.48); On ModelNet, SIOFS (93.03) has higher ACC than AllFeat (92.71). We will include these details in the final version.

Rebuttal Table B. ACC (%) on AID. Six FS methods are randomly selected for comparison.

Q3: Maintain the spirit of open source.

We appreciate the valuable comment. The code will be released once accepted.

Q4: Add an essential reference.

As suggested, we will include this paper in the Related Work section in the final version.

Q5: Explain the transition from 3σ to Eq.(5), the adjustment of the coefficient "2" in line 189, and the normalization in line 193.

As mentioned in lines 129-131 (right column) and lines 179-187, and combined with the revised Theorem 2, the transition to Eq.(5) is a heuristic reasoning that follows the conclusion in lines 183-187 (see "That is, ..."), and it is necessary to introduce SC for normalization. The original formula of $SC$ (see our response to Reviewer 5k2J, Q5) and the literature [1] (see the response to Reviewer 5k2j, Q2) clarify that $s^{(l)}\in(-3,3)$ is an empirical guideline, providing intuitive thresholds to assess skewness severity and guide data adjustments, and reflecting realistic bounds for most practical datasets. That is, we have $\frac{1}{3}s^{(l)}\in(-1,1)$, then $2-\frac{1}{3}s^{(l)}\in(1,3)$. As mentioned in lines 198-205, the $2-\frac{1}{3}s^{(l)}$ with its range of (1,3) is more rational than 2 for $\sigma^{(l)}$ to address the highest-density subcluster in the context of ADMHC. As demonstrated in Rebuttal Table C, the proposed $2-\frac{1}{3}s^{(l)}$ outperforms other formulas in most cases, and the average ACCs over the 11 datasets further quantitatively prove the superiority of our method. We will incorporate the above discussions in the final version.

Rebuttal Table C. Results of different adjustments and normalizations on some datasets. Our SIOFS is equipped with "$2-\frac{1}{3}s^{(l)}$". Additional settings are included in the table.

Q6: Minor errors.

We will revise them in the final version.

We sincerely thank Reviewer 52hZ for the recognition of our work and for providing constructive comments.

Q1: Explain the inconsistency between Theorem 2 and results about $\alpha$.

Sorry for the incomplete statement about Theorem 2. We clarify that the condition for $\alpha$ in Theorem 2 does not contradict the experimental results. When $\alpha$ is small, the conclusion in lines 183-187 (left column) still holds. For easy understanding, we add the following statement.

According to the definition of RDM center (see lines 96-98, right column), $\mathbf{u}^{(l)}$ is the average of the top $\lceil\alpha\cdot n_l\rceil$ high density values in $d_1^{(l)},\dots,d_{n_l}^{(l)}$. When $\alpha$ is smaller but $\lceil \alpha\cdot n_l\rceil=1$, we have u$^{(l)}=mode^{(l)}$. Following Theorem 1, let $\xi$ be the distance between the instance and the center of class $l$ and f(x) be the probability density function of $\xi$. Assume that the SC of $f(x)$ is greater than 0. There are two properties of statistical probability (Harris & Stocker, 1998): (i) $f(mode+\varepsilon)>f(mode-\varepsilon)$, for any $\varepsilon>0$. (ii) f(x) is monotonically increasing near the left side of the $mode$ and decreasing near the right side. In Theorem 2, $d_1^{(l)},\dots,d_{n_l}^{(l)}$ is a random sample of $\xi$, let $d_2,d_3$ denote the second and third largest density in $d_1^{(l)},\dots,d_{n_l}^{(l)}$, respectively, and $d_1=mode^{(l)}$. Obviously there is $f(d_1)>f(d_2)>f(d_3)$. When $\alpha$ is small but $\lceil\alpha\cdot n_l\rceil=2$, $\hat{\sigma}^{(l)}>0$ and $\hat{s}^{(l)}>0$. Given the chance of errors in the sampling process, the probability that $d_2>d_1$ holds according to condition (i) is 1, then u$^{(l)}=\frac{d_1+d_2}{2}>d_1$, i.e., at least u$^{(l)}+2\hat{\sigma}^{(l)}>mode^{(l)}$ holds with probability 1. When $\alpha$ is small but $\lceil\alpha\cdot n_l\rceil=3$, $\hat{\sigma}^{(l)}>0$ and $\hat{s}^{(l)}>0$. If $d_3>d_1$, then u$^{(l)}=\frac{1}{3}(d_1+d_2+d_3)>d_1$; If $d_3<d_1$ (see figure https://i.postimg.cc/GmzqF1fV/PDF.png), since $f(d_3)<f(d_2)$, combining the property (i), $d_3$ can only be very close to $d_1$ in this skewed distribution, i.e., u$^{(l)}=\frac{1}{3}(d_1+d_2+d_3)$ is very close to $d_1$. Combined with $\hat{\sigma}^{(l)}>0$, it holds with probability 1 that u$^{(l)}+2\hat{\sigma}^{(l)}>mode^{(l)}$ for $\lceil \alpha \cdot n_l\rceil=3$. When $\lceil\alpha\cdot n_l\rceil=4,5,\dots$, the same conclusion can be obtained. Similarly, when $\hat{s}^{(l)}<0$ and $\alpha$ is small, but $\lceil\alpha\cdot n_l\rceil=1,2,\dots$, u$^{(l)}+2\hat{\sigma}^{(l)}<mode^{(l)}$ holds with probability 1.

In addition, we rewrite "Based on Theorem 2..." (line 181) as "Combining Theorem 2..." to make the discussion and expression more rigorous. Based on the above explanations and Theorem 2, the value of $\alpha$ is reasonable in (0,1), and thus we set $\alpha=0.1,\dots,0.9$ in Figures 5 and 7. Since $\alpha$ is the ratio of higher density to all instances in a class (see lines 113-115), it is affected by different dataset characteristics. For example, due to the scattered distribution of clusters and multiple local high-density subclusters in CLL (see Fig. 1b), $\alpha$ has a significant impact on ACC (see Fig. 7). This problem can be solved by selecting more features. Above statements will be added in the final version.

Q2: Explanations of the adjustment of the coefficient "2" mentioned in the left column at line 189 and the normalization operation on line 193.

Please see our response to Reviewer aEFk, Q5.

Q3: Typos: conditions i-->(i), ii-->(ii) and iii-->(iii).

As suggested, we go over the paper and will revise them in the final version.

Q4: Whether SIOFS can still have a significant advantage over other methods when the number of classes is large?

As suggested, we add comparisons with the baselines on Caltech101 dataset, which has 101 classes and 3030 images. Like (Yuan et al., 2022), we also use the Fisher Vectors (262144 dimensions). Additionally, in order to reduce the computation time and preserve the main properties of the original representation, we uniformly sample these very high dimensional feature vectors with 50 components, i.e. $1,51,\dots,262144$, and obtain the final feature vector (5243 dimensions) for each image. Following the setting in (Yuan et al., 2022), we select 60%,70%,80%,90% of features. Same with the original paper, ACC and NMI are calculated over all baselines. And SIOFS outperforms all comparative methods in all cases. Some results are shown in Rebuttal Table A.

Rebuttal Table A. ACC (%) on the large-scale high-dimensional Caltech101 dataset. Details will be added in the final version.