Iterative Feature Space Optimization through Incremental Adaptive Evaluation

Dear reviewer WB9F：

Thank you very much for taking the time to thoroughly review our manuscript. Your valuable comments have provided us with many useful insights and directions for improvement. We have carefully read all the questions and suggestions and will address them one by one.

1.Regarding significance of the task of iterative feature space optimization

Indeed, existing traditional algorithms, such as Lasso, can perform feature selection in a single step. However, many iterative feature selection or feature generation algorithms can address the limitations of one-time selection or generation methods by capturing the complex interactions between features to improve performance. This approach has been mentioned in several recent works, as cited in the related literature in our introduction. The purpose of our proposed EASE framework is to provide a general evaluation method for these types of algorithms.

2.Regarding some expressions

(1)Regarding Equation (1): The detailed derivation of the loss between two consecutive iterations is explained in Section 4.1, "Sample Index Optimizer." In the t-th iteration, we use the sample loss from the previous iteration, so this sample evaluation spans across two iterations. Since this part mainly aims to explain the methodology, we do not provide a detailed explanation of the specific loss calculation here. In Section 4.3, we further clarify the specific loss function we used.

(2)Regarding terminology: In machine learning tasks, it is indeed common to refer to features as the data matrix. However, in this paper, we refer to the input data as the "feature space" to align with the common terminology used in feature selection, feature generation, and other feature optimization tasks. Specifically, "feature space optimization" is more specific than "data matrix optimization." If we used terms like "data matrix optimization" or "obtaining the data matrix," it could cause confusion with feature selection or feature generation tasks. Therefore, we chose to use the term "feature space" in this context.

(3)Regarding Equation (3): In Section 3 of this paper, we provide a detailed definition of F and also mention the definition of F_t. Since F_t is a feature space, it is clear that F_t is a matrix. To avoid redundancy, we did not redefine symbols that were already defined earlier in subsequent equations. The score is a scalar, and we may not have explained this part sufficiently, which could have led to a lack of clarity for the readers. Therefore, we have revised the relevant content in the latest version.

(4)Regarding Equations (2) and (10): Equation (2) is intended to clarify the incremental update method we use in the problem statement, specifically explaining that our incremental update strategy minimizes the empirical risk while constraining changes to some important model parameters. Equation (10), on the other hand, combines the feature subspace we construct to show how to constrain changes in model parameters, particularly through the use of Fisher information. Due to the complexity of the computation process for this principle, we provide the derivation in the explanation of Equation (10). Upon careful observation, it can be seen that the two terms in Equation (10) correspond to the empirical risk minimization loss and the structural risk minimization loss, which aligns with Equation (2).

3.Regarding feature selection and feature space optimization

The method we propose is not a feature selection algorithm, but an evaluator for the feature space selected by feature selection algorithms. Therefore, our method cannot be classified as a feature selection algorithm. As stated in Section 2 of the paper, we use feature selection algorithms as an example to evaluate the effectiveness of the EASE evaluator we propose.

4.Regarding time complexity

In order to ensure a fair comparison between the traditional evaluators and our proposed method, the traditional evaluators must first be optimized (i.e., parameter tuning). Only after this optimization can we compare them with our method; otherwise, the comparison would not be meaningful. The experiments in Section 5 demonstrate this point.

Dear reviewer eoma,

Thank you very much for your thorough review of our work and for acknowledging our contributions to feature space optimization. We appreciate your insights regarding the limitations of our paper, and we provide detailed responses below to address these points and improve our work further.

1.Regarding time complexity

We provide a detailed analysis of the time complexity of each evaluator in Section 5.2.2 of the paper and further supplement this with time complexity experiments under various scenarios in Appendix C.3 and Appendix C.6. The experimental results of the evaluators across different datasets effectively demonstrate the validity of the proposed method.

2.Regarding the state-of-the-art baseline methods

Our EASE framework is a general framework that can be applied to any algorithm requiring iterative optimization of feature spaces. Feature selection is just one application scenario; moreover, iterative feature generation is another scenario that can be used for validation. Therefore, the core focus of the method is on evaluation. We apologize for not using the state-of-the-art feature space selection algorithms, and we have conducted new experiments on the state-of-the-art feature selection algorithm SDAE at AAAI 2024. On 14 public datasets, our EASE achieved the best performance on 11 datasets and the second-best performance on 3 other datasets (See Appendix C.5). We can observe that in the feature space selected by the SDAE, our proposed EASE exhibits the best evaluation performance in both classification and regression tasks from Appendix C.5. This proves that EASE has stronger robustness.

3.Regarding large-scale datasets

Thank you for your valuable suggestions. We sincerely appreciate your feedback, which is crucial for improving our work. Based on your suggestion, we have further refined our experimental design and updated the results of EASE on large-scale datasets to provide a more comprehensive and detailed demonstration of its performance (See Section 5.1 for large-scale datasets and Section 5.2 and Appendix C.2, C.3, C.5 for results.). On the large-scale dataset AmazonEA, the EASE evaluator achieved an accuracy of 0.963, precision of 0.782, F1 score of 0.50, and recall of 0.503. On the Nomao dataset, EASE performed with an accuracy of 0.952, precision of 0.947, F1 score of 0.936, and recall of 0.922. Compared to other datasets, EASE demonstrated superior performance. The experimental results on large-scale datasets also validate the effectiveness of the method we proposed.

4.Regarding Innovation

In fact, the attention mechanism is not the focus of our innovation. Our main focus is on how to address sample bias and construct a general-purpose evaluator with strong generalization performance. The attention mechanism is merely a tool we use to capture important information between two consecutive iterations. In fact, the attention mechanism can be replaced by other feature learning methods, such as CNN for capturing local features. Our innovation lies in proposing a flexible and general framework that allows various feature extraction methods to be integrated, thus optimizing evaluation performance across different tasks.

Dear reviewer 67BS：

We sincerely appreciate the reviewer's time and effort in assessing our paper. We want to answer the posed questions to improve our paper's quality and clarity.

1.Regarding implementation details

Due to the limitations of the conference page length, we have included all details of the hyperparameter configurations, optimization strategies, specific training processes, and environmental settings for the experiments in Appendix C.1. Please refer to Appendix C.1 for more information. We apologize for not providing an explanation in the main text earlier.

2.Regarding the state-of-the-art baseline methods

Existing feature space evaluation algorithms typically rely on commonly used models, while research on new evaluators is almost nonexistent. This research gap is why we aim to propose a new feature space evaluator. Besides, we apologize for not using the state-of-the-art feature space selection algorithms. And we additional conducted new experiments on the state-of-the-art feature selection algorithm SDAE at AAAI 2024. On 14 public datasets, our EASE achieved the best performance on most datasets(See Appendix C.5). We can observe that in the feature space selected by the SDAE, our proposed EASE exhibits the best evaluation performance in both classification and regression tasks from Appendix C.5. This proves that EASE has stronger robustness.

3.Regarding theoretical analysis of EASE

In fact, the EASE method we proposed does not require special attention to the correlations between features and data distribution, as the neural network-based structure can adaptively adjust the usage of features. This is one of the advantages of our method. We have detailed the theoretical aspects of EASE in Section 4.1 and Section 4.3.

4. Regarding time complexity

In iterative feature space optimization algorithms, traditional evaluation methods often become highly complex when handling large-scale datasets. For effective evaluation, these methods typically need to continuously search for optimal parameters and retrain from scratch on the training dataset during each iteration. Particularly on large-scale or high-dimensional datasets, traditional evaluation algorithms may be unable to yield results due to excessive runtime. We encountered this issue when working with large datasets, which motivated us to propose the EASE framework. In the EASE framework, parameter selection only needs to be done during the initial training phase; in subsequent evaluations, only the parameters of newly emerging features need to be updated, significantly reducing algorithmic complexity. Our experimental results (see Section 5.2.2 and Appendix C.2) confirm this improvement.

5. Regarding parameter

(1) The regularization parameters have a significant impact on the stability of EASE in different scenarios. Our experimental results (see Section 5.2.1) confirm this effect, as EASE consistently performs best across various datasets. (2) As shown in the methodology section of this paper (see Section 4.1), the sampling strategy within the EASE framework is fixed. The loss evaluation for different samples in each iteration effects the sampling steps in the next iteration, so this parameter does not have a major impact on the results. (3) To address the issue caused by varying feature space length across different iterations, we set the embedding dimension of the attention mechanism to match the sample size of the feature subspace, effectively resolving the problem. For detailed experiments on the embedding dimension D (the sample size of the feature subspace), please refer to Appendix C.7. We have rephrased this part in the appendix. (4) Other parameters: We have also tested the effect of some parameters on the final evaluation results. For details, please refer to Appendix C.7.

6. Regarding tuning strategies

The details of the tuning strategies used in the experiments are provided in Appendix C.1.

7. Regarding generalizability

(1) Theoretical Support: We provide detailed theoretical support in Section 4.1 of the methodology. Specifically, in the "Sample Index Optimizer" and "Feature Subspace Construction" subsections, we explain how we leverage limited samples and features to enhance the generalization performance of the evaluator. First, for each iteration of training samples, we apply dynamic weighting to the samples based on the loss from the previous iteration before sampling. Secondly, during feature subspace construction, selection of subspace samples further improves generalization performance. (2) Effectiveness: In Section 5.2, we use as many datasets as possible to validate the effectiveness of EASE. The more datasets included, the greater the variety of feature spaces, allowing us to more comprehensively test EASE's performance across different feature spaces. In future work, we will continue to explore the effectiveness of this method on a wider range of datasets.