Distribution Shift Aware Neural Feature Transformation

We appreciate your acknowledgment that the paper is well-presented, and the topic is crucial.

Regarding your questions:

W1:

Our main contribution is addressing the distribution shift problem within the generative feature transformation framework. We re-summarize our contributions as follows:

1. We improved the embedded-optimization-reconstruction feature transformation framework to make it dataset distribution-aware rather than limited to token-level expressions.
2. We introduced a shift-resistant feature representation, flatness-aware generation, and incorporated normalization techniques to address distribution shift.
3. We conduct extensive experiments to demonstrate the efficiency, resilience, and traceability of our framework.

W2:

1. Our reported baseline results appear lower due to different dataset splitting methods. [1] and other baseline papers use random sampling, ensuring the train and test sets follow the same distribution. In our experiments, we split the data to create a distinct distribution shift detected by Kolmogorov-Smirnov test between train and test sets.
2. We did not report results on other datasets because these datasets are stable and it's hard to observe a significant distribution shift when splitting the data. We will clarify this in the dataset section.

W3:

We propose addressing the OOD problem in feature transformation within a generative framework by introducing shift-resistant feature set representation, flatness-aware generation, and normalization integration. Ablation experiments in Section 5.2 demonstrate the effectiveness of these three components.

We hope that our response can address your concerns.

Thanks for acknowledging that our rebuttal has answered and addressed your questions.

Regarding the final question about “why different experimental setting (total dataset number and ways of training/testing preparation) between our paper and [1]”:

Because our AI task is distribution shift-resistant feature transformation, not classic feature transformation, our data environment of “using 16 datasets” and “creating training/testing shifts” rigorously strengthens the solidness and comprehensiveness, instead of weakening experiments.

1. why we SHOULD use 16 datasets instead of all 23 datasets in [1]: When using our specialized sampling to create a distribution shift between training and testing data over the 23 datasets in [1], we found that the 16 of them show statistically-significant shifts rather than the other 7 datasets. We used the 16 drifted training/testing set pairs to ensure the fairness and solidness of our reported results. Instead, including the other 7 datasets without significant distribution shifts will reduce experimental solidness and fail to test shift resistance.

2. why the baseline [1] in our paper’s setting SHOULD BE lower-performant than in [1]’setting under the same data source and same code: We intentionally created distribution shifts between training and testing data. [1] reported classic feature transformation performance, but we reported shift-resistant performance under a drifted training/testing data environment.

3. why our experiments are comprehensive: Our experimental design (refer to paper and appendix) includes overall shift-resistant performance, impact of normalization, impact of flatness aware gradient search, impacts of reweighting, robustness check over diverse downstream predictors, robustness check over diverse strategies of shift creation, complexity analysis, visual and quality analysis of transformed features, visual and quality analysis of shift elimination.

I hope the area chair can check our clarifications of the misunderstanding.

We appreciate your acknowledgment that our research problem is new, our methodology is effective, and our experiments show a superior performance than baselines.

Regarding your specific concerns:

Organization and writing:

Q1: We intended to highlight the insights of our approach at a high level, so we placed the more detailed algorithmic components in the appendix. We will reorganize this section in the camera-ready version.

Q2: We removed the arxiv version.

Methodology:

Q1: We reorganized our contributions and stated the difference between our model and MOAT:

1. MOAT didn't consider the OOD problem.
2. MOAT embeds the token-level expressions so it can't be aware of data distribution. Our method regards a feature set as a feature-feature interaction graph to capture the potential relationship between features and embeds raw data to be aware of the distribution of the dataset.
3. We introduced anti-shift mechanisms to address the OOD problem.

Our contribution is summarized as:

1. We improved the embedded-optimization-reconstruction feature transformation framework to make it dataset distribution-aware rather than limited to token-level expressions.
2. We introduced a shift-resistant feature representation, flatness-aware generation, and incorporated normalization techniques to address distribution shift.
3. We conduct extensive experiments to demonstrate the efficiency, resilience, and traceability of our framework.

Q2: Data preparation is not a crucial step in our framework. So we follow the advanced paper GRFG [1] to sample high-quality and diverse training data.

[1] Wang, Dongjie, et al. "Group-wise reinforcement feature generation for optimal and explainable representation space reconstruction." Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining. 2022.

Q3&Q4: We address an essential AI task: The out-of-distribution problem in generative feature transformation, which is at the heart of ICLR. We propose a coherent approach to solve this problem. This is the first research task to solve the OOD of feature transformation. Several developed techniques in our method are generalizable:

• shift-resistant feature set representation: it can rectify and adjust the distortion and ensure reliable and transferable feature knowledge in the testing set.
• flatness-aware search: it's simple but effective to help identify the reliable in-distribution embedding. These computing thinking and insights are generic for feature engineering.

Experiments:

Q1: We did not report results on other datasets because these datasets are stable and it's hard to observe a significant distribution shift when splitting the data. We will clarify this in the dataset section.

We hope that our response can address your concerns.

Thanks for acknowledging that our rebuttal has answered and addressed most of your questions, except “experimental data environment and novelty”.

Regarding novelty:

Task novelty: the tasks of [1] and our paper are different: addressing feature transformation versus distribution-shift in feature transformation.

Technical novelty: shift robustness via three mechanisms: feature transformation experience reweighing and flatness-aware search under generative learning, as well as pre-normalization and post-denormalization

Regarding why using 16 datasets instead of 23 datasets in [1]:

Because our AI task is distribution shift robustness in feature transformation, not classic feature transformation, our data environment of “using 16 datasets” and “creating training/testing shifts” rigorously strengthens the solidness and comprehensiveness, instead of weakening experiments. This SHOULD NOT be a reason for rejection.

1. why we SHOULD use 16 datasets instead of all 23 datasets in [1]: When using our specialized sampling to create a distribution shift between training and testing data over the 23 datasets in [1], we found that the 16 of them show statistically-significant shifts rather than the other 7 datasets. We used the 16 drifted training/testing set pairs to ensure the fairness and solidness of our reported results. Instead, including the other 7 datasets without significant distribution shifts will reduce experimental solidness and fail to test shift resistance.

2. why the baseline [1] in our paper’s setting SHOULD BE lower-performant than in [1]’setting under the same data source and same code: We intentionally created distribution shifts between training and testing data. [1] reported classic feature transformation performance, but we reported shift-resistant performance under a drifted training/testing data environment.

3. why our experiments are comprehensive: Our experimental design (refer to paper and appendix) includes overall shift-resistant performance, impact of normalization, impact of flatness aware gradient search, impacts of reweighting, robustness check over diverse downstream predictors, robustness check over diverse strategies of shift creation, complexity analysis, visual and quality analysis of transformed features, visual and quality analysis of shift elimination.

I hope the area chair can check our clarifications of the misunderstanding.

We appreciate your acknowledgment that our research task is interesting and important.

Regarding your specific concerns:

W1:

We improved the embedded-optimization-reconstruction feature transformation framework to make it encode raw data and aware of data distribution, rather than limited to token-level expressions. Then we developed anti-shift methods to address the OOD problem in the feature transformation task. We will reorganize these contributions and avoid misunderstanding.

W2:

1. Our core task is to address the distribution shift problem in feature transformation.
2. We model the observed transformed feature set as a feature-feature interaction graph to capture underlying relationships between features and data distribution. We use inductive graph embedding algorithms to handle the dynamic changes in the number of nodes representing different observed transformed feature sets.
3. We use sample reweighting to debias shift.

W3:

Our main research task is to address the distribution shift problem in feature transformation. We regard the feature set as a graph for the following reasons:

1. To directly embed raw data and capture data distribution;
2. By representing the dataset as a feature-feature interaction graph, we can capture underlying relationships between features. Existing studies show that treating features or feature domains as nodes can effectively capture relationships between features.[1][2]

[1] He, Xiaofei, Deng Cai, and Partha Niyogi. "Laplacian score for feature selection." Advances in neural information processing systems 18 (2005).

[2] Li, Zekun, et al. "Fi-gnn: Modeling feature interactions via graph neural networks for ctr prediction." Proceedings of the 28th ACM international conference on information and knowledge management. 2019.

W4:

We did not report results on other datasets because these datasets are stable and it's hard to observe a significant distribution shift when splitting the data. We will clarify this in the dataset section.

We hope that our response can address your concerns.

We appreciate your acknowledgment that our research problem is crucial, the proposed method is comprehensive, and the experiments are extensive.

Regarding the weaknesses:

W1:

In each section, we emphasized the motivation behind each technique and we will further highlight this in the revised version.

1. Generative feature transformation framework: Prevents the exponential growth of possible feature combinations.
2. RL data collection: Leveraging RL’s exploration-exploitation mechanism automates the collection of high-quality, diverse training data to build a better continuous space.
3. Anti-shift mechanisms: In the generative feature transformation framework, training-testing shifts can distort the embedding space. Anti-shift mechanisms adjust for these distortions, ensuring reliable and transferable feature knowledge in the test set.

W2:

While some techniques we use, like flatness-aware methods, have been studied before, our integration of these techniques is a purposeful adaptation to tackle complex distributional shift challenges in feature transformation. Our experiments show that this approach effectively addresses OOD issues in feature transformation, which prior research has not considered. We believe this contribution is both innovative and practically impactful.

W3:

1. The datasets are public and widely adopted for validating feature transformation approaches, as previous studies have extensively used them to demonstrate effectiveness.
2. Scalability is very important. Small-scale experiments are a valid first step for concept validation. Appendix D shows that our model performs well in terms of time and space complexity compared to advanced baselines like MOAT and GRFG, indicating its potential for scalability to larger datasets.

W4:

We used the most classic and commonly adopted baselines in feature transformation and also compared recent advanced models like GRFG and MOAT.

W5:

Our original purpose was to demonstrate our workload and respect to ICLR reviewers. This paper is intensive, but we put effort into writing and the writing includes our computing thinking. Our logic pipeline is:

1. background
2. limitations
3. highlight research gap
4. summarize our insights
5. summarize our proposed method
6. our contributions

We will reorganize our introduction and remove some redundancy to make it more clear.

W6:

We provided a formal mathematical formulation for DSFT in the problem statement to enhance clarity:

$$\Gamma^{\ast}=\psi (\mathbf{E}^{\ast}) = \mathop{\arg\min}_{\mathbf{E}\in \varepsilon}\mathcal{A}(\mathcal{M}(\psi (\mathbf{E})(\mathbf{X}^{tr})), \mathbf{y}^{tr}),$$

where $\varepsilon$ is the continuous embedding space by transforming $\mathbf{R}$. $\mathbf{E}$ is the embedding vector in the space of $\varepsilon$, and we define the optimal embedding vector is $\mathbf{E}^{\ast}$. $\psi$ is a reconstructing function that can rebuild the feature transformation operation sequence relying on the embedding vector $\mathbf{E}$ from $\varepsilon$. $\mathcal{A}$ indicates the effectiveness of the downstream task and the $\mathcal{M}$ represents the downstream model.

Ultimately, we intend to employ $\Gamma^{\ast}$ to transform the original training feature set $\mathbf{X}^{tr}$ to the optimal training feature set $\mathbf{X}^{\ast}$. We expect the feature categories in $\mathbf{X}^{\ast}$ can also be the optimal feature categories on the test dataset $\mathcal{D}^{te}$ under distribution shifts.

W7:

We carefully checked these minor issues and fixed them.

We hope that our response can address your concerns.