Automated Feature Engineering by Prompting

Dear Reviewer gWBg,

Thank you for acknowledging our selection of datasets and tasks and the clarity of the paper. We want to address the highlighted weaknesses and answer the posed questions to improve our paper's quality and clarity. Next we reply to your comments.

---

The new feature space generated by this method is limited, and it can only generate new features based on given operators.

In this work, we explore an alternative to representation learning for automatically extracting feature information from datasets, i.e., AutoFE, which is a fundamental aspect of data-centric AI. AutoFE is particularly well-suited for small datasets, where the limited data may be insufficient to train a deep neural network for representation learning. Our LLM-based AutoFE approach generates meaningful features by performing mathematical feature-feature crossing while also providing semantic explanations. The use of pre-defined transformation operators enhances both the controllability of the feature generation process and the interpretability of the extracted features.

We select the same set of feature transformation operators as those in DIFER [Zhu22] to ensure a fair comparison. It is worth noting that the set of operators in our method is easily extendable, allowing for improved performance by generating features within a broader search space.

It is impossible to conduct feature engineering without a give set of operators. Even if features are specified by code, it still means a finite set of operators.

[Zhu22] DIFER: Differentiable automated feature engineering. The first International Conference on Automated Machine Learning, 2022

---

Innovation is limited.

We propose a novel approach to LLM-based automated feature engineering that addresses the weaknesses of the state-of-the-art approach CAAFE [Hollmann2023]. We add a discussion on the novelty in new Section G. We highlight the key differences between our work FEBP and CAAFE as follows

1. In FEBP, we provide top-performing constructed features in the prompt as learning examples, label them with performance scores, and rank them by score. CAAFE instead stores all previous instructions and code snippets in the conversation history. The advantages of our approach include
   • More effective in-context learning of feature patterns.
   • The LLM generation cost stays constant across feature search iterations, without consuming extra LLM context.
   • Stronger capability of performing feature search in large search spaces by enabling more iterations, providing pronounced benefits in presence of numerous feature attributes.
2. In FEBP, we represent features in the form of canonical RPN (cRPN), while CAAFE represents features with Python code. The advantages of our approach include
   • cRPN is more compact, reducing LLM generation costs.
   • The compactness of cRPN makes in-context learning of feature patterns easier.
   • The use of pre-defined operators reduces the search space and simplifies the learning process for optimizing feature construction.
   • Our approach gives better control, such as the number of operators to use in features, and helps avoid undesirable or unexpected LLM outputs, such as malicious code snippets.
   • It is convenient to import external features and export the results as individual features, providing compatibility with other AutoFE methods.
3. We demonstrate in this work that general-purpose LLMs like GPTs can effectively model recursive tree structures in the form of cRPN feature expressions and reason about the structures in the context of semantic information.

We hope this explanation makes the contributions of our work clearer.

[Hollmann2023] Large language models for automated data science: Introducing CAAFE for context-aware automated feature engineering. NIPS, 2023

Lacks more detailed analysis and module design.

We have examined the impact of dataset semantic context on performance and feature construction efficiency in Section 5.3. We have provided detailed analyses on the LLM-based feature search process in Section 5.4 and the effects of hyperparameters in Section 5.5.

We add a discussion on why we use canonical RPN in new Section A. We represent features in RPN because it is compact and unambiguous and it better encodes the recursive structure for sequential modeling. We canonicalize RPN because that ensures the consistency of feature representations and facilitates the in-context learning of feature patterns. Our canonicalization scheme also introduces left skewness to the expression tree that enhances the clarity of the recursive structure in cRPN.

We add an additional ablation study regarding RPN canonicalization in new Section E. We summarize the new experimental results as follows

Without RPN canonicalization, we observe performance degradation using all three downstream models. The Friedman-Nemenyi post-hoc test shows that the performance difference is statistically significant at the p=0.05 level for the cases with GPT-3.5 and post-AutoFE parameter tuning as well as GPT-4 without post-AutoFE parameter tuning. We also observe a decrease in the number of LLM responses without canonicalization. This is because the expression becomes more flexible, reducing the likelihood of duplication with existing features. Additionally, we find that when switching to prefix feature expressions, the LLM encounters difficulty generating syntactically valid feature expressions, leading to a failure to complete one single run in our experiments.

Furthermore, we add an additional feature analysis in new Section F.1. We compare the proportions of constructed features selecting each feature attribute in the datasets for both the full and semantically blinded versions of FEBP. In the blinded version, we observe that the LLM tends to prioritize earlier feature attributes in the dataset while paying less attention to later ones, reflecting an inherent bias of the language model. In contrast, in the full version, the selection of feature attributes is guided by the semantic information of the dataset rather than the positional order of the attributes. This demonstrates the role of dataset semantic information in the LLM-based feature construction process.

We also add an additional analysis of feature importance in new Section F.2. We find that the features constructed by our method have high importance in the downstream models. Moreover, our method adaptively adjusts the feature complexity to suit different downstream models.

---

I think the analysis of temperature is redundant.

The sampling temperature of the LLM is a relevant hyperparameter in our method, the other relevant one being the number of example features in the prompt. As discussed in lines 252-254 of our paper:

The sampling temperature of the LLM can be adjusted to balance exploration and exploitation, with higher temperatures encouraging more diverse solutions and lower temperatures favoring incremental changes to existing examples.

In essence, the temperature functions similarly to the learning rate in optimization algorithms. From our experimental results presented in Table 4, the temperature impacts both performance and feature construction efficiency of our method with statistical significance. We select the optimal temperature (1) based on validation performance for achieving the best test results.

The speculation in line 320 does not explain the reason for this phenomenon.

To verify our speculation in line 320, we conduct an additional experiment with GPT-4 by varying the number of example features in the prompt. We summarize the validation performance as follows

We observe improved performance as the number of example features increases. This indicates that incorporating more example features helps fully leverage GPT-4's enhanced in-context learning capabilities. We will include these results in the paper. For our evaluations, we set the number of example features to 10 for a fair comparison with GPT-3.5.

Dear Reviewer nSbg,

Thank you for acknowledging the importance of our research problem and the clarity of the paper. We are pleased that you like our attempt at formalism. We next address the highlighted weaknesses to improve our paper's quality and clarity.

---

The nature of the contribution.

We propose a novel approach to LLM-based automated feature engineering that addresses the weaknesses of the state-of-the-art approach CAAFE [Hollmann2023]. We add a discussion on our contributions in new Section G. We highlight the key differences between our work FEBP and CAAFE as follows

1. In FEBP, we provide top-performing constructed features in the prompt as learning examples, label them with performance scores, and rank them by score. CAAFE instead stores all previous instructions and code snippets in the conversation history. The advantages of our approach include
   • More effective in-context learning of feature patterns.
   • The LLM generation cost stays constant across feature search iterations, without consuming extra LLM context.
   • Stronger capability of performing feature search in large search spaces by enabling more iterations, providing pronounced benefits in presence of numerous feature attributes.
2. In FEBP, we represent features in the form of canonical RPN (cRPN), while CAAFE represents features with Python code. The advantages of our approach include
   • cRPN is more compact, reducing LLM generation costs.
   • The compactness of cRPN makes in-context learning of feature patterns easier.
   • The use of pre-defined operators reduces the search space and simplifies the learning process for optimizing feature construction.
   • Our approach gives better control, such as the number of operators to use in features, and helps avoid undesirable or unexpected LLM outputs, such as malicious code snippets.
   • It is convenient to import external features and export the results as individual features, providing compatibility with other AutoFE methods.
3. We demonstrate in this work that general-purpose LLMs like GPTs can effectively model recursive tree structures in the form of cRPN feature expressions and reason about the structures in the context of semantic information.

We hope this explanation makes the contributions of our work clearer.

[Hollmann2023] Large language models for automated data science: Introducing CAAFE for context-aware automated feature engineering. NIPS, 2023

Weakness in the experimental section.

We have examined the impact of dataset semantic context on performance and feature construction efficiency in Section 5.3. We have provided detailed analyses on the LLM-based feature search process in Section 5.4 and the effects of hyperparameters in Section 5.5.

We add an additional ablation study regarding RPN canonicalization in new Section E. We summarize the new experimental results as follows

Without RPN canonicalization, we observe performance degradation using all three downstream models. The Friedman-Nemenyi post-hoc test shows that the performance difference is statistically significant at the p=0.05 level for the cases with GPT-3.5 and post-AutoFE parameter tuning as well as GPT-4 without post-AutoFE parameter tuning. We also observe a decrease in the number of LLM responses without canonicalization. This is because the expression becomes more flexible, reducing the likelihood of duplication with existing features. Additionally, we find that when switching to prefix feature expressions, the LLM encounters difficulty generating syntactically valid feature expressions, leading to a failure to complete one single run in our experiments.

Additionally, we add a discussion on why we use canonical RPN in new Section A. We represent features in RPN because it is compact and unambiguous and it better encodes the recursive structure for sequential modeling. We canonicalize RPN because that ensures the consistency of feature representations and facilitates the in-context learning of feature patterns. Our canonicalization scheme also introduces left skewness to the expression tree that enhances the clarity of the recursive structure in cRPN.

Furthermore, we add an additional feature analysis in new Section F.1. We compare the proportions of constructed features selecting each feature attribute in the datasets for both the full and semantically blinded versions of FEBP. In the blinded version, we observe that the LLM tends to prioritize earlier feature attributes in the dataset while paying less attention to later ones, reflecting an inherent bias of the language model. In contrast, in the full version, the selection of feature attributes is guided by the semantic information of the dataset rather than the positional order of the attributes. This demonstrates the role of dataset semantic information in the LLM-based feature construction process.

We also add an additional analysis of feature importance in new Section F.2. We find that the features constructed by our method have high importance in the downstream models. Moreover, our method adaptively adjusts the feature complexity to suit different downstream models.

In carefully looking at the results, I am seeing hardly any difference between the authors' approach and baselines when using classifiers like Light-GBM and random forests.

We compute the mean percentage performance improvement of our method FEBP over the baseline methods with GPT-3.5 and GPT-4, respectively. We add the full results to our revised version as Tables 8 and 9 in New Section D.5.

With GPT-3.5,

With GPT-4,

To examine the performance difference when using Random Forests and LightGBM, we perform additional statistical tests excluding linear model results and add the new results to Section D.7. The p-values from the Nemenyi post-hoc test are summarized as follows

We note that FEBP with GPT-3.5 and post-AutoFE parameter tuning significantly outperforms all baselines except DIFER at the p = 0.05 level. With GPT-4, the performance difference between FEBP and CAAFE is statistically significant at the p = 0.05 level. We have highlighted the corresponding parts of Table 16 in our revised version for better clarity.

We hope this better shows the performance improvement of our approach over the baselines when using Light-GBM and random forests.

We would like to note that the reason why the performance difference is less salient when using Light-GBM and random forests is because these models can capture complex non-linear relationships from raw features themselves, which dilutes the effect of feature engineering. Thus, in our view, the performance improvement when using simple downstream models like linear models better demonstrates the efficacy of AutoFE.

Dear Reviewer ogYM,

We apologize for the delayed response. With the rebuttal period extended, we took additional time to thoroughly review and address your questions.

Thank you for your insightful review. We are pleased to find that the soundness, replicability, and explainability of our approach were recognized as strengths. We want to address the highlighted weaknesses and answer the posed questions to improve our paper's quality and clarity. Please find below replies to your comments.

The significance of some results is unclear.

We perform additional one-tailed paired t-tests to assess the statistical significance of the results presented in Tables 4 and 5. In Table 4, the performance improvement with the temperature at 1 compared to the temperature at 0.5 (0.8197 vs 0.8169) yields a p-value of less than 0.006. In Table 5, the performance improvement with 10 prompt examples compared to 1 prompt example (0.6840 vs 0.6823) yields a p-value of approximately 0.06. This shows that both the temperature and the number of example features in the prompt have a significant impact on the performance. We select the optimal temperature (1) and number of prompt examples (10) based on validation performance for achieving the best test results. We will include these significance results in the paper.

In addition, we compute the mean percentage performance improvement of our method FEBP over the baseline methods with GPT-3.5 and GPT-4, respectively. We have added the new results to our revised version as Tables 8 and 9 in New Section D.5. To examine the performance difference between our method and the baselines when using Random Forests and LightGBM, we perform additional statistical tests excluding linear model results and add the new results to Section D.7. We have highlighted the relevant statistical test results in Tables 14-16 for clarity.

How many responses on average are necessary to train a downstream model for classification and/or regression?

From our experiments in Section 5.3, the average number of responses to train a downstream model is 362.7 with GPT-3.5 and 331.9 with GPT-4.

We summarize the results as follows

Did you consider using reflection and providing previous prompts as input to the model apart from in context examples?

Thank your for the advice. We have experimented with providing previous prompts and LLM outputs as input to the LLM. However, we observe a degradation in performance as feature generation converges prematurely to suboptimal feature spaces. Additionally, the extra input significantly increases LLM generation costs. In contrast, our approach to providing feedback through in-context examples enables the LLM to explore a broader range of diverse features while remaining cost-efficient.

Dear Reviewer B4Hm,

Thank you for acknowledging the importance of our research problem and the clarity of the paper. We want to address the highlighted weaknesses and answer the posed questions to improve our paper's quality and clarity. Please find below replies to your comments.

---

W1. The paper does not provide enough justification/discussion on its difference to CAAFE.

We propose a novel approach to LLM-based automated feature engineering that addresses the weaknesses of CAAFE. We add a discussion on the differences in new Section G. We highlight the key differences between our work FEBP and CAAFE as follows

1. In FEBP, we provide top-performing constructed features in the prompt as learning examples, label them with performance scores, and rank them by score. CAAFE instead stores all previous instructions and code snippets in the conversation history. The advantages of our approach include
   • More effective in-context learning of feature patterns.
   • The LLM generation cost stays constant across feature search iterations, without consuming extra LLM context.
   • Stronger capability of performing feature search in large search spaces by enabling more iterations, providing pronounced benefits in presence of numerous feature attributes.
2. In FEBP, we represent features in the form of canonical RPN (cRPN), while CAAFE represents features with Python code. The advantages of our approach include
   • cRPN is more compact, reducing LLM generation costs.
   • The compactness of cRPN makes in-context learning of feature patterns easier.
   • The use of pre-defined operators reduces the search space and simplifies the learning process for optimizing feature construction.
   • Our approach gives better control, such as the number of operators to use in features, and helps avoid undesirable or unexpected LLM outputs, such as malicious code snippets.
   • It is convenient to import external features and export the results as individual features, providing compatibility with other AutoFE methods.
3. We demonstrate in this work that general-purpose LLMs like GPTs can effectively model recursive tree structures in the form of cRPN feature expressions and reason about the structures in the context of semantic information.

We hope this explanation makes the contributions of our work clearer.

---

W2. The experimental evaluation does not show significant improvement against CAAFE.

In Section D.7 (Section C.5 of the old version), we perform statistical tests to verify that the performance improvement of our method FEBP over CAAFE is statistically significant. The p-values from the Nemenyi post-hoc test are summarized as follows

We note that the performance improvement of our method FEBP is statistically significant at the p=0.01 level.

To examine the performance difference when using Random Forests and LightGBM, we perform additional statistical tests excluding linear model results. The p-values from the Nemenyi post-hoc test are summarized as follows

We note that FEBP with GPT-3.5 and post-AutoFE parameter tuning significantly outperforms CAAFE in all cases at the p = 0.05 level. With GPT-4, the performance difference between FEBP and CAAFE is statistically significant at the p = 0.05 level.

We have highlighted the corresponding parts of Tables 14 and 16 in our revised version for better clarity.

W3. Adding some ablation studies will be appreciated. There lacks of experimental evaluations to support the design choice, e.g. cRPN.

Thank your for the advice. We add an additional ablation study regarding RPN canonicalization in new Section E. We summarize the new experimental results as follows

Without RPN canonicalization, we observe performance degradation using all three downstream models. The Friedman-Nemenyi post-hoc test shows that the performance difference is statistically significant at the p=0.05 level for the cases with GPT-3.5 and post-AutoFE parameter tuning as well as GPT-4 without post-AutoFE parameter tuning. We also observe a decrease in the number of LLM responses without canonicalization. This is because the expression becomes more flexible, reducing the likelihood of duplication with existing features. Additionally, we find that when switching to prefix feature expressions, the LLM encounters difficulty generating syntactically valid feature expressions, leading to a failure to complete one single run in our experiments.

Moreover, we add a discussion on why we use canonical RPN in new Section A. We represent features in RPN because it is compact and unambiguous and it better encodes the recursive structure for sequential modeling. We canonicalize RPN because that ensures the consistency of feature representations and facilitates the in-context learning of feature patterns. Our canonicalization scheme also introduces left skewness to the expression tree that enhances the clarity of the recursive structure in cRPN.

Furthermore, we add an additional feature analysis in new Section F.1. We compare the proportions of constructed features selecting each feature attribute in the datasets for both the full and semantically blinded versions of FEBP. In the blinded version, we observe that the LLM tends to prioritize earlier feature attributes in the dataset while paying less attention to later ones, reflecting an inherent bias of the language model. In contrast, in the full version, the selection of feature attributes is guided by the semantic information of the dataset rather than the positional order of the attributes. This demonstrates the role of dataset semantic information in the LLM-based feature construction process.

We also add an additional analysis of feature importance in new Section F.2. We find that the features constructed by our method have high importance in the downstream models. Moreover, our method adaptively adjusts the feature complexity to suit different downstream models.

Dear Reviewer B4Hm,

Thank you very much for your response. We are glad that we have addressed your concern on W3. Next we provide additional information to address your concern on W2.

W2. The experimental evaluation does not show significant improvement against CAAFE.

We compute the mean percentage performance improvement of our method FEBP over the baseline methods with GPT-3.5 and GPT-4, respectively. We add the full results to our revised version as Tables 8 and 9 in New Section D.5.

With GPT-3.5,

With GPT-4,

We observe that our method has a significant amount of performance improvement over CAAFE. The greatest performance improvement over CAAFE is observed when using linear models.

We would like to note that the reason why the performance improvement is less salient when using RF and LGBM is because these models can capture complex non-linear relationships from raw features themselves, which dilutes the effect of feature engineering. Thus, in our view, the performance improvement when using simple downstream models like linear models better demonstrates the efficacy of AutoFE.

Dear Reviewers,

Thank you again for your review. We summarize the key points in our responses

Reviewer B4Hm

• We explain the differences between our work and CAAFE. Our approach addresses the weaknesses of CAAFE by adopting compact feature representations and providing example features in the prompt, leading to stronger feature search performance.
• We compute the relative performance improvement of our method over CAAFE and perform statistical tests to show that the performance improvement of our method over CAAFE is significant.
• We conduct an additional ablation study on RPN canonicalization. We add additional explanations on why we use canonical RPN for feature representation.

Reviewer ogYM

• Statistical tests show that the performance differences in our studies on the LLM temperature and the number of example features are significant.
• We explain the number of LLM responses on average to train a downstream model.
• We find that providing previous prompts to the LLM would lead to premature convergence to suboptimal feature spaces. In contrast, our approach enables the LLM to explore a broader range of diverse features while remaining cost-efficient.

Reviewer gWBg

• AutoFE suits small datasets and provides controllability of the feature generation process and the interpretability of the extracted features, which requires a give set of operators.
• We propose a novel approach to LLM-based automated feature engineering that addresses the limitations of the state-of-the-art approach.
• We conduct an additional ablation study on RPN canonicalization and perform additional analyses on the selection of feature attributes and the feature importance.
• Temperature is a relevant hyperparameter that impacts both the performance and the feature construction efficiency of our method.
• The performance with GPT-4 improves as the number of example features increases. Incorporating more example features helps fully leverage GPT-4's enhanced in-context learning capabilities.

Reviewer nSbg

• We propose a novel approach to LLM-based automated feature engineering that addresses the limitations of the state-of-the-art approach.
• We conduct an additional ablation study on RPN canonicalization and perform additional analyses on the selection of feature attributes and the feature importance.
• The relative performance improvement of our method over the baselines and additional statistical tests show that there are significant differences in performance when using Light-GBM and random forests.

In this work, we introduce a novel LLM-based AutoFE approach that addresses the limitations of the state-of-the-art method. Through extensive experiments, we demonstrate that our approach achieves superior performance compared to state-of-the-art baselines. We provide detailed analyses on the impact of dataset semantic information, RPN canonicalization, and hyperparameter settings. Additionally, we present details of the LLM-based feature search process, the selection of feature attributes, and the evaluation of feature importance. We believe our work makes valuable contributions to the fields of automated feature engineering and LLM-driven applications. We respectfully request your consideration of our paper for acceptance. Thank you for your time and thoughtful review.

Best regards,

Authors