Optimized Feature Generation for Tabular Data via LLMs with Decision Tree Reasoning

Dear Reviewer 8DXj,

We sincerely appreciate the time and effort you dedicated to reviewing our manuscript and providing insightful comments. Below, we address each of your points individually.

---

[W1, 2] The framework is costly and time-consuming (with no training time comparison) and involves some manual operations.

Thank you for your comment. As we described in Section 5 of our manuscript, one potential limitation is that it may be time-consuming if the prediction model requires extensive training. In this regard, we acknowledge that our method could be slower compared to competing methods like OpenFE [3]. Thus, we focused on evaluating the effectiveness of our method rather than the run time. However, once the feature generation rules are optimized, new column features can be generated simply by applying these rules to any additional data that becomes available.

As we also described in our draft, there are several approaches to further scaling our method:

• Feature transfer: As demonstrated in Table 6, it is possible to first generate features using a simpler model like XGBoost, and then transfer these features to the more complex target model like MLP. This allows for faster rule generation for more complex prediction models.
• Use of open LLMs: Unlike methods that rely on proprietary LLMs such as ChatGPT, we demonstrated that open models like Llama 2, with minimal additional fine-tuning, can be highly effective in our feature generation framework. This approach avoids the cost associated with paid APIs.
• Use of data subsets: We have shown our method is effective across datasets of varying sizes. This suggests that during rule generation, a suitable subset of the training data can be used to accelerate optimization, enabling scalability to larger datasets.

Lastly, we would like to point out that the only significant manual task involved is designing an appropriate prompt for the rule generator LLM. The version used in our experiments will be fully released alongside the code.

---

[W3] Scalability: Number of features.

Thanks for the valuable suggestion. In response, we evaluate the scalability of our method on datasets with hundreds of features (e.g., 501) using datasets from the OpenML repository. We chose XGBoost as the baseline because it was the most competitive baseline in our main experiments (see Table 2). Additionally, we used GPT-3.5-Turbo as the rule generator because the Llama2-based model we primarily used in our work is constrained by a maximum context length of 2048, which becomes limiting as the prompt size increases with the number of features. However, we emphasize that our method is compatible with an arbitrary LLM as the rule generator, as shown in Tables 1 and 3 of our manuscript.

As shown in the table below, our method scales effectively to datasets with a larger number of features. For example, on the madelon dataset, which contains 501 columns, our method reduces the relative error by 6.3% compared to the XGBoost. Here, we report test error, with values in parentheses indicating the reduction in relative error rates. We will incorporate these results into the final draft.

\begin{array}{lccc} \hline \text{Dataset}&\text{\# features}&\text{Baseline}&OCTree (Ours)\newline\hline \text{madelon}&501&21.54&20.19 (6.3\%)\newline \text{nomao} &119&3.08&2.84 (7.8\%)\newline\hline \end{array}

---

[W4] Scalability: Number of samples.

Thanks for the valuable suggestion. Following the suggestion, we evaluate the scalability of our method on larger datasets from the OpenML repository using XGBoost. As shown in the table below, our method scales effectively to datasets of a much larger scale. For example, in the nyc-taxi-green-dec-2016 dataset, which contains over 500,000 samples, our method achieves a 13.7% reduction in relative error compared to the baseline. Here, we report test error, with values in parentheses indicating the relative error reductions. We will incorporate these results into the final draft.

\begin{array}{lccc} \hline \text{Dataset}&\text{\# samples}&\text{Baseline}&OCTree (Ours)\newline\hline \text{nyc-taxi-green-dec-2016}&\text{581,835}&2.91&2.51 (13.7\%)\newline \text{Covertype}&\text{423,680}&2.79&2.15 (22.9\%)\newline\hline \end{array}

---

[Q1] Are there other works that use LLMs for feature generation? How does this approach differ from them?

As described in Section 2 of our manuscript, CAAFE [1] proposes a context-aware automatic feature engineering method that leverages LLMs but is distinct from OCTree in several key ways. Please refer to the global response above for more details. We thank the reviewer for the question and will incorporate the results in the final manuscript.

---

[Q2] Associated time cost.

Thanks for your question regarding various aspects of our method's efficiency.

First, the primary time cost comes from (a) using the LLM to suggest rules and (b) training the prediction model on the suggested features to compute validation scores. Directly training an XGBoost would involve a single training run on the original dataset, whereas our method involves multiple iterations until the optimal features are generated. However, once the rules have been generated, applying these rules to any additional data is trivial.

Second, while dataset size and feature dimensionality mainly affect the training time of the prediction model, they have minimal effect on LLM inference. Additional features may slightly increase the prompt length, but the overall size of the training dataset has little impact on inference time.

Lastly, please refer to the approaches to scaling the method we outlined in our response to [W1, 2]. These include using simpler prediction models and leveraging feature transfer, using open LLMs of moderate size like Llama 2 at the 7B scale, and using a subset of the training data during rule generation to improve efficiency.

Dear Reviewer 245r,

We sincerely appreciate the time and effort you dedicated to reviewing our manuscript and providing insightful comments. Below, we address each of your points individually.

---

[W1] Difficulty following the notation and the algorithm. Authors should release the code.

Thank you for your feedback. While several reviewers have noted that the paper is generally well-structured and clearly written, we acknowledge that some notations could be clearer. We will review and clarify these notations to improve understanding.

Regarding the availability of our code, we plan to release it in full. However, due to anonymity requirements, we are providing a version with any author-identifiable components removed. Per the guidelines, we have submitted the link to the AC, so that it can be made available for your review.

---

[W2] How does the proposed method tackle the limitations of other feature engineering methods that require setting the manual search space?

We would like to clarify that our approach enables the LLM to automatically generate rules without requiring a predefined optimization space. In contrast, other feature engineering methods, such as OpenFE [3], depend on predefined search spaces (e.g., using predefined operators like + or -) to generate candidates. For example, we use prompts like “Give me a new rule that is different from the old ones and has a score as high as possible,” therefore generating features with the rules we get from LLM’s suggestions alone, without any specification of the search space (see Section A.3 of our manuscript). Consequently, our LLM-based framework fully automates the optimization process without the need for a predefined search space.

---

[W3, Q1, Q2, L1] Comparison with CAAFE [1] and why orthogonal?

We would first like to clarify that, as mentioned in Section 4.1 of our manuscript, we originally excluded CAAFE from our comparison as it requires a language-based context for the dataset. This requirement limits CAAFE's applicability in cases where language descriptions are not explicitly provided, such as when feature names and values are obfuscated to protect confidentiality, as is common in financial and medical datasets. In contrast, the methods that we evaluated in Table 4, as well as our approach, are context-agnostic and can be applied to datasets without such linguistic descriptions. However, our method does benefit from clear language descriptions, as shown in Table 1, when available.

To further address your concern, we have compared our method to CAAFE and highlighted some key differences and distinctions. Please refer to the global response above for more details. We thank the reviewer for the suggestion and will incorporate the results in the final manuscript.

---

[W4, Q4] Decision tree reasoning: How it improves OCTree is not explained in the paper and needs evidence on how including rules increases the algorithm’s efficiency.

We would like to first clarify that decision tree reasoning refers to the language description of a decision tree constructed from the entire dataset, including any newly generated features. We have already conducted an ablation study, detailed in Table 5, to assess the effects of providing decision tree reasoning as feedback to the rule generator LLM. Here, our findings indicate that incorporating decision tree reasoning leads to the generation of higher-quality features, resulting in an additional performance improvement of up to 5%. This is because decision tree reasoning provides valuable insights learned from the entire training set, as it highlights the columns that are considered more significant (as nodes in the tree) and the corresponding values (as thresholds of the nodes) used for prediction. We will incorporate these points more clearly into our final manuscript.

---

[Q3] Clarification of notation: Meaning of $D \oplus r$.

We would like to clarify that the notation $D \oplus r := \\{x_i \oplus r(x_i), y_i \\}_{i=1}^{N}$ represents the dataset in which each sample $x_i$ is augmented with an additional feature generated by applying the rule $r$. To address your concerns and enhance clarity, we plan to update this notation to $D^r$ in the final draft.

---

[L2] Risk of memorization.

We would like to emphasize that our method is completely free from the risk of memorization because it generates new column features that are not present in the original dataset. Even if the LLM has memorized information from open datasets, our approach does not rely on the specific data; instead, it generates and introduces entirely new columns to the dataset.

To further address your concern regarding the potential impact of the LLM being trained on open datasets, we would like to highlight that most of the datasets used in Table 1 (e.g., Tesla, Enefit, Disease, Clinical) were released after the release of Llama 2, our base LLM. The fact that our method enhances the performance of various baseline models on this dataset demonstrates its capability to generate useful features without relying on dataset-specific information. Moreover, Table 2 presents the results of experiments conducted on datasets that lack semantic information about the columns. We used generic indicators (e.g., 'x1', 'x2') for columns and applied ordinal encoding and a min-max scaler to transform all feature values. Even in this setting, our method was able to enhance the performance of a range of baseline models across datasets of varying sizes and characteristics.

if ‘Has difficulty breathing’:
     if ‘Has fever’:
          ‘Subject has a disease’
     else:
          ‘No disease’
…

I have some rules to predict {Objective} with attributes listed below.
- {Column #1 Name} (Numerical value of {Column #1 Min} ~ {Column #1 Max}):  {Column #1 Feature Description}
- {Column #2 Name} (Boolean):  {Column #2 Feature Description}
- {Column #3 Name} (Categorical value of {Value #1}, {Value #2}, {Value #3}):  {Column #3 Feature Description}
...

We also have corresponding decision trees (CART) to predict {Objective} from the attributes listed above along with predicted {New Column Name}.
The rules are arranged in ascending order based on their scores evaluated with XGBoost classifier, where higher scores indicate better quality.

Rule to predict {Objective}:
{Rule #1}
Decision tree (CART):
{Decision Tree #1}
Score evaluated with XGBoost classifier:
{Score #1}

Rule to predict {Objective}:
{Rule #2}
Decision tree (CART):
{Decision Tree #2}
Score evaluated with XGBoost classifier:
{Score #2}

...

Give me a new rule to predict {Objective} that is different from the old ones (but should use the listed attributes above) and has a score as high as possible.

Improved rule:

Dear Reviewer Pmbp,

We sincerely appreciate the time and effort you dedicated to reviewing our manuscript and providing insightful comments. Below, we address each of your points individually.

---

[W1] Isn’t it contrary to the original intent of feature engineering if validation scores are not used as an evaluation metric?

We would like to first clarify that in our framework, validation scores are indeed a key objective for optimization and are provided as feedback to the rule generator LLM. However, instead of providing just the most recent validation score, we additionally provide a history of the scores to help the LLM better understand the optimization landscape and iteratively refine and improve the rules it generates. We also found that incorporating decision tree reasoning as input is beneficial (see Table 5 in our manuscript). This reasoning serves as a summary of the training dataset, giving the LLM a deeper understanding of the data's structure and relationships. In summary, utilizing information such as a history of validation scores and insights into the training data is essential for enabling the rule generator LLM to effectively handle this complex optimization task.

---

[W2] In-depth discussion: The authors only claim that validation scores are flawed feedback mechanisms but need to explore some of the newer methods in the main text.

Thanks for your feedback and for pointing out the related work. As we highlighted above, we want to clarify that validation scores are indeed a key objective we optimize for in our framework. However, we have empirically shown that relying solely on validation scores in a greedy manner can be suboptimal. Our findings show that it is crucial to consider sufficient history of the scores to effectively navigate the optimization landscape. Additionally, incorporating inputs like decision tree reasoning can improve feature quality by providing a richer context for the task.

Regarding related methods, framing feature generation as a sequential decision-making problem allows techniques such as reinforcement learning to be applied. Similar to our basic approach, these methods generate feature candidates and evaluate them on downstream tasks for iterative improvement. However, some of these methods require solving multiple Markov decision processes [4, 5], which can be complex and computationally demanding. In contrast, our framework offers a conceptually much simpler approach that is easier to implement. Our experiments show that an open LLM of moderate size can effectively serve as a rule generator across various datasets and prediction models. With optional fine-tuning to enhance chat or code generation capabilities, the model performs well without needing additional training or customization for each specific setting.

We will incorporate these points more clearly into our manuscript.

---

[W3-1] If the rules generated in step 1 are inaccurate, the generated column might also be inaccurate.

While it is true that the rule generator LLM can occasionally suggest suboptimal rules, we provide feedback on previously generated rules, guiding the LLM to iteratively refine and enhance its rule generation. This feedback loop is an important component of our framework, allowing the LLM to converge toward more effective and accurate rules.

---

[W3-2] Can the concept automatically be formed during the generation of decision trees or the training of neural networks rather than requiring such an inaccurate combination?

You are indeed correct that decision trees inherently represent a disjunction of conjunctions of feature values and that neural networks learn latent features through hidden layers. However, our empirical findings suggest that generating explicit features using our approach and incorporating them as additional inputs yields better results than relying solely on these models to infer implicit features. We suspect this is because providing relevant features as explicit inputs enables these models to allocate their capacity more effectively, focusing their degrees of freedom on learning more sophisticated combinations of features that capture subtle patterns within the data.

---

[W3-3] If not, do the authors have experiments to validate this claim?

We have indeed conducted empirical evaluations to examine the impact of our generated features on model performance. Specifically, we compared the performance of XGBoost and MLP models trained without additional features (relying solely on the implicit features learned by the models) against models trained with the features generated using our framework as explicit input.

As shown in Tables 1 and 2 in our manuscript, we have observed performance improvements on most datasets, with some showing relative performance gains of more than 20%. These results show that providing the generated features as explicit input can significantly improve the model's ability to converge and perform well, compared to relying solely on implicit feature learning.

---

[W4] Ensuring generated column names are valid.

We would like to clarify that we have already verified that the LLM indeed generates valid column names. Specifically, we have confirmed that (i) it is beneficial to use the actual values of the generated columns if they are available, and (ii) when given a candidate for a new column name, the LLM can effectively distinguish between columns that are more relevant to the target task (see Section 4.2 of our manuscript).

---

[W5] Table 2: Some datasets did not improve or only showed general improvement. Why?

We hypothesize that the limited effectiveness observed with some datasets with non-linguistic descriptions stems from the absence of meaningful semantic information that the LLM could leverage during rule generation. However, we would like to emphasize that for most datasets, the generated features enhanced the performance of the prediction models evaluated.

Dear Reviewer 6JM8,

We sincerely appreciate the time and effort you dedicated to reviewing our manuscript and providing insightful comments. Below, we address each of your points individually.

---

[W1, Q1] The authors did not compare their method to other feature generation techniques, in particular, CAAFE [1].

We want to first clarify that we have already compared our method with other techniques for automated feature generation, such as AutoFeat [2] and OpenFE [3], and verified that our method performs considerably better (see Table 4 in our manuscript). Furthermore, unlike competing methods, our approach generates semantically meaningful features that human annotators can use to provide additional labels, which can significantly enhance model performance. For example, as shown in Figure 3, collecting real values for the suggested features significantly improves patient mortality prediction.

To address your concern further, we compared our method to CAAFE. Please refer to the global response above for additional details. We thank the reviewer for the suggestion and will incorporate the results in the final manuscript.

---

[W2-1, Q2] Number of prompting rounds and results across multiple executions.

To find an effective feature generation rule, we execute 50 rounds of LLM prompting. In each round, LLM is provided with an optimization trajectory, including reasoning information highlighting past experiments. Consequently, the input prompts vary across rounds, with later rounds incorporating more accumulated information.

To further address your comment, we describe how the output rules evolve throughout the optimization rounds on the electricity dataset. Due to space constraints, we show the first and last five output rules below. In the early optimization stages, the LLM produces diverse outputs, suggesting active exploration of possible rules. In contrast, during the later stages, the LLM refines the solution space around previously discovered solutions, making only minor adjustments. We hope this additional analysis provides deeper insight into our method.

First five:

• x12 = x3 + (x5 * x8) * x2 - x6
• x12 = x5 * x2 + (x3 * x4 - x6) - x1
• x12 = x11 + (x7 * x10) * x2 - x8
• x12 = x3 + (x6 * x8 * x2) - x1
• x12 = (x1 * x6 + x2 * x8) * x3 - x7

Last five:

• x12 = ((x2 - x1) * (x11 - x4)) ** 2 - (0.5 * (x1 - 0.3))
• x12 = ((x2 - x1) * (x11 - x4)) ** 6
• x12 = ((x2 - x1) * (x11 - x4)) ** 14 - (0.02 * (x5 - x7)) - 0.5
• x12 = (x11 - x1) + (x11 * (x6 - x7))
• x12 = ((x2 - x1) * (x11 - x4)) ** 3 + (0.1 * (x5 - x6)) ** 2

---

[W2-2, Q4] Handling hallucinations and their frequency across different LLMs.

While it is true that the LLM can occasionally suggest suboptimal or semantically incoherent rules, our method is designed to overcome these hallucinations. Specifically, we provide feedback on previously generated rules to guide the LLM to iteratively improve its rule generation. This feedback loop helps the LLM avoid hallucinations, which have likely resulted in low validation scores in subsequent iterations. Empirically, we have found that these issues occur more frequently in the early stages when the LLM explores the rule space more extensively and for less capable LLMs, e.g., those without additional training on dialogue or code generation data.

---

[Q3] Case studies for different LLMs (e.g., results, patterns).

In our ablation study, we evaluated three different LLMs. As shown in Table 3 of our manuscript, our custom model (the Llama 2 trained additionally on a high-quality dialogue dataset) achieved the lowest average error rate, followed by the Code Llama and then the Llama 2 chat base, which still outperformed the baseline XGBoost. In terms of feature generation patterns, we observed several differences between the models.

Code Llama: This model utilizes various numpy operations (e.g., np.sin), which can generate mathematically sophisticated features.

• x9 = ((x4+0.24) * x1) + ((x5+0.27) * x2)
• x9 = np.sin(x1) * np.cos(x4)
• x9 = x4 * np.tan(x8)

Llama 2 chat base: This model favors various polynomial combinations.

• x9 = x4 * x1 * (x2 + x6) ** 2
• x9 = x4 * x1 ** 3 * (x2 + x6) ** 4 * (x7 + x8)
• x9 = x4 * x1 ** 2 * (x2 + x6) ** 3 * (x7 + x8) * (1 + x5 ** 4)

Ours: Our custom model also tends to explore polynomial space of features, often using built-in Python functions such as abs(). However, our model explores a considerably broader space of features more effectively than the Llama 2 chat base.

• x9 = x1 ** 2 * (x2 - x3)
• x9 = abs(x1) ** x1 + x2 - x3 - x4 - x5 - x6 - x7 - x8
• x9 = x1 * (x1 + 2) - x2 * (x2 - 0.5) * (x2 - 0.5)

These observations suggest that while all three models have strengths in feature generation, our custom model's broader exploration and more effective use of the feature space contribute to its superior performance. However, we believe equipping our custom model with enhanced code generation capabilities, such as utilizing scientific computing libraries like numpy, could lead to even more performance gains. We will incorporate these points into our final manuscript.

---

[Q6-2] Figure 3: Details on the dataset and evaluation with various metrics.

The dataset used in Figure 3 is nearly balanced with a class distribution of 55:45. Since it is a binary classification, the model outputs the class with the higher probability as the prediction result (i.e., no cut threshold). While accuracy is a suitable metric, we also report the ROC-AUC, a threshold-independent metric, in Figure 1 in the attached PDF to provide additional insights. Here, we highlight that using the real values improves all metrics, demonstrating that OCTree effectively recommends useful columns for the target task.

---

[Q6-1, 7] Editorial comments on Figure 3 and Table 7.

Thanks for your feedback. In the final draft, we will revise Figure 3 to be monochromatic (as in Figure 1 in the attached PDF) and make the best results bold in Table 7.