LLM Meeting Decision Trees on Tabular Data

R to W1: Thanks for your valuable comment.

(i) The intuition. Traditional methods like CART use information gain to make a series of locally optimal decisions, allowing a data-driven algorithm to find the best feature splits. This can lead to rules that are brittle or overfit to the specific data sample used for that tree. DeLTa, instead, utilizes the LLM to analyze diverse rules from a Random Forest and performs a logic-based reasoning task: (a) It recognizes patterns across multiple ''expert'' rules to identify features and decision paths that are consistently important. (b) It abstracts the core logic splits from individual trees. (c) It synthesizes a new, globally coherent rule that represents a consensus of the most robust patterns.

(ii) A concrete example of a better rule. As shown in Fig. 10 and Fig. 11 of Appendix, we provide a set of decision tree rules from Random Forest and refined rule by the LLM respectively. This is from the Credit dataset, where each sample (person) is classified as good or bad credit risks according to the set of features. To provide deeper insight into how the refined rule improves upon the original ones, we also provide the average SHAP values [1] of features under the Random Forest and our method respectively as follows. SHAP assigns each feature an importance value (higher is more important) for a particular prediction, thus it serves an unified framework for interpreting predictions.

The results show that, after LLM refinement, the SHAP importance of features such as ''credit_history'' (feature_16) and ''employment'' (feature_10) are increased, which aligns with the financial expertise. Let us take a closer look into Fig. 10 and Fig. 11 of Appendix, for ''credit_history'' (feature_16), this feature appears near the top of the second tree, indicating its potential significance. However, only 1 out of 3 decision trees in the original Random Forest rule set explicitly splits on this feature, which may dilute its influence in the ensemble prediction. For ''employment'' (feature_10), although it appears in 2 out of 3 rules, its splits occur late in the tree, leading to a low overall contribution in prediction. In contrast, the LLM-refined rule elevates the prominence of these features.

(iii) Why do traditional techniques not capture such a rule? More broadly, a single decision tree with information gain has the risk of overfitting. Although Random Forest ensembles the outputs from all tree rules, the inherent relationships and interactions among these rules are ignored. As the number of trees increases, analyzing these independent rules is gradually becoming more and more difficult, let alone utilizing these rules to partition feature space. To this end, we propose to leverage LLMs to analyze and summarize these rules into a refined rule due to the powerful logical reasoning ability of LLM. This enables the generation of improved partitioning strategies over the feature space, thereby promoting statistical coherence among samples assigned to the same leaf node. To verify it, we compute the intra-node sample distance over all leaf nodes partitioned by LLM refined rule $r^*$, and observe that the distance of $r^*$ is lower than original rule $r\in \mathcal{R}$ from Random Forest, as illustrated in Fig. 2 of original paper and Table 17 of Appendix.

[1] A Unified Approach to Interpreting Model Predictions. NeurIPS 2017.

R to W2: Following your suggestion, in addition to the computational cost comparison between DeLTa and LLM-based baselines reported in Table 3 of original paper, we now add a comparison with non-LLM baselines such as XGBoost and CART. The comparison is summarized below, where we provide runtime in seconds of different methods for the training and inference phase, conducted on Adult dataset. ''# Num'' indicates number of training samples. ''T'' and ''L'' correspond to traditional and LLM-based methods, respectively. It can be observed that, although DeLTa’s training time is longer than that of traditional tree-based models such as CART, XGBoost, and CatBoost, our DeLTa’s training cost is lower than neural methods designed specifically for tabular data, such as FT-Transformer and MNCA, and substantially lower than LLM-based baselines. The reason is that DeLTa only queries LLM via API to generate one refined rule for one dataset, without the need for per-sample prediction or fine-tuning LLM. Furthermore, at the testing stage, DeLTa makes decision by combining the trees and the Gradient Net, without querying LLM. Therefore, we can see that DeLTa exhibits similar inference costs to these tree-based methods and neural network-based methods, and incurs substantially lower cost compared to most of LLM-based baselines. To sum up, designed in this way, DeLTa enables significantly more efficient use of LLM resources, whose training and inference workflow are summarized in Algorithm 1 in the Appendix A.3.

R to W3: Thank you for raising this important point. We kindly remind you that our primary objective was to evaluate DeLTa's effectiveness in the context of LLM-based tabular prediction, and to ensure fair comparison with existing methods in this category. Accordingly, the majority of our classification datasets were selected based on their widespread use in recent LLM-based tabular learning studies such as TabLLM and FeatLLM, both of which primarily focus on classification tasks. To extend our evaluation beyond classification, we also incorporated regression datasets from the TALENT benchmark [2][3], which has recently gained attention for assessing tabular regression performance. In response to your suggestion, we have updated our experimental setup to include additional datasets from the WhyTrees benchmark. Specifically, we selected a subset of representative datasets that span different scales and feature characteristics, including the large-scale, high-dimensional "Year" dataset (as also recommended by Reviewer tga2), to evaluate DeLTa's scalability and generalizability. Due to time constraints during the rebuttal phase, we were only able to include a subset of WhyTrees datasets. However, we plan to expand this evaluation to a broader range of WhyTrees datasets in the final version. The updated results, summarized below, further demonstrate DeLTa's strong and consistent performance across a more diverse and standardized set of tabular tasks.

[2] A Closer Look at Deep Learning on Tabular Data. 2024.

[3] TALENT: A Tabular Analytics and Learning Toolbox. 2024.

R to W4: Thanks for the comment. We will carefully revised Section 4.2 to make it more clear. The core purpose of Section 4.2 is to introduce a calibration method for the original decision tree ensemble, $F(x)$. This is done by computing a sample-specific error correction vector, $\Delta_x$, to steer the prediction in the direction of error reduction. The process is as follows. Objective: We aim to adjust $F(x)$ by moving it toward the negative gradient of the loss function, which we define as the "error-reducing" direction. Partitioning: We use the refined rule $r^*$ (from Section 4.1) to partition the "Gradient Set"—a collection of training features and their corresponding negative gradients—into disjoint leaf nodes. This groups samples with similar structures. Learning: For each leaf node, we learn a specific mapping function $\phi_l$ that models the relationship between sample features and their negative gradients (Eq. 4). Inference and Calibration: For any new input $x$, we use $r^*$ to assign it to a leaf node. The corresponding mapping function $\phi_l$ then predicts the error correction vector $\Delta_x$ (Eq. 5). The final calibrated prediction is the sum of the original prediction and this correction: $F(x)+\Delta_x$ (Eq. 6).

R to W1: Thanks for your thoughtful comment. You are right that DeLTa shares some conceptual resemblance with boosting. Traditional Boosting would train a new, simple weak learner (e.g., a shallow decision tree) on the residuals, allowing a data-driven algorithm to find the best feature splits. DeLTa, instead, utilizes the LLM to analyze diverse rules and then synthesize a feature space partitioning rule, $r^*$. This rule then structures the error-correction stage. The LLM's role is not a greedy search on data but a logical synthesis of existing rules. While DeLTa undoubtedly benefits decision trees, its essence is to explore a novel use of LLMs for tabular data, which fundamentally differs from existing LLM-based tabular prediction paradigms.

Most current LLM-based approaches typically first serialize tabular data into natural language descriptions, and then tune LLMs or directly infer on these serialized data. There are two key limitations: (1) it is challenging to design such suitable template due to the modality gap between unstructured text and structured tabular data. And it may also pose privacy risks through direct textual exposure, and (2) LLM fine-tuning methods struggle with tabular data, and in-context learning scalability is bottle-necked by input length constraints (suitable for few-shot learning).

In contrast, DeLTa integrates LLM into tabular data via logical decision tree rules as intermediaries: the LLM is not used to predict sample-level outputs directly, but instead to generate a refined decision rule by understanding multiple tree rules. This rule-level interface has several distinct advantages: it (a) avoids serialize tabular data into natural language format, (b) preserves data privacy and does not require domain-specific information, and (c) can be applied in full data learning setting without LLM-finetuning. In essence, while the error-correction mechanism is boosting-inspired, the core novelty is using an LLM for logical rule refinement, establishing a new and more robust way to bridge the gap between LLMs and structured data.

R to W2: Thanks for your comment. We will cite these papers in our main manuscript.

R to W3: Thanks for pointing this out. We will carefully revised the corresponding parts of manuscript to make it more clear.

R to Q1: Thanks for your comment. As discussed in Appendix A.4, including the prompt example in Fig. 6, although we need to submit the rules to LLM, we do not need to upload additional domain-specific semantic information when designing prompt, such as explicit feature names or detailed task background knowledge. While the decision tree rules may contain feature threshold, all features fed into LLM are anonymized. Moreover, decision tree rules represent global feature space partitioning rule rather than individual samples. That is to say, the LLM operates only on abstracted, symbolic representations that reflect model logic rather than raw data. Moving beyond some LLM-based tabular learning methods that expose the serialized tabular samples containing raw feature values to LLMs, ours significantly mitigate privacy concerns.

R to Q2: Thanks for your comment. We agree that evaluating the quality of the rule rewriting process itself, independent of final task performance, is crucial for understanding why our method is effective. We assess this in a principled way from three perspectives: (i) SHAP value analysis, (ii) intra-node distance comparison and (iii) the transparency of the LLM's process.

(i) SHAP value analysis. As shown in Fig. 10 and Fig. 11 of Appendix, we provide a set of decision tree rules from Random Forest and refined rule by the LLM respectively. This is from the Credit dataset, where each sample (person) is classified as good or bad credit risks according to the set of features. To provide deeper insight into how the refined rule improves upon the original ones, we also provide the average SHAP values [1] of features under the Random Forest and our method respectively as follows. SHAP assigns each feature an importance value (higher is more important) for a particular prediction, thus it serves an unified framework for interpreting predictions.

The results show that, after LLM refinement, the SHAP importance of features such as ''credit_history'' (feature_16) and ''employment'' (feature_10) are increased, which aligns with the financial expertise. Let us take a closer look into Fig. 10 and Fig. 11 of Appendix, for ''credit_history'' (feature_16), this feature appears near the top of the second tree, indicating its potential significance. However, only 1 out of 3 decision trees in the original Random Forest rule set explicitly splits on this feature, which may dilute its influence in the ensemble of Random Forest prediction. For ''employment'' (feature_10), although it appears in 2 out of 3 rules, its splits occur late in the tree, leading to a low overall contribution in the ensemble of Random Forest prediction. In contrast, the LLM-refined rule elevates the prominence of these features.

[1] A Unified Approach to Interpreting Model Predictions. NeurIPS 2017.

(ii) Intra-node distance comparison. Although Random Forest ensembles the outputs from all tree rules, the inherent relationships and interactions among these rules are ignored. As the number of trees increases, analyzing these independent rules is gradually becoming more and more difficult, let alone utilizing these rules to partition feature space. To this end, we propose to leverage LLMs to analyze and summarize these rules into a refined rule due to the powerful logical reasoning ability of LLM. This enables the generation of improved partitioning strategies over the feature space, thereby promoting statistical coherence among samples assigned to the same leaf node. To verify it, we compute the intra-node sample distance over all leaf nodes partitioned by LLM refined rule $r^*$, and observe that the distance of $r^*$ is lower than original rule $r\in \mathcal{R}$ from Random Forest, as illustrated in Fig. 2 of original paper and Table 17 of Appendix.

(iii) Transparency of the LLM's Process: The Self-Explanation. Finally, our method is not a complete black box. The LLM provides a human-readable rationale for its actions, which allows us to inspect and understand its reasoning process. As shown in Fig. 11 of Appendix, the LLM generates a detailed "Explanation" for its refined rule. It explicitly breaks down its logic into two parts: "Choosing Features and Thresholds" and "Structure". It articulates why feature_2 was chosen (as a "critical deciding factor across multiple trees" ) and how it constructed the new tree (the "initial split on feature_2 immediately classifies a region into leaf 1" ).

In summary, we evaluate the LLM’s rule-rewriting performance through a multi-faceted approach. Together, these methods provide a robust and principled way to measure the success of this core component of our work. We will revise our manuscript to clarify it.

R to Q3: $\eta$ is set to 0.2 by default. In Eq. 6 of original paper, we need to move prediction produced by the Random Forest $F(x)$ in the direction of negative gradient by one step, and $\eta\in \mathbb{R}_+$ is the hyperparameter controlling the step size. In this context, $\eta$ plays a similar role to the learning rate in gradient descent, and it determines the strength of the correction. We found that $\eta=0.2$ works robustly.

R to Q4: Thanks for this insightful observation. You are correct that DeLTa relies on the LLM's ability to generate a useful rule $r^*$. However, we do not assume this process is deterministically perfect. Notably, DeLTa is designed with two specific mechanisms that make it highly robust to occasional, suboptimal generations from the LLM. First, we mitigate the risk of a single poor LLM output by querying the LLM 10 times by default and aggregating the results, as discussed in Appendix (Line 558). This helps to reduce variance and smooth out occasional underperforming generations. We have indeed observed rare cases (e.g., on the Adult 64-shot setting) where the LLM failed to synthesize a novel rule and instead largely replicated an input tree. Our aggregation strategy is designed to minimize the impact of such infrequent events. Second, and more importantly, the error correction (EC) stage of DeLTa acts as a powerful safety net, ensuring a strong final performance even if the initial LLM-generated rule is imperfect. The LLM refined rule $r^*$ is not the final predictor. Its primary role is to structure the problem for Gradient net, which then learns to correct the errors of the original Random Forest . Even a modestly effective $r^*$ can provide a useful partitioning of the problem space for this error correction mechanism to build upon. The effectiveness is demonstrated in ablation study in Table 4 . The results show that the DeLTa significantly outperforms the variant that uses only the LLM-refined rule for prediction (w/o ''EC''). This proves that the EC stage provides a substantial performance uplift and ensures the final model is robust. Therefore, while the LLM refinement is a key driver, DeLTa is not brittle. Through a combination of robust aggregation and powerful error correction, DeLTa is designed to consistently achieve performance improvements, even when the effectiveness of a single LLM invocation cannot be strictly guaranteed.

R to W1 & Q1: Thanks for your valuable comment. We kindly remind you that we have provided a detailed example in Appendix A.12. As shown in Fig. 10, Fig. 11 of Appendix A.12, we provide a set of decision tree rules from Random Forest and LLM refined rule respectively. This is from the Credit dataset, where each sample (person) is classified as good or bad credit risks according to the set of features. To provide deeper insight into how the refined rule improves upon the original ones, we also provide the average SHAP values [1] of features under the Random Forest and our method respectively as follows. SHAP assigns each feature an importance value (higher is more important) for a particular prediction, thus it serves an unified framework for interpreting predictions.

The results show that, after LLM refinement, the SHAP importance of features such as ''credit_history'' (feature_16) and ''employment'' (feature_10) are increased, which aligns with the financial expertise. Let us take a closer look into Fig. 10, Fig. 11 of Appendix, for ''credit_history'' (feature_16), this feature appears near the top of the second tree, indicating its potential significance. However, only 1 out of 3 decision trees in the original Random Forest rule set explicitly splits on this feature, which may dilute its influence in the ensemble prediction. For ''employment'' (feature_10), although it appears in 2 out of 3 rules, its splits occur late in the tree, leading to a low overall contribution in the ensemble prediction. In contrast, the LLM-refined rule elevates the prominence of these features.

More broadly, although Random Forest ensembles the outputs from all rules, the inherent relationships and interactions among these rules are ignored. As the number of trees increases, analyzing these independent rules is gradually becoming more and more difficult, let alone utilizing these rules to partition feature space. To this end, we propose to leverage LLMs to analyze and summarize these rules into a refined rule due to the powerful logical reasoning ability of LLM. Although the features in our method are anonymized, the LLM is still able to infer and capture latent patterns and relationships among the original Random Forest rules. This enables the generation of improved partitioning strategies over the feature space, thereby promoting statistical coherence among samples assigned to the same leaf node. This is also demonstrated by Fig. 2 of main paper and Table 17 of Appendix.

[1] A Unified Approach to Interpreting Model Predictions. NeurIPS 2017.

R to W2: Following your suggestion, we further add experiments to verify the generalizability of our DeLTa on the large-scale dataset with high dimensionality. The dataset details and experimental results are summarized as follows. The results show that DeLTa also achieves well performance on the high dimensional dataset. We will include more datasets in the revision.

R to W3's first part: Thanks for raising the important point about the dependencies of DeLTa on powerful LLMs. As we show in Appendix A.6, DeLTa with Qwen3-32B (open-source) already achieves performance comparable to that with GPT-4o. To further strengthen this point, we have conducted new experiments with an even more accessible Qwen3-8B (open-source). The results are summarized below. The results clearly show that DeLTa maintains its strong and stable performance across all three LLMs, even with the much smaller Qwen3-8B. This is because we task the LLM with a structured, logic-based synthesis of decision rules, not a complex, open-ended creative task. The core patterns in the provided rules are distinct enough to be identified by a range of capable models.

R to W3's second part & Q4: Thanks for your comment. The design of our prompt is shown in Eq. 3 of main paper and Fig. 6 of Appendix. To assess the sensitivity of DeLTa’s performance to the structure and wording of the prompt, we conduct an ablation study on its design. We consider the following variants: Variant 1 directly removes the whole part of meta information $p_{meta}$. Variant 2 softly rephrases the first sentence of the requirement part $p_{requirement}$ to: ''Based on the above information, please understand these rules and provide a refined rule.'' Variant 3 removes the second sentence of requirement $p_{requirement}$, which originally states: ''Please not just copy, please refine ...'' The results conducted on Blood dataset ($\uparrow$) show that removing the meta information or slightly rewording the requirement does not lead to significant performance degradation, suggesting that the model is robust to moderate variations in prompt phrasing. However, removing the instruction of ''Please not just copy, please refine ...'' leads to a notable performance drop. This indicates that explicitly guiding the LLM to go beyond copying and to discover interactions among Random Forest rules is crucial for the effectiveness of DeLTa.

R to Q2: Following your suggestion, we analyze the performance of DeLTa with different complexity of the Random Forest by increasing the maximum number of leaf nodes allowed in each decision tree (ML: max_leaf_nodes) and the maximum depth of each individual decision tree (MD: max_depth). The following results conducted on Blood dataset ($\uparrow$) show that moderately increasing model complexity enhances the LLM’s ability to refine the decision rules. However, further increasing the complexity yields limited benefit. Based on these observations, we conclude that DeLTa does not require an overly complex base model. A Random Forest with moderate complexity is sufficient to provide the necessary rule diversity for DeLTa to achieve a desired performance, highlighting a practical and efficient aspect of DeLTa. We will add this analysis into our revision.

R to Q3: Thanks for your comment. To verify whether LLM can use its own knowledge related to feature names to refine the rules, we further conduct ablation study. We add feature names in the format of ''feature_0: name, ...'' to the meta information of prompt (Eq. 3 of main paper, Fig. 6 in Appendix). We observed that the performance difference was marginal, when feature names are anonymized or not anonymized, indicating that the model’s ability to generate effective rules does not significantly rely on embedded domain knowledge from feature names. This supports our claim that the method operates in a domain-agnostic manner. Notably, operating on the anonymized features can significantly mitigates privacy concerns when using LLM.

R to Q5: Thank you for this insightful question. Regarding logically inferior rules, we observed a rare failure on the Adult 64-shot dataset where the LLM did not synthesize a novel rule from multiple decision tree rules, but instead replicated the structure of one of the input trees with only minor threshold changes. Our framework is designed with two specific mechanisms to be robust against these occasional suboptimal generations. First, we mitigate the risk of a single poor LLM output by querying the LLM 10 times by default and aggregating the results. As shown in our sensitivity analysis in Appendix A.11 (Fig. 7), this aggregation process demonstrably improves and stabilizes performance, smoothing out the impact of any single underperforming generation. More importantly, the error correction stage of DeLTa acts as a powerful safety net. The LLM's refined rule $r^*$ is not the final predictor; its role is to structure the problem for a Gradient Net that learns to correct the errors of the original Random Forest. The effectiveness is validated by our main ablation study in Table 4. Even in the aforementioned failure case where the initial LLM-generated rule was suboptimal on its own, the final performance of the full DeLTa framework still surpassed the original Random Forest baseline. In summary, while we acknowledge that the LLM is not a perfect synthesizer, DeLTa is not brittle. Through a combination of query aggregation and a powerful error correction mechanism, our framework is designed to be robust and consistently deliver state-of-the-art performance.

Response to Weaknesses: Thank you for the comment. We agree that the process of refining rules without direct data access must be clearly defined and validated. Conceptually, even without access to raw training data, LLMs are capable of identifying structural patterns in decision rules. This includes detecting which features appear at root nodes, recognizing feature recurrence across trees, and noting threshold values. These patterns are meaningful and often correlate with model salience. By semantically analyzing these patterns, LLMs acts as a semantic compression engine—reducing complex, low-level decision rules into a higher-level, interpretable representation. In addition, we would like to clarify our framework and demonstrate its soundness by: (1) Clearly defining the LLM's task and its objective. (2) Providing a rigorous, step-by-step analysis of the example in our paper to show the LLM's reasoning is grounded, not random. (3) Proposing a new, qualitative validation to further bolster our claims.

(1) The LLM's Task: Logical Synthesis of Model Components. First, we wish to clarify the LLM's role. The proposed approach does not ask the LLM to analyze the data. Instead, its task is one of logical synthesis: it analyzes a set of the decision rules derived from Random Forest to create a new, improved rule. In this context, a "better rule," as requested in the prompt, is defined by superior structural and statistical properties that can be inferred from the provided rules. Specifically, a better rule is one that: Creates more coherent partitions: It groups samples in a way that is more statistically homogeneous. We provide quantitative evidence of this in our paper (Fig. 2 of original paper and Table 17 of Appendix), where the LLM-refined rule consistently achieves a lower average intra-node distance.

(2) A Grounded Reasoning Process instead of random guess. To address your question whether the LLM's explanations are "random guesses", we walk through the example in Appendix A.12 to demonstrate that the LLM's reasoning is directly grounded in the structural evidence provided in Fig. 10. The LLM states, "Feature 16 was chosen due to its significance in Tree 1 with a prominent threshold of 0.82." which is not a guess. In the input provided (Fig. 10), feature_16 is the root node of Tree 1. Being the first split in a tree is the strongest possible structural indicator of a feature's importance for that particular data subset. However, only 1 out of 3 decision trees in the original Random Forest rule set explicitly splits on this feature, which may dilute its influence in the ensemble prediction. Instead, with our design, the LLM correctly identified this and reported it. For feature_10, although it appears in 2 out of 3 rules, its splits occur late in the tree, leading to a low overall contribution in the ensemble of Random Forest prediction. In contrast, the LLM-refined rule elevates the prominence of this feature. Moreover, the LLM identifies feature_2 as a "critical deciding factor across multiple trees". This observation is also grounded in the data: feature_2 is the root node of both Tree 0 and Tree 2. The LLM correctly inferred that a feature used as the primary split in two out of three "expert" opinions is fundamentally important. The LLM's refined rule (Fig. 11) is a logical synthesis of these grounded observations. This behavior reflects a structured and interpretable reasoning process. It leverages decision tree architecture—particularly the hierarchical importance implied by root-level splits—to produce a coherent explanation and synthesis. This is not randomness; it is an informed analysis.

(3) A New Proposal for Rigorous, Qualitative Validation. As shown in Fig. 10 and Fig. 11 of Appendix, we provide a set of decision tree rules from Random Forest and refined rule by the LLM respectively. This is from the Credit dataset, where each sample (person) is classified as good or bad credit risks according to the set of features. To provide deeper insight into how the refined rule improves upon the original ones, we also provide the average SHAP values [1] of features under the Random Forest and our method respectively as follows. SHAP assigns each feature an importance value (higher is more important) for a particular prediction, thus it serves an unified framework for interpreting predictions. The results show that, after LLM refinement, the SHAP importance of features such as ''credit_history'' (feature_16) and ''employment'' (feature_10) are increased. These features are well-recognized as critical indicators in credit scoring, and their elevated importance in the refined rule reflects closer alignment with established financial domain expertise.

We are grateful for your detailed feedback, which has pushed us to more clearly articulate and validate the core mechanism of our work. Based on your review, we will revise the paper to include a dedicated section that formally defines the LLM's task, includes the step-by-step logical analysis of our example, and presents the new SHAP validation. We believe these changes will provide the rigor you have requested and substantially improve our paper.

[1] A Unified Approach to Interpreting Model Predictions. NeurIPS 2017.