Can a Single Tree Outperform an Entire Forest?

1. Novelty: I believe the paper combines together several ideas explored in soft DT literature:

• Gradients based learning via sigmoid approximation is well-known approach to train soft trees;
• Based on my understanding, iterative scaled sigmoid approximation is similar to annealing mechanism, which has been previously explored, e.g. in [1] (although this is not the earliest work). Note that [1] also discuss alternative function to sigmoid.
• Subtree polishing reminds weaker version of Tree Alternating Optimization [2,3]. It's good to know that combing these all works quite well, but authors are encouraged to expand on this context and specify differences, if any.

2. Experiments are conducted on small-to-medium datasets where CART is already performing quite well and using RFs are not bringing significant benefits. Authors are encouraged to use more practical high-dimensional (and possibly large-scale) datasets. Moreover, I'd suggest adding [2,3] as additional non-greedy baseline. Note that [3] claim SOTA performance on some datasets used in this paper.

[1] Hussein Hazimeh, Natalia Ponomareva, Petros Mol, Zhenyu Tan, and Rahul Mazumder. ICML (2020). The Tree Ensemble Layer: Differentiability meets Conditional Computation.

[2] M. Carreira-Perpinan and P. Tavallali, 2018. Alternating optimization of decision trees, with application to learning sparse oblique trees.

[3] A. Zharmagambetov and M. Carreira-Perpinan, 2020. "Smaller, More Accurate Regression Forests Using Tree Alternating Optimization".

1. The use of simulated annealing for training soft decision trees lacks novelty (e.g., [1], [2])
2. The "accuracy" metric reported throughout the paper is not formally defined
3. The training time appears exponential in the number of parameters, due to the soft branches routing the entire dataset across all decision nodes ($2^{D + 1} - 1$). This is further exacerbated by the Polish Strategy, which applies the algorithm to all subtrees.
4. It is unclear how a complete binary oblique decision tree can be considered interpretable without incorporating regularization terms for sparsity in decision node weights.

[1] Thomas M. Hehn, Julian F. P. Kooij, and Fred A. Hamprecht. 2020. End-to-End Learning of Decision Trees and Forests. Int. J. Computer Vision 128 (April 2020), 997–1011.

[2] Ajaykrishna Karthikeyan, Naman Jain, Nagarajan Natarajan, and Prateek Jain. 2023. Learning Accurate Decision Trees with Bandit Feedback via Quantized Gradient Descent. Trans. Machine Learning Research (Sept. 2023).

1. The title of the paper is ambiguous. The comparison between trees and random forests requires conditional constraints. Random forest is essentially an ensemble learning framework, in which decision trees are the base learners. Does your title mean that ensemble learning frameworks cannot work on the tree model you proposed? If you are comparing the proposed tree model with the original version of RF, the significance of this comparison is not significant.
2. In my opinion, this article should focus on the comparison with different oblique decision tree algorithms, especially adding the latest pruning techniques, as this method includes a pruning mechanism.
3. The method presented in this article lacks a theoretical guarantee. I believe that in a structured model such as a tree model, theoretical explanations would be more convincing than experimental results after parameter tuning.
4. Tree models still have different structures at the same depth, and Tree depth is not very convincing. It is recommended that the number of nodes in the tree model be displayed.