Rebuttal Reviewer knMV

We thank knMV for their review. We are encouraged that the reviewer found the paper well-written and easy to follow and that the author found our approach very flexible and valuable for tabular regression problems with UQ. Below, we provide additional results and discussion that answer the questions and concerns that the reviewer presented. The new experiments definitely strengthened the paper!

Weaknesses

W1. Treeffuser must solve a stochastic differential equation to generate samples, which can become expensive W2. No complexity or empirical runtime analysis provided

Thank you for the great comments. In the paper, we provided some runtime analysis of Treeffuser lines 257-261. Yet, we agree that more runtime analyses are helpful for readers and practitioners. So we generated three new figures running Treeffuser with default parameters on a 2020 MacBook Pro.

1. Figure 1: runtime in seconds for training Treeffuser on subsets of different sizes of the M5 dataset (section 5.3). The shape of the training data is indicated on the plot. Error bars represent the standard error over five runs of Treeffuser.
2. Figure 2: runtime in seconds for training Treeffuser on the other datasets used in the paper. Error bars represent the standard error of over five runs of Treeffuser.
3. Figure 3: average runtime in seconds for producing one sample after training Treeffuser. The average is computed by sampling five thousand points from Treeffuser using its default settings of 50 discretization steps.

From these results, we find that:

• Treeffuser is very fast at fitting moderately sized datasets, with runtime increasing linearly with dataset size;
• sampling a single point is very fast ($\approx 10^{-4}$ seconds) and can be parallelized;
• yet, drawing many samples can take more time, e.g., with a large test set.

We also agree with the suggestion of including a complexity analysis and will add a detailed discussion in the appendix. To be brief here, the complexity for training is the same as for LightGBM with the subtelty that the dataset size is multiplied by n_repeats (see Alg 1 and Appendix C), and a different model is trained for each dimension of $y$. The complexity for sampling is the same as evaluating the trees n_sampling_steps * y_dim times.

W.3 The comparison to other methods (Section 5.2) could be improved. For example, iBUG's hyperparameter [...] which may improve its performance on datasets in which the output is not Gaussian.

Thank you for your suggestion. Following your advice, we

1. increased the search space of $k$ using the method for optimal tuning of $k$ (Algorithm 3 in iBUG's paper)
2. Implemented iBUG with KDE likelihood -- by tuning the kde bandwidth in [0.0001, 1000].

The results are available here.

The conclusions are unchanged in both cases: Treeffuser outperforms iBUG (with and without KDE). Thank you for suggesting these great baselines; we have updated section 5.2 of our paper with them. We hope that this addresses your concerns.

Questions

Q1. Do the authors have negative log likelihood (NLL) results?

We thank the reviewer for this question. We do not provide NLL results. While NLL can be approximated using KDEs on the samples outputted by Treeffuser, we found that KDEs are sensitive to the choice of bandwidth. We also note that exact likelihood computations are possible for diffusion models when the score estimator is continuous everywhere, as with neural-based diffusions (see section D.2 [8]). However, this is not applicable for GBT, because trees are not continuous.

Fortunately, this apparent limitation has no practical implications:

• For evaluation, we use CRPS, which can be evaluated from samples. CRPS is a proper scoring rule [16] with advantages over NLL, such as better handling of tail probabilities [17]. CRPS is commonly used for probabilistic predictions [16,18].
• For concrete downstream applications, we argue that being able to generate samples from $p(y\mid x)$ is more important than raw values of $\log p(y\mid x)$.

Q2. I'm surprised Treeffuser performs better than iBUG for point predictions [...] Do the authors have a hypothesis as to why Treeffuser performs better than XGBoost in terms of point performance?

Thank you for your question. We also anticipated that iBug would excel in point prediction since it relies on XGBoost. However, we are not surprised by the results.

1. First, we nuance that Treeffuser only slightly outperforms iBug for point prediction, with iBug often very close behind.
2. Second, we point out that iBug's point predictions do not mathematically coincide with XGBoost's. Indeed, the point predictions of a probabilistic method are computed as the expected mean: the empirical average of 50 samples from the modeled conditional probability distribution $p(y\mid x)$. So if the conditional probability returned by iBUG has errors, then the point prediction might also have errors.

In light of your comment, we conducted additional experiments for point predictions against XGBoost and LightGBM directly (with hyperparameter tuning optimization).

The results are shown in this table, where we also updated iBUG's point predictions with your suggested hyper-parameter tuning improvements which yielded similar but slightly improved results.

As expected, vanilla XGBoost and LightGBM outperform or tie with all the probabilistic methods. In particular, they are often comparable with Treeffuser, suggesting that Treeffuser provides probabilistic prediction without sacrificing point predictions of the mean.

We updated Table 4 with these results and added further discussion in the paper for clarity. Thank you for raising this point and suggesting the addition of more baselines; they strengthened the paper!

I thank the authors for their thorough response, and appreciate the additional experimental results. Overall, my concerns have been largely addressed and I have updated my score accordingly.

Rebuttal Reviewer 3Apn

We thank Reviewer 3Apn for their valuable feedback. We are pleased to hear that they appreciate our application to the newsvendor problem. Based on their comments and questions, we clarify our contributions, the applicability of our method, and its implementation below.

Weaknesses

W1. The work is not clear about its contribution and what parts of the diffusion-based setup are new vs borrowed from existing works.

Please see our answer to Q1.

W2. There could be more and better empirical comparisons.

Thank you for the comment. We selected five strong approaches that practitioners might choose for probablisitc predictions. However, we look forward to comparing the performance of Treeffuser (our method) against other methods.

What are some specific datasets or methods you think would be valuable for comparison in our paper?

W3. Notation and clarity could be improved

Thank you for the feedback.

• The vector a is a dummy vector to explain notations, it is normal to not find it in the equation. We rephrased the sentence to avoid confusion.
• The derivation of Eq. 12 was shown in Appendix Eqs. 19 to 22. We updated the paper to make sure this is also clear in the main text.

W4. The proposed method may not be easily used or widely applicable because of the complex process needed to generate samples involving solving differential equations.

Thank you for raising this point, usability is very important to us. But we are surprised by this comment, because Treeffuser is actually straightforward to use and widely applicable. This is a key contribution of our work. We argue this is the case for the following reasons:

• Concretely, we designed a simplified user experience by wrapping Treeffuser in a plug-and-play Python package that solves the differential equations for the user. With the accompanying package in the supplementary material, using Treeffuser is as simple as:

  model = Treeffuser()
  model.fit(X,y)
  samples = model.sample()
  

• Offering such a simple user interface is only possible because we built Treeffuser with a flexible estimator (trees) and flexible model (diffusions). As a result, this simple training procedure is the same regardless of the type of data (e.g. heavy tailed, multi-modal, heteroskedastic, categorical, missing).

• In practice, Treeffuser is accurate on diverse datasets, even without tuning (see our studies in Table 1, 2, and 4).

To reiterate, while solving differential equations is indeed complex, it is not more complex than fitting boosted trees or optimizing thousands of neural network parameters. Efficient packages such as xgboost, sklearn, and torch made these tasks accessible to any user. With our treeffuser companion package, we aim to make diffusion-based probabilistic predictions for tabular data equally accessible.

Questions

Q1.1 What part of the contribution is due to the specific way the diffusion process was modeled and what part is due to the use of trees?

Thank you for the question. We think that both parts cannot really be separated. The main contribution and goal is to use trees, but we had to adapt the diffusion model process to do so. (For example, trees only have a 1d output.)

Yet, you are right that the way we modeled the diffusion is important. For example, the score reparametrization on line 133 (and Eq. 13) is crucial for performance.

To highlight the importance of our design choices, we added this discussion to the main text, and included an ablation study on the score reparametrization in the Appendix.

Q1.2 How does the method differ from other diffusion-based approaches to probabilistic prediction such as the cited CARD paper?

The differences between CARD[9] and Treeffuser are:

1. CARD uses a specialized neural network architecture, while we use out-of-the-box gradient-boosted trees (e.g. XGBoost, LightGBM).
2. CARD uses a discrete time approach [13-14], while we formulate our conditional diffusion problem with continuous SDEs [8], which we believe renders the theory easier to follow.
3. Our approach requires much fewer function evaluations (i.e. 50 vs 1000) to generate a sample.
4. CARD requires two training steps: a first model is trained to predict the mean of the target only, then a second model learns the distribution of the data using a diffusion that takes the prediction from the first model as input. Treeffuser directly trains the diffusion model on the target. It doesn't require training any auxiliary model for conditioning and is thus simpler.
5. We attempted to use CARD in our experiments for comparison but were unable to obtain good results.

Q2. Tagasovska and Lopez-Paz's Single-Model Uncertainties for Deep Learning (NeurIPS 2019) is a relevant work to compare to here. [...] How would the method perform with neural networks?

Thank you for sharing this reference. We were not aware of it but now include it in the text. Indeed you are correct, the baseline quantile regression used in our paper implements the same idea using trees instead of neural networks.

In fact, we originally tried a similar version of quantile regression with neural networks, and we found that on all our experiments, the tree based quantile regression performed better. This is in line with the observed dominance of GBTs over neural networks on tabular data [10-12].

Q3 How easy is the SDE solver to stitch with the outputs of the trees?

Integrating the SDE solver with the outputs of the GBT is easy, akin to neural network based diffusions. Concretely, the trees are wrapped into a class with a predict function that returns the learned score, which is then fed to the SDE solver.

We ensured our code is modular to support any SDE solver, while hiding all the SDE complexity from the users. For example, model.sample(n_samples) calls the standard Euler-Maruyama SDE solver that handles the numerical integration automatically.

Rebuttal reviewer iWzb

We thank reviewer iWzb for their comments. We appreciate that the reviewer found our paper to be well-written and clearly presented. We appreciate the great questions and suggestions by the reviewer. We provide answers and discussions below.

Weaknesses

W1. There should be more related methods to compare with in the experiment in section 5.3, especially more recent ones.

We thank the reviewer for the suggestion. We designed Section 5.3 to be a concrete example showing how simple it is to use Treeffuser for real-world applications. This section was not intended to be an extensive comparison like Section 5.2.

We selected the methods in Section 5.3 because they performed the best in Table 2, including the recent iBUG [1]. They also represent two types of methods with interesting properties for comparison with Treeffuser: a “hand-crafted” log-likelihood specially suited for the problem (NGBoost Poisson), and a “likelihood-free” tree-based model (amortized quantile regression). We omitted other methods in Figures 2 and 3 to enhance the figures’ legibility.

Questions

Q1. What are the potential applications of the proposed method, especialy given that (1) it's designed for tabular data, and (2) it is generative but cannot provide density evaluation?

Thank you for your question:

1. Methods for predictions from tabular data are one of the most frequent uses of machine learning in practical industrial applications (e.g. sklearn [2], or methods such as xgboost [3]). Uncertainty estimation in the form of modeling $p(y \mid x)$ is equally important in those applications and for risk-aware decision making [4-7].

2. We do not focus on providing density evaluation because it is unnecessary for our purposes. In practice, we aim to model $p(y \mid x)$ to make real-world decisions based on quantities such as $\mathbb{E}[f(y, x) \mid x]$. These quantities are purely evaluated from samples of $y \mid x$. For example, a user might be interested in the expected output of their factory given specific settings, the standard deviation, the expected profit, or the probability that profit falls below a certain threshold $c$, where their profit function $f(y, x)$ is a black-box function. With Treeffuser, we can model and compute quantities by using the samples to compute $\mathbb{E}[y \mid x]$, $\mathbb{E}[y^2 \mid x] - \mathbb{E}[y \mid x]^2$, $\mathbb{E}[f(y, x) \mid x]$, and $\mathbb{E}[\mathbb{1}_{f(y, x) \leq c} \mid x]$, respectively. In contrast, with density estimation $p(y \mid x)$, evaluating $\mathbb{E}[f(y, x) \mid x]$ or even $\mathbb{E}[y \mid x]$ would likely require advanced techniques to sample from $p(y \mid x)$ and use Monte Carlo estimators to compute these quantities. Treeffuser performs these computations directly.

To re-emphasize, while having an exact density evaluation could be useful in some scenarios, we believe having sampling is as important if not more important. In fact, the ability to sample from a density is an active research area (inverse transform sampling, rejection sampling), and Treeffuser enables sampling directly without modeling the density.

We have now expanded the paper’s introduction with the discussion above to emphasize that the focus of the paper is sampling from $p(y \mid x)$ instead of approximating the value $p(y\mid x)$. Thank you for helping us clarify any confusion about the goal and impact of Treeffuser.

Limitations

L1. My main concern about this proposed method is that it cannot provide exact density evaluation. This is also discussed in the limitations section by the authors.

As detailed above, we did not aim to do exact density evaluation because the downstream applications we are focusing on do not need it. We focused on designing a method that provides high-quality samples, which is more important in practice. We believe that focusing on sampling is a strength, not a weakness.