Fast Estimation of Partial Dependence Functions using Trees

Thank you for your comments.

In our response to the first reviewer we have conducted further experiments of our method on real data (see 1. Real-world application in our response to R1). In our response to the second reviewer, we advice using shallow trees to reduce complexity for large-scale datasets with many features (see 2. High-complexity models in our response to R2). We believe that our method is still very applicable even in high-dimensions, provided that the model is not overly complex. We have not encountered numerical instability in our experiments with real data; since with many samples, the estimation of partial dependence (PD) functions becomes more reliable, not less. Moreover, practitioners are typically interested in marginal effects and low-order interactions, because they are easier to interpret. In all cases, the actual quantity being computed is a just an empirical average, for which FastPD can provide a speed-up when the model is a tree-based model.

We agree that not all functional components could or should be acquired, since for higher-order interactions the effects of these components are usually very close to zero, we will make this point more clear in the paper. This further motivates the use of shallow trees, as they limit the fitting of higher-order interactions and preserve interpretability. As a remark, in the package implementation of our method, we provide the user the option to specify the order of the effect they wish to acquire.

For the reference that we had inadvertently missed, perhaps it was the following you meant?

Fumagalli, F., Muschalik, M., Hüllermeier, E., Hammer, B., & Herbinger, J. (2024). Unifying Feature-Based Explanations with Functional ANOVA and Cooperative Game Theory. arXiv preprint arXiv:2412.17152.

We have already cited Hiabu et al. multiple times, but were not yet aware of Fumagalli et al. and will ensure it is cited in the revised version.

Finally, we agree that the paper title could be clearer. A possible revision would be:
"Fast Estimation of Partial Dependence Functions using Trees",
which more accurately reflects the tree-based nature of our approach.

Thank you for your comments, we address them below.

1. Real-world applications:
   We have addressed this in our reply to the first reviewer. Please see 1. Real-world application in our response to R1.

2. High-complexity models:
   The computational complexity of both our algorithm and the baselines scales exponentially with depth, which motivates the use of smaller depth values. Additionally, increasing depth captures higher-order interactions that are inherently less interpretable. Gradient Boosting methods such as XGBoost fit shallow trees well; we therefore recommend using a depth of 5 or less in practice. We will clarify this point in the paper.

3. Referencing SHAP scalability:
   We will add some references for non-tree-based SHAP algorithms in our introduction.

4. Distinguishing model PD and ground-truth PD:
   We will add the following to the application section:
   "We see that FastPD is well-suited for estimating the model PD, though it may deviate from the underlying ground truth PD. In practical settings where accurately capturing the relationship between the predictors and response is crucial, we recommend using FastPD as an initial visualization tool, which is to be complemented by other statistical methods for estimating continuous treatment effects, see e.g. Kennedy, E. H., Ma, Z., McHugh, M. D., & Small, D. S. (2017). Non-parametric methods for doubly robust estimation of continuous treatment effects. Journal of the Royal Statistical Society Series B: Statistical Methodology, 79(4), 1229-1245. and Chernozhukov, V., Chetverikov, D., Demirer, M., Duflo, E., Hansen, C., Newey, W., & Robins, J. (2018). Double/debiased machine learning for treatment and structural parameters. The Econometrics Journal, 21(1), C1–C68.."

5. Choice of background sample size:
   We now note in the main text that in practice, one should use all available data as background. If this is not feasible, a smaller background sample size can be chosen (e.g., $n_b = 100$) and compared to a slightly larger one (e.g., $n_b = 150$). If the resulting PD functions are similar, then the smaller sample size is likely sufficient.

   As a remark, in our experiments we have observed that augmenting the tree is typically fast, so the computational cost of using more background samples is low. The majority of computation time is instead spent on evaluating the explanation points.

Thank you for your comments, we address them below.

1. Real-world application:

   • Benchmark:
     For an additional comparison between FastPD, FastPD-100, FastPD-50, and the path-dependent method, we have now added a benchmark considering 33 regression and 29 classification datasets from the OpenML-CTR23 Task Collection and the OpenML-CC18 Task Collection. For a set $S \subseteq \{1,\dots,d\}$, we compute the importance measure of the estimated functional component $\hat{m}_S$ by $\hat{E}[|m_S(x)|]$, where $\hat{E}[\cdot]$ denotes the empirical mean.
     For each dataset, we record the importance of the components $\hat{m}_S$ that are ranked in the top 5 by any of the four methods (FastPD, FastPD-100, FastPD-50, path-dependent).
     The results are available here:
     https://www.dropbox.com/scl/fi/ujdikzu8iyrn53ha03qc4/summary.pdf?rlkey=ypnkbdhjwgylr38ylj16ugjzj&st=f97bol4d&dl=0
     We observe that while results are consistent across methods for some datasets, substantial differences can emerge in others. In several cases the differences are >100%, i.e., the feature importance values from the path-dependent method are more than twice as high as those from FastPD.

   • Concrete example:
     We highlight one dataset (adult) where FastPD and the path-dependent method gave qualitatively different insights. The following plot based on a single model obtained from one run of hyperparameter search. We have plotted the interaction effect between age and relationship status:
     $m_{\text{age,relationship}}$.
     https://www.dropbox.com/scl/fi/8wx0xj34rglhst6tm8eb0/adult.pdf?rlkey=nrgz74xjrd1udoxgrhohwx6au&st=yclm7q6o&dl=0 The path-dependent method indicates no significant interaction effect during working age (35–60), suggesting that age affects husbands and wives similarly in this range. Outside this interval, it estimates a more positive effect for wives than for husbands.
     Using FastPD, we get a different picture: Between ages 30 and 65, the effect is more positive for husbands than for wives, with the advantage reversing outside this age range.

2. Flowcharts/diagrams:
   We agree that a flowchart and illustration would help in explaining the algorithm, and will add it for the camera-ready version.

3. Reproducibility details:
   We will modify the beginning of the experiments section to include an overview of our hyperparameter settings.

4. Package implementation:
   A package implementation exists, including automated plot functions. However, due to anonymity, we prefer not to share it at this point, but it will be made available in the camera-ready version.