Unmasking Trees for Tabular Data

The training and inference procedure is not clear in writing. Can the authors show a more precise algorithm?

Thanks for the suggestion. We have added algorithms to the Appendix for further clarity.

The paper do limited ablation to show how the hyperparameters effect the results, like tree depth or duplication factor, especially for large datasets.

Given that the benchmark of Jolicoeur-Martineau et al uses default hyperparameters for other methods, we thought it would be unfair if we tuned our hyperparameters to do well on this benchmark. Thus, we chose hyperparameters that looked good on the Two Moons toy dataset (shown in Figure 1), and then just ran on the Jolicoeur-Martineau et al benchmark without further tuning. We can try to add further ablations, either for the camera-ready if accepted or for a resubmission if rejected, but were unfortunately unable to do this for the rebuttal, given our limited computing resources (a single 2015 iMac). Philosophically though, we think that a good method is one that doesn't require per-dataset hyperparameter tuning, and better yet, one that works well without even carefully tuning the default hyperparameters.

The results for each dataset in tabular data generation should be shown in the appendix. For tabular datasets the results are close to each other in value, so the comparison with mean rank of different methods can show the results more clear.

We've added results for each dataset to the Appendix. Please let us know if there's anything else you were hoping to see, and we can include it in a comment.

Unclear how the proposed hierarchical handling of regression sampling is better than a simple discretized approach with well chosen bins

We include an ablation analysis in Table 2, showing that our proposed approach performs better than vanilla quantization, either with bin boundaries chosen via KMeans ("UTrees-kMeans") as was done by TabMT (Gulati & Roysdon, 2024), or with bin boundaries chosen via KDI ("UTrees-KDI").

The idea is very simple and low-effort in terms of putting it to work

That's not a weakness -- it's a strength! To quote Ilya Sutskever from his NeurIPS 2024 Oral (https://www.youtube.com/watch?v=-uyXE7dY5H0) on "Sequence to Sequence Learning with Neural Networks" that just won a test-of-time award, "We use minimum innovation for maximum results." Frankly, the preference of ML conference reviewers for complicatedness and high-effort set back our field by a decade. This preference, driven not by practicality or promise, but by the desire to appear smart to each other with fancy equations, explains why autoregressive-based methods have been underexplored for tabular data. We make no apologies for our taste for simple, low-effort, elegant minimalism.

The results are not great. The tables feel a little to designed: boldening non-standard things in the table, i.e. the winner of a subset of the methods or the 2 best methods.

This is incorrect. We highlight the best of ALL methods (not a subset of the methods), to show which method is best. And we boldify the best of the XGBoost-based methods (ForestVP vs ForestFlow vs ours) as a natural ablation, to control for the type of model, and focus on diffusion vs flow-matching vs autoregression.

I think if one wants to make a scientific contribution at ICLR level it would be the right thing to train a model that generates data for filling in in one forward pass, instead of relying on traditional discriminative models only. Maybe this is worse, but then it would be an important ablation.

I'm not sure how to interpret this.

If you are referring to our autoregressive joint modeling approach: by this ideology, research on LLMs (which are autoregressive and which involve training a token discriminator) does not belong in ICLR. Really?

If you are referring to our BaltoBot hierarchical partitioning conditional modeling approach: as we stated above, we already include an ablation experiment. Are you suggesting that this idea should be a priori rejected by ICLR for ideological reasons, regardless of its surprising empirical effectiveness?

With all due respect, avoiding this type of ideological, closed-minded reviewing is why ICLR was founded in the first place. We hope the reviewer will reconsider their perspective, but if not, we hope this review will be disregarded by the Area Chair and Program Committee.

The paper fails to properly analyze how the method performs under different missingness mechanisms (MAR, MCAR, MNAR).

Our experiments are all on MCAR missingness. We note that the GAIN (Yoon et al, ICML 2018), ForestDiffusion (Jolicoeur-Martineau et al, AISTATS 2024), and TabMT (Gulati & Roysdon, NeurIPS 2024) papers only evaluated on MCAR. So we vigorously disagree with the reviewer's demanding of this as a condition for acceptance.

Although the method is a novel combination of existing techniques, it also lacks substantial theoretical innovation or justification.

This is blatantly false. Hierarchical partitioning (BaltoBot) is a novel technique for conditional generative modeling. We gently request the reviewer to reread Section 2.2.

The method is presented as a general solution, but its limitations and assumptions are not adequately discussed. The paper also lacks a clear discussion of how the method handles discrete variables and different data types.

Our paper has a full yet short Limitations section (Section 4). This section is short, because our method has few limitations. If the reviewer thinks otherwise, they are welcome to say precisely what they think those are!

With regards to discrete variables, our method has no limitations, though we inadvertantly failed to mention this in the original manuscript. We have clarified this in Section 2.1: "We model categorical features via softmax-based classification with cross-entropy loss; our approach for continuous features is described in Section 2.2." This is also clarified in Algorithm 1 in Appendix A.

The description needs more rigorous mathematical formulation and clear algorithmic steps.

We have added an Equation to Section 2.1 to formalize UnmaskingTrees, and have added algorithms to Appendix A.

Writing issues

We have addressed these. Note that we are now using the variables of H (for height of the meta-tree) and h (for height of a node in the meta-tree).

Datasets assumptions

We follow Jolicoeur-Martineau et al.'s exact setup and even their experimental scripts. Anyone who wants these details can read their paper.

The abstract fails to clearly communicate the paper's contributions and methodology.

Without any concrete details about what is allegedly lacking or missing in the abstract, this criticism is too vague to respond to. Can you clarify what you mean?

How does the choice of masking order affect the results? Is there any theoretical justification for random ordering?

There is typically no inherent ordering to tabular features. Random ordering shows as many possible orders to the model as possible during training. This is especially useful for imputation, where any subset of the features may need to be conditioned on at inference time. The theoretical justification for this is formalized and discussed extensively in the case of tabular data in TabMT (Gulati & Roysdon, 2024) and DP-TBART (Bastellon et al, 2023) (we added a citation to the latter in our revised manuscript), and, more generally, for joint modeling in (Kitouni et al., 2024).

What are the computational complexity implications for high-dimensional data?

This was already fully discussed in Section 2.3 (Computational Complexity). Please read and let us know if you have remaining questions.

We did add a discussion of empirical runtime to Appendix C, noting that our method is actually comparatively fast, especially on the biggest datasets in the benchmark:

"Timing results are in Table 9, and depicted in Figure 5. Our method is relatively efficient at both imputation and generation. The datasets on which we are slowest for imputation are Libras (1976 seconds) and Bean (1929 seconds), on our ancient 2015 iMac with 16Gb RAM. On Libras, ForestVP imputation took 12439 seconds (without RePaint) and 14715 seconds (with RePaint); on Bean, ForestVP took 898 seconds (without RePaint) and 1318 seconds (with RePaint), on their cluster of 10-20 CPUs with 64-256Gb of RAM. The datasets on which we are slowest for generation are also Libras (2987 seconds) and Bean (4346 seconds). On Libras, ForestFlow generation took 9481 seconds and ForestVP took 9042 seconds; on Bean, ForestFlow took 869 seconds and ForestVP took 947 seconds, once again on their much more powerful computing cluster."

What are the method's limitations when dealing with discrete variables?

As stated above, there are no limitations.

Was the experiment code (for the 27 datasets) available

We have added them to the repo; these are from the ForestDiffusion public repo, with additional function calls for our approach.

there were no unit tests

Our deanonymized repo has a variety of things including unit tests that were stripped from our anonymized repo. Frankly, we find this complaint rather nitpicky and unnecessary.

add ForestDiffusion to the requirements.txt

Done.

"The biggest downside of the proposed method would seem to be the need for duplicating the training set K*D times..."

True, while [1]'s benchmark includes Bean (N=13611, D=16) and Libras (N=360, D=90), it is mainly focused on small datasets. Note that we had no difficulty applying UnmaskingTrees to the benchmark on our ancient 2015 iMac with 16GB RAM. We have clarified in the updated manuscript that our approach is expecially geared towards smaller-sized datasets. Nevertheless, our method is sometimes faster than its diffusion-based (ForestDiffusion) and flow-based (ForestFlow) counterparts. See Appendix C in our revised Supplemental:

"Full imputation results are in Table 7. Full generation results are in Table 8. Timing results are in Table 9, and depicted in Figure 5. Our method is relatively efficient at both imputation and generation. The datasets on which we are slowest for imputation are Libras (1976 seconds) and Bean (1929 seconds), on our ancient 2015 iMac with 16Gb RAM. On Libras, ForestVP imputation took 12439 seconds (without RePaint) and 14715 seconds (with RePaint); on Bean, ForestVP took 898 seconds (without RePaint) and 1318 seconds (with RePaint), on their cluster of 10-20 CPUs with 64-256Gb of RAM. The datasets on which we are slowest for generation are also Libras (2987 seconds) and Bean (4346 seconds). On Libras, ForestFlow generation took 9481 seconds and ForestVP took 9042 seconds; on Bean, ForestFlow took 869 seconds and ForestVP took 947 seconds, once again on their much more powerful computing cluster."

The benchmark proposed by [1] also does not carry out any hyperparameter tuning as far as I understand. ... However, I do sympathize with the authors that there are too many degrees of freedom to such an analysis and it is not fun.

Correct, [1]'s benchmark doesn't tune hyperparameters. But we believe their choice was appropriate for such small datasets, where as you mention, one wants to avoid too many degrees of freedom. And we note that hyperparameter tuning is even more not-fun for practitioners. They may lack the data, time, and skills to do it, so a method like ours that doesn't require hyperparameter tuning, but works well by default, is genuinely valuable to them.

Please consider providing the full table of results and standard errors for every metric in an appendix.

We have added the raw data to the Appendix C and to the code repo.

The explanation of BaltoBot could be improved...

Thanks for this feedback! We have updated the manuscript accordingly: "Thus, the input space to every node is partitioned into two with the splitting point determined by KDI."

Fix missing K in line 092

Thanks; we have fixed this.

The BaltoBot evaluation could be further developed. As a non-parametric conditional estimation method for numerical variables it could have wider applicability as it seems competitive to other proposed methods for uncertainty estimation. However, evaluation on more than one real-world scenario would be required to draw meaningful conclusions.

Indeed, it would probably be possible to write an entire paper around the BaltoBot idea. However, such experiments are probably out of scope for this conference paper. Here, we think it suffices to show that we have superior results (including the Table 2 ablation comparing it with vanilla KMeans quantization and vanilla KDI quantization) using it as a subroutine for the task our paper addresses.

"Would similar strategies to ForestDiffusion using a data iterator to avoid materializing...?"

This is a fantastic idea. This would be more efficiently implemented by marginalizing over random permutations of autoregressive masking orders, which yields uniform-rate masked language modeling. These two are equivalent -- see TabMT (Gulati & Roysdon, 2024) and (Kitouni et al., 2024) -- but the latter is more amenable to using with the XGBoost data iterator as done by Jolicoeur-Martineau et al.

The density estimation aspect could be explored further.

Thanks for the reference. We hope to investigate this further, either for camera-ready if accepted, or in a future submission if rejected.

Since errors in auto-regressive generation could accumulate faster given certain feature permutations

Indeed. We use a different random permutation for each generated or imputed sample, as a way to contribute to sampling diversity. But each sample could also incorporate multiple different permutations to improve each sample's quality. However, how one would use the model's own outputs to evaluate generations is still an open research question (eg, for LLMs, the varentropy-based Entropix sampler (Shrek & Frog, 2024) is one such exploratory idea).