MotherNet: Fast Training and Inference via Hyper-Network Transformers

This is a very fair and insightful review, demonstrating the reviewer’s profound understanding and expertise in this field.

Thank you for your comments. We want to emphasize that we do not claim "MotherNet is better than TabPFN and tree-based methods", so this is not an argument that we would like to defend. MotherNet overcomes one of the limitations of TabPFN, while still providing competitive performance. We would like to claim that we show that on small datasets, MotherNet is highly effective without any dataset specific tuning, and that in-context learning of MLPs using Mothernet outperforms gradient descent training of similar models even if hyper-parameter tuning is performed.  

weaknesses

1. How do you suggest we include pre-training time into the methods cost? Generally, per-dataset training cost is considered. The baseline methods benefit from years of algorithm tuning by a large community of practitioners; that is usually also not included in the per-dataset cost, as it is not relevant to applying the model to a new task. We used two established benchmarks for evaluating our algorithm. Since you conclude this is not an effective experiment, what alternative would you suggest? It would be easy to create a benchmark that favors our method, however, we opt to provide a broad comparison based on established benchmarks; that the results are not statistically significant points to the hardness of rigorous testing and the lack of appropriate benchmarks – unless you can suggest a different benchmark, that you judge would be more effective. Our conclusion is that given the existing benchmarks, our method is statistically equivalent to the top performing methods, while providing extremely fast training and prediction speeds, using a novel approach that we think is noteworthy as it outperforms well-established gradient descent training.

2. If you could be more specific in your criticism, we are happy to address your concers about the presentation. We are happy to address the errors in the references. All the main results are included in the paper, and the figures in the appendix are additional results. We are happy to move some of the figure from the appendix to the main paper if you prefer; this seems a simple matter. To the best of our knowledge, the critical difference diagram is the main way the ML community compares algorithms across datasets. We are happy to provide a review of the methodology in the final version.

3. This work indeed is an extension of TabPFN, however, in a way that we think is interesting, that overcomes a limitation of TabPFN which TabPFN has been criticized for (long prediction time) and in a way that was consider impossible to achieve by the TabPFN authors (as per personal communication).

To justify my original short review, I believe the work suffers from a fundamental weakness and a lesser drawback: performance results and novelty. It is argued that MotherNet is "competitive," but the performance numbers contradict this claim. On CC-18, the normalized AUC is worse than TabPFN, and on TabZilla if one disregards the wide error bars, MotherNet comes in seventh for average rank. Reviewer DaxP and reviewer QK8H share a similar opinion. On the other hand, the fact that the method is an incremental one detracts from the contribution of the paper. The contribution of an interesting result (reviewer QK8H) is below the qualification of ICLR as a top conference. Note that it is the author's responsibility to produce convincing results on an accepted benchmark. The issue of presentation is a separate discussion. It is already pointed out that substantial arguments in Section 4 relies on results and figures in the appendix, so it is important to add necessary results to the main text. Unfortunately, there has not been a revision at this time; therefore, I maintain my evaluation.

As mentioned above, the goal of this work is not to improve the accuracy of TabPFN, but to improve the inference speed. We consider outperforming a tuned gradient boosting model a competitive approach; we do not claim setting a new state of the art of accuracy.

We are happy to produce results on an accepted benchmark; the question was which benchmark you consider accepted. Our results on both CC-18 and TabZilla are quite strong, in particular given that we perform pure in-context learning, using orders of magnitude less compute than competing approaches, and no hyper-parameter tuning.

We added some of the results to the main text, please let us know if that addresses your concern. We consider the validation set results supplemental but we moved them to the main text to address your feedback.

It seems you are concerned that the novelty of the architecture and the fact that MotherNet is ranks below TabPFN in terms of accuracy are your main concern. What evidence or experiment would convince you that the improved prediction time is a trade-off that can be beneficial?

We would like to thank reviewer DaxP for their comments and suggestions. We want to point out that the submission has an anonymized github link that contains the full code for training and prediction, as well as code to download pre-trained weights. We will address the other feedback below.

Benchmarks

There are indeed three results, two main experiments and additional results on the tabpfn validation set. To clarify the validation set, this methodology is common in AutoML work, where there is a chance to overfit model selection or hyper-parameters to specific datasets. All the development of MotherNet (and TabPFN), i.e. architecture, learning rates, prior, etc, have been done on the validation datasets, and only a final evaluation has been done on the test dataset. This is often not done in ML outside the AutoML community and can lead to overly optimistic benchmark results. Therefore the validation set is relegated to the appendix; however, it’s likely that most competing methods where also developed including the datasets in our test set.

We chose to evaluate on two benchmarks, exactly because for any method, the relative performance will always change depending on the benchmarks. We provide two benchmarks to provide a broader picture, instead of providing a strong message on a cherry-picked selection. We used two widely available benchmark datasets; as mentioned above, if you have another benchmark dataset that you would find more convincing, we can try to include it; however, doing so is extremely expensive for all competing methods (but not for mothernet). We could also consider the AutoML benchmark or the Grinsztajn datasets, whoever both focus more on larger datasets and are therefore less relevant to this work.

The results from Table 2 and Figure 3 and Table 6 that are not for MotherNet are taken directly from the TabZilla benchmark which has been published in the NeurIPS benchmark track. We obtained the raw numbers from the authors of that work. If these numbers seem, I don't think that can be considered a shortcoming of our work.

Hyperparameters

We tried to adhere to the methodology from the TabPFN paper for CC-18 to be more directly comparable; "Revisiting Deep Learning Models for Tabular Data" for example uses a much broader search space, which is what was used for the TabZilla benchmark (though potentially with less budget, as described in the TabZilla paper). We are using randomized search as in the TabPFN paper, so ordering is indeed ignored.

Related work

We are happy to discuss classical algorithms like tree-based ones in the related work section; we compare against tree-based algorithms and deep learning baselines like FT-Transformer in Table 2, we are happy to also mention them in the related work section, though a thorough review of all of tabular ML is out of the scope of the paper.

Analysis

1. We apologize for not properly calling out the experiments addressing 1) in the submission. The results are actually included in Table 3 in the appendix. We found that with one hour of hyper-parameter tuning, we cannot improve upon the weights produced by the hyper-network using gradient descent. We are happy to include a discussion of this experiment in the final version of the paper.

2. We did extensive hyper-parameter tuning on a space including the architecture of the child network to produce the MLP results (minus the low-rank constraint with is an optimization that did not alter accuracy in our experiments). Are you suggesting that limiting the hyper-parameter space would provide a fairer comparison? Or are you suggesting the low-rank constraint should be included in the hyper-parameter search?

We would like to thank reviewer QK8H for their comments and suggestions, which we'll address below.

Why limiting to small datasets

While we were able to computationally handle larger datasets in inference, the training data included up to 2000 points. While we expect the model to generalize somewhat beyond this (as shown in the TabPFN paper), we think it’s unlikely to scale to orders of magnitude more training data and perform well. Training on much larger datasets would be quite expensive with the current procedure.

Practical applicability and future research

As you say, that the approach works is somewhat surprising and compelling. This is a first step towards the full potential of the approach, and we think further research can improve accuracy much beyond what we demonstrate, for example by using other priors, other encodings and other architectures. As with all foundational models, training the model is quite expensive, which makes it difficult to fully study the effects of modifications in a single paper. This paper is meant as a feasibility study for the approach. We also disagree with the absence of a niche; there are many datasets we studied (unfortunately many of them in the validation set) on which linear models do not perform well, and MotherNet excels. Given the very good performance of TabPFN, it is likely that this model can be improved; on small datasets it already out-performs the default configurations of GBRT implementations that have a decade of tuning behind them.

Hyperfast

The main contribution beyond hyperfast is that this model is accurate on small datasets, and does not require hyper-parameter tuning and gradient descent. While they hyperfast paper discusses gradient descent and hyperparameter tuning to be optional, in practice they are not, as shown by our results. Hyperfast without hyper-parameter tuning will fail catastrophically on iris, depending on the specific split chosen (this has been observed by others, but has not been published so far – we are happy to add a more detailed analysis of the many ways in which HyperFast fails in the final version). As we discuss in the paper, HyperFast is outperformed on CC-18 by essentially all baseline algorithms, even though most of the CC-18 datasets are in the HyperFast training set. As an added benefit, the models generated by MotherNet are orders of magnitude smaller than those generated by HyperFast. In summary, MotherNet shows that accurate hyper-networks for tabular data are possible without per-dataset gradient descent, which HyperFast alluded to but does not actually provide. MotherNet outperforms HyperFast consistently at 25,000x less per-dataset computational cost compared to HyperFast, as mentioned in the paper. We could provide an in-depth comparison to the architecture of HyperFast, but since we have not been able to produce reasonable results with HyperFast (as observed by other researchers), even on its training set, the specific architectural choices made by HyperFast seem less interesting.

Embedding vector size

The embedding vector size in the paper is indeed wrong, thank you for pointing that out. The correct number should be 2hr + Nh=37888 (since the final layer is not low-rank, since 10<32 having low-rank would not make much sense).

Experimental results

As you observe, the (also untuned) model outperforms the untuned XGBoost model, while operating under a severe limitation of only accessing at most 3000 datapoints - and training and predicting faster than XGBoost in most cases. This is unmatched by any other deep learning architecture as far as we are aware (TabPFN predictions are much slower than XGBoost).

Thank you for your feedback and questions. It would be great if you could provide more details about where you find gaps in the presentation, so that we could close them.

Comments regarding your specific questions and criticism below.

Limited Domain Exploration

Exploring other domains is certainly interesting, though out of the scope of this work. Pretraining has been extremely successful for textual and image data, so from-scratch learning of new models is rarely if ever done, in contrast to tabular data. A possible angle would be to generate task-specific heads on pre-trained models; however, that might already be possible with the mothernet architecture we provide. Tabular data has the benefit that comparatively small models can be made to perform well, and the issue is mostly around regularization. For other modalities, models are usually much larger, and generating them from scratch would result in even larger hypernetworks.

Scalability Constraints:

We are investigating alternative attention approaches with promising results. However, the results are not at the stage yet where they could be included in the paper.

Comparison Depth:

We provide a comparison with results on TabZilla, which includes a wide variety of recent deep learning models. Is there specific models that you’d like to see added to the comparison? We do not compare against these models on the CC-18 dataset since the hyper-parameter tuning would be prohibitively expensive, and restrict our experiments only to the most similar hyperfast model.

We updated the PDF to move more of the figures into the main body of the paper, highlight the main distinction over hyper-fast, and elaborate on the fine-tuning experiments for MotherNet in the main text.