Dynamic Post-Hoc Neural Ensemblers

Dear reviewer, thank you for your feedback and interesting insights. Below we answer your questions:

Weaknesses

1. While the paper mentions using a fixed set of hyperparameters across all experiments, it does not explore the sensitivity of the Neural Ensemblers to hyperparameter changes. A more detailed analysis of hyperparameter tuning could provide deeper insights.

We appreciate your emphasis on the importance of hyperparameter analysis. In response, we conducted an ablation study on the number of layers and the hidden dimension of our Neural Ensembler. The results, presented in Appendix J (Figure 13) of the updated manuscript (or here, demonstrate that our default configuration strikes a balance that provides robust results across all the metadatasets.

2. Does the author carefully tune the hyper-parameters of other baselines e.g., XGBoost?

We used the default settings recommended in the respective libraries. These methods are well-researched with established default hyperparameters. Similarly, we kept the hyperparameters of our method fixed throughout the experiments to ensure a fair comparison.

3. The author should also compare their method with other advanced methods, e.g., LightGBM, CatBoot, which can be directly used in the open source libraries.

We included additional baselines such as CatBoost and XGBoost in Appendix I (Tables 18 and 19) of the updated manuscript (also available here and here). We did not include LightGBM as it is similar to the Gradient Boosting baseline already evaluated.

4. Lack of references and comparisons to Ensemble NN [1] and SCARF [2]. Both of these papers explore the ensemble or pretraining in Deep neural networks on tabular datasets.

Thank you for bringing these references to our attention. Unfortunately, we cannot find an open-sourced implementation of Ensemble NN to include it as a baseline. However, we discussed it in the related work, Section 5, of the updated manuscript. While SCARF is indeed a very interesting approach, it focuses on pretraining for tabular data rather than ensembling, thus falling outside the primary scope of our study.

---

Finally, thank you very much for taking the time to review our paper and providing us with valuable feedback, prompting several improvements to our submission. We hope we have clarified some of your concerns and would appreciate it if you consider improving the rating in your review. We welcome any further suggestions or feedback that you would be willing to share.

Thank you for your continued engagement and for acknowledging the additional results we provided.

To address it, we run LightGBM and compare it against all baselines. See results in Table 18 and 19 from the submitted PDF or here and here. Our results are consistent with previous works that have shown CatBoost and XGBoost outperforms LightGBM across various datasets [1, 2, 3].

We appreciate your feedback and hope our response addresses your concerns which would reflect in the score increase.

References:

[1] Kohli R., Feurer M., Eggensperger K., Bischl B., Hutter F. (2024). Towards Quantifying the Effect of Datasets for Benchmarking: A Look at Tabular Machine Learning. ICLR 2024 Workshop on Data-centric Machine Learning Research.

[2] Salinas, D., Erickson, N. (2024). TabRepo: A Large Scale Repository of Tabular Model Evaluations and its AutoML Applications. AutoML Conference 2024 (ABCD Track).

[3] McElfresh, D., et al. (2023). When Do Neural Nets Outperform Boosted Trees on Tabular Data? Advances in Neural Information Processing Systems.

4. It would be helpful to include the raw numbers in the appendix.

We appreciate your suggestion for greater transparency. In response, we added tables with the unnormalized results in Appendix E.1 of the updated manuscript (also available here and here).

---

Finally, thank you very much for taking the time to review our paper and providing us with valuable feedback, prompting several improvements to our submission. We hope we have clarified some of your concerns and would appreciate it if you consider improving the rating in your review. We welcome any further suggestions or feedback that you would be willing to share.

4. The key feature of the proposed Neural Ensembler is its ability to adjust the weights of each ensemble member in an instance-specific manner, either implicitly (in a stacking setup) or explicitly (in a model-averaging setup). However, there is currently no direct evidence regarding its effectiveness from this perspective. [...]

We kindly want to point out that our proof-of-concept experiment, detailed in our paper (Figure 3), demonstrates the advantage of dynamic ensembling in a one-dimensional toy problem. Additionally, our extensive empirical results in a broad range of modalities and datasets provide evidence of the effectiveness of instance-specific weighting. While we value the experiment that you suggested to test this hypothesis, we believe that learning a Neural Ensembler that could approximate the oracle would be challenging, as there could be many possible ensemble weights that also fit the data (besides the oracle). In other words, we might learn a set of weights, different from the oracle, that also fits the data.

In our experiments, we consider ensembles with constant weights that do not depend on the instance (see table 1 and 2, baseline MA), which lean fixed weights so that the output is $y=\theta_1 z_1 + …\theta_n z_n$, where z are the model predictions. As we can see in these tables, learning dynamically the ensemble (NE-MA method) outperforms the fixed weight approach MA.

5. The fact that nearly all results overlap within the range of standard deviations suggests that the experimental results may not be statistically significant. [...]

We appreciate your observation regarding statistical significance. To understand the significance of the results, we included Critical Difference (CD) diagrams in Appendix H of the updated manuscript. These diagrams (Figure 11h or here) show that our method achieves top performance across multiple datasets and performs best when aggregating all datasets. There is no baseline better than our approach, and, given the ablation studies, it still has room for improvement.

We additionally found out that our method is significantly better than Greedy (the second best approach) in the metadatasets with a lot of number of classes (appendix N). We hypothesize this occurs because of the neural ensembler's design, as it shares parameters across different classes (section 3.1).

Moreover, our method is faster than the Greedy baseline (Figure 9), which is the second-best method. We achieved these results using the same Neural Ensembler configuration across all metadatasets, highlighting its robustness. We could further improve the performance per metadataset by carefully tuning the hyperparameters (Appendix J (Figure 13) or here).

We also emphasize the theoretical contribution of our work, including proofs that our method provides a lower bound on ensemble diversity, which is crucial to avoid overfitting during ensemble learning. For further details, please refer to the response to reviewer Svxb (first weakness point).

Questions

1. I think the assumption of having a sufficiently large amount of validation data [...] is quite strong [..].

As previously mentioned, we do not use assume the availability of a large validation set. Our ablation study in Appendix K (Figure 14) (or here demonstrates that our method maintains strong performance even with significantly reduced validation data. For further details, please refer to the response to reviewer Svxb (second question).

2-3. One can approximate the model probability in Bayesian model averaging using the Bayesian Information Criterion (BIC), where the likelihood is estimated from the validation set. [...] Following BIC, the harmonic mean estimator and its variants come to mind.

Thank you for highlighting the relevance of Bayesian methods for increasing performance in the context of ensemble learning. In response, we would like to argue that our method approach uses a modified version of dropout, which can be interpreted as a Bayesian approximation, as discussed in [1]. We acknowledge that methods like BIC and harmonic mean estimators could potentially improve performance. To this end, we included a brief discussion of these methods in Appendix B of the updated manuscript.

[1] Gal, Y., & Ghahramani, Z. (2016, June). Dropout as a bayesian approximation: Representing model uncertainty in deep learning. In international conference on machine learning (pp. 1050-1059). PMLR.

Dear reviewer, thank you for your valuable feedback and insightful comments. We have addressed your concerns below:

Weaknesses

1. There are concerns about fair comparison since the proposed Neural Ensemble ultimately learns from additional validation data. While baseline methods also train statistical models based on validation data, the issue is that the Neural Ensemble is significantly larger than the baseline method. [...]

We understand your concern regarding the use of additional validation data and the size of the Neural Ensembler. **It is important to note that all methods in our comparison, except for the Random baseline, use validation data. For instance, Greedy approach uses the validation data for selecting iteratively the model that decreases the loss if added to the ensemble. SVM and RF uses the validation data to optimize the respective model parameters.

In fact, some baselines overfit the validation data, as observed by comparing Figures 4 and 8 in our manuscript, which means they are expressive enough, and in turn, that this is not an issue of lack of expressivity or underparametrization of the baselines. To address the concern about Neural Ensembler’s dependence on validation data, we conducted an ablation study varying the percentage of validation data used for its training. The results, presented in Appendix K (Figure 14) of the updated manuscript (or here), demonstrate that our method remains robust when the amount of validation data is significantly reduced. This indicated that our method does not heavily depend on a large validation set.

2. I believe the assumption of having a sufficiently large amount of validation data, which can be used to train the neural network model, is quite strong. [...]

Thank you for highlighting this point. As mentioned, our ablation study in Appendix K (Figure 14) (or here) shows that our Neural Ensembler performs well even with limited validation data.

3. Additionally, the Neural Ensembler is implemented using an MLP architecture with a hidden dimension of 32 and a depth of four. Such architectural choice also warrants an ablation study; a model that is too large may incur significant costs and increase overfitting, while one that is too small may not perform adequately. [...]

We appreciate your suggestion regarding the architectural choices of the Neural Ensembler. In response, we conducted an ablation study varying the number of layers and the hidden dimension of the MLP architecture. The results are presented in Appendix J (Figure 13) of the updated manuscript (or here). Our results show that the original architecture strikes a balance between performance and complexity. While modifying the architecture can lead to performance improvements on certain datasets, it decreases the performance on others. This demonstrates that our chosen configuration is robust across diverse datasets, providing consistent results without the need for extensive hyperparameter tuning.

As suggested by the reviewer, we included a baseline that considers akaike-weighting [1] or pseudo bayesian model averaging. We can see that the method obtains interesting results in some metadatasets such as NB-Mini, however it does not outperform our method. The reviewer is invited to check Table 18 and 19 from the updated PDF, or here and here.

[1] Wagenmakers, E. J., & Farrell, S. (2004). AIC model selection using Akaike weights. Psychonomic bulletin & review, 11, 192-196.

The logic here remains unclear. Ablation studies indicate that deviating from the originally presented configuration, by modifying the number of hidden layers or hidden dimensions, results in significant performance changes. It is unclear how this aligns with the claim that the method works "without the need for extensive hyperparameter tuning"?

In the original paper version, we explain in Lines 352-254 that we find the hyperparameters (HPs) using a separate metadataset split from Quicktune which is not used for the final test. This set of HPs is fixed because they do not change across metadatasets, even though they are from different data modalities.

As I understand it, the Normalized NLL in Figure 14 uses the original results obtained with the full validation data as a baseline value of 1, with relative differences shown accordingly. However, considering that the performance gap between the proposed method and the baselines wasn’t particularly large to begin with, wouldn’t a result of 1.25 times the original indicate performance worse than the baselines? Perhaps it could be worth considering using a simple yet competitive baseline like Greedy (applied to a subset of the validation data) as the baseline for this ablation.

As we mention in the first bullet point from our response, all baselines except Random use validation data, so we expect that they also will have a drop in performance, as they will use less validation data.

Moreover, why do QT-Micro and QT-Mini exhibit worse performance at 20-40% compared to 10% (or even 5% and 1%)? Intuitively, performance should tend to degrade as the validation data decreases from 100% to 0%. Additionally, there are instances—presumably around 1%—where the Normalized NLL approaches 1 (e.g., NB, QT-Micro, QT-Mini), which also seems odd.

We thank the reviewer for raising this point. We realize a problem in the plotting for this rebuttal experiment, which we fixed in the updated version (see Figure 14 or here). Indeed, we can see that our method is affected by decreasing the validation data size. We also revise the replies regarding the discussions around this plot. For some metadatasets the decrease is relatively small (FTC, NB), while for some is larger (QT-Micro). We hypothesize this occurs because the average number of samples in the validation split is already relatively small for QT-Micro (see Table 5).

While examining Table 3 (to see the baseline performance), I noticed something strange. How is it possible for Greedy, which achieves a value of <1 in NB (100), to achieve =1 in NB (1000)? When Greedy's performance is 1, it suggests that after selecting the initial single best candidate, all other candidates provide no improvement in validation performance when added to the ensemble, meaning the ensemble construction ends with just the single best candidate.

We invite the reviewer to bear in mind that our results are reported on a separate test data split. We will likely find an ensemble that improves the performance over a single model on the validation data split. However, it is possible that this ensemble is not generalizing well, thus having lower performance in the test split.

Imagine a situation where you tune model hyperparameters using a validation split, and you find a good hyperparameter configuration on this split. It can happen that it does not translate to a test because we probably overfit the validation split. This behavior has been seen in HPO (hyperparameter optimization) [1][2], and it is a plausible situation here as well.

[1] Lévesque, Julien-Charles. "Bayesian hyperparameter optimization: overfitting, ensembles and conditional spaces." (2018).

[2] Makarova, Anastasia, et al. "Automatic termination for hyperparameter optimization." International Conference on Automated Machine Learning. PMLR, 2022.

Dear reviewer, thank you for your valuable feedback and insightful comments. We appreciate the opportunity to address your concerns and clarify aspects of our work.

Weaknesses

1. The study lacks consideration of computational efficiency. The Neural Ensemblers’ performance depends on a strong subset of 50 base models, selected by DivBO, which may increase computational requirements.

We apologize for the misunderstanding. In reality, our Neural Ensembler does not depend on a strong subset of base models, we discuss this in L431. As demonstrated in our experiments addressing RQ2, our method can effectively ensemble base models selected at random.

To address the computational efficiency, we included a comprehensive analysis in Appendix F (Figure 9) of the updated manuscript (or here). This analysis compares runtime costs of different post-hoc ensembling methods, including both training and inference times. The results indicate that our method exhibits lower runtime than other high-performing baselines, demonstrating that our Neural Ensembler is both effective and computationally efficient, even without relying on a subset of strong base models.

2. My review of the author's code confirmed that the meta-datasets are limited to a tabular format, making the paper's description somewhat vague. A clearer explanation of the data structure would have been beneficial. Otherwise, the application scope should be limited to tabular data or metadata in a tabular format from fields like computer vision or natural language processing. [...]

Thank you for bringing this to our attention. While we store base model predictions in tabular form for computational efficiency, our method is not limited to tabular input data. We included experiments on various modalities, including computer vision (e.g. QuickTune, NASBench) and natural language processing datasets (FTC). Storing the base model predictions as tables simply allows us to simulate the ensembles faster.

Consider a regression task with two samples and three base models. The predictions from the base models are:

$$\begin{bmatrix} z_{1,1}, z_{1,2}, z_{1,3} \newline z_{2,1}, z_{2,2}, z_{2,3} \end{bmatrix}$$

Here, $z_{i,j}$ is the prediction of the j-th base model on the i-th sample. These predictions could come from any modality - for example, images processed by ResNet or ViT models.

We assign weights to the base models:

w_1 \newline w_2 \newline w_3 \end{bmatrix}

We can simulate any combination of weighted ensembles as we just need to do a matrix multiplication for computing the ensemble prediction Y.

$Y=ZW$

Notice that we are still working with vision models; this tabular abstraction of the predictions facilitates the evaluation of baselines. Previous works have applied this technique to explore AutoML approaches (e.g. [1)].

[1] Salinas, D., & Erickson, N. (2023). TabRepo: A Large Scale Repository of Tabular Model Evaluations and its AutoML Applications. arXiv preprint arXiv:2311.02971.

3. The author selects some base models and computes their predictions for each example. These predictions are then stored in tabular data format as examples to generate the training set. Lastly, train the Neural Ensembler on this set.

Thank you for summarizing this approach. Indeed, this is precisely the methodology we employ in our work. We apologize for the confusion.

Thank you for providing further clarification. Based on your response, I have summarized my understanding of your work as follows:

(1). Based on Appendix F (Figure 9), Neural Ensemblers outperform each single model but do not demonstrate computational efficiency advantages, which seems to contradict the authors' claims.

(2). As per your explanation, the Dynamic Post-Hoc Neural Ensemblers cannot be considered end-to-end models. In the data pipeline, predictions from 50 models are first collected and saved as tabular input data, which is then processed by the Ensemblers.

(4). To strengthen the argument, I recommend conducting quantitative ablation studies. Without considering the total number of model parameters or FLOPS, it becomes difficult to argue that multiple smaller models outperform a single large model. Even in scenarios with multiple GPUs and limited memory, a single large model can still be executed efficiently using Tensor Parallel (TP) or Pipeline Parallel (PP) techniques.

(6). Similar to point 3, the response does not fully address the concern about computational efficiency in comparison to single large models.

(7). The author's response, "Additionally, in Appendix M (Figure 15) we further show that, particularly for tabular data, using the raw input can lead to overfitting, thus adding complexity without significant benefits." is confusing. If the Ensemblers were run in an end-to-end mode without saving/loading tabular data file, the final output should not differ. Could you clarify why this approach would lead to overfitting?

2. More data is needed to train the parameters of the dynamic ensemblers It is unclear if we can reuse the training data used for base models or do we need more training data? [...]

Most of the baselines (except the Random baseline) use additional data (validation data) to optimize the ensemble. We run an experiment when we decrease the validation data size, and see that the performance of our method barely decreases in some metadatasets. For these experiments refer to Appendix K (Figure 14) (or here).

Additionally, we run an experiment where we train the base models and the ensemblers on the same data (merging the training and validation split). We can see the results in Appendix L (Figures 15 and 16) (or here), showing that training the base models and the ensemblers leads to overfitting. Thus it is better to train the ensemble in a separate split. This behavior is exhibited by other ensemble strategies such as Greedy baseline.

---

Finally, thank you very much for taking the time to review our paper and providing us with valuable feedback, prompting several improvements to our submission. We hope we have clarified some of your concerns and would appreciate it if you consider improving the rating in your review. We welcome any further suggestions or feedback that you would be willing to share.

3. It remains unclear what are the main reasons due to which the proposed methods improves on top of simple ensembling baselines. [...]

Thank you for your thoughtful comment. We'd like to clarify the main reasons why our proposed method outperforms simple ensembling baselines. Our method introduces two key innovations that collectively enhance performance.

Dynamic, Instance-Dependent Ensembling: Unlike traditional ensemble methods that assign fixed weights to base models, our approach computes ensemble weights dynamically for each input instance. This means that for each data point, the ensemble can emphasize the base models that are most relevant to that specific instance. This adaptive weighting allows the model to tailor its predictions based on the unique characteristics of each input, leading to improved accuracy.

Regularization through Dropout and Parameter Sharing: During training, we apply dropout to the predictions of the base models. This serves as a regularization technique, promoting diversity among the ensemble members and preventing the model from overfitting to the training data. Additionally, our Neural Ensembler employs a shared neural architecture with parameter sharing across different components. This reduces the total number of parameters and encourages the model to learn more generalizable features.

No prior work or baseline considers these aspects jointly. Traditional ensembling methods typically use fixed weights and do not incorporate instance-dependent weighting or these specific regularization strategies.

4. Post-hoc ensembling in the name can be misleading for someone who is not aware of the literature. In particular, to obtain the ensemble we need to train the parameters of the ensemble model.

Thank you for bringing this to our attention. We understand that the term "post-hoc ensembling" may be unfamiliar to some readers and could potentially cause confusion. In our work, "post-hoc ensembling" refers to methods where the base models are already trained, and the ensemble model is focused on aggregating their predictions without retraining the base models themselves.

While it is true that we need to train the parameters of the ensemble model, the key aspect of post-hoc ensembling is that it operates on pre-trained base models. This approach is discussed in previous works [1, 2], where the ensemble aims to enhance performance by optimally combining existing models.

[1] Purucker, Lennart Oswald, and Joeran Beel. "CMA-ES for Post Hoc Ensembling in AutoML: A Great Success and Salvageable Failure." International Conference on Automated Machine Learning. PMLR, 2023.

[2] Shen, Yu, et al. "DivBO: diversity-aware CASH for ensemble learning." Advances in Neural Information Processing Systems 35 (2022): 2958-2971.

Questions

1. It is not clear when dynamic ensemblers would benefit? Is it the case that dynamic ensemblers improve because base models underfit the data?

Thank you for your question. Dynamic ensemblers benefit in situations where the optimal combination of base models varies across different instances, making instance-dependent aggregation advantageous. This means that each sample can be better predicted by weighting base models differently based on its specific characteristics. As highlighted in our motivation section (Section 2), dynamic ensembling allows the model to adapt to diverse patterns within the data.

Importantly, this benefit is not limited to cases where base models underfit the data. Our experiments with overparameterized base models (*Appendix G (Figure 10)**) demonstrate that dynamic ensemblers continue to improve performance even when base models are sufficiently expressive. This suggests that the advantage of dynamic ensembling comes from its ability to adapt predictions to individual instances, enhancing performance beyond what is achievable with static ensemble methods.

Finally, Quicktune [1] and Nasbench [2] metadatasets have been used in previous papers, demonstrating that they contain well-tuned models.

[1] Arango, Sebastian Pineda, et al. "Quick-tune: Quickly learning which pretrained model to finetune and how." arXiv preprint arXiv:2306.03828 (2023).

[2] Ying, Chris, et al. "Nas-bench-101: Towards reproducible neural architecture search." International conference on machine learning. PMLR, 2019.

Dear Reviewer, thank you for your valuable feedback and insightful comments. We have addressed your concerns below:

Weakneses

1. The benefit of the proposed method is not super clear in Table 2. Given the variance of different methods, the efficacy of the proposed method is unclear. [...]

Thank you for your valuable feedback and for pointing out the need for a clearer demonstration of our method's benefits in Table 2. We understand that, given the variance among different methods, it might not be immediately evident how our method stands out in terms of efficacy. We want to clarify the benefits by pointing out the following aspects of our work:

• Comprehensive Performance Metrics: In the original manuscript, Table 2 and Table 3 reported various metrics: the average normalized error, AUC, and average normalized Negative Log-Likelihood (NLL), showing that our method consistently achieves top performance across various datasets and modalities. A more detailed list of results can be found in Appendix E.

• New Critical Difference (CD) Diagrams: To add a layer of statistical interpretability, we computed Critical Difference diagrams, which visualize the performance differences among methods while considering statistical significance. As shown in Figure 11 of the updated manuscript (or here), our method achieves the best average rank and is significantly better than many baselines. Additionally, statistically, no other method is better than ours.

• New Computational Efficiency Analysis: Recognizing that practical efficacy also depends on computational cost, we analyzed the runtime performance of our method compared to others. Figure 9 in the updated manuscript (or here) illustrates that our method not only excels in performance but also is faster than several baselines.

• New Insights about Significance on Datasets with a lot of Classes: In section N (figure 17) we demonstrate that our method is statistically significantly better than the second best method (Greedy) in datasets with a lot of classes. The reviewer can also see the results here. We hypothesize this occurs because of the neural ensembler's design, as it shares parameters across different classes (section 3.1).

• Theoretical Guarantees: To support our empirical findings, we provide theoretical proofs, ensuring that our method promotes ensemble diversity, which is crucial for limiting overfitting and enhancing generalization. These theoretical contributions are detailed in Propositions 1, 2, and 3 of our paper. Our ablation study in Figure 6 configure that using the base models DropOut is indeed very helpful, i.e. DropOut rate >0 is yields the best performance across many datasets.

2. POC example given in the paper can be misleading as their dynamic ensemblers train more parameters as compared to baseline. It remains unclear if this would be a problem in the first place if we could train base models that are sufficiently overparameterized. Does benefit of these ensemblers persist in cases when the base models are over parameterized neural networks?[...]

Thank you for raising this important point. To address your concern, we extended our proof-of-concept experiment to test whether the benefits of our dynamic ensemblers persist with overparameterized base models.

We increased the complexity of the base models by using polynomials of degree 10 instead of degree 2. The results, shown in Appendix G (Figure 10) (or here), demonstrate that even with these more expressive base models, our dynamic ensemblers continue to outperform the baselines.

This indicates that the advantages of our method are not limited to scenarios with underparameterized base models. Even when base models are sufficiently overparameterized, our dynamic ensemblers provide improved performance, due to their ability to mitigate overfitting and adaptively combine model predictions.