Linear Mode Connectivity in Differentiable Tree Ensembles

Thank you for your review.

Empirical Validation Scope: The empirical results are primarily based on tabular datasets. Extending the validation to additional types of data (e.g., image or sequential data processed via tree ensembles) could further strengthen the findings.

We have additionally conducted experiments using the MNIST dataset, confirming that LMC can be achieved when the image data is flattened into a one-dimensional input. While this is a single instance of other types of data, it provides a foundational proof of concept. Results from the MNIST experiment are included in the Appendix of our revised paper. Since soft tree ensembles are primarily designed as an alternative to MLPs, they are not inherently suited for direct processing of images or sequential data. Therefore our main experiments focus on tabular datasets— the primary domain for tree ensembles. Although it is technically possible to create an end-to-end model by pairing a CNN or RNN module to extract features as a one-dimensional vector, including CNNs or RNNs would complicate the discussion. Therefore, we opted for a simpler experimental setup in this study.

Scalability of Additional Invariance Handling: The paper discusses the computational expense associated with handling additional invariances for deeper trees. However, a more detailed discussion of potential optimizations or heuristic approaches to manage this complexity would be beneficial.

As discussed in the final paragraph of Section 4.2, the importance of considering additional invariants with an efficient (potentially heuristic) matching algorithm is relatively low. There, we have noted that deep perfect binary trees are not particularly significant, which is supported both empirically and theoretically by the NTK theory [1], and such an efficient matching algorithm would only be necessary for handling those deep perfect binary trees. We have also presented a modified decision list as one of possible solutions to the problem. When exact solutions are not required and heuristic methods are acceptable, one possible approach for subtree flip invariance, for example, could involve using a greedy algorithm. For example, to consider subtree flip invariance, this algorithm would sequentially determine the orientation of inequalities, starting from the root of the tree.

Comparative Analysis: While the results demonstrate LMC in tree ensembles, the paper could benefit from a more in-depth comparison with related methods or model architectures, particularly alternative approaches to model merging or parameter interpolation that do not rely on LMC.

Thank you for your comment. It is important to note that model merging (i.e., linear parameter interpolation) without explicitly assuming LMC often relies on the condition that all training processes start from the same initial parameters. For example, in techniques like Model Soup [2], parameters are merged simply by averaging, without explicitly addressing LMC. However, this approach works because all training starts from the same pretrained model checkpoint, inherently (and implicitly) satisfying the LMC condition. In contrast, our study explores the merging of two models trained from different initial parameters, offering a more generalized framework.

Equation Reference: Please use "Equation (1)" instead of "Equation 1."

Thank you for your feedback. For ICLR, the LaTeX template is provided on the Author Instruction page, which includes a math reference module (math_commands.tex). By using this module, the formatting is naturally set to omit parentheses around equation references.

% Reference to an equation, upper case
\def\Eqref#1{Equation~\ref{#1}}

Currently, we are following this convention. We would appreciate keeping it as is to contribute to a consistent format across the conference.

---

[1] Kanoh & Sugiyama, A Neural Tangent Kernel Perspective of Infinite Tree Ensembles, ICLR2022

[2] Wortsman et al., Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time, ICML2022

Thank you for your review.

Lack of Rigorous Theory: the paper would have been strengthened significantly had the authors provided a rigorous analysis of the effect of the stated invariances (permutation, subtree flip, and splitting order) on the ability to interpolate 2 (or more) models - even if only on extremely simple cases. Were that so, the empirical results would have been used to support rigorous theory, which would have been a much more powerful argument.

We agree that providing such a theory would be ideal. However, as mentioned in the Introduction, even in the field of neural networks, where research is far more advanced, a theoretical understanding remains an open problem. We believe that our contribution, which examines LMC from a different perspective beyond neural networks, contributes to a deeper understanding of LMC.

Linear Mode Connectivity was proposed to understand how deep neural networks could be sparsified (Frankle et al, 2020). Can a similar analysis of methods to sparsify decision trees be done, provided the insights proposed in this work? An analysis of this sort - where the need to interpolate models becomes evident - would have also noticeably improved this work.

Yes, the Lottery Ticket Hypothesis (LTH) is closely linked to the concept of LMC as explored in prior research [1] and we believe our work could further connect to these perspectives by expanding on this line of research. The LTH within the context of soft tree ensembles remains largely unexplored, presenting a promising avenue for future study. However, it is important to note that our research focuses on examining LMC between models with different random initializations and different random SGD noise. This setup differs from [1], which analyzes connectivity between two models trained from the same parameters but with different SGD noise, offering a more constrained framework. Our primary motivation is to deepen the understanding of non-convex optimization and explore techniques like model merging. In this regard, our study aligns more closely with [2].

The authors do not highlight specific tasks for which interpolation of trees would be of significant benefit. This weakens the argument motivating the need for this work. For instance, does model interpolation in tree ensembles improve robustness to distribution shift?

We believe that soft tree ensembles will become a practical choice for a wide range of real-world applications, such as federated learning and continual learning, as highlighted in the conclusion of our revised paper. Soft tree ensembles and neural networks such as MLPs are already being used in practice as one of the options for machine learning models. Our research offers a broader pool of models for trial and error in the context of model merge, which we regard as a valuable contribution. Furthermore, we believe that assessing the robustness of model merging under distribution shifts is a significant topic in its own right. Expanding comparisons across different model types, such as decision trees and neural networks, would further broaden the scope of this research. These considerations represent an important avenue for future work.

---

[1] Frankle et al., Linear Mode Connectivity and the Lottery Ticket Hypothesis, ICML2020

[2] Ainsworth et al., Git Re-Basin: Merging Models modulo Permutation Symmetries, ICLR2023

Thank you for your review.

The authors do not provide a more efficient matching algorithm other than exhaustive search.

This is because the importance of considering additional invariants with an efficient (potentially heuristic) matching algorithm is relatively low, as discussed in the final paragraph of Section 4.2. There, we have noted that deep perfect binary trees are not particularly significant, which is supported both empirically and theoretically by the NTK theory [1], and such an efficient matching algorithm would only be necessary for handling those deep perfect binary trees. When exact solutions are not required and heuristic methods are acceptable, one possible approach could involve using a greedy algorithm. For example, to consider subtree flip invariance, this algorithm would sequentially determine the orientation of inequalities, starting from the root of the tree.

it would be necessary to include some experiment similar to the one shown in Figure 8, applied to the proposed architecture.

Thank you for your suggestion. We have added experimental results on decision lists in the Appendix. Similar to the case of the perfect binary tree, we observed performance improvements when using model merging with the split dataset. Additionally, decision lists offer more efficient matching, which could be advantageous in certain practical situations.

In Figure 6, I don't quite understand why AM considering only permutation increases the barrier more than naive interpolation.

When AM ignores subtree flip invariance, linear interpolation between parameter vectors oriented in opposite directions is likely to occur. As discussed in Section 3.1, subtree flip invariance involves reversing the signs of $w_{m,n}$ and $b_{m,n}$​ to flip inequality signs. Ignoring this can lead to interpolations between oppositely oriented parameters, driving the combined model parameters toward zero. We believe this is one of the reasons why the barrier becomes larger than that of naive interpolation.

Why do the authors think that the barrier after matching increases as depth increases?

We think that the shape of the loss landscape, influenced by the model's structure, plays a role. As the depth of the tree increases, the loss landscape is likely to become more complex. If the loss valleys have irregular shapes, convergence within the same valley could still face larger barriers due to these distortions in the loss landscape. By contrast, in soft tree ensembles, increasing the number of trees tends to simplify the loss landscape, often making training easier. This phenomenon is partially explained by theories such as NTK [1]. These insights may help explain the observed results.

Is there a compelling reason to use activation matching over weight matching, given that it does not seem to offer any clear advantages? Or is it simply intended to replicate the methodology of Ainsworth et al. (2023) for validation purposes?

Yes, we have investigated both AM and WM, not only to follow previous research but also because the theoretical validity of both AM and WM has been proven in [2].

It would be useful to know the actual runtime required for matching. Compared to neural networks commonly considered in standard LMC research, such as ResNet, the number of parameters here is significantly lower. Given this, even with a large number of invariance patterns, it may not take very long to check all of them.

Thank you for your comment. While we agree that there is a potential for high-speed operation, our Python code for handling tree-structured data is not well optimized. As a result, the computational overhead required to prepare data for LAP can be large, making direct runtime comparisons challenging at this stage. Therefore, we focused on discussing computational complexity instead in our paper. With implementation optimizations, we believe it is possible to achieve comparable or even faster processing speeds depending on the number of invariance patterns involved. As noted in our paper, while multi-layer neural networks typically require performing LAP multiple times due to coordinate descent, this iterative operation is unnecessary in tree ensembles. This generally reduces the computational cost for tree ensembles.

Some minor suggestions

Thank you for your suggestion. We have fixed Figure 6 and clarified the specific meanings of terms to make them explicit. To avoid any confusion, we have explicitly defined all the notations, "Naive," "Tree Permutation”, and "Ours", at the end of the first paragraph in Section 4.2.

---

[1] Kanoh & Sugiyama, A Neural Tangent Kernel Perspective of Infinite Tree Ensembles, ICLR2022

[2] Zhou et al, Going Beyond Linear Mode Connectivity: The Layerwise Linear Feature Connectivity, NeurIPS2023