LoRA-Ensemble: Efficient Uncertainty Modelling for Self-attention Networks

W4. & Q4. Existing methods cannot be applied to Transformer architecture

We would like to clarify that we do not claim it is impossible to adapt methods like BatchEnsemble, or similar approaches to transformers. Instead, as mentioned in Appendix M, we state that it is challenging due to the unique computational structure of transformers compared to CNNs and MLPs. To illustrate, we attempted to implement FiLM-Ensemble on transformers, which required significant time and effort. While we achieved some preliminary results by adapting LayerNorm to emulate FiLM-Ensemble's behavior, the improvements were minimal and inconsistent.

Even for simpler methods like Dropout, adapting them to transformers is not straightforward. For example, significant efforts have been made to adapt Dropout to transformers, such as DropAttention (Lin et al., 2019) and DropKey (Li et al., 2023). These approaches involve substantial modifications to the standard Dropout technique, including mechanisms like re-normalization in DropAttention and advanced dropping schedules in DropKey, to make them compatible with transformers. This highlights that even the simplest implicit ensembling methods often require significant re-engineering for transformers.

Furthermore, methods like BatchEnsemble have not shown promising results, even on simpler architectures like ResNets. For example, on CIFAR-10, BatchEnsemble achieves a performance of 95.9 compared to 95.7 for MC-Dropout and 95.3 for a single network. While slightly better than MC-Dropout, the improvement is quite insignificant, with no reported error bars, and the Expected Calibration Error (ECE) does not show meaningful improvement either. Additionally, these methods often rely on limited experimental evaluations and do not establish themselves as robust or generalizable approaches.

Given these challenges and the lack of promising results from these methods, we do not see the merit in re-implementing them for transformers, which is itself an active area of research, when their existing performance does not justify such effort. Additionally, we have now included PackedEnsembles in the Related Work section for completeness.

References

Lin, Zehui, Pengfei Liu, Luyao Huang, Junkun Chen, Xipeng Qiu, and Xuanjing Huang. "DropAttention: A regularization method for fully-connected self-attention networks." arXiv preprint arXiv:1907.11065, 2019.

Li, Bonan, et al. "DropKey for vision transformer." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023.

W5. Absence of experiments on text data

Thank you for highlighting the absence of experiments on text data. While LoRA has indeed been popular in the text domain, our work primarily focuses on developing an ensemble approach applicable across diverse tasks, rather than targeting any specific domain like natural language processing. To evaluate uncertainty and demonstrate the robustness of our method, we chose well-established benchmarks such as CIFAR-10 and CIFAR-100, a real-world medical image dataset, and out-of-distribution (OOD) tasks. These datasets and tasks are widely used in the literature on ensemble methods, ensuring that our evaluation is in line with existing works and provides a practical and comprehensive test bed for assessing the generalizability of our approach. Incorporating text data is certainly an interesting direction, but integrating additional domains such as NLP would require significant extensions beyond the scope of this paper. We aim to ensure our experiments remain focused and aligned with the primary goal of developing an efficient ensembling method for self-attention networks. That said, we appreciate the suggestion and may explore applications in the text domain as part of future work.

Q1. How effective is using a mixture-of-experts approach as an “ensemble” for uncertainty evaluation

Comparing our method to mixture-of-experts approaches in terms of efficiency and uncertainty quality would be an intriguing direction for future work. However, such comparisons are beyond the scope of this paper. Expanding the discussion to include mixture-of-experts and other uncertainty estimation techniques, such as Bayesian approaches, would significantly broaden the focus and dilute the paper’s emphasis on efficient ensembles for self-attention networks. [1] can be an interesting read for you.

References

[1] Jiang, Yuchang, et al. "Mixture of Experts with Uncertainty Voting for Imbalanced Deep Regression Problems." arXiv preprint arXiv:2305.15178 (2023).

Q2. how well does it work for MLPs or CNNs?

It would indeed be interesting to explore how the method performs for MLPs or CNNs. However, this is beyond the scope of our paper, as our work specifically targets efficient ensembles for self-attention networks, as reflected in the title: LoRA-Ensemble: Efficient Uncertainty Modelling for Self-Attention Networks. This could be an interesting direction for future work.

We thank the reviewer for the detailed and constructive feedback. Below, we address all the concerns in detail.

W1. The method is primarily limited to attention-based models…

Developing a generic algorithm that works seamlessly across all types of network architectures, such as RNNs, CNNs, MLPs, and Transformers, is extremely challenging when the algorithm involves modifying the structure of computation. To the best of our knowledge, no existing work demonstrates such applicability across all architectures for methods that fundamentally alter network architecture.

Our work explicitly focuses on self-attention networks, as reflected in the title: LoRA-Ensemble: Efficient Uncertainty Modelling for Self-Attention Networks. This focus was intentional, given the widespread adoption and popularity of attention-based models in a variety of domains. By constraining ourselves to this architecture, we aim to provide a robust and efficient solution tailored to its specific structure and use cases.

Additionally, while our method is limited to self-attention models, we demonstrate its versatility across multiple domains, including medical image classification and sound classification. Furthermore, we evaluate our approach on different versions of transformers (e.g., ViT, DeiT) and compare their performance to validate its robustness and flexibility. These experiments highlight that our method can address a wide range of problems across diverse application areas, showcasing its broad applicability within the realm of attention-based models.

W2. & Q3 Similarity to existing method BatchEnsembles… - What specific advantages..

While there is a technical similarity, as noted in the Related Work section, where both methods constrain the parameter space using a low-rank approach, the methods are fundamentally different in design. First, the formulations are distinct, which already makes the two approaches quite different. In deep learning, formulation is critical, similar to how GRU and LSTM are considered separate architectures despite slight differences in their computational structure.

Furthermore, the two methods are developed for entirely different architectures. BatchEnsemble was designed for MLPs, whereas our method is specifically developed for Transformers. This architectural difference further highlights the distinct nature of the methods. Additionally, there are numerous technical distinctions, such as how the parameters are initialized and how pre-trained weights are utilized in our method, which is not the case for BatchEnsemble. Beyond introducing a new ensembling method, our work also explains why it performs well, as detailed in the newly added Section 4. Considering these differences in formulation, experimental focus, and analysis, we believe there is no concern regarding the novelty of our paper.

While re-implementing the BatchEnsemble idea for comparison would be interesting, adapting this algorithm to work effectively with Transformers would require significant re-engineering efforts, careful design choices, and substantial experimentation, which is beyond the scope of this work.

W3.1. It is unclear where the performance gain comes from…

To provide better insight into the effectiveness of LoRA-Ensemble, we have added a new section in the Appendix (see Section A.4, Table 7), where we compare a high-rank LoRA model with a low-rank LoRA model that have a similar number of trainable parameters. The results clearly demonstrate that the low-rank LoRA model outperforms its high-rank counterpart, showing that the performance improvement is not solely attributable to the number of trainable parameters. Instead, the results highlight the unique learning dynamics enabled by LoRA-Ensemble.

Additionally, we have introduced a new section, "Enhanced Diversity in LoRA-Ensemble," which empirically analyzes why LoRA-Ensemble outperforms methods like Explicit Ensemble. This section provides a much deeper understanding of the underlying reasons behind LoRA-Ensemble’s superior performance across most of the experimental settings. Please refer to Section 4 for further details.

W3.2. Mixed results… - ESC-50 experiments

We would like to clarify that, while we do not claim that LoRA-Ensemble outperforms all the baselines in every setting, we do demonstrate that LoRA-Ensemble outperforms or is on par with Explicit Ensemble, while being significantly more computationally efficient. In the ECS-50 experiment, although Explicit Ensemble achieves better performance on most metrics, the differences between LoRA-Ensemble and Explicit Ensemble are statistically insignificant.

We also provide a potential explanation for this in the paper: the initial pre-training was conducted on ImageNet, where there is a large distributional shift between pre-trained dataset and target dataset. We believe if the models were initialized with more appropriate pre-training, LoRA-Ensemble would outperform for this task as well.

We apologize for any confusion and would like to clarify that there may have been a misunderstanding regarding the rebuttal deadline. ICLR extended the discussion period by an additional 6 days, until December 3rd. As a result, we did not submit our rebuttal by the original deadline, as we were focused on completing the revision of the paper. We kindly request that our response be considered for a fair evaluation of our work, given the extended rebuttal period.

We sincerely thank the reviewer for their thoughtful feedback and valuable suggestions, below you can find our response to individual concerns in detail.

W1.1. Limited Theoretical insights and empirical validation

Providing a theoretical analysis of LoRA-Ensemble's relationship with the underlying probability distribution and its ability to prevent ensemble member degeneration is highly challenging. This difficulty arises from the complex interactions in high-dimensional parameter spaces, where pre-trained weights, LoRA updates, random initialization, optimization dynamics, and non-linear transformations collectively shape the learning process. Developing an analytical model would require strong assumptions about data, optimization trajectories, and architecture, which may not universally apply and would constitute a significant contribution to the field in itself.

Since such an analytical proof is beyond the scope of this work, we have instead added a new section, "Enhanced Diversity in LoRA-Ensemble," which empirically analyzes why LoRA-Ensemble outperforms methods like Explicit Ensemble. This section investigates the diversity of ensemble members in both function and weight spaces. The results demonstrate that LoRA-Ensemble achieves significantly greater diversity compared to Explicit Ensemble, which is crucial for constructing effective ensemble.

Additionally, our spectral analysis of the weight space reveals that LoRA-Ensemble produces distinct parameter updates by exploring new high-ranking dimensions that are near orthogonal to the initial weights. In contrast, Explicit Ensemble members tend to adhere more closely to the structure of the initial weights. This enhanced diversity enables LoRA-Ensemble to explore the loss landscape more effectively and capture a broader range of representations, leading to improved robustness and epistemic uncertainty estimation. We also provide visualizations of the training evolution of ensemble members in the loss landscape, demonstrating that LoRA-Ensemble members converge across a broader area, reflecting diverse learning dynamics.

W1.2 & Q1. Large Scale Datasets

We are currently conducting large-scale experiments on the iNaturalist dataset to demonstrate the scalability and robustness of our method on large, unbalanced, and challenging datasets. Due to computational constraints, the results are not yet complete, but our initial findings are promising, showing improvements over Explicit Ensemble baselines. We commit to including the full results for iNaturalist in the final version of the paper.

W2. Paper Formatting

We fixed formatting.

W3. Code

The PyTorch implementation of LoRA-Ensemble, along with a detailed README file, has already been included in the supplementary zip file.

W4. Post-hoc techniques

We have added a new section, Appendix E, to address this concern. We conducted analysis of the impact of Temperature Scaling (TS) as a post-hoc method for improving the calibration of the evaluated approaches. The results indicate that TS enhances calibration across all compared methods, except for the Single Net w/ LoRA. Importantly, while post-hoc methods like temperature scaling further improve calibration, the LoRA-Ensemble consistently outperforms all other compared methods, maintaining its superior performance in both diversity and uncertainty estimation.

W5. Missing Literature

Thank you for your suggestion. While we appreciate the importance of works like EDL and second-order UQ, incorporating them would be beyond the scope of this paper, as our focus is specifically on ensemble-based approaches. Including such methods would significantly broaden the scope, potentially leading to a loss of focus, as other uncertainty quantification techniques, such as Bayesian approaches, would also need to be considered.

Q2. OOD experiment

To address this concern, we added a new experiment. Please see Table 5 for results. We conducted an additional OOD experiment using CIFAR-100 as the in-distribution dataset and SVHN as the out-of-distribution dataset. Our method significantly outperforms all other methods. Notably, LoRA-Ensemble achieves approximately a 10% increase in both metrics compared to the closest baseline. In this experiment, the performance of Explicit Ensemble degrades compared to a single network. This finding aligns with the results reported for Split-Ensemble, where Explicit Ensemble performance also degrades under the same OOD scenario.

No rebuttal was provided by the authors and my concerns remain. I've decided to lower the score to 4 (weak reject).

We thank the reviewer for the detailed and constructive feedback. Below, we address all the concerns in detail.

W1 & W2. Theoretical Analysis and Degeneration of Ensemble Members

Providing a theoretical analysis of LoRA-Ensemble's relationship with the underlying probability distribution and its ability to prevent ensemble member degeneration is highly challenging. This difficulty arises from the complex interactions in high-dimensional parameter spaces, where pre-trained weights, LoRA updates, random initialization, optimization dynamics, and non-linear transformations collectively shape the learning process. Developing an analytical model would require strong assumptions about data, optimization trajectories, and architecture, which may not universally apply and would constitute a significant contribution to the field in itself.

Since such an analytical proof is beyond the scope of this work, we have instead added a new section, "Enhanced Diversity in LoRA-Ensemble," which empirically analyzes why LoRA-Ensemble outperforms methods like Explicit Ensemble. This section investigates the diversity of ensemble members in both function and weight spaces, using metrics such as pairwise disagreement rates, Jensen-Shannon divergence, and weight-space cosine similarity. The results demonstrate that LoRA-Ensemble achieves significantly greater diversity compared to Explicit Ensemble, which is crucial for constructing an effective ensemble.

Additionally, our spectral analysis of the weight space reveals that LoRA-Ensemble produces distinct parameter updates by exploring new high-ranking dimensions that are near orthogonal to the initial weights. In contrast, Explicit Ensemble members tend to adhere more closely to the structure of the initial weights. This enhanced diversity prevents LoRA-Ensemble from converging to a point estimate and enables it to explore the loss landscape more effectively, capturing a broader range of representations and leading to improved robustness and epistemic uncertainty estimation. We also provide visualizations of the training evolution of ensemble members in the loss landscape, demonstrating that LoRA-Ensemble members converge across a broader area, reflecting diverse learning dynamics.

W3. Large Scale Datasets.

To further strengthen our evaluation, we are currently conducting large-scale experiments on the iNaturalist dataset to demonstrate the scalability and robustness of our method on large, unbalanced, and challenging datasets. Due to computational constraints, the results are not yet complete, but our initial findings are promising, showing improvements over Explicit Ensemble baselines. We commit to including the full results for iNaturalist in the final version of the paper.

Q1.OOD Dataset.

To address this concern, we added a new experiment. Please see Table 5 for the results. We conducted an additional out-of-distribution (OOD) experiment using CIFAR-100 as the in-distribution dataset and SVHN as the out-of-distribution dataset, as this is a well-established benchmark in the literature. The results further emphasize the potential of LoRA-Ensemble for OOD tasks, as it significantly outperforms all other compared methods, including Explicit Ensembles and Split-Ensembles, in terms of AUROC and AUPRC. Notably, LoRA-Ensemble achieves approximately a 10% increase in both metrics compared to the closest baseline, Split-Ensemble. In this experiment, the performance of Explicit Ensemble degrades compared to a single network. This finding aligns with the results reported for Split-Ensemble, where the performance of Explicit Ensemble also degrades under the same OOD scenario. These results highlight the robustness of LoRA-Ensemble and demonstrate its superiority for OOD tasks.

We apologize for any confusion and would like to clarify that there may have been a misunderstanding regarding the rebuttal deadline. ICLR extended the discussion period by an additional 6 days. As a result, we did not submit our rebuttal by the original deadline, as we were focused on completing the revision of the paper. We kindly request that our response be considered for a fair evaluation of our work, given the extended rebuttal period.

We thank the reviewer for the detailed and constructive feedback. Below, we address all the concerns in detail.

W1.1 Additional Insights

We significantly extend our analysis in the newly added section, "Enhanced Diversity in LoRA-Ensemble," where we investigate the diversity of ensemble members in both function and weight spaces. Using metrics such as pairwise disagreement rates, Jensen-Shannon divergence, and weight-space cosine similarity, we show that LoRA-Ensemble achieves significantly greater diversity compared to Explicit Ensemble, which is crucial for constructing an effective ensemble. In light of our spectral analysis of the weight space, the results reveal that LoRA-Ensemble leads to very distinct parameter updates compared to Explicit Ensemble. LoRA-Ensemble explores new high-ranking dimensions that are near orthogonal to the initial weights, while Explicit Ensemble tends to adhere more closely to the structure of the initial weights. This enhanced diversity allows LoRA-Ensemble to better explore the loss landscape and capture a broader range of representations, leading to improved robustness and epistemic uncertainty estimation. Additionally, we provide visualizations of the training evolution of ensemble members in the loss landscape, demonstrating that LoRA-Ensemble members converge across a broader area, reflecting diverse learning dynamics.

W1.2 Combining ensembling and LoRA

Science often progresses through multiple groups independently exploring similar ideas, which highlights the significance and relevance of this research direction. Wang et al. work is a concurrent preprint that has not undergone peer review and focuses on fine-tuning LLMs on small text corpora with limited experimental evaluation and comparison. Additionally, their implementation details are unclear, and no code is available, making it difficult to fully assess their approach. In contrast, our work provides a much broader scope and deeper analysis. We conduct experiments across five datasets with diverse application domains, including real-world tasks where epistemic uncertainty is crucial, such as medical image analysis. We also evaluate our method on out-of-distribution scenarios, demonstrating its robustness and generalization. Finally, our work also investigates key factors such as initialization strategies and LoRA rank, which play a crucial role in the method’s effectiveness. By presenting a more extensive evaluation, covering diverse application domains, and introducing insights into the method’s design, our study significantly advances the understanding and applicability of LoRA-Ensemble.

W2. Small scale experiment.

To further strengthen our evaluation, we are currently conducting large-scale experiments on the iNaturalist dataset to demonstrate the scalability and robustness of our method on large, unbalanced, and challenging datasets. Due to computational constraints, the results are not yet complete, but our initial findings are promising, showing improvements over Explicit Ensemble baselines. We commit to including the full results for iNaturalist in the final version of the paper.

W3. Q1. More baselines

Thank you for this feedback. In response, we have added a new section in the Appendix (see Sec. A.4, Table 7) comparing a high-rank LoRA model with a low-rank LoRA model that have a similar number of trainable parameters. The results demonstrate that the low-rank LoRA model outperforms the high-rank counterpart, highlighting that the performance improvement is not solely due to the number of trainable parameters. This finding is in line with our new analysis and the analysis by Shuttleworth et al. (2024) [1], which show that as the rank of LoRA increases, its parameter update structure begins to resemble that of explicit ensemble models (or single model if #member=1), closely adhering to the structure of the initial weights. This similarity reduces the effectiveness of LoRA's unique low-rank adaptation capabilities. Additionally, we provide a new insight about this phenomenon in the Discussion section, further emphasizing its implications for LoRA's performance. We believe these additions provide deeper insights into the advantages of our approach and significantly strengthen the contributions of the paper.

References:

[1] Reece Shuttleworth, Jacob Andreas, Antonio Torralba, and Pratyusha Sharma. “Lora vs full fine-tuning: An illusion of equivalence.” arXiv preprint arXiv:2410.21228, 2024

Since there is no rebuttal and the reviewers are in consensus I maintain my initial assessment and score.