Promoting Ensemble Diversity with Interactive Bayesian Distributional Robustness for Fine-tuning Foundation Models

We would like to thanks reviewer KcS3 for their supportive review and feedback. We would like to address some question and concern as follow:

1. What is the limit on IBDR performance improvement as the number of particles increases? Thank you for the helpful suggestion. Following your comment, we conducted additional experiments using a greater number of particles. The results are reported in the table below for your convenience. As shown, the performance of our method generally improves as the number of particles increases. Notably, we observe a slight performance gain when using 12 particles. However, this appears to be the upper limit of improvement—using 16 particles yields comparable or even slightly lower performance than using 12 or 8 particles. | Accuracy | Camelyon | EuroSAT | Resics45 | Retinopathy | |-|-|-|-|-| | 1p | 82.4 | 93.1 | 84.2 | 73.8 | | 2p | 84.8 | 93.9 | 86.6 | 74.3 | | 4p | 85.1 | 95.0 | 87.3 | 76.5 | | 8p | 85.8 | 95.5 | 87.4 | 77.0 | | 12p | 86.1 | 95.3 | 87.6 | 77.3 | | 16p | 85.9 | 95.0 | 87.6 | 77.2 |
2. The accuracy-runtime tradeoff when choosing between IBDR and other baseline approaches.

We thank the reviewer for the insightful suggestion. We have already conducted experiments comparing the runtime of our method with several baseline approaches. As shown in the table below, our method has a slightly higher runtime than some baselines such as SA-BNN and SVGD. Deep Ensemble, being the simplest among the compared methods, achieves the fastest runtime—this aligns with its relatively lower accuracy, as reported in Table 1 in our paper. These results highlight the performance-runtime trade-off among different methods, as the reviewer suggested. We will extend our runtime analysis to include all datasets and baselines, and incorporate the findings into the revised version.

3. Regarding some typos in our paper

• Typo in running title and in Section title: “B.2. Data augmentations”: Thank you for the helpful suggestions. We will correct the typo in the running title by removing the extraneous "s". Additionally, we will capitalize “Augmentations” in Section B.2 to ensure consistency with the formatting of other appendix section titles.

We thank the reviewer for your feedback. We provide detailed responses to the main concerns as follows:

1. l195: 'conventional Bayesian frameworks' can't enforce diversity

• Indeed, in Section 3.3, we introduce what we meant by traditional Bayesian framework. Given a training set $S$, we have the closed from for the posterior $p(\theta|\mathcal{S})$. To handle this posterior, variational approach aims to learn an approximate posterior $q(\theta)$ using the ELBO. At the end, we sample $K$ models $\theta_1,\dots,\theta_K$ from $q(\theta)$ for ensemble the predictions. Certainly, this framework does not allow the particle interaction.
• Our main contribution is the joint distribution $Q^K$, which models interactions among particles and enables distributional robustness. This links to sharpness-aware minimization and allows flexible definitions of interaction (e.g., prediction divergence), unlike prior work like SVGD, wherein particle diversity is primarily enforced through similarity kernels over weight vectors.

2. The method is evaluated only for the subtask of fine-tuning ...

• A key limitation of ensemble-based methods is the need to store and train multiple models, which becomes computationally expensive when dealing with large-scale models. Besides, in the era of model finetuning, where fine-tuning pre-trained models is a dominant paradigm, our method becomes particularly advantageous. It mitigates the primary drawbacks of traditional ensemble approaches, by requiring only the storage of multiple lightweight adapters rather than full models.
• Following your suggestions, we have also conducted additional experiments involving training from scratch on CIFAR100 as follows: |Accuracy|Resnet18|Resnet34| |-|-|-| |Standard Training|76.29|77.31| |IBDR (4 particles)|77.45|77.92|

3. ... why (5) is still supposed to be a valid upper bound as it contains a multitude of changes

• Eq (5) is a relaxation of the one in Corolarry 4.3 that keeps its spirit. Particularly, for the one in Corolarry 4.3, given $\theta_{1:K}\overset{iid}{\sim}Q$, we need to find out the perturbation $\theta_{1:K}^{'}$ all in once that maximizes the loss while staying closely to $\theta_{1:K}$ via minimizing $\frac{\lambda}{K}\sum_{k=1}^{K}c(\theta_{k},\theta_{k}^{'})$. In the relaxation version, we isolate $\theta_k$ and $\theta_k^{'}$ in the sense that we find the perturbation $\theta_{k}^{'}$ around $\theta_k$ by maximizing the loss and minimizing the distance $c(\theta_k^{'}, \theta_k)$, while replacing $\theta_{-i}^{'}$ by the fixed $\theta_{-i}$. This is tolerable because $\theta_{-i}^{'}$ stays closely to $\theta_{-i}$. Besides being easier to implement, this relaxation still preserves the spirit of the original one.

4. The theory depends on the fact that the loss function has to be bounded...but this constraint is dropped completely during application...

• We do not break or violate the bounded loss constraint in our application. In particular, we mainly focus on classification tasks with the CE loss, whose formulation can be simplified as $-\log p_y$ where $y$ is the ground-truth label of $x$ and $p_y$ is the model prediction. We acknowledge that $-\log p_y$ tends to infinity if and only if $p_y$ tends to 0. This is impossible due to 2 reasons:
  • At the beginning of training, the model lacks prior knowledge and typically makes near-uniform predictions. This causes the predicted probability $p$ to fluctuate around $1/M$, where $M$ is the number of classes, rather than approaching zero.
  • During training, the objective is to increase the predicted probability for the correct class (maximize $p_y$). Therefore, the training process inherently pushes $p_y$ closer to 1, not 0.
• In addition, we notice that in the SAM paper, the McAllester PAC-Bayes bound was used to develop its theory, which is only applicable to the 0-1 loss. In this paper, we leverage with more advanced PAC-Bayes theorem, leading to a less restrictive family of loss functions.

5. .., paper's discussion of the field is mostly limited to varying MCMC methods. Other Bayesian approaches,... is completely omitted

• We will definitely discuss [D'Angelo & Fortuin, 2021] in the revised version. However, we notice that our approach is fundamentally different from these Stein approaches.

6. The LoRA paper reference:

• Thanks for your comment. This is indeed an error in our bibtex. We will definitely fix them in the revised version.

7. l370: "Table 1, IBDR outperforms all baselines by large margins..."

• We will change the wording. Indeed, Table 1 includes a column labeled AVG, and our method outperforms all other baselines by more than 2% in terms of average accuracy.

8. Why are the baselines between the two experimental setups different

• We chose baselines closely related to our method for image classification, and followed BLoB’s setup for commonsense reasoning, including comparisons with popular Bayesian methods in LLMs.

We appreciate the reviewer’s comments and respond to the key concerns as follows:

"While IBDR is evaluated on multiple datasets, computational overhead is not thoroughly analyzed. Given that the method involves interactive Bayesian sampling, a study on efficiency would be valuable."

Thank you for the insightful suggestion. We would like to point out that an analysis of performance and runtime with varying numbers of particles is already provided in Appendix A.1. For your convenience, we report these corresponding tables below.

We thank the reviewer for the constructive feedback and would like to address the concerns as follows:

1. Regarding the Theoretical Claims:

• Regarding the citation of prior work in the proof of Theorem 4.1 : Thank you for the suggestion. Our proof relies on Theorem 2.1 from the cited study. We will clarify this in the revision to better guide readers.
• Regarding the $\beta$ value in our proof: We appreciate the reviewer for poiting out our typos. In line 858, ${1}/{\delta}$ should be $\log\frac{1}{\delta}$. In line 878, the correct value $\beta$ should include a nested square root over the divergence and $\log\frac{1}{\delta}$, as reviewer suggested. Also, “equation (6)” in line 878 should refer to line 858, which is equation (7). We carefully double-check our proof and except these typo, our proof remains true. Specifically, the RHS of this equation (7) actually becomes $\sqrt{\frac{\text{D}_\text{KL}\left(Q^{K} \| P^{K}\right)+\log\frac{1}{\delta}}{2N}}\times L$ with the correct value of $\beta$ above.

2. Regarding our claim about promoting diversity

• Conventional Bayesian frameworks lack explicit mechanisms to model interactions between particles θ1:K during training: Indeed, in Section 3.3, we introduce what we meant by traditional Bayesian framework. Given a training set $S$, we have the closed from for the posterior $p(\theta|\mathcal{S})$. To handle this posterior, variational approach aims to learn an approximate posterior $q(\theta)$ using the ELBO. At the end, we sample $K$ models $\theta_1,\dots,\theta_K$ from $q(\theta)$ for ensemble the predictions. Certainly, this framework does not allow the particle interaction.
• The novelty of our promoting diversity mechanism: Our main contribution is the joint distribution $Q^K$, which models interactions among particles and enables distributional robustness. This links to sharpness-aware minimization and allows flexible definitions of interaction (e.g., prediction divergence), unlike prior work like SVGD, wherein particle diversity is primarily enforced through similarity kernels over weight vectors.

3. Comparison with nonlinear Stein VI approach:

• Thanks for pointing out us the nonlinear Stein paper. This is an interesting paper that extends Stein Variational Gradient Descend (SVGD). These two share the same form of the objective function $\max_{\rho}{ \left(F(\rho)+H(\rho)\right)}$ where $H\left(\rho\right)=-\int\log\rho d\rho$ is the entropy. As pointed out in Eq.(8) in Theorem 1 of the non-linear Stein paper, the second term in this equation relevant to $\nabla_{\theta}k\left(\theta,.\right)$ is known as the repulsive term derived from maximizing the entropy $H\left(\rho\right)$. This term encourages the particles to spread out for avoiding the mode collapse.
• Differently, our proposed approach enables the model interaction through $l_{div}\left(\theta_{1:K},x,y\right)$. Interestingly, we can define this term appropriately to encourage various kinds of diversity (e.g., diversity in the model span or diversity in model predictions). Evidently, in our practical method, motivated by the theory of Determinantal Point Processes, we propose a diversity loss to encourage the diversity in model predictions that is proven to improve the ensemble performance.
• Finally, we will cite and discuss the non-linear Stein paper.

4. Typographical comments: We thank the reviewer for catching these typos. We will correct these typos in the revised manuscript.
5. Comparison results with IVON LoRA: As reported in Section 5.2 of our paper, we conduct experiments on CommonSense Reasoning, which are also used in IVON-LoRA’s study. For the reviewer’s convenience, we will present our results alongside theirs as follows:

6. Other Reviewer's Question

• The meaning of "non-maximal prediction probabilities"?: These are the predicted class probabilities excluding the one assigned to the ground-truth class. For example, if the true class is $y_3$ and the original model prediction probability is $[p_1, p_2, p_3, p_4]$ for classes classes $y_1, y_2, y_3, y_4$, then the non-maximal prediction is $[p_1, p_2, p_4]$
• Regarding question about entropy-based regularizer: In our proposed approach, we use the diversity loss $l_{div}\left(\theta_{1:K},x,y\right)$ to encourage the diversity of model predictions instead of the model weight diversity from the entropy maximization.