Thank you for your insightful review. We hope our response below addresses any remaining concerns. If you have any further questions, please let us know. Otherwise, we kindly request that you reconsider your assessment accordingly. Thank you again for your valuable feedback!

In line 174, what does $w_{\text{MAP}}$ mean? Is it a typo for $w_{\text{MAP}}^\ast$?

Apologies for the confusion caused by the typos. We have identified that asterisks were missing in lines 174 and 232. Thank you for your meticulous review!

In line 179, the paper concludes that "it encountered difficulties in accurately estimating the diagonal covariance matrix." However, this conclusion is not clearly supported by the results presented in Figure 2. Please provide more evidence or clarification for this statement.

We initially concluded this because the SWAG-Diag baseline seemed ineffective at performing BMA. Without knowing the true posterior, we assessed the quality of the approximate Gaussian posterior through MAP and BMA performance. Good MAP performance suggested the mean was acceptable, but poor BMA indicated a problem with the variance estimate. However, after conducting additional experiments, we improved the SWAG baseline results, as detailed in the General Response. This resolves the issue with the SWAG baseline, which will be updated in the camera-ready version. Thank you for raising concerns about the SWAG results.

The paper relies on existing techniques for both ensembling and quantization. It primarily presents an empirical study on the combination of these two approaches rather than introducing a new method.

Our main contribution is offering a fresh perspective by using quantization errors--often seen as challenges in quantization research--as a source of diversity in ensemble methods. Consequently, although each individual approach (quantization and ensemble) may not be novel by itself, we believe showcasing the potential of low-precision ensembles for modern large-scale models represents a novel and significant contribution.

The practical utility of the proposed LPE-BSR method is somewhat constrained. Large models typically incur high inference costs, and ensembling them—even with low precision systems—can amplify these costs, particularly with ensemble sizes of 10 or 20, as discussed in the paper. Furthermore, according to the experimental results in Table 3, the performance improvement of the quantized ensemble over the full precision model is marginal. As an empirical study, the authors should better justify why ensembling in low precision number systems is a meaningful objective.

As you noted, the practicality of ensembling large models is constrained, which we believe explains the decreased preference for ensemble methods in the current era of large models. Our research addresses this by demonstrating that scalability issues with ensemble techniques can potentially be mitigated through the use of low precision number systems, particularly for modern large models. In doing so, we aim to rekindle interest in ensemble methods within the context of current large model trends, as mentioned in lines 289-293 of the conclusion section. We hope that our work on LPE-BSR will be seen as a pioneering effort in showcasing the potential of low precision ensembles.

Besides, following Reviewer 9d5B's recommendation, we have extended the memory budgets plot from Section 4.4 to include the experiments from Sections 4.2 and 4.5. The figures in the attached PDF demonstrate that our LPE-BSR approach is located in the lower-left corner, signifying that it achieves lower NLL while being more memory-efficient. Even with a smaller ensemble size of five, it outperforms both the BMA baselines and the original checkpoint.

The related work section does not provide enough context or comparison with existing research. It would be beneficial to include a more detailed literature review.

Thank you for your constructive comment, which strengthens our paper! We have reviewed relevant studies on low precision ensembling, especially those involving Bayesina methods with quantization-aware training, but have struggled to find further research on this topic. Any recommendations for further literature would be immensely helpful. Meanwhile, we will revise the paper to address your feedback by expanding the related work section to include additional paragraphs on ensemble methods and quantization.

The subfigures in Figures 3, 4, and 6 are difficult to interpret. Adding captions or labels for each subfigure would improve clarity.

Even though lines 210-216 of the main text explain the radial landscape plots, it appears that Reviewer bKfp also expressed similar concerns about Figure 3. Including captions and labels for each figure, as you recommended, should improve readability in the camera-ready version. We appreciate your constructive feedback!

I appreciate the authors' prompt response. As mentioned, the LPE-BSR (INT-5) ensemble model does outperform a single FP32 model in terms of both memory footprint and performance. However, if we were to simply adopt a model with more parameters and better performance, applying standard quantization, it might lead to a single dense model with similar memory requirements while performing better than the ensemble. Additionally, dense models typically benefit from a higher level of parallelism in practice compared to an ensemble approach. Therefore, the practical effectiveness of the proposed method is not sufficiently convincing for me. Given this, I will maintain my initial score.

Thank you for recognizing the potential and interest in our idea. We appreciate your detailed and constructive feedback. We have addressed your comments below and are confident that incorporating these revisions into the final version will significantly strengthen our work. If you have any additional concerns, please let us know. Otherwise, we kindly ask that you revise your assessment accordingly. Thank you again for your valuable feedback!

Another possible baseline to compare with is to build an ensemble by adding Gaussian noise with a fixed variance for all values tuned (for ensembling) on a validation set. Although very simple, that approach can be quite effective.

We recognized that the mentioned baseline offers a simple approach for training-free ensemble construction. As a result, we included additional comparative experiments with the Gaussian baseline in the General Response, along with the MCDropout baseline mentioned by another reviewer. The results validate that LPE-BSR outperforms these baselines. Thank you for your constructive suggestion!

Compare with others without adding its contribution. (...)

Our LPE-BSR method forms a low precision ensemble from a given checkpoint in a training-free manner, making it dependent on existing checkpoint. In our paper, we conducted a comparative study using fine-tuned checkpoints from SWAG, IVON, and SSE. Therefore, we respectfully disagree with the claim that SGD plus LPE-BSR should be compared to the MAP of IVON and SWAG; both IVON and SWAG are simply optimizers with additional features that provide an approximate Gaussian posterior. Specifically, SWAG is an enhanced version of SGD with Polyak-Ruppert averaging (cf. Izmailov et al., 2018), and IVON is a variation of Adam interpreted through variational learning (cf. Shen et al., 2024). As a result, it is entirely possible for the most basic optimizer, SGD without momentum (NLL = .492), to perform worse than the more advanced optimizers, SWAG and IVON (NLL = .488 and .489), and there is no reason why LPE-BSR should be limited to use with SGD. Importantly, LPE-BSR consistently improves performance over the given checkpoint (MAP), with improvements even surpassing those seen with BMA in IVON and SWAG ($\Delta$):.

We have further included results using Adam as well as SGD in the table for readers who may not be familiar with the SWAG and IVON optimizers. Given that LPE-BSR generates low precision ensemble members around the provided checkpoint (MAP), it is anticipated that LPE-BSR will perform better with a more advanced optimizer, such as Adam with an NLL of 0.487. Again, the main point is that LPE-BSR reliably produces a low precision ensemble that exceeds the performance of the original FP32 solution without additional training, achieving quality on par with leading Bayesian methods like SWAG and IVON.

Compare with Deep ensembling. (...)

Following the suggestions from Reviewers yx79 and bKfp, we compared LPE-BSR with deep ensembles (DE) and batch ensembles (BE). While LPE-BSR does not involve fine-tuning, these comparisons are valuable, similar to those with the BMA approach. Due to character limits, please see our response to Reviewer yx79. These comparisons show that LPE-BSR is a memory-efficient method achieving performance levels bounded by DE, similar to BE, but without requiring fine-tuning. Thank you for the constructive feedback!

Weight averaging. (...)

In short, averaging the weights of LPE-BSR ensemble members results in the original central FP32 weights, similar to how averaging samples in SWAG and IVON results in the original Gaussian mean. Due to space constraints, please refer to our response regarding Model Soups to Reviewer 9d5B for more details.

Figure 3, what is the difference among the three subfigures?

Details about radial basis plots are provided in lines 210-216. In short, the first subplot illustrates where each model is positioned within the loss landscape, while the next two subplots show how much the models are diverse from one another. Reviewer svQu also raised similar concerns regarding the clarity of the radial basis plots. Consequently, we intend to enhance the captions and labels in the camera-ready version. We appreciate your feedback on this matter.

Table 2, why not showing all configs for SGD?

Thank you for bringing that to our attention. We initially removed the content to save vertical space and did not reinclude it. The missing results for SGD presented below, which show trends consistent with other results, will be added to the camera-ready version. We appreciate your thorough feedback!

Thank you for recognizing the significance of the problem we addressed, the clarity of our writing, the technical robustness of our proposed method, and the effectiveness of our experimental results. We have addressed your comments below and believe that incorporating these revisions into the final version will significantly enhance the contribution of our work. If you have any further concerns, please let us know. Otherwise, we kindly ask that you adjust your assessment accordingly. We appreciate your valuable feedback once again!

Firstly, I think the paper is lacking some novelty. While interesting, it is not very surprising to me that an ensemble of randomly sampled low precision models can improve the generalization performance and uncertainty calibration. There have been numerous works of similar flavor.

We respectfully disagree with the novelty claim and would like to quote Michael Black’s remark: "If it is easy to explain and obvious in hindsight, this in no way diminishes the creativity (and novelty) of the idea." To our knowledge, the concept of employing quantization erros to obtain ensemble diversity has not been widely explored. Could you please provide specific examples of the "numerous works of similar flavor" you mentioned?

I think the paper lacks some important benchmark comparisons. For instance, how does the proposed method compare to other sampling based ensembling approaches like dropout ensemble [1] and batchnorm ensemble [2]?

Thank you for your insightful comment! Both MCDropout (Gal and Ghahramani, 2016) and MCBatchNorm (Teye et al., 2018) leverage dropout and batch normalization techniques--typically used during training--at the inference stage to build ensembles. These methods indeed enable training-free ensemble construction, making them valuable for comparison with our proposed method. However, since modern transformer architectures typically do not include batch normalization layers, we concentrated our additional experiments on MCDropout. MCDropout is particularly relevant as it uses a $q(w)$ form similar to Eq. (5) of LPE-BSR, employing $\delta(0)$ and $\delta(w)$. The General Response provides additional comparative results with the MCDropout baseline and the Gaussian baseline mentioned by another reviewer. The results confirm that LPE-BSR outperforms both of these baselines. Thanks again for your constructive suggestion!

How does the diversity of low precision ensembles compare with other popular ensembling techniques like deep ensemble and batch ensemble?

While our LPE-BSR method does not involve fine-tuning (though we do include fine-tuning experiments to compare LPE-BSR’s ensemble quality with Bayesian methods in our paper), a comparative study with other ensemble techniques like deep ensembles (DE) and batch ensembles (BE), as you suggested, would be valuable. Below are the results using the Adam optimizer; the number of ensemble members for each method is indicated in parentheses; the metrics (a), (b), and (c) correspond to those discussed in the diversity analysis of our paper; (a) average loss, (b) ambiguity (i.e., ensemble diversity), and (c) ensemble loss.

In our experiments using the Adam optimizer, we found that in LPE-BSR, (a) each ensemble member had relatively lower performance (NLL = 0.513). However, (b) due to high ensemble diversity (ambiguity ≥ 0.025), (c) there was a significant improvement in the final ensemble performance. Consequently, it achieves performance comparable to another memory-efficient method available in fine-tuning scenarios, BE. In BE, ensemble members are similarly centered around one solution, with BE members derived from shared weights by multiplying rank-one matrices, and LPE-BSR members derived from the center weights using stochastic rounding. This comparison with BE, a well-known memory-efficient ensembling strategy, highlights the potential of low precision ensembling with LPE-BSR.

Furthermore, it is worth noting that LPE-BSR forms low precision ensembles without requiring additional training on any given checkpoint. This suggests that, similar to the enhancements shown in fast ensembling experiments, deep ensembles can also achieve further improvements with LPE-BSR. Figure 12 in the attached PDF illustrates this point clearly.

It would also be interesting to benchmark the performance of the proposed method in terms of computation cost (in addition to the memory budgets shown in Figure 5).

We understand that you are addressing computation cost in terms of time complexity, such as latency, rather than space complexity like memory budgets. Unfortunately, quantized models typically have slower inference speeds without specialized hardware and software. As we do not have access to such hardware, we employed fake quantization for research purposes, which makes direct benchmarking challenging. However, since our work is based on the standard symmetric uniform quantization scheme, it aligns with current and future advancements in neural network quantization. We also want to highlight that our contribution, which showcases the potential of low-precision ensembling in large modern models, remains significant and relevant. We will include this point in the limitations section.

We are pleased with the positive feedback that highlights our work as both inspired and interesting. We hope the responses provided below address any remaining concerns. Please let us know if there are any further issues. Thank you for your valuable comments!

The paper presents NLL comparisons with SWAG-Diag and IVON. It would be great to see accuracy numbers as well to understand the actual performance differences as all the results are presented on classification tasks.

The ERR values presented in our paper represent classification error, computed as one minus the classification accuracy (i.e., ERR = 1.0 - ACC). We opted to report classification error instead of classification accuracy to ensure that our evaluation metrics (i.e., NLL, ERR, ECE) align with the “lower is better.” However, if using ACC enhances readability, we are willing to make that change in the final camera-ready version.

In addition, Table. 1 should have error estimates for LPE-BSR given that it relies on randomness.

We empirically checked that the variability in LPE-BSR performance due to randomness is minimal (≤ 0.002). This is likely due to our experimental setup utilizing the same pre-trained model. We also agree that error bars should be included whenever possible, regardless of how small they are, and the camera-ready version will include results with error bars in the appendix. Thanks for the constructive feedback!

Also given that LPE-BSR is not really a Bayesian method, it would be great to also compare with other ensembling approaches like Model Soups to see where it lands on the pareto curve of compute-vs-accuracy.

Model Soups (Wortsman et al., 2022) involves averaging the weights of models fine-tuned with different hyperparameters from the same pre-trained model. Thus, it is distinct from our LPE-BSR method, which does not involve any fine-tuning. Model Soups boosts the performance of the final fine-tuned checkpoint by averaging multiple fine-tuned checkpoints, whereas LPE-BSR works directly on a single provided checkpoint. Consequently, our LPE-BSR method can also be applied to a checkpoint obtained through Soup. Given that LPE-BSR generates low precision ensemble members around the provided checkpoint (such as MAP or Soup here), it is anticipated that LPE-BSR will achieve better performance with a more advanced checkpoint, like one from Soup with an NLL of .477. Notably, LPE-BSR consistently improves performance whether the given checkpoint comes from MAP or Soup, showcasing the flexibility of the proposed low precision ensembling method.

In addition, while weight averaging is straightforward with standard FP32 weights, it becomes more complex with low precision number systems because the averaged points do not align with the low precision system grid. More importantly, regardless of the low precision grid, averaging the weights of LPE-BSR ensemble members results in the original central FP32 weights, similar to how averaging samples in SWAG and IVON results in the original Gaussian mean. We will clarify this point further in the final version of our paper.

Also, the SWAG algorithm shows significant performance differences between SWAG, and SWAG-diag. While it may not be possible to compare for all the presented models, an exemplar comparison would also help support the conclusions presented in the paper.

Fig. 2 shows that SWAG does not improve with increasing ensembl sizes. This is a bit counter-intuitive. I may have misunderstood the figure, and invite some clarification if so.

We are happy to share the improved SWAG baseline results in the General Response. We hope it resolves your concerns and appreciate your constructive feedback.

Are the LLAMA results presented on MMLU 5-shot or 0-shot?

The MMLU results pertain to the 0-shot case, and we realized this detail is only mentioned in the caption of Figure 7. We will update Appendix A to include this information. Thanks for bringing it to our attention!

It would also be great to see some additional analysis on the actual compute savings incurred using LPE-BSR.

As advised, we have expanded the memory budgets plot from Section 4.4 to cover the experiments in Sections 4.2 and 4.5. Please refer to the figures in the attached PDF, which clearly illustrate the compute savings achieved with LPE-BSR, for further details. Thank you for the suggestion!

Thank you for your additional experiments. Could you please provide more details about the hyper-parameters of MCDropout and Gaussian noise used to build the ensemble? Which hyper-parameters did you consider? Which values? Did you select the optimal hyperparameters? Did you put as much effort in finding the optimal hyper-parameters as in your method? Thanks