BARNN: A Bayesian Autoregressive and Recurrent Neural Network

We appreciate the reviewer for their time and valuable feedback. In particular, we are delighted that the reviewer believes our work is of pragmatic interest due to its broad applicability in uncertainty-aware sequence modeling, and that their appreciate the experiment section. We answer the questions raised below point by point:

Weak claim(promise): autoregressive models are also prone to overfitting to the specific tasks they are trained on, challenging their application outside their training domain. -> Not explicitly demonstrated

• We thank the reviewer for analysing in depth our results section, we appreciate the time and effort. We agree that we did not stress in our experiment analysis autoregressive model overfitting. However, we want to highlight that our Table 2 results show the overfitting behaviour of autoregressive models. In particular, SMILES LSTM (it has dropout on the recurrent layers) and BARNN can be thought of as regularized versions of the vanilla LSTM. The latter, compared to the regularised models, is more prone to reproduce molecules already seen during training, shown by lower novelty and uniqueness, which is an indication of overfitting to the training data distribution.

Essential References Not Discussed: I think literature [1] , which also introduces a variational Bayesian scheme for RNNs, is highly relevant and should be discussed.

• We thank the reviewer for pointing out the methodology to us, which shares the similarity of modelling uncertainties in RNNs, and we will add in section 2. However, differently from our methodology, the approach is only suited to RNNs, while our approach can be applied to various autoregressive or recurrent models. Furthermore, but most importantly, the model in [1] can hardly scale to large networks mainly due to two reasons (i) weights are sampled directly, in contrast we use the local-reparametrization allowing scaling to large networks (ii) the number of parameters doubles because they model mean and standard deviation directly, while we use a variance proportional to the mean.

Weaknesses: Hard to position in context: The incorporation of Bayesian methods into RNNs has a long history, with key prior work (see "Essential References Not Discussed") that is not acknowledged or discussed.

• See above in “Essential References Not Discussed” answer.

Weaknesses: Combined with the presentation style, this gives the impression of overclaiming novelty, as if applying Bayesian inference to RNNs is itself a novel contribution.

• We thank you for the comment, and we do not want to claim that applying Bayesian inference to RNNs is our novel contribution. However, to the best of our knowledge, we are the first approach that can transform any autoregressive or recurrent model into a Bayesian one with minimal modification, obtaining similar or superior performances to non-Bayesian counterparts, scale to large networks and at the same time provide UQ.

Weaknesses: Ablation study of the prior: The tVAMP prior is a key contribution, yet the paper does not include an ablation study comparing it to a simpler isotropic Gaussian prior.

• We thank the reviewer for the suggestion, which is shared with reviewer 4bFX. We conducted a test on a synthetic time-series dataset to show how tVAMP excels over standard log uniform prior and fixed dropout rates. A simple isotropic Gaussian would not lead to a KL which is independent on the likelihood model parameters, therefore losing the regularization property of VD. Please see in reviwer's 4bFX answer the result table and explanation (not inserted here for words limit).

Weaknesses: Motivation for using RNNs over stronger sequence models: While this may not be a direct weakness, it is unclear why the paper focuses on Bayesian RNNs when stronger sequence models (e.g., S4) exist.

• We thank the reviewer for the comment, also suggesting a possible application to BARNN for SSMs. We wanted to focus on RNNs because the model in eq. (2) (3) is heavily recurrent. Of course, extending to SSMs and Transformers models would be interesting, but a specific ancoder $E$ is needed, and further research needs to be conducted. In summary, with the paper, we wanted to show a general methodology for Bayesian sequence modelling, hoping to inspire potential applications to future work.

Eq.~8: should it be $E_{\phi}(\mathbf{y}_{0:t-1}^k)$?

• Yes, we write it more compactly as $\alpha^l_t(\mathbf{y}_{0:t-1}^k)$, we will add a note on the paper, thank you for spotting it!

Intuition Behind Long-Term Dependency Claims

• We thank you for the interesting question, and we claim that this is due to the time-adaptivity of the model weights. Indeed, both SMILES LSTM (with Dropout) and standard LSTM obtain similar performances, worse than BARNN, which only has temporal adaptivity differently from them.

Practical Setting of $N$ in Eq.~(8)

• Correct! $N$ is set to the batch size. We will update the paper to be clearer.

We appreciate the reviewer's time and effort in making constructive comments and providing us with additional references. We are glad that the reviewer finds our experiments nice and acknowledges that the choices of applications are real-world driven. We answer below the questions raised by the reviewer, and we also ask if there are other concerns about the paper which not lead to a higher score in the initial review round; in that case, we will be glad to discuss.

For Eq.(6), why is the normal distribution assumed to have variance being the square of its mean?

• Thank you for pointing out this crucial step in our methodology. We choose to model the variance as the square of the mean, as also done in vanilla Variational Dropout, for efficiency proposes. This parametrization allows us to not use extra model parameters, permitting scaling to very large neural networks. Also, this approach allows us not to change the original (non-Bayesian) model and add the BARNN methodology on top of it, which, as 4bFX highlighted, is “easy to implement”

I do have a question regarding he confidence intervals presented in Figure 2. It is said in the paper that the presented CIs are 99.7%, which are still very narrowly concentrating near the mean prediction/posterior predictive mean. To me, it appears that the variational inference approach might have signficantly underestimated the uncertainty.

• We thank the reviewer for the comment on the CIs. We agree with the reviewer that in the picture, the confidence interval seems concentrated near the mean prediction/posterior predictive mean. However, this is mainly a plotting issue because we used a very thick line width to have three nice displayed figures side by side. To convince you, please look at the ECE and NLL in Table 1. Underestimating the variance (as in the case of Refiner, for example) would lead to a very high ECE and exploding NLL (which is not our case).

The design of the factorization format of variational posterior in Eq.(3) reminds of the line of work on modular Bayes inference, where we deliberately cut off certain dependencies between parameter blocks and parts of observations, in order to improve computational efficiency / reduce potential influence of model misspecification due to incorrect inclusion of certain dependencies.

• We thank the reviewer for pointing out this reference, which we believe is relevant. Indeed, we cut off non-causal dependencies in eq (3) (i.e. the current weights are only dependent on the past). We will look more in depth into modular bayes inference and add a connection to eq. (3) in the final manuscript. If there are other concerns or suggestion, we are happy to discuss.

We thank the reviewer for their time and comments, and we appreciate that the reviewer finds the paper well written with the experiments sufficient to demonstrate the validity of the model. We address the reviewer's comments point by point below in the hope of clarifying some misunderstandings, improving the paper and eventually raise their score:

Some work attempts to probabilistic model attention in order to obtain good uncertainty estimation ability

• We thank the reviewer for the references; we find Bayesian Attention Modules very relevant and will cite them in the related works. A key difference between Bayesian Attention Modules and our methodology is that our method explicitly models uncertainties in time using a joint state-weight model and applies to any autoregressive or recurrent model, while Bayesian Attention Modules are a technology specifically tailored to Transformers models, and the Bayesian weights are static.

Weakness The proposed method can be seen as an application of variational dropout to an autoregressive model but does not provide any new insight.

• We thank the reviewer for their time in reading the paper and finding parallelisms with Variational Dropout (VD). We understand that our method shares similarities with VD, from which we inherit the scalability to large networks. However, in variational dropout methods (i) the weights do not evolve in time, and (ii) the prior is empirical, while we provide a time-dependent new prior, which is the best for the objective in eq. (4). As reviewer 4bFX highlighted, the derived prior (tVAMP) and time-dependent dropout mechanics feel both novel and easy to implement, and the reviewer WqtE believes that our methodology is of pragmatic interest due to its broad applicability in uncertainty-aware sequence modeling. Finally, we also found that static weights lead to less accurate uncertainty metrics and adaptivity. For example, see Table 1, where ARD Dropout or Dropout shows lower accuracy in terms of NLL and ECE compared to our methodology.

Autoregressive models based on attention mechanisms are not discussed.

• We thank the reviewer for pointing out attention-based models, which we agree we did not mention exhaustively. We tried to cover a wide variety of references on autoregressive models, not focusing only on a specific model. Nevertheless, we did mention the Vaswani, A. et al. Attention Is All You Need paper for transformer reference, and the Radford, A., et al. Language Models are Unsupervised Multitask Learners for moder decoder-only language models. If the reviewer thinks we missed other relevant papers which will improve the manuscript, we will be happy to add them if pointed to.

Questions For Authors: Is there any difference between probabilistic modeling of autoregressive models and normal neural network models, like convolutional neural networks?

• We thank the reviewer for the very interesting question. We did find that standard techniques used to model uncertainties in non-autoregressive models tend to obtain less accurate uncertainty metrics when applied to autoregressive solvers (see, for instance, Table 1. where Ensemble Dropout does not obtain as good performances as our methodology). This is mainly due to error accumulation and long-term dependencies, which non-autoregressive models do not need to deal with.

Why not use Transformer, a more general and powerful autoaggressive neural network?

• We thank the reviewer for the question, which we are glad to discuss. In our experiment, we focused in the AI4Science field since we believe uncertainty quantification will be key there. In the AI4Science field, we choose PDEs and Molecule generation as tasks since they are very different and use different models. For PDEs, autoregressive models are mostly based on multiple-step predictions using a Neural Operator. For molecule generation, RNNs were first applied, and just more recently, Transformers have been used, and it is not clear that transformers are more powerful in this specific task. In summary, with the paper, we wanted to show a general methodology for Bayesian sequence modelling, hoping to inspire potential applications to future work given the simplicity to implement and the generality of the methodology.

Why not experiment on text to better illustrate the effectiveness of the model on large data sets?

• We agree with the reviewer that the NLP task is an interesting avenue of research. Even though not an NLP task, we did employ text for the molecule generation task where the SMILES syntax is learned by the model to generate new molecules. For a more specific NLP task (e.g. language translation), a complication for us was how to define uncertainty properly, which seems to be an ongoing open research. Indeed, we choose our experiment tailored to support each claim in the paper, as reviewer WqtE also highlighted (all the claim are supported by clear and convincing evidence).

We are delighted that the reviewer finds our approach elegant and general, allowing flexible application across domains and the tVAMP prior and time-dependent dropout mechanics novel and easy to implement. We appreciate that the reviewer considers that our claims are generally supported by well-chosen experiments. Finally, we thank the reviewer for the detailed comments and suggestions, which we address point by point below:

From a high-level view, the presented proofs appear logically consistent. However, a more detailed or step-by-step check of the proofs would help confirm there are no hidden assumptions that might be restrictive.

• We think the reviewer understood correctly the core of the proofs. For the temporal variational lower bound proof (Appendix A.1), given the specifics of the generative model distribution and the posterior distribution, no assumptions (e.g. independence or Markovian-memoryless properties) are made for deriving the lower bound. For the temporal VAMP prior proof (Appendix A.2), the approximation comes in eq. (19) while the general formulation of eq. (18) does not make any assumption.

Weaknesses: The method’s computational overhead for extremely large models (like multi-billion-parameter LLMs) is not fully explored

• We acknowledge the fact that we did not test, due to computational resources, the methodology on large-scale models. However, we want to highlight that our methodology can scale to large models due to the decoupling between posterior and main model weights. Indeed, the temporal dropout coefficients are obtained from a posterior model with shared weights, which contains order of magnitude lower number of parameters, compared to the main network which is left untouched in the parameter size.

Weaknesses: Real-world PDEs often involve high-dimensional grids (2D, 3D); the current experiments may not fully reflect potential complexities in multi-dimensional domains.

• We thank the reviewer for the suggestion, which we also find interesting to investigate. We believe our results will also scale to 2D and 3D experiments as we decouple the dropout rates from the main model weights. In our paper, we did not apply to 2D and 3D experiments for computational resources limitations (as GPU memory was limited).

Weaknesses: More thorough ablations could be done on alternative priors besides the new “tVAMP prior

• We thank the reviewer for this suggestion, which is shared with reviewer WqtE. To show how tVAMP excels over standard prior and fixed dropout rates, we conducted a test on a synthetic time-series dataset. This test was handmade to show how the prior affects the network modelling (i) for varying amplitudes and frequencies in the states, (ii) understand the effect of time adaptivitly of the prior. We also add a simpler static dropout model for reference in the hope of answering the question raised in Other Comments or Suggestions. The results report RMSE, NLL, and ECE statistics. Static method reports the RMSE obtained if the initial state is not propagated, while MLP is the base (non-bayesian) architecture (2 layer of with 64 with relu activation). BARNN uses the same MLP architecture, and it is ablate on different priors.

Dataset:

$

\begin{cases} x_t &= x_{t-1} + \frac{3\pi}{100}, \quad x_0=0\\ y_t &=\frac{1}{5}\sum_{j=1}^5\sin(\alpha_i x_t +\beta_i), \quad \alpha_i\sim U[0.5, 1.5],, \beta_i\sim U[0, 3\pi],, t\in{1, 2, \dots, 100} \end{cases}

$

Table:

Why did you pick a tVAMP prior specifically, instead of simpler approximations?....

• We thank the reveiwer for the interesting question. We picked the tVAMP prior because during preliminary experiments we found that using a standard log-uniform prior (commonly used for VD) did not give us good uncertainties, and we attribute it to the fact that the prior pushed to heavily spared network weights. This is also confirmed by the time-series experiment results or the PDE (Table 1) results, where log-uniform and ARD priors obtain suboptimal results.