Bridging the Gap between Variational Inference and Stochastic Gradient MCMC in Function Space

$**Q: **$ At what scale of BNNs can we use FVIMC feasibly? Can you make an inference on a ResNet-sized network (11-25k params)?

$**Reply: **$ The network parameter is 10,401-dimensional and 261-dimensional (maximum) for the synthetic extrapolation and uci regressions (maximum), respectively. The params for the image classifications is 42,310-dimensional, and for the contextual bandit, it is 12,501-dimensional, which illustrates that our FVIMC can scale to large models (at least 10-50k params). Hence, it should be feasible to apply it to ResNet-sized networks. We will consider deploying FVIMC to more complex architectures such as CNN and RNN in our future work.

$**Q: **$ Why does the collapse occur in FVIMC Fig. 1a? Are there ways to address or mitigate it?

$**Reply: **$ We consider this to be the result of a minor program implementation bug, as it happens to be near the 0 point. The problem is definitely controllable, and we will double-check and debug our code to solve it.

$**Q: **$ The computational cost of their method.

$**Reply: **$ Thanks for the suggestion! We add the analysis of computational complexity in Appendix G.2. To compare the efficiency between FVIMC and other inference approaches, we provide running time for all 2000 training epochs of all inference methods in the uci regression on the small Boston dataset(455 training points) and the large Protein dataset(41157 training points) in Tab4 in Appendix G.2. In the small Boston dataset, our FVIMC shows similar computational efficiency to the other functional fVI, GWI, IFBNN, and is nearly 5 times faster than FBNN. With the fast convergence of the V-Stage to locally high probabilities, our FVIMC demonstrates its computational efficiency advantage on the larger Protein dataset: the running time of GWI and FBNN is nearly 1.7-2.5× higher than FVIMC. The mixing rate of fSGLD, which performs well on small datasets, has deteriorated at this time and is significantly slower than our hybrid method. Moreover, the convergence processes of training loss for all VI baselines and the V-Stage of FVIMC are shown in Fig7, where FVIMC achieves high convergence speed and training stability.

$**Q: **$ How to choose hyperparameters?

$**Reply: **$ Our experimental tasks align with the benchmark tasks commonly used in the literature on functional and parameter posterior inference for BNNs. Initially, we referred to the general experimental settings reported in prior studies to define a broad range of hyperparameters. Through pre-training or initial trials, we selected specific configurations that enable all baseline methods to achieve satisfactory results. For instance, we employed an RBF kernel GP for all functional methods to facilitate smoothing, as it is a versatile kernel capable of capturing the primary characteristics of most polynomial functions. Additionally, an extensive ablation study on the choice of kernel functions is provided in Appendix H.2. Regarding convergence criteria for the V-Step and M-Step, we relied on the training loss convergence curve for fVI and the mixing time trajectory for fMCMC, respectively, as detailed in Appendices G.1 and G.3. For the transformation, we utilized a bias function $b(x;w)$, modelled as a two-hidden-layer fully connected neural network with 100 hidden units per layer. This architecture is flexible and expressive, making it well-suited for fitting complex posterior distributions.

$**Q: **$ State the T-Stage preserves the shared evolution of posteriors between the V-Stage and M-Satge.

$**Reply: **$ Thanks for the suggestion! We add the plot of the rmse comparisons between the posterior in the M-Stage and the transformed posterior obtained in the T-Stage in the uci regressions to quantify the transformation errors empirically in Appendix G.4. We can see that as the algorithm proceeds, this transformation error can be significantly reduced and effectively controlled. Additionally, the training losses in the T-stages converge quickly and smoothly. Therefore, the parameterized posterior obtained by the T-Satge transformation is a reasonable approximation to the implicit posterior of M-Stage.

We express our gratitude to Reviewer aL6E for positive and insightful feedback. In response to your comments, we prepared a revised version of the manuscript and address specific questions below.

Questions:

$**Q: **$ Could you clarify whether proposition 3.2 still holds with data subsampling?

$**Reply: **$ Proposition 3.2 remains valid even with data subsampling. Firstly, the stochastic gradient approximation of the likelihood term does not alter the target functional of the Wasserstein gradient flow in fVI. Secondly, it is well-established that the stationary distribution of stochastic gradient Langevin dynamics converges to the target posterior when using a decreasing step size, as the stochastic gradient noise diminishes faster than the injected Gaussian noise. Hence, the validity of Proposition 3.2 is independent of the stochastic gradient approximation. We will include a note on this in the final version.

$**Q: **$ The paper states that FVIMC can incorporate meaningful prior information into the posterior. The experiments are done through empirical Bayes, i.e., using a GP learned from data as a prior; however, it is not done with BBB. Why can't you cannot use empirical Bayes to find the prior for BBB?

$**Reply: **$ The reason is that functional prior like GP can easily encode prior knowledge about the underlying functions, e.g., periodicity, shape and smoothness through corresponding kernel functions. For example, suppose we want to predict the daily average temperature of a country. We know that such a function should show some periodicity, so we can give GP prior with a periodicity kernel, which can help with function learning. In contrast, it is hard to encode such periodicity into each neural network weight. As network weights are not interpretable, formulating a good prior on them is virtually impossible.

$**Q: **$ Could you please clarify which pathologies "potential pathologies in deep learning" (line 147) are referring to?

$**Reply: **$ We apologize for not elaborating on these pathologies due to space limitations. There is existing literature has highlighted potential issues with isotropic Gaussian priors in parameter-space inference. For instance, as network depth increases, function samples derived from Gaussian priors over parameters tend to exhibit horizontal behavior [1][2]. Moreover, in architectures with ReLU activations, the prior distribution of unit outputs becomes increasingly heavy-tailed with greater depth. These pathologies in parameter-space priors may lead to problems in posterior inference. Refer to Fig. 1 in [3] for a more intuitive illustration.

[1] David Duvenaud et al. Avoiding pathologies in very deep networks. AISTATS, 2014.

[2] Matthews et al. Gaussian process behaviour in wide deep neural networks. ICLR, 2018.

[3] Tran et al. All you need is a good functional prior for Bayesian deep learning. JMLR, 23(74):1–56, 2022.

$**Q: **$ On line 54, the paper calls the isotropic Gaussian prior non-informative. Should this be understood as non-interpretable in the sense introduced by Tran et al. (2022)?

$**Reply: **$ Actually, the default isotropic Gaussian prior is commonly thought of as uninformative and may even carry incorrect beliefs about the posterior (uni-modality, independent weights, etc.[4][5]). Moreover, due to the complex architecture and non-linearity of BNN, the corresponding prior over functions induced by this prior over parameters is also non-interpretable and hard to control, as shown in Tran et al. (2022).

[4] Knoblauch et al.Generalized variational inference: Three arguments for deriving new posteriors, 2019.

[5] Vincent Fortuin. Priors in Bayesian deep learning: A review, 2022.

$**Q: **$ What is the size of the networks used for experimentation?

$**Reply: **$ In the synthetic extrapolation, we use a $2 \times 100$ fully connected tanh neural network for all inference methods. In the uci regressions, the model is a two-hidden-layer fully connected neural network, with each layer having 10 hidden units. In the contextual bandit, we use fully connected neural networks with input-100-100-output architecture. And for the image classification and OOD detection experiments, the model is a two-hidden-layer fully connected neural network, each with 50 hidden units. We add the details of the experiment setting in Appendix E.

With the author's rebuttal and the updated article, the authors have adequately addressed most of my concerns with the paper. Therefore, I have increased the score to a 5, Soundness to 2 and the Presentation to a 3. To improve the score to a 6, I would like the authors to comment on question 8.

Thanks for sharing the connection between KL and $W_1$. I follow your argument, and your empirical evidence is persuasive. However, I have one point of technical clarification: for KL and $W_1$ to agree, you’d need the difference function f - g to be 1-Lipchitz; otherwise, it’s not a test function. Is this true for $f = \log q$ and g = $\frac{q}{p} \log p$?

We express our gratitude to Reviewer qXT6 for positive and insightful feedback. In response to your comments, we prepared a revised version of the manuscript and address specific questions below.

Questions:

$**Q: **$ Typo: 1. In line 132, the term $\mathcal{X}^n = \cup [\mathbf{X} \in \mathcal{X}^n |\mathcal{X}^n \in \mathbb{R}^n ]$ is confusing.

$**Reply: **$ We are sorry for this typo. Actually, it should be $\mathcal{X}_{\mathbb{N}} \doteq \cup _{n \in \mathbb{N}} [\mathbf{X} \in \mathcal{X}_n \mid \mathcal{X}_n \subseteq \mathbb{R}^{n \times p}]$, which denotes the collection of all finite sets of marginal measurement points in the input domain and $p$ is the input feature dimensions.

$**Q: **$ In line 166, "this factorized Gaussian will be modified later rather than being fixed, as in fVI, so it does not impose a strict restriction." I believe that the choice of variational distribution family $q_{s0}(\mathbf{w}{s0},\boldsymbol{\theta}{s0})$ does, in fact, constitute a restriction for fVI. Can you clarify this point more clearly?

$**Reply: **$ We understand that our earlier explanation might have been unclear, and we apologise for any confusion. The key point we intended to convey is this: Unlike the naive pure fVI, which typically imposes rigid distributional assumptions on the variational posterior, our hybrid algorithm is inherently more flexible, allowing it to model arbitrarily complex posteriors. Specifically, given a simple initial approximate posterior over parameters $q_{s0}(\mathbf{w}{s0},\boldsymbol{\theta}{s0})$, e.g., it can be a factorized Gaussian has $\boldsymbol{\theta}{s0} = (\mathbf{\mu}{s0}, \mathbf{\Sigma}{s0})$, which induces the posterior measure over functions denoted by $Q_{f_{s0}}$. Then, after the first V-Stage, we can obtain the updated functional posterior $Q_{f_{s1}}$ with the corresponding updated variational posterior over parameters $q_{s1}(\mathbf{w}{s0}; \boldsymbol{\theta}{s1})$. In the following M-Stage, the probability measure of the collected function samples is denoted as $Q_{f_{s2}}$. Notice that at this point, the distribution $q_{s2}(\mathbf{w}{s2})$ of the corresponding parameter samples becomes assumption-free and implicit, meaning it is no longer restricted to the form of a fully-factorized Gaussian. In contrast, pure fVI retains its initial assumption about the family of variational distributions throughout the optimization process. For example, if the initial variational distribution $q(\mathbf{w},\boldsymbol{\theta})$ is a fully-factorized Gaussian, pure fVI continuously optimizes the variational parameters $\boldsymbol{\theta} = (\mathbf{\mu}, \mathbf{\Sigma})$ within this framework, without altering the underlying distributional type. This oversimplified variational assumption in pure fVI may fall short in accurately approximating the posterior for complex models and induce pathologies such as over-pruning.

Weaknesses:

$**W: **$ In the results for BNNs, the RMSE scale seems inconsistent with previous BNN results reported in [1] and [2].

$**Reply: **$ Our experimental setup is different from Sun et al. In their uci experiment, they used different network architectures and training epochs, but the model implementation exactly follows their published code. The results of all methods reported in our paper share the same architecture and training iterations for fair comparison.

$**W: **$ In the experimental results, an ablation study comparing KL divergence and 1-Wasserstein distance in fVIMC would be helpful.

$**Reply: **$ In the experiment part of the main paper (Sec.5), FBNN (Sun et al., 2019) refers to the KL divergence-based functional variational method, while fVI represents the alternative function variational inference approach utilizing the Wasserstein distance. Experimental results demonstrate that the pure fVI method, leveraging the Wasserstein distance, outperforms FBNN in both predictive performance and uncertainty estimation. Moreover, KL-based FBNN was excluded from the V-Stage in our hybrid FVIMC for the ablation study due to its prohibitively high computational complexity (you can see Appendix G.2 for detailed computational complexity comparisons), which limits its scalability to models with high-dimensional inputs or targets.

We express our gratitude to Reviewer K6pQ for positive and insightful feedback. In response to your comments, we prepared a revised version of the manuscript and address specific questions below.

Questions:

$**Q: **$ I do not believe this general approach has been taken for parametric Bayesian neural networks, is this the case?

$**Reply: **$ Our method can indeed be applied to parameter space inference in a similarly smooth manner, specifically by simply replacing the fVI optimization in the V-Stage and the fMCMC procedure in the M-Stage with the corresponding parameter-space VI and MCMC, respectively, which reflects the universality of our proposed hybrid algorithm.

Weaknesses:

$**W: **$ The mixing times of fMCMC, especially in high dimensions, require a stand-alone investigation. Additionally, showcasing some loss curves and other metrics throughout training for real-world experiments could also showcase this.

$**Reply: **$ Thanks for the suggestion! We add the analysis of mixing Time in Appendix G.1. We plot the trajectories of fMCMC in the M-Stage, the parameter-space SGLD and the functional fSGLD for the synthetic extrapolation and uci regression (Yacht) in Fig6. The parameter dimension of the nn used for the extrapolation experiments is 10401 dimensions, and the parameters of the uci regression model are 191 dimensions. With the aid of the explicit optimization of fVI in the V-stage, which quickly converges to local high-probability target regions, the fMCMC in the M-stage of our hybrid FVIMC exhibits a rapid mixing rate for high-dimensional parameter space. In the extrapolation example, the fMCMC in the M-stage converges rapidly to the stationary measure within 800 iterations, while fSGLD and SGLD show stabilizing trends until after 2000 iterations. Similarly, our FVIMC reaches the stationary for the uci regression in only 400 iterations, while SGLD and fSGLD require more than 500 iterations for mixing. Additionally, the convergence processes of training loss for all VI baselines and the fVI in the V-Stage of FVIMC are shown in Fig7 in Appendix G.2, where FVIMC achieves high convergence speed and training stability.

We express our gratitude to Reviewer 7yD4 for positive and insightful feedback. In response to your comments, we prepared a revised version of the manuscript and address specific questions below.

Questions:

$**Q: **$ In the section 5.1, the authors said they make fair comparison between their iterative method and the non-iterative methods by specifying certain number of epochs and iterations. I wonder how those number are determined in terms of the same computational budget between all methods?

$**Reply: **$ Our experimental tasks align with the benchmark tasks commonly used in the literature on functional and parameter posterior inference for BNNs. Initially, we referred to the general experimental settings reported in prior studies to define a broad range of training iterations. Through pre-training or initial trials, we selected specific configurations that enable all baseline methods to achieve satisfactory results (e.g., in the extrapolation example in sec.5.1, all methods appeared to converge with more than 10,000 training epochs). This study's experiments are designed to show the different performances between different parameter/function-space inference methods for BNNs. Hence, we do not thoroughly adjust such factors to obtain the best performance, and we just set the same number for all methods for fair comparison. To analyse computational complexity, we provide running time comparisons for the same 2000 training iterations of all inference methods in the uci regressions in Tab4 of Appendix G.2, where our method demonstrates high convergence speed and training stability.

Weaknesses:

$**W: **$ I am a bit confused about how to validate the iterations between the V-stage and the M-stage. As the paper mentioned in the page 4, there is a problem when altering between these two stages, because the M-stage outputs a non-parametric distribution while the V-stage requires a parametric one as its input. I wonder how the transformation from $Q_{fs1}$ to $Q_{fs2}$ is defined specifically.

$**Reply: **$ Based on the $Q_{fs1}$ with corresponding explicit posterior over parameters as $q_{s1}(w_{s1}; \theta_{s1})$ obtained in the last V-Stage, we can fit an approximated parametric transformation in T-Stage for the posterior $Q_{fs2}$ obtained in the M-Stage with implicit parameter posterior $q_{s2}(w_{s2})$. Specifically, 1) we first collect function samples $f_{s2_{{1:N}}}^{X_M} \sim Q_{fs2}$ based on the corresponding parameter samples $w_{s2_{{1:N}}}$ obtained from M-Stage, where $X_{M}$ is the randomly sampled measurement set; 2) Based on the local linearization (Rudner et al., 2022), approximating functional $Q_{f_{s1}}$ as a Gaussian process given by $GP(f_{s1}|f_{s1}(\cdot, \mu_{s1}), J_{\mu_{s1}}\Sigma_{s1}J_{\mu_{s1}}^{T})$, where $J_{\mu_{s1}}$ denotes the Jacobian $\frac{\partial f_{s1}(\cdot, w_{s1})}{\partial w_{s1}}| {w_{s1} = \mu_{s1}}$, and the finite marginal distribution of $Q_{f_{s1}}$ is reduced to an analytical Gaussian distribution; 3) with the three elaborately designed invertible bijections in eq(7)-(9), where the arbitrarily flexible bias function $b(x, w)$ is parametrized as a $2 \times 100$ fully connected nn to achieve high expressive ability, maximizing the log-likelihood in eq(5) to optimize the bijection parameters $\lambda = [a, w]$ (we implement an automatic selection mechanism that chooses one of the three bijections with the highest likelihood during the optimization). Finally, we can obtain the transformed $F_{\lambda_{1}} \circ Q_{f_{s1}}$ for $Q_{fs2}$ with the optimal bijective function $F_{\lambda_{1}}$.

Dear Reviewer 7yD4,

Thank you very much for taking the time to review our responses. We are glad to hear that we have effectively addressed your concerns. I sincerely thank you for your thoughtful feedback and for assigning a higher score to our work.

Thank you once again for your support!

Best Regards,

Authors

$**Q5: **$ The transformation step will probably introduce errors.

$**Reply: **$ Thanks for the suggestion! We add the plot of the rmse comparisons between the posterior in the M-Stage and the transformed posterior obtained in the T-Stage in the uci regressions to quantify the transformation errors in Appendix G.4. We can see that as the algorithm proceeds, this transformation error can be significantly reduced and effectively controlled. Additionally, the training losses in the T-stages converge quickly and smoothly. Therefore, the parameterized posterior obtained by the T-Satge transformation is a reasonable approximation to the implicit posterior of M-Stage.

Weaknesses:

$**W1: **$ I think the method can probably be applied in the more simple non-functional setting, which can make the description more easy to understand and get across.

$**Reply: **$ Our approach can indeed be applied to parameter-space inference smoothly, but this paper is concerned with the functional posterior inference for BNNs and is dedicated to combining fMCMC and fVI to inherit their respective strengths effectively. We will carry out careful proofreading work and further improve the expression of some sentences in the final version.

$**W2: **$ It would also be nice to see experimental results on uncertainty quantification.

$**Reply: **$ We add the plot of calibration curves of all parameter/function-space inference approaches for the synthetic extrapolation example in Appendix G.3, which assesses how well the predicted probabilities of a model match the true frequencies. Our FVIMC demonstrates competitive calibration capability. We think the strong and reasonable uncertainty presented in Fig1(a), and the favourable nll and auc results in Tab2 and Tab3 in the main paper can also indicate the reliable uncertainty quantification of our method.

$**W3: **$ The paper did not compare against other baselines that are "not as Bayesian".

$**Reply: **$ In this study, our experiments are mainly designed to demonstrate the advantages of our hybrid method compared with other parameter/function-space VI and MCMC baselines. Our baselines are consistent with those used in the existing literature on functional /parameter posterior inference for BNNs. We will consider extending the comparisons with other non-Bayesian methods in our following study.

$**W4: **$ My worry with the paper is the technical complexity and computational expense.

$**Reply: **$ To compare the efficiency between FVIMC and other inference approaches, we provide running time comparisons for all 2000 training epochs of all inference methods in the uci regression on the small Boston dataset(455 training points) and the large Protein dataset(41157 training points) in Tab4 in Appendix G.2. In the small Boston dataset, our FVIMC shows similar computational efficiency to the other functional fVI, GWI, IFBNN, and is nearly 5 times faster than FBNN. With the fast convergence of the V-Stage to locally high probabilities, our FVIMC demonstrates its computational efficiency advantage on the larger Protein dataset: the running time of GWI and FBNN is nearly 1.7-2.5× higher than FVIMC. The mixing rate of fSGLD, which performs well on small datasets, has deteriorated at this time and is significantly slower than our hybrid method. Moreover, the convergence processes of training loss for all VI baselines and the V-Stage of FVIMC are shown in Fig7, where FVIMC achieves high convergence speed and training stability.

We express our gratitude to Reviewer bU3Q for the insightful feedback. In response to your comments, we prepared a revised version of the manuscript and address specific questions below.

$**Q1: **$ Which specific fVI method do you use?

$**Reply: **$ Eq.(1) in the main paper is the general KL-based functional ELBO to be maximized in fVI. However, considering the potential definitional and estimation problems of kL divergence for distribution over functions, we use the alternative Wasserstein distance-based variational objective with an additional enhanced second-order moment matching term for fVI [1] in practical optimization as: $L_{W} = -\frac{1}{M} \sum_{(x, y) \in \mathcal{B}}E_{Q_{f}}[\log p(y \mid f(x;\mathbf{w}))] + W_1(Q_{f} (f^{X_{M}}), P_{0} (f^{X_{M}})) + |var(Q_{f}(f^{X_{M}}))-var(P_0(f^{X_{M}}))|$. It is a fully sampling-based procedure that does not depend on the closed form of $Q_{f}$ and is more flexible than the KL-based objective. We will carefully check the notation in the pseudocode in the final version.

[1] Wu et al. Functional Wasserstein bridge inference for Bayesian deep learning. UAI, 2024.

$**Q2: **$ I think $Q_{fsk}$ in Prop 3.1 should be $Q_{fs1}$?

$**Reply: **$ Actually, it should be $Q_{fsK}$ since $Q_{fsi}$ is continuously updated. Based on the transformed posterior $F_{\lambda_{1}} \circ Q_{f_{s1}}$ obtained in the first T-Stage1 of the main paper for the posterior $Q_{fs2}$ of samples from the second M-Stage2, we can continue to perform the next V-Stage3, and the resulting approximate posterior measure is denoted as $F_{\lambda_{1}} \circ Q_{f_{s3}}$ with the corresponding updated posterior over parameters as $q_{s3}(w_{s3}; \theta_{s3})$. In the following M-Stage4, the probability measure of the collected function samples is denoted as $F_{\lambda_{1}} \circ Q_{f_{s4}}$ and the corresponding implicit posterior over parameters is $q_{s4}(w_{s4})$. We perform the second T-sage2 to transform this $F_{\lambda_{1}} \circ Q_{f_{s4}}$ into $F_{\lambda_{2}} \circ F_{\lambda_{1}} \circ Q_{f_{s3}}$ based on the $F_{\lambda_{1}} \circ Q_{f_{s3}}$ with parametric $q_{s3}(w_{s3}; \theta_{s3})$ obtained in the last V-Stage3, then it is able to run the next V-Stage5 smoothly and obtain the updated $F_{\lambda_{2}} \circ F_{\lambda_{1}} \circ Q_{f_{s5}}$ with the corresponding $q_{s5}(w_{s5}; \theta_{s5})$. We continue in this fashion by successively alternating between V-Stage and M-Stage with intermediate T-Stage, so the final approximate posterior measure is in a composite form: $F_{\lambda_{(K-1)/2}} \circ \cdots F_{\lambda_{2}} \circ F_{\lambda_{1}} \circ Q_{f_{sK}}$ with a total $K$ V-Stages and M-Stages. The base parametric $q_{si}(w_{si}; \theta_{si})$ is being continuously updated, so the corresponding final base posterior over functions is no longer $Q_{fs1}$, but should be $Q_{fsK}$. You can see Appendix B.1 for a more detailed derivation of Prop 3.1.

$**Q3: **$ The forms of the invertible transformations are very simple. Can they really capture sufficiently complex posteriors?

$**Reply: **$ The bias function $b(x; \omega)$ in the invertible transformations can be arbitrarily complex since the computation of Jacobian does not depend on it. So, we can form $b(x; \omega)$ as neural networks parametrized by $\omega$ with arbitrarily flexible structures to fit the complex posteriors. For example, in our experiment settings, we made it a two-hidden-layer fully connected neural network, and each layer with 100 hidden units, which is flexible and expressive to fit the complex posteriors.

$**Q4: **$ I don't think there's a precise statement of Prop 3.2. Can you furnish this? I feel that Prop 3.2 is just saying that the fVI and fMCMC are "going in the same direction".

$**Reply: **$ We extend the parameter-space conclusion that Langevin SDE and the probability flow ODE derived from the Wasserstein gradient of VI share the same marginals $\{q_{t}(\mathbf{w})\}_{t\geq0}$ [2][3] to function space. Similar to the parameter space derivation process, we derived that the Wasserstein gradient flow of fVI reads the Fokker-Planck equation of functional Langevin SDE, and further proved that they share the same probability measure evolution marginals.

[2] Hoffman et al. Black-box variational inference as a parametric approximation to Langevin dynamics. ICML, 2020.

[3] Yi et al. Bridging the gap between variational inference and Wasserstein gradient flows. arXiv:2310.20090, 2023.

Dear Reviewer bU3Q,

Thank you very much for taking the time and effort to review our manuscript, which has helped us refine our work. As the discussion phase between authors and reviewers is nearing its conclusion, we sincerely hope that the revisions and responses have clarified your concerns. If you have any remaining doubts or require further clarification, please do not hesitate to let us know. We are more than willing to continue the discussion to address them.

Thank you once again for your time and valuable suggestions.

Best Regards,

Authors