Decision Making under the Exponential Family: Distributionally Robust Optimisation with Bayesian Ambiguity Sets

Thank you for providing feedback that greatly improves the paper. Please see this anonymous link for Tables and Figures: https://github.com/ICML-anon-2025/paper-11717

1. Table 1: We have changed the table and caption to distinguish between variables and the input size of the dual (see link above). As you suggested, we will also add a brief discussion of Table 1 inside Sec. 3.5 with a more detailed explanation in the appendix.

2. Practical choice of $\epsilon$ and cross-validation (CV): We have now implemented a 10-fold CV procedure for choosing $\epsilon$ from a set $E$ for BAS-PE and BAS-PP. As suggested, we took Fig. 7 for the Normal DGP Newsvendor. We train our model only on the training folds, then after solving for each $\epsilon \in E$, we visualise the CV mean and variance on the validation folds across all replications in Fig. 9 of the link provided above. On Fig. 9, we have marked the $\epsilon$ that achieves 1) the smallest CV mean, 2) the smallest CV variance, 3) the smallest CV mean/standard-deviation tradeoff.

3. Statistical benefits of DRO-BAS over BDRO: One key difference is that we have a single worst-case objective, leading to an interpretable closed-form worst-case distribution (Sec. 3.6), whereas BDRO does not (see response 2 to fJ7Y). At the limit of infinite observations, both DRO-BAS and BDRO converge to a KL-based ambiguity set based on the DGP. However, our analysis is straightforward and more intuitive compared to BDRO, which uses a much more complex argument (p.1286-1292) to prove where the expected worst-case concentrates. In terms of statistical properties, we follow suggestions from the DRO literature (Shapiro et al. 2023, Gotoh et al. 2021) and focus on the mean-variance frontier, which is a type of statistical trade-off, showcasing our domination over BDRO. See also response 2 to V1H1 (on the statistical behaviour of the solution).

4. Interpretation of BAS in comparison to PDRO: Although the dual formulations of BAS and PDRO with the KL are both model-based and look related, they are fundamentally different. Our ambiguity sets are a-posteriori informed, integrating prior beliefs and data evidence. Consequently, the resulting nominal distribution ($\mathbb{P}_n$ for BAS-PP and $\mathbb{P}\_{\hat{\eta}}$ for BAS-PE) contains all the information from our posterior beliefs, including their uncertainty quantification, unlike the point estimator approach of Iyengar et al., 2023. This allows us to propagate uncertainty from the posterior beliefs about the parameters to the ambiguity set. We will highlight this important distinction by expanding the lines 53-63 with the points above.

5. Scalability: If the posterior is not available in closed-form, then the sampling time will increase (similarly to all Bayesian DRO methods) unless one performs approximate inference such as Variational or Laplace approximations. In this paper, we provide efficient formulations for a big class of models - the exponential family (which goes beyond just the Normal and Exponential distributions). See also response 3 to oPX6. Moreover, working with the exponential family enables our tractable formulations of the DRO-BAS problem and leverages the conjugacy property, which greatly enhances the scalability of our method.

6. Monte Carlo guarantees: It would be valuable to theoretically examine the effect of the Monte Carlo sampling size to the out-of-sample cost, similar to our empirical analysis in Sec. 4. This is a non-trivial analysis as it requires establishing bounds on the worst-case KL objectives based on a nominal distribution and its empirical approximation. This is also likely the reason why Iyengar et al., 2023 only provide these types of guarantees for the Wasserstein and $\chi^2$-divergence, and why Shapiro et al., 2023 omit them for the KL-based BDRO. Such guarantees would be a notable contribution on their own as they would likely lead to similar results for other model and KL-based DRO methods (like PDRO and BDRO). We will include this as future work in Sec. 5.

7. Choice of divergence: Please see response 3 to fJ7Y for a discussion of other $\phi$-divergences and response 4 to oPX6 for the benefits of KL-based BAS. The advantages of the KL have also been widely discussed in the DRO literature (Hu et al. 2013, Shapiro et al., 2023) and the closely related DR Bayesian Optimisation literature (Husain et al., 2023). We see extending BAS to other distances or divergences as an interesting future research direction.

8. Other comments: We will correct the vector notation for $h$ in the newsvendor objective. Concerning visualisations in Fig. 2 & 3, we have kept the $\epsilon$ markers because we do not want to give a false impression that the line is continuous and the values of $\epsilon$ are equidistant, but we will work to make the figures more legible.

[1] Husain H. et al. Distributionally robust Bayesian optimization with ϕ-divergences. 2023.

Thank you for carefully considering our work and providing feedback that greatly improves the paper:

1. Upper bound in Eq. 13: There is potentially a misunderstanding here: our work does not rely on this bound and goes beyond this inequality to achieve the strong duality result (with equality) in Theorem 3.6. In detail:

   • "What's the point of the upper bound in Eq. 13?": It provides a weak duality result for general parametric Bayesian models without closed-form posteriors, e.g. Bayesian neural networks. The discussion after Proposition 3.3 explains the challenge associated with BAS-PE leading to the upper bound in Eq. 13. Exactly for this reason we study the widely used exponential family models in Sec. 3.3, allowing us to move from an upper bound (weak duality) to an equality (strong duality) in Theorem 3.6.
   • "It resembles Eq. 8, but is not quantitative since it depends on $f$ and $\theta$ which makes the generalization property unclear": The goal of the duality result is to reformulate the maximisation in Eq. 11 as a minimisation so that the objective can be jointly minimised in the Lagrange variable $\gamma$ and decision variable $x$. As a duality result, it necessarily relies on $f$ and $\theta$ as these are part of the worst-case risk (the same dependence is in Proposition 3.2, Theorem 3.6, and in the duality results of BDRO and other methods).
   • "Is it possible to provide a more quantitative bound, [...] Otherwise, the theoretical analysis in the paper, except for the conjugate, looks like trivial corollaries of results in Hu & Hong, 2013." The theoretical analysis in Sec. 3.3 is far from trivial and cannot be seen as corollaries of Hu et al. 2013 as their analysis is for divergences and not for expected divergences. The goal of Sec. 3.3 is to go beyond the weak duality bound in Eq. 13 and provide stronger results. In particular, it provides an important formulation of the expected KL divergence in Lemma 3.5, which is crucial to overcoming the difficulty of obtaining the convex conjugate of the expected KL. This leads to a strong duality result in Theorem 3.6 along with the results in Corollary 3.7, Eq. 20 and Sec. 3.6, which have not been derived before for an ambiguity set based on an expected divergence (rather than just a divergence). To avoid similar misunderstandings in the future, we will include the clarifications above before Proposition 3.3.

2. Optimal radius selection: Such a probabilistic guarantee would be very interesting but is challenging future work that we add to the discussion. It would require finite sample concentration results about the KL divergence with respect to the posterior distribution. This is challenging as evident from the lack of such results in the DRO/KLD literature. To provide an alternative method of selecting the tolerance level, we now also perform a cross-validation radius selection analysis (see response 2 to GxWu)

3. Restriction to the exponential family: As discussed in Sec. 5, this is an interesting topic for future work. The key property is that the family of distributions should have a closed-form expression for the expected KL divergence, such as we have in Lemma 3.5. Otherwise, one can apply Proposition 3.3 to obtain an upper bound on the worst-case risk. We will include this comment in the discussion at the beginning of Sec. 3.3. Thankfully, we have obtained such a result for the exponential family, which is widespread across probabilistic graphical models, as well as models in spatial statistics and time series analysis, which also rely heavily on the exponential family. Moreover, it plays a central role in Variational inference, where it is the de facto approximating family, making our work broadly applicable across Machine Learning domains.

4. Choice of KL divergence (KLD) & missed paper in lit. review: Firstly, the KLD is a natural choice as the Bayesian posterior itself targets the KLD minimiser between the model family and DGP. Hence, intuitively, the expected KLD and the KLD with respect to the posterior predictive become good measures of separation between our posterior beliefs and the DGP (see main text lines 119-127 and Shapiro et al. 2023). The KLD further allows us to obtain a strong dual formulation in Theorem 3.6 for the DRO-BAS-PE setting and tractable reformulations (see Sec. 3.5). Moreover, by using the KLD, DRO-BAS admits the formulation of the optimal radius and the worst-case distribution, which are very important for interpretability and applicability of the method, overcoming shortcomings of the KL-based BDRO. We will further discuss the challenges associated with extending the framework to other distances/divergences in future work (Sec. 5) (see response 3 to fJ7Y) and include the missed paper in our literature review.

5. Typos and other comments: We will correct the typos and properly define the function $h$, which corresponds to the scaling constant of the exponential family.

Thank you for carefully considering our work and providing feedback and suggestions that greatly improve the paper:

1. Additional comparisons to other DRO ambiguity sets: Firstly, the suggested empirical-based ambiguity sets, based on the KL-divergence and Wasserstein distance, are fundamentally different to model-based approaches (e.g. DRO-BAS, BDRO) as they are fully data-driven and are not appropriate in cases where the decision-maker relies on a model to make a decision. There is already a discussion of the model-based vs empirical KL DRO setting in Appendix E.3, however, we will add a short statement in the introduction pointing the reader to this discussion. To this end, we have already included a comparison to the standard, empirical KL divergence ambiguity set that you suggested in Appendix E.3 and Figure 7. To further strengthen our empirical comparisons, and following your suggestion, we have also now conducted a comparison to the Wasserstein-based ambiguity set centred on the empirical measure for a well-specified (Normal DGP) and a misspecified (Truncated Normal DGP) Newsvendor example (see Figure 8 of this anonymous link: https://github.com/ICML-anon-2025/paper-11717). We observe that, for M sufficiently large, DRO-BAS outperforms all empirical methods. We will include this new comparison in Sec. 4 and in the Appendix.

2. How do the worst case distributions identified by BAS and BDRO differ?: Crucially, for BDRO, a single worst-case distribution does not exist because BDRO considers an expected worst-case approach (see Eq. 2, Figure 1 and the discussion on lines 67-79) rather than a worst-case approach, as advocated by DRO methods. In contrast, since both DRO-BAS formulations correspond to a worst-case risk minimisation objective, we obtain the worst-case distributions in Sec. 3.6. If we sample $\theta_j \sim \Pi$ from the posterior for BDRO, then one can obtain a worst-case distribution for each $\mathbb{P}\_{\theta_j}$ via the argument in Hu & Hong (2013), eq. (8). However, notice that in general, the minimiser of an expected objective is not the same as the average of the minimisers of the individual objectives as $\min\{f(x) + g(x)\} \geq \min\{f(x)\} + \min\{g(x)\}$ for any functions $f$ and $g$. Hence, even looking at the posterior mean or mode of the worst-case minimisers of the inner worst-case objectives in Eq. 2 would not necessarily give us a single distribution $p$ that yields a worst-case risk of the form $\mathbb{E}_{\xi \sim p} [f(x,\xi)]$ corresponding to the risk minimised by BDRO. In the limiting case of infinite data observations ($n \rightarrow \infty$) all of BDRO, BAS-PE and BAS-PP concentrate to a KL ambiguity set based on the data-generating process $\mathbb{P}^\star$ hence the worst case distribution is asymptotically the same for all methods but it is different (or non-existent for BDRO) for finite sample sizes. This further highlights a fundamental difference between BDRO, which opts for an expected worst-case objective, and DRO-BAS, which follows the common DRO route of defining a worst-case objective. DRO-BAS provides the decision-maker with an exact and interpretable formulation of the worst-case risk being minimised. We will expand further on these points in lines 60-63 where we discuss the main difference of our method to BDRO and also include the above points in Sec. 3.6, which deals with the DRO-BAS worst-case distribution.

3. Extension to general phi-divergence metrics?: This is something we are actively working towards and believe would be very valuable future work.

   • The analysis for DRO-BAS-PP would involve the derivation of the dual formulation of the problem based on convex conjugate results for $\phi$-divergences. Results about the radius and the worst-case distribution might be more challenging and would rely on properties of the posterior predictive and $\phi$-divergences.
   • The analysis for DRO-BAS-PE appears even more challenging: one would need to derive, in closed-form, an expression for the expected $\phi$-divergence under the posterior (similarly to Lemma 3.5). This would allow us to derive the convex conjugate form of the expected $\phi$-divergence, which is otherwise challenging to obtain. The convex conjugate expression of the expected divergence is essential in order to derive the strong duality result in Theorem 3.6, but the closed-form of the expected KL divergence further allowed us to obtain the results about the tolerance level selection (Sec. 3.4), a closed-form objective in the Gaussian case with linear cost function (Sec. 3.5) and the closed form of the worst-case distribution (Sec. 3.6).
   • In the discussion, we will explain the challenges associated with extending BAS to other $\phi$-divergences and suggest this as important future work. We believe this framework opens the door to alternative BAS formulations grounded in expected distances or divergences, such as the $\phi$-divergence you mentioned.

Thank you for carefully considering our work and providing feedback that greatly improve the paper:

1. Clarification of line 248 right side: $\epsilon^{\star}\_{\text{PE}}(n)$ in Eq. 20 is defined as the expected KL divergence between the data-generating process $\mathbb{P}^\star$ and model $\mathbb{P}\_\eta$, indexed by the natural parameter $\eta$. For any $\epsilon \geq \epsilon^\star\_{\text{PE}}(n)$ we have that $\mathbb{E}\_{\eta \sim \Pi}[d\_{KL}(\mathbb{P}^\star || \mathbb{P}\_\eta)] = \epsilon^\star\_{\text{PE}} \leq \epsilon$ and hence $\mathbb{P}^\star \in \mathcal{A}\_{\epsilon}(\Pi)$ by definition of the BAS-PE ambiguity set $\mathcal{A}\_\epsilon(\Pi)$. Since $\mathbb{P}^\star$ belongs to the ambiguity set, the worst-case risk will be at least as big as the risk under $\mathbb{P}^\star$ (since the supremum over a set is always at least as big as the value attained by any of its elements), resulting in Eq. 22. We will add this explanation to the main text and highlight its importance: if one chooses the radius at least as large as $\epsilon^\star\_{\text{PE}}(n)$ then the worst-case risk minimised by DRO-BAS-PE will upper bound the true risk under $\mathbb{P}^\star$.

2. Explanation of variance behaviour for increasing values of $\epsilon$ in Fig. 2: The reason is well-understood but quite challenging to convey compactly: the behaviour of the OOS variance $v(\epsilon)$ in the Newsvendor experiments is due to the behaviour of the variance of the solution (denoted by $v\_{x}(\epsilon)$) (see Fig. 10 in this anonymised link: https://github.com/ICML-anon-2025/paper-11717) across epsilons. In turn, the variance of the solution is driven by the Newsvendor asymmetric cost function and its interplay with the worst-case distribution for each $\epsilon$. The Newsvendor objective is piece-wise linear with 2 pieces, and hence, the true risk (expected cost under the DGP, in this example a Normal) has a similar two-piece behaviour. The optimal solution lies at the intersection of the two pieces of the true risk. The piece corresponding to larger solutions has a significantly smaller slope (see 4th plot of Fig. 10), leading to a smaller true risk.

   • Let’s consider a fixed value of $M=25$. For small values of $\epsilon$, the variance of the solution $v\_{x}(\epsilon)$ is smaller because fewer distributions are included in the ambiguity set; hence, the obtained solution is fairly stable over replications (see first plot of Fig. 10). However, because BAS likely does not include the DGP, and the decision-making is risk-prone, the OOS variance of the cost function $v(\epsilon)$ is large (see the Pareto curve of the first plot of Fig. 4). In this regime, the OOS variance reduces as we increase epsilon and better capture the DGP.
   • As $\epsilon$ increases further, the ambiguity sets start including a lot more distributions, leading to the mean solution moving to values larger than the optimal solution where the slope of the true risk is smaller (see 4th plot of Fig. 10). This makes sense as, as we increase $\epsilon$, we become more conservative. However, as $\epsilon$ increases to very big values (in this case $> 0.5$), the ambiguity set contains a lot of arbitrary distributions, significantly increasing the OOS cost. The methods then push the solution towards smaller values of $x$ as can be observed by the big variance $v\_{x}(\epsilon)$ on the first plot for larger values of $\epsilon$.
   • This behaviour is common for all values of $M$, however, there is an important distinction: for smaller values of $M$ (25, 100), the error bars extend way below and above the optimal solution. If we associate this with the form of the true risk on the last plot, we can expect a very high variance $v(\epsilon)$ of the OOS cost. On the other hand, the empirical variance of the solution $v\_{x}(\epsilon)$ for large M ($M = 900$) does not extend below the optimal solution by a high degree meaning that as the optimisation becomes more exact, the methods suggest staying on values bigger than the optimal solution which correspond to the smaller slope of the true risk. This is why in the third plot of Fig. 4 in the main text ($M = 900$), the OOS variance $v(\epsilon)$ does not increase by a lot for big values of $\epsilon$.
   • We believe this plot and the above explanation give great intuition on how the specific form of the cost function can affect the behaviour of the method and will hence include it in the Appendix.

3. Other comments: We will change the statement in line 102 to your suggestion. We will further add a reference to Definition 3.4, noting that we are following the notation of Murphy et al. 2023 for the conjugate exponential family. We accidentally omitted the definition of $h(\xi)$, which is referred to as the scaling constant.

4. Typos and references to proofs: We will correct the typo and add direct references to the location of each proof after each mathematical statement.