The Bias-Variance Tradeoff in Data-Driven Optimization: A Local Misspecification Perspective

We sincerely appreciate your detailed review and thoughtful comments on our paper.

W1: We completely understand your concern, and have run and discussed new experimental results. Please refer to our response to Reviewer JBev for details.

W2: We very much understand the reviewer's point on practical applicability. As in many other theoretical studies, the goal of our paper is to provide insights, in our case the bias-variance dissection and directional impacts of model misspecification, and also to bring in the new theoretical tools that allow us to derive the results and conclude these insights. There is certainly an important question on practical implementation. To this end, in our mind our work is not a one-time study in this problem domain, instead there will be many follow-up works on other aspects including practical and computational issues.

Thus, we are careful not to over-state our contributions (both in the paper and in this response). That being said, our theory does give some clear insights. First, in terms of choosing among the three methods: if the misspecification error is larger than the sampling error, then SAA should be used; if the model is almost well-specified, then ETO should be used; if the misspecification error and the sampling error are of the same scale, or are generally unknown, then IEO is usually a robust choice. Second, our main results reveal that the orderings of the bias and variance of the three methods are universal regardless of (almost) all directions $u(\boldsymbol{z})$, but only depend on the degree of misspecification. That is, the deviating directions $u$ of $Q$ turn out to be an invariant factor under most scenarios.

W3: We will add discussions on the extensions of our framework to other settings in the paper. These extensions, including contextual optimization, constrained optimization and the incorporation of robustness, range from immediately doable to the need of a potentially substantial expansion of our framework: For contextual optimization, the extension is quite immediate. Specifically, we introduce the feature $\boldsymbol{x}$ in additional to the random parameter $\boldsymbol{z}$. We write the joint distribution of $\boldsymbol{x}$ and $\boldsymbol{z}$ as $Q(\boldsymbol{x},\boldsymbol{z})=Q(\boldsymbol{x})Q(\boldsymbol{z}|\boldsymbol{x})$. In this case we need to estimate $Q(\boldsymbol{z}|\boldsymbol{x})$ using parametric family $Q_{\boldsymbol{\theta}}(\boldsymbol{z}|\boldsymbol{x})$. Note that in the contextual case, the SAA method is not interesting because it allows to choose any map from context to decision, thus easily overfitting the finite-sample problem. In this case, IEO and ETO are the more legitimate approaches. In the mildly misspecified case, ETO is better than IEO while in the balanced and misspecified case, IEO is better than ETO. For constrained optimization, if the constraints are deterministic, then we expect our analysis and insights to continue to hold, but it would introduce additional technical complexity from the KKT conditions instead of using simpler optimality conditions. On the other hand, if the constraints are stochastic, then there is an additional question of feasibility (e.g., in chance constrained settings where random safety conditions need to be enforced with high probability). These extensions to constrained settings warrant future work in our view. Lastly, to incorporate robustness, a natural consideration is the possibility of distributional shifts. In this setup, we would need to consider the worst-case regret or other decision-theoretic counterparts, and the involved dissection is not only confined to statistical bias and variance but also deterioration of the decision quality coming from the fundamental shifts. This extension is substantial and would be worth another future work.

Q1: As described in our response to W1, we completely understand your concern, and refer you to our new experimental results and discussions in our response to Reviewer JBev.

We sincerely appreciate your detailed review and thoughtful comments on our paper.

W1: We completely understand and have added experiments to validate our theoretical results. We validate our findings by conducting numerical experiments on the newsvendor problem, a classic example in operations research with non-linear cost objectives. We show and compare the performances of the three data-driven methods under different local misspecification settings, including different directions and degrees of misspecification, to validate our theoretical analysis as well as providing additional insights about the effect of model parameters.

The newsvendor problem has the objective function $c(w,z)={a}^\top(w-z)^++d^\top(z-w)^+$, where for each $j\in[d_z]$: (1) $z^{(j)}$ is the customers' random demand of product $j$; (2) $w^{(j)}$ is the decision variable, the ordering quantity for product $j$; (3) $a^{(j)}$ is the holding cost for product $j$; (4) $d^{(j)}$ is the backlogging cost for product $j$. We assume the random demand for each product are independent. We assume the holding cost and backlogging cost is uniform among all products by setting $a^{(j)}=5$ and $d^{(j)}=1$ for all $j\in[d_z]$.

We describe the local misspecified setting by using the framework of Example 5 and building a model and generating a random demand dataset as follows. We denote the training dataset as $\{z_i^{(j)}\}_{i=1}^n$, where $n$ is the training sample size. The model assumes that the demand for each product $j\in[d_z]$ is normally distributed with the distribution $N(j\theta,1)$ where $\theta$ is unknown and needs to be learned. We first describe the well-specified setting, where the demand distribution for product $j$ is $N(3j,1)$. To describe the local misspecification, we need to specify the direction and degree of misspecification. We set (1) $\alpha=0.1$ for the severely misspecified setting, (2) $\alpha=0.5$ for the balanced setting and $\alpha=2$ for the mildly misspecified setting.

We discuss two types of directions: (1) $u(z)=\sum_{j=1}^{d_z} (z^{(j)})^2$; (2) ${u}(z)=\sum_{j=1}^{d_z} j(z^{(j)}-3j)$.

We show experimental results in Tables 1-3 to support our theoretical results in Section 3, using the mean of the regret for three methods, under three directions and three misspecified regimes. When ${u}(z)=\sum_{j=1}^{d_z} (z^{(j)})^2$, in the mildly specified case, ETO has a lower regret than IEO, and IEO has a lower regret than SAA. However, in the severely misspecified regime, the ordering of the three methods flips. This is consistent with our theoretical comparison results in Theorems 3, 4 and 6. In the balanced regime, experimental results show that IEO has the lowest regret among the three methods. This is also consistent with the theoretical insight in Theorem 2 that IEO has the advantage of more readily achieving the bias-variance trade-off in terms of the decisions and corresponding regrets. It also gives us some practical guidelines in choosing among the three methods: if the misspecification error is larger than the sampling error, then SAA should be used; if the model is almost well-specified, then ETO should be used; if the misspecification error and the sampling error are of the same scale, or are generally unknown, then IEO is usually a robust choice.

W2: We note that (and we believe this is precisely what the reviewer thinks as well), as in other asymptotic analyses, our "negligibility" is theoretically framed by comparing the orders of different sequences. Consequently, the theory by itself does not reveal the absolute magnitudes of the various quantities; instead it serves primarily to provide comparative insights. While this is a typical limitation of any asymptotic study, through our new experiments we can get a sense of the absolute magnitudes, as shown in Tables 5-7, when $u(z)=\sum_{j=1}^{d_z} (z^{(j)})^2$ under three different misspecified regimes. We demonstrate the bias and variance terms as scalars by taking the norm of sample mean and sample covariance matrix of $n^{-\min\{1/2,\alpha\}}(\hat{w}^\square-w^*_n)$. In the mildly misspecified case, the bias term is dominated by the variance term; in the severely misspecified regime, the dominance is flipped; in the balanced case, no significant dominance exists. Moreover, the results are consistent with the bias and variance comparison results in Theorems 1, 3, 4, 6.

W3: In our new experiments, we include the setting when ${u}(z)=\sum_{j=1}^{d_z} j(z^{(j)}-3j)$, an impactless direction discussed in Theorem 5. In this case, in Table 4 the ordering of the three methods is different from previous scenarios (Table 1-3). This is because the direction of misspecification ${u}(z)$ in this case coincides with the score function itself. Note that the score function at $\theta\_0=3$ is $s_{\theta_0}(z)=\nabla\_{\theta}\log p\_{\theta\_0}(z)=\nabla\_{\theta}\log\exp(-\sum\_{j=1}^{d_z} (z^{(j)}-3j)^2/2)=\sum\_{j=1}^{d\_z} j(z^{(j)}-3j)$. Even in the severely misspecified setting, all three methods have asymptotically $o(n^{-\alpha})$ decision bias and corresponding $o(n^{-\alpha})$ regret, i.e., $b^\textup{SAA}=b^{\textup{IEO}}=b^\textup{ETO}=0$ and $R^\textup{SAA}=R^\textup{IEO}=R^\textup{ETO}=0$. Intuitively, through this particular direction, regardless of the degree of misspecification, the ordering of the performance of the three methods is consistent with the mildly misspecified setting, where ETO is better than IEO and IEO is better than SAA. The experiments validate our theory and show our theory is robust. Not only the theory holds in the asymptotic regime but also we have seen the same finite-sample behaviours and trends for a wide range of sample sizes.

Q1 and Q2: As described in our responses to your comments above, we have provided and discussed new experimental results that aim to give a sense of the finite-sample manifestation.

We also want to point out that, in terms of building additional theory that sheds light on finite-sample behaviors, we can likely derive corresponding bounds, but it is unclear that this would reveal better insights than our asymptotic analysis. Please see our response to W2 for Reviewer JYdJ regarding the comparison and motivation of our work with [1] that focuses on finite-sample bounds.

[1] Elmachtoub, Adam N., Henry Lam, Haixiang Lan, and Haofeng Zhang. "Dissecting the Impact of Model Misspecification in Data-driven Optimization." arXiv preprint arXiv:2503.00626 (2025).

Note: the caption stands for the table above.

Table1: mild case, first direction

Table 2: balanced case, first direction

Table 3: severe case, first direction

Table 4: severe case, second direction

Table 5: mild case (b: bias; v: variance)

Table 6: balanced case

Table 7: severe case

Thank you for recognizing our significance and strengths, and for the thoughtful comments. We address your comments and questions as follows:

W1: We understand your comment and have run new experiments. Please refer to our added experiment part in the response to reviewer JBev.

W2: We have already introduced an important lemma, LeCam's third lemma in the appendix. We will add more explanations and references for the contiguity theory, if needed in the appendix. Specifically, we would start from the definition of contiguity for two sequences of measures $P_n$ and $Q_n$. Let $\{P_n\}$ and $\{Q_n\}$ be sequences of probability measures on a common measurable space. We say that $\{Q_n\}$ is contiguous with respect to $\{P_n\}$, written as $Q_n \triangleleft P_n$, if for every sequence of measurable events $A_n$, $P_n(A_n) \to 0 \quad \Rightarrow \quad Q_n(A_n) \to 0$. If both $Q_n \triangleleft P_n$ and $P_n \triangleleft Q_n$, then we say the sequences are mutually contiguous and write $P_n \sim Q_n$. We will introduce the criterion for contiguity (Le Cam's first lemma): Let $\{P_n\}$ and $\{Q_n\}$ be sequences of probability measures on $(\mathcal{X}_n, \mathcal{A}_n)$ such that $Q_n \ll P_n$ for all $n$, and define the likelihood ratio $Z_n = \frac{dQ_n}{dP_n}$. Then the following are equivalent: (1) $Q_n \triangleleft P_n$, i.e., $Q_n$ is contiguous with respect to $P_n$. (2) If along some subsequence, $\frac{dP_n}{dQ_n} \to U$ in distribution under $Q_n$, then $\mathbb{P}(U > 0) = 1$. (3) If along some subsequence, $\frac{dQ_n}{dP_n} \to V$ in distribution under $P_n$, then $\mathbb{E}\_P[V] = 1$. (4) For any sequence of statistics $T_n$, if $T_n \to 0$ in $P_n$-probability, then $T_n \to 0$ in $Q_n$-probability. We will introduce asymptotic log normality for two mutually contiguous $P_n$ and $Q_n$: $\log \frac{dP_n}{dQ_n} \xrightarrow{d} e^Z, \quad \text{under } Q_n, \quad \text{where } Z \sim \mathcal{N}(\mu, \sigma^2)$, then we have: (1) $Q_n \triangleleft P_n$, and (2) $P_n \triangleleft Q_n$ if and only if $\mu = -\frac{1}{2} \sigma^2$. We will review quadratic mean differentaiblity and local alternatives, definitions and properties of local asymptotic normality (LAN) in further details.

Q1: Let us respond to four aspects of your question:

1. We emphasize that in Theorems 1-6, while the bias and variance terms depend on specific expressions involving influence functions and score functions, the orderings of the bias and variance are universal among all cost function structures (influence functions) and parametric families (score functions). The regret ordering in the mildly and severely regimes are also universal among all influence functions and score functions. The only undetermined regime is the balanced regime, where the orderings of bias and variance are opposite, thus making the regret (the sum of bias and variance) ordering undetermined. However, we empirically demonstrate the regret ordering in the added numerical experiments and find that IEO may achieve the best bias-variance tradeoff and the lowest regret among all three methods in the newsvendor problem.

2. We only assume the smoothness of the cost function in expectation, instead of the cost function itself. The former assumption is much weaker and general than the latter. That is, the cost function itself can be possibly non-smooth in our framework. For example, the cost function of the newsvendor problem has kinks and is not smooth. Nonetheless, if the distribution is continuous, then the expected cost will be smooth and satisfy our framework.

3. If by "heteroskedastic" the reviewer means that the cost function taken in the expectation can have different noise depending on the decision, then yes our framework allows that. In fact, we don't even make an explicit distinction between "heteroskedastic" and "homoskedastic" in the paper.

4. Our framework focuses on nonlinear objective function, and does not apply to linear programming settings per se. However, we should point out that the latter setting is easier than our nonlinear setting from a statistical perspective. This is because if the cost function is linear, then it suffices to predict the expectation of the cost vector, which is a point prediction [1]. Because of this, fitting the whole distribution like we do is not needed. In contrast, our setting require the notion of model misspecification that is at the distributional level, since the nonlinearity of the objective function forces a more complex relation between the randomness and the decision variables.

Q2: Our framework can be extended to other settings. For contextual optimization, the extension is quite immediate. Specifically, we introduce the feature $\boldsymbol{x}$ in addition to the random parameter $\boldsymbol{z}$. We write the joint distribution of $\boldsymbol{x}$ and $\boldsymbol{z}$ as $Q(\boldsymbol{x},\boldsymbol{z})=Q(\boldsymbol{x})Q(\boldsymbol{z}|\boldsymbol{x})$. In this case we need to estimate $Q(\boldsymbol{z}|\boldsymbol{x})$ using parametric family $Q_{\boldsymbol{\theta}}(\boldsymbol{z}|\boldsymbol{x})$. Note that in the contextual case, the SAA method is not interesting because it allows to choose any map from context to decision, thus easily overfitting the finite-sample problem. In this case, IEO and ETO are the legitimate approaches. In the mildly misspecified case, ETO is better than IEO while in the balanced and misspecified case, IEO is better than ETO. For constrained optimization, if the constraints are deterministic, then we expect our analysis and insights to continue to hold, but it would introduce additional technical complexity from the KKT conditions instead of using simpler optimality conditions. On the other hand, if the constraints are stochastic, then there is an additional question of feasibility (e.g., in chance constrained settings where random safety conditions need to be enforced with high probability). These extensions to constrained settings warrant future work in our view.

[1] Elmachtoub, Adam N., Henry Lam, Haofeng Zhang and Yunfan Zhao. "Estimate-Then-Optimize versus Integrated-Estimation-Optimization versus Sample Average Approximation: A Stochastic Dominance Perspective." arXiv preprint arXiv:2304.06833 (2023).

Thank you for the detailed review and the list of thoughtful comments. In the following, we address all your comments/questions, including those in the proof details in the appendix. Regarding many of your presentation concerns, in case of acceptance, we will use the additional page allowed for the camera ready to move more intuition and definitions to the main body of the paper. Specifically, we will add Assumption 3 and 4, the explanation on line 969-970 and the numerical experiments in the additional page. Below, you can find our responses to each concern.

W1.a: No, the structural assumption in line 901 (exponential tilt, essentially Example 5) is NOT necessary to prove Lemma 4. Definition 1 is more general and sufficient. We apologize for adding the exponential tilt assumption and making readers confused. We will delete this redundant assumption in the revision.

W1.b: The equality comes from the quadratic mean differentiability property in standard asymptotic statistics (Page 94, Section 7.2 of [1]). Specifically, Definition 1 indicates that $\frac{1}{2}u(z)\sqrt{p_{\theta_0}(z)}$ is the derivative of the map $t\to \sqrt{q_t(z)}$ at $t=0$ (note that $q_0=p_{\theta_0}$) for almost every $z$. In this case, $u(z)=2\frac{1}{\sqrt{q_0(z)}}\frac{\partial}{\partial t}\sqrt{q_t(z)}=\frac{\partial}{\partial t}\log q_t(z)$ at $t=0$.

W1.c: In line 128, we assume that $\Omega$ is an open set, thus every point in $\Omega$ lies in the interior. Because of this, it is valid to set the gradient to $0$.

W1.d: In line 866, we assumed the existence of minimizers ($w\_t$, $w\^{\*}$, $w\_n^\*$) in the open feasible region in Assumption 3, which is a standard assumption in asymptotic statistics (assumptions in Theorem 5.7 and Theorem 5.23 of [1]).

W1.e: In line 906, once we get the functional limit for $t$, the sequential limit for $n$ is an immediate result (since $1/n^\alpha$ is a sequence that approaches to $0$ for all $\alpha>0$) from the sequential criterion for functional limits (Theorem 4.2.3 of [Abbott, 2015]). Please note that the notation for $\boldsymbol{w}_n^*$ and $Q_n$ already implicitly considers the order $\alpha$ in line 249-250 and Definition 2.

W1.f: Very similar explanations as the above hold for Lemma 5.

W1.g: Line 966 is the consequence of the former analysis (962-966) on the middle term. Line 967 is the beginning of analyzing the first term based on using definitions of $O_p(\cdot)$.

W2.a: Thanks for your suggestions. We have provided some descriptions on bias and variance in Theorem 1 and provided the exact definition for regret in Definition 3. We will make it clearer in the final version.

W2.b: We aim to demonstrate our own novel framework and results in the main body. We defer Assumptions 3 and 4 to Appendix (1) for the sake of page length; (2) since they are standard regularity assumptions in the existing asymptotic statistics literature. That said, we will use the additional page for the camera ready to discuss these in the body of the paper.

W2.c: Thanks for your suggestions. We have justified the validity of the assumption by citing previous literature in line 262. We will move further explanations (currently in line 969) here.

W2.d: We hope our explanations for W1.e answers these questions.

W2.e: Thanks for your suggestions. We will display the notations in a more concentrated manner in the final version.

W2.f: Lemma 2 is sufficient to prove using Example 5 and Definition 2 (they are already mentioned in the statement). Lemmas 4 and 5 are both based on the same assumptions in Definition 1 (and interchangeability assumptions). The difference is the result rather than the assumptions, where Lemma 4 focuses on $\boldsymbol{w}$ while Lemma 5 focuses on $\theta$. The limit is taken by sending $t\to 0$ thus also holds by taking $1/n^\alpha\to 0$ for any $\alpha>0$ and $n\to\infty$ (and thus for all $1/n^\alpha$ with $\alpha>0$).

W2.g: We recognize that Proposition 1 is mostly an existing result from previous literature [3] (that is why we call it proposition instead of theorem). We aim to demonstrate that the existing result can be extended beyond the well-specified case to the mildly-misspecified case. We will add the reference in Proposition 1 and some clarifying text around it.

W2.h: Definition 1 is to describe the relationship between the ground truth distribution and the parametric distribution (model class). Definition 1 is rather general and we have included many examples in the Appendix. We will make it more explicit by adding the description "the distribution $Q_t$ is a local perturbation of the parametric family $\{P_{\boldsymbol{\theta}}:\boldsymbol{\theta}\in\Theta\}$ at $\boldsymbol{\theta}_0$ if it satisfies for all $t>0$, $q_t$ is differentiable and satisfies the following quadratic mean differentiablity condition (line 194)" in the revision.

W2.i: We have extensively discussed the term "local misspecification" in related works section (line 112 to 124) with many classical literature in statistics, econometrics and machine learning, e.g. [2]. Specifically, [2] uses the terminology "local misspecification" in Section 2. We have also technically explained the terminology in line 206-217. We build a universal, general and rigorous local misspecification theoretical framework based on these literature and on the technical tools from [1], despite the fact that the original "local asymptotic normality (LAN)" is used for other settings. For these reasons, we believe our use of the term "local misspecification" and the relation to past literature are sufficiently clear, but we are happy to revise the exposition accordingly if the reviewer still sees the need.

Q1: Please see W1.a

Q2: Thanks for your suggestions. The exposition in Table 1 is primarily to give readers a rough sense of our results, before loading them with many technical details. We have stated in the caption that $\approx 0$ means ``asymptotically negligible", and we have described in our main results much more accurately how it is in comparison to other terms and how the relative scale may depend on different regimes. Specifically, in the severe case, we build a convergence in probability result. In this case, the limit is a fixed vector (or, say, a degenerate distribution with zero variance). If you feel strongly, we can directly state the convergence result in the table, but it might hurt readability a bit.

Q3: The technical assumptions including the invertibility of the Hessian/Fisher information matrix, twice continuous differentiability, and the full-rankness, mainly indicate the well-posedness of the problem and are often used in standard statistical literature (e.g. Theorems 4.6, 5.23, 5.39 in [1] have discussed the full-rankness, invertibility of Fisher information matrices and second-order Taylor expansion, etc.). We will add further discussions on the satisfaction of these assumptions in our application examples in Appendix A.1 (note that, on the other hand, we have provided many examples in Appendix A.2 regarding local perturbation, as this is the more interesting concept in this paper).

Q4: The notation $v(\boldsymbol{w},\theta)$ (not $v(\boldsymbol{w}, z)$ as you wrote) is a shorthand for $v(\boldsymbol{w}, P\_{\theta})=\mathbb{E}\_{P_{\theta}}[c(\boldsymbol{w}, z)]$. The notation $v(\boldsymbol{w}, Q_t)$ stands for $\mathbb{E}\_{Q_{t}}[c(\boldsymbol{w}, z)]$. They are consistent, but we will make it more clear in the revision. Specifically, we will stop using $v(\boldsymbol{w},\theta)$ and instead stick to $v(\boldsymbol{w}, P_{\theta})$.

Q5: Please see W2.e

Q6: Yes we will update all references, including [3] for which the reviewer has kindly noted there is an updated version.

Q7: We will fix this typo in the revision.

[1] Van der Vaart, Aad W. Asymptotic statistics. Vol. 3. Cambridge university press, 2000.

[2] J. Fan, K. Imai, I. Lee, H. Liu, Y. Ning, and X. Yang. Optimal covariate balancing conditions in propensity score estimation. Journal of Business & Economic Statistics, 41(1):97–110, 2022

[3] Elmachtoub, Adam N., Henry Lam, Haofeng Zhang and Yunfan Zhao. "Estimate-Then-Optimize versus Integrated-Estimation-Optimization versus Sample Average Approximation: A Stochastic Dominance Perspective." arXiv preprint arXiv:2304.06833 (2023).

I would like to thank the authors for going through my comments carefully. I have several remaining concerns/questions and will start with the ones that I found to be more critical. I may add more subsequently.

Re: W1.c: Your statement seems at least not very rigorous. Consider (the many variations of) the one-dimensional problems: $\min x: -1<x<1$ and $\min x^2: 1<x<2$. Here, the feasible regions are open sets. Assigning gradient of objective to zero does not work.

Re: W1.d: It is unclear to me where the existence of these solutions was stated in Assumption 3, nor is there seemingly explicitly mentioning of interior of the feasible region. Meanwhile, there is a mysterious $\epsilon$ defined but unused in clause 2 of that assumption.

Re: W1.e: Please provide some more relative detail. In particular, based on your notation, as $t$ changes, the sample size $n$ used to obtain, e.g., the SAA solution $w_t^{\ast}$ should vary accordingly. However, when you assign $t = 1/n^\alpha$, the notation $w_n^{\ast}$ does not appear to correctly represent the sample size for the corresponding SAA solution.

Re: Q3: My concern was that the simultaneous satisfaction of the conditions in the list may appear uncommon and hard to verify, even though some subsets of (variations of) those conditions have been discussed in the literature. To resolve my concern, I wonder if the authors could provide some examples of problems/functions meaningful to machine learning or data-driven decision-making in their response.

We thank the reviewer for the response. We would like to respond as follows:

Re: Re: 1.c: Since in Assumption 3 we assume the existence of the minimizer(or maximizer) that belongs to the open set, we can set the gradient to $0$. In your first example, the minimizer does not exist in the open interval $(-1,1)$. Our paper mainly focus on the case where the optimizer lies in the interior of the feasible region. The case is standard in statistical theory. As a remedy for the boundary case, potentially it is possible to extend the framework to the constrained case. In your example, we can take $x\in\mathbb{R}$ ($\mathbb{R}$ is an open set) and add two constraints $x+1\leq 0$ and $x-1\leq 0$. More generally, for the constrained case, we can add equality constraints and inequality constraints on the feasible set. To address the constraints, we can consider the Lagrangian function and corresponding Lagrangian multipliers. Under standard KKT conditions (such as Assumption A, B, C and D in [1], including complementary slackness, linear independent constraint qualification, and some assumptions on the relative interior, see Assumption B), potentially, the asymptotic behaviour of the three methods may still hold. This is left as our future research direction.

Re: Re: 1.d: We apologize for missing a notation in the clause 2 in Assumption 3. The revised version is : there exists $\zeta$ $\zeta\^\ast=\arg\max\_{\zeta}\mathbb{E}\_Q[m\_\zeta(z)]$, for all $\varepsilon>0$, $\sup\_{\zeta:\|\zeta-\zeta\^\ast\|\geq \varepsilon}$, .... The existence of these solutions is due to the existence of the maximizer $\zeta^\ast$. Also, we revise in line 877 "for IEO $m_\zeta(z)=-c(w_\theta,z)$" since we missed a minus sign there. For further technical details of Assumption 3, we refere to the assumptions in Theorem 5.7 and Theorem 5.23 in [2].

Re: Re: 1e: We recall the definition of $w\_n^\ast:=\arg\min\mathbb{E}\_{Q^n}[\frac{1}{n} \sum_{i=1}\^n c(w, z)]$, where $Q\^n:=Q\_{1/n^{\alpha}}\^{\otimes n}$ is defined in line 214 as the $n$-fold product distribution. By setting $t=1/n^\alpha$, now $w\_n^\ast=\boldsymbol{w}\_{1/n\^\alpha}$, by the definition of $w\_t$ in line 895-896. To be even more explicit, for fixed a $\alpha$, we have $w\_n^\ast=\arg\min_{w\in\Omega}\mathbb{E}\_{Q\_{1/n\^\alpha}}[c(w, z)]$.

Re: Re: Q3: We verify our assumptions by (1) citing a generic example and (2) providing a more concrete example. The cited generic example is from [3]. Consider the newsvendor problem $c(w,z)=a^\top(w-z)^++d^\top(z-w)^+$. We can see that Proposition 3 in [3] shows that the newsvendor problem where each product $j$ with demand distribution $N(\theta j, \sigma_j)$ satisfies their Assumptions 1, 2 (which corresponds to our Assumption 3) and Assumption 3 (which corresponds to our Assumptions 1 and 4).

For a concrete example, we consider the portfolio optimization problem with $c(w, z) = \frac{1}{2} w^\top (z - \mathbb{E}[z])(z - \mathbb{E}[z])^\top w - w^\top z + \frac{1}{2} w^\top w$, where $z \in \mathbb{R}^3$ denotes the assets’ return and follows a multivariate Gaussian distribution $z \sim \mathcal{N}(B(\theta), \Gamma)$ where $\theta \in \mathbb{R}$is an unknown scalar parameter, $B(\theta) = (\theta, 2\theta, 3\theta)^\top \in \mathbb{R}^3$, $\Gamma \in \mathbb{R}^{3 \times 3}$ is a known, positive definite covariance matrix. We get: $\mathbb{E}\_\theta[c(w, z)] = \frac{1}{2} w^\top (\Gamma + I) w - w^\top B(\theta).$ We define this expected cost as: $v(w, \theta) := \frac{1}{2} w\^\top (\Gamma + I) w - w\^\top B(\theta)$. We now calculate the gradient and Hessian. Let $Q :=\Gamma + I$, which is symmetric and positive definite. Then: $\nabla_w v(w, \theta) = Q w - B(\theta)$, $\frac{\partial v}{\partial \theta} = -w^\top \frac{d B(\theta)}{d \theta} = - (w_1 + 2w_2 + 3w_3)$ The Hessian of $v(w, \theta)$ with respect to $(w, \theta)$ has the blocks corresponding to our Assumption 1: $V=Q$ (invertible), and $\Sigma= -(1,2,3)$ (full rank). To minimize $v(w, \theta)$, solve the first-order condition: $\nabla_w v(w, \theta) = Q w - B(\theta) = 0$. Using $B(\theta) = \theta \cdot b$, where $b = (1, 2, 3)^\top$, we get: $w_\theta = \theta Q^{-1} b$. First derivative: $\frac{d w_\theta}{d\theta} = Q^{-1} b$. Second derivative is $0$. Fisher information: $\mathcal{I}(\theta) = \left( \frac{d B(\theta)}{d\theta} \right)^\top \Gamma^{-1} \left( \frac{d B(\theta)}{d\theta} \right)=b^\top \Gamma^{-1} b$, which is positive, thus invertible (it is a scalar).

[1] Duchi, John C., and Feng Ruan. "Asymptotic optimality in stochastic optimization." (2021): 21-48.

[2] Van der Vaart, Aad W. Asymptotic statistics. Vol. 3. Cambridge university press, 2000.

[3] Elmachtoub, Adam N., Henry Lam, Haofeng Zhang and Yunfan Zhao. "Estimate-Then-Optimize versus Integrated-Estimation-Optimization versus Sample Average Approximation: A Stochastic Dominance Perspective." arXiv preprint arXiv:2304.06833 (2023).

Thank you for your response; please see my questions regarding your second-round responses below:

Re: Re: 1.d: Thanks for your explanation and for pointing out how to correct the notations. However, if my understanding were correct, even after fixing the previous mistake in the notations, there seem to be several remaining notational issues. In particular, Clause 2 in Assumption 3 does not ensure the existence of optimal solutions within the interior of the feasible region in that there is no mentioning of feasible set in the statement.

Re: Re: 1e: In view of your response, could you explain why the conclusions before line 907 would be consistent with the second formula in Line 939--940?

Please see my additional questions regarding your first-round responses below:

Re: W1.b: Please provide proper references to the sources to improve clarity and give proper credit.

Re: W2.f: Definition 1 alone is not sufficient for Lemma 4 and 5. For an example, the existence of minimizers/maximizers in the interior of the feasible region has to be assumed.

One last additional comment:

Re: Re: Q3: It appears that neither of the examples (newsvendor and portfolio optimization) provided in your response satisfies all your assumptions. In particular, the newsvendor problem does not seem to admit an invertible hessian matrix and the distribution estimation problem in the portfolio optimization example does not admit an optimal solution in the interior of $\Theta$.

Again, I appreciate the author's time and effort in carefully responding to my questions and concerns.

This reviewer remains concerned that the presentation of Assumption 3 and the implicit assumptions claimed in the authors’ response, among others, reflect room for improving the rigor in the analysis and for adding essential expositions in the presentation. The reviewer is also concerned that the applicability of the results is either limited or not clearly articulated, due to the seemingly restrictive nature of the assumptions.

Re: Portfolio optimization problem: In view of the authors' response, the authors might revisit the proof of Lemma 5 using their example, where a linear cost function is involved. In this special case, the step of setting the gradient to zero appears problematic and warrant reconsideration.

Thank you. Based on the author's rebuttal, $\mathbb{E}_\theta[c(w, z)] = \frac{1}{2} w^\top (\Gamma + I) w - w^\top B(\theta)$, which is linear (affine) in $\theta$.

Thank you for your recognition of our significance and strengths, and your detailed review. We address your comments and questions as follows:

W1.1: Let us respond in three parts:

1. Our paper focuses on non-linear cost objectives, and our assumptions are in line with the standard statistics [2, 3] and optimization literature [4, 5, 6] that presume openness of the decision space and hence that an optimal solution is an interior point. Note that this assumption can be relaxed by adding equality constraints and inequality constraints -- We expect analogous results hold by utilizing KKT conditions, but the conceptual insights would be similar while adding significant technical complexity, and hence we leave this extension as future work.
2. The uniqueness assumption is also standard in nonlinear optimization; See [4, 5, 6, 7] (some of them even assume strong convexity). Note that this assumption may be weaker than you would initially think, because the uniqueness applies to the expected cost, not the cost itself. For example, in the newsvendor problem, a classic example in operations research, the cost function itself is piecewise linear and is not strongly convex. However, the expected cost is indeed strongly convex if the distribution of customer demand is continuous and has a lower bounded density, therefore, the optimal solution becomes unique.
3. In fact, the linear programming setting raised by the reviewer is easier than our nonlinear setting from a statistical perspective. This is because when the cost function is linear, it suffices to predict the expectation of the cost vector, which is a point prediction [1]. Because of this, it is not necessary to fit the whole distribution, as we must. In contrast, our developments require the notion of model misspecification that is at the distributional level, since the nonlinearity of the objective function forces a more complex relation between the randomness and the decision variables.

W1.2: Yes, the assumption on $Q$ is reasonable. The form originates from the local minimax theory and semi-parametric statistics in [2] (Section 25, Eq.(25.13)). In other words, it is a smoothness condition saying that the distribution $Q_t$ is differentiable at $t=0$ with score function $u$ in the $L_2$ norm sense. This condition plays a central role in ensuring that the log-likelihood ratio behaves locally like a quadratic form, which is the cornerstone of the local asymptotic normality (LAN) framework. On the other hand, in practice this misspecification direction is not known (otherwise, if it is known to the modeler, it would have been incorporated in the fitting so that the misspecification is reduced). Nonetheless, we highlight that, as a strength of our main results, the orderings of the bias and variance of the three methods are universal regardless of (almost) all directions $u(\boldsymbol{z})$, but only depend on the degree of misspecification. That is, the deviating directions $u$ of $Q$ turn out to be an invariant factor under most scenarios.

W2: First, note that the statement "there is no smooth transition in between that captures the impact of varying the misspecification amount." in Line 57 refers to [5], not [8]. Indeed, [5] divides the analysis into either misspecified or well-specified cases and, unlike the current work, there is no notion of ``local misspecification" there that allows the ground-truth data distribution and the model to be close in a certain asymptotic sense.

Now, as the reviewer rightly points out, [8] has a similar favor as us in that they consider the degree of misspecification. However, the derived bounds of [8] are opaque and loose. Specifically, their degree of misspecification is denoted by $\delta$ and $B_0$, which is problem and method dependent, is not interpretable and just for technical convenience. They use the so-called Berry-Esseen bounds, a uniform tool, to compare the performance between ETO and IEO by tail probability bounds, thus making the analysis coarse. Even though they can conclude the partial ordering between IEO and ETO, the involved conditions including the threshold for $B_0$ and $\delta$ in Assumption 4 and Corollary 1 are difficult to understand. In contrast, and in fact as a remedy to [8], we introduce the notion of local misspecification that allows us to explicitly derive asymptotic limits, dissect the bias and variance, and even understand the impact from the directions of misspecification that are well beyond [8].

Finally, we cannot set $B_0=1/n^\alpha$ in [8] because $B_0$ there is assumed (and required by the analysis) to be a fixed number that cannot be changed with respect to the sample size $n$. In our framework, however, we allow dynamics where, roughly speaking, the distance between the ground truth distribution $Q$ and the model $P_{\theta}$ has a comparable order of magnitude with the sampling error in estimating the parameter $\theta$. This, together with our developed asymptotic framework, empowers us to dissect bias-variance accurately, especially in cases where these errors are comparable (the interesting case), as well as directional impacts.

Q1: As explained in our response to W2 above, finite-sample results are interesting, but like in [8], they are anticipated to be intricate and do not offer the clean analysis that allows us to dissect bias-variance and the impact of misspecification directions. We do understand that there are merits for finite-sample bounds (just like in other ML problems), but at the same time it is also the case that these bounds very often are conservative and loosely depend on model factors. A tight enough finite-sample analysis that allows us to dissect bias-variance tradeoff meaningfully would be very interesting. Lastly, please kindly refer to our response to Reviewer JBev on our new experimental results that validate our asymptotic theory and its demonstration in finite-sample manifestation.

Typo: Yes we will add the norm for the three terms in our final version.

[1] Hu, Yichun, Nathan Kallus, and Xiaojie Mao. "Fast rates for contextual linear optimization." Management Science 68.6 (2022): 4236-4245.

[2] Van der Vaart, Aad W. Asymptotic statistics. Vol. 3. Cambridge university press, 2000.

[3] Shao, Jun. Mathematical statistics. Springer Science & Business Media, 2008.

[4] Duchi, John C., and Feng Ruan. "Asymptotic optimality in stochastic optimization." (2021): 21-48.

[5] Elmachtoub, Adam N., Henry Lam, Haofeng Zhang and Yunfan Zhao. "Estimate-Then-Optimize versus Integrated-Estimation-Optimization versus Sample Average Approximation: A Stochastic Dominance Perspective." arXiv preprint arXiv:2304.06833 (2023).

[6] Nocedal, Jorge, and Stephen J. Wright, eds. Numerical optimization. New York, NY: Springer New York, 1999.

[7] Qi, Meng, Paul Grigas, and Zuo-Jun Max Shen. ``Integrated conditional estimation-optimization.'' arXiv preprint arXiv:2110.12351 (2021).

[8] Elmachtoub, Adam N., Henry Lam, Haixiang Lan, and Haofeng Zhang. "Dissecting the Impact of Model Misspecification in Data-driven Optimization." arXiv preprint arXiv:2503.00626 (2025).