End-to-End Learning under Endogenous Uncertainty

Thank you very much for the thoughtful comments, reviews and time carefully reading the paper. These have greatly improved the paper. We would like to address each of your points, and have also revised the paper based on your suggestions.

1. We agree this is not strictly end-to-end in the same way as existing work like those you cite. We also considered the formulation you described when we first began work --- this is definitely the most direct connection to current end-to-end work. Unfortunately, it is not solvable as it is. Specifically because we cannot evaluate the expectation as the distribution $Z(\hat{w}(f(x)), x)$ is unknown and we have no data for it. This approach would require knowing the counterfactual distribution for a different action taken. In exogenous case, this distribution was only $Z(x)$ and hence historical observations sufficed but here it depends on actions $w$.

While our paper does have strong connections to offline policy learning, our approach still takes ideas from the traditional end-to-end methods. Specifically, that ${\hat{f}_{end-to-end}}$

aims to make the best prediction (of the random variable $z$) that results in the best approximation of the reward. In the traditional end-to-end setting, $\hat{f}_{{end-to-end}}$ aims to make the best prediction that results in the best decision. This was the connection we were trying to express.

2. Our claim that "optimally minimizing our proposed objective produces optimal decisions"is perhaps too generic and we have re-framed this in the revised paper. However, optimally solving for $f^*$ to achieve equation (2) indeed produces optimal decisions. This is primarily what we referred to.

Overall, we restructured section 3 and placed the results in the previous section to better convey the meaning. First, we show that $f^*$ as described in equation (2) can exist in the first place. This is given by proposition 3.1 (now proposition 2.1 in the revised draft). Second, the hypothesis class likely does not contain $f^*$ but we can increase its complexity to get better results and be closer to $f^*$. Theorem 3.2 (now 2.3) explains the tradeoff to increasing hypothesis class complexity when we have limited data. We hope the revised paper provides this clearer message.

We agree our paper does not make any theoretical guarantees in regards to learning "optimal decisions", although this would be an interesting future direction. We have included many new experiments to at least empirically show strong performance.

Thank you for bringing up some of the smaller issues in the theorem. Indeed, the expectation should be over $x$ as well. Second, you make a good point, we do implicitly assume that $z$ is bounded (even in the examples we give, it would be unnatural to have unbounded demand for instance). We have amended the paper with these points.

Thank you for your quick response and very helpful feedback. We have amended the computational results with your comments. To address each of your points:

points 1/2) Specifically, we run each experiment on 20 datasets, and report percent error compared to the optimal solution as well as standard deviations.

We would like to note that we use the same model architecture for each approach. So, our method does not use a simpler model. Using the same complexity model, we find our approach performs better. Your comments also gave us the idea of comparing method performance as we increase the complexity of the predictive model --- in this case the width of the hidden layer of the network. We see that the reward-learning methods requires 500-1000 hidden layer width to achieve the same accuracy as our approach which only needs width 30.

For the noise experiments, there was likely some mistake in the implementation, and we have fixed this. Indeed, as noise increases performance becomes worse for all methods. We still observe our method performs better than the remaining methods at higher noise values. For the reward learning method we compared against both a network of width 50 (the same we use for our approach) and one of width 500. Even with width 500, the reward-learning method under-performs.

point 3) We decided not to include the MIP formulation since it is quite expensive to compute, and from the experiments we find simply using gradient descent instead nearly optimally solves the problem. We moved this to the appendix in case it is of interest.

point 4) We agree that our method may not be best-suited when there is sampling bias. This may be remedied by existing work in the literature, which for instance constrains the decision-space to be "in-distribution" with the training data. This would maintain the accuracy of our method on the constrained space of decisions. This is similar in idea to offline reinforcement learning work which constrains the distance between state-action visitation of the policy and the state-action pairs contained in the data. See for instance [1]. This is perhaps out of scope of our paper, although we are happy to add discussion on this point in the paper.

point 5) The new results don't exhibit any strange behavior. As data increases, all methods perform better overall; as noise increases, all methods perform worse.

We believe section 3 is an interesting application -- for instance, it combines with the traditional exogenous end-to-end method as the endogenous aspect appears due to the two stage nature of the problem. This type of problem has not been studied in the literature as far as we know. We could potentially move this section to the appendix if other reviewers feel it is distracting. To answer your question, yes one can simply learn the value/reward/cost of making a waiting decision (this is the reward-learning approach we compare against). Similar to the setting considered in the previous part of the paper, there is value in making an intermediate prediction of what the observed samples would be.

[1] Off-Policy Deep Reinforcement Learning without Exploration. Scott Fujimoto, David Meger, Doina Precup

We appreciate your time in carefully reading the paper and your thoughtful comments. These have been helpful in restructuring parts of the paper to make it clearer and easier to read. In addition, we would like to let you know of some new computational results we ran as suggested by other reviewers. We hope you find these interesting as well. We discuss this in the common response as well as in the revised draft.

To address each of your points:

1. Our approach is indeed focused on more structured decision-making problems where objectives, constraints etc. are well-defined. We've made this more clear from the beginning of the introduction to better place the setting of the paper. All of the examples we give from the start of the introduction fall within the setting we consider in the rest of the paper.

2. This is an interesting area of work, thank you for bringing it to our attention. We have included a short discussion on this in the related works section. In this area of work, a performative prediction is one which affects the outcome/distribution of the target. For example, this area would focus on problems like how traffic predictions will affect driver behavior and hence affect traffic itself. This is similar to our setting where instead the decision we take affects the distribution of the target. Another difference in performative learning is the online aspect whereas we consider the offline setting.

3. Yes, for the optimal solution to the optimization problem in (3) to be the wanted $f^*$ which satisfies (2), we do need to make the assumption that $f^*$ is contained in our hypothesis class. Proposition 3.1 (now proposition 2.1 in the revised draft) shows that $f^*$ exists in the first place, although it may fall outside the hypothesis class. Nevertheless, solving problem (3) still brings value. In the new experiments we consider the mis-specified setting and see strong results. We have added this as a point of discussion in the paper following equation (3).

4,5,6) The non-convexity of the objective function is an issue from the point of view of solving the problem to optimality. The integer program is mean to address the issue. However, we agree section 2.1 feels somewhat out of place. After running fairly extensive experiments, solving equation (3) simply by gradient descent (in spite of non-convexity) performs very well, and very simple to implement compared to the other ideas in section 2.1. For better clarity and continuity of the paper (as well as to save space for more computational results) we decided to move section 2.1 to the appendix.

7. Thank you for pointing this out, section 3 indeed does not flow particularly well. We have moved each proposition/theorem to a more appropriate place within the paper to connect better. In particular, we describe the following flow.

First, that $f^*$ as described in equation (2) can exist in the first place. This is given by proposition 3.1 (now proposition 2.1 in the revised draft). Second, the hypothesis class likely does not contain $f^*$ but we can increase its complexity to get better results and be closer to $f^*$. Theorem 3.2 (now 2.3) explains the tradeoff to increasing hypothesis class complexity when we have limited data. We hope the revised paper provides this clearer message.

8. In the information gathering setting we consider a slightly different problem than the initial endegenous setting. But we show that it can reduce to the endogenous setting. In the information gathering setting, $z$ is independent of the actions taken (in the numerical experiments, $z$ is electricity of demand for each of the next 24 hours). However, we can query to gain information about $z$. In the experiment, this amounts to waiting $t$ hours and observing the first $t$ entries of $z$. The first decision (the amount of time $t$ to wait) can be solved by our approach in the endogenous setting. Intuitively, this is because the observations we make on $z$ depends on our decision $t$. Similarly in the endogenous setting, the distribution of $z$ depends on our action.

In general, the information gathering setting assumes we can query to gain information about some entries of $z$.

Thank you for your time in reading the paper and the constructive feedback. It has been very helpful in improving the paper. We have taken your advice on running more experiments across multiple datasets, and across a wider range of parameters. We wrote a more detailed description of these new results in a general response, since it would be of interest to all reviewers as well. Please see the revised draft of the paper as well for details.

In terms of benchmarking against other end-to-end approaches as you mentioned, this is not possible since they are not applicable to our endogenous setting. We have a discussion of why this is the case in section 2.2. Intuitively, every existing end-to-end approach, including all that you cite, one needs to train by being able to evaluate the cost of any decision for a given training datapoint/observation (exogenous demand setting). However, in the endogenous uncertainty setting we consider in this paper this is not possible because the distribution of the random variable changes for every decision. This is in large part the reason we studied this endogenous problem and proposed our method.

We did add another benchmark and compared against a guassian process method which learns to predict the cost of a decision directly (skipping the intermediate prediction of the random variable).

Thank you for your time in reading our paper. We agree that further experimental results are crucial to demonstrate the performance of the method. We have done our best in the short review period to add an extensive set of new results. Please see our general response as well as the revised draft of the paper for details.

If there are any other specific methods you would like us to compare against which can be applied to our problem setting, please let us know. With the new computational results, we consider 5 benchmarks. (1) a predict-then-optimize method, also known as a two-stage method, which train a model to learn the uncertainty (demand in this case) as a function of actions (inventory in this case). (2) A cost-learning method which trains a model to learning cost directly as a function of $\mathbf{w}$. This does not take into account the intermediate demand data or the structure of the cost function. It only observes the final cost of a decision. (3) Similar to method (2), we predict cost using a gaussian process method instead. (4) A K-NN-based method which, for any action, approximates cost by computing the average of the costs of the $K$-nearest neighboring actions. (5) A K-NN method which instead predicts demand. This is a fairly broad range of methods, both considering predicting intermediate demand as a function of actions, learning cost directly as a function of actions, and using both parametric and non-parametric methods for each.

We would like to update the computational results based off of new experiments as suggested from follow-up conversation with reviewer tP9r. Please see the revised paper.

We now run each experiment on 20 datasets, and report percent error compared to the optimal solution as well as standard deviations. Using the same complexity model, we find our approach performs better. We also run a new experiment comparing method performance as we increase the complexity of the predictive model --- in this case the width of the hidden layer of the network. We see that the reward-learning methods requires 500-1000 hidden layer width to achieve the same accuracy as our approach which only needs width 30.

Similarly for the noise experiments, we still observe our method performs better than the remaining methods at higher noise values. For the reward-learning method we used both a network of width 50 (the same we use for our approach) and one of width 500. Even with the width 500 network, the reward-learning method under-performs compared to our approach. We hope these results strengthen the claim that our approach can perform better or as well as other methods while using a lower-complexity model.