Distributionally Robust Performative Prediction

We thank the reviewer for the valuable feedback. We are glad to know that the reviewer appreciated our work from multiple aspects. We address the questions below, and we look forward to interacting with the reviewer during the discussion period.

“L77: it is easy to see..” Aren’t there cases where the optimization problem is still computationally, if not statistically, intractable?

The reviewer is correct that even when the optimization problem is statistically tractable, it can still be computationally intractable. The recipe laid out in L77-79 relies on knowledge of the distribution map and an argmin oracle. Without the argmin oracle, it is often still possible to obtain a local minimum (with say first-order methods). We will clarify this in a revised version of the manuscript.

Could you provide a formal statement showing why one would still be able to tractably solve this modified version of the problem, assuming that the first version is tractable?

To our understanding, the reviewer refers to PO as the first version of the problem, and DRPO as the modified version of the problem. As long as it is possible to at least solve the PO problem, then Algorithm 1 provides a way to reduce solving the DRPO problem to solving a sequence of PO problems.

Could you spell out in more detail exactly what the grid search is over when you state that L270 “for general problems, these calibration methods necessitate a grid search….”

For the second calibration method "calibration set", the grid search performs the three steps listed between L265 and L268. For the first calibration method "post-fitting calibration", the grid search executes the following procedure according to the equations between L258 and L259: for a sorted candidate set of $\rho$'s, $\rho_1 < \rho_2 < \ldots < \rho_m$, we try $\rho_i~(1\leq i \leq m)$ from small to large and return the first $\rho_i$ such that $\max\_{\mathcal{D}\_{\operatorname{true}} \in \Xi} \widehat{D}(\mathcal{D}\_{\operatorname{true}}(\theta\_{\operatorname{DRPO}}(\rho\_i)) \|\| \mathcal{D}(\theta\_{\operatorname{DRPO}}(\rho\_i))) \leq \rho\_i$.

We thank the reviewer for the valuable feedback. We are glad to know that the reviewer appreciated our work from multiple aspects. We address the questions and concerns below, and we look forward to interacting with the reviewer during the discussion period.

The novelty is simply in bringing together the DRO framework and performative prediction.

The components of our approach are not new, but we are combining them in a novel way to solve a relevant problem. The novelty has been discussed in Section 6 between L371 and L375.

One of the steps in the proposed alternative minimization algorithm for DRPR itself is a performative risk minimization problem, which needs multiple deployments to solve.

It depends on how one models the distribution map. When the distribution map is properly modeled (see the pushforward in Appendix E between L540 and L552), DRPR minimization doesn't require more deployments than standard PR minimization. For the most general case of distribution map modeling, Algorithm 1 needs additional cost of deployments.

Can you try to extend the result on regularizing effect of DRPR beyond the toy example in section 2? It seems like this should be possible.

We extend the uni-variate example in Section 2 to its multi-variate case in Appendix A between L487 and L494. Regarding the general form of the loss function and distribution map, it is not expected that there will be a concise and elegant closed form formula, similar to the one found in the toy example.

What is the effect of setting the parameter in tilted performative risk minimization to a negative value? It will be nice to compare this setting to the corresponding setting in TERM.

A negative tilt parameter in TERM suppresses the hard examples (the samples with high loss values) by assigning them less weight. When interpreting the hard examples as outliers, TERM with a negative tilt parameter is outlier-robust because it ignores the outliers to some degree. Similarly, tilted performative risk minimization with a negative tilt parameter can be viewed as an outlier-robust performative risk minimization approach. Unlike TERM, tilted performative risk minimization considers performativity. As a result, tilted performative risk minimization with a negative tilt parameter can be used when the distribution map (a family of distributions) is contaminated by a portion of outliers, but not just a single distribution. A negative tilt parameter, on the other hand, breaks the connection between the distributionally robust and tilted performative predictions. Because distributionally robust performative risk minimization focuses on hard examples but not ignoring them. In theory, the duality results necessitate a tilt parameter that is non-negative.

We thank the reviewer for the valuable feedback. We address the questions and concerns below, and we look forward to interacting with the reviewer during the discussion period.

Connection between numerics and theoretical guarantees.

The numerical experiments and theoretical findings support the benefits of using DRPO over PO from different aspects. The theoretical findings demonstrate DRPO’s potential to outperform PO in achieving lower performative risk at a single distribution map. The numerical experiments show DRPO’s capacity to attain a more favorable worst-case performative risk across a collection of distribution maps, in comparison to PO.

The paper proposes two algorithms for the distributionally robust framework. Which algorithm is used in each of numeric examples and are there benefits using one over the other? Or is one algorithm just a special case of the other?

The first experiment uses Algorithm 1 to find DRPO, while the second and third experiments use Algorithm 2 to find TPO. While both DRPO and TPO certify a certain level of robustness to misspecification of distribution maps, each has its own benefits over the other. On the one hand, DRPO exactly accounts for the uncertainty set size, whereas TPO does not. On the other hand, TPO is computationally more efficient, whereas DRPO requires more run time. The two algorithms are not special cases of each other. Their relationship should be understood as that the tilted performative risk minimization implicitly solves the corresponding DR performative risk minimization problem, and there is an implicit correspondence between the solution path $\theta_{\operatorname{DRPO}}(\rho)$ and $\theta_{\operatorname{TPO}}(\alpha)$.

Tightness of the theoretical guarantees? Can more insight be shed on the differences between the two excess risk bounds (prop 3.2 and prop 3.3)? In what settings are the two bounds and subsequently the excess risk significantly different?

The excess risk bound for DRPO (3.3) is sharp in the asymptotic regime $\rho \to 0$. This is a consequence of the sensitivity property of KL divergence-based DRO. The excess risk bound for PO (3.2) is sharp in its rate, but the constant is likely loose. The key insight from the two bounds is the excess risk for DRPO has a localization property (the leading term of the excess risk vanishes even for fixed $\rho$) which is not shared by the excess risk of PO. This suggests that the DRPO bound is sharper than the PO bound.

We thank the reviewer for the valuable feedback. We are glad to know that the reviewer appreciated our work from multiple aspects. We address the questions and concerns below, and we look forward to interacting with the reviewer during the discussion period.

What are the pros and cons of such approach? How general is it when the family of distribution maps is more complex than a linear function? When should one use this approach?

The cons of our approach is additional computational cost. Because the true distribution map is unknown, the learner must rely on a nominal distribution map estimated from data, resulting in modeling or estimation errors. Due to this reason, as long as the additional computational cost is affordable, one should always use our approach, as the pros of our approach is that it is robust to distribution map misspecification. The linear distribution map is a standard setting that people consider in the research area of performative prediction (see simulations in Section 5.2 of Perdomo et al. (2020)). However, our approach continues to work for more complex distribution maps.

Experiments are on semi-simulations and use very simple model. Is that common in this literature?

Yes, that is common in the literature. Our experiments align with the established convention in performative prediction research area, which utilizes synthetic/semi-synthetic data and simple model with almost no exceptions. A semi-synthetic dataset is often a real-world data simulator, which comprises real data as the base distribution and a synthetic performativity setup. The usage of semi-synthetic data is common in the literature because it is difficult to come by a dataset to learn the real distribution map.

There is little discussion of the challenges of this DR approach when distribution maps are more complex (non-linear), high-dimensional.

The theory and algorithms of our DR approach extend to the non-linear and high-dimensional cases. Conceptually, when dealing with non-linear and high-dimensional distribution maps, our approach can provide more benefits because distribution map estimation errors can be larger.

Example 2.1 is a bit ill-defined because neither u nor c depend on \theta. In Section 5, it is a bit clearer as it is instantiated on a simple example.

We will change $u(x)$ to $u_{\theta}(x)$ to emphasize that the utility function depends on the model parameter. As the reviewer pointed out, this is actually instantiated in an example in Section 5 (particularly Subsection 5.1).

In the end the Kaggle data you cite is only used to estimate a base distribution but then all data is simulated, right?

The reviewer is correct that the data is semi-synthetic: the base distribution comes from the Kaggle data, whereas the performativity setup is synthetic.

Figure 1 (left) shows that for a large interval of values of \epsilon_{true}, the performative risk obtained with the nominal map is actual lower than the distributional risk. I feel like the comment on this result l.

Figure 1 (left) shows the curves for some discrete values of $\rho$'s. We didn't show the result of an "infinitesimal" $\rho$ (i.e. a $\rho$ extremely close to $0$), of which the performative risk curve should be "almost surely" slightly lower than that obtained with the nominal map, except for the point at $\epsilon_{\operatorname{true}} = 0.5$. Now let's consider a continuous spectrum of $\rho$'s. For values of $\rho$ ranging from small to moderate, DRPO outperforms PO in terms of performative risk (and similarly for worst-case performative risk). Conversely, for large values of $\rho$, PO is better than DRPO. There exists an “sweet spot” of $\rho$ where DRPO yields maximal benefits over PO. This trade-off between DRPO and PO is demonstrated in any “vertical slices” of the left plot of Figure 1 (and similarly in the lines of the middle plot of Figure 1 for worst-case performance).

300-302 could be a bit expanded and insist on why this is typical for DR approaches.

As $\rho$ increases, the DRPO aims to achieve low worst-case performative risk over a wider range of $\epsilon_{\operatorname{true}}$. This requires a trade-off between regions of $\epsilon_{\operatorname{true}}$ with low and high performative risk. Figure 1 (left) shows that as $\rho$ increases, the DRPO achieves more uniform performance ("flatter" performative risk curve) across a wider range of $\epsilon_{\operatorname{true}}$.