Revisiting the Predictability of Performative, Social Events

The authors consider a classic problem in social science--how can we make accurate predictions about the world if our predictions affect the world--from a learning theory perspective. In this setting, there are features $x$, a binary outcome $y$, and we wish to make probabilistic predictions $f(x)$ to predict $y$. However, once we pick $f$, the world changes so that $(x, y) \sim \mathcal D(f)$, and outcomes now depend on our predictions in an arbitrary way (the performative prediction setting). Classic work using fixed-point analysis shows that there exist predictors $f$ such that $f(x)$ predicts $y$ when $(x, y)\sim \mathcal D(f)$, but does not provide a method of computing such predictors. Here, the authors adapt online multicalibrated algorithms to performative prediction. They show that it is indeed possible to efficiently learn a predictor $f(x)$ that is (multi)calibrated (on whichever subsets of $x$ we care about) with the induced outcomes drawn from $\mathcal D(f)$. Moreover, they show a connection to outcome indistinguishability, so that their predictor's outputs are computationally indistinguishable from the world outcomes. On the other hand, through a construction, the authors also demonstrate that the learned predictor can utterly fail to explain the variance in $y$, despite being perfect calibrated. Intuitively, they show that in some cases, the only way to achieve calibration is to steer the world into higher variance outcomes, where are predictions are essentially useless (despite being calibrated). In contrast, if we were willing to accept some small bias in our predictions in this construction, we could steer the world into highly predictable outcomes (consistently y=0 or y=1).

Update after rebuttal

I continue to think this is a very nice paper and I'm glad the other reviews agree on acceptance.

This paper formalizes the question of whether predictions can remain accurate when the act of predicting affects the state of the world. The authors address this question theoretically and show that a predictor can maintain some bounded level of calibration/validity. The paper provides a bound on how far a predictor might be from accuracy and show that high quality predictions can be found in polynomial time. The paper ends by showing that while the calibration criteria which has been used through the paper has very bad worst-case performance.

• This paper investigates multicalibration problems in performative settings.
• The models assume the performative prediction framework, where data distribution depends on the deployed model $(x,y)\\sim \\mathcal{D}(f)$.
• The main result is a convergence bound for the performative multicalibration loss, achieved through an online-to-batch reduction (Theorem 3.4). The result is modular, converting online learning algorithms with multicalibration guarantees into batch algorithms with performative multicalibration guarantees.
• Section 4 shows that predictors which are approximately performatively multicalibrated are also approximately performatively stable for the quadratic loss function. Finally, Section 5 presents a construction showing that perfectly multicalibrated classifiers can have the worst possible performance with respect to the quadratic loss.

Update after rebuttal

Thank you for the clarifications, and I look forward to seeing the improvements in the next revision of the paper.

The paper at hand claims to explore the predictability of socil events. Social predictions do not merely describe the future. They also influence it. Such predictions can affect market prices, voter behavior, and policy outcomes. This interaction complicates the ability to forecast accurately. Early theorists, including Morgenstern and Simon, discussed these challenges extensively. This paper addresses these questions anew, using recent concepts from machine learning. Specifically, it utilizes the framework of "performative prediction," where predictions themselves shape outcomes. The authors establish that accurate forecasting of binary social events remains computationally feasible despite these dynamics. A key contribution is demonstrating the existence of predictors that remain accurate even as they actively shape events. These predictors satisfy strong conditions like multicalibration and outcome indistinguishability. Algorithms to achieve these conditions are presented, which are efficient both statistically and computationally.

However, the paper also identifies critical limitations. Although accurate predictions are always achievable, they might not always lead to desirable outcomes. Calibrated predictions may sometimes create poor social equilibria. Such predictions could, paradoxically, maximize prediction error measured in terms of performative risk. Thus, the paper shows a tension between accuracy and social desirability in forecasting social events. The authors conclude by suggesting a reconsideration of historical forecasting methods. They call for further research into what goals predictions in social contexts should ultimately serve.

This paper studies the predictability of social event within a modern performative prediction setting. The paper theoretically establishes positive results for the binary case within this setting, while highlighting its limitations. Authors were generally positive about the paper, were appreciative of the aim of recreating classic results in contemporary settings, and found the theoretical results to be sound and interesting. Some reviewers encourage the authors to better frame their contribution in the precise setting in which results hold, namely stateful performative prediction, rather than a general thesis about the predictability of social events in general. Statefulness in particular seems to be a common simplifying assumption in the performative prediction literature, but that does not necessary apply broadly. Several reviewers found clarity to be improvable on several fronts – the authors are encourage to take these into account in their next revision. Adding a final discussion section is also advisable as it is the norm for almost all ICML papers.