Adjusting Pretrained Backbones for Performativity

Q1: “... It seems possible to model a function that corrects the classifier's logits without using a neural network module. What exactly is the neural network module learning in the proposed method?”

In our problem setting we only assumed that there is a relationship between the class-wise accuracies and the next distribution label probabilities (it can be any function that maps class-wise accuracies to label marginals). We model this cause and effect relationship by parameterizing a neural network.

Q2: “In real-world scenarios, what situations would the performative label shift tested in the experiments apply to”

We assume that the shift is determined by the model performance on individual classes.
Our main motivation are adversarial sampling strategies. Here, the model performance impacts the decision of a model developer about which datapoints to collect in the next round. The strategy could be adversarial with the goal to challenge models where they perform poorly as in dynamic benchmarks. Alternatively imbalance sampling could be done for the purpose of achieving fairness. It could also happen more passively without the intervention of the developer, but through a self-selection mechanism. Meaning that individuals that experience bad performance drop out of the system due to the limited utility they get from the service. These are all instances of our setup. Typically these mechanisms involve behavioral aspects and they are hard to explain exactly. But our approach can learn from experience some relevant aspects impacting model performance.

Q3: “What is the most common type of performative shift in real-world applications?”

The performative label shift on $P(Y)$ studied in this work is one of the most common types for applications involving image/text classification, as we discuss in Q4 and the paper.
For regression tasks on tabular data, perhaps the most common type is the shift on the conditional distribution $P (Y| X)$. The relevant applications include traffic systems where traffic forecasts influence drivers’ decisions and in turn the future traffic, stock markets where stock forecasts influence the stock values through the holders, and loan interests where individuals predicted to have a high default risk are assigned with a high loan interest and this in turn increases their propensity to default. But in image classification the concept shift is less natural. Please kindly refer to $1$ , $2$ for more examples in this setting.

$1$ Performative Prediction. Perdomo et al. ICML2020.

$2$ Anticipating Performativity by Predicting from Predictions. Mendler-Dünner et al. NeurIPS 2022.

W1.1 “Some of the definitions in the paper is clear”.

We have addressed your specific questions on definitions individually below. Please kindly let us know if there are any further doubts and we would be happy to clarify.

W1.2: “The benchmark dataset setting is not clearly stated.”

In Appendix (line 882), you can see all the Dataset Details. In section 4.1 (Performative Label Shift), we defined our sufficient statistics (Equation 5) and how we calculated the exact distribution shift given the sufficient statistic (Equation 6). In line 362, you can see the paragraph “Evaluation metric” explaining how the model’s performance on round t affects the sufficient statistic $S_t$ which leads to $P_{t+1}$. We repeatedly evaluated the baselines mentioned on lines 310-323 under the very same setting over 200 rounds.

Q1: “What is the clear definition of distribution shift?”

”Distribution shift” refers to the variation between the training and the testing distributions in general. A performative distribution shift can be a distribution shift of any kind, thus we use the general term distribution shift. While the conceptual idea of anticipating performativity based on recording a sufficient statistic applies to any distribution shift, we are very explicit that our primary focus is label shift. See background line 128, where we say:

“We primarily focus on label shift in this work. Label shift refers to the shift of the marginal distribution P(Y), while the class conditionals P(X|Y) remain fixed (Manski & Lerman, 1977; Storkey et al., 2008; Zhang et al., 2013; Lipton et al., 2018).”

Q2: “What is the definition of label marginals, how do we calculate it?”

Label marginals refer to the marginal distribution of the labels P(Y) and marginalizing out the data X. It is computed empirically by counting the occurrence of each class in a given dataset.

Q3: “What is the memory M in the algorithm 1? I still did not get why we need M”, “I did not get the point how did you get the gradient to update T”

We kindly point the reviewer to line 269:

"… we collect the statistic $S_{t−1}$ of the deployed model $f_{t−1}$, together with the induced label marginals $Λ_t$ over $P_t$ and store it in a memory buffer to learn the predictor T in a supervised manner."

Having a memory buffer is a common technique for online learning algorithms and is also widely used in reinforcement learning. By storing the previously seen samples, we can create an auxiliary dataset and treat it as a classic supervised learning setting rather than online learning. Therefore, the memory buffer is our dataset and we apply simple mini-batch updates iterating the dataset.

Q4: “In Table 1, PaP estimated performance is reported here. How did you calculate it?”

A detailed explanation of the results in Table 1 is given in the paragraph “Anticipating performativity.” (line 468). Briefly, we created balanced holdout sets on CIFAR100 (for Table 1) and on ImageNet100 (for Table 5), and trained a pool of models where we mainly change the number of epochs and random initializations. Please also refer to the common response.

To estimate the post deployment performance, we use PaP to estimate the label probabilities on the next round. Then, we sample instances from the current distribution proportionally to the predicted label probabilities, in order to simulate the next shift. Using these sampled instances, we can estimate the future performances of the models without observing the shift. We will clarify this in the draft.

Thank you for your dedicated time to reviewing our manuscript. We address your comments individually below and indicate our edits thanks to your feedback. We hope that these have increased the readability of our paper and please let us know if any confusion remains. Thank you!

Q1.1: “it's unclear what "sufficient statistic" refers to.”

Thank you for the suggestion. We updated the document by providing an example statistic earlier.

Q1.2 “ “have trouble understanding what to take from Table 1 and how to interpret the ranking.”

We hope the general response was able to clarify this. We updated the caption so that it is more self-contained.

In summary, the table shows that the performance of models pre-deployment phase can be misleading for assessing the performance post-deployment. Depending on the self-induced shift, the ranking before the shift and after the shift can change, which creates the need of accounting for the shift. Here, we show that PaP’s estimates are aligned with the model performances after the shift and gives a good proxy on which model will perform the best.

Q2: “what is the key advantage of the adapter since it does not work proactively either.”

There might be a misunderstanding here. PaP estimates the label probabilities on the next round without observing the next round samples. Thus it can anticipate what update it is going to do to the backbone’s predictions before moving on to the next round. The results in Table 1 demonstrate this ability to foresee shifts. Model 1-Model 3 are models that have not been deployed during training and still PaP can reasonably predict their post-deployment performance.

Q3.1 and Q3.2: how does the parameter "round T" affect the final adapter from the training stage? Given that in Algorithm 1 the output is one single adapter module at time T, is this adapter used to test throughout the 100 rounds shown in Figure 2?

On Figures 1 and 2, we initialize PaP at round 1, and train it in an online manner while performing the model updates. The figures show the online performance, and we see it gets stable very fast – after the sudden drop at round 10, it quickly recovers. Therefore, the effect of T on the adapter quality is visible on the trajectory. We choose this setting purposefully to showcase how our adapter accommodates the distribution shift with only a few rounds.

Q4: “ some conjectures / explanations for the drastic drop in Oracle finetuning at Round 1 from Figure 2 and 3”.

Figure 7 on appendix shows an ablation for Oracle-finetuning baseline with the following settings:

• 5 epochs for finetuning
• 40 epochs for finetuning
• Training only the last linear layer

However, these models are very sensitive to the distribution shifts too, mainly due to the fact that the finetuned datasets (self-induced distributions) are imbalanced.

As an extreme case, If we increase the severity of the performative shift, the distribution can even oscillate between sets of classes which are disjoint, resulting in very poor performance as the model will not see the corresponding classes during its finetuning stage.

Q1: “in Lines 362-363: what exactly is the 'retraining trajectory'?”

The “retraining trajectory” refers to the repeated training and evaluation of a specific model over multiple rounds/steps. Since the model’s training induces new distribution shifts for the next round, which in turn influences its evaluation performance on this new distribution, the performance varies over rounds. We are interested in this entire process.

We have added the text “... trajectory (i.e., the repeated training and evaluation over multiple rounds/steps)” to line 363 to clarify this. Please let us know what you think.

Q2: “how is this definition of 'Acc' distinct from 'Acc in S'??”

Note that we can only prepare the model using the current distribution $P_t$, but we always report the performance on $P_{t+1}$ since, when we deploy the model, the distribution changes.

Therefore, Acc denotes the accuracy of the model on the deployment distribution, signaling the start of the next round. On the other hand, class-wise accuracies in S refer to the model's performance in the current round. Reported accuracies on the Figure 2 and 3 always demonstrate the model’s performance on the shifted distribution.

Q3: “in Algorithm 1, does 'Deploy' imply training on the sample, and does 'Observe sample' indicate the same?”

The term deploy means releasing the model to the environment, and evaluating it on whatever distribution arises. Observe $P_t$ refers to the part where one can prepare the model for the future distribution $P_{t+1}$ using current data obtained from the distribution $P_t$.

Q4: “in Section 3.2, what is the purpose of the discussions on 'Dynamic benchmarking' and 'self-selection'?”

These are two real-world examples where performative shifts arise naturally, that we wanted to discuss to showcase where our method could be applied.

• Dynamic benchmarks are an example of a model-induced distribution shift. On these benchmarks, the model is put in a feedback loop over different distributions and iterated finetuning is applied to obtain a better performing model. Here, PaP could be used to mitigate the model-induced shift in a very lightweight way, which has interesting implications for this practice.
• Self-selection is presented similarly to support an application where PaP is relevant. Model’s poor performance for subgroups can result in disengagement of users. Instead, PaP can be used very naturally to balance underrepresented classes/groups on the distribution by its reweighting strategy.

Q5: “First, what are the results for other datasets in Table 1?”

We have included experiments on ImageNet100 in the Appendix (line 1047). The results are consistent with the reported results and confirm the benefit of PaP for model selection.

Q6: “which pre-training datasets were used, and how might the distribution of the pre-training data impact the results?”

We created balanced holdout sets on CIFAR100 (for Table 1) and on ImageNet100 (for Table 5), and trained a pool of models where we mainly change the number of epochs and random initializations.

The distribution of the pretraining data would affect the model performance more drastically if it is now balanced. In our experimental setting, we sample from classes inversely proportionally to the model’s performance. Imbalanced datasets can lead to a poorer performance on particular classes, resulting in a more severe shift. Therefore, the gap between the pre deployment performance and post deployment performance increases.