Long-Term Impacts of Model Retraining with Strategic Feedback

Thanks for your comments. Please see our responses point by point.

Weakness 1: The paper is incremental in proposing solutions to biases and unfairness it uncovers.

Answer: The key of the section on fairness is to reveal the long-term fairness dynamics, where we discover that even when the systematic bias of human annotation always exists, simply retraining both groups for several rounds will lead to a fair classifier without any other explicit corrections. In the updated draft, we first provide an analysis of the long-term fairness dynamics under the retraining process when the two groups have different initial qualification rates (as suggested by Reviewer QsNX) and then analyze the dynamics of $a_t, q_t, \Delta_t$ while applying fairness intervention at each round on both original/refined retaining process. All details are reflected in Sec. 4 and App. D.2 of the updated draft.

Weakness 2: Simplifying assumption that the joint distribution $P_{XY}$ is fixed over time, neglecting user retention and representation disparity effects.

Answer: In this paper, we mainly focus on retraining with different data sources and strategic agents, which is a novel setting that was never considered by previous literature. Adding participation dynamics may indeed model reality better, but it will add more complexity to the model, which may be a future direction.

Question 1: Justification for Assumption 3.1 regarding algorithmic bias and its variation with $D_t$

Answer: When we consider a specific class of models, the algorithmic bias may not be negligible in practice, but the focus of our paper lies in the effects brought by the retraining process. Instead, algorithmic bias has already been extensively studied in the statistical learning theory and it is inevitable to some degree. With model-annotated samples, the sample complexity becomes very large, which already makes algorithmic bias as small as possible (the remaining bias is caused by the distribution itself). Our empirical results contain algorithmic bias, and it still demonstrates the validity of the theoretical results.

Question 2: lower limit in Thm 3.5

Answer: Due to the monotonic likelihood assumption, the best response will not make agents' qualifications lower than $q_0$, so the lower limit will be $q_0$.

Question 3: why does the decision-maker use one-shot classifier?

Answer: There are frameworks considering using repeated risk minimization to update a single classifier in long term [Perdomo et al., 2020]. However, the training strategy needs the loss function to satisfy rigorous assumptions and is not likely to converge to an optimal point. Thus, as a more spontaneous solution, the decision-maker tends to gather data each time to update the classifier, which is the common practice nowadays. Our paper aims to investigate whether this practice will bring unexpected outcomes.

Question 4: Implementation details about the early stopping mechanism in the presence of unknown systematic bias and group identities.

Answer: In practice, although the decision-maker may not know the exact magnitude of systematic bias, but it usually has sufficient prior knowledge of the group identities and the direction of the systematic bias. For example, in a college admission scenario, the admission may already have good knowledge on which ethnic group (race) is historically underrepresented, thereby assuming a negative systematic bias exists.

Question 5: Conclusion that training solely with human-annotated samples could be more beneficial?

Answer: In our current experiment, we assume the decision-maker can obtain 2000 human-annotated samples each round, and the underlying distribution $P_{XY}$ is relatively easy to learn. In practice, the availability of human-annotated samples may be more limited, making the algorithmic bias significantly larger due to the low sample complexity.

Thanks for your comments. We provide our response point by point as follows.

Weakness 1: Assuming groups have equal base rates is entirely unrealistic. It would be interesting to have explored how strategic feedback could've helped with improving fairness using different base rates.

Answer: The assumption that groups have equal base rates is only used in the discussion of section 4 on fairness. In particular, we are concerned about how retraining with systematic bias under the strategic setting will influence fairness given the two groups are equally qualified and should be treated equally, so we make the equal base rate assumption. Furthermore, Thm 4.1 holds when group $i$ has a base rate larger than or equal to the one of group $j$ (not necessarily equal). For other situations under unequal base rates, we discuss it in detail in the general response to reviewers.

Weakness 2: The proposed solution for fairness can't be "stop retraining the model". And relying on the same decision-maker to self-identify and correct acceptance bias is not ideal

Answer: The key of the section on fairness is to reveal the long-term fairness dynamics, where we discover that even when the systematic bias of human annotation always exists, simply retraining both groups for several rounds will lead to a fair classifier without any other explicit corrections. Instead of implementing the early stopping mechanism, we can consider a more common setting where a "hard fairness intervention" is performed at each round to only let the decision-maker choose from fair models. This is also discussed in detail in the general response.

Weakness 3: Early stopping can improve DP because the retraining process increases acceptance rate, showcasing modelling flaws.

Answer: What we hope to show in Sec. 4 is that although the acceptance rate increases as the retraining process goes on, it is possible to have positive impact on fairness if we apply the early stopping mechanism.

Weakness 4: Dataset and real-world applications

Answer: The theoretical model needs us to know $P_{XY}, P_X$. This is why we need to estimate the distribution through the KDE estimator. The common datasets for strategic learning settings are limited in samples, so many previous works have a protocol to generate samples. Moreover, except for the uniform synthetic dataset, all other datasets violate some of the assumptions in the main paper, and the results still verify the validity of our theoretical results.

Question 1: Why does the qualification rate only change for model-annotated samples under strategic feedback and not for human-annotated samples?

Answer: We assume the human-annotated samples are drawn from prior-best-response distribution $P_{XY}$ because the human annotation process is independent of the decision-making process. At each round $t$, each human agent is classified by the model $f_t$ and best respond to $f_{t-1}$, the decision-maker never confuses the agents by simultaneously using human experts to label agents. Instead, human experts never participate in the interaction and human annotation is another process for the decision-maker to obtain information about the whole population by acquiring data from public dataset or some third-parties, and then label them to recover the population distribution. This distribution is free of best response because the samples are not subject to any decision. To better illustrate this idea, consider an example as follows. In a loan application scenario, the bank obtains personal finance data from credit agency and other demographic data from government to produce human-annotated data, while they keep using an automated decision model to reject or approve loan application and update this model with the aid of human-annotated samples. We also agree with the interesting nature of considering the human-annotation for post-best-response distribution and our model can incorporate this setting. Details are discussed in the general response.

Thanks for your comments. Please see our responses point by point as follows.

Issues in Section 4

Answer: The objective of section 4 is to study the effects on fairness solely caused by the retraining process with strategic agents, so we consider the setting where the two groups have no "innate" qualification rate difference (equal base rate). Thm 4.1 reveals that even when the systematic bias of human annotation always exists, simply retraining both groups for several rounds will lead to a fair classifier without any other explicit corrections. Regarding the questions on more substantive results on fairness, we can first provide an analysis of the long-term fairness dynamics under the retraining process when the two groups have different initial qualification rates (as suggested by Reviewer QsNX) and then analyze the dynamics of $a_t, q_t, \Delta_t$ while applying fairness intervention at each round on both original/refined retaining process. All details are reflected in Sec. 4 and App. D.2 of the updated draft.

Convexity in Thm 3.5

The convexity of $F_X$ and $P_{Y|X}(1|x)$ will ensure that no matter where $f_t$ is and the cost matrix $B$ is, $q_t$ will always be decreasing. If any of the convexity constraint is not satisfied, $q_t$ will not be guaranteed to decrease. Consider an example on 1-d feature space where $P_X = 2 - 2x$, $P_{Y|X}(1|x) = x$, $B = 0.1$, and all new samples are model-annotated. Ideally, $f_0 = f_1 = \mathbf{1}(x \ge 0.5)$. Then agents having $x \in [0.4, 0.5)$ will best respond. However, at $t = 2$, since all agents are labeled 1 when $x \in [0.4, 0.5)$, $f_2$ is likely to set a threshold smaller than $0.5$. Assume the threshold is $0.5 - \theta$, then agents having $x \in [0.4 - \theta, 0.5 - \theta)$ will best respond. Note that $P_X$ is decreasing, so the improvement of agents at $t = 2$ will be larger than the improvement of agents at $t = 1$, and $q_t$ is not decreasing.

However, if the conditions in Prop. 3.4 of the main paper is satisfied, $a_t$ will approach 1, this means $q_t$ will still eventually approach to $q_0$, which is the qualification rate of the prior-best-response distribution.

Thanks for the comments. Please see our responses point by point as follows.

Weakness 1: Focus on marginal quantities rather than conditional quantities makes results derivation unclear. It might improve clarity for the authors to consider one of these at a time and describe what we expect to happen.

Answer: Firstly, our model mainly consists of 3 parts as shown in Eqn.(3) of the main paper: human-annotated samples, model-annotated samples, and agents' strategic best responses. We study them together since each of them is likely to play a role under the strategic retraining setting. Moreover, our model admits different proportions of human-annotated samples and model-annotated samples, which means the results are suitable for a process updated with only model-annotated samples or human-annotated samples. In App. G.3 (Lemma G.2), the proof details reveal that both $\mathbb{E}[f_t(X) | X = x]$ and $\overline{q}_t$ are increasing. However, note that the increasing $a_t$ is not due to the increasing $q_t$. As Prop. 3.4 shows, $q_t$ can decrease over time. It is just because the model "misannotate" some agents to be qualified.

Weakness 2: Assumption of covariate modification translating to genuine label changes is strong. How about gaming?

Answer: Firstly, our work takes interest in how the qualification rate changes as the retraining goes on, so we assume the covariate shift because feature shifts will not produce a qualification rate change. Although the earliest works on strategic classification consider only the feature shifts, it is quite common for recent works to consider covariate shifts ([Rosenfeld, et al., 2020; Raab & Liu, 2021; Guldogan et al., 2022]. Moreover, the works in causal strategic learning ([Miller et al., 2020]) suggest that the decision-maker can only consider causal features to ensure improvement.

Weakness 3: The covariate distributions for model-annotated and human-annotated samples are different without clear motivation.

Answer: We assume the human-annotated samples are drawn from prior-best-response distribution $P_{XY}$ because the human annotation process is independent of the decision-making process. At each round $t$, each human agent is classified by the model $f_t$ and best responds to $f_{t-1}$, the decision-maker never confuses the agents by simultaneously using human experts to label agents. Instead, human experts never participate in the interaction and human annotation is another process for the decision-maker to obtain information about the whole population by acquiring data from public datasets or some third parties, and then label them to recover the population distribution. This distribution is free of best response because the samples are not subject to any decision. To better illustrate this idea, consider an example as follows. In a loan application scenario, the bank obtains personal finance data from credit agency and other demographic data from the government to produce human-annotated data, while they keep using an automated decision model to reject or approve loan application and update this model with the aid of human-annotated samples. We also agree with the interesting nature of considering the human-annotation for post-best-response distribution and our model can incorporate this setting. Details are discussed in the general response.

Question 1: Clarity on how we should view human-annotated samples in terms of reliability and strategic supply.

Answer: The model can either let human-annotated samples be reliable or not, but it is likely that they are not accurate and have a systematic bias due to several reasons (App. B gives examples). However, all results in Sec. 3.1 - 3.2 hold no matter whether systematic bias exists. This is why we do not state it until Sec. 3.3. The systematic bias exists not because of strategic supply. It exists mainly because of the complexity of labeling a sample under human-related decision-making scenarios. Sec. 4 and App. B illustrates this in detail.

Question 2: the setting where agents are strategic with only model-annotated samples.

Answer: Our model can demonstrate this setting because the ratio between model-annotated and human-annotated samples is not constant. Specifically, Thm. 3.3, Prop. 3.4 will always hold when there are only model-annotated samples. Other results also need no modification.

Question 3: It would be helpful to see a discussion of the different retraining strategies

Answer: thanks for this helpful suggestion. Just as the answer of the previous question, the ratio between model-annotated and human-annotated samples should not be a problem. For the prior-best-response and post-best-response strategies, we discuss it in the general response.