From Individual Experience to Collective Evidence: A Reporting-Based Framework for Identifying Systemic Harms

Thanks for your time writing the review! We have grouped responses to your comments below. If there are any further weaknesses in the work that are concerning for you, please don’t hesitate to let us know.

Experiments

“For the simulated experiment, it would be better to show the result from synthetic experiment with different parameters for analysis.”

To clarify, the mortgage experiments in 5.2 are only partially simulated: they draw from real-world data, and the component that we simulate is reporting behavior, which is exactly the main parameter of interest for our problem setting.

To this end, we simulate three different patterns of reporting, as discussed on L396. In Appendix D.2, we give more details on these simulated reports, as well as how they relate to the parameters that affect modeling (i.e. $\rho_G/\rho$) that were discussed in Section 3.1. As our computations (in D.2) show, the three ways that we simulate reports do correspond to meaningfully different values of the parameter $\rho_G/\rho$.

If you meant something else by “parameter,” please let us know!

“a explicit false positive/negative rate analysis could help reader understand better.”

In our problem setting, we felt that false positives/negatives were not quite the right abstraction for understanding the performance of algorithms. Thus, while we covered measures that are conceptually similar, we did not use the language of false positives/negatives explicitly. We will outline how these ideas relate below.

For false positives: The notion of “false positive” in our setting is subtle. For the pure hypothesis testing setting, a false positive would be a group for which $\mu_G \leq \beta\mu_G^0$ but is returned by the algorithm; the likelihood of this error is what is provably controlled at level $\alpha$. For both the vaccine and mortgage experiments, all tests identified groups where $\mu_G \geq \mu_G^0$, i.e. “FPR” of 0.

For our application, we are also interested in a notion of “true” harm — i.e., we hope that the groups identified by the algorithms actually reflect groups that (post-hoc) we know to have been harmed. For the vaccine experiment, this was broadly the “young men” category; all our algorithms only identified the groups (M, 12-17) and (M, 18-24), also suggesting a “FPR” of 0. For the mortgage experiment, we wanted to identify groups with a high relative risk of denial in general, but there are not necessarily hard cutoffs for what would have counted as a “true/false positive.” Tables 2-3 show that our algorithms generally found groups with high true relative risk, suggesting a low “FPR” overall.

For false negatives: Because our tests are sequential and could run for arbitrarily many timesteps, it is impossible to ever fully conclude that a non-null hypothesis has not been rejected. (In fact, our power results indicate that, in the limit of $t \to \infty$, both our proposed tests will, with probability 1, identify any group with $\mu_G > \beta\mu_G^0$.) Furthermore, all of our algorithms stop fairly quickly (see Tables 1, 2, 3) in almost all the settings we test. As we discuss in Table 3, there are a handful of settings where a group has not been identified within 40k steps; heuristically, these could be considered “false negatives” and we report those rates in Table 3.

Writing (clarity & structure; problem definition)

While our work does involve many moving parts, we have done our best to keep the presentation modular. Section 2 is focused fully on notation and model; the beginning of Section 3 gives the problem statement explicitly; and the beginning of Section 4 outlines our general solution concept.

We would love to hear if there are specific aspects of the presentation that were unnecessarily confusing, or any concrete suggestions for revision that would improve clarity.

Knowledge of base rates

“Assumption that baseline rates (μ0G) are known, which is unclear in practice.”

We believe that there is a strong case to be made that it is reasonable to expect $\mu_G^0$ to be tracked by system owners. For example, it is already mandatory, by the Home Mortgage Disclosure Act, for banks to record and publicize the demographic details of all home loan applicants, and the CDC tracks vaccine uptake rates). Generally, we expect that most organizations track some internal metrics for system usage even if this information is not released publicly. On the other hand, we hope that for future systems, our work is one motivation to actively to ask or mandate organizations to collect/share this data.

Other methods

We considered the question of baselines for this problem carefully, but for our problem setting, the two algorithms proposed in section 4 are adaptations of the main approaches to sequential testing given in the literature. One notable alternative that we excluded was Wald’s SPRT, which requires making more parametric decisions and is thus not directly comparable.

Thanks for your time writing the review and reading our paper! It is a good catch on the 4.1 proof, and we agree it would be clearer to break it up as you suggested — we’ll do so in the revision!

To answer your question about handling variations in $\mu_G^0$, [1] show in Section 3 how to extend their standard algorithm to handle a variety of settings where the problem varies over time. These properties come almost “for free” from the testing by betting setup, and doing something analogous for our betting-style algorithm is straightforward despite our tests themselves being different. We glossed over this point a bit in the version of the draft we submitted but will be more explicit in our revision.

[1] Chugg et.al. Auditing Fairness by Betting

Thanks for your time writing the review and the thoughtful questions!

(Q1) We are overall optimistic about how our methods might work for a real-world system, and in future work we hope to develop and/or highlight collaborations with practitioners with real incident reporting databases. The linked AI incident database is not directly compatible with our framework (see also our discussion in Appendix A). While incidentdatabase.ai collects one-off stories of problematic incidents, they are not necessarily linked to specific systems — any incident with any AI system is eligible for report — which makes it hard to make claims about patterns of problems with specific systems. While perhaps those incident reports could be separated by an associated AI system, an additional challenge is to formalize and identify appropriate “base rates” or “subgroups” for this setting.

(Q2) This is definitely a question on our minds for future work. One related work we mention [1] identifies subgroups by clustering online, but the task of running a hypothesis test is much more stringent than making predictions. Dealing with the intricacies of a sequential hypothesis test is the main technical challenge in this setting — such an approach must address data reuse for both identifying subgroups and running a valid test (i.e., avoiding the sequential analogue of p-hacking).

[1] Dai et.al. Learning With Multi-Group Guarantees For Clusterable Subpopulations

Thanks for your time writing this review and the thoughtful questions!

(Q1) This is a good question --- differential rates of (access to) reporting is something that we’ve thought about a lot. In the current version of this work, this can be modeled with the group-specific reporting parameters discussed on section 3 — and, though we don’t focus on it in the main exposition, in principle a different $\beta$ could be set for each group. That said, while it is natural to model known underreporting (e.g. that arises due to access reasons), the current version of our framework doesn’t help with identifying or estimating the degree of underreporting (which, e.g., some of the related work on L80-86 addresses). We think better understanding this question is an important direction of future work.

(Q2) Our theory shows that the stopping time increases by only a logarithmic factor in the number of groups, and only additively (rather than multiplicatively). Thus, for settings where the number of groups is combinatorially large (e.g. we have $2^d$ groups in the case of $d$ binary features), we would expect an additive impact on the stopping time of approximately $O(d)$. Our experiments with 29 and 115 groups suggest that the impact of Bonferroni in practice is even less pronounced than the $\log(|\mathcal{G}|)$ suggested by theory, and we suspect this to be true in general.

As for algorithmic improvements, it is not obvious that current mathematical tools allow for any improvement over the Bonferroni correction. Some recent developments in sequential testing with e-values can handle composite null testing (e.g. [1]) — however, their guarantee is subtly different, in that they are only able to confirm that a harm has occurred to one of the groups, but not identify which one it was. This is an area of future work we are definitely interested in, though it seems that it will require developing more sophisticated theory.

[1] Cho et.al. Peeking with PEAK: Sequential, Nonparametric Composite Hypothesis Tests for Means of Multiple Data Streams