Model Explanation Disparities as a Fairness Diagnostic

$**Weakness: Fairness Discussion**$

This is a great point and one that we looked to provide a more robust discussion of by the addition of two experiments in our new Section 4.3 connecting our work to existing fairness work. First, we used the open source GerryFair code by Kearns et al. 2018 to find rich subgroups that maximally violate false positive rate. We find that these rich subgroups also have high AVG-SEPFID, which in some cases are on the same features found using our methodology. These AVG-SEPFID values are lower than ones from our approach but the connection between the two highlights the relevance of our work towards fairness. We also evaluated traditional fairness metrics (independence, separation, and sufficiency as defined by Barocas et al. 2023) on our subgroups versus the whole dataset and found disparities along these metrics. These disparities were not always very large, but they did consistently appear. Given these two results, our work remains consistent with established literature on fairness.

We note that regardless of the fairness metric used, establishing our results without optimizing for a specific fairness metric allows our method to be generalizable to all definitions of fairness. As discussed in Limitations: "It is known that even the most popular and natural fairness metrics are impossible to satisfy simultaneously, and so (choosing a fairness definition) would run up against the problem of determining what it means for a model to be fair." The results of our methodology may then be observed under any users' chosen perspective of fairness and still be accurate.

$**Q: Local Separability Assumption**$

We do not believe that a local separability assumption is too strong. The most popular model explanation methods such as the ones we use in our experiments (LIME, SHAP, and Vanilla Gradient) satisfy local separability. With rising interest in understanding individual model outputs, this will possibly continue to be the case. Furthermore, as seen in the experiments section, we also ran a successful experiment without the local separability condition, using linear regression coefficients as the importance notion. This shows that even if local separability is not satisfied, we can still identify subgroups with high AVG-SEPFID.

$**Q: Section 3/Algorithm 1 Clarity**$

Thank you for pointing this out. While we discuss this in full detail in Appendix B, we have added additional clarifying text to Section 3 to address this and other concerns about the presentation in this section. $L$ is the Lagrangian function that is formed in order to solve the constrained optimization problem in Equation 1. Because we are optimizing over randomized classifiers in $\Delta(\mathcal{G})$, we can apply strong duality to solve the dual problem:

$(p_g^*, \lambda^*) = argmin_{p_g \in \Delta(\mathcal{G})} argmax_{\lambda \in \Lambda} \mathbb{E}_{g \sim p_g}[L(g, \lambda)]$

$= argmin_{p_g \in \Delta(\mathcal{G})} argmax_{\lambda \in \Lambda}L(p_g, \lambda)$

with $L(g, \lambda) = \sum_{x \in X} g(x) F(f_j,x,h) + \lambda_L \Phi_L + \lambda_U \Phi_U, \quad L(p_g, \lambda) = \mathbb{E}_{g \sim p_g}[L(g, \lambda)],$

where $p^*_g$ is the optimal rich subgroup distribution, and $\lambda^*$ is the optimal value of the dual variable. $L(g,\lambda)$ as represented above is the sum of the total importance of feature $f_j$ on subgroup $g$ and the size constraint penalty functions $\Phi$ which are weighted by the dual $\lambda$ variables. This is the objective function that both the min and max player are trying to optimize when applying the result of Freund and Schapire 1996, each optimizing over different arguments.

$**Q: Practical applications**$

This is a good point and we have added additional discussion in the introduction along with the two substantial experiments mentioned above to address this. At a high level, our method is best viewed as an exploratory data analysis technique that can result in hypothesis generation about potential sources of bias. For example, if a given feature is found to have a significant effect in a given subgroup, this could result in the hypothesis that there is some measurement error in the way it is collected, or alternatively that the underlying dynamics are actually different for that subgroup, in which case a reasonable solution may be to fit a separate model for that subgroup.

Our additional experiments highlight two other scenarios where our methodology may be useful. First, the FID metric may be used to audit a subgroup which we already know violates some fairness metric to aid a practitioner in quickly identifying features which may be the source of bias or error. Second, Algorithm 1 can be used to find rich subgroups that might face some kind of bias, as we showed that high FID subgroups tend to have other fairness metric disparities.

$**c: Definition 3 typo**$

Thank you for pointing this out. We have corrected this typo.

$**q: What does the bound B in Algorithm 1 represent intuitively?**$

The bound constant $B$ is a hyperparameter that represents the magnitude of the cost of violating the size constraints. Intuitively, it can be thought of like a coefficient for a gradient function where using too small of a value means the algorithm will not converge to a subgroup of appropriate size while using too large of a value means the proposed subgroup will jump around too much between iterations. As discussed in Appendix G, we found an appropriate value to be $B = 10^4 \cdot \mu(f_j)$ where $\mu(f_j)$ is the average feature importance value.

$**q: Subgroup functions g are linear functions. Is this a restriction imposed by the proposed algorithm?**$

We set the hypothesis class of $g$ to be linear but that is not a restriction of our proposed algorithm. Following Kearns et al 2018 we picked a linear threshold function because it is (i) interpretable, (ii) generalizes out of sample, and (iii) is still significantly more powerful than looking at marginal attributes alone. Hebert-Johnson et al. use rich subgroups defined by conjunctions of attributes which would also be a reasonable choice for our setting. The framework is flexible enough to accommodate any hypothesis class assuming we have access to an approximate optimization oracle for that class.

$**q: FID usefulness**$

You are correct that FID itself is not representative of the true importance a separable explanation model assigns because it depends on the size of the subgroup. This is why we use AVG-SEPFID as the metric to optimize for and which we report all of our results for. The presentation of Definition $1$ builds the foundation for optimizing AVG-SEPFID and is also useful for non-separable notions of importance which do not run into the issue you pointed out.

On a broader scope, we do not view FID or AVG-SEPFID as fairness notions in and of themselves. As discussed in the primary response, the overarching use of this work is to audit models for rich subgroups with potential fairness concerns, which we expanded upon with additional discussion in the introduction and the addition of the experiments in Section 4.3. We did not pick a traditional fairness metric to optimize for in order to remain agnostic about the "correct" fairness metric to use. As discussed in the Limitations section: "Importantly, we eschew any broader claims that large FID values necessarily imply a mathematical conclusion about the fairness of the underlying classification model in all cases. It is known that even the most popular and natural fairness metrics are impossible to satisfy simultaneously, and so we would run up against the problem of determining what it means for a model to be fair."

$**Q1: Comparison to existing rich subgroup work**$

While there is existing literature exploring rich subgroups, we are the first to do work in the context of feature importance disparities. Prior rich subgroup discovery methods would not be applicable for optimizing for FID. We did, however, run an additional experiment in Section 4.3 using the GerryFair code of Kearns et al. 2018 to find rich subgroups that maximize false positive rates and audit them with respect to AVG-SEPFID. We find that these subgroups also have significant AVG-SEPFID on some features, but not as large as those found by our Algorithm 1 that optimizes for AVG-SEPFID. While this experiment was primarily run to address other reviewers' concerns about connections to fairness, it also highlights the connection between our work and rich subgroup discovery algorithms designed for fairness metrics like equalized odds.

$**Q2: Runtime concerns**$

This is a good question and a reasonable concern that we found in practice not to be an issue. As discussed in Appendix M: "While Theorem 1 states that convergence time may grow quadratically, in practice we found that computation time was not a significant concern -- empirically our method converged must faster than the theory alone would suggest. The time for convergence varied slightly based on dataset but for the most part, convergence for a given feature was achieved in a handful of iterations that took a few seconds to compute. Features which took several thousand iterations could take around 30 minutes to compute on larger datasets."

$**Q3: Steps forward**$

These are great questions and we have made additions to the paper's introduction that better address the usefulness of our approach. We've added the following text to the introduction:

"As we later discuss in Section 5, disparities in feature importance do not necessarily imply that a subgroup has fairness disparity as measured by conventional metrics like equalized odds or calibration, although we show in Subsection 4.3 that this is empirically often the case. Nor does finding a subgroup with high feature disparity come with a pre-defined “fix”– the disparity could be caused by many factors including (i) true underlying differences between subgroups, or (ii) differences in measurement of the features or outcome variables across subgroups. Since the specific “fixes” are highly context dependent, our method should be viewed as a tool for generating hypotheses about potential sources of bias, which can then be addressed by other means. For example, in Figure 4 we find that the feature arrested-but-with-no-charges is highly important when predicting two-year-recidivism on the population as a whole, but carries almost no importance on a subgroup which is largely defined by Native-American males. This could motivate further research into if this subgroup is policed in a different way than the population as a whole. Alternatively, a technical solution may be to train a separate model for this subgroup."

With regards to the second question, the exact values of AVG-SEPFID are dependent on the feature importance notion and the dataset. Thus a threshold value would not be appropriate for signaling concern. The better signal for concern would be the AVG-SEPFID value relative to the average feature importance value. We represent it like this in Figure 2, where this ratio value ranges from a small multiplicative factor to over $225$ times the average value. We also note that AVG-SEPFID is not itself a bias metric, rather it may help identify sources of bias as discussed earlier.

$**Q4: Approximation bound in Theorem 1**$

I think there is a slight misunderstanding here -- this is a stronger result than a high probability guarantee or an expected error guarantee; the theorem is that (assuming access to CSC oracles) we return an approximately optimal rich subgroup distribution in $O(4n^2B^2/\nu^2)$ iterations, with probability $1$. This follows from the fact that the regret bound for the exponentiated gradient algorithm is with probability $1$.

$**Weakness: Interpretability of subgroup**$

In our methodology we select the best subgroup out of the distribution generated. This subgroup is a threshold function so the outcome subgroup is still binary. From Section 3: "However, in practice we simply take the groups $g_t$ found at each round and output the ones that are in the appropriate size range, and have largest FID values. The results in Section 4 validate that this heuristic choice is able to find groups that are both feasible and have large FID values. This method also generalizes out of sample showing that the FID is not artificially inflated by multiple testing (Appendix J). Moreover, our method provides a menu of potential groups $(g_t)^T_{t=1}$ that can be quickly evaluated for large FID, which can be a useful feature to find interesting biases not present in the maximal subgroup."

$**Q(a): Clarity of Algorithm 1 and additional questions on rich subgroup class and size violations**$

Thank you for highlighting concerns about the presentation. We have added additional context to the presentation of the algorithm including clarification of the approach after Equation 2 and more description of algorithm parameters.

We provide algorithmic details of the rich subgroup class in Appendix G. The rich subgroup hypothesis class is a linear threshold function, which we will state more clearly in the draft. This matches prior work on rich subgroup fairness and multicalibration, which use the same rich subgroup function. Note that our rich subgroup class contains the class of conjunctions as a subclass.

The size violations correspond to the upper and lower bounds on the rich subgroup size. As discussed in Section $3$, we need to constrain the rich subgroup size to $[\alpha_L, \alpha_U]$ since we are solving the constrained FID optimization problem over a discretization of $[0,1]$ and then taking the subgroup with maximal AVG-SEPFID. The proof and details for this approach are discussed in Appendix C.

$**Q(b): Connection to fairness metrics**$

This is a great point and one that we have addressed by making significant additions to the paper via the additional experiments in the new Section 4.3. We ran two additional experiments to tie in our notion of AVG-SEPFID with established fairness work. First, we used the open source GerrFair code by Kearns et al. 2018 to find rich subgroups that maximally violate false positive rates. We find that these rich subgroups also have high AVG-SEPFID, which in many cases load on the same feature found using our methodology. These AVG-SEPFID values are lower than ones from our approach, but the connection between the two highlights the relevance of our work towards fairness. We also evaluated traditional fairness metrics (equalized odds, calibration) on our subgroups g versus the whole dataset and found disparities along these metrics. These disparities were not always very large, but they did consistently appear.

We also added some discussion about the role of our method to the introduction:

"As we later discuss in Section 5, disparities in feature importance do not necessarily imply that a subgroup has fairness disparity as measured by conventional metrics like equalized odds or calibration, although we show in Subsection 4.3 that this is empirically often the case. Nor does finding a subgroup with high feature disparity come with a pre-defined “fix”– the disparity could be caused by many factors including (i) true underlying differences between subgroups, or (ii) differences in measurement of the features or outcome variables across subgroups. Since the specific “fixes” are highly context dependent, our method should be viewed as a tool for generating hypotheses about potential sources of bias, which can then be addressed by other means. For example, in Figure 4 we find that the feature arrested-but-with-no-charges is highly important when predicting two-year-recidivism on the population as a whole, but carries almost no importance on a subgroup which is largely defined by Native-American males. This could motivate further research into if this subgroup is policed in a different way than the population as a whole. Alternatively, a technical solution may be to train a separate model for this subgroup."

Finally we note in the Limitations section: "Importantly, we eschew any broader claims that large FID values necessarily imply a mathematical conclusion about the fairness of the underlying classification model in all cases. It is known that even the most popular and natural fairness metrics are impossible to satisfy simultaneously, and so we would run up against the problem of determining what it means for a model to be fair." We still stand behind not using a fairness metric as our primary result measure, but we believe these two additional experiments added in the new Section 4.3 highlight the connection of our work to the broader fairness literature.

$**Q: Algorithm 1 Clarity**$

Thank you for pointing this out, we have added some clarifying text to the description under Equation $2$ that improves the interpretability of algorithm 1. Regarding the relation to Kivinen and Warmuth, in short, to solve the constrained optimization problem in (1), we first form the Lagrangian and apply strong duality (which we can do because we are solving an LP after relaxing to optimizing over randomized classifiers in $\Delta(\mathcal{G})$), which corresponds to solving the dual problem:

$(p_g^*, \lambda^*) = argmin_{p_g \in \Delta(\mathcal{G})} argmax_{\lambda \in \Lambda} \mathbb{E}_{g \sim p_g}[L(g, \lambda)]$

$= argmin_{p_g \in \Delta(\mathcal{G})} argmax_{\lambda \in \Lambda}L(p_g, \lambda)$

with $L(g, \lambda) = \sum_{x \in X} g(x) F(f_j,x,h) + \lambda_L \Phi_L + \lambda_U \Phi_U, \quad L(p_g, \lambda) = \mathbb{E}_{g \sim p_g}[L(g, \lambda)],$

where $p^*_g$ is the optimal rich subgroup distribution, and $\lambda^*$ is the optimal value of the dual variable. The $*min*$-player corresponds to optimizing over $p_g$ which aims to maximize the subgroup disparity (minimize the negative disparity), and the $*max*$-player corresponds to optimizing $\lambda$, which is the vector of dual variables, one for each constraint. Both players have the same objective function $L(g, \lambda)$ since it is a zero sum game, but they are optimizing different arguments. We discuss this in full detail in Appendix B.

$**Q: What is expectation over in Definition 1?**$

The expectation is over $X$ which is our dataset of points drawn i.i.d. from some distribution $\mathcal{R}$. $X$ and $\mathcal{R}$ are both defined earlier in the preliminaries.

$**Q: Locally Separable intuition**$

Intuitively, if the underlying mechanism to compute an explanation value can be used on an individual data point, then the feature importance notion is locally separable. For example, LIME takes a single data point, computes a local model around it, and creates explanation values from that. On the other hand, coefficients from linear regression are often used by practitioners to explain feature importance but a linear regression may only be computed over a group of data points. In general, most modern popular model explanation methods are locally separable since current explainability work that can be applied to more complex models focuses on understanding individual predictions.

$**Q1: Discussion of Balagopalan**$

Thank you for highlighting this paper, we have added discussion of it to the related works section. Balagopalan et al. 2022 is similar to Dai et al. 2022, which we already discuss. Those two papers investigate sensitive subgroups and their relationship with model explanation quality, as measured by fidelity, stability, and other metrics they define, whereas we look at the difference in the magnitude of the explanations. Furthermore, we generalize the problem significantly by considering rich subgroups and our algorithm for searching an exponentially large subgroup space is novel and is essential for any fairness work regarding rich subgroups. We have highlighted these differences in the related works discussion but these papers are certainly relevant and could be explored as a future work in conjunction with ours.

$**Q2/Weakness 1: Model utility and next steps for user**$

This is a great point -- we have both added significant discussion of this point in the introduction, and substantial experiments to make this more clear. At a high level, our method is best viewed as an exploratory data analysis technique that can result in hypothesis generation about potential sources of bias. For example, if a given feature is found to have a significant effect in a given subgroup, this could result in the hypothesis that there is some measurement error in the way it is collected, or alternatively that the underlying dynamics are actually different for that subgroup, in which case a reasonable solution may be to fit a separate model for that subgroup. While our method does not prescribe a given solution since it is very context dependent, it is a method to uncover these possible disparities in the first place. We've added the follow text to the introduction:

"As we later discuss in Section 5, disparities in feature importance do not necessarily imply that a subgroup has fairness disparity as measured by conventional metrics like equalized odds or calibration, although we show in Subsection 4.3 that this is empirically often the case. Nor does finding a subgroup with high feature disparity come with a pre-defined “fix”– the disparity could be caused by many factors including (i) true underlying differences between subgroups, or (ii) differences in measurement of the features or outcome variables across subgroups. Since the specific “fixes” are highly context dependent, our method should be viewed as a tool for generating hypotheses about potential sources of bias, which can then be addressed by other means. For example, in Figure 4 we find that the feature arrested-but-with-no-charges is highly important when predicting two-year-recidivism on the population as a whole, but carries almost no importance on a subgroup which is largely defined by Native-American males. This could motivate further research into if this subgroup is policed in a different way than the population as a whole. Alternatively, a technical solution may be to train a separate model for this subgroup."

$**Additional Experiments**.$ We also run new experiments investigating the relationship between our notion of subgroup disparity and fairness metrics like subgroup calibration and equalized odds, which summarize in a new Section 4.3 in the updated draft. We find that across all data sets (i) our found subgroups often also correspond to subgroups with significant disparities in fairness metrics but (ii) consistent with the theory in fairness, which says that even common fairness metrics can not be simultaneously satisfied, we also find some subgroups that don't have a large fairness disparity. We also run the reverse experiment, taking rich subgroups that maximize the false positive rate disparity using the GerryFair code from Kearns et al. 2018, and auditing these rich subgroups with respect to AVG-SEPFID. We find that the AVG-SEPFID of these subgroups is significant, but not as large as those found by our Algorithm $1$ that explicitly optimizes the AVG-SEPFID.

$**Additional Comments:**$

We have added text to Section 3 of the paper that adds clarity to the presentation of Algorithm 1. We have made additional minor changes to the submission to address concerns about readability.