SureMap: Simultaneous mean estimation for single-task and multi-task disaggregated evaluation

Thank you for your detailed review! We address your concerns and questions below:

1. [Can the authors provide some analysis or insights when the data contradicts with Gaussian assumption?]
   a. Please see our discussion of this issue in the general response (Issue 1: Assumption).
2. [Can the authors visualize/show the cost when incorporating data from clients?]
   a. Please see our discussion of efficiency in the general response (Issue 2: Computation) and in particular Rebuttal Figure 2. We will include this analysis and visualization in the revision.

Thank you for your detailed review! We address your concerns and questions below:

1. [The efficiency discussion of the method needs to be expanded. It would be better to list the complexity of the proposed method with existing methods and report the actual runtime.]
   a. Please see our discussion of efficiency in the general response (Issue 2: Computation). We will include this analysis in the revision.
2. [There are certain cases that the proposed method performs worse than the pooled estimation, such as in Figure 2. It would be better to analyze the reasons behind such results.]
   a. In Figure 2, pooling outperforms our approach at extremely low data regimes where the typical number of samples per group is 1. In this extreme regime, there may not be enough information to estimate group-level performance well and so the best thing to report is the overall mean (which is what the pooled estimator does).
3. [For subpopulations with few samples, how does this paper compute their ground truth and evaluate the estimation?]
   a. As noted in the first paragraph of Section 5, we exclude subpopulations with fewer than twenty samples, as we cannot obtain a reasonable ground truth for them.
4. [It would be better to clarify the pooled estimator. For example, what does the notation $h$ in Equation 3?]
   a. The pooled estimator just takes the average across all data samples, or equivalently a weighted average across the average performance on each subpopulation, with weights corresponding to the number of samples on each subpopulation. Equation 3 makes the latter explicit, and uses $h$ to index each subpopulation.
5. [It would be better to explain the working of the baselines, such as the Bock method.]
   a. The Bock estimator is described in Section 3.1, and specified explicitly in Equation 6. All other baselines are also specified in Equations 2, 3, and 5. We will make this clearer in the paper.

Thank you for your detailed review! We address your concerns and questions below:

1. [I would like to know if there are any rules that need to be followed in the selection of disaggregated evaluation datasets to ensure fairness, as well as the reason why the disaggregated evaluation goal is set as the mean 0-1 error.]
   a. The goal of disaggregated evaluation is to evaluate fairness and thus help surface any fairness issues that need to be addressed. The question of selecting the right dataset and metric for disaggregated evaluation is an active area of research but beyond the scope of this paper. Some considerations include how well the data represents the intended uses of the AI system and how well the metric captures potential harms / benefits; see Barocas et al. (2021) for an in-depth discussion. In our case, for dataset selection we simply use all the available data, while for the performance metric note that 0-1 error is just one possible measure among many that our method can be applied to, including MAE, MSE, WER, AUC, and so on. In addition to the 0-1 error our experiments include results for MAE and WER.
2. [The fairness assessment is an application biased towards real-world scenarios. Does using Gaussian distribution as a prior align with real-world evaluations?]
   a. Please see our discussion of these points in the general response (Issue 1: Assumption). In particular, please note that we do not assume that the prior and individual-level distributions are Gaussian, only that the summary statistics are. As discussed in the response, this approximation is quite reasonable for numerous performance metrics of practical interest, including the 0-1 error, MSE, WER, AUC, and so on.

References

Barocas, Guo, Kamar, Krones, Morris, Wortman Vaughan, Wadsworth, Wallach. Designing disaggregated evaluations of AI systems: Choices, considerations, and tradeoffs. AIES 2021.

Thank you for your detailed review! We address your concerns and questions below:

1. [The Gaussian assumption may be too strong for some real-world settings [...] Have you considered non-Gaussian models? For example, a t-distribution might better handle heavy-tailed performance data that can occur in practice.]
   a. Please see our discussion of these points in the general response (Issue 1: Assumption). In particular, as noted in the paper we view a derivation of a robust version of the SureMap objective using Student’s t-distribution as a good direction for future work.
2. [The multi-task setting assumes clients are willing to share summary statistics. How realistic is this assumption? Could you explore privacy-preserving ways to share this information?]
   a. There are many settings when this is not a concern, e.g. in situations when the companies are already sharing the performance statistics publicly as a form of disclosure to their customers or government regulators. In settings where this is a concern, it should be possible to apply techniques from the differential privacy (DP) literature to overcome these limitations (many DP techniques exist for releasing summary statistics, e.g. Biswas et al. (2020)). We leave a full investigation to future work.
3. [How does SureMap compare to GP-based approaches like in [(Piratla et al., 2021)]? The method aims to handle low-data regimes well, which seems relevant here.]
   a. Piratla et al. (2021) study a different low-data setting where they allow the user to actively sample points on which to evaluate performance. This difference makes it somewhat difficult to compare our methods.

References

Biswas, Dong, Kamath, Ullman. CoinPress: Practical private mean and covariance estimation. NeurIPS 2020.
Piratla, Chakrabarti, Sarawagi. Active assessment of prediction services as accuracy surface over attribute combinations. NeurIPS 2021.

Thank you for your detailed review! We address your concerns and questions below:

1. [While the method is theoretically sound, the practical implementation of SureMap may be complex, particularly the coordinate descent algorithm used for tuning parameters. This could pose challenges for practitioners. [...] Can the authors provide more detailed pseudocode or a step-by-step guide for implementing the coordinate descent algorithm used in SureMap?]
   a. As discussed in the general response (Issue 2: Computation), we have updated the parameter optimization scheme post-submission to be L-BFGS. While SureMap is more involved than baselines such as using the naive estimate or the Bock estimator, it is as simple as or simpler than other recent approaches such as Structured Regression (Herlihy et al., 2024) and AAA (Piratla et al., 2021). The pseudo-code for our current method—which we will add in revision—can be summarized in two steps: (1) apply L-BFGS with SciPy’s default settings to the objective in Equation 10 (or Equation 11 for the multi-task case) and (2) run MAP estimation with the resulting mean and covariance.
2. [The method relies on Gaussian assumptions for the prior and noise distributions. In real-world scenarios, data distributions may deviate significantly from Gaussian, potentially affecting the performance of SureMap. [...] How robust is SureMap to deviations from the Gaussian assumptions? Have the authors considered alternative distributions, and how would the method need to be adapted for such cases?]
   a. Please see our discussion of these points in the general response (Issue 1: Assumption). In particular, please note that we do not assume that the prior and the noise are Gaussian; we only assume that the summary statistics are.
3. [The scalability of the method in very large-scale settings is not fully explored. The computational cost associated with MAP estimation and SURE tuning might be prohibitive for very large datasets or a very high number of tasks. [...] What are the computational requirements of SureMap for very large datasets or a high number of tasks? Can the authors provide any benchmarks or comparisons in terms of runtime and memory usage?]
   a. Please see our discussion of these points in the general response (Issue 2: Computation). In particular, SureMap is not computationally expensive for large datasets (it relies on cheap-to-compute summary statistics and so scales weakly with dataset size) nor for many tasks (as shown in Rebuttal Figure 2); in practice, we expect the cost of SureMap (less than a few seconds) is dominated by model inference costs. We will include this cost analysis in the revision.
4. [While the empirical results are promising, they are primarily based on synthetic and semi-synthetic datasets. More extensive evaluation on real-world datasets across diverse domains would strengthen the findings. [...] Can the authors provide examples of real-world applications where SureMap has been or could be successfully applied? This would help in contextualizing the method’s practical utility.]
   a. Please note that none of our benchmarks are fully synthetic, in the sense of being entirely generated by artificial distributions, and many do not involve any synthetic aspects at all. For example, our single-task ASR setting involves a widely used model (Whisper) evaluated on a real-world dataset (Common Voice). In terms of specific applications, the Diabetes task is motivated by a real-world use-case (Obermeyer et al., 2019) and has been used in previous disaggregated evaluation research (Miller et al., 2021; Herlihy et al., 2024). At a higher level, disaggregated evaluation is the central step in any fairness assessment (Barocas et al., 2021; Herlihy et al., 2024), so we expect SureMap to be of use in a broad range of real-world settings by AI companies, regulators, journalists, and researchers.

References

Barocas, Guo, Kamar, Krones, Morris, Wortman Vaughan, Wadsworth, Wallach. Designing disaggregated evaluations of AI systems: Choices, considerations, and tradeoffs. AIES 2021.
Herlihy, Truong, Chouldechova, Dudík. A structured regression approach for evaluating model performance across intersectional subgroups. FAccT 2024.
Miller, Gatys, Futoma, Fox. Model-based metrics: Sample-efficient estimates of predictive model subpopulation performance. ML4H 2021.
Obermeyer, Powers, Vogeli, Mullainathan. Dissecting racial bias in an algorithm used to manage the health of populations. Science, 2019.
Piratla, Chakrabarti, Sarawagi. Active assessment of prediction services as accuracy surface over attribute combinations. NeurIPS 2021.