Fair Anomaly Detection For Imbalanced Groups

Thank you very much for your detailed and constructive comments. We are glad to provide the following responses to your questions.

Weakness 1: Unclear how the proposed approach can minimize the upper bound given in Theorem 4.6.

Answer 1: In Appendix E.4, we present the detailed proof of how our designed $\mathcal{L}_\text{FAC}$ minimizes the upper bound in Theorem 4.6.

Weakness 2 & 3: Please clarify whether there is a theoretical relationship between the risk difference bound and the recall difference metric used in the experiments. It would be better to explicitly discuss how to leverage the theorems in Sec 4 to analyze the fairness issue in anomaly detection tasks, especially the reconstruction-based anomaly detection models.

Answer 2 & 3 : The risk difference bound places a theoretical upper limit on disparities in recall, assuming the loss function explicitly penalizes false negatives. If risk differences are bounded tightly, recall differences are also constrained, since both rely on group-level error dynamics. While recall difference focuses specifically on positive cases, the risk difference bound incorporates all types of errors (false positives, false negatives, etc.). Therefore, a high risk difference often correlates with high recall disparities, but the risk metric provides a broader error perspective. And that is why we provide a risk difference bound for the fairness guarantee of our fair anomaly detection method.

Thank you very much for your detailed and constructive comments. We are glad to provide the following responses to your questions.

Major Weakness 1: Question in the statement 'deep anomaly detection methods demonstrate significantly better performance than shallow anomaly detection methods.'

Answer 1: We agree with you that shallow models might have better performance on some easy datasets, such as tabular data. We mean to say that deep models tend to outperform shallow model in some complex datasets. We have revised our introduction by emphasizing that usually deep anomaly detection methods perform better than the shallow methods on more complicated tasks.

Major Weakness 2 & 3: The use of an autoencoder as an anomaly detector relies heavily on an assumption stated by the authors that " Anomalies tend to exhibit large reconstruction errors because they do not conform to the patterns in the data as coded by the autoencoder." This is an assumption which has been heavily questioned in recent studies. The authors assume in section 3.2 that the model is only capable of fitting 2 out of 4 subgroups. Yet this seems like a strong assumption with little theoretical foundation.

Answer 2&3: There are several reasons for us to choose vanilla autoencoder as the anomaly detector.

1. Vanilla autoencoder is simple yet effective to identify the anomaly from both protected and unprotected group. In the early stage of the algorithm development, we try to incorporate the idea of the MemAE [1] by using it as the backbone for our method. However, the experimental results show that the performance of MemAE tends to have worse performance than a vanilla autoencoder architecture on most datasets.
2. We try to avoid the over-fitting issue by using vanilla autoencoder, specifically for the tabular data with less than 10 features.
3. The theoretical analysis is built upon the assumption that the model is only capable of capturing the normal samples for protected and unprotected group, and thus the anomalies from both groups will have the larger reconstruction error. Otherwise, the model cannot distinguish anomalies from normal samples. However, a complex model could not fulfill such an assumption. Thus, we select vanilla autoencoder as the backbone for our method.
4. We consider the efficiency of the vanilla autoencoder over the complex backbone in other anomaly detection methods.
5. We would like to point out that "Anomalies tend to exhibit large reconstruction errors " is still a widely accepted assumption in the field of anomaly detection, such as [2][3][4].

[1] Dong Gong, Lingqiao Liu, Vuong Le, Budhaditya Saha, Moussa Reda Mansour, Svetha Venkatesh, and Anton van den Hengel. Memorizing normality to detect anomaly: Memory-augmented deep autoencoder for unsupervised anomaly detection. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 1705–1714, 2019.

[2] Guo, Jia, Lize Jia, Weihang Zhang, and Huiqi Li. "Recontrast: Domain-specific anomaly detection via contrastive reconstruction." Advances in Neural Information Processing Systems 36 (2023).

[3] Z. You et al., “A unified model for multi-class anomaly detection,” in Advances in Neural Information Processing Systems (S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, eds.), vol. 35, pp. 4571–4584, Curran Associates, Inc., 2022.

[4] H. Deng and X. Li, “Anomaly Detection via Reverse Distillation from One-Class Embedding,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9737–9746, 2022.

Major Weakness 4: For the CelebA dataset, all detectors perform much worse than should be expected, with almost all detectors being close to random. This indicates that perhaps it is not a well-suited benchmark dataset, or other detectors should be shown which do perform well.

Answer 4: We agree that CelebA dataset is a very challenging dataset. Following [5], we evaluate the performance of all detectors on this dataset. Notice that [5] assumes that the label information is partially available for samples in the training set and [5] achieves 95.1% AUC-ROC on this dataset. While all detectors do not perform well on this dataset in the unsupervised setting, it is still important to evaluate the performance of anomaly detection methods on this benchmark dataset in the field of fair machine learning.

[5] Pang, Guansong, Chunhua Shen, and Anton Van Den Hengel. "Deep anomaly detection with deviation networks." In Proceedings of the 25th ACM SIGKDD international conference on knowledge discovery & data mining, pp. 353-362. 2019.

Thank you for your answers to my comments. I will reply point-by-point again.

1: Tabular data is by no means "easier" than computer vision data. Computer vision data, tabular data, time series data, etc. should in many cases just be considered separately.

2/3: Thank for you for elaboration. I would like to see this elaboration reflected in the final manuscript as well.

• 1. Perhaps a better alternative to memAE would be the Normalized autoencoder (https://arxiv.org/abs/2105.05735). Please also include these experimental results in the paper.
• 2. I do not see how using a vanilla autoencoder leads to less overfitting compared to different architectures.
• 3. I am still unconvinced by the validity of this assumption. It would be better to either not assume this at all, or to substantiate the assumption further.
• 4. I do not understand what the author means by this.
• 5. I think it's extremely problematic that an assumption which is widely questioned is further propagated, even in recent works. Note that none of these works validate the assumption, so they do not add any weight to the assumption itself.

4:

• If other authors in slightly different settings achieve substantially better results, than the manuscript should better reflect the change in settings which account for this difference.

Thank you very much for your detailed and constructive comments. We are glad to provide the following responses to your questions.

Weakness 1&2: The definition of protected groups is rather simplistic, and it might not be easy to identify the underrepresented groups automatically. Why restrict to one single protected group?

Answer 1&2: Our method can be easily extended to the multi-valued multi-group setting and it is not necessary for us to identify one single protected group from different subgroups. To be more specific, In our loss design, $L_\text{REC}$ is an auto-reweighted reconstruction error, and the weight is in proportion with $L_0^{G} - L_G$, where $G$ denotes a certain group. In the multi-valued, multi-group case, we can extend the design of $L_\text{REC}$ by weighting each reconstruction error of group $G$ with $\frac{L_0^{G} - L_G} { \sum_g L_0^{g} - L_g}$. Then, in $L_\text{FAC}$, $L_\text{fair}$ encourages the representation similarity between different groups, and we can extend it to the similarity between different combinations of multi-valued, multiple attributes. $L_\text{unif}$ encourages the uniformity within each group, and we can add the uniformity term for each group in the multi-valued, multi-group case.

Weakness 3: Comparison with separate anomaly detectors for each (protected/unprotected) group?

Answer 3: Thanks for the question. There are several reasons why we do not include it as a baseline in our work.

1. It's inefficient to train two separate anomaly detection detectors.
2. We do not know the ratio of anomalies in protected and unprotected group. If we train two separate anomaly detector model, it's hard to determine the number of anomalies in each group.
3. We also try to train two separate anomaly detectors for two groups. During the test time, we set the k for top-k selection in proportion to the group ratio (i.e., set top-k as k * protected group ratio for protected group model, and set top-k for unprotected group model as k * unprotected group ratio). In the following table we present the results (mean and std) of training separate detectors for different groups on the four datasets. Compared with this baseline, our method performs much better in both task performance and fairness. One possible explanation for the failure of the baseline is that separate models may overfit on certain groups and thus cannot distinguish anomalies from normal samples.

Weakness 4: Confusion of pink and red dots in Figure 2.

Answer 4: Thank you for the suggestion. We have updated Figure 2 in the revision.

Weakness 5: Confusion in Line 194.

Answer 5: Sorry for the confusion. We mean that 'encouraging uniformity with contrastive learning can alleviate this issue by pushing examples to be uniformly distributed in the unit hypersphere, as illustrated in Figure 2b'.

Weakness 6: Confusion in Lines 195-196.

Answer 6: In contrastive learning, for each data point, different "views" of the same instance are created to form positive pairs, while this instance forms negative pairs with other instances. Contrastive loss maximizes the alignment of positive pairs and the separation of negative pairs.

Weakness 7: Confusion in Line 208.

Answer 7: We have revised the statement as 'maximize the cosine similarity between the representations of the protected and unprotected group'. We have also added more detailed explanations in the revision. If there is more than one protected group, the loss function will still encourage the similarity among different groups while encouraging the uniformity within each group. To be more specific, $L_\text{fair}$ encourages the representation similarity between different groups, and we can extend it to the similarity between different combinations of multi-valued, multiple attributes. $L_\text{unif}$ encourages the uniformity within each group, and we can add the uniformity term for each group in the multi-valued, multi-group case.

Thank you very much for your recognition of our work and your constructive comments. We are glad to provide the following responses to your questions.

Weakness 1: How does the method scale with multi-valued, multiple protected attributes setup?

Answer 1: In our loss design, $L_\text{REC}$ is an auto-reweighted reconstruction error, and the weight is in proportion with $L_0^{G} - L_G$, where $G$ denotes a certain group. In the multi-valued, multiple protected attributes case, we can extend the design of $L_\text{REC}$ by weighting each reconstruction error of group $G$ with $\frac{L_0^{G} - L_G} { \sum_g L_0^{g} - L_g}$. Then, in $L_\text{FAC}$, $L_\text{fair}$ encourages the representation similarity between different groups, and we can extend it to the similarity between different combinations of multi-valued, multiple protected attributes. $L_\text{unif}$ encourages the uniformity within each group, and we can add the uniformity term for each group in the multi-valued, multiple group case.

Weakness 2: The choice of $\alpha$ hyperparameter is arbitrary.

Answer 2: In Appendix F.4 we can observe that our method is robust to the choice of $\alpha$, and we show the results of $\alpha=4$ in the experiments. In real-world scenarios, since our method is robust to the choice of $\alpha$, there should be no much concern on how to choose $\alpha$. You can simply set $\alpha=4$ for convenience.

Weakness 3: Lack of the discussion on hyperparameter settings for the competitors.

Answer 3: Thanks for the suggestion. In Appendix F.1, we include the training details. For all the competitors, we use suggested hyperparameter settings in their original papers and the official code implementations.

Question 1: Are there simpler considerations for $\epsilon$ that can be informed from the imbalance factor of groups? How does $\epsilon$ work for multi-valued attributes and multiple protected attributes?

Answer Q1: In Appendix F.3, we compare our method with the variant which sets $\epsilon$ as the imbalance factor of groups. From Table 12 we can observe that our designed learnable $\epsilon$ can effectively better enhance task performance and meanwhile promote fairness. For multi-valued multiple attributes, as discussed in Answer to Weakness 1, we can set $\epsilon=\frac{L_0^{G} - L_G} { \sum_g L_0^{g} - L_g}$ for each subgroup $G$.

Thank you very much for your detailed and constructive comments. We are glad to provide the following responses to your questions.

Weakness 1: The strong coupling between the challenges of data imbalance and representation disparity makes the necessity of the fairness-aware contrastive learning module questionable.

Answer 1: We would like to clarify that the challenge of imbalanced data is two-fold: (1) the imbalance between the normal examples and the anomalies; (2) the imbalance between the groups. The fairness-aware contrastive learning loss consists of two terms, $L_\text{fair}$ and $L_\text{unif}$. $L_\text{fair}$ focuses on the imbalance between the protected and unprotected groups while $L_\text{unif}$ deals with the imbalance between normal examples and the anomalies by encouraging uniformity. In the ablation study in Section 5.4, we compare our method with the one without $L_\text{fair}$ (FADIG-N) and the one without $L_\text{unif}$ (FADIG-D), and replacing our designed fairness-aware contrastive learning loss with traditional contrastive loss (FADIG-C), and we can observe that our proposed method always has the best performance in both task performance and fairness, compared with all the variants. This validates the necessity of the fairness-aware contrastive learning module.

Weakness 2: Question of novelty in addressing accuracy parity by solving group imbalance in the task of anomaly detection.

Answer 2: In the task of anomaly detection, aside from the group imbalance, we are also faced with the imbalance between the anomalies and the normal samples. The mixture of the two types of imbalance makes the task of fair anomaly detection much more challenging. From the empirical comparison in Sections 5.2 and 5.3, we can see that our method usually outperforms all the existing fairness-aware anomaly detection baselines, which emphasizes the significance of our work in this field. In addition, we would like to further emphasize our contribution, including the novel re-balancing autoencoder, fairness-aware contrastive learning loss and the theoretical support for our proposed method.

Weakness 3: Lack of a discussion of related work

Answer 3: Please see our updated related work in Appendix B.

Weakness 4: Suggestions in writing and figure 2.

Answer 4: We have clarified the use of the fairness metrics in the revision to avoid confusion, and updated Figure 2 and its caption.