PLS-based approach for Fair Representation Learning

• My main concern is with the empirical results and the settings.
  • The learned fair representations in the downstream tasks are evaluated using different target models but are not compared with other baseline methods, which makes the effectiveness of the proposed methods less convincing. The authors need to demonstrate that through PLS, the learned fair representations can achieve better accuracy, fairness, or efficiency compared to other methods (for example, using VAE or other disentanglement methods).
  • The chosen datasets are all tabular, and the dimensionality is not very high. Therefore, I am concerned about the proposed method’s performance with high-dimensional data (e.g., image data).
• The motivation for the proposed method is weak. Although incorporating fairness constraints into PLS is a novel attempt, the justification for using PLS is not strong. In the introduction (lines 121-141), the authors introduce existing fair representation learning methods but do not sufficiently justify the benefits of using PLS. The only comparison made is with PCA, stating that PLS-built features are more accurate than PCA components, which is not enough to fully support the choice of PLS.
• In the paragraph (lines 274-283), the authors explain why Fair PLS cannot be formulated in closed form. As a result, the algorithm requires iterations to solve for the weight $w_h$ . In each iteration, it requires the re-computation of the eigenvectors, when the dimension is large, this would increase computation cost and decrease the efficiency.

Although I appreciate the presentation of the work, I have the following concerns.

1. The empirical comparison to previous fair PCA is only tested on one dataset (Adult Income) and provided in the Appendix. It's not convincing the proposed work will outperform previous work in most cases.
2. Although possible application to fairness in LLM is discussed, it's rather superficial. Unless you have done experiments on LLM to measure fairness, I do not suggest this as a separate section in the main paper.
3. The experiments are conducted on simple tabular data. Will it be possible to test on more complex and high-dimensional datasets because you are considering dimension reduction?
4. Several related works in fair representation learning are missing for discussion [1,2,3].

[1] Kim, Jin-Young, and Sung-Bae Cho. "Fair representation for safe artificial intelligence via adversarial learning of unbiased information bottleneck." SafeAI@ AAAI. 2020.

[2] Shui, Changjian, et al. "Fair representation learning through implicit path alignment." International Conference on Machine Learning. PMLR, 2022.

[3] Zamani, Amirreza, Borja Rodríguez-Gálvez, and Mikael Skoglund. "On information theoretic fairness: Compressed representations with perfect demographic parity." arXiv preprint arXiv:2408.13168 (2024).

I have the following questions and concerns regarding the contribution and evaluation of the work:

1. Motivation for PLS-based Framework: It would be helpful for the authors to clarify the motivation behind selecting PLS as the foundation for their framework. There are various types of approaches for fair representation learning, such as adversarial learning [1], disentanglement [2], and distribution alignment [3]. What are the specific advantages of a PLS-based approach in comparison to these methods?

2. Applicability and Practical Constraints: The proposed method requires annotations for both the target label $Y$ and the sensitive attribute $S$, which can limit practical applications. Compared to existing approaches in unsupervised fair representation learning or those that do not rely on sensitive attribute annotations (such as [4]), it would be beneficial for the authors to further clarify the unique advantages of their method, potentially in terms of efficiency, theoretical guarantees, or effectiveness.

3. Evaluation and Comparison: The evaluation of the proposed method is based on relatively small datasets and lacks comparisons with related approaches. The current experiments seem to focus on applying the learned representations to various classifiers (like in Figure 1) rather than comparing with alternative representation learning methods. The lack of thorough evaluation and comparison makes it challenging to validate the effectiveness of the proposed method.

4. Extension to LLMs: It's helpful that the authors discussed in Section 4.3 about extending their method to Large Language Models (LLMs), where their method could decompose the CLS-embedding from a transformer encoder for fairness constraints. However, it's suggested that the authors could provide a more detailed discussion (with math formulation) in this section and conduct experiments to validate this extension.

5. Coupling Issue in Fair Representation Learning: Fair representation learning often involves a trade-off between fairness constraints and downstream task performance. It would be insightful if the authors could discuss how their method might address or mitigate this issue.

6. (Minor point on notations) In the previous context, the authors use x to refer to the input data, but in Section 4.3, x refers to the transformed data in the latent space. The authors may use a different symbol to avoid confusion.

[1] Madras, David, et al. "Learning adversarially fair and transferable representations." International Conference on Machine Learning. PMLR, 2018.

[2] Balunovic, Mislav, Anian Ruoss, and Martin Vechev. "Fair Normalizing Flows." International Conference on Learning Representations. 2022.

[3] Creager, Elliot, et al. "Flexibly fair representation learning by disentanglement." International conference on machine learning. PMLR, 2019.

[4] Chai, Junyi, and Xiaoqian Wang. "Self-supervised fair representation learning without demographics." Advances in Neural Information Processing Systems 35 (2022): 27100-27113.

• The paper suggests using Gradient Descent for optimization without discussing convexity. Even empirical testing would be valuable - such as showing convergence over epochs to identify patterns reflecting non-convex problems (like bumps).
• Section 4.3 appears disconnected and seemingly unrelated, particularly as it doesn't address all challenges in seq-to-seq fairness problems, only covering the relatively straightforward fact that text can be encoded.
• A main concern is that no comparison to other methods exists in the main text. Given this paper doesn't focus on groundbreaking theoretical results (fair decomposition methods are not new), such comparisons should be of utmost importance. For example, for fair representation there are more than a few algorithms already addressed in this paper, as well as other leading candidates like [1] or [2]. This is specifically true for the fair-classification setup where many libraries have been published, like IBM's [3] and [4] to name a few.

[1] "Fair Normalizing Flows" [2] "Efficient Fairness-Performance Pareto Front Computation" [3] "AI Fairness 360" [4] "Aequitas: A bias and fairness audit toolkit"