FACTER: Fairness-Aware Conformal Thresholding and Prompt Engineering for Enabling Fair LLM-Based Recommender Systems

Thank you for your positive assessment. We address your points in the following paragraphs.

Weakness 1:

A1.

Our approach requires multiple hyper-parameters (e.g., $\lambda$, $\gamma$, $\tau_\rho$), which we tune via grid search on a 20% hold‑out calibration subset (see Section 4.2 and Appendix §A.2 for detailed ablation tables). Other methods such as Bayesian optimization ([1],[2]) can also be used. Although the number of hyper-parameters may appear excessive, it is comparable to (or smaller than) the parameter sets in other post‑hoc fairness algorithms (e.g., adversarial fine‑tuning or distribution‑level constraints, Madras et al., 2018; reweighting or threshold methods, Dwork et al., 2012; Angelopoulos et al., 2023). Empirically, moderate deviations in these hyperparameters do not substantially change fairness or accuracy, as demonstrated by our ablation studies in Tables 6–7. The need for such tuning is indeed common across state‑of‑the‑art fairness frameworks (Hua et al., 2023).

In our final manuscript, we will revisit and refine these hyperparameter discussions, incorporating the new references and clarifying our tuning strategies in the text.

References: [1] Shahriari, Bobak, et al. "Taking the human out of the loop: A review of Bayesian optimization." Proceedings of the IEEE 104.1 (2015): 148-175. [2] Frazier, Peter I. "A tutorial on Bayesian optimization." arXiv preprint arXiv:1807.02811 (2018).

Weakness 2:

A2.

As detailed in Section 3.4 of the original manuscript, we address scalability by using approximate nearest neighbor (ANN) methods for all offline calibration steps, reducing the naive complexity from O(n^2) to about O(n log(n)). We also employ GPU batch processing and other parallel optimizations, so in practice, the runtime sometimes scales sublinearly with n, as larger batches can be processed more efficiently. Table A4 illustrates this: MovieLens‑1M (around 6k users) takes around 40–65 minutes, whereas MovieLens‑20M ( around 138k users) extends to 6–8 hours, which is acceptable for overnight jobs.

Table A4: Approximate Calibration Times

In the final version, we will emphasize that ANN was used for all reported studies and add the above table and discussion to clarify how GPU batch processing and approximate search heuristics yield observed calibration times that sometimes grow sublinearly with dataset size, while theoretical complexity remains O(n log(n)).

Weakness 3:

A3.

Please refer to A1 in the Reviewer 49dx rebuttal section.

Thank you for carefully reading our work and your valuable comments. We address your concerns in the following paragraphs.

Weakness 1:

A1.

While FACTER leverages existing techniques such as conformal prediction and prompt engineering, to our knowledge, no prior work has unified these methods into a closed-loop, adaptive fairness calibration framework that dynamically refines prompts and thresholds to mitigate fairness violations in black-box LLM recommenders with statistical guarantees. Specifically, FACTER introduces the use of conformal prediction with fairness constraints, an iterative violation-triggered prompt refinement strategy that limits token bloat and generalizes to unseen biases, and provides a paradigm shift toward adaptive, statistically grounded fairness for black-box LLMs.

Unlike static fairness methods (e.g., UP5 [Hua et al., 2023]), FACTER uses conformal prediction ([Angelopoulos et al., 2023]; §3.2) to adaptively tighten thresholds based on violation rates while ensuring 1−α coverage (Eq. 7). This closed-loop calibration addresses distribution shifts and emergent biases, reducing violations by 95.5% vs. UP5 (Table 1 of the original paper). Prior works treat fairness and accuracy as separate objectives ([Dwork et al., 2012]; [Gallegos et al.]). FACTER’s score $S_i = d_i+λ Δ_i$ (Eq. 5) jointly optimizes both, enabling a principled tradeoff validated by ablation studies (Table 5 of the original paper).

Unlike static prompts (e.g., “avoid bias” in [Yang et al., 2022]), FACTER injects concrete bias patterns (e.g., “Gender=F → Romance-Only”) from a violation buffer (§3.3). This iterative refinement reduces token bloat while generalizing to unseen biases, outperforming Zero-Shot by 22× in violations (Table 1 of the original paper). Our proofs for embedding robustness (Theorem 1) and threshold convergence (Theorem 2) provide formal guarantees absent in prior fairness frameworks ([Shafer & Vovk, 2008]; [Madras et al., 2018]).

Therefore, FACTER is not a simple combination of tools but a paradigm shift toward adaptive, statistically grounded fairness for black-box LLMs.

Weakness 2:

A2.

Please refer to A1 in the Reviewer 49dx rebuttal section.

Weakness 3:

A3.

Thank you for your excellent suggestion. The choice of α directly controls the conformal coverage guarantee (1−α), ensuring that the probability of falsely flagging a fair recommendation as biased (Type I error) is bounded by α. This aligns with the theoretical foundations of conformal prediction (Angelopoulos et al., 2023). Specifically, setting α=0.10 provides a 90% coverage guarantee, meaning 90% of fair recommendations will not be erroneously flagged.

However, lowering α (e.g., α=0.05) tightens the threshold, reducing Type I errors but potentially increasing Type II errors (failing to detect true violations). Conversely, higher α (e.g., α=0.20) relaxes the threshold, increasing Type I errors but improving detection power. To validate this trade-off empirically, based on the reviewer’s comment, we conducted an ablation study (Table 1 below) on MovieLens-1M, measuring violations, Type I/II errors, and recommendation quality. The results are reported in Table A1.

Table A5: Impact of α on Fairness-Accuracy Tradeoff

As the results show, lower α (e.g., 0.01) prioritizes strict fairness (fewer violations, low Type I errors) but risks missing true biases (higher Type II errors). Higher α (e.g., 0.20) improves detection power (lower Type II errors) at the cost of increased false alarms.

In the final manuscript, we will expand Section 3.2 to explicitly discuss how α governs the Type I/II error trade-off, referencing Eq. (7) in our paper and Theorem 1 in Appendix J.1.1.

Weakness 4:

A4.

Please refer to A2 in the Reviewer b7t4 rebuttal section.

Moreover, regarding your concern about essential references, we will include the mentioned [3,4] references, which are survey papers, to the final version of the paper. Unfortunately, we do not know what other missing references you have in mind. If there is an opportunity for you to anonymously provide these additional references to us, we will be grateful and will add them (and even provide comparisons with the most relevant ones) to the final manuscript.

Thank you for your thorough review. We address your comments/concerns in the following paragraphs.

Weakness 1:

A1.

Our theoretical guarantee (Theorem 1) assumes that the calibration set is approximately exchangeable with future test data. Hence, the quality and diversity of the calibration set are essential, and we have considered them in our assumptions. Moreover, we can follow some guidelines to ensure the exchangeability of the test distribution with the calibration set. These guidelines are as follows: (i) stratified sampling across key user demographics to ensure diverse coverage of protected attributes, (ii) balancing user groups so that each protected category is well-represented, and (iii) periodic refresh of calibration data in real-world deployments to track shifting user populations or model updates. In addition, we performed experiments across 3 calibration seeds and found that fairness metrics (e.g., CFR, violations) varied by <5%, indicating stability under reasonable data variations. These explanations and experimentation will be added to the final manuscript. Finally, regarding the embedding bias robustness, please refer to A1 in the Reviewer 49dx rebuttal section.

Weakness 2:

A2.

The non-conformity score $S_i = d_i+ λ Δ_i$ integrates accuracy and fairness by combining a predictive error term ( $d_i$ , cosine distance between recommendations and user preferences) and a fairness penalty ($Δ_i$ , maximum embedding divergence across demographic groups for similar users). The tradeoff parameter λ=0.7, selected via grid search, balances Pareto-optimal fairness-accuracy tradeoffs, reducing violations by 95% while retaining 98.7% recommendation quality (Appendix Table 5 of the original paper). Grounded in multi-objective optimization, this additive design enforces individual fairness by penalizing disparities between comparable users, validated empirically to outperform multiplicative alternatives. The score ensures equitable relevance in black-box LLMs without internal access.

We will provide this additional discussion on the design rationale in the final version.

Weakness 3.1:

A3.1.

The embedding $e^{y}_{\text{new}}$ is computed as:

• Directly, if the ground-truth $y_{new}$ is available: $e^{y_{\text{new}}} = \mathrm{Emb}(y_{\text{new}})$

• Via Approximate Nearest Neighbor (ANN mentioned in Section 3.4) , if $y_{new}$ is not available (e.g., cold-start), we retrieve the closest calibration item using ANN and use its embedding.

We will clarify this process comprehensively in the final version.

Weakness 3.2:

A3.2.

Thank you for your comment. Yes, you are correct the three LLMS have nearly the same parameter counts. What we meant to say is that these models are architecturally different or at least different versions (in case of LLaMA models.) While LLaMA3-8B, LLaMA2-7B, and Mistral-7B have similar parameter counts, their architectures differ significantly: LLaMA uses a pure decoder, while Mistral integrates sliding window attention. Architectural diversity tests FACTER’s generalizability, a key strength highlighted in §4.2.

We will revise the text to correct the sentence.

Thank you for your detailed feedback. We address your comments in the following paragraphs.

Weakness 1 and Q1:

A1.

As noted in Section 3.4, we acknowledge that any single embedding model can carry bias. Our theoretical analysis (Appendix §J.1.1, Theorem 1) shows that if embeddings drift by $\epsilon_{emb}$, the fairness score Si changes by at most $ΔS_i$ ≤2(1+λ)$\epsilon_{emb}$. We initially used the simplified bound 3ϵemb for λ≤0.5, but our chosen λ=0.7 yields $ΔS_i$≤3.4$\epsilon_{emb}$, which remains manageable for typical $\epsilon_{emb}$ ≈0.05–0.1. Empirically, FACTER reduces fairness violations by 95.5% (Table 1), showing that real‑world performance is robust despite minor embedding shifts. By using violation‑triggered prompt updates, FACTER incrementally mitigates embedding flaws, adding explicit “avoid” examples to steer the LLM away from bias (e.g., Table 3). We further tested two additional embedders (Sentence‑BERT‑base and RoBERTa‑large‑nli‑stsb‑mean‑tokens) in Table A1, finding similar reduction in violations and near equal NDCG@10, indicating minimal impact from different initial biases:

Table A1: Embedding Consistency Test

In the final version, we will expand Section 3.4 with a concise overview of domain‑tailored fine‑tuning, adversarial or hard‑debiasing methods (Bolukbasi et al., 2016), and multi‑embedder ensembles to reduce reliance on a single model. We will outline how practitioners can retrain or lightly tune embedding layers using curated, bias‑filtered corpora, integrate adversarial loss terms that penalize demographic correlations, or combine embeddings from different SentenceTransformer variants. We will also include Table A1 and its discussion in the final paper.

Weakness 2 and Q2:

A2.

We store only the 50 most recent bias‑avoid examples. This prevents unbounded prompt growth. As shown in Table 7 (Appendix section of the original manuscript), despite this constraint, this approach reduces violations by ~90% without hitting token limits.

Q3:

A3.

Thank you for the suggestion. Although we did not include real-user feedback in the original manuscript, it is straightforward to do in FACTER. To illustrate, we simulated a scenario where synthetic users correct 10–30% of flagged violations. We augmented FACTER’s threshold updates with these “corrected” examples and observed how violations fell over three online calibration iterations on MovieLens‑1M. Table A2 shows that even modest correction rates (10–30%) accelerate violation reduction. With 30% correction, violations drop to 1 (versus 5 in the baseline), confirming that user feedback can enhance FACTER’s fairness calibration. In the final paper, we will add a subsection in Section 5 (Future Work) discussing how automated detection and human‑in‑the‑loop validation can be combined.

Table A2: Synthetic User Feedback Impact

We will include the table and corresponding results in the final paper.

Weakness 3:

A4.

While our current focus is on counterfactual fairness via minimal attribute changes, we agree that real-world applications often require multi-attribute fairness. Our formulation naturally extends to this case by treating the sensitive attribute as a vector $a=(a_1,…,a_k)$, and requiring that for any (x,a) and (x,a′) differing in at least one component, ∥y(x,a)− y(x,a′)∥≤δ. Under conformal calibration, the non-conformity score becomes $S = d + λ max_{j∈N(x)} ∥Emb(y)−Emb(y_j)∥$, where N(x) includes calibration points with the same non-sensitive features but differing in at least one attribute dimension. The same coverage guarantee (Eq. (7)) applies due to exchangeability. To validate this, we conducted a multi-attribute evaluation on the MovieLens dataset (using both Gender and Age). As shown in Table A3, FACTER remains effective under this setting, with minimal accuracy drop.

Table A3: Multi-Attribute Fairness Evaluation (Preliminary Results)

We will include this discussion and additional experiments in the final version of the manuscript.