Ensemble Debiasing Across Class and Sample Levels for Fairer Prompting Accuracy

Thank you for your valuable review and the constructive comments! We appreciate that you point out the citation issue and will cite properly.

Regarding the question on how $a_k, b_k, c_k$ of the triangular membership functions are determined: we use a finite, pre-determined set of triangular membership functions ($\mu_k$'s), with $a_k, b_k, c_k$ controlling their active ranges. Our SA algorithm performs a constrained combinatorial optimization to search for the optimal combination of which specific $\mu_{\xi_i}$ and $\omega_{\xi_i}$ to assign to each class i, where $\xi_i$ is the selection index. The Heaviside functions H(⋅) in Eq. (4) dynamically selects $\mu_{\xi_i}$ or $\omega_{\xi_i}$ based on a fuzzy decision index $D_F$ (typically set between 10 to 20, covering 4 fuzzy slope partitions). Specifically, if $\xi_i \le D_F$, $\mu_{\xi_i}$ is selected. For instance, $\mu_1$ with $a_1=0, b_1=0, c_1=0.5$ applies linear downscaling from probability 0 to 0.5, peaking at 1 when $p=0$, enabling smooth low-range probability transformations.

Each SA step proposes a new combination of weight and membership correction functions, accepted if it improves the objective or satisfies the Metropolis criterion. This discrete optimization over function combinations allows flexible, post-hoc corrections at both class and sample levels - our key contribution beyond prior work.

Thank you for your insightful review and the constructive comments! We will correct the citation issue and appreciate your suggestion on visualizing the ablation of sample-level and class-level rebalancing. We will include bar charts (temporarily linked as https://anonymous.4open.science/r/colm-0D36/DCS_visualization.pdf).

Regarding hyperparameter tuning, generalizability and efficiency: most hyperparameters are fixed in practice, including the number of triangular membership functions and most SA parameters. Only five scalars are tuned: the weight correction scale, $\beta, \tau$ in Eq. (5), and $\lambda_1, \lambda_2$ in SA. Our post-hoc optimization averages 46 seconds (as computed from Figure 3 in the paper), significantly lighter than LLM fine-tuning — especially at the 70B scale, where bias remains significant and finetuning is costly. Our method adapts outputs directly, making it applicable to large or closed models that expose only logits or probabilities.

Regarding baselines: while class-balanced loss or other related losses can be used in fine-tuning smaller models, it does not guarantee balanced class accuracy, as it remains an indirect training surrogate. We appreciate the suggestion to compare with N-shot fine-tuning and will consider it in future or supplementary work.

Thank you for your valuable review and for highlighting a key practical consideration! We appreciate the opportunity to elaborate on the real-world applicability of our proposed DCS method.

Accessibility

DCS is model-agnostic, requiring only output probabilities, making it particularly suitable for large or closed LLMs where only logits/output probabilities are accessible.

Post-hoc optimization

The post-hoc, offline optimization of DCS requires no model architecture modification and prompt engineering. The annealing time averages 46 seconds (computed from Figure 3 in the paper). Hyperparameter tuning can be done efficiently because most hyperparameters are pre-configured in practice, with only a few scalar values (e.g., $\beta, \tau$) requiring tuning. Moreover, since the class-level correction method DNIP demonstrates low-data optimization capability, we'd infer that DCS is amenable to low-data scenarios. These make DCS particularly valuable in specialized applications where fairness and accuracy are paramount.

Negligible overhead

The learned correction functions are reused on-the-fly with mere milliseconds in prediction, introducing virtually no computation overhead or latency.

Scalability

Our method exhibits linear scaling with the number of classes ($N$). At each temperature in simulated annealing, the number of searches remains within several multiples of the search space size $N(D_F + D_W)$, ensuring practical feasibility across small and large classification tasks.

Thank you for your constructive review and valuable suggestions! We will add examples and discussions of how our method improves performance and on the underperforming case, and experiments on more model families.

Regarding concrete examples: we add visualizations on how sample-level and class-level corrections benefit the classes, with bar charts temporarily linked as https://anonymous.4open.science/r/colm-0D36/DCS_visualization.pdf.

Regarding more model families: we also appreciate the suggestion on experimenting with more model families. We experimented with a more recent LLM, Gemma-2-2B, and obtained consistent improvements similar to the Llama-2 cases, demonstrating that our method is applicable to LLMs of varied sizes and families. Additional results are shown in the table temporarily linked as https://anonymous.4open.science/r/colm-C8E3/DCS_gemma_results.pdf.

Regarding the underperforming case: DCS underperforms on the binary classification RTE task because the binary setting may not best exploit the ensemble flexibility. Compared to multi-class cases, with three configuration options (both classes use class-level corrections, a class uses class-level correction and the other uses sample-level correction, both classes use sample-level corrections), the benefit of diversity diminishes. FuRud (sample-level) performs best on accuracy for its capacity to push borderline samples to the correct class, while DNIP (class-level) excels on COBias via consistent global reweighting. DCS adds complexity, but its broad flexibility does not outperform in the simpler binary setting.