Debiasing Language Models Using Energy-Guided Ordinary Differential Equations

Bias-NLI

• Task Type: Natural Language Inference
• Prompt: Hypothesis: {hypothesis}\n\nPremise: {premise}\n\nResult: {label}
• Example: Hypothesis: A person on a horse jumps over a broken down airplane.\n\nPremise: A person is outdoors, on a horse.\n\nResult: entailment
• Settings:
  • Training Dataset: SNLI dataset
  • Evaluation Dataset: Bias-NLI dataset
  • Epoch: 3
  • Learning rate: 1e-4
  • Optimizer: paged_adamw_32bit
• Metrics:
  • Net Neutral (NN): The average probability of the neutral label across all sentence pairs.
  • Fraction Neutral (FN): The fraction of sentence pairs labeled neutral.
  • T: A parameterized measure that reports the fraction of examples whose probability of neutral above t: we report this for t = 0.5 and t = 0.7.
• Results:

Bias-in-Bios

• Task Type: Classification
• Prompt: CLS: {biographies}\n\nResult: {profession}
• Example: CLS: Prior to law school, Brittni graduated magna cum laude from DePaul University in 2011...\n\nResult: 2(attorney)
• Settings:
  • Training Dataset: Bias-in-Bios train split
  • Evaluation Dataset: Bias-in-Bios test split
  • Epoch: 3
  • Learning rate: 1e-4
  • Optimizer: paged_adamw_32bit
• Metrics:
  • acc_a: the overview accuracy.
  • acc_m: the accuracy of male instances.
  • acc_f: the accuracy of female instances.
  • TPR-GAP: the gap between acc_m and acc_f.
  • TPR-RMS: Root-mean-square deviation between male and female predictions.
• Results:

We thank the reviewer for the comments and questions! We hope the response can address all the raised concerns and questions.

Q1: Intrinsic and Extrinsic Evaluation

We conduct extrinsic evaluations on three benchmarks shown below. All of them employed the sequence-to-sequence style. For the classification tasks, we limited the vocabulary list to ensure we could access the logit of each class. The results revealed that our method exceeded the performance of the majority of baselines within these extrinsic benchmarks and was successful in striking an optimal balance between maintaining language modeling capability while minimizing bias.

Regarding the intrinsic benchmark [1], which is designed for the masked language models. It is not compatible with the generative language models. Meanwhile, recent studies [2, 3] have brought the validity of SEAT into question due to the counter-intuitive results it often produces. As referenced in [4], SEAT is found to lack a consistent correlation with extrinsic metrics. This finding implies that a model can potentially receive high scores from SEAT, while simultaneously rendering biased judgments in downstream conditions. Therefore, we have chosen to disregard SEAT as an evaluation metric in our work.

WinoBias

• Task Type: Coreference
• Prompt: COREF: {sentence}\n\nResult: {sentence_with_bracketed_coreference}
• Example: COREF: The laborer eats more than the receptionist as [he] is hungry. \n\nResult: [The laborer] eats more than the receptionist as [he] is hungry.
• Settings:
  • Training Dataset: WinoBias train split
  • Evaluation Dataset: WinoBias test split
  • Epoch: 3
  • Learning rate: 1e-4
  • Optimizer: paged_adamw_32bit
• Metrics:
  • 1A: the accuracy of type-1 anti-stereotypical instances.
  • 1P: the accuracy of type-1 stereotypical instances.
  • 2A: the accuracy of type-2 anti-stereotypical instances.
  • 2P: the accuracy of type-3 stereotypical instances.
  • TPR-1: the gap between 1P and 1A.
  • TPR-2: the gap between 2P and 2A
• Results:

Thank you for the rebuttal, the downstream experiments indeed strengthen the paper results, so I will increase the score to 5.

We thank the reviewer for their thoughtful feedback and hope the response can address the raised concerns and questions.

Q1: How is your sampling method connected to diffusion models?

We employ an ODE solver in this work, which can be regarded as a sort of ``deterministic'' diffusion model [1].

Our motivation is to use the ODE solver to gradually converge the latent variable obtained from the VAE encoder to the energy area we expect under the guidance of EBM. This time-variant process can be regarded as a reverse diffusion process. [1] stated the reverse of a diffusion process is also a diffusion process. Crucially, this reverse process satisfies a reverse-time SDE, which can be derived from the forward SDE given the score of the marginal probability densities as a function of time. Moreover, for all diffusion processes, there exists a corresponding deterministic process whose trajectories share the same marginal probability densities as the SDE. This deterministic process satisfies an ODE [2]. Above is the connection between our sampling method and diffusion models.

Q2: Why you have opted for ODE instead of Langevin dynamics?

Regarding why we opted for ODE instead of Langevin dynamics, here are reasons from these aspects:

• [Diversity] The latent variable is envisioned as a low-dimensional manifold embedded within a higher-dimensional latent space. Consequently, most points from Langevin Sampling (without noise) won't align with the manifold. This implies that the likelihood of these points will be zero, rendering the score function $\log p(x)$ undefined for these points. Hence, SGLD tends to easily settle into local optima. In our work, cyclic annealing was leveraged to introduce scaled noise to the sample data. As a result, the generated latent variable is not only more robust but also converges more quickly.

• [Speed] Similar to the first problem, LD without noise struggles to garner enough sample data to effectively guide the score function in areas of sparsity. Consider the energy distributions depicted in Figure 6, where most samples are tightly clustered leaving other areas distinctly sparse. LD might employ a smaller learning rate alongside an increased number of learning steps to work around this problem, however, this would inevitably compromise the speed. The DICE method proposed in our paper manages this issue by initiating scale disturbance as well. The Runge-Kutta method is a deterministic process, in contrast to Langevin sampling which necessitates MCMC sampling, which samples from the approximated distribution for every gradient update. Therefore, the Runge-Kutta method proves to be more efficient than Langevin sampling in this case.

• [Joint Modelling] In our empirical experiments, we found both SGLD and ODE could generate fluent completions with desired attributes for a single-bias control, while SGLD is slower and less diverse. However, the performance of SGLD drastically deteriorates under joint debiasing settings. This decrease in performance might be attributed to SGLD's naive disregarding of the weight each joint score function carries. For example, consider a joint score function $\log p(x) = \log (w_1 * p_1(x) + w_2 * p_2(x))$, SGLD employs $\nabla_x \log p(x) = \nabla_x \log p_1(x) + \nabla_x \log p_2(x)$ to sample from $p(x)$ which does not rely on these weights.

• [Optimization] The SGLD approach is highly sensitive to hyperparameters, often requiring substantial time for optimization without guaranteeing fluent completion (see the empirical experiment). On the other hand, the Runge-Kutta method exhibits consistent performance across all tested LLMs(GPT2-base, GPT2-large, Llama-7b, and Llama2-7b) using the same hyperparameters. This trait makes it much more practical and adaptable in real scenarios.

Thanks for your great effort in responding and revising. After reading the rebuttal, I am still very confused.

For Q1, you argue that the ODE is a probabilistic flow of the SDE. However, I can't understand what's the SDE of your model. What's your time-varying probability p_t and how do you learn a neural network to approximate p_t. To me, it seems that you just want to sample from the target distribution $p \propto \sum_{i} f_i(a_i | z)$, and you run a gradient decent to find its local optimal. Therefore, the resulting method is not a valid sampler.

For Q2, "... while SGLD is slower and less diverse". It is really supervised to me. Why a stochastic sampler is less diverse to a deterministic gradient decent?

For Q5, I think you misunderstood my question. It is true that if you just run a few steps, the resulting samples are not far away from the latent space learned by VAE. However, it is noteworthy that in eq.8, there is no constraint that ensures the samples should be sampled in the latent space of VAE, and it's very dangerous that your samples can be far away from the learned spaces. Therefore, I think doing projected gradient descent is better than doing gradient descent.

Thank you for your feedback. We are more than willing to answer your questions:

Q1:

Consider a diffusion process that iteratively blurs an image to reach a Gaussian distribution ($\mathcal{N}(0, I)$) in the forward process, and in reverse, iteratively samples an image back from a Gaussian distribution ($\mathcal{N}(0, I)$). In this context, the time-varying probability $p_0$ depicts the image data distribution, and $p_T$ represents the initial distribution $\sim\mathcal{N}(0, I)$. An illustrative example can be found in Fig. 1 of [1]. Our model emulates the reverse diffusion process, which can be represented as solving an initial value problem (IVP) via ODE. We aim for the latent variable to gradually accommodate the desired attribute, enabling us to quantitively control the bias level. To this end, we employ an ODE solver (Runge-Kutta of order 5 of Dormand-Prince-Shampine) to approximate the target distribution given the original encoder output. Unlike conventional SDE which samples from a normal distribution, we treat each training data as a sample and then convert it into a representation in the latent space through the encoder. We can change the name if you feel it is inappropriate to call it sampling.

We hypothesize that our main divergence lies in how to sample from the latent space effectively. Though gradient descent seems an intuitive approach to approximate the target distribution, it exhibits significant limitations in our scenario. In our empirical experiments, we found that gradient descent is laborious to train and tends to produce unstable output—an issue also noted in [1,2,3]. Moreover, our model is designed to debias multiple attributes simultaneously, a scenario that SGLD may not adeptly accommodate. In fact, SGLD's performance significantly deteriorates under joint debiasing settings. This decrease in performance can potentially be attributed to the naive disregard by SGLD of the weight that each joint score function holds. For instance, in the case of a joint score function $\log p(x) = \log (w_1 * p_1(x) + w_2 * p_2(x))$, SGLD employs $\nabla_x \log p(x) = \nabla_x \log p_1(x) + \nabla_x \log p_2(x)$ to sample from $p(x)$, without considering the corresponding weights.

Q2:

Slower: When two modes of a given data distribution are divided by areas of low density, Langevin dynamics may not effectively recover the relative weights of these modes within a reasonable time, and thus may not converge to the true distribution [1]. This analysis also holds when different modes have approximately disjoint supports - they might share the same support but be linked via areas of low data density.

In a joint debiasing scenario, where biases related to "gender", "race", and "religion" are to be addressed simultaneously, the common support may not be adequate to recover the target distribution using SGLD. In theory, Langevin dynamics could indeed generate accurate samples, but this might necessitate a very small step size and a huge number of steps for effective mixing.

Less diverse: The latent variable is a low-dimensional manifold embedded within a higher-dimensional latent space. If we consider the vanilla SGLD without annealed noise, most points from Langevin sampling won't align with the manifold [1]. This implies that the likelihood of these points will be zero, rendering the score function undefined for these points. Hence, SGLD tends to easily settle into local optima and cluster together. To address this problem, we introduce scaled noise to perturb the supports, despite the ODE's deterministic nature. Consequently, our approach can generate a wider range of diverse results.

Q5:

I agree with your point. There is indeed potential danger if the sample strays significantly away from the learning space. Current empirical observations indicate less impact on the decoder (likely attributable to the fewer than 10 ODE iterations performed). To further confirm these findings, we plan to conduct additional experiments, projecting parameters into the decoder’s constraint space, and providing theoretical boundaries for the latent variables. Your suggestions are greatly appreciated!

[1] Song, Yang, et al. "Score-based generative modeling through stochastic differential equations." arXiv preprint arXiv:2011.13456 (2020).

[2] Nie, Weili, Arash Vahdat, and Anima Anandkumar. "Controllable and compositional generation with latent-space energy-based models." Advances in Neural Information Processing Systems 34 (2021): 13497-13510.

[3] Liu, Guangyi, et al. "Composable text controls in latent space with odes." arXiv preprint arXiv:2208.00638 (2022).

Thank you for your swift response and insightful question.

I understand your quoted claim originated from the answer to your second question. Prior to delving into the discussion, let's establish that we will discuss diversity within the framework of the joint-debiasing scenario.

We hypothesize that the observed difference may be due to the SGLD's inability to account for the individual weights within each joint score function. For instance, given a joint score function $\log p(x) = \log (w_1 * p_1(x) + w_2 * p_2(x))$, SGLD utilizes $\nabla_x \log p(x) = \nabla_x \log p_1(x) + \nabla_x \log p_2(x)$ to take samples from $p(x)$. This approach does not rely on weight factors. Thus, it could be inferred that SGLD might not possess the capacity to effectively manage the multiple-mode distribution. This is a primary motivator behind our decision to opt for ODE over LD. This particular scenario is also highlighted in references [1,2,3] for your information.

It's a good question. A comprehensive comparison and analysis of LD and ODE might be substantial enough to merit a separate paper. We will leave it to future work.

[1] Nie, Weili, Arash Vahdat, and Anima Anandkumar. "Controllable and compositional generation with latent-space energy-based models." Advances in Neural Information Processing Systems 34 (2021): 13497-13510.

[2] Song, Yang, et al. "Score-based generative modeling through stochastic differential equations." arXiv preprint arXiv:2011.13456 (2020).

[3] Liu, Guangyi, et al. "Composable text controls in latent space with odes." arXiv preprint arXiv:2208.00638 (2022).

We appreciate your prompt reply!

The corresponding description can be found in Appendix 2.

$\mathcal{L}_{vae}(x)=-E\_{q(z|x)} [\log p(x|z)] + \beta \cdot \rm{KL} (q(z|x) \cdot noise || p\_{prior}(z))$

Briefly, this trick imports a coefficient $\beta$ that regulates the weight of the KL divergence on the VAE objective. The coefficient is gradually changed from 0 to 1. The motivation for doing this is as follows:

• Since the definition domain of Gaussian noise is the entire latent space, adding noise to the original data solves the zero probability problem of low-dimensional manifolds (otherwise the score function will not be able to steer the ODE solver).

• Adding noise essentially expands the range of each mode in the distribution, allowing the low-probability areas in the data distribution to get more supervision signals. In particular, this is very effective for the joint-debiasing scenario.

• The noise scale selection is not a simple problem. Adding too much noise will cover more low-probability space and drastically change the original data distribution, while too little noise will be useless to the less dense space.

Therefore, the final solution we ended up using was to add a 4-loop annealing factor to rescale the KL divergence.

Thank you for your valuable review and constructive suggestions! I hope the response can address all the raised concerns and questions.

Q1: Given longer enough context, the autoregressive model will ignore the effects of the latent variables, especially with powerful enough decoders. I see you use the cyclic annealing in Bert+GPT2 experiment to address this issue. However, I want to know more results and evaluation for more powerful models such as LLaMA-2.

In this study, we employ cyclic annealing and infusion mechanisms (AtM and PSA) to prevent mode collapse from the decoder. Noteworthily, our empirical results show that more powerful decoders even demonstrate better debiasing performance in our approach. We postulate this enhanced performance may be attributed to the fact that these more potent decoders have a refined comprehension of bias, thus enabling sounder debiasing effects through the guidance of the latent variable.

In our experiments, we have reported results for the GPT2, Llama-7B, and Llama 2-7B models respectively. Regarding the larger Llama 2 models (such as 13B and 70B configurations), we have commenced evaluation processes. We will update the data once we have gathered the necessary results as soon as possible.

Q2: Do you consider using learnable prior in the training rather than in a two-step manner? We leverage the feature of isotropic Gaussian to provide an initial distribution for the ODE solver. A more meaningful prior distribution can indeed alleviate the issue of VAE mode collapse. However, our concern lies in the ODE solver's potential inability to function adequately if the distribution deviates significantly from the Gaussian distribution. Consequently, we will conduct additional experiments to discern an effective way to strike a balance. Thank you once again for your valuable suggestion!

Q3: Latent space EBM citations.

We thank you for pointing out several important references that were missing in the original submission. We will carefully incorporate them in the final revision paper.

We greatly appreciate your valuable feedback! It is our sincere hope that our responses can effectively address your concerns.

Q1: More details about the AtM and PSA are needed.

Both AtM [1] and PSA [2] operate as the infusion mechanism connecting the encoder and decoder.

AtM projects the latent variable $z$ into both the attention key and value spaces by a MLP. It then concatenates them with the key and value vector in each attention layer within the decoder. This process can be thought of as the insertion of a "soft" prompt prefix ahead of the context. In this scenario, the latent variable functions as Q, with the Key and Value (KV) being projected by Q:

$k' = v' = mlp(z)$

$q = [q;z] \quad k = [k;k'] \quad v = [v;v']$

PSA is similar to AtM, except it has different K and V:

$k' = mlp_{k}(z) \quad v' = mlp_{v}(z)$

$q = [q;z] \quad k = [k;k'] \quad v = [v;v']$

Q2: What are the labels in Algorithm 1 line 9? The labels in Algorithm 1 line 9 can be found in Table 9.

Algorithm 1 represents the DICE training and inference procedure, whereas Crows-Pairs / StereoSet benchmarks are denoted as the evaluation procedure. Labels bear different meanings within these two procedures.

Within the context of Algorithm 1, labels can guide DICE to produce completions with various bias inclinations. For instance, in a race-bias context, the label "Black" might prompt DICE to produce completions that have a bias towards black-racism. Conversely, the label "Neutral" may direct DICE to yield completions with less racial bias.

In the Crows-Pairs Benchmark, the labels are defined as {more, less}, and the bias direction is categorized as {stereotypical, anti-stereotypical}. For instance, For example, the statement "Blacks never listen to their parents" is deemed more stereotypical compared to "Whites never listen to their parents". We propose that racial identifiers (“Blacks” and “Whites”) contribute to the formation of bias. Consequently, we employ the label "Neutral" in Algorithm 1 to generate a more "neutral" latent variable, and therefore mitigate the race bias in LLM completions.

[1] Li, Chunyuan, et al. "Optimus: Organizing sentences via pre-trained modeling of a latent space." arXiv preprint arXiv:2004.04092 (2020).

[2] Fang, Le, et al. "Transformer-based conditional variational autoencoder for controllable story generation." arXiv preprint arXiv:2101.00828 (2021).