Seeing the Unseen: How EMoE Unveils Bias in Text-to-Image Diffusion Models

Dear Reviewer,

Thank you for your insightful comments. Please find below our responses and the corresponding changes, which have also been highlighted in the provided PDF document in red.

Comment regarding MoE unfamiliarity and Pseudocode

MoE creates an efficient ensemble within a submodule of the generation pipeline by taking predictions from each expert—viewed as individual ensemble components—and aggregating them into a final output. EMoE separates this aggregation process, treating the outputs from each expert independently to estimate epistemic uncertainty. Rather than creating $M$ distinct models, we leverage the experts as our ensemble within a network submodule, making our approach significantly more efficient in terms of both computation and parameter count. To clarify this further, we have added the following text to the paper:

The parameter count for the Segmoe model is 1.63 billion parameters, while a single model contains 1.07 billion parameters. This highlights the efficiency of using a sparse MoE approach compared to creating 4 distinct models, as the Segmoe model is only 153% the size of a single model, rather than 400%.

EMoE then evaluates the discrepancies among these experts at a predefined stage in the reverse diffusion process. For clarity, we have also added pseudocode in Appendix, Algorithm 1 with reference to Figure 2 and 3.

Comment regarding gating

The gating network used in our mixture of experts approach is not novel to this paper and follows established practices within the community. In our setup, each expert is associated with a positive and a negative descriptor prompt. The positive descriptor highlights the expert’s strengths, while the negative descriptor outlines its limitations. When a positive and negative prompt are provided for image generation, the system calculates gating weights for each expert based on the distance between the input prompts and the corresponding descriptor prompts in a latent space weighting them. To provide further clarity, we have included a detailed explanation in the appendix.

Comments regarding typos

Thank you for pointing these out, we have changed shorter to longer and defined $CA^i$ in the paper:

Note that $CA^i$ denotes the cross-attention output from the $i$th component Attention($Q^i, K^i, V^i$).

Comment regarding small deviation in CLIP score

The relatively small differences in CLIP scores in Figure 4 and Table 1 can be attributed to each expert's prior exposure to the training data from Q1, Q2, Q3, and Q4. Consequently, well-trained experts are expected to consistently generate high-quality images for all prompts within the training distribution. A notable feature of EMoE is its ability to detect subtle variations in uncertainty, even with in-sample data. To further clarify, we have added the following text:

Given that each expert has been trained on all data in the COCO dataset, EMoE’s ability to detect subtle differences in uncertainty on in-sample data is a notable feature

Figure 7 underscores the languages these models may or may not have been trained on, highlighting EMoE’s capability to accurately capture model biases toward specific languages. To enhance this point, we have added the following text for clarity:

This section and Section 4.2 illustrate the model’s bias toward certain languages and reveal its unfairness toward non-European languages. This demonstrates how EMoE can be utilized to detect biases and identify the data necessary for training to mitigate these issues.

Comments regarding Figure 6

The purpose of this image is to demonstrate that Finnish prompts containing the word "pizza" were more text-aligned, indicating a bias in the model towards certain prompts. We acknowledge the reviewer's suggestion that the image could be clearer in conveying this point. To address this, we have updated both the image and the accompanying text for clarity. The revised image now compares a prompt featuring the word ``pizza” and a random Finnish prompt. Please note that the model received the Finnish prompt directly, while the English translation is provided for reader reference.

Comments regarding how Segmoe works

Given the complexity of the code, which builds on numerous submodules such as VAEs, UNets, tokenizers, attention mechanisms, diffusion processes, and more, it is not feasible to cover the entire pipeline exhaustively. Instead, we have selectively highlighted sections of the code that are crucial to explaining the distinct aspects of EMoE. We believe that the clarifying text, particularly in the appendix, will enhance the overall understanding and clarity of the code. In addition, the code is provided for full transparency.

I thank the authors for their response.

On MoE/Gating explanation

I still find the entire setup of EMOE poorly explained. To my knowledge, MoE using pretrained DMs with SegMOE is not common in the research literature, so it is important to explain clearly how the gating and MoE mechanism works. The pseudocode is a good start. I suggest the authors also include a pseudocode on the gating process, e.g., explicitly how to calculate the weights assigned to each expert.

It took me some effort to realize the various experts are full diffusion models trained on different datasets, and the 'positive' and 'negative' prompts that define the gates are user-injected knowledge based on the training dataset. This is buried in App E on model cards. I think this must be more clearly stated in the main text as this definition of MoE departs significantly from the MoE referred to in the literature on LLMs, for example, where the gates are trained.

How does the sparse MoE work? This seems like a crucial detail missing which forms a huge advantage of the work. I'm not sure how one can combine four diffusion models and only require 153% parameters of a single model without training some specific routing mechanism. It looks like the magic comes from the implementation of SegMOE, which is an open-source library on GitHub. However, this is a research paper, so I think crucial details like this should be explained without expecting the reader to consult code of a large library as I believe sparse MOEs using pretrained DMs is not common in the research literature.

Typo

I think the typo is still there in line 359: it says Q1 and Q2 have shorter character count.

On Motivation

On reading the comments by the other reviewer, I encourage the authors to more clearly motivate why measuring EU is important in text-to-image modeling beyond measuring bias in datasets, which I think is a small point. For instance, authors have hinted at this as higher EU means lower CLIP score in Fig 4. Authors can be clear about this as an advantage of measuring EU, and perhaps supplement with additional experiments showing relationship of image quality (eg. FID) and EU. Otherwise, it seems from Fig 1 that qualitatively, the model has no issues producing high-quality images that correspond to the prompt even for high EU prompts.

Dear Reviewer,

Thank you for your insightful comments. Please find below our responses and the corresponding changes, which have also been highlighted in the provided PDF document in red.

Comment regarding Figure 1

Epistemic uncertainty provides practitioners with insights into inputs that were absent during training and areas where the model may be extrapolating. Accordingly, Figure 1 illustrates that images featuring Black male, White female, and Black female presidents exhibit higher uncertainty, suggesting that the model may not have encountered many such examples during training. This highlights bias within the training dataset and aligns with historical context, given the absence of female presidents and the presence of only one Black president. In general, diffusion models are capable of producing high-quality outputs regardless of whether the prompts have been encountered before or not during training.

Comment regarding not well-posed mathematically

Here is an intuitive explanation for our choice of estimator for epistemic uncertainty using the theory of Gaussian Processes. Each expert can be viewed as a sample from the posterior distribution of functions given an input y, denoted as $p(f (y)|y)$. By calculating the variance across these experts, we obtain the variance $\sigma^2$ of $p(f (y)|y)$, which serves as an estimate of epistemic uncertainty within the Gaussian Process framework. In general, other works have used the differences among ensemble components to represent epistemic uncertainty [1,2,3]. We have included this explanation, with additional mathematical rigor, in the appendix of our paper and highlighted it in the main text as follows:

The intuition behind this choice of epistemic uncertainty estimator is detailed in Appendix C.

Comment regarding CLIP score

Although CLIP score is known to exhibit bias [4], it remains a primary method for evaluating the quality of text-generated images, alongside the FID [5,6]. Both metrics rely on auxiliary networks (CLIP and the Inception network) for evaluation and are therefore both susceptible to bias. FID, however, requires a large number of samples for reliable estimates, whereas the CLIP score allows for more direct text-to-image alignment evaluation with fewer samples [7,8]. Given these factors, we prioritized the CLIP score for its suitability to our research objectives and its broad acceptance within related studies [5,7,8,9]. To further address potential concerns, we conducted an additional experiment evaluating EMoE’s performance using SSIM, detailed in Appendix B. The following text was added:

The CLIP score, despite its known biases (Chinchure et al., 2023), remains a widely-used method for evaluating the alignment between text prompts and generated images, alongside FID (Shi et al., 2020; Kumari et al., 2023). Both metrics, however, rely on auxiliary models (CLIP and Inception, respectively), making them susceptible to inherent biases. While FID requires a large number of samples for reliable estimation, the CLIP score facilitates a more direct assessment of text-to-image alignment with fewer samples (Kawar et al., 2023; Ho et al., 2022a). Considering these trade-offs, we prioritized the CLIP score due to its relevance to our research objectives and its broad acceptance in related studies.

To further validate our findings and address any potential concerns related to metric biases, we conducted an additional experiment using the Structural Similarity Index (SSIM) as the evaluation metric. Unlike CLIP or FID, SSIM does not depend on any auxiliary models for its calculation, thereby mitigating the risk of bias. We computed SSIM between generated images and corresponding ground-truth images from the COCO dataset and analyzed the results for each uncertainty quartile. As shown in Table 5, EMoE effectively categorized prompts into the appropriate uncertainty quartiles based on model performance. This provides further evidence of EMoE’s efficacy in estimating uncertainty for MoE text-to-image models, highlighting its robustness across different evaluation metrics.

You are correct that the CLIP score is generally lower for non-English prompts due to limited training data for CLIP in other languages. However, when evaluating the images generated from Finnish prompts, English prompts were then used to assess their CLIP score. We have added the following text for clarification:

Note that when evaluating the CLIP score for images generated from non-English prompts, the English version of the prompt was used for assessment. This was done to account for the fact that CLIP was primarily trained on English data.

Comment regarding MoE as an ensemble

We refer the reader to [10], which defines MoE (Mixture of Experts) as an ensemble method in Chapter 4.3. The ensemble methodology is defined as follows in [10]:

The main idea behind the ensemble methodology is to weigh several individual pattern classifiers and combine them, in order to obtain a classifier that outperforms all of them.

Under this definition, MoE qualifies as an ensemble method, even if it selects the top expert and assigns zero weights to the remaining experts. EMoE extends this concept by disentangling the ensemble and treating each expert component as an independent model for estimating uncertainty.

References

[1] Yarin Gal, Riashat Islam, and Zoubin Ghahramani. Deep bayesian active learning with image data. In International Conference on Machine Learning, pp. 1183–1192. PMLR, 2017.

[2] Stefan Depeweg, Jose-Miguel Hernandez-Lobato, Finale Doshi-Velez, and Steffen Udluft. Decomposition of uncertainty in bayesian deep learning for efficient and risk-sensitive learning. In International Conference on Machine Learning, pp. 1184–1193. PMLR, 2018.

[3] Lucas Berry and David Meger. Normalizing flow ensembles for rich aleatoric and epistemic uncertainty modeling. Proceedings of the AAAI Conference on Artificial Intelligence, 37(6):6806–6814, 2023b.

[4] Aditya Chinchure, Pushkar Shukla, Gaurav Bhatt, Kiri Salij, Kartik Hosanagar, Leonid Sigal, and Matthew Turk. Tibet: Identifying and evaluating biases in text-to-image generative models. arXiv preprint arXiv:2312.01261, 2023.

[5] Zhan Shi, Xu Zhou, Xipeng Qiu, and Xiaodan Zhu. Improving image captioning with better use of captions. arXiv preprint arXiv:2006.11807, 2020.

[6] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bj¨orn Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10684–10695, 2022

[7] Nupur Kumari, Bingliang Zhang, Richard Zhang, Eli Shechtman, and Jun-Yan Zhu. Multi-concept customization of text-to-image diffusion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1931–1941, 2023.

[8] Bahjat Kawar, Shiran Zada, Oran Lang, Omer Tov, Huiwen Chang, Tali Dekel, Inbar Mosseri, and Michal Irani. Imagic: Text-based real image editing with diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6007–6017, 2023.

[9] Jonathan Ho, Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, and David J Fleet. Video diffusion models. Advances in Neural Information Processing Systems, 35:8633–8646, 2022b.

[10] Rokach, Lior. Ensemble learning: pattern classification using ensemble methods. 2019.

Dear Reviewer,

Thank you for your insightful comments. Please find below our responses and the corresponding changes, which have also been highlighted in the provided PDF document in red.

Comment regarding training of the gating network

You are correct in their assertion that the gating function of MoEs, as commonly used since their inception, requires training/fine-tuning. However a new gating method that does not require fine-tuning or re-training has emerged with LLMs and has become an established practice in the community (many models based on this approach are available on hugging face). The gating network used in our MoE approach is based on established community practices [1]. In our configuration, each expert is assigned to a positive and negative descriptor prompt. The positive descriptor emphasizes the expert’s strengths, while the negative descriptor highlights its limitations. During image generation, when a positive and negative prompt are provided, the system computes gating weights for each expert by measuring the distance between input prompts and corresponding descriptors in a latent space. This gating approach allows the re-combination of arbitrary expert models, that could be trained at differing times, on differing datasets, without fine-tuning. We have added a detailed explanation in Appendix D for further clarity.

Comment regarding computational burden of more experts

Using EMoE, it is possible to estimate uncertainty without generating an image for each expert. Instead, the variance of the midblock can be estimated at the initial reverse diffusion step, while employing the standard aggregation procedure from MoEs during subsequent image generation steps. This approach effectively reduces the computational burden associated with image generation. This enhanced version of EMoE results in only a minimal increase in computational time. The following addition to the paper describes this technique and quantifies the associated computational cost:

When running the SegMoE model in its standard mode, generating an image from one prompt takes an average of 3.58 seconds. In comparison, using EMoE typically requires an average of 12.32 seconds to generate four images from a single prompt. However, for scenarios where only one image per prompt is needed, EMoE’s output can be optimized by estimating epistemic uncertainty during the initial diffusion step, followed by standard MoE-based image generation. This optimized version of EMoE achieves an average generation time of 5.5 seconds. Table 10 provides further details.

Comment regarding the small deviation in CLIP score in Figure 4 and Table 1

The differences in CLIP scores in Figure 4 and Table 1 are relatively small because each expert has previously encountered the training data from Q1, Q2, Q3, and Q4. Consequently, if the experts are well-trained, they are expected to consistently produce high-quality images for all prompts within the training distribution. The capacity of EMoE to detect subtle variations in uncertainty, even with in-sample data, is a significant feature. To further clarify, we have added the following text:

Given that each expert has been trained on all data in the COCO dataset, EMoE’s ability to detect subtle differences in uncertainty on in-sample data is a notable feature.

Comment regarding background section

Given the complexity of the pipeline, we believe it is important to provide a thorough overview of uncertainty, UNets, and MoE. Accordingly, we have expanded the discussion on the methodology in the appendix.

Comment regarding diversity of experts

We provide an ablation study on the diversity of experts. In the Runway ML MoE, each expert represents a different version of Runway ML, illustrating an example of very similar models. EMoE demonstrates strong performance in this experiment (Figure 8d), showcasing EMoE's ability to detect uncertainty even among highly similar models. To clarify further, we have added the following text:

Additionally, this demonstrates that EMoE can detect uncertainty even within the context of very similar models.

References

[1] Goddard, Charles, et al. Arcee's MergeKit: A Toolkit for Merging Large Language Models. arXiv preprint arXiv:2403.13257 (2024).

Comments on hallucinations

We address the connection between EMoE and the concept of hallucinations in generative models in the related works section, lines 510-513. The concept of hallucinations in such models is still in its early stages of development, as evidenced by the reference you provided [10], published in the summer of 2024 and still undergoing formal definition. Our work does not aim to formally define hallucinations; rather, we have offered a brief commentary on the topic. Additionally, we do not attempt to capture hallucinations such as in [10] since we did not train the networks ourselves and lack detailed knowledge of the training data, making it impossible to accurately determine what constitutes a hallucinated response. Therefore, it does not make sense to use [10] as a baseline given our experimental setup.

Comments on Figure 1 and Detecting Biases and Fairness

We acknowledge that the paper does not specifically address racial and gender biases as illustrated in Figure 1. However, we present multiple experiments demonstrating how EMoE can be employed to detect language biases in MoE diffusion models, as discussed in Sections 4.2 and 4.3. This highlights which languages are underrepresented by these models, showcasing their implications for fairness. Practitioners may leverage EMoE similarly, as shown in Sections 4.2 and 4.3, to fine tune diffusion models on underrepresented languages. It is worth noting that Figure 1 was intended as an accessible and illustrative motivating example. We have added the following text to the paper to clarify this:

This section and Section 4.2 illustrate the model’s bias toward certain languages and reveal its unfairness toward non-European languages. This demonstrates how EMoE can be utilized to detect biases and identify the data necessary for training to mitigate these issues.

References

[1] Zhuobin Zheng, Chun Yuan, Xinrui Zhu, Zhihui Lin, Yangyang Cheng, Cheng Shi, and Jiahui Ye. Self-supervised mixture-of-experts by uncertainty estimation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pp. 5933–5940, 2019.

[2] Lucas Luttner. Training of neural networks with uncertain data, a mixture of experts approach. arXiv preprint arXiv:2312.08083, 2023.

[3] Rongyu Zhang, Yulin Luo, Jiaming Liu, Huanrui Yang, Zhen Dong, Denis Gudovskiy, Tomoyuki Okuno, Yohei Nakata, Kurt Keutzer, Yuan Du, et al. Efficient deweahter mixture-of-experts with uncertainty-aware feature-wise linear modulation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp. 16812–16820, 2024.

[4] Aditya Chinchure, Pushkar Shukla, Gaurav Bhatt, Kiri Salij, Kartik Hosanagar, Leonid Sigal, and Matthew Turk. Tibet: Identifying and evaluating biases in text-to-image generative models. arXiv preprint arXiv:2312.01261, 2023.

[5] Zhan Shi, Xu Zhou, Xipeng Qiu, and Xiaodan Zhu. Improving image captioning with better use of captions. arXiv preprint arXiv:2006.11807, 2020.

[6] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bj¨orn Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10684–10695, 2022

[7] Nupur Kumari, Bingliang Zhang, Richard Zhang, Eli Shechtman, and Jun-Yan Zhu. Multi-concept customization of text-to-image diffusion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1931–1941, 2023.

[8] Bahjat Kawar, Shiran Zada, Oran Lang, Omer Tov, Huiwen Chang, Tali Dekel, Inbar Mosseri, and Michal Irani. Imagic: Text-based real image editing with diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6007–6017, 2023.

[9] Jonathan Ho, Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, and David J Fleet. Video diffusion models. Advances in Neural Information Processing Systems, 35:8633–8646, 2022b.

[10] Aithal, Sumukh K., et al. Understanding Hallucinations in Diffusion Models through Mode Interpolation. arXiv preprint arXiv:2406.09358 2024.

Dear Reviewer,

Thank you for your insightful comments. Please find below our responses and the corresponding changes, which have also been highlighted in the provided PDF document in red.

Comments Regarding Uncertainty in Previous MoE Papers

We appreciate you bringing up the missing related work. We would like to clarify and expand upon the treatment of uncertainty in these references:

• [1] does not estimate epistemic uncertainty; instead, the uncertainty estimation is isolated to the critic component, where the variance of a Gaussian distribution is estimated using negative log-likelihood as the loss function. Importantly, the MoE utilized in [1] is separate from the critic, meaning that the uncertainty estimation does not originate from a MoE component. Furthermore, their approach necessitates training the entire pipeline, precluding zero-shot capabilities.

• [2] specifically addresses aleatoric uncertainty in MoE models. Our work, in contrast, focuses on epistemic uncertainty, offering a distinct contribution in this space.

• [3] does not report any uncertainty estimates. Instead, it leverages uncertainty to assign experts more efficiently within MoE models. Additionally, it is noteworthy that [3] requires training its uncertainty-aware routing (UaR), whereas our proposed EMoE approach estimates uncertainty without requiring any training.

It is important to highlight that none of these approaches—[1], [2], or [3]—estimate epistemic uncertainty and all require training from scratch. Moreover, none of these works address text-to-image generation tasks. Specifically, [1] focuses on regression and classification problems, [2] evaluates on reinforcement learning (RL) tasks, and [3] concentrates on the removal of weather artifacts from images.To address these distinctions, we have included the following addition in the related works section:

Further, while previous methods have integrated uncertainty into model pipelines using MoE (Zheng et al., 2019; Luttner, 2023; Zhang et al., 2024), these approaches neither address epistemic uncertainty nor consider text-to-image generation tasks and are not applicable in a zero-shot manner.

Comments regarding CLIP score

Although CLIP score is known to exhibit bias [4], it remains a primary method for evaluating the quality of text-generated images, alongside the FID [5,6]. Both metrics rely on auxiliary networks (CLIP and the Inception network) for evaluation and are therefore both susceptible to bias. FID, however, requires a large number of samples for reliable estimates, whereas the CLIP score allows for more direct text-to-image alignment evaluation with fewer samples [6,7]. Given these factors, we prioritized the CLIP score for its suitability to our research objectives and its broad acceptance within related studies [4,6,7,8]. To further address potential concerns, we conducted an additional experiment evaluating EMoE’s performance using SSIM, detailed in Appendix B. The following text was added:

The CLIP score, despite its known biases (Chinchure et al., 2023), remains a widely-used method for evaluating the alignment between text prompts and generated images, alongside FID (Shi et al., 2020; Kumari et al., 2023). Both metrics, however, rely on auxiliary models (CLIP and Inception, respectively), making them susceptible to inherent biases. While FID requires a large number of samples for reliable estimation, the CLIP score facilitates a more direct assessment of text-to-image alignment with fewer samples (Kawar et al., 2023; Ho et al., 2022a). Considering these trade-offs, we prioritized the CLIP score due to its relevance to our research objectives and its broad acceptance in related studies.

To further validate our findings and address any potential concerns related to metric biases, we conducted an additional experiment using the Structural Similarity Index (SSIM) as the evaluation metric. Unlike CLIP or FID, SSIM does not depend on any auxiliary models for its calculation, thereby mitigating the risk of bias. We computed SSIM between generated images and corresponding ground-truth images from the COCO dataset and analyzed the results for each uncertainty quartile. As shown in Table 5, EMoE effectively categorized prompts into the appropriate uncertainty quartiles based on model performance. This provides further evidence of EMoE’s efficacy in estimating uncertainty for MoE text-to-image models, highlighting its robustness across different evaluation metrics.