Towards Understanding Text Hallucination of Diffusion Models via Local Generation Bias

Thank your time and thoughtful suggestions! Here are the responses to your problems.

1. Limited Experiments.

The experimental setup appears less practical and somewhat limited in scope. Most experiments were conducted on small, toy datasets and within narrow domains, such as digit generation. Could larger datasets and real-world applications, like image generation on ImageNet, be explored? Additionally, would textual hallucination issues appear in other real-world contexts?

The reason we use synthetical dataset is to gain a well-controlled data setting for clean and comprehensive study, therefore the lack of practical meaning is inevitable. Nevertheless, we do verify our findings and insights in more practical scenarios in appendix. For instance, we rendered 1000 common English words and Chinese characters and conduct similar study, discovering that a high LDR is also spotted. And the local generation bias persists.

We also conduct LDR analysis for current models such as FLUX1 and SD3.5, please refer to general response and appendix.

2.Does VLM have similar issues?

Labeling this issue as "textual hallucination" might cause confusion with hallucination problems in large language models (LLMs). It’s advisable to adjust the task description or add a notice at the start of the main paper for clarity. Following up on the previous question, would this issue also occur in Vision Language Models (MLLMs)?

It is interesting and meaningful to investigate whether VLM also exhibits such issues. However, as VLM is not generative model, we can not directly sample from its internal model to visualize its understanding on texts. But there does exist concurrent work[1] finding that VLM is not so good at recognizing text content, i.e. OCR task.

[1] Vision language models are blind. Pooyan Rahmanzadehgervi, Logan Bolton, Mohammad Reza Taesiri, Anh Totti Nguyen

3. Limitation of Theoretical Study.

Thank you for your suggestion. Please refer to general response 4.

Thank you for your thoughtful and constructive feedback on our submission. We have carefully addressed your concerns and provided detailed responses in our rebuttal. We kindly remind you to review our rebuttal at your earliest convenience. If there are any remaining issues or further clarifications needed, we would be more than happy to address them.

Thank you very much for your positive rate and valueable feedback. In our rebuttal, we have further addressed your concerns with detailed explanation and results. If you have any further questions, we are more than glad to answer them.

Also, we would sincerely appreciate it if you could increase your score. A clearer acceptence would make AC's job much easier for current boarderline score and enable this work get opportunity to be exposed at conference. Thank you once again!

Thank your time and thoughtful suggestions! Here are the responses to your problems.

1. Difference between Dyck and Parity.

In the dyck parenthesis experiment in Section A.4.2, it seems like the diffusion model correctly figures out the grammar beneath valid dyck sequences. Does this suggest the denoising network can overcome the local generation bias and avoid hallucination, contrary to the paper's conclusion? If not, what is the difference between this experiment and other results?

That is a good question. First, Dyck does not satisfy one of our central assumption-the pairwise independence of tokens. Any two tokens in parity distribution have independent marginal distribution. In real-world text, different letters of a text (especially long range) are also nearly independent. But Dyck grammar has strong correlation between context and each token. For example, if there have already been a lot of ( in the prefix, then the chance of having a ) would be much higher than (. Moreover, even the model can successfully generate Dyck sequences, it still can not uniformly cover all the unseen data to correctly extrapolate. Therefore, the success of Dyck is subtle, model learns some relation and is able to combine valid sub-sequences, but it still does not fully understand it.

2. Explanation on Figure 5 and Experiment on Real Models.

It would be better if there's an LDR comparison on the sampling of pre-trained models, between generating hallucinated samples and generating correct samples.

That is a very good point. We renew figure 5 and put explanation in general response 3. Please refer to it.

3. Why Distinguish In-Dataset and Extrapolation?

in terms of resolving hallucinations, is there a meaningful difference between the generation of extrapolated data and memorized data? Memorized output is also valid, and memorization by overfitting may not be due to the intrinsic symbol correlation itself...Also, for some datasets, especially real-world texts, the correct extrapolation from the training dataset is almost impossible, and being contained in the database becomes the only way to be valid.

We acknowledge that unconditional extrapolation on complex real-world distributions, such as text, may not be meaningful for all the generation tasks. However, this is beyond the scope of the current paper. The main point of the paper is that the models we concern do not have the flexibility to enable extrapolated generation. This implies that the models cannot really understand the underlying grammar, and we need to use them with caution in practice.

Moreover, The core objective of developing generative models is to create content that extends beyond the specific instances present in the training dataset, otherwise we just need to retrieve nearest neighbor from the dataset. Although focusing on text issues, the discovered local generation bias may have broarder and deeper meaning in other artifacts. All these requires a clear distinguishment between in-dataset sample and extrapolated ones.

Thank you for your thoughtful and constructive feedback on our submission. We have carefully addressed your concerns and provided detailed responses in our rebuttal. We kindly remind you to review our rebuttal at your earliest convenience. If there are any remaining issues or further clarifications needed, we would be more than happy to address them.

Thank your time and thoughtful suggestions! Here are the responses to your problems.

1. Generalize Results to Current Models.

..., it would be key for the paper's evaluation to investigate them using the proposed LDR metric. Specifically, insights into whether these models leverage more global structure or if improvements are tied to the strong priors of text conditioning.

We agree that testing our LDR metric on latest model is important to validate the intuition and proposed mechanism. We first sample images with rich text content generated by SD3.5 and FLUX1, and text issue is still happening. For more details, please refer to general response 2. We also conduct an approximated LDR analysis for recent models, for more details please refer to general response 3.

The reason we use synthetical dataset is to gain a well-controlled data setting for clean and comprehensive study, therefore the lack of practical meaning is inevitable. Nevertheless, we do verify our findings and insights in more practical scenarios in appendix. For instance, we rendered 1000 common English words and Chinese characters and conduct similar study, discovering that a high LDR is also spotted. And the local generation bias persists.

2. Parameter and Overfitting.

When comparing UNet and DiT architectures, do the authors ensure equal parameter count?

Yes, please refer to general response 3. Actually, even successful memorization at large-scale model does not harm our argument. We do not mean that model can never learn global relation even by overfitting. But under relatively few data and insufficient training, hallucination persists to happen and we aim to understand its mechanism, namely local generation bias. Note that overfitting usually require 5x training iterations than just achieving low FID for plausible visual clues, where most real-world diffusion training stops here.

3. About Timestep and Locality.

First, could the authors clarify if in their notation the generative/reverse diffusion process goes from T→0. I.e. higher values of t correspond to earlier/more noise steps in the denoising process. If that is the case, the results are somewhat counter-indicative to prior observations of diffusion models . It seems to be the current consensus that earlier steps generate larger/global image aspects with later steps refining localized features.

We renew figure 5 and explain it in general response 3. The $t$ refers to forward procedure timestep, and denoising is considerd as $T\to 0$, higher $t$ means close to pure noise.

On question about localized generation, you raised a very good question. Actually, the locality for DiT against timestep is a U-shape dependence. LDR is large for small and large timesteps $t$, but small for intermediate $t$. This is because for large $t$, the denoising function is close to identity $s_{\theta}(x)=x$, which is actually a localized function. Along the decrease of $t$, SNR increases and content gradually emerges, LDR decreases and model start to draw in sketch, caring more about global structure. Finally when $t$ is small, model only refine local details, thus LDR increases again.

4. Progression Difference.

do the authors have an intuition/explanation why the UNet seems to have a clear progression over timesteps from between more global and local attention, whereas the DiT does not exhibit such a seperation?

I guess you mean DiT has more clear regression in the local-to-global attention transition. This is because DiT is a pure-transformer structure operating on latent tokens, therefore its ability to overfit and memorize dataset (especially token's composition grammar) is stronger than attention-agumented UNet. But still, even equipped with this capacity, DiT struggled with local generation for a long time before it eventually memorize the dataset.

5. Typos and Typesetting.

Thank you for pointing out these problems! We renew figure 5 and correct typos in our draft. We also add more intuitive explanation for theory and will recommend more related work that potentially gives solution in our future versions.

Thank you for your thoughtful and constructive feedback on our submission. We have carefully addressed your concerns and provided detailed responses in our rebuttal. We kindly remind you to review our rebuttal at your earliest convenience. If there are any remaining issues or further clarifications needed, we would be more than happy to address them.

Thank your time and thoughtful suggestions! Here are the responses to your problems.

1. About Architecture Influence.

However, the comparative experiments in Figure 4 only demonstrate that different model architectures exhibit hallucinations, it does not completely decouple architectural influences.

Yes, architecture does make a difference. It is impossible to rule out any potential model architecture which can achieve text generation, and the best we can do is to test this on all existing models to verify. For more information, please refer to general response 1 and 3.

2. Insufficient experiments.

The article conducts comparative experiments under two different experimental settings on only two datasets. Including more validation experiments would enhance the reliability of the paper.

In our appendix, we conduct experiments beyond Quater-MNIST and Parity Parenthesis, including Dyck, English words and Chinese characters. All of them yields similar conclusions. We also add new experiments on up-to-date models like SD3.5 and FLUX1 in appendix, and the local generation mechanism still holds. Please refer to general reponse 2.

Thank you for your thoughtful and constructive feedback on our submission. We have carefully addressed your concerns and provided detailed responses in our rebuttal. We kindly remind you to review our rebuttal at your earliest convenience. If there are any remaining issues or further clarifications needed, we would be more than happy to address them.

Thank your time and thoughtful suggestions! Here are the responses to your problems.

1. Insufficient Parameter to Learn Global Relation.

Regarding the experiments in section 4.2, it is possible that the network capacity might be insufficient to learn global relations. There should be more details on the model architecture.

The total number of parameters of attention augmented UNet of init_dim=64 (35.7 M) and DiT-S (33 M) are comparable. It should have been written in appendix and we apologize for our negligence. Model architecture does play a role, since DiT can eventually overfit but UNet can not. But both models undergo a hallucination phase and shares similar mechanism in terms of local generation, which is what we aim to understand. (Please refer to general response 3.)

Also, according to our experiment, recent work (SD3.5, FLUX1) does not solve unconditional text generation task. Hallucination still exists, even for some conditional generation. Please refer to general response 2.

2. The Model Capture the Global Relation Eventually.

The model does not need to learn the true latent relation (s1+s2=s3+s4) if it can simply memorize all training samples. This mitigates the validity of the claim that diffusion models cannot learn the global relation.

Yes, we also confirm in our experiments that after elapsed time of training, the model can overfit and memorize the training data, which is impossible without capturing global relation. But we are not claiming that model can never generate valid data even by memorizing, but to understand why model hallucinates at under-trained and insufficient-data (with respect to text generation) regime by generalizing incorrectly, which is real-world models like SD3.5 lie in.

3. More Explanation on Parity Dataset.

The description "both have even numbers" about the Parity Parenthesis dataset seems incorrect. The claim "random guess has 50% chance of satisfying parity requirement" seems questionable

In our experiments, there are total 8 or 16 number of tokens $x_i$. Therefore, parity condition $\prod_{i=1}^L x_i=1$ is equivalent to set $X^-=\\{i\in[L]:x_i=-1\\}$ has even number of elements. If that is the case, then $X^+ = \\{ i\in[L]:x_i=1 \\}$ will have $L-|X^-|$ number of elements, which is also even. Therefore, if all $x_i$ are randomly drawn from $\\{\pm 1\\}$, then with half chance $|X^-|$ is even, so independent random generation has half chance of being valid for parity.

4. Limitation of Theoretical Analysis.

Our theoretical study is a conceptual framework to provide insight rather than rigorous disproof. Please refer to general response 4.

Thank you for your thoughtful and constructive feedback on our submission. We have carefully addressed your concerns and provided detailed responses in our rebuttal. We kindly remind you to review our rebuttal at your earliest convenience. If there are any remaining issues or further clarifications needed, we would be more than happy to address them.

Thank you for your further response. Actually we did have a response on supporting our claim in general response 1 to 4. To summarize, we further foster our claim from following perspectives

• Empirical Evaluation In the rebuttal (general response 2,3), we add new test of empirical large model like SD3.5 and FLUX1 to the original paper, finding that their unconditional text generation still suffers from hallucination issue. (appendix, page 18-20) We also conduct an approximated version of our LDR analysis for these modern models, demonstrating that it also exhibits local generation bias. All these results show that various models suffer from such bias.
• Theoretical Study. We further explain the insight behind our theoretical analysis in general response 4 and revised draft. We find that when marginal distributions of tokens are nearly independent, the trained dynamics of score matching is provably biased towards local information in MLP. The model's weight receives a very sparse gradient, only getting significant information within local region. This means the gradient descent will provably lead to a denoising function which primarily focusing on local region despite global receptive field.
• New Paradigm. We agree that a new training paradigm might mitigate hallucination. But that is beyond our scope. We are not claiming that any paradigm could not solve the hallucination issue (see general response 1). We are focused on why current paradigm without explicit prior knowledge and direct symbolism modeling on text content suffers from this hallucination, and how it is related to the underlying distribution's property and gradient descent.

Given these theoretical results and empirical validation, could you please inform us what are your further concerns? Although the revision period will soon be closed so we cannot conduct new experiments, we are still more than happy to hear constructive advices from you.

Indeed, hallucination is not an already solved issue, but has been greatly alleviated by new architectures. Overall, I still hold the opinion that the empirical evaluation is too weak to support the central claim. Therefore, I would like to keep my rating.