Can Diffusion Models Learn Hidden Inter-Feature Rules Behind Images?

Thanks for your time and efforts reviewing our paper. We now address raised questions as follows.

Q1: Further ablation studies across different architectures and training configurations.

Section D.3 (Lines 964-1032) considers different architectures (U-Net, SiT, DiT) and training configurations including training epochs, training data size and image size. Experimental results show DMs still have limitations in learning fine-grained rules with different settings.

Q2: The proposed mitigation strategies do not fully resolve the issue.

Thanks for your comments. Our strategy is an initial effort to solve this issue. Importantly, we identify a potential bottleneck: the signal of fine rules is too weak to be captured by the classifier, a problem have not been highlighted in classical and Imagenet task-based conditional DDPMs (see Section 5.2). We hope bottleneck analysis can provide valuable insights for future exploration.

Additionally, the goal of our work is to reveal the limitations of rule learning in DMs through experiments and theory. Fully addressing this challenge requires further work like rule-specific datasets and metrics, which are currently lacking. We leave the complete resolution to future work.

Q3:Theoretical proofs are based on simplified models.

Thanks for your question. Theorem 4.4 highlights for non-linear two-layer neural networks, there is constant errors due to the variance of diffusion noise. Theorem 4.5 explicitly derives the constant error. We believe the conclusions from Theorem 4.4 and 4.5 extend beyond the simplified setups. Intuitively, the score function is required to satisfy a low-dimensional constraint that holds for every noised input. However, without explicitly embedding this constraint into the model or the training objective, learning it from data becomes inherently difficult. Since the constraint must hold globally, neural networks lack the inductive bias needed to recover such structure from finite samples.

Q4:Real-world images.

Thank you for your good points. We conduct additional experiments on real-world datasets to further demonstrate DMs can learn coarse rules but struggle with fine ones.

• SynMirror [1] presents objects and their reflections, where rules link their features like color, size, and shape. We find DDPM captures coarse rules (e.g., matching colors between objects and reflections) but struggles with fine ones, showing shape mismatches.
• Cifar-MNIST pairs specific CIFAR and MNIST classes (e.g., Cats/Dogs with 0/1). We find DDPM satisfies coarse rules (e.g., always generating two digits and two objects), but only 20% of generations follow fine-grained rules requiring specific class pairings.

See Real-world Data for more details. We will add these into revised manuscript.

Q5:The sensitivity of feature extraction process to hyperparameters (e.g., predefined HSV ranges).

Sorry for the confusion. There is no hyperparameters in the feature extraction process. The HSV values used are predefined during training data construction. For example, in Task A, the sun’s HSV is set to yellow with hue [0, 30], saturation [100, 255], and value [200, 255]. The same HSV range is used during feature extraction (see Lines 747–806 for details).

Q6: Potential future directions.

Inspired by [2,3], one potential direction is optimizing the sampling process during inference. We can introduce additional reward signals from human feedback or reward models to guide DMs during sampling. Additionally, improving the tokenizer to better learn semantic information related to rules could also enhance rule learning. We will add this discussion to revised manuscript.

Q7: Can you provide more insights into why these approaches yield limited gains?

Thank you for mentioning this question. Section 5.2 (Lines 408-424) shows limited improvements is due to weak signals of the fine-grained rule which make the guidance from the classifier isn't strong enough to completely correct the sampling. Specially,

• Figure 16 shows inseparable CLIP representations of contrastive data, making classifier training challenging.
• Figure 17 demonstrates that training on simple contrastive data results in test accuracy below 90%, highlighting the difficulty in distinguishing subtle differences.
• Visualization demonstrates the weak differences between different classes in raw contrastive data.

[1] Reflecting Reality: Enabling Diffusion Models to Produce Faithful Mirror Reflections.

[2] Aligning Text-to-Image Models Using Human Feedback.

[3] Human preference score: Better aligning text-to-image models with human preference.

We hope above response can address your concern and we are open to discuss more if any question still hold.

We thank the reviewer's efforts on reviewing this paper. We now address the questions raised as follows.

Q1: Broader datasets / Real-world data.

Thanks for your good question. To supprot that DMs can learn coarse rules but hard to learn fine-grained rules, we conduct additional experiments on two real-world datasets, SynMirror and Cifar-MNIST.

• SynMirror [1] displays objects and their reflections, where inter-feature rules manifest as constraints between objects and their reflections in terms of color, size, and shape. The results show that the generation by DDPM can capture some coarse rules, such as objects and their reflections share the same colors, but it hard to learn fine-grained rules, such as there are siginificant differences in the shapes and contours.

• Cifar-MNIST combines specific classes from CIFAR and MNIST, such as pairing Cats and Dogs from CIFAR with 0 and 1 from MNIST. The results show that generations by DDPM satisfy coarse rules, such as ensuring that each generated image contains two digits (MNIST) and two non-digit objects (CIFAR). But only 20% of the generations satisfy predefiend fine-grained rules, where only specific categories from CIFAR and MNIST are allowed to pair.

Real-world Data Results provide more details and visualizations. We will add these results into the revised manuscript.

Q2: Mitigation strategy is not novel ... / any alternative strategies that might address these limitations more effectively?

Thanks for your good points.

• Novel Method. The main goal of our paper is to clearly identify the shortcomings of DMs in rule learning from both experimental and theoretical perspectives. At the end of the paper, we make initial attempts to address this issue. Importantly, we highlight that a potential bottleneck is the signals of fine-grained rules are too weak to be captured (see Section 5.2). This issue has not been reported in traditional DDPMs, such as those targeting ImageNet tasks with classifier guidance. We hope our initial strategies and bottleneck analysis provide valuable insights for further exploration.

• Alternative Strategies. Inspired by existing work [2,3], for further exploration, we can introduce additional reward signals from human feedback or powerful reward models to better guide DMs during sampling. Additionally, improving the tokenizer to better learn semantic information related to rules could also enhance rule learning. We will include the discussion in the revised manuscript.

Q3: Some of the definitions, like the rule-conforming error, lack intuitive explanations / It would be beneficial to include more detailed explanations ... such as the rule-conforming error.

Thank you for the question. The accuracy of score learning is inherently tied to the generation quality of diffusion models [4]. As we have shown in Theorem 4.2, in order to sample from the data distribution (with rule conformity), the score function should satisfy a constraint $\langle \nabla \log p_t(x_t) + x_t/\beta_t^2, [u; v] \rangle = \alpha_t/\beta_t^2$. By designing the network function in eq. (1) (in the main text), the constraint is equivalent to $\psi_t(x_t) = \alpha_t/\beta_t^2$ that holds for any $x_t$ (as in Definition 4.3). The rule-conforming error, defined as the mean squared deviation from this value, exactly measures how closely the learned score aligns with the ideal constraint (as $x_t$ varies). In practice, this relates to generating images that respect structural properties such as fixed object size. The smaller the rule-conforming error, the smaller the estimation error of the score function and thus the more likely the samples generated by diffusion models adhere to the constraint.

We will revise the text to clarify this connection.

[1] Reflecting Reality: Enabling Diffusion Models to Produce Faithful Mirror Reflections.

[2] Aligning Text-to-Image Models Using Human Feedback.

[3] Human preference score: Better aligning text-to-image models with human preference.

[4] Sampling is as easy as learning the score: theory for diffusion models with minimal data assumptions.

We hope the above response can resolve your concern and if there is further concern please let us know.

Thanks for your time reviewing this paper. We now address your questions as follows.

Q1: Theory

Thanks for your good points. First, Gaussian setup in [W2025] is a special case of our theory. Particularly, as Theorem 4.2, the score can be written as $\nabla \log p_t(x_t^{(1)}, x_t^{(2)}) = - \frac{1}{\beta_t^2} x_t + \frac{\alpha_t}{\beta_t^2} \begin{bmatrix} \gamma(x_t) u , \\ (1- \gamma(x_t)) v \end{bmatrix}$ where $\gamma(x_t) = E_{\zeta} [\pi_t (\zeta, x_t) \zeta]$. When $\zeta$ follows Gaussian distribution, the score can be simplified into a linear function over $x_t$, which can be learned by a linear model. However, we aim to cover more general setup where the above score is non-linear in $x_t$, i.e., $\zeta$ can be any bounded distribution. In such setting, the score function can be generally formulated as a combination of a fixed linear term $-1/\beta_t^2 x_t$ and an additional non-linear term, which motivates us to apply the two-layer network with the residual connection as the score network.

Moreover, our current patch-separated configuration follows prior work (Han et al. 2024a), while similar results can also be extended to network handling dependent patches. Particularly, we can consider $s^{(1,2)}_w(x_t) = - \frac{1}{\beta_t^2} x_t^{(1,2)} + W \sigma(W^\top x_t^{(1,2)})$ for some polynomial activation function $\sigma(\cdot)$ and $W \in \mathbb R^{2d \times m}$. Then, it can be seen that only when $\langle W \sigma(W^\top x_t^{(1,2)}), [u; v] \rangle = \alpha_t/\beta_t^2$ for any $x_t$ which concludes the network learns the rule (*). However, with the new network, we can still follow Theorem 4.4 to show: (1) parameters $W$ will only be the function of $u$, $v$ and the initialization and (2) the network function will be basically a polynomial function of $\langle u, \epsilon_t\rangle$ and $\langle v,\epsilon_t\rangle$ and their cross terms (not appeared in patch-separated network), where $\epsilon_t$ denotes the added diffusion noise in $x_t$. As $x_t$ varies, the function output also varies, which results in a non-vanishing rule conforming error that depends on the variation of $\epsilon_t$. Then similar results in Theorem 4.4 can be obtained and our main theoretical arguments still hold.

Q2: Interpretation of failure / threshold of Figure 5A / Raven’s matrices

Interpretation of failure:

• Compared to the training data with $R^2$ close to 1 (Figure 3), synthetic tasks yield significant lower $R^2$ with 0.6–0.8, indicating weaker rule learning.
• $R^2$ measures linear fitting quality, not rule accuracy. Differences in coefficients are also important, e.g., Task A’s estimation line $\beta_1 = 0.82$ is smaller than ground truth $\beta_1 = 1$. And Error metric in Table 2 which combines coefficient deviation and MSE further quantifies rule learning limitation of DMs.

Figure 5A: we adopt a strict threshold to show DMs can generate high-quality samples even under such strict conditions. Since DDPM perform well under strict settings, they naturally perform well under more relaxed thresholds.

Raven Matrices: fine-grained rules can also be measured in discrete state space. Specifically, we divide features within [0,1] into 20 intervals. A generation is rule-conforming if measured features fall within the interval satisfies predefined rules. Discrete Results show 95% of training data satisfy rules, while only 50–80% of generations do, highlighting DMs' limitation in learning rules.

Q3: line 299 is not correct.

We will modify line 299 to sum of projection.

Q4: Coarse/fine grained rules.

Fine-grained rules impose stricter requirements than coarse ones. As shown in Section 4, coarse rules only require the network to discover key features $\mathbf{u}$ and $\mathbf{v}$, while fine rules additionally require satisfying constraints between them—e.g., their projections summing to a constant (Definition 4.3). Thus, learning coarse rules doesn't guarantee fine-grained rule learning.

Q5: Spatial/Non-spatial rule.

• Spatial rules involve spatial arrangements like positions and layouts (e.g., Light-shadow in Figure 1), while non-spatial rules relate to features independent of space, such as size and texture in Figure 1. Section 3.3 explains non-spatial rules are harder to learn, possibly due to the lack of explicit cues like positions and lengths in spatial rule, which may explain why Task C is more hard than others.
• Our theory focuses on spatial rules with clearly defined patch-level dependencies but also can extend to non-spatial rules. For example, by introducing proper tokenizer, inseparable non-spatial rules in pixel space can be separable in latent space, allowing our theory to be applied.

We hope this addresses your questions. Please let us know if you have any further concerns.

We thank the reviewer's efforts on reviewing this paper. We now address the questions raised as follows.

Q1:The proposed approach to facilitating fine-grained rule learning appears to have no direct connection with the theoretical analysis. /The proposed method is unrelated to the main analysis of this paper

Thank you for the question. Our theoretical analysis in main text highlights that the failure of classical DDPMs in learning inter-feature rules is mainly due to the objective of denoising, which does not explicitly capture the hidden inter-feature rules. Therefore, classical DDPMs, when trained solely with the standard objective, lack the inductive bias necessary to learn fine-grained inter-feature rules. This naturally inspires us to introduce additional guidance to steer the sampling process, encouraging DDPM to generate rule-conforming samples.

Additionally, Figure 5 in experimental analysis in main text shows that DMs can generate high-quality samples that meet fine-grained rules, but the process is unstable and prone to rule violations. Therefore, our proposed method introduce additional information to help DDPM stably sample from high-quality regions (more discussion in Lines 300–322).

We will add more discussions in our revised version.

Q2: The proposed method to facilitate fine-grained rule learning is a bit straightforward/ The proposed approach to facilitating fine-grained rule learning d achieves only limited improvements.

Thank you for your question. The main focus of our work is to identify the limitations of DMs in learning fine-grained rules through experiments and theoretical analysis, rather than to propose a complete solution. This limitation has been overlooked (as discussed in Section 2) and represents 'a relatively underexplored area' (as noted by Reviewer 9KzS). Our work aims to 'extend previous findings on compositionality and factual consistency in diffusion models' (as noted by Reviewer gKDf).

Additionally, the proposed method is an initial attempt to enhance rule learning. Importantly, we identify a key bottleneck: the signal of fine-grained rules is too weak for the classifier to capture, a phenomenon not been highlighted in traditional DDPMs, such as those targeting ImageNet tasks with classifier guidance (see Section 5.2). We hope that these early attempts and bottleneck analysis can provide valuable insights for future exploration.

[1] Inference-Time Alignment in Diffusion Models with Reward-Guided Generation: Tutorial and Review.

We hope the above response resolved the questions and if there is further concern please let us know.