Seeds of Structure: Patch PCA Reveals Universal Compositional Cues in Diffusion Models

We thank the reviewer for the feedback and constructive questions, and for recognizing our analysis that explains how the initial noise correlates with the generated final image.

1. On the scale of the editing experiments.

The only weakness raised is the scale of the editing experiments.

The authors aimed to exploit their findings and propose an editing scheme using the extracted PCA components. The PCA-based noise editing requires considerably more experimentation to support and, in the end, hurts the overall presentation of the paper. The authors claim that the proposed zero-shot Patch PCA editing enables compositional control in the generated images. However, the experiments to support this are limited, with the only example being the interpolation between generated images and a 'Target' reference image. The reference image has a very distinct structure, which, although it seems to affect the CIFAR-10 and ImageNet generations, in no way guarantees that the proposed algorithm works in a more general setting. In the supplementary, the authors demonstrate editing with another reference image, but the results in Fig. 18 are nowhere near as successful. To claim compositional control, a larger-scale experiment is necessary, such as the one performed by Li et al. [11], which does numerical composition, i.e., generating K distinct objects in the image.

We think there might be a misunderstanding of our paper's core contribution compared to works like Li et al. [11], which we hope to clarify.

Our paper's primary contribution is a fundamental analysis of the image diffusion model: we identify that low-frequency components of initial noise are key to compositional control. As the reviewer noted, this provides an "elegant explanation" of the noise-to-image correlation.

Li et al. [11] leverage the intuition of reliable noise and focus on the downstream application of mining reliable seeds to improve text-to-image generation. While large-scale experiments are crucial for their goal, our editing method is intended as proof-of-concept—a clear demonstration of our core finding about Patch PCA components of initial noise. Its success validates our main thesis about frequency bands.

We clarify that we define image composition in Lines 32-33 as illumination and major structural lines that contribute to perceived composition, which can be measured by SSIM. The numerical composition is a completely different concept. Whether some principal components from Patch PCA reflect numerical composition is an interesting topic for future study.

In our response to other reviewers, we have incorporated feedback to:

1. Add SSIM scores and color histogram correlation to quantitatively evaluate compositional alignment and color preservation (Item 1 in rebuttal to Reviewer t5kH)
2. Include new experiments on high-frequency noise editing influence (Item 2 in rebuttal to Reviewer t5kH)
3. Verify findings hold across different deterministic samplers like VE and VP (Item 3 in rebuttal to Reviewer HRUH)
4. Verify findings hold for flow-matching models (Item 4 in rebuttal to Reviewer HRUH)

Given that our paper's central scientific contribution is distinct from [11] and the strengthened evaluation, we believe our work makes a valuable contribution to understanding diffusion models.

Clarification on Compositional Control Success in Figure 18

In the supplementary, the authors demonstrate editing with another reference image, but the results in Fig. 18 are nowhere near as successful

We respectfully disagree that editing in Fig. 18 is unsuccessful. For Starry Night, the main perceived structure is diagonal movement from bottom left to middle right. Generated images with increasing weights for the reference show a transition from original generation toward variants with diagonal movement similar to the reference, showing compositional structure alignment. We conducted SSIM computation comparing baseline (no edit) and edited case with interpolation factors, showing a monotonic increasing trend:

This quantitatively demonstrates successful progressive alignment toward reference composition.

2. On Translating Patch PCA to DiTs

Does the proposed Patch PCA translate to diffusion transformers (DiTs)? The convolutional models utilize patch-specific features to perform denoising, but diffusion transformers are not biased towards using local patches. Including DiTs in the analysis could help drive the main argument of the paper further.

Thank you for this insightful question. Yes, our findings and methods apply to transformer-based diffusion architectures like DiTs. Transformer-based diffusion models, such as DiT, have an initial "Patchify" step where the image is broken into tokens, creating an inductive bias compatible with our patch-based analysis. Additionally, as shown in recent work [24], different architectures—including U-Nets, U-ViT, and Consistency Trajectory Models—produce remarkably nearly identical images when given the same initial noise (see Fig. 1 and Fig. 11 in [24]). Consequently, both frequency perturbation phenomenon and editing apply to diffusion architecture with transformers that we empirically verify below.

We couldn't find the DiT pre-trained model on ImageNet-64, FFHQ, or AFHQ, so we used U-ViT instead. We generated 1000 images with identical seeds for EDM and U-ViT, both of which are publicly available pre-trained models on ImageNet and Patch PCA, as well as edited versions using the same edited initial noises, following Figure 1 settings with an interpolation factor of 0.1. We show SSIM and MSE scores between the same seed (paired) and random (unpaired) images.

These results confirm our Patch PCA based observations and that our editing method applies across different architectures, with edited images showing strong structural alignment regardless of underlying model architecture.

3. On the Dataset for Patch PCA Computation

The specific dataset used to compute Patch PCA basis is not critical for small patch sizes we use. In Appendix Figure 9, we show that patch covariance matrices computed from ImageNet, FFHQ, and AFHQ are nearly identical for patches smaller than 16×16 (achieving >0.99 correlation). This is because small patches capture universal, low-level statistics of natural images that transcend specific content of any dataset. When using small patches, high-level structures like aligned faces in FFHQ are broken down into generic components (edge detectors, textures), making their statistics highly similar to those from general-purpose datasets like ImageNet. Therefore, we did not observe components specific to features like mouths or eyes.

4. On Limitations

The only limitation mentioned is that the proposed editing is focused on low-frequency components. However, the most significant limitations would be regarding the applicability of the proposed editing, which has not been discussed thoroughly. The authors should expand their experimental section to demonstrate and address potential limitations of the proposed editing algorithm.

As we clarified earlier, our applicability to Fig. 18 is confirmed by SSIM scores. This paper is intended as a proof-of-concept—a clear demonstration of our core finding about Patch PCA components of initial noise. A comprehensive evaluation remains a future goal.

---

[11] Li, Shuangqi, Hieu Le, Jingyi Xu, and Mathieu Salzmann. "Enhancing compositional text-to-image generation with reliable random seeds." ICLR 2024.

[24] Huijie Zhang, Jinfan Zhou, Yifu Lu, Minzhe Guo, Peng Wang, Liyue Shen, and Qing Qu. The emergence of reproducibility and consistency in diffusion models. ICML 2024.

We thank the reviewer for the encouraging assessment and for highlighting both the empirical insight of universal compositional cues and the practical value of our zero-shot editing method. Below, we respond to each weakness and question.

1. The Influence of High-Frequency Noise

The analysis focuses heavily on the role of low-frequency components for composition, while the influence of high-frequency noise components on texture and fine details is mentioned but not explored in depth. A more complete picture would involve a deeper analysis of how different frequency bands contribute to various aspects of the final image.

We agree that a comprehensive exploration of all frequency bands would be valuable. However, in this paper, our focus is on establishing the dominant structural role of low frequencies. To demonstrate that the editing method naturally extends to the high-frequency region, we conducted preliminary experiments on high-frequency perturbation. Using "Starry Night" as the reference image (Figure 17, appendix) with 5×5 Patch PCA basis, we find:

1. Frequency band [19:75], interpolation factor 0.3: Creates blue and yellow oil-painting texture like in Starry Night while maintaining the original composition.
2. Frequency band [20:75], interpolation factor 0.9: Generated image remains almost identical to the unedited version except for additional curly textures mimicking Starry Night's swirling patterns.

These experiments demonstrate that our method can effectively manipulate texture. We will include these results in the updated paper.

2. Theoretical Justification for Neural Network Denoisers

The theoretical justification relies on a simplified Patch PCA denoiser, which, while effective, may not fully capture the complex, non-linear dynamics of large-scale neural network denoisers. This simplification could limit the direct applicability of the theoretical claims to state-of-the-art diffusion models.

We appreciate the opportunity to clarify the relationship between our theoretical analysis and practical neural network denoisers.

Our theoretical framework handles nonlinear settings. Our main result, Theorem A.2, provides closed-form expressions for the optimal denoiser within a broad family of "admissible local functions," which includes both linear and nonlinear functions. This family is motivated by the local score and equivariant score functions introduced in [2] and offers a natural generalization to settings where equivariance is broken—for example, near boundaries. Notably, our analysis reveals that the equivariance condition is redundant; the key structural constraint is what we define as the "patch shape" in Definition A.1.

Theorem 5.1 follows as a corollary of Theorem A.2, establishing that when the patch distribution is Gaussian, the optimal denoiser within the admissible local function family coincides with the Patch PCA denoiser. This suggests that even complex neural networks, if locally determined, should approximate this theoretical optimum for Gaussian-like patch distributions during training.

For neural network denoisers trained on non-Gaussian but approximately Gaussian patch distributions (e.g., close in Wasserstein distance), we hypothesize that the neural network denoisers can be viewed as a bounded perturbation of the optimal Patch PCA denoiser. Based on this assumption, if we denote the neural network denoiser as $f_{NN}$ and Patch PCA denoiser as $f_{PCA}$, assuming $||f_{NN}(x, t) - f_{PCA}(x, t)||_2 \leq \epsilon(t)$ and bounds on the Lipschitz constant $L_t$ of flow maps $\Psi_t$, we can adapt the Alekseev–Gröbner stability argument from [1] to bound trajectory discrepancies. This framework implies that final samples from neural denoisers remain close to those from our theoretical model when the assumptions are satisfied.

3. Editing on More Complex Scenes

While the editing results on CIFAR-10 and ImageNet are promising, the method's effectiveness on more complex scenes with multiple objects or intricate layouts is not fully demonstrated. The compositional control appears to be primarily global, and its precision in more complex scenarios remains an open question.

We agree that extending the method to multi-object scenes and finer spatial precision is compelling future work. Our primary goal here is to introduce a new Patch PCA filter-based perspective for editing via single manipulation of the initial seed.

For reference images more complex than the target image, Figure 18 in the Appendix demonstrates editing results for CIFAR-10 using Starry Night as the reference, where our method successfully captures the characteristic bottom-left to middle-right diagonal movement of the reference composition. We will leave a comprehensive evaluation of more complex multi-object scenes as an important direction for future studies.

---

[1] Benton, Joe, George Deligiannidis, and Arnaud Doucet. "Error Bounds for Flow Matching Methods." TMLR 2024
[2] Kamb, and Ganguli. "An analytic theory of creativity in convolutional diffusion models". ICML 2025

We thank the reviewer for the constructive feedback and are encouraged by your recognition of the non-trivial nature of our findings and the practical value of the proposed zero-shot editing method. We address the questions below.

1. Quantitative Evaluation of Zero-Shot Editing

Zero-shot patch pca based noise editing is missing quantitative comparisons.

We agree that adding quantitative evaluation would further strengthen the experiments. To complement the qualitative results in Figure 7, we added quantitative evaluations across 256 random seeds using CIFAR-10. We use SSIM for structure alignment and Color Histogram Correlation for global color similarity. Color Histogram Correlation compares normalized RGB histograms using Bhattacharyya distance (range: 0 to 1, where 1 indicates identical distributions).

Results show that editing low-frequency Patch PCA components significantly improves SSIM, confirming better compositional alignment. Excluding color channels (c) achieves better color preservation, as demonstrated by higher correlation with the no-edit case and lower correlation with the reference image. We will add these results to the main text.

2. Relation to SDEdit and Other Zero-Shot Editors

I'm very curious about how this compares to methods like SDEdit, and if some of the insights of this technique can apply to SDEdit or vice-versa. It would be nice to show comparison with other zero-shot image and style editing techniques as well.

SDEdit interpolates a reference image with noise at selected timesteps, then denoises from that intermediate state. Our method manipulates Patch PCA frequency bands in the initial noise. These approaches introduce editing at different aspects: SDEdit at selected timesteps, ours via frequency bands. Combining our approach with SDEdit for finer-grained control is an interesting direction. Unfortunately, this year's policy does not allow us to present visual results, and we will leave it as future work. We agree that a full benchmark across major zero-shot editors would be valuable. However, it falls outside this paper's scope, as our primary goal is to expose the structural role of low-frequency noise and introduce a novel, lightweight, training-free editing tool. We will leave comprehensive benchmarking as future work.

3. Sampler Effect

How does the choice of sampler affect the findings of the paper? If I'm not mistaken, the study is currently using DDIM. Does this change for DDPM (given the stochasticity of each denoising step) or other VE/VP samplers?

Yes, our study uses DDIM (EDM scheduling). Other deterministic samplers (VE, VP) produce identical results for the same model and initial noise when given sufficient steps. Our observations and methods apply to all deterministic samplers, empirically verified by near-identical images across samplers:

However, stochastic samplers like DDPM add noise at every step, so the same seed no longer yields unique output. Our observations about low-frequency components and our editing method don't apply to stochastic samplers. One workaround is to extend frequency-band editing to intermediate denoising steps, which fall into the combination of our method with SDEdit, as you mentioned in the review.

4. Flow-Matching Based Models

How would this analysis change for flow-matching based models? I don't see a discussion of this, but I'm curious to know if similar patterns emerge.

Flow-matching models extend diffusion models with deterministic sampling. We trained a flow-matching (FM) model with the OT scheduling function (or rectified flow) on AFHQ and compared it with pretrained EDM:

This suggests flow-matching models (FM AFHQ) learn similar noise-to-image mappings as EDM AFHQ. This aligns with existing studies on noise-to-image map reproducibility across different architectures. For example, the work [24] shows different architectures (U-Nets, U-ViT, Consistency Trajectory Models) produce remarkably similar images given identical initial noise in their Fig. 1 and Fig. 11. Consequently, our observations and methods are directly applicable to flow-matching models. We will add these discussions to the main text.

5. [L114] Resampling the Frequency Band [n, 48]

[L114] Can you please describe line 114 in more detail? i.e. "resampling the identified frequency bands [n,48] on each patch." An algorithm like algorithm 1 would be nice (can be included in the appendix).

We decompose initial noise into 48 orthonormal Patch-PCA eigenvectors (3 channels × 4×4). To "resample" band [n,48], we set coefficients in range [n,48] to new i.i.d. Gaussian samples. Since the basis is orthonormal, overall noise remains Gaussian. We are happy to present this as an algorithm in the revision.

6. [Fig 1b] Row Description

[Fig 1b] what do the three rows correspond to?

Each set of three rows shows three independent examples (top to bottom) of images generated after perturbing initial noise in the specified frequency band (labeled above the first row).

7. [L132] Models Used

[L132] Are different models are being used here? Are you using an ImageNet trained model A and a FFHQ trained model B and then generating images with the same seed?

Yes. We use pixel-diffusion models separately trained on ImageNet, FFHQ, and AFHQ. For each model, we generate images with identical seeds to test whether similar global structure emerges across models trained on different datasets.

8. [Tab 1, Fig 5] LPIPS Metric Results

[Tab 1, Fig 5] How do the differences measure in terms of the LPIPS metric?

LPIPS results mirror SSIM trends. Same-seed images always receive lower scores (higher similarity) than random pairs. Differences are most pronounced in pixel-space models and smaller but evident for Patch PCA in latent space.

Pixel-space models:

Latent-space models:

9. Patch-Size Sensitivity

Was a patch size of 4x4 used for all experiments? How do results from the noise editing algorithm differ when using different patch sizes?

Figure 1's frequency perturbation used 4×4 patches to divide 64×64 (or 32×32) images, preserving Gaussian properties. Image editing experiments used 5×5 patches for both pixel and latent space. We tested 3×3, 7×7, and 9×9 patches with similar qualitative results, though optimal eigenvector numbers and interpolation weights varied. Larger patches generally require more eigenvectors for comparable results.

---

[24] Huijie Zhang, Jinfan Zhou, Yifu Lu, Minzhe Guo, Peng Wang, Liyue Shen, and Qing Qu. The emergence of reproducibility and consistency in diffusion models. ICML 2024.

Thank you for responding to my questions. I maintain my recommendation for acceptance as I think this paper has insightful findings.

To "resample" band [n,48], we set coefficients in range [n,48] to new i.i.d. Gaussian samples.

I realize this is late, so no worries if there is not enough time, but I would like to see the algorithm for this portion.

LPIPS Metric Results

The difference in SSIM seems much higher than for LPIPS. Do you know why this might be the case?

We thank the reviewer for the thoughtful and constructive feedback. We are encouraged that you found our central observations compelling, particularly the role of low-frequency components of initial noise in shaping compositional structure and the universality of this phenomenon across architectures and datasets. Below, we address the questions raised:

---

1. Patch PCA in Latent Diffusion Models

While the Patch PCA denoiser is theoretically justified in pixel space under Gaussian patch assumptions, its extension to latent diffusion models is only evaluated empirically. Since the latent space is itself a learned low-dimensional representation, further theoretical or statistical analysis of PCA behavior in latent space would strengthen the argument for universality.

The core theoretical justification relies on the assumption that image patches follow a Gaussian distribution. Whether this assumption holds in latent space for latent diffusion models is unclear and not discussed.

We appreciate the reviewer's questions regarding Patch PCA in the latent space. Due to the nonlinearity of the VQ-VAE encoder, a Gaussian patch distribution may no longer remain statistically Gaussian in latent space. Extended theoretical analysis is an interesting direction, but it can be quite complicated. However, we observe that the encoder's dimensional reduction aligns remarkably well with PCA for image patches: the leading principal directions in pixel space strongly correspond to the principal directions in latent space—precisely the components our PCA-based editing method leverages.

To support this, we conducted a subspace similarity analysis:

• Step 1: Extracted 8p × 8p patches from natural images and computed the top k PCA directions in pixel space, where k ranges from 1 to 4p² (the dimensionality of the encoded latent patches).
• Step 2: Encoded these PCA directions with the LDM encoder, yielding k vectors in latent space.
• Step 3: Computed PCA directions directly on p × p latent patches from encoded images.
• Step 4: Compared the two k-dimensional subspaces via mean cosine similarities of principal angles. The score ranges from 0 (orthogonal) to 1 (identical).

Below are the results for 5×5 latent patches used in the paper:

First ten principal directions:

Coarser grid up to full latent-patch dimensionality (100):

These results confirm our hypothesis with consistently high alignment values (>0.93 on average), supporting the validity of our patch PCA-based observations and methods in latent space. We will include this analysis in the revision.

2. Quantitative Evaluation of Zero-Shot Editing

Although the paper provides promising qualitative results for the zero-shot editing method, it lacks quantitative metrics (e.g., compositional similarity scores, human preference, or alignment accuracy) to objectively assess the success of compositional transfer. Moreover, claims that both structure and color can be selectively preserved or transferred through PCA manipulation are not fully substantiated by quantitative evidence.

We agree that adding quantitative evaluation would further strengthen the experiments. To complement the qualitative results in Figure 7, we added quantitative evaluations across 256 random seeds using CIFAR-10. We use SSIM for structure alignment and Color Histogram Correlation for global color similarity. Color Histogram Correlation compares normalized RGB histograms using Bhattacharyya distance (range: 0 to 1, where 1 indicates identical distributions).

Results show that editing low-frequency Patch PCA components significantly improves SSIM, confirming better compositional alignment. Excluding color channels (c) achieves better color preservation, as demonstrated by higher correlation with the no-edit case and lower correlation with the reference image. We will include these quantitative results in the revision.