Varying Manifolds in Diffusion: From Time-varying Geometries to Visual Saliency

Thank you for your detailed review! Below we address the questions.

Q1. Larger Dataset for Visual Saliency Evaluation.

We have expanded our visual saliency evaluation to include the full MIT saliency benchmark CAT2000, consisting of 2000 diverse-category images. Our results indicate that 81% of randomly chosen salient points exhibit higher curve variance compared to non-salient points. Furthermore, we conducted rigorous statistical analyses as suggested by Reviewer xDCq on the above data:

• Point-Biserial Correlation: $(r = 0.297, p=1.36e^{-69})$

• Independent Samples t-test: $(t = 18.050, p=1.36e^{-69})$

The statistical tests further confirm a significant positive correlation between generation curve fluctuations and visual saliency, and statistically significant differences between salient and non-salient curves. Visual results are presented in the https://limewire.com/d/3h4XH#SSSDAgZ0rL Figure 2.

Q2. Quantitative Evaluation on Applications.

We have included quantitative evaluations for all four proposed applications in Tab.1, Appendix D (lines starting from 648). Following related works, we employ large Vision-Language models such as CLIP, alongside user studies, to quantitatively assess the performance. Results demonstrate our method outperforms or is competitive with current state-of-the-art approaches across evaluation metrics. Detailed discussions of quantitative comparisons can be found in the final paragraph of each application subsection within Appendix D.

Q3. Comparison with Recent Image Blending Baselines.

According to our investigation, recent relevant works include [Wu2019, Zhang2020, Zhang2021, Xing2022], summarized in a comprehensive survey [Niu2025](page 8-10). Unfortunately, [Zhang2021] and [Xing2022] do not provide publicly available implementations. Therefore, our comparisons primarily focus on [Wu2019] and [Zhang2020], which were contemporaneously published in 2019.

Upon evaluation, [Zhang2020](Deep Image Blending) frequently introduces undesirable background color leakage into composite objects, leading to noticeable color distortion in natural images. We illustrate these issues with specific examples in https://limewire.com/d/3h4XH#SSSDAgZ0rL Figure 4. Consequently, we select [Wu2019] as the primary baseline due to its consistently better performance in maintaining visual coherence in blended images.

[Wu2019] Huikai Wu, Shuai Zheng, Junge Zhang, and Kaiqi Huang. GP-GAN: Towards realistic high-resolution image blending. ACM MM, 2019.

[Zhang2020] L. Zhang, T. Wen, and J. Shi (2020). Deep Image Blending. WACV, 2020.

[Zhang2021] He Zhang, Jianming Zhang, Federico Perazzi, Zhe Lin, and Vishal M Patel. Deep image compositing. WACV, 2021.

[Xing2022] Yazhou Xing, Yu Li, Xintao Wang, Ye Zhu, and Qifeng Chen. Composite photograph harmonization with complete background cues. ACM MM, 2022.

[Niu2025]Li Niu, Wenyan Cong, Liu Liu, Yan Hong, Bo Zhang, Jing Liang, Liqing Zhang. Making Images Real Again: A Comprehensive Survey on Deep Image Composition. arXiv preprint arXiv:2106.14490(v6), 2025.

Q4. Clarification of Curves in Figure 5.

We clarify that the curves presented in Figure 5 are generated at the pixel level, intended to illustrate the approximation of curve shapes discussed in Section 4.1. Since the objective is solely to compare shape approximations rather than interpret specific semantic content, pixel-level representation sufficiently serves this purpose. We will explicitly mention this clarification in the figure caption to avoid potential confusion.

Q5. Image Inversion for Real Images.

Yes, image inversion is required and employed for all real-image analyses and subsequent editing tasks discussed. To enhance clarity, we will explicitly mention the diffusion-model inversion at the conclusion of Section 3.1 (around line 134).

Thank you for your detailed review! Below we address the questions.

Q1-1. Expanded Discussion on Baselines - Object Removal.

Methods for object removal generally fall into two categories: image inpainting and instruction-based editing (as detailed in the recent survey [Huang2025]). We selected effective representatives from both ([Podell2024, Li2024]).

Regarding the specific baselines:

• RePaint[Lugmayr2022]: our SDXL-inpainting baseline [Podell2024] is fundamentally built on Repaint's principles with a more advanced base model (SDXL) and specialized post-training on inpainting datasets. The theoretical justifications (Repaint+[Rout2023]) for Repaint-based inpainting is discussed around line 300.

• Inverse-Problem-Based Methods[Chuang2022, Chuang2023]: We quantitatively compared with the more updated [Chuang2023] (see Table 1 and Figure 3 in https://limewire.com/d/3h4XH#SSSDAgZ0rL). Our method outperforms in editing direction-likely due to guidance from our reference curves-while they demonstrate greater visual changes.

• SDEdit[Meng2022]: As an early diffusion-based editing method that incorporates guiding instructions, SDEdit is often regarded as a form of instruction-based editing. Subsequent methods, such as InstructPix2Pix, have reported significantly improvement upon SDEdit. Since our selected SOTA baseline, Zone[Li2024], demonstrated clear superiority over these methods, we consider explicit comparison to SDEdit unnecessary.

Q1-2. Expanded Discussion on Baselines - Image Blending.

[Chen2019] focuses on replacing an object with a semantically similar but visually distinct one (closer with image-to-image translation), whereas our method emphasizes seamless pixel-level blending of an inserted object with its surrounding. We discussed an extensive comparison in response to Reviewer Zb6M's Q3, identifying our baseline [Wu2019] as the most relevant and strongest open-source baseline.

In the original paper, additional qualitative results for four tasks are provided in Appendix Figure 14-21.

[Rout2023]Rout et al.. A theoretical justification for image inpainting using denoising diffusion probabilistic models. 2023.

[Huang2025] Huang et al.. Diffusion Model-Based Image Editing: A Survey. TPAMI 2025.

Q2. Essential References Not Discussed.

We will integrate the discussion in Q1 and the following to our Appendix D.

• LaMa[Suvorov2022] Although effective, the GAN-based LaMa frequently introduces grid-like artifacts when compared to diffusion-based inpainting as also noted in [Lugmayr2022].
• ObjectStitch[Song2023] They leverage guided diffusion models for object compositing, focusing on semantic consistency rather than boundary smoothing. Compared to our method, they provide more diverse outputs at the expense of precise pixel alignment.

Q3. Derivation of Equation Eq.8 from Eq.4.

Eq.8 can be explicitly derived from Eq.4 by treating $X_{t-\Delta t}$ as a function $g(X_t)$: $X_{t-\Delta t} =g(X_t) = \sqrt{\frac {\alpha_{t-\Delta t}}{\alpha_t}} X_t + (\sqrt {1-\alpha_{t-\Delta t}}+ \sqrt{\frac{\alpha_{t-\Delta t}(1-\alpha_t)}{\alpha_t}}) \epsilon_\theta^t(X_t)$. Taking the Jacobian of $D_g(X_t)$ with respect to $X_t$ yields: $D_{g}(X_t) = \sqrt{\frac {\alpha_{t-\Delta t}}{\alpha_t}} I + (\sqrt {1-\alpha_{t-\Delta t}}+ \sqrt{\frac{\alpha_{t-\Delta t}(1-\alpha_t)}{\alpha_t}}) D_{\epsilon_\theta^t}(X_t)$, where $I$ is the identity matrix. Thus, applying $D_g(X_t)$ to a vector $v$ produces Eq.8.

Q4. Clarification on Perceptual Metrics and Generation Rate.

The generation rate at data point level $X_t$ makes sense since it can also be interpreted as the rate at which noise is removed as $X_t$ moves closer to the original image $X_0$. In practice, this is effectively captured by the perceptual distance change computed through metrics such as LPIPS between the estimated $\hat X_0(X_t)$ and the real image $X_0$.

Q5. Obtaining Reference Generation Curves via Image Inversion.

Yes, we use diffusion ODE inversion to obtain generation curves for real images. We will clarify this clearly in Section 3.

Q6. Leveraging Generation Curves for Image Editing Applications.

The positive correlation of saliency with curve fluctuation is leveraged as follows:

• Saliency Manipulation and Blending: By enhancing or suppressing the curve fluctuation of pixels, we directly control their saliency; disharmonious object boundaries, identified as inherently salient regions, are smoothed by reducing their curve fluctuations.

• Semantic Transfer and Object Removal: The overall shape of curves encodes broader visual semantics, as discussed in lines 204-210. We transfer these semantics by matching the entire shape of generation curves; object removal is treated as a special case where semantics are shifted from the object to its background.

These methods can be applied to arbitrary image regions, which do not necessarily have to be salient.

Thank you for your detailed review! Below we address the questions.

Q1.Theoretical Justification of Contractive $D_{f_t}$.

The observation of contractility is primarily empirical, demonstrated through experiments where tangent vectors {$v_i$} are scaled under the forward differential $D_{f_t}[v]$. This empirical contractility serves to intuitively illustrate our idea rather than being a necessary condition for defining the generation rate. Even without it, the scaling factor can still meaningfully quantify generation rates.

Q2. Clarification of 'Dominates Overall Shape' and Visibility of Yellow Curve in Fig.5.

We clarify the words 'dominates the overall shape' as indicating that the shape of $\|D_{\epsilon_\theta^t}(X_t)[\text{Proj}(v)]\|$ closely resembles the shape of the original $\|D_{f_t}(X_t)[\text{Proj}(v)]\|$. Regarding the visibility of the yellow curve in Figure 5, it overlaps significantly with the red curve in the first plot due to their high similarity, particularly visible upon closer inspection in the timestep range $t\in [0,40]$.

Q3. Expanded Preliminaries and Manifold Notations.

Following your suggestion, we will revise Appendix A to explicitly introduce the notations of diffeomorphism induced by the diffusion ODE. We agree to replace $D_{f_t}$ consistently with the clearer notation $D_xf_t$ throughout the paper.

Q4. Additional Details of Power Method in Appendix.

The power method and the identification of $h_t$ as a contractive map are discussed extensively in [Park2023], specifically in their Appendix F. We will expand our Appendix to briefly summarize these foundational details.

Q5. Formula and Computational Efficiency of $\text{Proj}(v)$.

Given an orthonormal tangent basis {$u_i$}$^k_{i=1}$ and an arbitrary ambient-space vector $v \in R^n$, the projection operator can be explicitly expressed as: $\text{Proj}(v) = UU^Tv$, where $U=[u_1...u_k]$ is an $n \times k$ matrix of tangent vectors. The above matrix multiplication is efficient, with primary complexity arising from calculating the tangent basis. Using the power method, this procedure typically requires tens of seconds on a GPU. We note this complexity is inherently necessary due to the high dimensionality of the ambient space.

[Park2023]Park et al.. Understanding the latent space of diffusion models through the lens of riemannian geometry. NeurIPS, 2023.

Thank you for your detailed review! Below we address the questions.

Q1. Continuous-Time Formulation of Generation Rate

We agree that leveraging a continuous-time formulation provides a scheduler-independent definition. For the diffusion ODE $\frac d {dt}X_t=h(X_t,t)$, the variation of the norm of a unit tangent vector $v_t$ can be expressed as $\frac d {dt}\|v_t\|= \left \langle v_t, D_{h(X_t,t)}[v_t]\right \rangle$. We will clearly present and discuss this formulation in Section 3.

Q2-1. Necessity of Projection onto Tangent Space

In diffusion ODE, the data manifold $M_0$ is continuously transformed into $M_t=\phi_t(M_0)$ by induced diffeomorphism $\phi_t$, preserving a low-dimensional manifold structure at all times. Therefore, the projection onto $T_XM_t$ remains necessary. Although ideally one can track the $T_XM_t$ by pushing forward $T_XM_0$ via the diffusion ODE, under computational constraints we approximating the tangent space at each timestep. This is reliable at moderate to low noise levels despite some loss of precision at very high noise levels.

Q2-2. Alternative Projection Methods

Dataset PCA is impractical because it requires full dataset access at each noise level {$X_t^i$}$_{i=1}^N$, and PCA on single trajectories (as in [WV2023]) yields overly restricted subspaces. The adopted power-based approach, while dependent on the UNet architecture, is a practical solution. If there are other viable methods for high-dimensional nature image analysis, we will incorporate ablation studies with them.

Q3. Clarification on Optimization Algorithm

At each iteration, for variable $X_t$, we samples a random timestep $t_s$ and traverse the diffusion ODE from $X_t$ to $X_{t_s}$ without gradients. We then calculate gradients at $X_{t_s}$ and propagate them back to $X_t$ using adjoint method incorporated in PyTorch. We will clarify these details further in Section 4.2 and in the algorithm box.

Q4. Theoretical Explanation Connecting Generation Curve and Visual Saliency

• Critical Period and Peak Positions According to [WV2024], the estimated $\hat{X_0}$ undergoes a critical period of rapid change. Since the rate of change of $\hat{X_0}$ parallels our generation rate (as discussed in Section 3.2), this period directly corresponds to the dominant peak in our generation curves. The relation between critical period and feature variance[WV2024] also explains how peak positions encode detailed visual semantics essential for our semantic transfer task.

• Non-salient Curves with Peaks Near $t=0$: [WV2024] notes that high-frequency features tend to have lower variance, causing their critical periods to occur later in the diffusion process. For non-salient backgrounds (like a plain wall), low-frequency components are largely captured by the distribution mean approximated by $\hat{X_0}(X_T)$. Consequently, only the high-frequency details—such as subtle textures—require explicit generation, which consistently results in peaks near $t=0$. Moreover, [WV2024] Figure 3.D analytically demonstrates that when the critical period occurs near zero, the corresponding change rate surges sharply, a result that aligns with our empirical findings (lines 191–194) of non-salient curves.

• Curvature and saliency: Considering the continuous diffusion ODE expressed by score function $dX_t = -g(t)s(X_t,t)dt$, the generation rate can be related to the Hessian of the log probability density: $r(X_t,v) = |g(t) \left \langle D_{s(X_t,t)}[v], v \right \rangle | = |g(t) \left \langle D_{\nabla logP(X_t)}[v], v \right \rangle |= |g(t) \text{Hessian}_{log P}(v,v) |$. Intuitively, the Hessian measures curvature-the sensitivity to perturbations in a given direction. Visually salient areas, rich in semantic details, may exhibit larger curvature: For example, the change of boundary pixels can disrupt the outline of an object, resulting in the deviation from image manifold. This geometric perspective explains why salient regions have greater fluctuations.

Q5. Saliency and Curve Fluctuation Experiments

• Visual examples (Figure 2 left): We provide saliency maps (via EML-NET) in https://limewire.com/d/3h4XH#SSSDAgZ0rL Figure 1; Figure2 left highlights salient areas such as the mantis and the center of the lotus leaf, with additional curve fluctuations from leaf veins.

• Statistical correlation: We first converted the discrete salient points in dataset to continues maps via Gaussian blurring as suggested. Statistical tests conducted on dataset yields:

• Point-Biserial: $(r = 0.297, p=1.36e^{-69})$

• t-test: $(t = 18.050, p=1.36e^{-69})$

These confirm a strong positive correlation between fluctuation and saliency. This evaluation was conducted on an expanded dataset of 2000 images.

Q6. Suggestions for Style.

We will adjust the plots and update the missing link as suggested. We refer to our model as 'unconditional' because we exclusively use its unconditional mode (with text prompt 'None').