StochSync: Stochastic Diffusion Synchronization for Image Generation in Arbitrary Spaces

Thank you for highlighting the novelty of our method, particularly noting that "the idea of using multi-step backward equipped with non-overlapping view sampling is novel." Below, we provide detailed responses to each comment.

---

Q1. Ablation Study for Non-Overlapping View Sampling.

A1. We kindly remind the reviewer that an ablation study is indeed presented in our paper, addressing the effectiveness of Non-Overlapping View Sampling (N.O. Views) both qualitatively and quantitatively. The results are detailed in Fig. 3, Tab. 2, and Sec. 7.1.2 of the main paper.

As the reviewer noted, blending the outputs of multi-step computation in deterministic sampling (DDIM) can introduce visible seams (row 3, Fig. 3 of the main paper). To address this, we explored increasing the level of stochasticity to its maximum, which achieves global coherence with reasonable quality (row 4, Fig. 3 of the main paper). Finally, by incorporating N.O. Views, we produce sharper outputs with crisp details, resulting in $`StochSync`$ (row 6, Fig. 3 of the main paper). Quantitatively, we observe similar trend on most of the metrics reported in Tab. 2 of the main paper of the main paper. Notably, the utilization of N.O. Views results in improvements across all metrics, as shown in the transition from row 4 to row 6 in Tab. 2 of the main paper.

Q4. Use of Video Diffusion Model.

A4. We appreciate the insightful suggestion regarding the use of video diffusion models. Video diffusion models conditioned on camera trajectories can be utilized to generate $360^\circ$ panoramas. For our experiments, we provided camera parameters simulating a full $360^\circ$ rotation at a fixed position to a pose-conditioned video diffusion model, CameraCtrl [18], to generate a 36-frame video. From these frames, we selected six views at 6-frame intervals, corresponding to instance views with $60^\circ$ azimuth differences, and used them for panorama generation. In Fig. 15 of the main paper, we present qualitative results of $360^\circ$ panorama generation with $`StochSync`$ using a pose-conditioned video diffusion model, CameraCtrl [18]. While the results demonstrate promising spatial consistency across views (rows 1–3), we often observed failure cases (row 4) where the model struggles to generate plausible panoramas. This is likely due to errors propagating from the pose-conditioned video diffusion model which struggles to generate precise frames captured at the conditioned camera pose, as shown in Fig. 16 of the main paper. For future work, we believe that leveraging more accurate pose-conditioned video diffusion models could significantly improve the quality of panorama generation.

---

Q5. 3D Mesh Texturing with PBR Materials.

A5. We thank the reviewer for suggesting an interesting extension of 3D mesh texturing using $`StochSync`$. For 3D mesh texturing with PBR materials, we replace the RGB texture image of the 3D mesh with PBR texture maps. Since $\mathbf{x} _{0|t}$ cannot be directly unprojected to PBR maps, we employ a differentiable renderer [19] to update the canonical sample parameters through gradient descent. The qualitative results are shown in "Output" column of Fig. 14 of the main paper, along with examples of specular map modifications in "Spec.$\uparrow$" and "Spec.$\downarrow$" columns. For future work, extending $`StochSync`$ to a light-conditioned diffusion model [20] to enable the generation of more realistic PBR materials would be a promising direction.

---

Q6. Conditional Generation using $`StochSync`$.

A6. In Tab. 3 and Fig. 4 of the main paper, we present quantitative and qualitative results comparing $`StochSync`$ with DS-based methods for 3D mesh texturing, a conditional generation setup utilizing depth maps. While DS-based methods perform well with sufficient conditioning, $`StochSync`$ also shows comparable results. Notably, when the conditions are insufficient, $`StochSync`$ significantly outperforms DS-based methods, as shown in Tab. 2 of the main paper.

---

Q7. Generating High-Resolution Outputs.

A7. Fig. 13 of the main paper shows 8K panorama images generated using $`StochSync`$. To generate high-resolution panoramas, we adapted $`StochSync`$ by narrowing the field of view for individual views and increasing the number of samples, thereby raising the resolution of the canonical sample. However, increasing the number of views poses the risk of repetitive objects appearing in the scene. To address this, we employed the refinement technique from SDEdit [1]. Specifically, a panorama is first generated using the original setup described in Sec. B of Appendix. The resulting image is then perturbed with noise at $t=600$, and the sampling process is restarted from this timestep.

We thank the reviewer for noteworthy praises of our work, such as "the authors clearly illustrate their motivation" and "the paper provides detailed ablation studies." Below, we provide detailed responses to the comments.

---

Q1. Merging SDS and DS.

A1. We respectfully disagree with the statement that our contribution is limited to simply combining Diffusion Synchronization (DS) and Score Distillation Sampling (SDS). SDS and DS have been regarded as fundamentally distinct frameworks, and combining them in a meaningful way is far from trivial. Below, we present contributions of our work. First, by re-interpreting SDS with respect to $\mathbf{x} _{0|t}$, we identified algorithmic-level similarities between the two frameworks. Second, as discussed in Sec. 5 of the main paper, we provide a comprehensive analysis of the key components of both frameworks, highlighting their similarities and differences. Third, based on the analysis, in Sec. 6 of the main paper, we propose a new framework that effectively integrates the benefits of the two frameworks. To the best of our knowledge, this work is the first to address the level of stochasticity in DS. Additionally, we introduce $`StochSync`$, with two novel techniques: Multi-Step $\mathbf{x} _{0|t}$ Computation and Non-Overlapping View Sampling, which play a critical role in further enhancing the quality. Last but not least, our proposed zero-shot method achieves new state-of-the-art performance, even outperforming finetuning-based methods [3, 6, 17].

---

Q2. Discussion of Computational Efficiency.

A2. Please refer to A1 of the global rebuttal, where we address the concerns regarding the computational efficiency of $`StochSync`$. We provide a comparison of running times and discuss techniques to reduce the computational costs. Specifically, we tuned the hyperparameters of our method and replaced the ODE solver with a more advanced one. As a result, surprisingly, $`StochSync`$ achieves the fastest running time across the baselines in $360^\circ$ panorama generation.

---

Q3. Multi-View 3D Results.

A3. Thank you for the valuable feedback. We also agree that multi-view consistency is a crucial factor in 3D mesh texturing. We present the results of 3D mesh texturing using an interactive 3D viewer, available at the following anonymous website: https://stochsync.github.io/.

Dear Reviewer DvFj,

The authors sincerely appreciate the reviewer for their constructive feedback and suggestions, as well as for viewing our work positively. While the discussion period has been extended, we now have just a few days to address any further comments from the reviewers. We kindly ask if there are any additional concerns or questions regarding our work.

In response to the feedback, we have clarified the contributions of our work and included running time comparison results. Furthermore, we have outlined several potential extensions of StochSync, including high-resolution panorama generation, the use of video diffusion models, and 3D mesh PBR texturing. Please feel free to share any additional questions or concerns.

We appreciate the reviewer for highlighting the strengths of our work, such as "the paper makes a good effort at comparing existing methods" and "the algorithms are very clearly presented." Below, we address the comments, including further elucidation of the components of $`StochSync`$.

Q1. Analysis of Maximum Stochasticity and Non-Overlapping View Sampling.

A1. We fully understand the concern raised by the reviewer and sincerely appreciate the constructive feedback on our paper. Analyzing $`StochSync`$ presents challenges due to the interplay of its various components, which complicates a straightforward analysis.

To elucidate the intuitions behind the algorithmic details of $`StochSync`$—Max $\sigma _t$, Impr. $\mathbf{x} _{0|t}$, and N.O. Views—we simplify the setup by: (1) considering only a single instance view, $N=1$, (line 8, Alg 4), (2) using an identity as the projection operation (line 9, Alg. 4), and (3) modifying the objective function (lines 6, 13 of Alg. 4).

As an example of such case, we consider image inpainting, where the objective is to generate a realistic image $\mathbf{x} _0$ that aligns with the partial observation $\mathbf{y} = \mathbf{M} \odot \mathbf{x} _0$, where $\mathbf{M} \in \{ 0, 1 \}$ represents a binary mask. To guide the sampling process, the generation is conditioned by replacing $\mathbf{M} \odot \mathbf{x} _{0|t}$ with $\mathbf{y}$.

Under these simplifications, the update rule for $\mathbf{z}$ becomes:

$

\mathbf{z} = \text{argmin} _{\mathbf{z}} \left[ \lVert (1 - \mathbf{M}) \odot (\mathbf{z} - \mathbf{x} _{0|t-1}) \rVert^2 + \lVert \mathbf{M} \odot (\mathbf{z} - \mathbf{y}) \rVert^2 \right].

$

To analyze the effectiveness of the level of stochasticity on synchronization, we examine the convergence rate of measurement error, $\mathcal{L}(\mathbf{x} _{0|t}) = \lVert \mathbf{M} \odot \mathbf{x} _{0|t} - \mathbf{y} \rVert^2$, for two cases: $\sigma _t = 0$ and $\sigma _t = \sqrt{1-\alpha _{t-1}}$ (Max. $\sigma _t$), respectively. As discussed in Sec. 4 of the main paper, when $\sigma _t = 0$, the sampling process becomes fully deterministic. To better illustrate our intuitions, we make two reasonable and straightforward assumptions:

• The initial sample $\mathbf{x} _T \sim \mathcal{N}(\mathbf{0}, ***I***)$ satisfies $\mathcal{L}(\mathbf{x} _{0|T}) \gg 0$ and $\mathcal{L}(\mathcal{G}(\mathbf{x} _T)) \gg 0$.

• The pretrained noise prediction network $\epsilon _\theta(\cdot, \cdot)$ is $K$-Lipschitz, satisfying $|\epsilon _\theta(\mathbf{x} _t, t) - \epsilon _\theta(\mathbf{x} _{t-\Delta t}, t - \Delta t)| < K |\mathbf{x} _t - \mathbf{x} _{t-\Delta t}|$ for some constant $K$.

Under these assumptions, the reformulation of a one-step denoising process from the perspective of $\mathbf{x} _{0|t}$ yields:

$

\nonumber
\mathbf{x} _{0|{t-\Delta t}} &= \mathbf{x} _{0|t} + \sqrt{\frac{1-\alpha _{t-\Delta t}}{\alpha _{t-\Delta t}}} \left( \boldsymbol{{\epsilon}} _t - \boldsymbol{{\epsilon}} _{t-\Delta t} \right), \\\\
\nonumber
\therefore |\mathbf{x} _{0|t - \Delta t} - \mathbf{x} _{0|t}| &= \sqrt{\frac{1-\alpha _{t-\Delta t}}{\alpha _{t-\Delta t}}} | \boldsymbol{{\epsilon}} _t - \boldsymbol{{\epsilon}} _{t-\Delta t} | < \sqrt{\frac{1-\alpha _{t-\Delta t}}{\alpha _{t-\Delta t}}} K |\mathbf{x} _t - \mathbf{x} _{t-\Delta t}| \approx 0,

$

where the approximation equality holds when $\Delta t \approx 0$. This implies that $\mathbf{x} _{0|{t-\Delta t}}$ is largely dependent by the previous sample $\mathbf{x} _{0|t}$, and as a result, the measurement error $\mathcal{L}(\mathbf{x} _{0|t})$ can remain large even after a few steps of the denoising process, thereby slowing down the convergence of $\mathbf{x} _{0|t}$ to $\mathbf{y}$.

On the other hand, when setting $\sigma _t = \sqrt{1-\alpha _{t-1}}$ (Max. $\sigma _t$), $\mathbf{x} _{0|{t-\Delta t}}$ is no longer dependent on $\mathbf{x} _{0|t}$, allowing $\mathbf{x} _t$ and $\mathbf{x} _{t-\Delta t}$ to differ significantly, even for small $\Delta t$.

This process can be interpreted as resetting the denoising trajectory based on $\mathbf{x} _{0|t}$, enabling the exploration of $\mathbf{x} _{t-\Delta t}$ that minimizes the measurement error. While it is also true that the newly sampled $\mathbf{x} _{t-\Delta t}$ could potentially deviate from the desired trajectory and increase $\mathcal{L}(\mathbf{x} _{0|t-\Delta t})$, our empirical observations show that, in most cases, it converges to the measurement within a few denoising steps.

Dear Reviewer yJqb,

We sincerely thank the reviewer for taking the time to review our work and for providing valuable and constructive feedback, which has significantly contributed to improving our paper. While the discussion period has been extended, we now have just a few days to address any further comments from the reviewers. We kindly ask if there are any additional concerns or questions regarding our work.

To address previous feedback, we have provided detailed explanations of the algorithmic components of StochSync and clarified the statement in L316: "Increased Stochasticity Leads to Greater Deviation." Furthermore, we have included additional experiments on inference time and evaluation metrics (user study). Please feel free to share any additional questions or concerns.

I have reviewed the analysis (although I haven't checked every single equation), and it appears to be very important for the paper. It should be included in the final version or added to the appendix.

However, I find some of the analysis inaccurate:

The forward process under maximum stochasticity diverges and cannot be approximated by a Stochastic Differential Equation (SDE) as the timestep interval approaches zero.

Not really true. The stochasticity of SDE can be controlled with an extra parameter. See [1][2]. The proposed method (resetting the denoising trajectory) can also be interpreted as a numerical solver of such SDE, similar to adding the S churn parameter in K-diffusion [3]. (I suggest citing these papers in the final version.)

For the claim in L316, I feel the following is a better explanation:

Unfortunately, as the one-step prediction computed using Tweedie’s formula [11] often deviate from the clean data manifold, sampling process using Max $\sigma_t$ leads to being placed in low-density regions of the noisy data distribution, ultimately degrading the quality of $x_0$

Another minor inaccuracy I noticed:

the measurement error can remain large even after a few steps of the denoising process, thereby slowing down the convergence

"Slowing down" seems inaccurate here. My understanding is that the error cannot be reduced and the final trajectory always deviates from real data no matter how many sampling steps are used in deterministic ODE sampling.

[1] Xue et al. SA-Solver: Stochastic Adams Solver for Fast Sampling of Diffusion Models
[2] Cao et al. Exploring the Optimal Choice for Generative Processes in Diffusion Models: Ordinary vs Stochastic Differential Equations
[3] Karras et al. Elucidating the Design Space of Diffusion-Based Generative Models

I sincerely appreciate the authors' detailed responses and their efforts in adding the evaluation and providing the inference times. Unfortunately, the formatting issues with LaTeX formulas on OpenReview made it challenging to review everything thoroughly. Nonetheless, I maintain my original rating.

Edit: Sorry about that. It seems that the formatting issues are gone after refreshing. I will review these comments soon.

Refrerences

[1] Chenlin Meng, Yutong He, Yang Song, Jiaming Song, Jiajun Wu, Jun-Yan Zhu, Stefano Ermon. "SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equations." International Conference on Learning Representations (ICLR), 2022.

[2] Zhipeng Cai, Matthias Mueller, Reiner Birkl, Diana Wofk, Shao-Yen Tseng, Junda Cheng, Gabriela Ben-Melech Stan, Vasudev Lal, Michael Paulitsch. "L-MAGIC: Language Model Assisted Generation of Images with Coherence." IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024.

[3] Shitao Tang, Fuyang Zhang, Jiacheng Chen, Peng Wang, Yasutaka Furukawa. "MVDiffusion: Enabling Holistic Multi-view Image Generation with Correspondence-Aware Diffusion." Advances in Neural Information Processing Systems (NeurIPS), 2023.

[4] Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, Jun Zhu. "DPM-Solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling in Around 10 Steps." Advances in Neural Information Processing Systems (NeurIPS), 2022.

[5] Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, Jun Zhu. "DPM-Solver++: Fast Solver for Guided Sampling of Diffusion Probabilistic Models." arXiv preprint arXiv:2211.01095, 2022.

[6] Yuxuan Zhang, Yifan Jiang, Zhaoyang Liu, Zhangyang Wang. "Taming Stable Diffusion for Text to 360° Panorama Image Generation." IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024.

[7] Jaihoon Kim, Juil Koo, Kyeongmin Yeo, Minhyuk Sung. "SyncTweedies: A General Generative Framework Based on Synchronized Diffusions." Advances in Neural Information Processing Systems (NeurIPS), 2024.

[8] Yaron Lipman, Ricky T. Q. Chen, Heli Ben-Hamu, Maximilian Nickel, Matt Le. "Flow Matching for Generative Modeling." International Conference on Learning Representations (ICLR), 2023.

[9] Xingchao Liu, Chengyue Gong, Qiang Liu. "Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flow." International Conference on Learning Representations (ICLR), 2023.

[10] Xingchao Liu, Xiwen Zhang, Jianzhu Ma, Jian Peng, Qiang Liu. "InstaFlow: One Step is Enough for High-Quality Diffusion-Based Text-to-Image Generation." International Conference on Learning Representations (ICLR), 2024.

[11] Herbert E Robbins. "An empirical bayes approach to statistics." In Breakthroughs in Statistic, 1956.

[12] Jiaming Song, Chenlin Meng, Stefano Ermon. "Denoising Diffusion Implicit Models." International Conference on Learning Representations (ICLR), 2021.

[13] Ben Poole, Ajay Jain, Jonathan T. Barron, Ben Mildenhall. "DreamFusion: Text-to-3D using 2D Diffusion." International Conference on Learning Representations (ICLR), 2023.

[14] Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, Ren Ng. "NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis." European Conference on Computer Vision (ECCV), 2020.

[15] Bernhard Kerbl, Georgios Kopanas, Thomas Leimkühler, George Drettakis. "3D Gaussian Splatting for Real-Time Radiance Field Rendering." ACM Transactions on Graphics (SIGGRAPH), 2023.

[16] Jiaxiang Tang, Zhaoxi Chen, Xiaokang Chen, Tengfei Wang, Gang Zeng, Ziwei Liu. "LGM: Large Multi-View Gaussian Model for High-Resolution 3D Content Creation." European Conference on Computer Vision (ECCV), 2024.

[17] Xianfang Zeng, Xin Chen, Zhongqi Qi, Wen Liu, Zibo Zhao, Zhibin Wang, Bin Fu, Yong Liu, Gang Yu. "Paint3D: Paint Anything 3D with Lighting-Less Texture Diffusion Models." IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024.

[18] Hao He, Yinghao Xu, Yuwei Guo, Gordon Wetzstein, Bo Dai, Hongsheng Li, Ceyuan Yang. "CameraCtrl: Enabling Camera Control for Text-to-Video Generation." arXiv preprint arXiv:2404.02101, 2024.

[19] Samuli Laine, Janne Hellsten, Tero Karras, Yeongho Seol, Jaakko Lehtinen, Timo Aila. "Modular Primitives for High-Performance Differentiable Rendering." ACM Transactions on Graphics (SIGGRAPH), 2020.

[20] Yuxuan Zhang, Yifan Jiang, Zhaoyang Liu, Zhangyang Wang. "DreamMat: High-quality PBR Material Generation with Geometry- and Light-aware Diffusion Models." ACM Transactions on Graphics (SIGGRAPH), 2024.

A1. We appreciate the reviewers for providing insightful feedback regarding the computational efficiency of our method. The concern raised by the reviewers is that $`StochSync`$ may suffer from long inference times due to the multi-step computation of $\mathbf{x} _{0|t}$, which could introduce significant computational overhead. Here, we show that this is not the case as our method with optimized hyperparameters already achieves a runtime comparable to L-MAGIC [2] and even outperforms MVDiffusion [3]. To our surprise, our method achieves the fastest running time when integrated with a more efficient ODE solver in $360^\circ$ panorama generation.

As discussed previously, our method with optimal hyperparameters, denoted as $`StochSync`^\ast$, effectively reduces the number of denoising steps to $8$, which significantly saves the computation cost. Furthermore, more advanced samplers [4, 5] can be utilized to solve ODEs more efficiently. Specifically, we plug DPM-Solver [4] to $`StochSync`^\ast$. The resulting method, $`StochSync`^\ast+`DPM-S`$, facilitates efficient computation of multi-step $\mathbf{x} _{0|t}$ using fewer denoising steps, thereby further reducing the runtime of our method.

In Tab. 1 and Tab. 2, we present a runtime comparison of $`StochSync`$ with the baselines for $360^\circ$ panorama generation and 3D mesh texturing, respectively. For the runtime comparison, the vanilla StochSync is evaluated using the same setup described in Sec. B of the Appendix. For the baselines, we utilize the default parameters specified in the implementations, setting the number of denoising steps to $50$ for MVDiffusion [3] and PanFusion [6], $30$ for SyncTweedies [7], and $25$ for L-MAGIC [2]. For 3D mesh texturing, the running time results are sourced from SyncTweedies [7]. As shown in Tab. 1, $`StochSync`^{\ast}$ significantly reduces the runtime compared to the vanilla StochSync while achieving better performance. Furthermore, as presented in Tab. 1 and Tab. 2, $`StochSync`^{\ast}+`DPM-S`$ reduces the number of ODE steps from $50$ to $20$, further improving computational efficiency without significant quality degradation. Notably, in $360^\circ$ panorama generation, $`StochSync`^{\ast}+`DPM-S`$ achieves the fastest runtime.

Q1. Inference Time Comparison.

Table 1: Quantitative results of panorama generation using the out-of-distribution prompts provided in PanFusion [6]. GIQA is scaled by $10^3$. The best result in each column is highlighted in bold, and the runner-up is $\underline{\text{underlined}}$.

Table 2: Quantitative results of 3D mesh texturing. The best result in each column is highlighted in bold, and the runner-up is $\underline{\text{underlined}}$.