Neural Residual Diffusion Models for Deep Scalable Vision Generation

Q1: Explanation of some details in Fig.1 & 2
R1: Thanks. We will explain these details below:

1. In Fig.2, (a), (b) and (c) respectively represent Flow-shaped Networks (e.g., Transformer), U-shaped Networks (e.g., U-Net) and Unified Stacking Network (our Neural-RDM). As depicted in line 98-118, (a) and (b) are both special cases of (c), i.e., $f_{\theta_i}$ in (a) corresponds to $F_{\theta_i}$in (c), whereas $f_{\theta_i^{(l)}}$ and $f_{\theta_i^{(r)}}$ in (b) jointly correspond to $F_{\theta_i}$ in (c). Note that (c) covers both data streams mentioned earlier with a unified dynamics formula, which is a significant difference from (a).
2. The dotted lines in Fig.1 & 2 indicate the residual connections.
3. The coloured arrows (i.e., blue arrows) denote the data streams of the stacked networks.

We will carefully supplement these details in the final version.

Q2: Dimensions of the variables
R2: We first want to clarify that this paper is actually a foundational work applicable to a variety of DMs' deep-scalable training, e.g., image, video and others, thus we do not specifically emphasize the dimensions of the variables, but they are actually easy to specify, e.g., $\boldsymbol{z}\in\mathbb{R}^{B \times F \times C \times H \times W}$($B$ is bach size, $F$ is video frames, $C$ is channels, $H$ and $W$ denote height and width respectively), $\alpha\in\mathbb{R}^{L \times D}$ and $\beta\in\mathbb{R}^{L \times D}$ ($L$ is the number of network layers, $D$ is dimension of the hidden layer) in video tasks. Note that the image tasks are slightly different from the videos in frame dimensions. If necessary, we'll add them in the final version.

Q3: Whether to denoise partially or completely
R3: We suspect this may be a serious misunderstanding, and we want to clarify that although the Neural-RDM network ($F_{\theta}$) is composed of a series of mrs-unit $F_{\theta_i}$, but it is still performed for single-step denoising, i.e., $\boldsymbol{z}(t)\rightarrow\boldsymbol{z}(t-1)$ for time t, as illustrated in Fig.2(d). Note that this point is completely consistent with the previous baseline LDM, DiT and Latte. Moreover, we want to further highlight: we introduce continuous-time Residual-Sensitivity ODE into the neural networks of depth L, NOT into the diffusion process (e.g., $\boldsymbol{z}(0)\rightarrow\boldsymbol{z}(T)$). Moreover, for U-shaped stacking networks, we want to specify $\mathcal{F} _ {\theta _ {i}}$ stands for both $f_{\theta_i^{(l)}}$ and $f_{\theta_i^{(r)}}$ in the i-th residual unit, i.e., the skip-connection $\boldsymbol{z} _ {i+1}\rightarrow\boldsymbol{z}_{2L-2-i}$ in UNet or U-ViT. The goal is to build a unified Denoising-Dynamics ODE under consistent dynamics formula, so as to facilitate the improvement and optimization of all stacked networks (including U-shaped and Flow-shaped) for better deep scalable training. We will make these points clearer in the final revision and hope the above clarifications can help understand our work better.

Q4: Explanation for the difference in performance between variant 0 and variant 4
R4: We understand the reviewer's concern, but that's actually unnecessary. Firstly, we want to clarify that the $\hat{\beta} _ {t,\phi}$ is not a simple scalar, but rather a series of learnable time-dependent parameters that are used to fit the mean-related scheduler (i.e., $\mu(z_t,t)$) in our Denoising-Dynamics ODE, thus the adjustment of $\hat{\beta} _ {t,\phi}$ directly impacts the denoising performance (i.e. the FVD score). More broadly, the $\hat{\alpha} _ {t,\phi}$ and $\hat{\beta} _ {t,\phi}$ respectively parameterized the variance- and mean-related scheduler $-\frac{1}{2}\sigma(t)^2$ and $\mu(\boldsymbol{z} _ t,t)$, which replaces human design in previous works as,

$\frac{d\boldsymbol{z} _ t}{dt} = \mu(\boldsymbol{z} _ t,t)-\frac{1}{2}\sigma(t)^2\cdot\Big[\nabla _ {z} \log p_t(\boldsymbol{z}_t)\Big] = \hat{\alpha} _ {t,\phi}\cdot\mathcal{F} _ \theta(\boldsymbol{z} _ t,t) + \hat{\beta} _ {t,\phi}.$

Whereas variant 4 contains only a scaling tensor $\hat{\alpha}_{t,\phi}$, making it difficult to fit the Denoising-Dynamics ODE, thus resulting in worse results than variant 0.

Q5: Details of the Loss Function
R5: Yes, the final loss function with sensitivity control is as stated in lines 162-163, which can be expressed as follows,

$\mathcal{L} _ s = ||\mathcal{F} _ \theta(\boldsymbol{z} _ {t}, t) - \nabla_z \log p _ t(\boldsymbol{z} _ t)||^2_2 + \gamma \cdot \sum _ {L}|| \hat{\alpha}_ {t,\phi} \cdot\frac{\partial f _ \theta(\hat{\boldsymbol{z}} _ {t},t)}{\partial\hat{\boldsymbol{z}} _ {t}}-\hat{\beta}_{t,\phi}||^2_2.$

The total loss is obtained by summing the loss per layer, with sensitivity of each mrs-unit layer $F_ {\theta_i}$ adaptively modulated and updated by $\hat{\alpha} _ {t,\phi}$ and $\hat{\beta} _ {t,\phi}$.

Q6: Determination of hyper-parameter $\gamma$
R6: The value of the $\gamma$ is determined via hyper-parameter search experiments. As shown in the table below, we conducted extensive experiments with different values of $\gamma$ on C2I & T2I tasks, benchmarking on ImageNet and JourneyDB datasets respectively. Based on the observation from experimental results, we select $\gamma=0.35$ as the optimal value.

Q1: Complexity of the model after introducing gated residual parameters
R1: We understand the reviewer's concern, but that's actually unnecessary. Firstly, we want to highlight the introduction of these gated residual parameters is a simple yet meaningful change to the common architecture of deep generative networks (e.g., U-Net or Transformer), which only requires adding learnable scaling and bias tensors in each vanilla residual connection layer to fit the mean-variance scheduling dynamics of the proposed Denoising-Dynamics ODE. Secondly, we designed an error correction loss and integrated it into the conventional score matching loss at a smaller proportion (i.e., $\gamma=0.35$), balancing the model's denoising ability and error correction performance while ensuring that the model is easy to train.

Q2: Computation requirements
R2: Neural-RDM focuses on helping existing or brand-new generative backbone networks support large-scale and deep-scalable training, which supports both full parameter training and partial parameter fine-tuning. In our experiments, 2/4*A100 80G GPUs are respectively adopted to fine-tune the learnable gated parameters for image tasks (C2I & T2I) and video tasks (N2V & C2V) and 8*A100 80G GPUs for the full parameter training from scratch.

Q3: Model parameters and GFLOPs
R3: Thanks for this helpful suggestion, we have added the details of model parameters and GFLOPs in the table below, which will be included in the final version.

Q1: Definitions and clarifications on scalability metrics
R1: Thanks for the helpful comment. We first want to clarify that the "Scalability" columns in Tab.1 & Tab.2 indicates the scaling capability (i.e., parameter scale and architecture stackability) of the evaluated models, meanwhile ensuring that errors do not accumulate as the network deepens, to facilitate the deep scalable training of the generative models (i.e., scaling law). Moreover, we want to re-clarify the meaning of the three metrics (i.e., cross, tick and double tick): 1) GAN: Non-scalability (cross), limited by the instability of deep network training between the generator and the discriminator; 2) U/F-shaped: Low-scalability (tick), supporting a certain deep-scalable training, but error accumulation will occur as the network deepens during gradient update; 3) Ours: High-scalability (double tick), supporting deep-scalable training with the integration of error correction mechanism. We will make these points clearer in the final revision and hope the above clarifications can help understand our work better.

Q2: Supplements of scalability experiments
R2: Thanks for this valuable suggestion, we have supplemented the scalability experiments versus the baseline (i.e., Latte-XL) in following table, benchmarking on more tasks (T2I, N2V & C2V) and more datasets (JourneyDB, Taichi-HD & UCF-101). As shown in the table, in the T2I task, as the network depth increases from 12 to 32, the IS-score of our Neural-RDM shows a significant improvement compared to the baseline, which implys the superiority of the error control mechanism in supporting deep scalable training. A similar trend can be observed in the N2V and C2V tasks, which consistently validate the deep scalability advantage of our Neural-RDM. We will supplement the above results to the final version to better illustrate the performance of the proposed method.

Q3: Regarding the imprecise claim.
R3: Thanks for this very thoughtful comment. We will refine this claim to make it more precise in the final version.