Sequential Data Generation with Groupwise Diffusion Process

We appreciate the reviewer's constructive and detailed comments. Especially, our strengths recognized by the reviewer are:

• Originality of the group-wise strategy
• Convincing experiments
• Controllability

In the following, we answer the specific questions raised by the reviewer.

Presentation-level comments

Thank you for pointing out the typos, grammatical errors, and other presentation-level issues. They were very helpful. We reflected them in our revised version, so please refer to it.

Questions about grouping (Sec. 3.1.)

...does this mean we divide the data into any possible integer in the interval $1 \leq k \leq d$, then that integer value represents the number of groups?

This is right. We revised the paragraph as follows:

...we divide data $\boldsymbol x \in \mathbb R^d$ into $k\in \{1,...,d\}$ disjoint groups and assign different noise schedules for each group. Specifically, we divide the indices $\{1,...,d\}$ into a partition $\{S_j\}_{j=1}^k$ where $\sum_j |S_j|=d$. $S_j$ is used for determining membership of $\boldsymbol x_i$ to $j$-th group by testing $i \in S_j$.

For example, in the block-wise grouping, $j$-th group corresponds to the $j$-th block of an image (there are $k$ blocks in total). $S_j$ is the set of the indices of each block.

The latent group

What is a latent group? When was the name latent group introduced? Or what variable corresponds to such a latent group?

Throughout the paper, "group of latent variables" and "latent group" were used interchangeably. In the revised version, we clarified this in the last paragraph of the introduction. Also, there was a typo in Sec. 3.1 which we replaced "latent group" with "group" during the revision. Thank you for pointing out.

Why do we use $\boldsymbol u_{\boldsymbol \theta}$ instead of $\boldsymbol v_{\boldsymbol \theta}$?

The problem of directly predicting $\mathbf A'(t)(\boldsymbol x - \boldsymbol z)$ is that a neural network has to infer the derivative of $\mathbf A(t)$ from $\mathbf A(t)$. On the other hand, $\boldsymbol u_{\boldsymbol \theta}$ parameterization does not face such challenges since the derivative of $\mathbf A(t)$ is already given as a part of it. In practice, we tried both and found that $\boldsymbol u_{\boldsymbol \theta}$ indeed outperforms $\boldsymbol v_{\boldsymbol \theta}$.

Effect of traversing other elements

In the Fig. 6 where traversing a single latent variable is associated to a particular featured changed in the images, what is the effect of traversing other elements like (2), (3), (10) or (20)?

We have tried traversing other elements, but in some cases the factors captured are not interpretable by humans. We argue that this is not a limitation of our work in particular but due to the ill-posedness of the problem itself. For example, in the first column of Fig. 1 in $\beta$-VAE paper [1], one can see that changing the azimuth also affects e.g. hair color. Without additional assumptions or inductive biases, it is too ambitious to hope that a factor captured by neural networks is always interpretable by humans because the generative model that transports $p(\boldsymbol z)$ to $p(\boldsymbol x)$ is not unique [2].

In which scenarios should we apply GDM or GDM-F?

Compared to standard diffusion models, GDM can be particularly useful when we have some apriori knowledge of the underlying data-generating process. For instance, in video data, temporal ordering can be a natural design choice, so sequentially generating each frame could be more effective. In this case, the pixels in each frame consist of a group, and $k$ is equal to the number of frames.

One could further consider generating some frames in parallel rather than generating each frame sequentially because the information in neighboring frames would be highly redundant. One could even generate some key-frames first and then interpolate between them, or generate key pixels/frames first and then fill in the rest across the spatial/temporal axis, which has shown to be effective in practice [3]. GDM unifies all these grouping and ordering strategies into a single unified framework.

One might consider applying GDM-F if the most salient features of the data are captured by low-frequency components as in images. If this is the case, GDM-F opens up many interesting applications such as hierarchical latent space, extracting independent generative factors that are semantically meaningful, editing, etc., as we have shown in the paper.

[1] β-VAE: LEARNING BASIC VISUAL CONCEPTS WITH A CONSTRAINED VARIATIONAL FRAMEWORK

[2] Challenging Common Assumptions in the Unsupervised Learning of Disentangled Representations

[3] Imagen Video: High Definition Video Generation with Diffusion Models

Dear Reviewer efZH,

We thank the Reviewer for the constructive comments. As the end of the discussion period is approaching, we would like to ask whether our responses have addressed your concerns and questions adequately. If not, we would be happy to discuss further.

Regards,

the authors.

We appreciate the reviewer's comments. The review contains some significant misconceptions, which we will address below.

Limited novelty

Overall, I find this method has limited novelty. It is a simple extension of the model such as rectified flow by using a matrix interpolation function.

Our method uses a very specific interpolation function in Eq. 10, not just any matrix-valued function. The connection between autoregressive models and cascaded diffusion models is only valid because of this specific choice of the interpolation function. To our knowledge, extending diffusion models to support the sequential generation of each group has not been done previously. The other two reviewers also agreed that the connection with autoregressive and cascaded diffusion models is indeed novel.

Other contributions

Our unified model also accompanies useful properties including hierarchical latent space, finding independent generative factors, etc., which have not been explored in previous work.

Why do certain grouping and ordering work better than others?

It is not explained and supported that why we should partite the data by a certain order and why such partition gives better generation quality.

The aim of this paper is not to propose the best grouping and ordering strategy but to show that grouping and ordering is important design choices in sequential generative models, and we are already very explicit about this in the paper. The reviewer is correct that the reason why certain grouping and ordering work better than others is often not human-interpretable. Indeed, the progressive growing strategy of CDM is outperformed by bottom->top ordering or even CDM-inverse, and it is hard to interpret the meaningful distinction between the best and worst performing orderings of the block-wise grouping in Fig. 5. This is not a limitation of our work but is in fact a significant observation because it implies that the commonly used groupings and orderings, even if they appear to be a good design choice from a perspective of a human, could be not the best choice.

Dear Reviewer 5sF6,

We thank the Reviewer for the constructive comments. As the end of the discussion period is approaching, we would like to ask whether our responses have addressed your concerns and questions adequately. If not, we would be happy to discuss further.

Regards,

the authors.

We appreciate the reviewer's insightful comments, highlighting our strengths in

• interesting connection between diffusion and autoregressive/cascaded diffusion models,
• intriguing properties of GDM supported by promising experiment results
• extension to the frequency domain

Regarding your specific concerns:

The number of groups $k$

$k$ can be chosen based on the user's demand. For example, if a user wants autoregressive models, $k$ should be set to $d$. If a user wants cascaded diffusion with two stages, $k$ should be 2. What types of sequential generation a user needs depends on the data type and the user's prior knowledge of it.

Also, setting $k$ to be greater than 1 enables various use cases such as obtaining hierarchical representation, extracting independent generative factors, image editing, and so on.

If we set the number of groups equal to 1, how does it compare with DDPM or DDIM?

GDM is equivalent to previous diffusion models (or rectified flow) when $k=1$. For example, if we use the interpolation coefficients in Eq. 1, GDM reduces to the variance-preserving (VP) diffusion models. We made this even more clear in the last paragraph of Sec. 3.1. DDPM is a discretization of VP diffusion, and DDIM is a discretization of the probability-flow ODE.

Comparison with VAEs, GANs, and guided diffusion

We would like to emphasize that the primary focus of our paper is to propose a unified framework for diffusion models and other sequential generative models and dissect interesting properties arising from it.

Since we do not claim state-of-the-art generation performance, we believe it is not necessary to compare with VAEs and GANs, which are distinct families from diffusion and autoregressive models. Also, guided diffusion [4] is a method that utilizes classifier gradients for making a diversity-fidelity trade-off, which is not directly aligned with the scope of our work.

When $k=1$, GDM reduces to standard diffusion models, so the sample quality is on par with other diffusion models, given that the same architecture and training configuration are used. For $k>1$ case, please refer to Appendix A.2.

[1] Cascaded Diffusion Models for High Fidelity Image Generation

[2] PixelCNN++: Improving the PixelCNN with Discretized Logistic Mixture Likelihood and Other Modifications

[3] simple diffusion: End-to-end diffusion for high resolution images

[4] Diffusion Models Beat GANs on Image Synthesis

Dear Reviewer Mi2L,

We thank the Reviewer for the constructive comments. As the end of the discussion period is approaching, we would like to ask whether our responses have addressed your concerns and questions adequately. If not, we would be happy to discuss further.

Regards,

the authors.