Reply to Reviewer Zc8J

We thank the reviewer for valuable feedback. Below, we address the concerned points:

1. Possible Reason for Suboptimal FID of SAR-TS

Actually, we find that the image quality of SAR-TS is good, but there are serious framing misalignment issues. For example, the randomly generated 8 images in Figure 15 in the Appendix. Among large number of generated images, the subjects frequently encounter problems of extremely large, extremely small, or only part displayed. We think it leads to the suboptimal FID. Further, the issue may come from the strong reliance of SAR-TS on position embedding. Classical AR methods rely on static relative position relation, i.e., the next-token relation, so AR does not rely on position embedding. But SAR-TS uses random order, it perceives position completely from position embedding. It also explains that SAR-AR gets better results than the AR baseline, but falls behind when using random orders.

2. Training/Inference Speed Gain from KV Cache

For a comprehensive comparison, we examine the time cost of FMT-XL (for text-to-image generation) when generating one 1024x1024 image using one A100 GPU. The generation order of AR is raster, while those of MAR and SAR are both random (actually the order does not affect speed). The results are as follows:

We empirically find that there is no visual difference between using 64 steps and more. And, the visualization in Figure 1 and Figure 13 (in the Appendix) are all sampled with 64 steps. We can see that the inference speed of SAR is > 60 times than that of AR with KV cache enabled. Notably, in the transformer decoder, MAR applies global attention across all tokens, while the number of tokens processed in AR and SAR-TS increases gradually. Consequently, even without KV cache, the inference time for SAR-TS is shorter than that of MAR; when KV cache is enabled, SAR-TS is three times faster than MAR with $64$ or $128$ steps.

3. Performance of Text-to-Image Generation

We test the performance with 64 steps and compare it with other existing methods/models. The results are as follows, where the test dataset is MJHQ-30K [1], results of other methods in the table is cited from SANA [2], and the FID metric and CLIP score is calculated using [3] and [4] respectively.

[1] Li et al. Playground v2.5: Three Insights towards Enhancing Aesthetic Quality in Text-to-Image Generation

[2] Xie et al. SANA: Efficient High-Resolution Image Synthesis with Linear Diffusion Transformers

[3] https://github.com/mseitzer/pytorch-fid

[4] https://github.com/Taited/clip-score

[5] Podell et al. SDXL: Improving latent diffusion models for high-resolution image synthesis

[6] Esser et al. Scaling rectified flow transformers for high-resolution image synthesis

[7] Black Forest Labs. Flux, 2024. URL https://blackforestlabs.ai/

4. Questions

i) Thanks for indicating this. The $n$ in Eq. (1) is a typo. ii) 'random-2-random' means random order, 2 sets, random intervals. During training, MAR randomly divides tokens into 2 sets with random order, but it only calculates loss on the masked part. Therefore we said that ‘it can be derived by removing the loss of the first token set in ‘random-2-random’. iii) m_e, m_{ds}, m_{dc} mean causal masks for encoder self-attention, decoder self-attention and decoder cross-attention. iv) There is no strong relation in training and inference in SAR. One can adjust the output intervals to achieve sampling with any number of steps as in Algorithm 2. v) Thanks for your suggestion and we will explore the scaling behavior in the future.

5. Supplementary on Training/Inference Speed Gain from KV Cache

For a comprehensive comparison, we examine the time cost of FMT-XL (for text-to-image generation) when generating one 1024x1024 image using one A100 GPU. The generation order of AR is raster, while those of MAR and SAR are both random (actually the order does not affect speed). The results are as follows:

We empirically find that there is no visual difference between using 64 steps and more. And, the visualization in Figure 1 and Figure 13 (in the Appendix) are all sampled with 64 steps. We can see that the inference speed of SAR is > 60 times than that of AR with KV cache enabled. Notably, in the transformer decoder, MAR applies global attention across all tokens, while the number of tokens processed in AR and SAR-TS increases gradually. Consequently, even without KV cache, the inference time for SAR-TS is shorter than that of MAR; when KV cache is enabled, SAR-TS is three times faster than MAR with $64$ or $128$ steps.

Reply to Reviewer rmFX

Thank you very much for your careful reading and valuable feedback! Below we will address the concerns:

1. Further Clarification on Differences between SAR and other Methods

i) Compared to classical AR, a) SAR can be faster at inference, and b) SAR enables image editing, such as inpainting/outpainting due to the flexibility of order.

ii) Compared to MAR, a) the inference speed is three times faster (can be referred to 5). b) SAR offers a more flexible way for causal learning. During causal training, SAR can simply incorporate any modality just by setting or adjusting the generalized causal mask. For example in video generation, we may only modify the mask to reflect the time order, and it can be trained as a video generator. Moreover, due to the causal manner of language, SAR can also easily facilitate multimodal learning.

iii) Compared to VAR, a) SAR directly operates with common VAE, while VAR needs a customized multiscale VAE. b) SAR is more flexible in processing images, videos and other modality or multimodality.

2. Further Discussion on Experiments

i) When compared with LlamaGen, all settings are the same except the model.

ii) Since SAR-TS is new to AR models, we mainly focus on evaluating SAR-TS while do not explore much of other settings.

iii) On the performance of SAR-TS. Actually, we find that the image quality of SAR-TS is good, but there are serious framing misalignment issues. For example, the randomly generated 8 images in Figure 15 in the Appendix. Among large number of generated images, the subjects frequently encounter problems of extremely large, extremely small, or only part displayed. We think it leads to the suboptimal FID. Further, the issue may come from the strong reliance of SAR-TS on position embedding. Classical AR methods rely on static relative position relation, i.e., the next-token relation, so AR does not rely much on position embedding. But SAR-TS uses random order, it perceives position completely from position embedding. It also explains that SAR-AR gets better results than the AR baseline, but falls behind when using random orders. We think better position strategy may solve this problem. And for further verifying the generation ability of SAR-TS, we also train a text-to-image model, which can generate high-quality and high-resolution images after very short training (200k iterations with batchsize of 256).

3. Further Clarification of Statements

i) The mismatch of MAR training. During training of MAR, it always randomly masks 70%-100% tokens; but at inference, the generated (unmasked) tokens grow gradually. For example, there are 70% unmasked tokens and the model is to predict the rest 30%. This scenario is not seen during training.

ii) Sorry for that, and m_e, m_{ds}, m_{dc} mean causal masks for encoder self-attention, decoder self-attention and decoder cross-attention.

iii) We think apart from 64-set and 256-set training, other variants lose too much performance. So we said that only 64-set training is suitable for few-step generation.

iv) The '2' in 'random-2-random' means 2 sets. During training, MAR randomly divides tokens into 2 sets with random order, but it only calculates loss on the masked part. And we use K to denote the number of output sets that is used in calculating loss. So K=1.

v) In the middle of Figure 9, the sampling steps are always 64. Sorry for misleading. It is true that "causal calculation" and "full calculation" refer to causal and full attention masks, respectively.

4. Questions

i) If using such model, in the attention calculation, the seen part conducts causal attention, and the position embedding part conducts self- and position-to-seen causal attention. It allows KV cache, but does not allow parallel causal training. Based on this model, we turn to use whole position embeddings and a encoder-decoder architecture. Your comments also coincide with our idea. By the way, it also works with only the decoder of FMT, but the performance may degrade.

ii) The sampling steps are 64 as said in 3 (v).

Thank you for the your response. I appreciate the comprehensive experiments presented in this work and the effort put into addressing my questions. However, I still have some concerns regarding the response:

A1. (a) The authors claim that MAR is not suitable for learning other modalities. More explanation is needed to support this statement. In my understanding, as long as sequences and tokens are properly defined, MAR can also perform generation in the same way it does for images. (b) I mentioned VAR in my initial review because it is a causal method that predicts a set of tokens in one step, independent of how tokens and sequences are defined (e.g., based on resolution). The authors have not explained how this method differs from SAR.

A2. While the authors have pointed out reasons for the suboptimal results, they have not addressed the problem itself. The practical value of SAR-TS remains unclear to me.

A3. Thank you for the clarification.

A4. Thank you for the clarification.

A5. The comparison of time between MAR and SAR seems unfair, as the authors do not show the performance of the two methods.

In Summary, the authors' response does not address my primary concerns: the insights provided by SAR are limited, and the results are not compelling enough.

Reply to Reviewer vUCC

We thank the reviewer for the feedback. And we will address the concerns as follows:

1. Possible Reason for Suboptimal FID of SAR-TS

Based on observation, we find that the image quality of SAR-TS is good, but there are serious framing misalignment issues. For example, the randomly generated 8 images in Figure 15 in the Appendix. Among large number of generated images, we find that subjects frequently encounter problems of extremely large, extremely small, or only part displayed. We think this leads to the suboptimal FID. Further, the framing misalignment issue may come from the strong reliance of SAR-TS on position embedding. Classical AR methods rely on static relative position relation, i.e., the next-token relation, so AR does not need position embedding. But SAR-TS uses random order, the perception of position completely comes from position embedding. It also explains that SAR-AR gets better results than AR baseline, but falls behind when using random orders.

2. The Claim of Fully Masked Transformer

Our claim about FMT is, it enables causal learning with any sequence order and any output intervals, which means: i) the training is causal and parallel, ii) at inference, it can output tokens in any order, and can output multiple tokens at once. LlamaGen only satisfies i), but not ii).

3. Questions

SAR is a paradigm on training/inference, and it is not limited to cross-entropy loss. With diffusion loss, it will work like MAR. K=1 in SAR means that the loss is calculated with only one group of output tokens at each training iteration, as shown in Figure 2. With causal mask disabled, conceptually it is the same as MAR (but the model and loss are different).

Reply to Reviewer 7W6R

We appreciate the reviewer for the helpful comments. Below, we address each of the points raised:

1. Using Cosine Schedule

We use cosine schedule for its simplicity in performance evaluation, and it is not our contribution.

2. Questions on Ablation study

Since FMT uses position embedding to indicate positions, it treats any sequence order equally. As a result, we cannot judge which order performs best in advance. For the results on aligned order, we will put them into the Appendix. Thanks for your advice.

Reply to Reviewer QJJo

We appreciate the reviewer's feedback and questions regarding our work. Below, we address each of the points raised:

1. The Meaning of SAR-TS

"TS" denotes transition state, i.e., the variant that can output tokens in any order with few-step inference. The definition is in the caption of Table 4.

2. Training/Inference Speed Gain from KV Cache

First, we would like to explain that the visualization in Figure 1 and Figure 13 (in the appendix) are all sampled with 64 steps (the total number of tokens is 4096). And we find that there is no visual difference between using 64 steps and more. For a comprehensive comparison, we examine the time cost of FMT-XL (for text-to-image generation) when generating one 1024x1024 image using one A100 GPU. The generation order of AR is raster, while those of MAR and SAR are both random (actually the order does not affect speed). The results are as follows:

We can see that the inference speed of SAR is > 60 times than that of AR with KV cache enabled. Notably, in the transformer decoder, MAR applies global attention across all tokens, while the number of tokens processed in AR and SAR-TS increases gradually. Consequently, even without KV cache, the inference time for SAR-TS is shorter than that of MAR; when KV cache is enabled, SAR-TS is three times faster than MAR with $64$ or $128$ steps.

3. Performance of Text-to-Image Generation

Since we only train the T2I model under the setting of SAR trasition state, the performance is optimal with causal attention and random order. Therefore we cannot compare different inference settings (AR, MAR, SAR-TS) fairly. Besides, the AR inference time with full 4096 steps is quite long (3 min for one image), so it may not be practical to evaluate it at high resolution. Here, we test the performance with 64 steps and compare it with other existing methods/models. The results are as follows, where the test dataset is MJHQ-30K [1], results of other methods in the table is cited from SANA [2], and the FID metric and CLIP score is calculated using [3] and [4] respectively.

[1] Li et al. Playground v2.5: Three Insights towards Enhancing Aesthetic Quality in Text-to-Image Generation

[2] Xie et al. SANA: Efficient High-Resolution Image Synthesis with Linear Diffusion Transformers

[3] https://github.com/mseitzer/pytorch-fid

[4] https://github.com/Taited/clip-score

[5] Podell et al. SDXL: Improving latent diffusion models for high-resolution image synthesis

[6] Esser et al. Scaling rectified flow transformers for high-resolution image synthesis

[7] Black Forest Labs. Flux, 2024. URL https://blackforestlabs.ai/