Lumina-Next : Making Lumina-T2X Stronger and Faster with Next-DiT

Q1: Whether identifiers such as [nextline] and [nextframe] are unnecessary when using 3D RoPE is not sufficiently discussed in the paper.

Good point! Lumina-T2X adopted learnable [nextline] and [nextframe] tokens to achieve flexible modeling of 2D/3D signals with 1D RoPE. However, we found that modeling any modality as the 1D sequence leads to certain limitations, e.g., misalignment with the image/video's physical properties and limited multi-dimensional resolution extrapolation ability (refer to Q1@Reviewer NAeC for more details). For these reasons, we introduced 2D/3D RoPE in lumina-next to replace 1D RoPE with learnable tokens. In this case, we naturally don't need learnable tokens to assist the model in understanding the 2D / 3D structures. We shall supplement the relevant discussion upon acceptance.

Q2: The paper lacks a comparison with Stable Diffusion 3, which is also a flow-based diffusion model.

Thanks for the suggestion! The SD3 model remains closed-source when submitting this paper. Here we further supplement the quantitative comparison results with SD3. To better illustrate the aesthetics and semantic ability, we conduct an AI preference study to evaluate Lumina-Next against other text-to-image models, since conventional metrics such as FID and CLIP-Score may not accurately reflect the generation quality. Following PixArt, we employ GPT-4o, the SoTA multimodal LLM exhibiting strong alignment with human preference, as our evaluator to vote based on image quality. As shown in the following table, Lumina-Next demonstrates competitive performance with advanced text-to-image models including PixArt and SD3. Note that SD3 uses over 1B text-image pairs, which is ~100x greater than our models, PixArt, which is already a training-efficiencient model, still uses 3x training compute than ours. However, we have to admit that Lumina-Next still underperforms these SoTA models in terms of text-image alignment and compositional generation, due to inadequate data and training.

Q3: Is the proposed Time Schedule universally applicable at low NFEs (<10)? Figure 7 contains too few examples to reflect the actual effect. Are there any numerical comparisons?

Thanks for posing this question. As a zero-shot method simply changing the time schedule, it is hard to get satisfactory results using extremely low NFEs (<10). Usually, using ~20 NFEs can get decent results. Besides, the design of the temporal condition in Time-aware scaled RoPE makes it intuitively more sensitive to the inference step number. Here, we give the results for NFEs of 40, 20, and 10 with NTK/Time- aware Scaled RoPE, and we can see that resolution extrapolation with Time-aware scaled RoPE is getting worse. However, the overall trend is consistent with the degradation of image quality in the case of using NTK-aware Scaled RoPE, and there is no significant nullification. We will add these experimental results to the main paper along with the visualization results.

Thanks for posing this question and sorry for our typos. The columns represent CLIP Score and FID, respectively.

Thank you for taking the time to respond. For Q3, what do the two columns of metrics represent respectively?

Q1: The work does not show comparisons with these state-of-the-art methods in terms of text-image alignment and image quality.

Thank you for the suggestion. We have further supplemented the quantitative comparison experiments. Due to time constraints, we only compared Lumina-Next with representative SoTA T2I models, SD3 and PixArt. To better illustrate the aesthetics and semantic ability, we conduct an AI preference study to evaluate Lumina-Next against other text-to-image models, since conventional metrics such as FID and CLIP-Score may not accurately reflect the generation quality. Following PixArt, we employ GPT-4o, the SoTA multimodal LLM exhibiting strong alignment with human preference, as our evaluator to vote based on image quality. As shown in the following table, Lumina-Next demonstrates competitive performance with advanced text-to-image models including PixArt and SD3. Note that SD3 uses over 1B text-image pairs, which is ~100x greater than our models, PixArt, which is already a training-efficiencient model, still uses 3x training compute than ours. However, we have to admit that Lumina-Next still underperforms these SoTA models in terms of text-image alignment and compositional generation, due to inadequate data and training.

Q2: The paper does not demonstrate the manifestation and properties of scaling laws in the generation domain.

Good point! The nature of good scaling up and the consequent emergence of modeling capabilities are essential properties for evaluating GenAI nowadays, which has been explored in Lumina-T2X -- scaling up DiT from 0.6B to 7B. However, in this paper, since the scale of data we use is much smaller than the leading open-source models (e.g., 30M in Lumina-Next V.S. over 1B in SD3), the benefits of further scaling up the model to more parameters are not significant. Therefore, we adopt a 2B model to achieve a sweet point between performance and efficiency. The focus of this paper is more on how to make the DiT-based models achieve more stable training, higher inference quality and speed. In addition, we would like to highlight the additional contribution presented in our paper, as detailed in the Appendix. These include (1) zero-shot multilingual generation, (2) flexible visual recognition, and (3) text-to-multiview/music/audio/point clouds generation, all of which are yet to be explored for flow-based diffusion transformers.

Q1: Quantitative evaluations on all generation tasks.

Thank you for the suggestion. We have further supplemented the quantitative comparison experiments. Due to time constraints, we only compared Lumina-Next with representative SoTA T2I models, SD3 and PixArt. To better illustrate the aesthetics and semantic ability, we conduct an AI preference study to evaluate Lumina-Next against other text-to-image models, since conventional metrics such as FID and CLIP-Score may not accurately reflect the generation quality. Following PixArt, we employ GPT-4o, the SoTA multimodal LLM exhibiting strong alignment with human preference, as our evaluator to vote based on image quality. As shown in the following table, Lumina-Next demonstrates competitive performance with advanced text-to-image models including PixArt and SD3. Note that SD3 uses over 1B text-image pairs, which is ~100x greater than our models, PixArt, which is already a training-efficiencient model, still uses 3x training compute than ours. However, we have to admit that Lumina-Next still underperforms these SoTA models in terms of text-image alignment and compositional generation, due to inadequate data and training.

Q2: Extra apples-to-apples evaluations on any resolution recognition.

Thanks for pointing this out! The lack of any resolution fine-tuning could indeed lead to unfair comparisons. To address this, we have added experiments with and without any resolution fine-tuning for any resolution inference. We also included two additional comparison methods: PVT-Base and Swin-Base. The experimental results show that under completely consistent settings, Next-DiT still achieves better performance over Deit-Base and PVT-Base, demonstrating the versatility of Next-DiT as a representation learning network for arbitrary resolutions. However, compared to Swin-Base, an advanced architecture with complicated architecture design for visual recognition tasks, Next-DiT still has room for improvement, objectively highlighting the superiority of specialized models in dedicated tasks. We have supplemented these results in the revised version of our paper.

Q3: Comprehensive quantitative ablations on the proposed technical extensions, a) 3D RoPE, b) extra normalization, c) frequency- and time- aware scaled RoPE, d) improved time scheduling and higher-order ODE solvers for reverse sampling, etc.

Thanks for the advice! Due to time constraints, we are unable to provide all independent ablation experiments for 3D RoPE and additional normalization here. However, our results in Fig. 6 demonstrate that Next-DiT, the combination of these aforementioned techniques, achieved an overall improvement on the ImageNet benchmark compared to the Flag-DiT used by Lumina-T2X. Additionally, In the table below, we supplement quantitative ablation studies on the Lumina-Next T2I model to verify the effectiveness of our resolution extrapolation and inference acceleration techniques. We have added these results to our main paper.

Q1: There is a lack of quantitative experiment comparison results on some Text2Image benchmarks to demonstrate the superiority and effectiveness of the proposed method.

Thank you for the suggestion. We have further supplemented the quantitative comparison experiments.

Due to time constraints, we only compared Lumina-Next with representative SoTA T2I models, SD3 and PixArt. To better illustrate the aesthetics and semantic ability, we conduct an AI preference study to evaluate Lumina-Next against other text-to-image models, since conventional metrics such as FID and CLIP-Score may not accurately reflect the generation quality. Following PixArt, we employ GPT-4o, the SoTA multimodal LLM exhibiting strong alignment with human preference, as our evaluator to vote based on image quality. As shown in the following table, Lumina-Next demonstrates competitive performance with advanced text-to-image models including PixArt and SD3. Note that SD3 uses over 1B text-image pairs, which is ~100x greater than our models, PixArt, which is already a training-efficiencient model, still uses 3x training compute than ours. However, we have to admit that Lumina-Next still underperforms these SoTA models in terms of text-image alignment and compositional generation, due to inadequate data and training.

In addition to the ImageNet experiments in our paper, we also conduct various ablation studies to verify the effectiveness of each proposed component.

Q1: Incremental Changes.

Good point! We agree that Lumina-Next is largely based on Lumina-T2X with several improvements. However, we would like to highlight that these changes are made after comprehensive examinations and are essential for scaling this flow-based diffusion transformer or applying it in diverse fields. For example, we discover the exploding network activation and then propose to use sandwich normalization, which effectively eliminates the instability during both training and inference. In addition, we would like to highlight the additional contribution presented in our paper, as detailed in the Appendix. These include (1) zero-shot multilingual generation, (2) flexible visual recognition, and (3) text-to-multiview/music/audio/point clouds generation, all of which are yet to be explored for flow-based diffusion transformers.

Q2: The advantage of 3D rope.

Well said! The current implementation of 3D RoPE is almost equivalent to normal 2D RoPE + temporal positional embedding. We use 3D RoPE to continue one of the excellent features of Lumina-T2X — a unified framework for different modalities. For example, we adopt the 3D RoPE for multiview generation as a natural extension, since it can be treated as a sequence of image frames. The high-quality and consistent multiview results shown in Figure 25 verified the effectiveness of 3D RoPE in modeling 3D signals. Due to current limitations in training resources and data availability, Lumina-Next is still a unified multimodal framework with independently trained models. However, we consider a unified multimodal generative model as a necessary future step, so this unified framework can serve as an essential foundation for the next stage.

Q3: The ImageNet experiments can hardly justify the improvements.

Sorry for the confusion. Our comparison in Figure 5 is completely fair — with the same parameter settings and model sizes. Besides, the results from the ImageNet benchmark are also widely regarded as consistent when model scaling up and are commonly used as a preliminary validation test to reduce trial-and-error costs, e.g., EDM [3] conducts extensive experiments on ImageNet to gain valuable insights, which are further proven to be transferrable to various scales and fields.

[3] Karras, Tero, et al. "Elucidating the design space of diffusion-based generative models." Advances in neural information processing systems 35 (2022): 26565-26577.

Q4: Lack of explanation about model size.

Thanks for posing this question. Since the scale of data we use is much smaller than some open-source models (e.g., 30M in Lumina-Next V.S. over 1B in SD3), we found that, given the current data scale, the benefits of further scaling up the model to more parameters are not significant. This aligns with the finding in Lumina-T2X, where the 3B and 7B models achieved similar results. To achieve a sweet point between performance and efficiency, we adopted a 2B model, making it more accessible for contributors to run Lumina-Next on their personal devices. We have added more explanation in the revised version of our paper.

Q5: The adoption of Gemma.

Please allow us to clarify that we chose Gemma 2B mainly for better efficiency, and it is not considered as one of our contributions. Actually, Gemma 2B is weaker than LLaMA 7B used in Lumina-T2X. The influence of using different text encoders is valuable and worth exploring. However, this is not the major focus of our paper. Besides, the effectiveness of our core contributions has been well validated as follows: (i) the experiments on the ImageNet benchmark demonstrate the improvements of Next-DiT over Flag-DiT, and (2) fair experiments on the Lumina-Next T2I model have confirmed the effectiveness of our resolution extrapolation and inference acceleration techniques. We have further supplemented the quantitative results of ablation studies here.

Q6: Lack of quadratic evaluation metrics.

Thank you for the suggestion. Due to time constraints, we only compared Lumina-Next with representative SoTA T2I models, SD3 and PixArt. To better illustrate the aesthetics and semantic ability, we conduct an AI preference study to evaluate Lumina-Next against other text-to-image models, since conventional metrics such as FID and CLIP-Score may not accurately reflect the generation quality. Following PixArt, we employ GPT-4o as our evaluator to vote based on image quality. As shown in the following table, Lumina-Next demonstrates competitive performance with advanced text-to-image models including PixArt and SD3. Note that SD3 uses over 1B text-image pairs, which is ~100x greater than our model, PixArt, which is already a training-efficiencient model, still uses 3x training compute than ours. However, we have to admit that Lumina-Next still underperforms these SoTA models in terms of text-image alignment and compositional generation, due to inadequate data and training.

Q7: What about simply using grad clip norm.

Good point! We have tried using techniques like gradient clip, but they are not that effective in preventing model training from crashing. This finding has also been mentioned in other papers such as ViT-22B [4]. We have included these empirical findings in the paper, hoping to provide some helpful insights to the community.

[4] Dehghani, Mostafa, et al. "Scaling vision transformers to 22 billion parameters." International Conference on Machine Learning. PMLR, 2023.