Boosting Generative Image Modeling via Joint Image-Feature Synthesis

We appreciate your insightful comments and efforts in reviewing our manuscript. Below, we provide our responses to each of your comments:

W1. Lack of some analytical experiments

• FID change curve during training.

We respectfully emphasize that in Figure 2, we demonstrate the progression of FID during training for both DiT-XL w/ ReDi and SiT-XL w/ ReDi, and compare with the vanilla DiT/SiT and REPA.

• Visualization of the "synthesized" representations sampling process.

We appreciate the reviewer’s suggestion. Because the NeurIPS 2025 rebuttal guidelines do not allow new figures to be uploaded at this stage, we cannot include the requested visualization in the rebuttal. We will, however, add a figure illustrating how ReDi’s synthesized representations evolve over the course of inference in the camera‑ready version.

• Comparison of the "synthesized" representations between REPA and ReDi.

We appreciate the reviewer’s suggestion to include a comparison between the features generated by ReDi and REPA. We will include a visual comparison of the generated features from both methods in the final version of our paper.

W2. Alternative Representations

We thank the reviewer for their suggestion to explore alternative representations. Although we selected DINOv2 for its state‑of‑the‑art performance across a wide spectrum of vision tasks, we agree that evaluating other representations can provide additional insight.

To address this concern, we conducted some preliminary supplementary experiments using 2 alternative representations, namely MAE and JEPA, with different number of principal components (#PC). The additional experiments suggested by the reviewer, such as LLM embeddings corresponding to prompts and representations from a REPA, are not trivial to implement within the short rebuttal timeline. However, we agree that these would be interesting to explore, and we plan to include such experiments in the final version of our paper.

For all experiments, we use SiT-B/2 as the base model and train for 400K steps. We observe that using DINOv2 outperforms the other self-supervised encoders. This result is consistent with the ablation conducted by REPA.

W3. A more in-depth explanation regarding Figure 7 (Principal Components Ablation)

We agree with the reviewer that further exploration is needed to better explain the performance degradation observed when increasing the number of DINOv2 PC's. To support our claim that this degradation occurs due to the model disproportionately allocating capacity to predict the visual representations at the expense of image latents, we conduct two additional experiments.

One experiment with PC=16 and $\lambda_z=0.5$ and additionally an experiment where we use the 8 components after the first 8 PCs i.e. $z[8:16]$. We highlight these new experiments with bold in the table below:

These results show that although the 8 components after the first 8 PCs ($z[8:16]$) contain useful information and match the performance of the first 8 PCs ($z[0:8]$), modeling both together negatively impacts performance. This holds true even when we adjust the loss weight to $\lambda_z=0.5$. These experiments support our explanation for the performance degradation observed as the number of DINO channels increases.

We appreciate your insightful comments and efforts in reviewing our manuscript. Below, we provide our responses to each of your comments:

W1.1 Quantitative analysis of the generated DINO features

We appreciate the reviewer’s suggestion to demonstrate the discriminative capabilities of ReDi's generated features. Although the primary aim of this work is to improve generative image modeling, we agree that this is an interesting experiment.

Since we operate on only the first 8 Principal Components (PC) of DINOv2 features, both ground-truth and generated representations have reduced discriminative capabilities, making linear probing less effective. Instead, we use Attentive Probing [1], which is more suitable for this low-dimensional setting.

Our evaluation setup is the following: We first train an Attentive Probing Head on 8-PC DINOv2 features from ImageNet. Then compare the classification accuracy on 50K (50 for each ImageNet class) generated samples between:

• Ground-Truth Features: PCA-reduced features extracted from generated images with DINOv2.
• Generated Features: Features directly generated by ReDi.

We employ SiT-XL w/ ReDi trained for 4M iterations and use Classifier-Free Guidance as in Table 2, to generate the samples.

We note that the top-1 accuracy is high for both the ground truth and the generated features, since they are produced in a class-conditioned setting with CFG. This pushes the generated samples to align with the condition, thereby limiting diversity. As such, these results are not directly comparable to those typically reported in representation learning papers, such as DINOv2.

The results demonstrate that ReDi’s generated features largely preserve the discriminative capacity of the ground truth DINOv2 features extracted from generated images.

W1.2 Visual comparison between ground truth and generated DINO features

We appreciate the reviewer’s suggestion to include a visual comparison between ground truth and generated DINO features. Because the NeurIPS 2025 rebuttal guidelines do not allow new figures to be uploaded at this stage, we cannot include the requested visualization in the rebuttal. We will include the suggested visual comparison in the final version of our paper.

W2. Higher Resolution (512 x 512)

We agree with the reviewer that it would be interesting to validate the effectiveness of ReDi in improving the generative quality of higher-resolution images. However, due to our limited computational resources and the limited time frame of the rebuttal period, we were unable to conduct an experiment on ImageNet512. However, we plan to include such experiments in the final version of our paper.

Q. How about the speed?

We respectfully emphasize that we have explicitly measured the sampling speed of ReDi in terms of throughput (generated images per second) and reported the results in Table 5. The Merged Tokens strategy (our standard approach), which preserves the original sequence length, achieves a throughput of 4.51, while Vanilla DiT-B/2 achieves 4.52. This demonstrates that ReDi (MR) maintains the same computational cost (99.78%) as Vanilla DiT-B/2.

References

[1] Scalable pre-training of large autoregressive image models. In ICML'24

We appreciate your insightful comments and efforts in reviewing our manuscript. Below, we provide our responses to each of your comments:

"A little improvement for class-conditional generation"

We respectfully clarify that ReDi's improvements on class conditional generation are significant both with and without CFG, even though we agree with the reviewer's assessment that "there's little room for improvement in the first place for ImageNet256" in the CFG setting. We summarize the performance gains over the baseline (SiT) and REPA in what follows:

As presented in Table 2 ReDi reaches an FID of 1.72 at 350 epochs, surpassing both the baseline SiT and REPA trained for 1400 and 800 epochs, respectively. These results show that even in the CFG setting, ReDi accelerates convergence by > x4 over SiT and by > x2 over REPA.

W1. Experimental setting limited to ImageNet256

We appreciate the reviewer’s concern regarding the applicability of our method to other datasets. To address this, we conducted an additional experiment on Food101 dataset [1]. While this serves as a preliminary result in a smaller-scale setting, it provides valuable empirical evidence that ReDi’s significant improvements are not limited to ImageNet256.

Due to our limited computational resources and the short timeframe of the rebuttal period, we were unable to conduct a large-scale experiment, e.g., on ImageNet512. However, we plan to include such experiments in the final version of our paper.

W2-W3. Alternative representations

We thank the reviewer for their suggestion to explore alternative representations. Although we selected DINOv2 for its state‑of‑the‑art performance across a wide spectrum of vision tasks, we agree that evaluating other representations can provide additional insight. As such, we have already highlighted the potential for evaluating a combination of different representations in the "Limitations and Future Work" section (Appendix D).

To address this concern, we conducted some preliminary supplementary experiments using 2 alternative representations, namely MAE and JEPA, with different number of principal components (#PC). Given the time constraints of the rebuttal period, a more thorough exploration was not possible. However, we plan to include additional alternative representations, such as CLIP, SIGLIP, and MOCOv3, in the final version of our paper.

For all experiments, we use SiT-B/2 as the base model and train for 400K steps. We observe that using DINOv2 outperforms the other self-supervised encoders. This result is consistent with the ablation conducted by REPA [2].

W4. Engineering a solution

We respectfully disagree with the reviewer's assessment of our method as "an engineering solution for natural image generation".

Modeling the joint distribution of precise low-level features (via the VAE latents) and semantic high-level features (via DINOv2) is well motivated by the recent literature [2] that established a connection between representation learning and generative performance.

Further, we emphasize that we did not apply any engineering tricks during training (e.g., adjusting LR, optimization parameters, etc.) nor any changes in the standard diffusion transformer architecture (e.g., RMSNorm [3], SwiGLU [4], and RoPE [5]). Thus, our work presents a fundamental finding and not an engineering solution: integrating image latents and semantic representations within one probabilistic model demonstrably simplifies and improves diffusion‑based generation.

References

[1] Food-101–mining discriminative components with random forests. In ECCV, 2014

[2] Representation alignment for generation: Training diffusion transformers is easier than you think. In ICLR, 2025

[3] Root mean square layer normalization. In NeuRIPS 2019

[4] Glu variants improve transformer. In Arxiv 2020

[5] Roformer: Enhanced transformer with rotary position embedding. Neurocomputing 2024

We sincerely appreciate the reviewer's thoughtful questions about the fundamental mechanisms behind our approach's success. While these open-ended inquiries deserve careful consideration, providing comprehensive answers within the rebuttal period is difficult. For now, we share below some additional experiments, our current perspectives and note that these questions inspire valuable future research directions - particularly a deeper theoretical or mechanistic analysis of the ReDi's joint diffusion mechanisms.

W1.1 Why and how joint modeling leads to improvements.

We agree with the reviewer's assessment that the significant performance improvement demonstrated by ReDi raises interesting research questions regarding the underlying mechanisms of our method.

Both our work and REPA are motivated by the observation that diffusion models tend to learn features that lack the depth and versatility seen in self-supervised approaches like DINOv2. Our hypothesis is that this limitation arises because the denoising objective does not differentiate between meaningful semantic information and irrelevant image details.

REPA showed that aligning the DiT representations with DINOv2 can mitigate the deficiencies of the denoising objective, enabling the model to learn stronger representations. This resulted in significant improvements, indicating that learning high-level semantic representations is essential for generative modeling.

This observation directly motivates our joint modeling approach. Rather than relying on distillation, ReDi simultaneously learns stronger semantic representations by learning to denoise the first principal components of DINOv2, which encode the high-level semantic information of the image.

W1.2 Loss weight and #PC

We thank the reviewer for their suggestion to further explore a more optimal weighting between ReDI's VAE and DINOv2 denoising losses. We agree that such an exploration is an interesting direction that we plan to do in future research. As an initial step in this direction, we conduct an experiment with PC=16 and $\lambda_z=0.5$. Additionally, we conduct an experiment where we use the 8 components after the first 8 PCs i.e. $z[8:16]$. We highlight these new experiments with bold in the table below:

These results show that although the 8 components after the first 8 PCs ($z[8:16]$) contain useful information and match the performance of the first 8 PCs ($z[0:8]$), modeling both together negatively impacts performance. This holds even when we adjust the loss weight to $\lambda_z=0.5$. While we agree that a deeper exploration is necessary, this preliminary result supports our explanation for the performance degradation observed as the number of DINO channels increases.

W1.3 Different DINO channels may encode fundamentally different types of information.

We once again thank the reviewer for their suggestion to explore which principal components (PCs) are most beneficial for ReDi. Indeed, different PCA components encode different types of information. Specifically, the first PCs capture low-frequency global structures, while the last PCs capture high-frequency details of DINOv2. Thus, we agree that determining whether low-frequency, high-frequency, or a combination of both is most beneficial for joint modeling is an interesting question.

In this direction, we conducted two additional experiments (in bold) where we fixed the number of PCs to 8 and varied their position, with the 1st PC indexed at 0:

These results indicate that the low-frequency semantic information is the most beneficial for ReDi.

W2. Why Representation Guidance (a form of self-guidance) is effective?

This is indeed an interesting observation, our intuition is that Representation Guidance (RG) can be viewed through the same framework as Classifier‑Free Guidance (CFG):

• In CFG $\nabla \log p(x_t|c) -\nabla \log p(x_t)$ acts as the gradient of an implicit classifier (amplifying the likelihood of $p(c | x_t)$), producing images that follow the condition $c$.
• In RG $\nabla \log p(x_t,z_t) -\nabla \log p(x_t)$ acts as the gradient of an implicit feature extractor (amplifying the likelihood of $p(z_t | x_t)$), producing images that follow the (noisy) semantic structure $z_t$ at each step $t$.

We will highlight this connection more explicitly in the final version of the paper.

Q1. Why and how jointly modeling “more stuff” (like DINO, which should make the task harder) actually leads to faster convergence?

More generally, jointly modeling multiple modalities has been systematically shown to benefit image and video generative models, offering strong scalability properties. We discuss such methods in the Multi-modal Generative Modeling paragraph of our Related Work section.

Specifically for ReDi, we have already analyzed our hypothesis for the observed performance gains in response to W1.1.

Q2. Would jointly modeling the class label outperform DINO?

We believe that the reviewer raises an interesting question. As an initial experiment, we attempted to encode the class label as a one-hot vector and jointly generate it as a separate token. However, we observed that this straightforward implementation is not successful. Specifically, the model was unable to learn the discrete "class label" distribution (potentially due to the denoising objective), and the training loss increased over time rather than converging. Still, we consider this an interesting direction, and further experimentation beyond the rebuttal period is needed.