However, the paper presents a significant methodological oversight—it appears as a mere combination of existing MAE (Masked Autoencoder) and MAR (Masked Autoregressive Model) methods without a critical analysis or ablation study showing how the MAE branch impacts the AR branch. Specifically, the absence of experiments where the MAE branch is removed casts doubt on the incremental benefit of this integration.

D-JEPA is not based on the MAE structure; it is an improvement upon the JEPA structure, both utilizing the same ViT structure as the backbone. The effectiveness and design details of JEPA have been well-validated in existing literature, such as in "I-JEPA: Self-Supervised Learning from Images with a Joint-Embedding Predictive Architecture." The primary improvement in D-JEPA is the integration of the generative capabilities of diffusion models into JEPA, enabling it to handle generative tasks.

The representation learning capability of D-JEPA is also questionable. According to Table 7, the best ImageNet accuracy is obtained when the model is trained on pixel-level data, while the generative experiments focus on a D-JEPA trained on VAE latent space. This discrepancy points to a lack of a truly unified model approach, as the performance seems to depend heavily on the training specifics rather than the model architecture itself.

For discussions on the representation learning capability of D-JEPA, please refer to the analysis provided here. As analyzed in Appendix E, the suboptimal representation learning capability of D-JEPA in latent space is primarily due to the VAE's inability to effectively encode images after data augmentation. This results in D-JEPA performing better in pixel space than in latent space, and our experimental results support this hypothesis. Additionally, similar analyses were conducted in MAGE.

Could the authors provide an ablation study that isolates the MAE branch to elucidate its specific impact on the AR branch's performance?

For an ablation study that isolates the impact of the JEPA branch on generative performance, please refer to the discussion here. In summary, the JEPA branch in D-JEPA effectively enhances the model's convergence speed.

How does the model's performance on representation learning tasks vary between training in pixel space and latent space, and what does this indicate about the model's ability to serve as a unified architecture?

D-JEPA performs better in pixel space than in latent space, likely due to the VAE's limitations in effectively encoding images after data augmentation. Currently, the core focus of this work is to address generative capabilities in multimodal models, while representation learning, which leans more towards understanding, is not within the scope of this work. However, our recent work has validated D-JEPA's effectiveness in constructing unified multimodal generative and understanding models, where multimodal joint training can significantly overcome these limitations.

It’s interesting to see from Tables 1 and 2 that D-JEPA has more parameters than MAR. I’m curious—shouldn’t these models have similar architectures? Can the authors explain why D-JEPA is larger? What additional components does D-JEPA include that MAR doesn't?

We provided a detailed explanation here. In summary, D-JEPA adopts the standard ViT B/L/H structure, whereas MAR uses a non-standard ViT B/L/H structure, leading to discrepancies in model parameters between the two.

Thank you to the authors for the clarification. I now have a better understanding of the method and architecture. However, I still have some concerns:

Ablation Study

I could not find the experiments I previously requested, where the I-JEPA branch is removed and MAR branch is evaluated in isolation, keeping the other design aspects unchanged. The link provided by the authors directs to the current page, which does not include experiments related to this specific ablation.

Representation learning

In the rebuttal, the authors state:

Currently, the core focus of this work is to address generative capabilities in multimodal models, while representation learning, which leans more towards understanding, is not within the scope of this work.

However, this claim contradicts the content of the paper. The authors explicitly position their work as aiming to unify multimodal models and emphasize balancing representation learning and diffusion learning. For example, on Line 980, the paper states:

Our work uniquely verifies the mutual benefits of representation learning and generative modeling

This demonstrates that representation learning is indeed within the scope of the work, as evidenced by the extensive experiments in the appendix. The rebuttal appears inconsistent with the paper’s claims.

It would be interesting to have generation results at higher resolution than 256x256 px to see if inference speed suffers when bi-directional attention meets lots of tokens.

We have showcased the generation results of 1024x1024 images here or you can click here. Some quantitative metrics are presented here. During our experiments, we observed that bi-directional attention does indeed lead to a significant decrease in inference speed when generating a large number of tokens. However, we can mitigate this issue by using a next set-of-tokens prediction approach, which allows us to generate multiple tokens at once, thereby greatly reducing the number of inference steps needed for image generation. In our given samples, we were able to generate high-quality 1024x1024 images using only 128 autoregressive steps.

The organization and the structure of the current version should be improved. The writing of chapter 3 is very confusing for me. And some important results and discussions should be re-located in the main paper not the appendix.

Thank you for your suggestions on improving the structure of the paper. We will make the necessary adjustments in the revision version. Here, I would like to provide some additional clarification regarding Chapter 3: Since our work is highly related to MAR and JEPA, we provided a brief introduction to these works in Chapter 2 (Background). I believe this background information will help readers better understand the content of Chapter 3. As mentioned in the introduction, the core focus of this paper is to demonstrate the importance of representation learning in generative tasks, rather than showcasing that D-JEPA can achieve the best results in both representation learning and generative tasks. Therefore, we emphasize D-JEPA's performance in image generation in the experiments section, while its performance in representation learning is presented in the appendix.

The novelity is limited: It seems that this work just replace the MAGE parts of MAR. The simple combination of I-JEPA and MAR's dfiffusion parts. It doesn't solve the core issue between the gap of representation learning and generative modeling. Such an archecture cannot bring huge improvement on representation learning with latent-level input (Results in AppendixE over down-stream tasks are good, just for pixel-level inputs), and the corresponding performance are very poor. Could you please provide the similar settings for MAR? Use both the pixel-level and latent-level inputs to check the representation performance? I think your design with JEPA cannot solve the inherent limitation (latent input -> bad representation results) of such an TOKEN learning + diffusion framework.

Firstly, D-JEPA is not aimed at unifying representation learning and generative tasks, but rather at serving as an effective framework to unify various continuous modality generation tasks, such as image, audio, and video generation. Therefore, we showcase D-JEPA's generative performance across different modalities in Appendix G. The experiments on D-JEPA's representation tasks in the appendix are supplementary, demonstrating that D-JEPA can also perform reasonably well in representation tasks. Its performance in representation learning still requires further exploration, which is beyond the scope and contribution of this paper.

Secondly, regarding the comparison between MAGE and D-JEPA, MAGE does not mention any concepts related to JEPA and is based on contrastive loss for feature learning, whereas D-JEPA uses L1 loss. These are fundamentally different approaches, and it is not rigorous to confuse them due to architectural similarities.

Lastly, concerning the performance of D-JEPA and MAGE in representation learning, especially in the latent space, we provided a detailed analysis in Appendix E. We attribute the suboptimal performance primarily to the pretrained VAE model's unsuitability for handling augmented image data. To support this hypothesis, we provided experimental results in the pixel space. The results (Tab 7 and Tab 8) indicate that D-JEPA trained on raw pixels can be comparable to mainstream representation learning methods.

To reiterate, the core of D-JEPA is not to unify representation and generative models; achieving optimal results in representation tasks is not within the research scope of this work.

Please provide the exps on Imagenet-512x512 to compare with SOTAs in a more comoutation situation.

To better showcase the effectiveness of D-JEPA, we provide the experimental results of D-JEPA-L on ImageNet-512:

Additionally, we have showcased the text-to-image generation results of D-JEPA at 1024 resolution here.

How about the D-JEPA-H linear and finetuning performance in Table 7/8.

We did not conduct experiments on D-JEPA-H for representation learning because D-JEPA-H is based on ViT-H. Current representation learning tasks typically do not use ViT-H as a backbone, mainly opting for ViT-B and ViT-L. Therefore, to ensure a fair comparison, we presented the results for D-JEPA-B and D-JEPA-L.

It is unfair to compare Huge scale models in Table1 and 2. The parameters of the proposed D-JEPA is not aligned with MAR-H.

For a fair comparison between D-JEPA-H and MAR-H, please refer to the detailed discussion here. In short, the D-JEPA-L model, with only 687M parameters and 480 epochs of training, already surpassed MAR-H (943M parameters) with 800 epochs of training (FID 2.32 vs. 2.35). This sufficiently demonstrates that D-JEPA outperforms MAR.

Line 200： “It is important to note that training in latent space is not a necessity for D-JEPA; it can be trained in raw space and still achieve excellent results.” WHERE is the corresponding generative results？ I haven't found it.

Since our method description and experimental results throughout the paper are based on latent space, the experimental results in Appendix E are conducted in raw pixel space. The note in L200 is primarily to prevent reader confusion, ensuring they do not mistakenly believe that D-JEPA can only be trained in latent space. As for generative results in raw space, we did not evaluate them but instead demonstrated their representation task performance in Appendix E.

The core of I-JEPA is the small-block-wise masks to perform the feature-level augmentation to achieve the better linear performance without the help of data-level augmentation. But in your work, the random mask strategy with high ratio is applied. Could you please provide some empirical evidence and intuitions about such a design？

Firstly, I-JEPA does not use small-block-wise masks. The masked region is an irregular polygon, covering up to 80% of the area. In D-JEPA, we randomly mask tokens as the smallest unit, with a mask ratio between 0.75 and 1.0. Thus, there is no significant difference in the masked area between the two. Regarding the choice of mask ratio in D-JEPA, we mainly followed the configurations of MAGE/MAR/MAE. Additionally, during experiments, we found that maintaining a relatively high mask ratio significantly reduces computational load and improves convergence speed. Therefore, we adhered to the configurations of previous works.

After detailed consideration, i am leaning to give the score of 5. Reasons:

1. The newly added ablation studies (Table 6) present that there is no obvious improvements after adding the JEPA loss. And more importantly, neither CFG or IS results are provided.

2. Even though the 512x512 results of D-JEPA in Table8 are good, the parameters of D-JEPA-L is much more than MAR-L. And the detailed training epochs are not provided (I guess D-JEPA 480 and MAR 800).

3. I would appreciate the fair comparisons with MAR series, with same parameters and same epochs, not current 1/3 more parameters and 1/2 less training epochs. Such an comparison is unfair.

The total representation results in my rating of 5.

Dear Reviewer dQp1,

I hope this message finds you well. On behalf of our research team, I would like to express our sincere gratitude for the time and effort you have dedicated to reviewing our manuscript. Your insightful comments and constructive feedback have been invaluable in guiding our revisions and improving the clarity and quality of our work.

We have carefully addressed each of your comments and have made detailed revisions to the manuscript, particularly in the introduction and experimental analysis sections, to ensure clarity and comprehensiveness. We hope that these changes effectively address your concerns and enhance the overall presentation of our research.

We kindly invite you to review the revised version of our manuscript. If there are any remaining concerns or additional feedback, please do not hesitate to reach out. We are eager to engage in further discussions and make any necessary adjustments to meet the high standards expected by the committee.

Thank you once again for your invaluable input and for considering the revisions we have made. We hope that our efforts will encourage you to reassess the manuscript favorably.

Warm regards

Dear Reviewer dQp1,

I hope this email finds you in good spirits. We are grateful for your detailed evaluation and have worked hard to address your concerns. With the discussion deadline fast approaching, we would appreciate it if you could revisit the manuscript to see if the adjustments align with your expectations. Your expertise continues to play a pivotal role in refining our work.

Thank you once again for your dedication.

Sincerely,

Authors

How does the model perform in unconditional generation tasks or on more complex datasets, such as COCO?

For performance on larger and more complex datasets, please refer to the analysis provided here and here. The D-JEPA model not only excels in complex text-to-image tasks but also performs well in multimodal tasks.

My main concerns with this paper relate to its performance compared to the similar approach, MAR. The absence of ablation studies on model components and evaluation on other datasets makes it challenging to assess the impact of each design choice.

I hope the above responses address your concerns. Overall, while D-JEPA is similar to MAR, its core lies in utilizing a representation learning architecture combined with a next-token prediction mechanism. These aspects significantly enhance D-JEPA's performance. Moreover, the design of D-JEPA significantly surpasses the performance of REPA. REPA is a recent work that also aims to integrate representation learning, but its overall performance is much lower than D-JEPA, as analyzed here.

The performance is comparable to similar approaches like MAR, which does not use the JEPA loss. In Table 1, the proposed method requires more training epochs (1400 vs. 800 for D-JEPA-B VS ) to achieve similar results to MAR-B. While D-JEPA-L/H outperforms MAR-L/H, it also involves more parameters. Similar trends are observed in Table 2.

1. Actually, our performance far exceeds MAR. For a detailed comparison and analysis regarding the parameter count and fair comparison between D-JEPA-L/H and MAR-L/H, please refer to the detailed discussion here.

2. Regarding the experiments of D-JEPA-B in Tab.1, we indeed ran for 1400 epochs, which is 600 more than MAR-B. However, the D-JEPA-B model already achieved comparable results to MAR-B at 600 epochs (FID 3.50 vs. 3.48), as shown in Fig. 6. The reason we continued to 1400 epochs was to further explore the performance ceiling of D-JEPA-B through sufficient training. The final experiments also demonstrate that while D-JEPA-B converges well at 600 epochs, its performance continues to improve with additional training.

3. Unlike D-JEPA, MAR is currently unable to achieve effective results in speech generation tasks, as analyzed here. However, the D-JEPA architecture can handle tasks such as speech and video generation. We have showcased generated speech segments on our homepage: https://d-jepa.github.io/.

There is no comparison to baseline methods, such as the effect of removing the JEPA loss.

There is indeed an ablation study in the paper, although not explicitly compared and analyzed. Here we provide the corresponding analysis. We conducted ablation experiments on the D-JEPA-B model. The results of MAR-B serve as the baseline (i.e., the results without the JEPA Loss), because MAR-B and D-JEPA-B have identical model structures and experimental parameters, making them a comparable baseline.

From Fig. 6, we can see that D-JEPA-B achieves the performance of MAR-B at 600 epochs. Additionally, from Tab.1 and Tab.2, we observe that D-JEPA-B significantly outperforms MAR-B at 1400 epochs. For a more intuitive comparison, we provide some key data points below:

These results are obtained without using classifier-free guidance, which we believe better reflects the true scenario.

Dear Reviewer gTwo,

I hope this message finds you well. On behalf of our research team, I would like to express our sincere gratitude for the time and effort you have dedicated to reviewing our manuscript. Your insightful comments and constructive feedback have been invaluable in guiding our revisions and improving the clarity and quality of our work.

We have carefully addressed each of your comments and have made detailed revisions to the manuscript, particularly in the introduction and experimental analysis sections, to ensure clarity and comprehensiveness. We hope that these changes effectively address your concerns and enhance the overall presentation of our research.

We kindly invite you to review the revised version of our manuscript. If there are any remaining concerns or additional feedback, please do not hesitate to reach out. We are eager to engage in further discussions and make any necessary adjustments to meet the high standards expected by the committee.

Thank you once again for your invaluable input and for considering the revisions we have made. We hope that our efforts will encourage you to reassess the manuscript favorably.

Warm regards

Dear Reviewer gTwo,

I hope this email finds you in good spirits. We are grateful for your detailed evaluation and have worked hard to address your concerns. With the discussion deadline fast approaching, we would appreciate it if you could revisit the manuscript to see if the adjustments align with your expectations. Your expertise continues to play a pivotal role in refining our work.

Thank you once again for your dedication.

Sincerely,

Authors

Conclusion

We sincerely appreciate the valuable comments from the reviewers. After the paper concludes the double-blind review process, we will open-source all the code and models for D-JEPA, including the code for audio, video, and multimodal generation as mentioned in Appendix G.

Regarding Model Scale and Experimental Results

Both Reviewer gTwo and Reviewer dQp1 have expressed concerns about the fairness of the experimental comparisons in Tab.1. They noted that D-JEPA-H has more parameters compared to MAR-H (1.4B vs. 943M). Consequently, although D-JEPA-H achieved superior results with just 320 epochs of training compared to MAR-H’s 800 epochs (FID 2.04 vs. 2.35), they question the fairness of this comparison.

Firstly, let us clarify the principle behind the D-JEPA model configurations at different scales. To fairly complete the representation learning experiments in Appendix E, we chose to base our work on the existing ViT-B/L/H frameworks. As shown in Table 7, existing tasks focusing solely on representation learning have been based on the standard ViT B/L/H. Thus, our D-JEPA B/L/H models respectively adopt the corresponding ViT B/L/H as the backbone, resulting in parameter sizes of 212M, 687M, and 1.4B. For the MAR series models, MAR-B also uses the ViT-B structure, but MAR-L/H do not follow the standard ViT-L/H, which restricts our ability to maintain parameter consistency.

Secondly, concerning the experiments in Table 1, it should be noted that our D-JEPA-L model already outperforms all previous works (including MAR-H) with 480 epochs of training and 687M parameters. This adequately illustrates the superiority of the D-JEPA architecture, even without considering the results of D-JEPA-H. Nevertheless, we trained the D-JEPA-H model to further demonstrate the scaling laws of the D-JEPA architecture, and unsurprisingly, it achieved even better results. Detailed analysis on this part is provided in lines 375 to 409 of the paper. Below is an excerpt from Tab 1, offering a more intuitive comparison:

Comprehensive Comparisons with Recent Methodologies

We have noticed that several highly relevant works have been published on ArXiv after the ICLR submission deadline. According to the guidelines, we are not required to compare with these works, but to fully demonstrate the superiority of the D-JEPA architecture, we have conducted a comprehensive comparison here.

The core idea of D-JEPA is to bring the advantages of representation learning into generative tasks, rather than pursuing a unified model for both representation learning and generative tasks, which Reviewer dQp1 has misunderstood. Based on this premise, REPA has made similar discoveries. REPA effectively shortened the training time of DiT by adding additional feature learning tasks. D-JEPA, on the other hand, is a novel architecture that, apart from being used for image generation, has more crucial potential roles. Firstly, it is capable of generating across various continuous modalities (something both MAR and REPA cannot achieve), including audio and video. Secondly, D-JEPA employs a Next-token-prediction approach, laying the groundwork for constructing unified multimodal large models in the future. These two key differences are at the heart of the D-JEPA work. For a better comparison with REPA, we present the results in the table below:

As seen in the table above, D-JEPA-L, with a comparable parameter size, achieves results far superior to REPA with 4M iterations using only 300K iterations. This clearly demonstrates the superiority of the D-JEPA-L architecture.

REPA: Representation Alignment for Generation: Training Diffusion Transformers Is Easier Than You Think, https://arxiv.org/abs/2410.06940

More Experiments on T2I

Recently, we have extended the T2I experiments in section G.2. We built a 2.6B text-to-image model based on D-JEPA, and on the GenEval metrics, we have surpassed the results of the highly representative diffusion model, SD3.0 (4B version) (0.65 vs. 0.64). The detailed metrics are listed in the table below:

It is noteworthy that this is the first work to surpass diffusion models in the T2I task using a next-token prediction-based architecture. This achievement lays a solid foundation for the future design of unified multimodal model architectures.