MMAR: Towards Lossless Multi-Modal Auto-Regressive Prababilistic Modeling

1. The citation format of this paper is a disaster. Many previous works are not cited when mentioned for the first time (e.g., MARS, LlamaGen, and Chamelon in Line 53, VQ-VAE in Line 76, Transfusion in Line 80, ROPE in Line 335, SiLU in Line 340, and Imagegen in Table 2); abuse of \citet and \citep (e.g., Line 109, Line141-142, Line 160-161, and Line 428), no space between the text and the reference (e.g., Line 144-146, Line 157-158, Line 173, Line 182, Line 184, and Line 187). Because there are so many wrong formats, I do not list them individually.
2. The article's logic needs to be improved. In section 3.3.1, when EmbeddingViT and VisProjector are first introduced, no explanation or citation is provided, which makes me quite confused. When introducing new terms, the authors should clearly indicate where their definitions can be found rather than making readers spend time searching throughout the entire paper. Additionally, the abbreviation CFG is used extensively without any introduction, and the authors inconsistently use both CFG and cfg (line 351). Even in Section 3.3.4, which discusses Classifier-Free Guidance, I could not find an explanation clarifying that CFG refers to Classifier-Free Guidance.
3. The assumption in Section 3.2 is unreasonable. In Section 3.2, the authors assume that $\epsilon_{\theta}(x^t, t, z_i)$ follows a standard normal distribution without providing any justification for the validity of this assumption. In fact, this assumption is unreasonable. Based on [1], the noise prediction network $\epsilon_{\theta}(x^t, t, z_i)$ models $E[\epsilon|x^t]$. As $t$ approx $0$, $\epsilon_{\theta}(x^t, t, z_i) \approx E[\epsilon|x^t]=E[\epsilon]=0$. The approximation error arises from the disparity between the model distribution and the true data distribution.
4. Notations are confusing. In Section 3.1, for autoregressive modeling, the authors use the same notation for both text and images (i.e., $p_{\theta}(x_i|x_{<i})$). This consistent notation suggests that both text and images employ a left-to-right next-token prediction modeling approach. However, in the experimental section, the modeling for images utilizes different masked autoregressive models.
5. The authors claim that this paper model the joint probabilistic $P(T, I)$. However, $P(T, I)$ is never defined. In Line 225, the authors claim that Eq. (4) is the overall loss for the image-text joint probability. Is Eq. (4) an exact likelihood for $p(T, I)$, or is it an upper bound? I do not see any convincing proof explaining the relationship between Eq. (4) and $p(T, I)$. In fact, according to the authors' notation, Eq. (4) does not reflect any interaction between text and images; it appears to model the marginal distributions of text and images separately.
6. The authors claim that this paper presents scaling-up laws of MMAR in the abstract and introduction. However, the main text does not present any scaling-up laws. The scaling laws [2,3] are established through a series of experiments that identify quantitative power-law relationships between loss or other metrics and factors such as model size, data size, and training compute. In this paper, all experiments related to scaling cannot be referred to as laws; "scaling behavior" is a more appropriate term.
7. The authors claim that v-prediction is the 'only' way to reduce the error between data distribution and model distribution in Line 113 and Line 258, such a claim is highly questionable. Possible solutions could also include adjusting optimizer parameters, using more advanced network architectures, and exploring different noise schedules. However, the authors hastily claim that v-prediction is the only solution without attempting these alternatives.
8. The evaluation of text-to-image results is insufficient. More metrics, such as Geneval[4] and T2i-bench[5], should be considered. Only zero-shot FID results are insufficient.

I strongly support the authors' motivation, but this paper is rife with flawed logic, formatting errors, unreasonable assumptions, confusing notations, overclaims, and insufficient evaluation. These factors lead me to reject this paper.

[1] Bao et al. Analytic-dpm: an analytic estimate of the optimal reverse variance in diffusion probabilistic models. ICLR2022.

[2] Kaplan et al. Scaling laws for neural language models. arXiv.

[3] Hoffmann et al. Training compute-optimal large language models. arXiv.

[4] Ghosh et al. Geneval: An object-focused framework for evaluating text-to-image alignment. NeurIPS 2023

[5] Huang et al. T2i-compbench: A comprehensive benchmark for open-world compositional text-to-image generation. ICCV 2023

• Lack of novelty. The first contribution does not look so novel as it was the main contribution of MAR[1]. The deduction of low-precision training of diffusion models, i.e., Eq. (5) & (6), looks straightforward to me while I do not think it could be a main contribution.
• The proposed two-stage training strategy is not proven to outperform existing state-of-the-art MLLM works according to the experiments on visual understanding and visual generation. Generally, the MMAR just performs the second best when compared to other joint image-text probabilistic models.

[1] Li, Tianhong, Yonglong Tian, He Li, Mingyang Deng, and Kaiming He. "Autoregressive Image Generation without Vector Quantization." arXiv preprint arXiv:2406.11838 (2024).

1. The paper is a simple extension of MAR and is of insufficient novelty.

2. I am not convinced by the paper's claim that discretization introduces a bottleneck and that using continuous probabilistic models is necessary for image modeling in multimodal modeling due to the following reasons.

i) A unified representation and network structure for multimodal modeling is simpler and easy to scale.

ii) It is hard to claim introducing continuous image representation can outperform purely discrete representations. It seems that purely discrete representations have not yet reached their full potential. For example, EMU3 [1] uses purely discrete representations and achieves state-of-the-art performance in many areas.

iii) Data is inherently discrete in computers.

3. Current experimental results in the paper are not strong enough to support the main claim. For example, does the MMAR beat domain-specific generative model such as stable diffusion like EMU3? The generation results in the appendix seems to be of low quality.

[1] Wang, Xinlong, et al. "Emu3: Next-token prediction is all you need." arXiv preprint arXiv:2409.18869 (2024).

• I’m not convinced by the argument in the introduction that the proposed MMAR, which uses masked modeling, is fundamentally better suited for image understanding tasks than Transfusion, which uses diffusion on the full image. Specifically, the authors argue that full image diffusion hurts understanding because the model is trained on noisy images (with varying levels of noise). However, MMAR is also trained on noisy images when modeling P(image|text), just instead of additive noise, the noise comes from dropped tokens. Furthermore, when modeling P(text|image) both Transfusion and MMAR can operate on noiseless and fully unmasked, respectively.
• The text-to-image quality of MMAR is somewhat weak compared to the image understanding quality. It would be good to see why this is, e.g. because the model relies on a pretrained LLM which is a adapted with LoRA. A image generation-only baseline, potentially trained from scratch, could elucidate this question.
• While the error/stability analysis of the n vs. v-parametrization makes sense to me, the authors never experimentally show that going from high-precision training to low precision training actually leads to issues. Would it be possible to show some training curves?