PUMA: Empowering Unified MLLM with Multi-granular Visual Generation

1. The novelty of this paper seems limited; although it claims to introduce multi-granular visual generation, many papers (e.g., SEED-X[1], Matryoshka[2]) already focus on multi-granularity visual features, both in MLLM and image generation. Furthermore, the implementation of multi-granularity merely involves using different pooled visual features and different diffusion decoders, which also introduces additional parameters and computation.
2. In Table 1, is PUMA using (1+4+16+64+256) visual tokens? This number is much larger than in previous works. When using the same number of visual tokens, PUMA appears to underperform compared to other methods.
3. In Figure 2, the input and output of the PUMA Multimodal Generative Model are visual features covering different levels of visual granularity. I wonder whether the model can perform single granularity image generation during the inference stage (or human-specified multi-granularity).
4. In Tables 3 and Table 4, the authors should indicate how many visual tokens each model uses.

[1] Ge, Yuying, et al. "Seed-x: Multimodal models with unified multi-granularity comprehension and generation." arXiv preprint arXiv:2404.14396 (2024). [2] Cai, Mu, et al. "Matryoshka Multimodal Models." arXiv preprint arXiv:2405.17430 (2024).

The simple design will lead to a model that's slow in inference time: a), the image feature sequence length is essentially doubled with a lot of redundant information. Leading to a larger context window in inference time, and 2x inference cost as the output sequence length will be doubled for visual generation tasks. b) several images has to be generated, each from a different diffusion decoder for the corresponding feature granularity. In the end, only one of these outputs will be used as the final output.

The result final model size will be much larger as well since there will be several SDXL UNet checkpoints, each is around 2B parameters.

In experiments, the paper fine-tunes separate MLLMs for different visual-linguistic tasks. This setup is inconsistent with the problem formulation, where one MLLM has to balance the needs of several tasks.

The authors did not discuss how to select the feature scale for final output. This is vaguely discussed in Appendix D. It should be more clearly discussed as it is an important part of the entire generation flow.

• The main concerns is that the paper does not fulfill what seems to be its main promise: To have a single integrated model that can perform generative and understanding tasks. The models that are evaluated for each of the four tasks are different model checkpoints that are produced by targeting each downstream task specifically through different finetuning datasets.
• I also have some concern with not including FID (Frechet Inception Distance) as is customary to evaluate generative tasks. While this metric is not perfect it seems the paper is currently bypassing this measure of quality and mostly focusing on diversity and adherence to prompt.
• There are a couple of comparisons that could have been included to other recent models that provide evaluation results for the same checkpoint across understanding and generation and/or provide publicly available checkpoints for evaluation such as GILL (Koh et al NeurIPS 2023) -- mentioned in the paper, Unified-IO 2 (Lu et al CVPR 2024), and GenLLava (Hernandez et al 2024). These models are not finetuned for each type of tasks, generation, editing or understanding but are finetuned jointly to perform well across types of tasks.

• Understanding is negatively affected by multi-granularity modeling. Looking at Table 5 in the appendix, it is not clear whether multi-granularity modeling adds to the understanding ability of the model. For half of the metrics the performance becomes worse, for one is the same, and only for a single metric the performance actually seems to improve. Do the authors have an intuition of why this is the case? Is spending part of the compute budget in the coarse level features the reason for this? Or is it the features themselves that confuse the model.

• Understanding ablations. It is not clear how the ablations in appendix C were done. Did the authors train multiple models with encodings from only a single granularity? Or did the authors remove the encodings at other granularities at inference time to evaluate on how well the understanding performs when given a single granularity encodings?

• Reconstruction performance. In the provided evaluations, the proposed method outperforms previous models in terms of reconstruction given image encodings at the finest granularity. Is this an effect of the multi-granularity learning? Or is it just an effect of the encoder / decoder combination. If it's just the encoder / decoder combination, does this result validate the proposed multi-granular idea of the paper?

• Use of multiple decoders. Is using multiple decoders necessary? Did the authors try using a conditioning signal to tell the diffusion decoders what type of inputs are being fed for generation?