Language Model Beats Diffusion - Tokenizer is key to visual generation

We appreciate the thoughtful comments from reviewer k12R and the acknowledgment that with the proposed tokenizer, LLM surpasses diffusion models in standard benchmarks, and the tokenizer also shows promising results in video compression and representation learning.

W1: Incorporating results from AR-LM.

In the Appendix, we have added experiments on the autoregressive language model (AR-LM) where the proposed tokenizer outperforms the prior work MAGVIT [R3-A]. The results, also provided in Table R3.1 below, show our tokenizer can also work with the AR-LM.

It is worth noting that we have observed that achieving optimal performance requires a much larger AR-LM model size. But due to time constraints, we have not been able to train a larger AR-LM model with an adequate number of training steps. That also partly explains our current choice of not building our work on the autoregressive language model, and we leave it as future work.

Table R3.1: Video generation results: class-conditional generation on UCF-101 with AR-LM models. We use the same transformer configuration as MLM experiments but without vocabulary factorization and weight tying. As a result, the AR-LM with our tokenizer uses more parameters in the embedding table and the softmax layer.

[R3-A] MAGVIT: Masked Generative Video Transformer. In CVPR 2023.

W2: Efficacy as a viable self-supervised pre-training target compared to baselines like pixel colors and image descriptors.

Following the reviewer’s suggestion, we have added pixel colors and HoG descriptors from MaskFeat [B] as baseline self-supervised pre-training targets into revised Table 4, with a distilled version in Table R3.2 below. As shown, pre-training with tokens from our model as the target achieves the best performance compared to the previous leading tokenizer MAGVIT as well as other baselines, including raw pixels, 3D VQ-VAE, and HoG descriptors.

Table R3.2 Video action recognition performance (classification accuracy↑ $\times$100) with different self-supervised pre-training targets as the transformer output.

Q1: Generation performance with LFQ at larger vocabularies?

In Table R3.3, we showcase an instantiation of LFQ with a much larger vocabulary size at $2^{40}$. As shown, it yields consistent improvement over $2^{18}$ in both reconstruction and generation FID metrics.

Table R3.3 Class-conditional image generation results on ImageNet 512$\times$512 with larger LFQ vocabularies. We use an MLM model and 12 decoding steps.

Q2: PSNR and MS-SSIM compared to standard video codec.

Our model uses GAN and perceptual losses to improve the reconstruction quality especially in terms of the realism of the generated video. At a similarly low bit rate, standard codecs based on block-level Fourier-related transformations may preserve better local details but introduce inter-block artifacts. As pointed out by [R3-B], traditional distortion metrics such as PSNR and MS-SSIM, “for very low bitrates … these distortion metrics lose significance as they favor pixel-wise preservation of local structure over preserving texture and global structure.” [R3-B] As a result, while standard codecs might appear to outperform on metrics like PSNR and MS-SSIM, they fall short in the gold-standard human rater study (see Figure 6). We have revised Section 4.3 to clarify this point.

[R3-B] Generative adversarial networks for extreme learned image compression

We appreciate the constructive comments from reviewer sfJe and the positive assessment. We provide responses to each individual question below.

W1: the completeness of exposition in the paper; assuming the reader is familiar with the state of the art in video tokenization.

Thank you for your understanding regarding the space limitations. To enhance the method's readability and self-containment, we have implemented the following changes. First, we have added a paragraph in Section 3.1, detailing all the training losses, which includes those from prior works. Second, we have added a section to the Appendix, which provides additional details on model design and hyperparameters. Third, a paragraph has been included in Section 3.2 to clarify the weight tying approach in the factorized prediction. We hope these can improve the completeness of the exposition, and we are open to additional recommendations from the reviewer.

Q1: Why masked LM and not AR LM for image / video generation?

We selected the masked language model (MLM) due to its competitive performance on benchmark datasets [R2-A, R2-B] compared to the autoregressive language model (AR-LM), where MLM also uses much fewer decoding steps than AR-LM. In Table R2.1, we also show experiments on the AR-LM where the proposed tokenizer outperforms the prior work MAGVIT [R2-A]. The results show our tokenizer can also work with the AR-LM, but the AR-LM lags behind the state-of-the-art. We have included this rationale in the revised Section 4.2 and added the new result to the Appendix.

It is worth noting that we have observed that achieving optimal performance requires a much larger AR-LM model size. But due to time constraints, we have not been able to train a larger AR-LM model with an adequate number of training steps. That also partly explains our current choice not to build our work on the autoregressive language model. We are investigating the integration of the tokenizer with the large AR-LM and will study this in our future research.

Table R2.1: Video generation results: class-conditional generation on UCF-101 with AR-LM models. We use the same transformer configuration as MLM experiments but without vocabulary factorization and weight tying. As a result, the AR-LM with our tokenizer uses more parameters in the embedding table and the softmax layer.

[R2-A] MAGVIT: Masked Generative Video Transformer. In CVPR 2023.
[R2-B] Discrete Predictor-Corrector Diffusion Models for Image Synthesis. In ICLR 2023.

Q2: No explicit definition of the objective for training the tokenizer (loss function).

Following the reviewer's recommendation, we have added a description of the tokenizer's training objective at the end of Section 3.1, detailing all the training losses including those from prior works. In addition to the entropy penalty in Equation 5, the overall training objective involves the standard combination of reconstruction, GAN, perceptual, and commitment losses from VQGAN [R2-C], excluding the codebook loss. In addition, we follow [R2-A] in using LeCAM regularization [R2-D] for improved stability. To provide better clarity on the details, we have added Appendix A.2 listing all relevant hyperparameters.

[R2-C] Taming Transformers for High-Resolution Image Synthesis. In CVPR 2021.
[R2-D] Regularizing Generative Adversarial Networks under Limited Data. In CVPR 2021.

Q3 No mention of decoder in VQ-VAE and VQ-VAE loss used when we use no lookup.

Initially, the decoder was only briefly mentioned in Section 2 and in the caption of Figure 2. We have improved the clarity by elaborating on the additional details on model design and hyperparameters in Appendix A.2 with an architecture diagram in Figure 7, as well as by providing a discussion of the loss functions in Section 3.1.

We appreciate the valuable feedback from Review rVyC and the acknowledgment of our method that is proved to benefit video generation, compression and recognition and potentially has a huge impact on the general audience of video understanding. Please see our response below.

W1: Video compression metric curves.

Following the suggestion, we have added Figure 9 in the Appendix showing the curves of compression metrics, including LPIPS, PSNR, and MS-SSIM at various bpp levels. The results are consistent with the findings from Table 3 that our model outperforms MAGVIT on all metrics and outperforms all methods on LPIPS. It is worth noting that LPIPS may correlate more closely with subjective assessments, which we have shown in the human rater study in Figure 6, than PSNR or MS-SSIM, according to [R1-A].

[R1-A] Generative adversarial networks for extreme learned image compression

W2: Necessity of detokenization in video recognition.

We appreciate your understanding that ViViT takes raw pixels as input. The detokenization experiments used the frozen ViViT model that was trained on RGB pixels. For proper inference, the detokenizer is needed to convert the tokens back into pixel inputs, aligning with the input requirements of the frozen ViViT model. As shown in Table 4, our model shows superior performance compared to MAGVIT [R1-B] and is very close to raw pixels while only using ~1/500 data bandwidth. This setting was mentioned in Appendix A.3 of the original submission. And we have revised Section 4.4 to clarify this.

Following the reviewer’s suggestion, we also train a BEVT model with tokens as both the input and the output for action recognition. The result is shown in Table R1.1. We find that directly using token inputs yields suboptimal performance in part because the current architecture and pre-training setups are tailored for pixel inputs. Further research is needed to investigate this token-in-token-out pretraining paradigm for action recognition, where both the input and output are based on discrete tokens.

Table R1.1: Video action recognition performance (classification accuracy↑ $\times$100) on SSv2 dataset.

[R1-B] MAGVIT: Masked Generative Video Transformer. In CVPR 2023.