LARP: Tokenizing Videos with a Learned Autoregressive Generative Prior

We thank Reviewer cs2m for their insightful and constructive review of our paper. We appreciate the positive feedback on our analysis of AR tokenization limitations, the proposed holistic tokenizer, and the strong experimental results demonstrating its effectiveness. We address each of the raised concerns and suggestions in detail below.

W1: Comparison with [1].

We would like to highlight that, as noted at the end of Section 2.1 in our original paper, [1] is a concurrent work to ours.

Although [1] also uses holistic tokens to represent visual signals, LARP and [1] differ significantly in several key aspects, as outlined below:

1. Order and Prior: [1] does not define an order for their latent tokens and does not use prior models or regularization losses to enhance the performance of their downstream generative models. In contrast, the AR prior model is a core contribution of LARP, substantially enhancing video generation quality, as demonstrated in Figure 1(c) of our main paper.
2. Generative Model: [1] uses MaskGIT [5] as their video generator, a masked language modeling (MLM) generative model. In contrast, LARP is designed for AR generative models, known for their exceptional effectiveness and scalability.
3. Focus: [1] focuses on image generation, but LARP focuses on video generation.
4. De-tokenization: [1]'s ViT decoder operates similarly to MAE [2], where multiple identical mask tokens are input to their ViTs to reconstruct the images patches. In contrast, LARP is inspired by DETR [3] and the Q-former introduced in [4], where distinct query tokens are used to extract holistic tokens and reconstruct video patches.
5. Training Strategy: [1] employs a two-stage training process. In the first stage, they train their image tokenizer with discrete codes generated by an off-the-shelf MaskGIT-VQGAN model, akin to a model distillation approach. In the second stage, they fine-tune the decoder using the VQGAN objective. In contrast, the LARP tokenizer is trained end-to-end with a VQGAN-like objective, making it simpler and eliminating the need for other pre-trained models.
6. Discriminator Design: For the GAN loss, [1] uses a ConvNet-based discriminator, while LARP employs a transformer-based discriminator, enhancing reconstruction fidelity and training stability. We observed that training LARP with a ConvNet-based discriminator is not stable and often leads to loss divergence.

In fact, we began this project well before the preprint of [1] was released.

W2: For the second contribution (AR prior model), there is no analysis of its accuracy or generalizability. Otherwise, it seems to me difficult for a small 21.7M AR model to predict the next tokens in a highly complex video latent space. The previous work VideoPoet [2] and Video-LaVIT [3] all require a 7B+ LLM to achieve this.

We would like to clarify that the AR prior model is used exclusively during the training of the LARP tokenizer. Its role is to align the discrete latent space of LARP with AR generation tasks, leading to significantly improved generation performance, as demonstrated in Figure 1(b) of the original paper. For this purpose, a small AR model, like the one we used, is sufficient. Using a larger AR model would significantly slow down the training of the LARP tokenizer.

It is important to note that the AR prior model is not involved in training the AR generator. As shown in Table 1 of the original paper, a 632M AR model is trained to achieve state-of-the-art FVD scores on the UCF101 class-conditional generation benchmark. This model size is appropriate given that UCF101 is a smaller-scale, academic-purpose video dataset, unlike the large-scale datasets used in [2] and [3].

Q1: The video data is already nicely ordered by timestamps, why not use this ordering directly for tokenization? Does the learned tokenizer reflect this natural order in its latent space?

Using timestamp order to tokenize videos is possible; however, this approach implies per-frame tokenization, which cannot effectively capture the temporal redundancy in videos, compromising the compactness of the latent space. Moreover, even if timestamp order is used, individual video frames cannot typically be represented with a single token, necessitating further decisions on how to tokenize the content within each frame.

That said, the concept of timestamp order tokenization is compatible with LARP and could play a significant role in future work when scaling up to longer videos. In such cases, LARP could transition from per-video tokenization to per-clip tokenization, where the latent tokens of a long video would consist of multiple groups of LARP tokens for its clips, concatenated along the timestamp dimension. We are excited to explore this promising direction in future work.

[1] Yu, et al. An Image is Worth 32 Tokens for Reconstruction and Generation. NeurIPS 2024.

[2] Kondratyuk, et al. VideoPoet: A Large Language Model for Zero-Shot Video Generation. ICML 2024

[3] Jin, et al. Video-LaVIT: Unified Video-Language Pre-training with Decoupled Visual-Motional Tokenization. ICML 2024

[4] Tran, Du, et al. Learning spatiotemporal features with 3d convolutional networks. Proceedings of the IEEE international conference on computer vision. 2015.

[5] Carreira, Joao, and Andrew Zisserman. Quo vadis, action recognition? a new model and the kinetics dataset. proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2017.

[6] Barratt, Shane, and Rishi Sharma. A note on the inception score. arXiv preprint arXiv:1801.01973 (2018).

[7] Yu, Lijun, et al. Language Model Beats Diffusion-Tokenizer is key to visual generation. ICLR 2024.

[8] Wang, Junke, et al. OmniTokenizer: A Joint Image-Video Tokenizer for Visual Generation. NeurIPS 2024.

[9] Luo, Zhengxiong, et al. Videofusion: Decomposed diffusion models for high-quality video generation. arXiv preprint arXiv:2303.08320 (2023).

[10] Skorokhodov, Ivan, et al. Hierarchical Patch Diffusion Models for High-Resolution Video Generation. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.

[11] Hong, Wenyi, et al. Cogvideo: Large-scale pretraining for text-to-video generation via transformers. arXiv preprint arXiv:2205.15868 (2022).

[12] Ge, Songwei, et al. Long video generation with time-agnostic vqgan and time-sensitive transformer. European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2022.

[13] Yu, Lijun, et al. Magvit: Masked generative video transformer. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.

We would like to thank Reviewer cs2m once again for their thoughtful review, which has greatly contributed to improving our work. If there are any additional concerns, we would be happy to address them. We respectfully request Reviewer cs2m to consider a higher rating in light of our detailed responses and revisions.

W3: FID and IS on UCF101.

The FID (Fréchet Inception Distance) measures the Fréchet Distance between the distributions of Inception v3 network features for real and generated images. It is important to emphasize that FID is a metric specifically designed for evaluating image generation, not video generation, and thus is not an appropriate metric for evaluating LARP. While some earlier video generation works report "FID" scores, these are essentially FVD (Fréchet Video Distance) scores calculated using an outdated video feature extractor, the C3D network [4]. In contrast, LARP and recent works, including all those compared in our main paper, utilize the modern version of FVD in Table 1 of the main paper, which employs the advanced I3D network [5] as the video feature extractor.

The Inception Score (IS) is not a reliable metric for evaluating generative models, as discussed in [6]. Moreover, calculating IS for videos relies on the outdated C3D network [4]. For these reasons, more recent works [7, 8] have moved away from using this metric to evaluate video generators. Consequently, we did not include IS comparisons in our original paper. However, as requested, we provide a comparison of the IS of LARP with other works reporting IS in the table below:

Here, GT denotes the IS calculated for ground truth UCF101 videos. Notably, LARP achieves the highest IS among all methods, with only a minimal gap compared to the ground truth videos. This comparison further highlights the exceptional video generation capabilities of LARP.

W4: Latent space analysis and visualization.

We thank the reviewer for the suggestion. To gain deeper insights into the structure and properties of LARP's latent space, we randomly select 10 videos as examples (shown in Figure 9 of the revised paper) and analyze the impact of individual LARP tokens on the reconstruction quality of these videos.

Using the LARP-L-Long tokenizer, we first calculate the latent LARP tokens for all selected videos and measure their reconstruction PSNR values based on the unmodified LARP tokens. Next, for each video, we systematically modify a specific LARP token by assigning it a random value, then reconstruct the video and compute the PSNR again. The influence of the modified token on reconstruction quality is quantified by the difference between the PSNR values obtained from the unmodified and the single-token-modified reconstructions. This process is repeated for all 1024 LARP tokens across the latent space and for all 10 videos. The resulting PSNR differences are visualized in the scatter plots in Figure 10 of the revised paper.

Interestingly, we observe that for each video, the majority of LARP tokens have minimal impact on reconstruction quality, while only a small subset of tokens carry the critical information necessary to represent the video accurately. However, the specific set of important tokens varies significantly across different videos, emphasizing the content-adaptive nature of LARP tokens, as different subsets of tokens are activated to represent different content.

Beyond the video-level analysis of LARP tokens conducted above, we also examine their impact at the spatial-temporal level.

For LARP tokens of 5 different videos, we select a specific token index and set the values of tokens at this index in all videos to 0. We then compare the spatial-temporal reconstruction differences before and after this modification and visualize these differences as heatmaps, as shown in Figure 11 of the revised paper. The brightness of the heatmaps has been enhanced for improved visibility.

It is evident that this token's influence spans the entire video, unlike patchwise tokens, which primarily affect specific spatial-temporal video patches. This finding confirms that LARP tokens serve as holistic representations for videos. Another intriguing observation is that the LARP token's impact is not random; rather, it is concentrated on specific objects and structures, reflecting the semantic properties of the LARP latent space.

This discussion has been incorporated into Section B.3 of the revised paper.

We thank Reviewer qHTB for their insightful and constructive review of our paper. We are grateful for the positive remarks on our novel approach to video tokenization, its compatibility with AR models, and the strong performance of our method. We address each of the raised concerns below.

W1: Benefit or penalty from using the proposed method on other tasks such as video editing and stylization.

Following recent works on video tokenization [6,7], we primarily focus our efforts on video generation as the primary task. While video editing and stylization are beyond the scope of this paper, the potential benefits of the proposed methods for these tasks are promising. Since LARP is trained for alignment with AR models, including multimodal large language models (MLLMs), it can naturally enhance instruction-driven editing when integrated with MLLMs. Instruction fine-tuning on mixed text and LARP tokens using AR NLL objectives is expected to leverage LARP's AR-aligned design, likely resulting in notable improvements in editing and stylization quality. We believe these are interesting future follow-up works of our paper.

W2 & Q1: Comparison with [1].

We would like to emphasize that, as noted at the end of Section 2.1 in our original paper, [1] is a concurrent work to ours.

Although [1] also uses holistic tokens to represent visual signals, LARP and [1] differ significantly in several key aspects, as outlined below:

1. Order and Prior: [1] does not define an order for their latent tokens and does not use prior models or regularization losses to enhance the performance of their downstream generative models. In contrast, the AR prior model is a core contribution of LARP, substantially enhancing video generation quality, as demonstrated in Figure 1(c) of our original paper.
2. Generative Model: [1] uses MaskGIT [5] as their video generator, a masked language modeling (MLM) generative model. In contrast, LARP is designed for AR generative models, known for their exceptional effectiveness and scalability.
3. Focus: [1] focuses on image generation, but LARP focuses on video generation.
4. De-tokenization: [1]'s ViT decoder operates similarly to MAE [2], where multiple identical mask tokens are input to their ViTs to reconstruct the images patches. In contrast, LARP is inspired by DETR [3] and the Q-former introduced in [4], where distinct query tokens are used to extract holistic tokens and reconstruct video patches.
5. Training Strategy: [1] employs a two-stage training process. In the first stage, they train their image tokenizer with discrete codes generated by an off-the-shelf MaskGIT-VQGAN model, akin to a model distillation approach. In the second stage, they fine-tune the decoder using the VQGAN objective. In contrast, the LARP tokenizer is trained end-to-end with a VQGAN-like objective, making it simpler and eliminating the need for other pre-trained models.
6. Discriminator Design: For the GAN loss, [1] uses a ConvNet-based discriminator, while LARP employs a transformer-based discriminator, enhancing reconstruction fidelity and training stability. We observed that training LARP with a ConvNet-based discriminator is not stable and often leads to loss divergence.

In fact, we began this project well before the preprint of [1] was released.

Q2: Would the holistic tokens hurt the appearance preservation capabilities in editing tasks such as inpainting and outpainting?

While video editing tasks are beyond the scope of this paper, we can reasonably expect that the use of holistic tokens would not hinder these tasks. Frame prediction can be considered a special case of video outpainting, specifically temporal single-direction outpainting. Experimental results in Table 1 of our main paper show that LARP achieves competitive performance on the Kinetics-600 video frame prediction benchmark. Therefore, we believe that LARP is well-suited for broader inpainting and outpainting generation tasks, as well as other video editing applications.

Q3: Frame prediction implementation details.

To create the 5-frame conditioning holistic tokens for the frame prediction task, we first take the initial 5 frames from the ground truth video, then repeat the 5th frame 11 times and pad these repetitions with the initial 5 frames (using the same padding). This forms a complete 16-frame conditioning video. This conditioning video is then input to the LARP tokenizer to generate the conditioning holistic tokens.

At inference time, the AR model predicts the holistic tokens for the full video, conditioned on the holistic tokens from the padded conditioning video.

This clarification has been included in Section A.2 of the revised paper.

[1] Yu et al. An Image is Worth 32 Tokens for Reconstruction and Generation. arXiv 2406.07550

[2] He, Kaiming, et al. "Masked autoencoders are scalable vision learners." Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2022.

[3] Carion, Nicolas, et al. "End-to-end object detection with transformers." European conference on computer vision. Cham: Springer International Publishing, 2020.

[4] Li, Junnan, et al. "Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models." International conference on machine learning. PMLR, 2023.

[5] Chang, Huiwen, et al. "Maskgit: Masked generative image transformer." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022.

[6] Yu, Lijun, et al. Language Model Beats Diffusion-Tokenizer is key to visual generation. ICLR 2024.

[7] Wang, Junke, et al. OmniTokenizer: A Joint Image-Video Tokenizer for Visual Generation. NeurIPS 2024.

We would like to thank reviewer qHTB once again for their review, which has been instrumental in strengthening our work. Should there be any further concerns, we would be glad to address them. We respectfully ask reviewer qHTB to consider a higher rating in light of our responses and revisions.

Q3: Despite the flexibility of supporting an arbitrary number of discrete tokens, the claim of “global and semantic representations” seems not validated in the paper. Are there experimental results supporting this claim?

The holistic tokenization scheme of LARP enables global and semantic representations, effectively capturing the spatiotemporal redundancy in videos and representing them accurately within a compact latent space. The semantic properties of LARP tokens are demonstrated by their exceptional generation quality. Moreover, we provide an extensive analysis and visualization of LARP's latent space in Section B.3 of the revised paper. Our visualizations reveal that LARP tokens influence the entire video, unlike patchwise tokens that primarily affect specific spatial-temporal patches. This finding underscores that LARP tokens act as holistic representations for videos. Interestingly, the impact of LARP tokens is not random; instead, it is focused on specific objects and structures, highlighting the semantic nature of the LARP latent space. We kindly refer you to Section B.3 for more details regarding the representations learned by LARP.

To further validate LARP's capability to represent videos globally and holistically, we manually introduce information redundancy into videos and measure the improvement in reconstruction quality.

Specifically, we fix the total number of video frames to 16, consistent with our main experiments, and repeat shorter videos multiple times to form 16-frame videos. This increases information redundancy as the number of unique frames decreases with higher repetition, implying improved reconstruction quality for tokenizers capable of exploiting global and holistic information.

The table below reports the reconstruction PSNR for various repetition levels. We also compare LARP against a patchwise video tokenizer, OmniTokenizer. Notably, although OmniTokenizer operates in a patchwise manner, its transformer architecture allows it to leverage some global information. For repetition levels, a value of 1 indicates no repetition, where all frames are unique, while a value of 16 represents full repetition, where a single frame is repeated to create the entire video.

The results in the table demonstrate the impact of increasing information redundancy on reconstruction quality for both OmniTokenizer and LARP. As the repetition level increases, both tokenizers show improved PSNR, reflecting their ability to leverage the added redundancy. However, LARP consistently outperforms OmniTokenizer across all repetition levels, with a significantly larger improvement in PSNR as the redundancy increases.

These results validate LARP's effectiveness in capturing spatial-temporal redundancy, enabling it to represent videos more accurately within a compact latent space.

This discussion has been incorporated into Section B.2 of the revised paper, with the results from the table above visualized in Figure 8.

Q4: What will the rFVD be if 1280 tokens are used?

We evaluate the rFVD of LARP-L-Long with 1280 tokens and compare it with the rFVD scores of other methods and configurations, as shown in the table below:

It is evident that increasing the number of tokens used by LARP from 1024 to 1280 leads to a significant improvement in rFVD. The use of 1280 tokens enhances the information capacity of LARP's latent space, resulting in better reconstruction quality.

[1] Ge, Songwei, et al. Long video generation with time-agnostic vqgan and time-sensitive transformer. European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2022.

[2] Yu, Lijun, et al. Magvit: Masked generative video transformer. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.

[3] Yu, Lijun, et al. Language Model Beats Diffusion-Tokenizer is key to visual generation. ICLR 2024.

[4] Wang, Junke, et al. OmniTokenizer: A Joint Image-Video Tokenizer for Visual Generation. NeurIPS 2024.

We sincerely thank reviewer 3WZp once again for their valuable feedback, which has played a crucial role in improving our work. If there are any additional concerns, we would be happy to address them. We are committed to ensuring clarity and providing comprehensive responses to all concerns. We kindly request reviewer 3WZp to consider a higher rating in light of the revisions and responses provided.

We thank Reviewer 3WZp for their constructive feedback and acknowledgment of our methodological and experimental contributions. We appreciate the thoughtful suggestions, which will help us improve our work further. We respond to each of the concerns raised below.

W1: Typos.

We have corrected the typos in the revised paper.

W2: Effectiveness of holistic video tokenization and autoregressive generation, detailed in the Questions section.

We address each question below.

Q1: If this transformer encoder is replaced by an encoder-decoder structure, will it serve as an autoregressive prior?

We emphasize that the learned autoregressive prior is agnostic to model architecture details. It operates in the discrete latent space to align it with the downstream autoregressive generation task, without imposing assumptions on how data is encoded into this space. Thus, replacing the transformer encoder with an encoder-decoder structure would not affect the applicability of the learned autoregressive prior.

In fact, we explored the encoder-decoder architecture during early experiments and observed the similar performance. However, the encoder-decoder structure introduces additional cross-attention layers, increasing model complexity and reducing parameter efficiency. For these reasons, we opted for the simpler transformer encoder architecture.

Q2: Could the authors further explain why an autoregressive prior during tokenizer training can lead to better generation performance? An argument is that, lower autoregressive prior loss does not necessarily lead to better generation performance, since in an extreme case, if all the tokens are the same, then the autoregressive prior loss can be extremely low, but the codebook will collapse and the downstream generation task will definitely fail.

The extreme case mentioned in Q2 is a valid concern. To prevent such collapse, we combine the AR prior loss with the reconstruction loss and use a small weight $\alpha$ for the AR prior loss, as detailed in Eq. 8 and line 406 of the original paper. This ensures the reconstruction loss dominates the training objective, preserving the integrity and effectiveness of the codebook. This is why LARP achieves strong performance in both reconstruction and generation tasks, as demonstrated in Table 1 of the original paper.

Learning an autoregressive prior during tokenizer training improves generation performance by aligning the tokenizer's discrete latent space with the autoregressive generation task. This alignment is achieved through the autoregressive prior loss, which encourages a more structured latent space, simplifying downstream autoregressive modeling.

Thanks for the authors’ response. I still have some questions.

• By increasing the number of tokens from 1024 to 1280, we can indeed observe better performance in rFVD. However, there is still a gap between MAGVIT-v2 (16 vs 8.6). Where do you think the gap comes from?
• I’m still wondering if the holistic video tokenization technique is necessary for incorporating with the autoregressive prior. Can the autoregressive prior be added to patchwise tokens and lead to improved generation performance?
• For the comparison in Section B.3, I acknowledge the authors’ effort in demonstrating the advantage of holistic video tokenization over patchwise video tokenizers for better leveraging redundancy. However, since MAGVIT-v2 is a much stronger baseline than OmniTokenizer, I’m still curious how MAGVIT-v2 will perform in this setting.

We thank Reviewer KrLv for their thoughtful and constructive review of our paper. We appreciate the positive feedback on our paper's novelty, experimental design, and performance. Below, we address each of the concerns raised.

W1: Computational cost and complexity of training LARP, especially with the AR prior model.

We acknowledge that training LARP as a video tokenizer is computationally intensive. However, LARP's training efficiency is significantly higher compared to other transformer-based tokenizers, such as OmniTokenizer [1]. For example, training the LARP-L-Long tokenizer required only two days on 8 H100 GPUs, whereas OmniTokenizer required two weeks on 8 A100 GPUs, as reported in their paper. Additionally, we emphasize that the computational demand of using the AR prior model is minimal, as it is a lightweight model with only 21.7 million parameters.

The table below compares the training speeds of LARP-L with and without the AR prior model, both conducted on 4 H100 GPUs. Notably, incorporating the AR prior model results in only a 1.19 iter/s reduction in speed, a trade-off well justified by the substantial improvement in generation quality enabled by the AR prior model, as demonstrated in Figure 1(c) of our main paper.

W2: It remains unclear how stable the training process is over longer periods or under different training regimes.

In Figure 7 of the revised paper, we present the training loss curves for LARP across different training epochs and regimes. As illustrated, all loss curves decrease smoothly and converge by the end of training, demonstrating LARP's strong training stability. Further details are provided in Section B.1 of the revised paper.

W3 & Q3: Limitations and failure cases.

While LARP significantly enhances video generation quality, it still has certain limitations. Like other transformer-based video tokenizers [1], LARP performs best with fixed-resolution videos due to the constraints of positional encoding. Additionally, artifacts may appear in LARP-generated videos when the scenes are particularly complex. Fortunately, scaling up the AR generative model is expected to improve video quality and reduce these artifacts, as suggested by the scaling laws of AR models [2,3].

This discussion has been added to Section D of the revised paper.

Q1: How does the training process scale with larger datasets or more complex video content?

In this paper, we follow standard practices established in the literature, particularly recent works on video tokenization [1,4], to evaluate LARP on the UCF101 and K600 datasets. We acknowledge that due to limited computational resources, we are currently unable to scale LARP to larger video datasets with more complex content. However, we believe that the extensive experimental results obtained on these widely used benchmarks are sufficient to showcase LARP's superior performance in video generation.

Q2: What are the hardware requirements for training and deploying LARP?

Training the LARP-L-Long tokenizer takes 2 days on 8 H100 GPUs with the default global batch size of 128, or 4 days on 4 H100 GPUs with a reduced global batch size of 96.

During inference, the LARP-L-Long tokenizer and the 632M LARP AR model can generate up to 1.8 videos per second on a single H100 GPU. The minimum GPU memory requirement to run the entire LARP system is only 3GB when using a batch size of 1.

[1] Wang, Junke, et al. "OmniTokenizer: A Joint Image-Video Tokenizer for Visual Generation." arXiv preprint arXiv:2406.09399 (2024).

[2] Henighan, Tom, et al. "Scaling laws for autoregressive generative modeling." arXiv preprint arXiv:2010.14701 (2020).

[3] Sun, Peize, et al. "Autoregressive Model Beats Diffusion: Llama for Scalable Image Generation." arXiv preprint arXiv:2406.06525 (2024).

[4] Yu, Lijun, et al. Language Model Beats Diffusion-Tokenizer is key to visual generation. ICLR 2024.

We thank reviewer KrLv once again for their review, which has been invaluable in improving our work. If there are any additional concerns, we would be more than happy to address them. We kindly ask reviewer KrLv to consider an increased rating based on our responses and revisions.