GenRec: Unifying Video Generation and Recognition with Diffusion Models

Thank you for your valuable comments. We will address your concerns point by point.

Q1: Provide more details on the derivation of Eq.14-16.

The derivation of Eqs. 14-16 is similar to Eq. 12 in [A]. Here are the details.

Firstly, Eq. 13 in our paper shows the formulation of the unconditional noise reduction residual.

To introduce the class control, we substitute the score function $\nabla p(\hat{z_i})$ with the conditional form $\nabla_{\hat{z_i}} \log p_{\theta} (\hat{z_i}|y)$, aiming it to be the conditional distribution we want. It is worth noting that the substitution introduces class information and is not an equivalent replacement. Next, by applying the Bayes` Theorem, the score function becomes:

Evidently, ${\nabla_{\hat{\mathbf{z}}_i}} \log p(y)=0$ and can be removed. As a result, the conditional noise reduction residual formulation will become,

which corresponds to the Eq. 15 in our paper. We will update this section with the above detailed derivation process in the revised version.

[A] Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. In NeurIPS, 2021.

Q2: The authors are encouraged to provide justifications for the EDM second-order correction ommision

We follow the setting of SVD implementation, which uses the EulerDiscreteScheduler within the EDM framework, as can be seen in the ``huggingface, stable-video-diffusion-img2vid-xt, scheduler/scheduler_config.json''. EulerDiscreteScheduler do the denoising step without second-order correction, so in our previous version, we omit the second order correction.

In addition, with the Eq. 17 that reveals the calculating way for the class-condition denoising residual $\epsilon^{*}_{\theta}(\hat{\mathbf{z}}_i;\overline{\mathbf{\mathbf{z}}_0})$,

we can substitute it into the algorithm 2 in EDM to get the second-order expression as below:

We will add the above illustration and the derivation process in the final revised version.

Q3: In Table 1, the generative performance of Baseline I should be included. Furthermore, information on parameter counts and computational overhead should also be presented.

Thank you for your insightful suggestions. We acknowledge the importance of including the generative performance of Baseline I. We have supplemented Baseline I with its generation scores to act as the generation baseline, shown as bellow.

Also the parameter counts information has been added in the updated PDF, Table 1.

As for the computational cost/time analysis, we compare the training and recognition inference time cost between MVD-H, baseline II and GenRec, fairly on the same hardware resource: 4-nodes * 8 V100s, and same batch size. Baseline II refers to fully fine-tuning the original SVD model with classification supervision only. As shown in the table, both GenRec and Baseline II consume more test time than MVD-H, primarily due to the higher number of parameters. GenRec requires additional decoder blocks for video generation, which slightly increases the training time compared to Baseline II. However, these blocks are not used during action recognition, so GenRec and Baseline II share the same test time. Note that the ‘‘✝’’ in MVD-H indicates the use of repeated augmentation, which significantly increases the training time. Our approach doesn't need to use that augmentation. All methods fine-tune downstream for 30 epochs.

The above discussions will be included in our revised version.

Q4: misspelling words and notation inconsistent

Thanks for your carefully reading. We will definitely check and revise, and make sure of the consistency in the revised version.

Many thanks for the authors' response. I've checked the authors' rebuttal and other reviewers' comments. I think the response has addressed my concerns. Therefore, I decide to raise the score to 5. Moreover, is it possible for the authors to provide the derivation from Eq. 3 to Eq. 4? The derivation is a bit confusing to me.

Thanks for the positive feedback! We address your questions below.

Q1: SOTA selection and completion.

We greatly appreciate your suggestion. We agree that including more comparable methods is very meaningful. We have updated the Table 1 in the uploaded PDF to include more comparable methods, specifically InternVid, InternVid2, OmniVec, and OmniVec2, and categorized them into single modality and multi-modality for clearer comparison.

Our initial version primarily focused on comparing with single-modality self-supervised methods. While the newly added methods incorporate multiple modalities, introducing multiple modality encoders for multi-modal alignment on large datasets, building upon previous self-supervised methods. Although SVD is a model that is pretrained on additional data, it does not explicitly perform multi-modal alignment during the generation training process, making it hard to surpass their performance. Indeed, unifying the generation task and visual-language multi-modal alignment is a promising research direction and can be achieved by extending our work.

Q2: Provide example videos for the competing methods, as reference points.

Thanks for your helpful suggestions. We have added more detailed example comparison with competing methods, as shown in the uploaded PDF, Figure 1 and Figure 2. In the update, we compare examples provided by SEER official website. Also we add more comparison among the closed-source generation methods (PIKA, Gen-2, KLING) and open-source method (SEER) on action control cases. From the cases in the PDF, we find that: SEER suffers from distortion issues and lacks accuracy, e.g, easy to get wrong for the left-right direction control. PIKA maintains strong visual consistency but fails to generate accurately according to instructions. Gen-2 often has problems with object deformation. KLING shows very high visual consistency, and for some instructions it can generate high-quality correct videos, e.g, cases of ``covering something<the toy> with something<a blue towel>''. But sometimes it struggles with fine-grained action control. Our method demonstrates good performance, with higher visual quality and more accurate fine-grained action control, compared with other methods.

More comprehensive cases comparison will be added in our final revision version.

Q3: Need Generation baseline.

Thank you for your insightful suggestions. We acknowledge the importance of including a generation baseline for SVD. Baseline I can serve as a robust generation baseline, and we have supplemented it with its generation scores to act as the generation baseline, shown as bellow table. We would update this in our revised paper version.

Q4: Paper should cite more enough works

Thanks. We would cite your mentioned related works in the revised version.

Thank you for your valuable comments. We will address your concerns point by point.

Q1: Why the loss weight of the recognition loss is not related with the noise level?

Thanks for your valuable feedback. Adding loss weighting related to noise is standard for diffusion training, but is unexplored for classification. Designing the dynamic classification loss weight brings additional hyperparameters, influenced by factors such as noise level and masking ratio. Note that in EDM generation, small loss ratio are set when the noise levels are large and small. However, for classification, we aim for robust partially video recognition with varing noise. Thus, we maintain a constant classification loss weight to ensure stability, and also to reduce the number of hyperparameters, making training more manageable. Experimental results demonstrate our design effectively balances both generation and classification.

Q2: Training and testing efficiency/computational cost, compared with other recognition models.

Thanks for your valuable feedback. We compare the training and inference time costs between MVD-H, Baseline II, and GenRec, using the same hardware (4 nodes * 8 V100s) and batch size. Baseline II refers to fully fine-tuning the SVD with classification only.

As shown in the table, both GenRec and Baseline II consume more test time than MVD-H, primarily due to the higher number of parameters. GenRec requires additional decoder blocks for video generation, which slightly increases the training time compared to Baseline II. However, these blocks are not used during action recognition, so GenRec and Baseline II share the same test time. Note that the ‘‘✝’’ in MVD-H indicates the use of repeated augmentation, which significantly increases the training time. Our approach doesn't need to use that augmentation. All methods fine-tune downstream for 30 epochs.

Q3: Model size.

Thanks. We have updated Table 1 in the uploaded PDF to include parameter counts.

Q4: Concern about whether the classification training will degrade the model generation ability.

Thank you for your valuable feedback. To demonstrate classification and generative can coexist well in our training without negatively impacting each other, we have constructed the comparison with "Baseline I (SVD baseline)", which adapts SVD on the target dataset without classification supervision, and then trains another classifier head freeze the generation backbone, ensuring no influence from classification supervision. We conduct comparisons on SSv2: FP (predicting future frames) and CFP (predicting future frames with class information).

By comparing the FP results, the similar scores between the two methods conclude that classification loss does not degrade the model’s generative ability in our approach. Moreover, the CFP results indicate that our method, with its more accurate classification performance, can guide the model to achieve higher-quality video frame predictions.

We will update the above discussion in our revised version.

Q5: Mask mechanism discussion with [A].

Thank you for your valuable suggestions. In [A], the authors choose to mask on the noisy latents directly, while we choose to mask the condition latents. Masking noisy latents can break the mathematical theorem basis and complicate the task. Therefore, we opt not to influence the latent noise but to change the conditional latents instead.

We will add the discussion about [A] in the related work section of our revised version.

[A] Masked Diffusion Transformer is a Strong Image Synthesizer, ICCV 2023.

Q6: Table 4 results show recognition and generation may be conflict.

Thanks. Table 4 presents an ablation study on the layer index for feature extraction, showing proper layer selection is needed for preventing conflict. Shallow feature extraction decreases performance due to insufficient complexity, while late feature extraction conflicts with generation. Thus, we selected an optimal layer to balance performance and prevent conflicts between generation and understanding tasks.

For further demonstration, we upload the PDF and compare with baseline performance. In the table 1, Baseline I train the diffusion process without classification, and Baseline II trains classification without generation. The results show our method outperforms Baseline I in generation and matches or exceeds Baseline II in classification. These findings confirm our final GenRec model does not exhibit conflicts between generation and classification.

Q7: SVD to extract features is conventional, and the design of unifying generation and recognition is not elegant.

Thanks. We would like to stress that applying diffusion models for video understanding is non-trivial due to the complicated nature of videos. Moreover, unifying generation and recognition through diffusion models is even more challenging, as conflicts can easily arise and mutual enhancement between the two tasks is difficult to achieve. This is precisely where our GenRec stands out.

By exploring the temporal modeling capabilities of SVD and integrating our novel random masking condition, GenRe effectively bridges the gap between generation and classification, and achieves significant mutual enhancement between the two types of tasks. The effectiveness confirms our design is innovative and pragmatic, and we believe GenRec represents an exciting direction for the computer vision community by combining generation and recognition tasks in a unified framework.

Thanks for the positive feedback! We address your questions below.

Q1:SoTA results completion.

Thank you for your valuable feedback. We have updated the Table 1 in the uploaded PDF, which includes more comparisons with methods like InternVid, InternVid2, OmniVec and OmniVec2. In addition, we have included the parameter counts for each method for a more comprehensive comparison.

It is important to note that, our initial version primarily focused on comparing with single-modality self-supervised methods. In contrast, the newly added methods, e.g, InternVid2, incorporate more modalities, introducing multiple modality encoders for multi-modal alignment on large data, and is built upon the previous self-supervised methods. Therefore, we have distinguished between these two types of methods in the updated table.

Q2:More analysis and explanation about the FVD scores.

Thank you for your valuable feedback. We have double-checked our FVD results and confirmed their accuracy. The reduction in FVD can be attributed to two main factors: The use of the SVD model and the guidance of an accurate classifier. Among two of them, the application of the SVD pretrained model is the primary reason.

To illustrate this point, we have added the FVD scores for baseline I for frame prediction and class-conditioned frame prediction. Results are shown as below. It is evident that using the fine-tuned SVD of "baseline I" for frame prediction (FP, given the first two frames and predicting the subsequent video frames) yields good scores as shown in the first line of the table below, thanks to the rich priors learned during the SVD pretraining on large-scale video data. In contrast, other methods primarily rely on image-based stable diffusion models and fine-tune them on limited video data, making it more challenging to learn temporal dynamics and resulting in lower performance. In addition, the table below also shows that precise classifier guidance, corresponding to CFP (class-condition frame prediction), leads to better FVD results.

In summary, the major cause for the low FVD is the usage of pretrained SVD model, along with the significant impact of accurate classification guidance. We will include the analysis and explanation of the FVD scores in our revised version.

Q3:Ablation on the loss balance ratio and the masking schemes.

Thank you for your valuable feedback. We conduct an ablation study on the loss balance ratio, and the results are presented below. The study shows that varying the loss ratio has a minor impact on action recognition accuracy. However, setting the ratio too low negatively affects the generation performance. In particular, when the ratio is set to zero, significant forgetting occurs, severely compromising the model’s ability to perform the generation task.

We also ablate the masking schemes with different masking ratio expectation. The results show that a larger masking ratio might be beneficial for generation, as it closely resembles our class-conditioned frame prediction with one or two given frames setting. However, too large masking ratio (87.5%) harms the action recognition accuracy, resulting in a 0.9% decrease compared to our chosen ratio.

These ablation results will be included in the final revised version.

Q4:Why is the focus on “particularly the unconditioned or image-conditioned models?

Thanks. This sentence highlights our focus is not on the type of text-conditioned video generation methods that requires video description as input. This is because, understanding videos requires the model to analyze video content, whereas text-conditioned generation relies on descriptions that predefine the content. Such conflict makes it harder to unify these two approaches.

Q5:Minor weakness about the expressions.

Thank you so much for your carefully reading. We will definitely revise the expressions in the revision version.

Thank you for your valuable comments. We will address your concerns point by point.

Q1: Concern about the novelty compared with [8,37,46].

Thanks. Previous works [8, 37, 46] applied diffusion models to understanding tasks, but there are some limitation that mainly differs from our paper:

• [8] abandons generation capabilities in the end for improved discriminative performance;
• [37] uses a frozen image diffusion model and only focuses specifically on low-level geometric correspondence;
• [46] uses a frozen image diffusion backbone only for mask location prediction;

In summary, they ignore visual generation, do not unify generation and recognition tasks, and also they are limited to the image domain. In contrast, our paper focuses on the video domain, addressing the following challenges:

• We demonstrate that video diffusion models can leverage generative priors for analyzing high-level actions, especially for tasks requiring thorough temporal information analysis.
• We demonstrate that video generation and understanding can coexist and enhance each other, improving recognition robustness with limited visual frames, and adherence to instructions for generation.

We believe these explorations are novel and significant as suggested by Reviewer 96MS.

[8] Deconstructing denoising diffusion models for self-supervised learning. arXiv preprint arXiv:2401.14404, 2024

[37] Emergent correspondence from image diffusion. In Neurips, 2023

[46] Open-vocabulary panoptic segmentation with text-to-image diffusion models. In CVPR, 2023

Q2: Why is noise incorporated during inference for video recognition, how does it affect result determinism, is multi-step denoising required during inference?

Thank you for your feedback. Our unified models are trained on noisy inputs, so maintaining consistency during inference is necessary to ensure robustness. Works [37, 46], as you previous mentioned, also need to add a certain-level noise before extracting visual features. We further do the testing on SSv2 action recognition for multiple times with different random seeds, shown as the table below. The results show that the accuracy is consistent with only 0.0125% standard variation, indicating the model’s robustness to the noise. For fully deterministic results, the random seed for noise sampling should be fixed.

During inference, only one single-step denoising is required for prediction. Thus, the inference process is efficient.

Q3: Training and inference time cost compared with previous classification methods.

Thank you for your valuable feedback. We compare the training and inference time cost between MVD-H, Baseline II and GenRec, fairly on the same hardware resource: 4-nodes * 8 V100s, and same batch size. Baseline II refers to fully fine-tuning the original SVD model with classification supervision only.

As shown in the table, GenRec and the Baseline II consumes more time for testing than MVD-H. The difference in time primarily arises from the varying number of parameters. Our model is derived from a generative model, and the complexity of the generative task necessitates a larger number of parameters for effective learning. Compared with Baseline II, since GenRec requires additional decoder blocks for video generation training, the training time will get increased a little bit. As the additional up-sampling blocks will not be used when doing action recognition, GenRec shares the same testing time with Baseline II. It is worth noting that ''✝'' in MVD-H is to highlight that MVD method would use repeated augmentation technique during training, but such augmentation will significantly increase the training time. Our approach does not need to use that augmentation. All the methods do the down-stream finetuning for 30 epochs.

I have carefully read the responses from the authors and other reviewers' comments. The response has addressed my concerns, I would like to raise my rating to 5.