On Learning Multi-Modal Forgery Representation for Diffusion Generated Video Detection

We thank the reviewer for the careful and helpful feedback. We appreciate all constructive comments on the novelty, clarity, and valuable contribution of the paper. We demonstrate additional experiments and answer all weaknesses and questions mentioned in the review.

Q1 The usage of VQVAE

As for diffusion detection, a series of previous works[1,2] found that the reconstruction process of autoencoders extracts discriminative features for diffusion. Following [2], our work introduces a pretrained vector quantized variational autoencoder (VQVAE) for an augmentation to diffusion forgery features. We display the contribution of VQVAE reconstruction in Figure B of our PDF material.

For brevity and legibility, we do not visualize an explicit reconstruction process in Figure 3 of our paper, as it is not the main contribution of our method. We have provided a detailed explanation of this process in Section 3.2, Lines 147-158 of our paper.

Q2 Analysis of IAFA

We provide analysis and experiments in Q1 of the overall response.

Q3: The relationship between experiment datasets and our proposed DVF

Both training and testing datasets are subsets of DVF. They do not overlap with each other and form the entire DVF together.

Specifically, the dataset DVF includes real videos from Internvid-10M and fake videos from 8 diffusion methods. In experiments, we choose one diffusion method, Stable Video Diffusion, as the training sets, and evaluate all baselines on the other 7 diffusion methods.

Q4: Contribution of DVF

We provide additional experiments to prove the contribution of DVF on performance in Table I. For our method, we emphasize the contribution of DVF in finetuning LMMs. For other methods, DVF provides diffusion traces for forgery detection. We prove the effectiveness of DVF by comparing pretrained baselines with ones fine-tuned on DVF, for which we choose UnviersalFD[5] and F3Net[6], and comparing our method between pretrained and fine-tuned LMMs.

Table I: Performance of pretrained and fine-tuned detectors on DVF in terms of AUC

As is shown, DVF benefits other baselines in $+22.0\\%$ and $+25.9\\%$ in AUC. For our method, the LMM finetuned on DVF also outperforms pretrained one by $+5.4\\%$ in AUC.

Q5: More experiments on validation of our method

We provide further analysis and experiments to validate our method. In the overall response, we provide

1. the functionality of IAFA compared with other baselines in Q1,

2. the generalization of our method on GAN and latest diffusion methods in Q2,

3. the performance of our method on different video durations and resolutions in Q2,

4. the ablation study on different LLMs in Q3.

In our PDF material, we provide

1. Figure A: attention heatmaps of our proposed IAFA compared with other spatiotemporal networks.

Q6: From the visualization results, the colors of the fake image are more colorful, is the author's method still valid if the brightness is adjusted to make them similar to the real image?

Our method is effective after adjusting the brightness. We provide more visualization results of our LMMs on both real and fake content with similar brightness in Figure C of our PDF material.

References

[1] Wang Z, Bao J, Zhou W, et al. Dire for diffusion-generated image detection

[2] Ricker J, Lukovnikov D, Fischer A. AEROBLADE: Training-Free Detection of Latent Diffusion Images Using Autoencoder Reconstruction Error

[3] Wang J, Yang X, Li H, et al. Efficient video transformers with spatial-temporal token selection

[4] Liu Y, Xiong P, Xu L, et al. Ts2-net: Token shift and selection transformer for text-video retrieval

[5] Ojha U, Li Y, Lee Y J. Towards universal fake image detectors that generalize across generative models

[6] Qian Y, Yin G, Sheng L, et al. Thinking in frequency: Face forgery detection by mining frequency-aware clues

Dear Reviewer Niir,

We are glad that our additional experiments solved your concerns. Thanks again for your valuable review.

Best regards,

Authors of Submission 2768

We thank the reviewer for all insightful comments and suggestions. We appreciate the appraisal of the contribution, soundness, and clarity of the paper. We provide additional experiments and answer the weaknesses and questions in the review.

Q1: Comparison with other spatiotemporal networks

We provide a comparison of IAFA with other ViT baselines and analyze our advantages in Q1 of the overall response.

Q2: Discussion and comparison with other multimodal baselines

In previous studies on forgery detection, DEFAKE[1] adopts multimodal representation for forgery detection by using semantic descriptive prompts as an augmentation for visual features. However, semantic descriptions do not have strong correlations with the authenticity of images, which is ineffective in forgery tasks. As a comparison, our method utilizes the powerful reasoning ability of finetuned Large Multimodal Models (LMMs) to provide more accurate features in the text space. Therefore, our method is more effective in forgery detection.

We provide a comparison of DEFAKE[1] with our method in Table H. Each model is trained on the same training dataset SVD, and evaluated on other diffusion datasets in our proposed dataset DVF.

Table H: Comparison between DEFAKE and our method in AUC

As is demonstrated, our proposed multimodal representation is effective in most diffusion forgery content.

Q3: Visualization of DVF

We will release sample video frames from DVF instead of individual frames for better visualization.

Q4: A clarification of the main table

To justify the effectiveness of our method both on the image-level and video-level detection, the main performance table contains both frame and video comparisons. For frame-level detection, frames from each video are treated individually for training and testing to demonstrate the generalization ability of our proposed Multi-Modal Forgery Representation(MMFR) without spatio-temporal information. For video-level detection, continuous video clips are forwarded during training and evaluation. In this case, both multimodal representation and spatiotemporal information take effect. We use these experiment settings to validate the effectiveness of MMFR and IAFA.

Q5: Discussion and comparison with more works

We provide additional analysis and experiments regarding more related works.

In the overall response, we provide

1. the functionality of IAFA compared with other baselines in Q1,

2. the generalization of our method on GAN and latest diffusion methods in Q2,

3. the performance of our method on different video durations and resolutions in Q2,

4. the ablation study on different LLMs for our method in Q3.

In the PDF material, we provide

1. Figure A: Visualization on attention heatmaps from IAFA.

References

[1] Sha Z, Li Z, Yu N, et al. De-fake: Detection and attribution of fake images generated by text-to-image generation models

[2] Arnab A, Dehghani M, Heigold G, et al. Vivit: A video vision transformer

Dear Reviewer LBha,

We are glad that our rebuttal has successfully addressed your concerns. We will include the clarifications into the revised manuscript. Thank you again for your valuable comments.

Best regards,

Authors of Submission 2768

We express our sincere gratitude to the reviewer for the insightful and valuable feedback. We appreciate all constructive comments on the novelty, effectiveness, and clarity of the paper. We provide additional experiments and answer the weaknesses and questions mentioned in the review as follows.

Q1: A clear justification on MM-Det

In our paper, We propose Multi-Modal Detection (MM-Det) for forgery detection. Briefly, the method integrates Multi-Modal Forgery Representations (MMFR) in the LMM branch with highly effective spatiotemporal features in the ST branch. In addition, A dynamic fusion strategy is applied to aggregate these features and adjust feature weights. Our proposed In-and-Across Frame Attention (IAFA) benefits our proposed method in effectively capturing forgery features. Besides, our method maintains a strong generalization ability on unseen forgery types with the help of large multimodal models.

We provide detailed analyses and experiments for the functionality and effectiveness of our proposed MMFR and IAFA in the overall response. Specially, we provide

1. the functionality of IAFA compared with other baselines in Q1,

2. the generalization of our method on GAN and latest diffusion methods in Q2,

3. the performance of our method on different video durations and resolutions in Q2,

4. the ablation study on different LLMs for our method in Q3.

Q2: Generalization ability

We provide additional performance results on GAN and latest diffusion methods in Q2 of the overall response to prove the generalization ability of our method on other types of forgery methods.

Q3: Improvements on more forgery content

Additional performance results on GAN and latest diffusion methods in Q2 of the overall response prove the effectiveness of our method, showing that our finetuned large multimodal model specializes in forgery detection and demonstrates generalization ability on most forgery types. For adaptation to more forgery content, enlarging the number and types of datasets and fine-tuning large multimodal models contributes to more promising performance.

Dear Reviewer H1bu,

Thank you for your recognition of our work. We greatly appreciate your valuable comments.

Best regards,

Authors of Submission 2768

We thank the reviewer for the valuable and insightful feedback. We appreciate all constructive comments on the novelty, soundness, and clarity of the paper. Here, we demonstrate additional experiments and answer the weaknesses and questions mentioned in the review.

Q1: Computational analysis

As for computational requirements, our method is implemented on a single 4090 GPU with 24G memory for both training and inference. We provide further computational analysis and inference speed for our methods and other baselines in Table F.

Table F: Comparison of computational cost and inference speed

Compared with other baselines, our method reaches 6x GFLOPs and 23x Params of UniversalFD[3], which also uses a CLIP encoder for detection. As for inference speed, we conduct video-level inference at 40 fps. Regarding the computational cost and efficiency, we argue that the bottleneck of our method lies in the integration of the large multimodal model, which makes up 94% FLOPs in inference and 98% Params in total. With the improvement of LMM techniques and the appearance of more computational-friendly LMMs, such limitations will be solved. In addition, LMM inference on cloud service is available in practical deployment. Therefore, we hold the opinion that the computational cost of our method will be relieved to a large extent in the future. Further discussion is beyond the scope of our paper.

Q2: Generalization ability

We report the performance of our method on multiple GAN-based methods in Q2 of the overall response, where we provide extensive experiments and results on

1. Generalization ability on non-diffusion videos,

2. Generalization ability on evolving diffusion videos,

3. Scalability to multiple durations and resolutions.

With all promising performance, we prove that our method is effective in most cases.

Q3: Robustness analysis

We provide additional experiments for robustness analysis in Table G.

Table G: Performance of MM-Det on multiple post-processing in AUC

Our method remains effective under all such post-processing conditions.

Q4: Transferability to other modalities

Our proposed MMFR takes advantage of feature spaces from multimodal encoders and text decoders in LMMs to form an effective representation for forgery detection tasks, which is a transferable method to other media. As long as the LMM is adapted to downstream tasks in audio or text, features from the encoder and decoder stand for a strong representation of unseen synthetic types. Therefore, our method is generalizable to other modalities.

Q5: Discussion on failure cases

While our paper makes significant strides in addressing diffusion video detection, our exploration does not extensively cover partially manipulated content. For diffusion manipulations that only happen in small areas, our method may fail to capture informative features. A possible reason is that small forgery traces disappear after multiple downsampling operations in the deep network of LMMs. Future research endeavors may benefit from investigating the limitation to tackle the challenge of minor forgery detection.

References

[1] Arnab A, Dehghani M, Heigold G, et al. Vivit: A video vision transformer

[2] Xu Y, Liang J, Jia G, et al. Tall: Thumbnail layout for deepfake video detection

[3] Ojha U, Li Y, Lee Y J. Towards universal fake image detectors that generalize across generative models

We extend our gratitude to the reviewer for the insightful feedback. We appreciate the constructive positive comments on the novelty, soundness, and valuable contribution of our paper. In response, we present additional experiments and address the weaknesses and questions highlighted in the review.

Q1 As for LLM, it isn't easy to see which module represents LLM from the paper and Figure 3.

We provide a more concrete explanation here. In our MM-Det framework, we apply a Large Multimodal Model to capture multimodal forgery representation, whose modules and outputs are denoted in dark blue in Figure 3 of our paper. The composition of the LMM branch is a typical structure of a multimodal large language model, including a tokenizer and an embedding layer for text input, a visual encoder for image input, and a transformer decoder layer as the language backbone.

Q2 If LLM is decoupled and divided into several modules, the article should give explanations.

Our proposed method obtains multimodal forgery representations (MMFR) by taking advantage of pretrained visual encoder and language decoder backbone from a LMM. Specifically, given an LMM composed of a vision encoder denoted as $\\mathcal{E}_v$, a text encoder denoted as $\\mathcal{E}_t$, and a language decoder backbone denoted as $\\mathcal{D}_t$. The input is a video sequence $\\{\\mathbf{x}\\}^N$ of $N$ frames and an instruction prompt $\\mathbf{p}$ from our forgery templates. We capture informative multimodal features $\\mathbf{F}_m$ by conducting text generation and extracting hidden states from the last layer of $\\mathcal{E}_v$ and $\\mathcal{D}_t$, which are feature representations in both visual and textual embedding space. This process can be expressed as:

\\mathbf{F}_v =\\mathcal{E}_v(\\mathbf{x})\

\\mathbf{F}_t =\\mathcal{D}_t(\\mathbf{F}_v, \\mathcal{E}_t(\\mathbf{p}))\

\\mathbf{F}_m=\\{\\mathbf{F}_v, \\mathbf{F}_t\\}\

where $\\mathbf{F}_v$ and $\\mathbf{F}_t$ denote features in the visual and textual embedding space, respectively.

Q3 Illustration of the functionality of IAFA

We provide further analysis and demonstrate additional experiments in Q1 of the overall response.

Q4 An ablation study on LLMs

We provide an extensive ablation study on the influence of LLMs in Q3 of the overall response.

Dear Reviewer X8oW,

We are glad that our rebuttal has addressed your concerns. Thanks again for your valuable comments.

Best regards,

Authors of Submission 2768