LOKI: A Comprehensive Synthetic Data Detection Benchmark using Large Multimodal Models

W1: Limited Performance in Certain Modalities.

Thank you for your detailed feedback. The limited performance of models in certain modalities aligns with the conclusions we reached in Section 4.2 of the paper, particularly regarding performance in 3D and audio modalities. In Appendix D.3 and D.4, we provide detailed analysis of such phenomenon:

• Compared to image and text modalities, existing LMMs exhibit significantly poorer performance on 3D and audio modality tasks. A key reason for this disparity is the relative scarcity of 3D and audio data in the training datasets of current LMMs.
• Currently, there are very few LMMs that support audio or 3D modalities, and most perform poorly on synthetic detection tasks. As noted in Appendix D.3, previous studies (Liu et al., 2024b) have shown that acoustic features are more critical than the audio content itself for detecting deepfake audio. However, the architecture and training data of current audio-language models are primarily focused on content understanding rather than capturing acoustic features, which is likely a key reason for their subpar performance.

Our perspective aligns closely with your views, which is also one of our key findings. Furthermore, we would like to clarify that the poor performance of LMMs on these modalities reflects the limitations of the existing models, not a weakness of the LOKI benchmark.

W2: Insufficient Details on Data Generation Methods.

We appreciate the reviewer’s attention to the details of the data synthesis methods. Our data collection process is described in detail in Appendix B.1. Our data primarily originates from online internet collections, reused from public datasets, and self-synthesized into new composite data. In this Table 6 of the Appendix, we provide detailed descriptions of the data sources for each modality and the models used for generating the data. To ensure diversity in synthetic data, each modality incorporates more than five different synthesis methods. Figures 9 and 10 also showcase additional examples of synthetic data. Importantly, we ensured the inclusion of the latest or SOTA models, which demonstrate outstanding performance in their respective fields, such as Video-Sora, 3D-CLAY, Text-GPT-4o, Image-FLUX, and Audio-Suno. In Appendix B.4, we provide additional explanations regarding the SPECIAL DATA synthesis methods, including document images, remote sensing imagery, and medical images. We believe the detailed descriptions in the supplementary materials will facilitate a better understanding of the data synthesis methods.

Q1: Quality and Realism of Synthetic Data.

We fully understand the reviewers' concerns regarding the quality of synthetic data. Below, we analyze this aspect from four perspectives: data synthesis methods, data categories and annotations, diverse task design, and the real-world relevance of the data.

Latest Multimodal Synthetic Data Generation Methods.

As outlined in Weakness 2, we ensure the inclusion of the latest or SOTA models, all of which demonstrate exceptional performance in their respective domains. These advanced synthetic data generation methods effectively reflect the latest progress in generative models, ensuring the production of high-quality synthetic data.

Heterogeneous Categories and Multi-level Annotations.

LOKI integrates a wide range of recently popular synthetic modalities, including video, image, 3D data, text, and audio, ensuring comprehensive data coverage. The dataset spans 26 detailed categories across these modalities. Moreover, LOKI provides both basic "synthetic or real" labels and fine-grained annotations of anomalous regions.

Diverse task design to evaluate the model's capabilities.

Our task design is diverse and challenging (as detailed in Section 3.3 of the paper), including judgment, multiple-choice questions, anomaly detection, and anomaly explanation. The LOKI dataset’s task design extends beyond traditional binary classification problems to include complex tasks requiring deep understanding and reasoning. Models must not only classify synthetic data but also identify global or local anomaly locations and generate reasonable natural language explanations.

Real-world relevance of the data.

LOKI includes heterogeneous data covering various application domains such as medical imaging, remote sensing, and natural scenes (as shown in Figure 2). This design ensures that the dataset can simulate complex real-world scenarios, making it closely aligned with practical application needs. This fine-grained design ensures that the dataset is suitable for various real-world scenarios and provides researchers with a highly realistic and challenging evaluation environment.

W3: The paper mentions that the Chain-of-Thought (CoT) approach can impact model performance in image and 3D reasoning tasks. However, it does not provide sufficient experimental details to clarify whether CoT significantly enhances performance across all types of tasks or if it is only effective for the specific tasks currently evaluated.

In LOKI, we primarily focused on 3D and image modalities when exploring the impact of different prompting strategies on model performance. This focus was chosen because these modalities provide sufficiently annotated data and diverse question types. However, due to the large sample size in the video modality and the high cost of conducting comprehensive prompting tests on closed-source models, we did not include video modality in our initial experiments.

To address your concern and further validate the effectiveness of CoT across various question types and modalities, we conducted additional experiments. These include testing on multiple question types within the image modality (e.g., multiple-choice and anomaly selection) and extending the evaluation to video and text modalities (e.g., binary classification tasks). The results are summarized in this table.

From the results, we observe that for most models, CoT significantly improves performance across different question types and modalities. While LLaVA-OV-7B struggles under CoT settings, other models, such as InternVL2-8B, Qwen2-VL, and GPT-4o, consistently benefit from CoT prompting, with noticeable performance gains across both tasks (e.g., anomaly detection, binary classification) and modalities (e.g., image, video, text).

These additional experiments strongly support the effectiveness of CoT prompting across various task types and modalities. We have included more details about the CoT implementation and additional results in Appendix E.3 (highlighted in blue).

W4: It is suggested to discuss and compare more related works such as [1,2] in this paper.

We appreciate the reviewer’s suggestion and acknowledge the important contributions of [1] and [2] in advancing the detection and grounding of multi-modal media manipulation. These works tackle a critical problem by not only detecting manipulated media but also localizing manipulated content, which provides valuable insights for reasoning across visual and textual modalities. Similar to [1] and [2], our work also aims to push the boundaries of model performance in detecting artificial data and emphasize the importance of reasoning in multi-modal tasks.

However, our work differs in its broader scope. While [1] and [2] concentrate on manipulation detection and grounding, LOKI focuses on synthetic data detection across a wider range of modalities, including video, audio, and 3D. Additionally, LOKI also introduces tasks such as anomaly selection and natural language explanations, allowing for a comprehensive evaluation of LMMs' reasoning and explainability. Once again, we would like to thank the reviewer for the reminder. We have included the citations in the revised version to confirm the relevance of [1] and [2] (highlighted in blue).

Q2: Could you provide additional insights into why Claude-3.5-Sonnet tends to misclassify synthetic images as real? For instance, does this result from limitations in the model's ability to recognize fine-grained abnormalities, or are there certain types of synthetic image features that are particularly challenging for the model?

The anomalous performance of Claude-3.5-Sonnet has indeed also drawn our attention. In the Abnormal Explanation experiment, we indeed observed a particular behavior from Claude. Specifically, in the image modality, when the model was prompted to explain anomalous phenomena (as illustrated in Figure 12), Claude frequently (98.3% of the time) outright rejected the premise of the task. A typical response would be: "Apologies, but I must disagree with your premise. Upon careful examination, this image does not exhibit any features indicative of being artificially generated or forged."

It is worth noting that this issue was independent of the type of input image or the nature of the anomaly. We suspect that this behavior arises from biases introduced during Claude’s post-training for safety alignment [1]. To prioritize "safety," Claude appears to avoid making definitive judgments in tasks involving AI-generated content, opting instead to decline participation as a risk-avoidance mechanism. While this approach ensures "safe" responses, it significantly undermines the model’s performance in this task. However, this limitation does not appear to be triggered in other modalities. We would like to note that, since we cannot bypass Claude's safety constraints, the performance of Claude on such tasks is likely underestimated in this context.

Reference:

[1] Anthropic. Claude 3.5 Sonnet URL: https://www.anthropic.com/news/claude-3-5-sonnet

W1: The current evaluation mainly relies on accuracy and NBI; however, at low recall rates, NBI may not adequately reflect model bias. Additionally, the design of NBI may be insufficient to comprehensively capture various types of bias exhibited by the model.

We appreciate your concerns about the reliability and comprehensiveness of the current NBI design under low recall rates. After a thorough review, we have critically reflected on the rationale behind the NBI formula proposed in Appendix C.2 of the paper, particularly its accuracy in scenarios with low recall rates.

1. Clarification of the Reasonableness of the Current NBI Formula

We would like to clarify that the current NBI formula is inherently reasonable across different recall rates, though there is room for improvement. Specifically, we provide the following mathematical discussions:

• When $R{\text{natural}}$ and $R{\text{generated}}$ both approach 0, it indicates that the model has completely reversed the real and fake classifications. We consider this scenario highly unlikely to occur.
• When either $R{\text{natural}}$ or $R{\text{generated}}$ approaches 0, the existing NBI formula will always result in a significant bias toward the other class, which is quite reasonable.
• In other cases, the original NBI formula can be rewritten as:

This demonstrates that the current NBI formula is not inherently inadequate in reflecting model bias under low recall rates. Instead, it fundamentally depends on the ratio of recall rates between real and generated data. We plotted the graph of the existing NBI function in this figure.

2. Ensuring Unbiased Evaluations of LMMs

We also wish to emphasize that in evaluating LMMs, we took rigorous precautions at multiple levels (e.g., data and task design) to prevent the introduction of extraneous biases, ensuring that the observed bias originates from the LMMs themselves. Detailed clarifications are as follows:

• Data Level:
  • We clarify that the LOKI benchmark proposed in this work minimizes prior biases. As outlined in Appendix B.1 Table 6, LOKI contains diverse sources of both real and synthetic data (e.g., multiple generative models and public datasets), ensuring data diversity and mitigating the risk of single-distribution dominance.
  • Additionally, for each modality, we ensure that real and synthetic data originated from identical domains to prevent any apparent content biases. This guarantees consistency and ensures that real and synthetic data share a comparable feature space.
  • Finally, we aimed for balanced quantities of real and synthetic data within each modality (e.g., for images). For modalities with imbalanced data quantities (e.g., 3D or video), we manually adjusted evaluation metrics to ensure balance by separately computing results for real and synthetic data and averaging the two, thereby eliminating biases caused by data imbalance.
• Task Design Level:
  • We acknowledge the potential influence of different prompts on LMM responses. To eliminate biases introduced by task design, we implemented positive and negative questioning during LOKI evaluation. For every input, we posed a positive query (e.g., asking whether the input is real data) and a negative query (e.g., asking whether the input is synthetic). The final result was derived as the mean of the two queries, effectively eliminating biases arising from differences in task design.

3. Future Directions

We sincerely appreciate your valuable feedback regarding the NBI design. As you suggested, biases exhibited by models may manifest at multiple granularities. In future work, we plan to design more targeted metrics to evaluate LMMs' biases at various fine-grained levels. Additionally, we aim to conduct further instance-based analyses to explore the underlying causes of biases across different granularities.

The author has addressed most of the concerns. The reviewer increases the score and is inclined towards acceptance of the paper.

W3: As deepfake detection is one of the most prominent synthetic data detection categories today, I believe the benchmark would benefit from its inclusion.

We appreciate the reviewer's constructive suggestion regarding the inclusion of the deepfake detection category[1][2]. We would like to clarify that our benchmark LOKI already includes partial deepfake datasets from audio and image modalities. However, we did not perform a standalone analysis for the Deepfake category. The existing data for the Deepfake category includes:

• Audio Modality: In audio tasks, some samples of speech and singing are sourced from their respective audio deepfake datasets ASVSpoof2019 and DCASE2023 Track 7，as mentioned in Table 6. These datasets are built on state-of-the-art voice cloning techniques, such as Text-to-Speech[3], Voice Conversion[4], and Singing Voice Conversion [5]. These technologies address major challenges in deepfake detection, including vulnerabilities in voice authentication systems and the misuse of synthetic voice technology to generate counterfeit content, such as fake artists.
• Image Modality: As shown in Table 6, our image data includes a portion of facial deepfake images, comprising partial datasets from FFHQ and Deepfakeface. These data are directly aligned with the core challenges in image-based deepfake detection, ensuring that our benchmark comprehensively evaluates models on this critical aspect of synthetic data detection.

In response to your suggestion, we have independently evaluated the deepfake-related components of our current dataset and presented the results below. From the results, it is evident that most models perform significantly worse on deepfake data compared to their overall accuracy. This discrepancy likely arises because the overall evaluation task includes simpler synthetic samples (e.g., basic synthetic data), which are easier for the models to distinguish, thereby inflating their overall accuracy. In contrast, deepfake data often involves more nuanced and fine-grained forgery features that are particularly challenging for general-purpose multimodal pretrained models to capture effectively. Interestingly, GPT-4o achieves the best performance on deepfake data among all evaluated models, demonstrating its relatively stronger ability to handle such complex forgeries.

To emphasize the importance of deepfake data, we have established it as a separate detection category and included detailed model results in Appendix E.2 (highlighted in blue). We sincerely thank the reviewers once again for their constructive feedback and reminders.

References:

[1] Wang, T., Liao, X., Chow, K. P., Lin, X., & Wang, Y. (2024). Deepfake Detection: A Comprehensive Survey from the Reliability Perspective. ACM Computing Surveys, 57(3), 1–35. https://doi.org/10.1145/3699710

[2] Pei, G., Zhang, J., Hu, M., Zhang, Z., Wang, C., Wu, Y., Zhai, G., Yang, J., Shen, C., & Tao, D. (2024). Deepfake Generation and Detection: A Benchmark and Survey. arXiv. https://arxiv.org/abs/2403.17881

[3] Ren, Y., Ruan, Y., Tan, X., Qin, T., Zhao, S., Zhao, Z., & Liu, T. Y. (2019). Fastspeech: Fast, robust and controllable text to speech. Advances in xinkneural information processing systems, 32.

[4] Sisman, B., Yamagishi, J., King, S., & Li, H. (2020). An overview of voice conversion and its challenges: From statistical modeling to deep learning. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 29, 132-157.

[5] Nachmani, E., & Wolf, L. (2019). Unsupervised singing voice conversion. arXiv preprint arXiv:1904.06590.

W2: It is unclear to me, how a user can methodically compare scores for different models across tasks/categories (e.g., in Table 2); perhaps the authors can address this, given the heterogenous and imbalanced nature of the data modalities and tasks, as well as the problem/domain "difficulty".

Since LOKI involves a large number of modalities and diverse task types, with each modality further encompassing heterogeneous data, methodically comparing scores for different models across tasks/categories is indeed a significant challenge. We sincerely appreciate the reviewers' concerns on this issue. Below, we provide additional elaboration based on existing metrics, potential future improvements, and detailed supplementary results.

1. As detailed in Appendix C.2, average accuracy is the primary metric for judgment, multiple-choice, and abnormal detail selection tasks. Table 2 in the main text evaluates the performance of large models across multiple modalities and tasks using average accuracy as the unified metric. This evaluation approach follows the metric design used in previous comprehensive benchmarks, such as MMMU and MMT-Bench. For instance, MMMU assesses capabilities across diverse disciplines and tasks, while MMT-Bench evaluates over 150 different subtasks. Both benchmarks calculate accuracy based on whether model responses align with the ground truths. This unified metric design ensures consistent evaluation across tasks, avoids task-specific metrics, and provides a standardized basis for comparison, which is a common strategy for assessing the multi-task and general capabilities of large multimodal models.
2. The reviewers’ suggestion to consider the “difficulty” of different problems or modalities as a factor for balancing model score comparisons is highly insightful. In Table 4, we have highlighted the evaluation performance across different modalities at varying difficulty levels. Additionally, we have supplemented the results with more tasks across finer-grained difficulty levels, as shown in this Table , and incorporated these results into the revised Appendix E.4 (highlighted in blue). As part of our future plans, we also intend to explore difficulty-weighted scoring for models to further improve the evaluation of comprehensive model capabilities and provide a more detailed summary of their strengths and weaknesses.
3. It is worth noting that, in addition to the more general overall evaluation presented in Table 2, we have provided more fine-grained evaluation results in supplementary materials. These results detail model performance under different tasks and modality divisions, enabling a more comprehensive assessment of model capabilities:

• Video Modality: Tables 7 and 8 respectively present the model's performance in judgment tasks and multiple-choice tasks under different video synthesis methods.
• Image Modality: Tables 10 and 11 illustrate the model's performance in selection and judgment tasks across different types of images.
• 3D Modality: Tables 13 and 15 evaluate the model's performance under different 3D synthesis methods (e.g., NeRF and GS) and point cloud input formats.
• Audio Modality: Table 16 demonstrates the model's performance across various audio types.
• Text Modality: Tables 17 through 19 assess the model's capabilities across different topics, text lengths, and both Chinese and English scenarios.

W1: While the differentiated modalities, categories and annotation levels are beneficial, the overall size of the dataset actually seems relatively small vis-a-vis related datasets (Table 1).

Thank you for your concern regarding the scale of our dataset. Table 1 provides a detailed comparison of LOKI with existing datasets, including traditional synthetic detection benchmarks and those tailored for evaluating LMMs.

1. Traditional synthetic detection datasets, such as Fake2M (Lu et al., 2023b) and HC3 (Guo et al., 2023), are primarily designed for training expert detection models. As such, their reported sizes include both training and testing data. In contrast, LOKI is specifically designed to test the capabilities of multimodal large models (LMMs), and its reported size exclusively represents the test set. Furthermore, these traditional datasets often use binary ground truth labels , eliminating the need for fine-grained annotation of anomalous regions as required in our dataset, thus facilitating the construction of larger-scale datasets.
2. Compared to other LMMs' synthetic detection benchmarks like FakeBench (Li et al., 2024b)(6K) and VANE (Bharadwaj et al., 2024)(0.9K) in the middle section of Table 1, LOKI has a clear quantitative advantage in size (18k) , which further demonstrates our advantages as a robust benchmark for evaluating synthetic data detection across a variety of scenarios.

In addition, the table below further provides a comparison between LOKI and other widely-used benchmarks for evaluating LMMs, including MMMU (Yue et al., 2024) (11.5K), MME[1] (2.37K), and MMBench[2] (6.67K).

In summary, we believe LOKI’s scale is sufficient to reveal model strengths and limitations, particularly in detecting multimodal synthetic data. Thank you for raising your concerns.

References:

[1] C. Fu et al., "MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models," arXiv preprint arXiv:2306.13394, 2024. [Online]. Available: https://arxiv.org/abs/2306.13394.

[2] Liu, Y., et al. (2025). MMBench: Is Your Multi-modal Model an All-Around Player? In Leonardis, A., Ricci, E., Roth, S., Russakovsky, O., Sattler, T., & Varol, G. (Eds.), Computer Vision – ECCV 2024. Lecture Notes in Computer Science (Vol. 15064).

W: The benchmark lacks a robustness test against common real-world conditions like compression artifacts. To enhance real-world applicability, the authors could include performance evaluations on compressed data.

Thank you for your valuable suggestion. We completely agree that incorporating robustness tests under real-world scenarios, such as compression artifacts, is essential for better evaluating the applicability of multimodal large models (LMMs) in practical settings. Following previous studies[1][2]，we conducted JPEG compression tests on judgment tasks under the image modality, based on three representative LMMs and an expert model AIDE，as previously introduced in the main text.

From these results, we observe that expert models AIDE[3] are more significantly impacted by compression artifacts. In contrast, LMMs demonstrate remarkable robustness, with compression levels having little to no effect on their detection capabilities. This distinction highlights the inherent advantages of LMMs over traditional expert models. There are several possible factors that contribute to the robustness of LMMs to compressed images.

1. Extensive Pretraining on Diverse Data: The vast amount of data available on the internet includes images with varying levels of compression. During pretraining, LMMs inherently learn features that are robust to such artifacts by including the internet-scale data, as demonstrated in works like UniversalFakeDetect[2] and CLIPping[4] .
2. Model Scale Advantage: Unlike traditional expert modules, which typically operate with parameter counts in the millions, LMMs are equipped with parameters in the billions. This scale significantly enhances their generalization and robustness, consistent with the emergent properties observed in large-scale models.

This finding further underscores the potential of LMMs for real-world applications, particularly in scenarios involving compression artifacts. We have included these experimental results and compression robustness analysis (highlighted in blue) in Appendix E.1, to provide a comprehensive understanding of the reliability and applicability of LMM. If you have additional questions, we welcome further discussion.

References:

[1] Wang, S. Y., Wang, O., Zhang, R., Owens, A., & Efros, A. A. (2020). CNN-generated images are surprisingly easy to spot... for now. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (pp. 8695-8704).

[2] Ojha, U., Li, Y., & Lee, Y. J. (2023). Towards universal fake image detectors that generalize across generative models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 24480-24489).

[3] Yan, S., Li, O., Cai, J., Hao, Y., Jiang, X., Hu, Y., & Xie, W. (2024). A sanity check for ai-generated image detection. arXiv preprint arXiv:2406.19435.

[4] Khan, S. A., & Dang-Nguyen, D.-T. (2024). CLIPping the deception: Adapting vision-language models for universal deepfake detection. In Proceedings of the 2024 International Conference on Multimedia Retrieval (pp. 1006–1015).

Dear Reviewer xU39,

We would like to once again express our sincere gratitude for your valuable comments and feedback. As the Rebuttal period is approaching its end, we kindly remind you that if you have any further questions or concerns, please do not hesitate to reach out to us at your convenience.

Thank you once again for your time and consideration.