Deepfake Caricatures: Amplifying attention to artifacts increases deepfake detection by humans and machines

Thank you for your comments. We respond to your concerns below.

Caption size in Figure 1.

We will modify Figure 1 and increase the size of the font corresponding to the loss for the camera ready version.

Need to make artifacts more detectable to human observers.

The main objective of our paper is not only to detect fake faces and regions, but to make them more obvious to human observers. This helps humans detect fakes early, avoid being fooled and avoid consuming fake information. The core motivation for this, as we describe in the second paragraph of our introduction and in Section 4, is that current methodologies for making a user aware that a video is fake rely on text labels in UIs (e.g. “This Video is Fake”, "Video modified by AI"), and this is suboptimal - easily missed, hardly believed, as many studies show (Introduction, paragraph 2, line 3). Making artifacts more detectable to human observers is key to increasing the chances that humans detect fakes: in fact, our proposed deepfake caricature visual indicator almost doubles human detection performance when compared to the unaided condition, and increases it by 20.5% when compared to a text-based indicator (Section 4, Paragraph 2). This further confirms the necessity for evidencing artifacts to humans instead of just showing text labels. The potential societal impact of this is large: our experiments essentially show that humans choose to not trust a text-based indicator (e.g. "Video modified by AI") 22% of the time, but only distrust our deepfake caricatures 6% of the time.

Our paper showcases, through several experiments, that humans are less easily fooled when a technique that amplifies artifacts is used to show them that a video is fake. We hope this clarifies the reason (and the importance) around why we choose to exacerbate artifacts for humans in this work.

Comparison to attention-based deepfake detection methods

We already showcase a performance comparison to the attention-based method most aligned with our framework in Table 4 (Shiohara 2022). We additionally provide comparisons to [1] in Table 1, row 7. Further, we bring comparisons with [2,3] in the Table below (cross dataset generalization, FF++ to CelebDF, showing AUC results). We will add these results in the supplemental.

In terms of functionality, the proposed attention mechanisms are substantially different from our own. Most rely on an internal measure of self consistency, while ours are sourced from large scale human annotations. Automatically generated heatmaps such as in [1, 2, 3] do not make compelling caricatures, as they highlight face boundaries or the whole swapped area, rather than individual artifacts (unlike our CariNet). Our ablation study shows the effect of simpler synthetic heatmaps (centered gaussian, Table 4) and heatmaps from a relevant work (Shiohara 2022, Table 4). Performance degrades in both cases when our artifact attention module is replaced. We will add short additional insights about attention mechanism differences to our related work section.

We hope these responses helped alleviate your concerns, and we are open to further discussion.

Thank you for your review and comments. We appreciate the recognition towards our dataset.

We respond to your comments below.

Application and Distinctiveness.

Caricatures are substantially different from satirical videos: they aim at evidencing artifacts instead of generating a satirical depiction of a character. They work by amplifying defects, highlighted by our artifact attention module, across frame embeddings, to generate a magnified version of those defects. They have a particular "uncannyness" in their motion (as can be seen in our gallery) that makes them distinct from a human-crafted satirical video. Attackers could potentially generate disturbances similar to the ones exhibited by caricatures to confuse a viewer, but the video itself would be detected as fake by our classifier. Additionally, our caricature framework ensures that only detected fake artifacts are magnified, which reduces the probability that a real video passed through our pipeline results in distortions.

Does our approach make fake videos more challenging to distinguish from other videos?

This is an interesting question - our caricatures could be making fake videos more similar to in-the-wild videos through their distortion. We propose that this effect is a) very unlikely to occur, and b) preferable to having fake videos indistinguishable from their real counterparts. First, the distortion applied by our caricatures are very particular: because they stem from a magnification of embedding differences gated by an attentional heatmap, they have a very specific signature that makes them substantially different to other videos in the wild that have been distorted in one way or another. This means that generally, caricatures will not become indistinguishable from another random subset of videos. Second, in the unlikely scenario where they do become hard to distinguish from another set of real distorted videos, we argue that it is still preferable to have fake videos in that state than to disseminate the perfectly crafted fake, where the alteration is made precisely to fool people into thinking they represent real faces (more likely to spread misinformation).

Qualitative Results

We will ensure that the main manuscript showcases more qualitative examples - thank you for your suggestion. We will add a figure to the appendix presenting more examples. We do highlight, however, that the caricatures are much better appreciated in video form. This is why we decided to use the limited space in the main manuscript to properly showcase our experimental results and figures, leaving most of the visual qualitative exploration to a better suited web gallery.

We hope these comments help address your concerns.

Thank you for your review. We respond to your main concerns below.

Bias of human intuition. We agree that human intuition is biased. We claim, however, that artifact annotations collected by many different humans over many videos do provide strong signal to improve detection and generate effective caricatures. Most experiments in our paper aim at showcasing the usefulness of our artifact attention module and caricature module, both built on human annotations at scale. We repeatedly show the advantage of these human annotations:

1. They can improve classification through supervising our artifact attention module, as CariNet beats 13 previous works over 4 datasets and 3 tasks. Our ablation studies in Table 4 further show that using the artifact attention module is better than not using it or replacing it with a different type of attention - crucially, using attention alternatives that are not derived from our human annotations degrades performance.
2. They can help humans recognize deepfakes better: Section 4 and Figure 5 show that caricatures, which are a direct product of artifact attention heatmaps, greatly improve human accuracy at the task of recognizing fakes. This improvement is important: humans become 20.5% better at picking up deepfakes with our caricatures (and thus, with the help of our human annotations) than when exposed to text based indicators (e.g. "THIS VIDEO IS FAKE" or "Video Modified by AI")

We recognize that human intuition can be biased, but as we show in this work, using large quantities of human annotations of fake regions improves performance unequivocally, across tasks and across domains. We agree that this result may seem surprising, but we believe that that makes it even more important for the community.

Novelty. Our framework introduces a larger variety of novelty points than the ones constrained to our attention mechanism. We present the following innovations:

• a) The integration of artifact attention maps into self-attention structures by directly gating the keys, which to the best of our knowledge has not been previously introduced in any current attention mechanisms (Figure 4B).
• b) The targeted embedding distortion framework using heatmaps (equation 3) which allows us to amplify specific defects in a video with precision, hijacking the residuals of contiguous time-sliced coded representations built by our encoder (Figure 2).
• c) Our specific combination of human supervision with machine supervision, leveraging 4 different losses at two key points of the network (to the best of our knowledge, a novel combination of losses), before feeding back the output of one task (artifact attention heatmap reconstruction) as input to the other (classification).

Additionally, our paper introduces two novel human annotation datasets (artifacts labeled on DFDCp and FF++) totaling more than 20k annotations on well-known benchmarks, novel visual indicators (Caricatures) as well as the framework to make them, and multiple human experiments of our own design that showcase the utility of our caricatures.

We hope this helps shed light on the strength of our contributions and the novelty they exhibit, as they go beyond the architectural choice of one part of our backbone.

Focus on generation and role of caricatures. We respectfully disagree with the statement that caricatures are closer to "a way of utilizing human abilities to provide auxiliary samples". First, our caricatures are not used as auxiliary samples: the classifier never sees caricatures, and instead leverages the artifact attention heatmaps produced by our artifact attention module (Figure 1A, Figure 4B). Second, our caricatures are much more than auxiliary samples: they are a novel visual indicator that is 20.5% more effective at evidencing deepfakes to humans than text indicators (Section 4, paragraph 2). That is, when humans are confronted with a fake video, distorting that video with our caricature framework increases hit rates to 0.94 (vs. 0.78 in text based indicator conditions, p<0.0001 for both), making it much more likely that humans will correctly recognize it as fake than showing text that says "THIS VIDEO IS FAKE". We believe this result alone makes caricatures particularly impactful.

Thank you for your comments. We really appreciate your thoughts on the experiments!

Here are our responses to the weaknesses raised:

Amplified Representations. We tried feeding the caricatures themselves to the classifier in early experiments, but that structure does not improve performance - the caricatures are distorted when they reach the classifier and not helpful enough for that module to be able to effectively utilize it. We did some initial experiments with CariNet-S in earlier stages of the project, and obtained an AUC of 64.5 on DFDCp - compare that to 72.9 when using artifact attention maps. We hypothesize that the signal of artifact location gets mixed with the movement of the pixels themselves. The artifact attention heatmaps, however, are consequential, as shown in our experiments - they are better than attention maps from contemporary papers (Siohara 2022) and synergize well with the modified attention layers of our classification module.

Caricature Evaluation. The caricatures are evaluated through our extensive human experiments: we show how humans exposed to caricatures become better at detecting fake videos in Section 4 and Figure 5. We compare to scenarios with no caricatures and with text labels, and observe a significant difference in hit rate when using Caricatures as the visual indicator of fakeness.

Our caricature system will typically not affect a real video, as showcased in our gallery. However, an attacker could hijack our caricature generation module and force a caricature on a real video. In that case, humans could indeed consider that the video is fake. This impression of fakeness could happen in many scenarios however: people will think a video is fake when the lighting is weird, when the people are funny-looking, when the video quality is low, when the subject matter clashes with their beliefs, etc. Given this, to truly answer the question of how often caricatured videos make humans think real videos are fake, we would first need to measure how often this happens without caricatures in the mix, which is out of the scope of this study.