DEFAME: Dynamic Evidence-based FAct-checking with Multimodal Experts

We thank for the time invested into the review. Please find our response to your concerns below.

First to handle multimodal claims and evidence

The work by Tahmasebi et al. (CIKM 2024), i.e. LVLM4F, was covered in our paper, most notably in the prior work overview (Table 1) and as a baseline in Table 3. Critically, LVLM4FV is not designed to handle multimodal claims: “Given a textual claim 𝑄 and a corpus of evidences C as input [...]”, p. 3.

Key novelty

Please refer to our response to Reviewer bvvH (“Originality and Theoretical Contribution”) for more details on the novelty of DEFAME.

Prior work

Thanks for pointing out SNIFFER. Please refer to our response to Reviewer K7Ta (“Comparison to Prior Work”).

”Incomplete” Table 3

The other methods in Table 3 have not been evaluated on all benchmarks for several methodological reasons. Most importantly, the methods have different task specializations, targeting only a particular subtask of fact-checking: CFR, GPT-CoT, LVLM4FV, and MetaSum are limited to text-only claims (not provided by VERITE and partially CR+). CHASMA and AITR focus solely on OOC detection and both require visual input, making them inapplicable to text-only claims (like in AVeriTeC, MOCHEG, and partially CR+). For CFR, the code and model weights have not been publicly released. GPT-CoT requires gold evidence as input to achieve the reported competitive numbers, not available for VERITE and CR+, where evidence retrieval is considered an integral part of the fact-checking task. Moreover, GPT-CoT mainly builds on GPT-3.5-Turbo, implying that the much stronger GPT-4o (CoT) baselines provide a better comparison than GPT-CoT. CHASMA lacks a publicly available, trained model. Furthermore, CHASMA and AITR both require training - which is impossible for CR+ as there is no training data available. LVLM4FV, MetaSum, and AITR require a predefined evidence corpus for retrieval, which is infeasible to create for CR+.

Thus, it is not possible to run the previous methods on the benchmarks they were not designed for (or if it is technically feasible, it will produce meaningless results). With DEFAME, we deliver a method that - despite its generality - is able to beat even the specialized methods.

Inconsistency in metrics

Metrics (F1, accuracy) match those used in the original benchmarks and prior work for consistent intra-benchmark comparison. If the reviewer finds it helpful, we are open to adding F1 score as an additional metric to AVeriTeC and CR+ or, alternatively, add accuracy to MOCHEG.

Efficiency

We agree that DEFAME has high token usage and acknowledge that there is room to reduce it. As correctly anticipated, the high token consumption is due to the processing of external evidence. External evidence can incorporate entire webpages, full PDFs, and other long documents that Firecrawl can turn into Markdown representation. We truncate inputs only when they exceed the maximum context window of the MLLM, which is 128k tokens for the GPT models used. Since the GPT baselines do not process any external evidence, unsurprisingly, their token consumption is fairly low - at the cost of random-like performance for unseen claims. Table 7 also reveals that the integration of planning (DEFAME with GPT-4o) reduces token consumption by about 20K tokens (almost a third) compared to the planning-ablated variant “Static Actions,” which executes all available actions. Since efficiency was not a goal of DEFAME, we offer to add it as a future direction to the discussion section.

It is hard to compare DEFAME’s efficiency with the methods in Table 3 due to (a) no reported resource usage (e.g., tokens or FLOPs), and (b) significant differences in architecture; also many are specialized for sub-tasks like OOC detection or evidence summarization and do not perform retrieval. Thus, direct efficiency comparison is not meaningful.

Claim on Novelty / Related Work

Thanks for pointing out SNIFFER, MMD-Agent (from the MMFakeBench paper), and LEMMA. Please refer to our response to Reviewer K7Ta.

No Theoretical Claims

We believe that absence of theoretical claims is not unusual for papers under “Application-Driven Machine Learning”, the ICML topic that we have submitted to. At the same time, we offer a more formal mathematical representation of the framework below.

No Supplementary Material, Details for CR+

Please refer to Appendix Sections A–L (pp. 16 ff.), which contain extensive additional material. Some of your questions are addressed in Appendix J. We also point you to our response to Reviewer K7Ta on “ClaimReview2024+ Details”, where these details are further elaborated.

Originality and Theoretical Contribution

DEFAME’s originality lies in its dynamic, modular integration of retrieval tools for handling both multimodal claims and multimodal evidence—capabilities not jointly supported by prior work. While it builds on established components like CoT prompting and tool use, DEFAME combines them into a unified framework that supports zero-shot fact-checking across a diverse set of benchmarks: AVeriTeC (text claims & text evidence), MOCHEG (text claims and multimodal evidence), VERITE (multimodal claims and potentially multimodal evidence), ClaimReview2024+ (both uni- and multimodal claims with potentially multimodal evidence). To our knowledge, it is the only system capable of operating across all these scenarios, while previous methods are specialized and typically incompatible with at least one modality setting (e.g., requiring only text, only images, or static corpora). We will make this contribution more explicit.

Formal Representation of DEFAME

We appreciate the suggestion to formalize the DEFAME pipeline. We agree that a formal view helps clarify the role of each stage.

Let $\mathcal{T}$ and $\mathcal{I}$ be the spaces of text and images. Define $\mathcal{M} := (\mathcal{T} \cup \mathcal{I})^*$ as the space of multimodal sequences, and $\mathcal{Y}$ the space of verdict labels. DEFAME is a function:

$$\mathcal{F} : \mathcal{M} \rightarrow \mathcal{M} \times \mathcal{Y}, \quad \mathcal{F}(c) = (R_\text{out}, y_\text{out}),$$

where, given a claim $c \in \mathcal{M}$, the output consists of a report $R_\text{out}$ containing the fact-check, and a predicted verdict $y_\text{out}$. DEFAME proceeds iteratively up to $N$ steps as follows:

• $(R^{(i+1)}, y^{(i+1)}) := \mathcal{F}_\text{iter}(R^{(i)})$
• $i^* := \min \\{ i \leq N \mid y^{(i)} \ne \text{NEI} \text{ or } i = N \\}$

Final outputs:

• $R_\text{out} := \mathcal{S}_6(R^{(i^*)})$ (justification)
• $y_\text{out} := y^{(i^*)}$ The justification stage $\mathcal{S}_6$ appends a rationale to the final report.

Each iteration $\mathcal{F}_\text{iter} = \mathcal{S}_5 \circ \mathcal{S}_4 \circ \mathcal{S}_3 \circ \mathcal{S}_2 \circ \mathcal{S}_1$ consists of:

1. Planning ($\mathcal{S}_1$): Select actions $A \subseteq \mathcal{A}$ based on $R^{(i)}$
2. Execution ($\mathcal{S}_2$): Retrieve evidence $E := \\{ \tau(a) \mid a \in A \\}$, where $\tau$ is a tool executing corresponding action $a$.
3. Summarization ($\mathcal{S}_3$): $R_1^{(i)} := \sigma(E, R^{(i)})$, where $\sigma$ summarizes evidence $E$ conditioned on the current report $R^{(i)}$ and appends it to the report.
4. Develop ($\mathcal{S}_4$): $R_2^{(i)} := \mathcal{S}_4(R_1^{(i)})$, where $\mathcal{S}_4$ is a generative model that performs structured reasoning and expands the report with the generated NLI sequence.
5. Verdict Prediction ($\mathcal{S}_5$): $(R_3^{(i)}, y^{(i)}) := \mathcal{S}_5(R_2^{(i)})$, where $\mathcal{S}_5$ is a classifier over multimodal sequences, returning a verdict $y^{(i)} \in \mathcal{Y}$ and an updated report $R_3^{(i)} \in \mathcal{M}$ which is the input report $R_2^{(i)}$ expanded by a summary of the key takeaways from the report alongside with the verdict.

We welcome feedback on whether including this formalization (or an extended/abbreviated version) in the main paper would be helpful.

We value the time invested in the review and thank for the feedback and suggestions! Please find our response to your concerns below.

Comparison to Prior Work

Thanks for pointing out SNIFFER, a related work that was unintentionally omitted. Critically, unlike DEFAME, SNIFFER is incapable of retrieving multimodal evidence (“we input both the news caption and the text from webpages retrieved [...] into the LLM”, p. 5). It performs no planning. SNIFFER has only one tool (entity extraction) that is always executed. It cannot dynamically decide to retrieve additional evidence. SNIFFER is limited to news content and does not generalize beyond OOC detection. It is inapplicable to text-only inputs. Additionally, it requires finetuning - which DEFAME does not.

MMD-Agent was missing from our initial submission as it was not peer-reviewed back then, but we are happy to integrate it now. MMD-Agent does not incorporate visual evidence. Specifically, Figure 4 and the published code indicate that the gathered evidence is text-only. The paper only mentions Wikipedia as the source of external evidence (pp. 7 and 10). Additionally, their pipeline is static and does not allow for follow-up evidence retrieval. It does not include systematic planning. MMD-Agent is restricted to news. Finally, the outputs lack justifications and, therefore, the overall explainability of MMD-Agent is limited.

You mentioned the work by Tahmasebi et al. (CIKM 2024). The method introduced there is referred to as LVLM4F, which our paper covers in detail, most notably in the prior work overview (Table 1) and as a baseline in Table 3. Critically, LVLM4FV is not designed to handle multimodal claims: “Given a textual claim 𝑄 and a corpus of evidences C as input [...]”, p. 3.

Finally, we did not include LEMMA for the following reasons. First, it is not peer-reviewed. Second, LEMMA only retrieves textual evidence: The "Vision Evidence" in Figure 4 of the LEMMA paper refers to "a list of web page's title" (p. 6). These web pages were retrieved via reverse image search using the input image - which is probably why they call it “vision evidence.” Third, it remains unclear if LEMMA applies to text-only inputs. Finally, it cannot retrieve further evidence after the first pass, does not involve any efficient tool use planning, and does not provide any comprehensible justification generation, limiting its explainability.

Considering all four references, our claim remains valid that no prior published MAFC method can handle both multimodal claims and multimodal evidence.

ClaimReview2024+ Details

Complementary to Appendix J, we are happy to add more details on CR+ in the following: Claims were collected via the Google Fact-Check API by issuing queries across ten broad topics (climate change, politics, health, …). We deduplicated results based on the review URL to avoid overlap. For each claim, we also collected the date and author (claimant) to preserve context. Label assignment was handled in two stages: Trivial labels were automatically mapped using an LLM-based script (code included in the release), while non-trivial cases were manually annotated by a PhD-level MAFC researcher. A full validation pass was conducted by a student who compared extracted content (text, label, date, claimant, image) with the original fact-check articles to ensure accuracy. Images were manually curated since the Google API only returns teaser images, which often contain overlays or composites. Manual curation ensured that claim images are as close as possible to the original ones referenced in the fact-check. Because the Google Fact-Check API aggregates claims from leading organizations such as Snopes, PolitiFact, and AFP—whose focus is on timely and harmful misinformation—we believe CR+ reflects real-world misinformation trends. We will clarify these details in the main paper for greater transparency. Thank you for the feedback.

AVeriTeC metric

We used accuracy as the main metric for AVeriTeC to align with recent follow-up work, where accuracy has become a common evaluation metric for veracity prediction [Singhal et al., 2024; Cao et al., 2023]. The original AVeriTeC paper conditions veracity scoring on alignment between retrieved and gold evidence, which can confound evaluation of general-purpose systems. Following subsequent work, we omit this step to enable broader comparability and support systems that retrieve evidence freely.

Evaluation “Not Comprehensive”

You point out that “evaluation of the proposed method is also not comprehensive” and refer to section "Claims and Evidence." There, however, you write that “the claims made in the submission are supported by clear and convincing evidence.” Please clarify.

We thank for highlighting that the “paper has great potential” and are grateful for pointing out the usefulness of the information provided in the appendix. We appreciate the time invested into the review and are happy to address the concerns in the following.

Summarize, Develop, and Justify Stages

• The summarize stage serves several roles. Efficiency and Comprehensiveness: Without it, retrieved web evidence (entire web articles, PDFs, etc.) would go directly into the report, resulting in an excessively long document containing unfiltered raw information, incl. ads and other irrelevant information. Report documents have length of typically a few thousand tokens, but raw evidence can go well beyond hundreds of thousands. This would increase the required computational cost by orders of magnitude and decrease the human readability of the report. Performance: Upon your request, we compared standard DEFAME against simple truncation at 1000 characters and observed a drop from 68.7% to 63.2% accuracy on CR+, confirming its practical necessity.
• The develop stage mainly targets to determine how well the claim is supported by the gathered evidence. This is done by contrasting the claim with the evidence and performing Natural Language Inference (NLI) through Chain-of-Thought (CoT). We expect from the NLI to deduce facts helpful for the fact-check. Moreover, we expect that gaps of missing evidence become evident during that process. This stage prepares the judge stage where the actual classification happens.
• The justify stage aims to produce a comprehensive summary of the report to serve the user as a quick explanation of the decision. It is purely explanatory and does not affect label prediction—ablating it would not change performance metrics.

GPT-4o Comparison (Single-turn vs. Multi-turn)

We conducted an additional experiment using multi-turn GPT-4o with CoT prompting leveraging the same reiteration criterion as in DEFAME. It achieved 40.7% accuracy on CR+, only slightly improving over the single-turn variant. The primary limitation remains GPT-4o’s lack of access to external evidence, leading to overuse of NEI.

Human Evaluation

As detailed in Appendix H, evaluators saw complete reports (claim, evidence, verdict, justification); baseline outputs were reformatted for fairness (Appendix K). While not trained fact-checkers, all evaluators had higher education and familiarity with MLLMs.

Other Models on CR+

Prior SOTA methods weren’t included on CR+ due to multiple reasons. Please refer to our response to Reviewer 3bbT (“‘Incomplete’ Table 3”) for more details.

Ablation Scope

Thank you for recommending additional ablations, which we now extended to CR+, see the results in the table below.

The values confirm that all components contribute meaningfully to DEFAME’s performance - removing any would hurt the method. We intentionally did not extend ablations to AVeriTeC due to its unimodal nature, which renders many multimodal ablations inapplicable.

Component Contributions

We observe distinct tool contributions across datasets due to the fundamental differences in the tasks: VERITE targets Out Of Context (OOC) detection which (as already shown by previous work) benefits highly from applying Reverse Image Search to the input image. MOCHEG benefits from Web Search strongly as, in contrast to VERITE, it is constructed from real-world claims only.

We performed an extra ablation DEFAME with GPT-4o + Web Search (GPT WS) on CR+, where GPT WS was allowed to perform a single round of evidence retrieval and to apply CoT. GPT WS achieved an accuracy of 54.7% which is clearly better than the native baselines but still strongly worse than DEFAME (accuracy 68.7%). Would you like to see additional ablations on MOCHEG and VERITE?

NEI Confusions in MOCHEG

Yes, many NEI errors result from outdated or time-sensitive labels, as discussed in Appendix G. Others stem from DEFAME’s current limitation of not assessing source credibility. For example, in one case, DEFAME classified the claim "Former President Jimmy Carter said 'America has no functioning democracy at this moment.’” as Supported based on an article from truthout.org. However, Snopes labeled the same claim as Not Enough Information, arguing that further verification from more authoritative or corroborated sources was necessary. This illustrates how differences in evidence sufficiency thresholds—and subjective judgments about source credibility—can lead to apparent disagreement.

Effect of Planning

Table 4 and the additional ablations on CR+ in the table above already include the removal of planning, referred to as “Static Actions” (all tools used every time).