We thank you for your time and for providing constructive feedback. We have provided a high-level overview of major concerns in the general rebuttal and address more targeted questions in this comment.

Additional Experiments on Loss Function Ablation Study

We have updated Table 3 to include the loss function ablation experiments on the best performing backbone (GPT-2-XL) as well as a different, smaller backbone (T5-Small). This allows for a comprehensive understanding of the most critical components of the loss function across two different backbone settings. We consistently observed that the distillation loss is the most relevant component of the loss function, and the contrastive and scoring loss components allow for further improvement in the quality of the learned embeddings.

Clarification on Distillation Loss and Cold Start

We appreciate your attention to the issue of cold start and your request for clarification on how our framework addresses it. Our method indeed takes the cold start problem into consideration, where during the early training stages, the knowledge model provides completely dissimilar embeddings to triplets in comparison to the claim embedding given by the language model. As you correctly pointed out, we tackle this with the $\mathcal{L}\_{distill}$ component in the loss function. We would like to clarify how this works by providing a fine-grained explanation of $\mathcal{L}\_{distill}$.

The $\mathcal{L}\_{distill}$ is defined in Section 3.4 as follows:

$

$

{\mathcal{L}_{distill}} = \sum_{j\in F^{+}} || \frac{F^{KM}_j}{||F^{KM}_j||_2} - \frac{F^{LM}_j}{||F^{LM}_j||_2} ||_p.

$

$

In the above equation, we have two main elements: $F^{KM}\_j$ and $F^{LM}\_j$, corresponding to the embeddings obtained from the knowledge model and the language model, respectively, for each fact, $j$, in the positive set $F^{+}$. To obtain $F^{LM}_j$, we generate embeddings for the verbalised fact triplets in the knowledge graph using a language model (e.g. GPT-2-XL). These embeddings capture the fact's textual representation and provide a way to compare it to the structured knowledge from the knowledge graph. This way of obtaining triplets embeddings from sequence-based models is standard in the state-of-the-art (see [1] for a recent example). These embeddings are computed only once during data preprocessing and stored to be used during training.

Therefore, during training, the $\mathcal{L}\_{distill}$ encourages the knowledge model to embed the triplets in the feature space as close as possible to the corresponding verbalised triplet embedding from the language model. As shown in the ablation study in Table 3, this loss component is critical for the performance of our method, as without it the model can not solve the cold start. Moreover, this is consistent with our findings at the end of Section 3.4, where we state that weighting more $\mathcal{L}\_{distill}$ than $\mathcal{L}\_{contrastive}$ or $\mathcal{L}\_{intra}$ is the best setting for optimal pre-training results.

To make this more clear, we have extended our explanation in the Claim-Fact Distillation Loss paragraph in Section 3.4.

Notation Typos and Simplification of Section 3

We appreciate you pointing out these typos and have made another pass through the work to correct errors. In particular, we have made the notation in Section 3 consistent throughout the paper, fixed notation typos and made it more clear and concise when possible.

Thank you again for helping us improve this work.

Please let us know if you would like to see additional experiments or have further feedback.

References

[1] Lu et al., “KELM: Knowledge Enhanced Pre-Trained Language Representations with Message Passing on Hierarchical Relational Graphs”. ICLR, 2022.

We thank the reviewer for the helpful comments, and would like to clarify the concerns raised. We have provided a high-level overview of major concerns in the general rebuttal and address more targeted questions in this comment.

Response to Weaknesses:

Experiments on Additional Datasets

As this is a common question between some reviewers, we have provided experiments on additional datasets in the general rebuttal. In particular, we benchmark SFAVEL's unsupervised pre-trained model in the FB15k-237 dataset, which is another common benchmark for fact checking tasks. When compared against state-of-the-art methods, our approach is able to achieve +5.3%, +3.6% and +6.3% for Hits@1, Hits@3 and Hits@10 for fact ranking, respectively.

On Performance Gap Between Previous Methods and SFAVEL

In our work, we leverage self-supervised learning to exclusively learn and understand the underlying structure of the claim-fact alignments, leading to high-quality embeddings that can be used in further fact-related downstream tasks, as shown in our new experiments. This is different to the approach taken by previous works (e.g. [1, 2, 3], to mention some), which aim to learn models for both feature representation and prediction at the same time, in a supervised setting. This leads to models that focus on learning specific and potentially spurious patterns associated with the labels, without encouraging the model to learn rich, meaningful and general representations of the data. For this reason, SFAVEL is able to learn intrinsic properties of the data that attain better transferability.

Clarification on Terminology: Proposed Unsupervised Pre-training Method vs Supervised Linear Probe

We would like to emphasise that the main contribution of our work is the unsupervised (self-supervised) pre-training framework. The design of our method is motivated by the goal of learning high-quality claim-fact embeddings useful for downstream fact verification tasks. The embeddings obtained with SFAVEL can be used as input for training any downstream model on different tasks related to fact checking. This is a standard practice in self-supervised learning (see e.g. [4, 5]).

In our experiments we use a linear probe, which is just a linear projection layer, as an example of a downstream model using SFAVEL’s claim embedding and facts pooled embedding as input for claim classification. This is required for the case of FEVER, as the task is to classify claims, but it is not for FB15k-237, where only ranking of claims given a fact is important. On the latter, we do not need a supervised stage, and only use the ranking given by SFAVEL directly for evaluation. The aim of this experiment is two-fold: first, we show the generalizability of our unsupervised pre-training method on fact verification tasks with different nature, and second we obtain state-of-the-art performance with a simple linear layer classifier relying only on pre-trained SFAVEL features, showing the quality of the learned embeddings.

We have now clarified this with the following changes: we added “Pretraining” in the paper title, clarified the optionality of the downstream task in Figure 1, and added more details in the explanation for the Linear Probe description in Section 4.1.

Thank you for your thoughtful and detailed comments on our work. We have provided a high-level overview of major additions in the general rebuttal and hope to address more targeted concerns in this comment.

Experiments on Additional Datasets

As this is a common question between some reviewers, we have provided experiments on additional datasets in the general rebuttal. In particular, we benchmark SFAVEL's unsupervised pre-trained model in the FB15k-237 dataset, which is another common benchmark for fact checking tasks. When compared against state-of-the-art methods, our approach is able to achieve +5.3%, +3.6% and +6.3% for Hits@1, Hits@3 and Hits@10 for fact ranking, respectively.

Experiments on Larger Language Models

As this is a common question between some reviewers, we have provided experiments on larger language models in the general rebuttal. In summary, we have added experiments with GPT-2-XL (1.5B parameters) as the distilled language model and updated the paper accordingly.

Clarification on Comparisons to Other Approaches

The use of top-k retrieval in our approach is motivated by the need to consider multiple pieces of evidence when verifying a claim in real-world applications, and we acknowledge this is a critical factor in our (and all recent) fact verification approach. Given the complexity of some of the claims in the benchmarked datasets (FEVER, FB15k-237), it is not possible to only use single-fact reasoning to predict whether a claim is true or false. For this reason, all state-of-the-art approaches we compare against have a (multiple) evidence retrieval stage (see e.g. [1, 2, 3] for some examples). Therefore, we would like to kindly emphasise that with respect to the top-k retrieval mechanism, the comparison of our method against others is fair and consistent with the literature.

Clarification on Self-supervision

We would like to clarify that the main contribution of our work is the unsupervised (self-supervised) pre-training framework. The design of our method is motivated by the goal of learning high-quality claim-fact embeddings useful for downstream fact verification tasks. The embeddings obtained with SFAVEL can be used as input for training any downstream model on different tasks related to fact checking. This is a standard practice in self-supervised learning (see e.g. [4, 5]).

In our experiments we use a linear probe, which is just a linear projection layer, as an example of a downstream model using SFAVEL’s claim embedding and facts pooled embeddings as the input for claim classification. This is required for the case of FEVER, as the task is to classify claims, but it is not for FB15k-237, where only ranking of claims given a fact is important. On the latter, we do not need a supervised stage, and only use the ranking given by SFAVEL directly for evaluation. The aim of this experiment is two-fold: first, we show the generalizability of our unsupervised pre-training method on fact verification tasks with different nature, and second we obtain state-of-the-art performance with a simple linear layer classifier relying on pre-trained SFAVEL features, showing the quality of the learned embeddings. We have now clarified this with the following changes: we added “Pretraining” in the paper title, clarified the optionality of the downstream task in Figure 1, and detailed the purpose for the Linear Probe in the description in Section 4.1.

We appreciate your feedback and will consider these points in our discussions and future work.

Please let us know if you would like to see additional experiments, and thank you again for helping us improve this work.

References

[1] Krishna et al., "ProoFVer: Natural Logic Theorem Proving for Fact Verification". TACL, 2021.

[2] Zhong et al., “Reasoning Over Semantic-Level Graph for Fact Checking”. ACL, 2020.

[3] Chen et al., “GERE: Generative Evidence Retrieval for Fact Verification”. SIGIR, 2022.

[4] Bouniot et al., “Proposal-Contrastive Pretraining for Object Detection from Fewer Data”. ICLR, 2023.

[5] Hamilton et al., “Unsupervised Semantic Segmentation by Distilling Feature Correspondences”. ICLR, 2022.

Firstly we would like to thank the reviewer for their thoughtful comments and support of our work. We have provided a high-level overview of major additions in the general rebuttal and would like to address more targeted questions in this comment.

Regarding Large Language Models (LLMs), we would like to point out that we do not state our method is targeting LLMs in particular, but more generally any Language Model (LM). However, we agree that assessing the performance of our unsupervised pre-training methodology by distilling from even larger models is an interesting experiment given the current landscape in terms of language model sizes.

Therefore, we have added an additional experiment in Table 2, where we utilise GPT-2-XL as the teacher/distilled language model. Despite not existing a strict definition of LLM, any language model with at least 1B parameters is usually deemed as “large”, and GPT-2-XL contains 1.5B parameters, which effectively makes it a LLM. To run the experiment, we utilise the same hyperparameter settings as the initial experiments.

As a result of this new experiment, we see a 90.32% label accuracy in the FEVER dev dataset. From this experiment we can conclude that our distillation method is still beneficial on larger models, and still provides performance improvements on the downstream task.

Finally, we added additional experiments on the FB15k-237 dataset to showcase the applicability of our method in a different setting.

Thank you again for the helpful feedback and support of our work!

We appreciate your thoughtful and detailed comments on our work. We have provided a high-level overview of major concerns in the general rebuttal and address more targeted questions in this comment.

Experiments on Additional Datasets

As this is a common question between some reviewers, we have provided experiments on additional datasets in the general rebuttal. In particular, we benchmark SFAVEL's unsupervised pre-trained model in the FB15k-237 dataset, which is another common benchmark for fact checking tasks. When compared against state-of-the-art methods, our approach is able to achieve +5.3%, +3.6% and +6.3% for Hits@1, Hits@3 and Hits@10 for fact ranking, respectively.

Clarification on Generation of Negative Instances

We understand your concern regarding the generalizability of our proposed negative examples construction technique and its simplicity. As you correctly pointed out, the individual triples used in our experiments capture relatively simple relationships, typically involving subject-predicate-object structures. While it is true that these triples might not encompass all the complexity of natural language claims, the effectiveness of such negative triplet generation procedures has been demonstrated in previous works (see e.g. [1, 2, 3], to mention a few). In this regard, one of our main contributions is to demonstrate the potential of leveraging structured knowledge from knowledge graphs to improve unsupervised pre-training for fact verification. Despite the fact that real-world claims can involve various aspects such as "who did what, where, when, and why", we argue that breaking down these statements into unitary components can be an effective strategy allowing to capture fine-grained factual differences. As an example, for the claim “Bob Ross created ABC drama The Joy of Painting” (see Section A.1 of the Appendix), our model is capable of obtaining a set of facts that when used together allows the model to perform multi-level reasoning.

Relation Between Ranking/Retrieval and Fact Verification

Your comment raises a valid concern about the choice of problem formulation, specifically the use of “ranking” in the context of fact verification, which may not appear intuitive in this context at a first glance. However, ranking is a crucial intermediate step in state-of-the-art fact verification machine learning approaches (see [1] for a comprehensive overview of relevance of ranking in fact verification). While our approach does provide a score for each fact, such as 0.9, it is intended as a measure of the relevance and credibility of the evidence. Regarding the difference between retrieval and ranking, the former aims to find relevant data points (i.e. facts), while ranking goes further by ordering them based on relevance or quality. In the context of fact verification, ranking helps users prioritise evidence, which is crucial when dealing with a large volume of potentially relevant information, as is the case for the benchmarked datasets. Your feedback has been instrumental in highlighting this aspect, and we will consider it for future improvements to our work.

Please let us know if you would like to see additional experiments, and once again, thank you for the helpful and constructive feedback to improve our work!

References

[1] Guo et al., “A Survey on Automated Fact-Checking”. TACL, 2022.

[2] Jiang et al., “Don’t Mess with Mister-in-Between: Improved Negative Search for Knowledge Graph Completion”. EMNLP, 2023.

[3] Komatani et al., “Knowledge Graph Completion-based Question Selection for Acquiring Domain Knowledge through Dialogues”. IUI, 2021.

[4] Zhang et al., “Simple and automated negative sampling for knowledge graph embedding”. VLDB, 2021.

I appreciated the author's response and the experiments. However, I still have concerns about using FB15k-237. First, this is not a fact-checking dataset, why can not use real fact checking claims from websites such as snopes, factcheck.org etc., Second, this is still a dataset consisting of triples, which has a big overlap with Wikipedia, which may be biased to author's method, similar to FEVER. Therefore, I still have concerns about the generalizability of the method on fact-checking.