CofCA: A STEP-WISE Counterfactual Multi-hop QA benchmark

Response 1: We sincerely appreciate the reviewer’s constructive feedback and valuable suggestions. Below, we address the concerns raised in the review point by point: We here compare the performance of several open-source and commercial LLMs on the 2-hop of the CofCA dataset and the HopotQA dataset.

Table 1. LLMs on CofCA and HotpotQA datasets.

The selected LLMs also exhibit varying extent of performance degradation on HotpotQA and our CofCA. Due to the time limit, we will add the full experiment results including 2WikiMultihopQA and MuSiQue datasets in the final version of our paper.

Besides, we compare the reasoning chain performance of GPT-4o and O1-preview:

Analysis: GPT-4o achieves the best performance of all models, but still a large margin between HotpotQA and the 2-hop subset. Due to the time limit, we will add the full experiment results on 3-hop and 4-hop subsets with the model performance and update Tables 3 and 4.

Compared with O1-preview. GPT-4o achieves comparable performance on the correct reasoning chain (45.2 vs 44.5 in the first row). However, GPT-4o is easier to be influenced by incorrect sub-answers and performs worse on incorrect reasoning chains. (16.6 vs 18.1 in the last row, 4.9 vs 5.8 in the fifth row). When correctly answering the first sub-questions, GPT-4o is harder to be influenced when the answer to the second sub-question is wrong (15.4 vs 12.7 in the fourth row.) Accordingly, we added the performance of GPT-4o in Table 8.

Response weakness 3：

The reviewer highlights a potential concern about whether the generated multi-hop questions and sub-questions exhibit sufficient diversity. We recognize the importance of this point and offer the following clarifications and plans:

The multi-hop question and corresponding sub-questions are reviewed by humans which means it must be related to the annotated passages (Answerability), which ensures the sub-questions are dependent on multi-hop questions. our questions mainly contain two question types, bridge and comparison. The bridge questions are single pattern q1-q2-q3. While the comparison question is not a single pattern, the answer final question relies on the comparison of all sub-answers.

Here we list two examples of our data: Bridge Question: "When did Africa's second public FM radio station launch?" Sub-Question 1:"What is Africa's second public FM radio station?" Sub-Question 2: "When did Permission launch?"

Comparison Question:"Which is later, the opening time of Gold or the opening time of the Mad Book?" Sub-question 1: "When is the opening time of Gold?" Sub-question 2: "When is the opening time of Mad Book?" Sub-question 2 :"Which year is later between 1935 and 2006?"

Response weakness 4: We appreciate the suggestion to benchmark CofCA on more recent and advanced models. We plan to address this by extending benchmarking:

Analysis: GPT-4o achieves the best performance of all models, but still a large margin between HotpotQA and 2-hop. Due to the time limit, we will experiment with the model performance on the whole dataset and update Tables 3 and 4.

Response Question: Details of computing Joint Score

The reviewer requested a more intuitive explanation of the JOINT COMPUTATION metric. Here’s an outline of how we clarify this in the revised submission:

We provide an example illustrating how the metric works in practice: The multi-hop question: "When did Africa's second public FM radio station launch?" Sub-Question 1: "What is Africa's second public FM radio station?" Sub-Question 2: "When did Permission launch?" Sub-answer 1: "Permission" Sub-answer 2: "4 August 2002" Final Answer: "4 August 2002"

We first compute the F1 score and EM score of each sub-question and final question. Then we compute the joint score based on each F1 score and EM score.

Since in the reasoning chain, once the first sub-question is answered incorrectly, the subsequent questions are prone to be wrongly answered. The joint score is a negative log of F1 scores and EM scores. Consequently, The higher the score, the lower the performance of the reasoning chain.

By emphasizing the dependency between sub-question answers and the final answer, this metric helps expose reasoning shortcuts and incomplete or flawed reasoning processes.

Response weakness 1:

We thank the reviewer for your constructive feedback, thoughtful evaluation, and recognition of the strengths of our work. We are pleased that the clarity of our paper, the introduction of the Partial Match metric, and our efforts to address limitations in traditional evaluation metrics were appreciated. Below, we address the concerns raised in the review point by point: We agree with the reviewer’s observation that our paper primarily focuses on sub-question accuracy and that reasoning abilities involve more than treating sub-questions as isolated tasks. To address this concern:

We conducted a new experiment based on the suggestions, where we randomly selected 100 data 2-hop dataset and 100 data from the HotpotQA dataset. We require LLMs to generate sub-questions based on the given multi-hop question and passage and compute the BLEU and Rouge-L scores.

Analysis: The question decomposition ability of LLMs has almost not decreased, which is mainly because LLMs just need to understand the question structure and all evidence given in passages, thus emphasizing the holistic nature of reasoning.

Response weakness 2: The reviewer raises a valid point about using GPT-4 for dataset validation while also evaluating GPT-4 on the dataset. To address this: Human Verification Process: While GPT-4 assisted in generating the dataset, all counterfactual paragraphs and question-answer pairs underwent manual verification by trained annotators to ensure correctness, informativeness, and alignment with the counterfactual modifications. We randomly selected 300 data (100 from the 2-hop subset, 100 from the 3-hop subset, and 100 from the 4-hop subset) and asked a third expert to review the data quality. The third expert checks: 1) grammar issue; 2) whether Exists in the real world or not; 3)Answerable to the passages and QA pairs. We show the third expert review results as follows:

Diversity Validation: We include an additional analysis comparing the diversity of question structures and reasoning chains, demonstrating that the dataset contains non-trivial multi-hop reasoning challenges beyond simple patterns.

Although questions are the same question type, their sentence structures are different.

Response weakness 1:

We sincerely appreciate the reviewer’s constructive feedback and valuable suggestions. Below, we address the concerns raised in the review point by point:

1. Counterfactual modifications were generated by GPT-4 and subjected to manual verification by annotators to ensure internal coherence and consistency with the provided context. Our instructions explicitly required annotators to confirm that counterfactual information introduced plausible yet alternative scenarios without conflicting with factual grounding. For example, in Figure 1 d, no such coach that serves the Chinese national basketball team and was born in 1997. Once we were given such counterfactual passages, LLMs could only generate answers based on the given passages.

2. While some degradation in performance is expected when models encounter counterfactual scenarios, our findings indicate that the extent of the performance gap between CofCA and factual datasets cannot be solely attributed to contradictions. Instead, as our quantitative and qualitative analyses highlight, this gap primarily stems from reasoning failures rather than mere unfamiliarity with the counterfactual context.

Response weakness 2:

The reviewer suggests that the qualitative analysis in Section 5 could be more thorough. We appreciate this feedback and provide the following modifications:

1. Expanded Error Analysis: We will enhance our error analysis to explicitly compare error categories between factual and counterfactual questions. This will provide deeper insights into the specific challenges posed by counterfactual reasoning. For example, we categorized errors into types such as reasoning failures, evidence retrieval errors, or contradictions and analyze their relative prevalence in factual versus counterfactual datasets.
2. Analysis of Contradictions and Shortcuts: As part of the expanded qualitative analysis, we examine whether any contradictions or shortcuts in the dataset affect model performance. This will directly address the reviewer’s concerns and reinforce the claims made in the paper.

In particular, we randomly sampled 40 factual data (20 bridge questions and 20 comparison questions) from the HotpotQA dataset and 40 counterfactual data (20 bridge questions and 20 comparison questions) from our 2-hop data. The answer is generated by GPT-4.

Table 1 Error Analysis on Question Types.

The total number of errors in factual data is 10, total errors in counterfactual data is 19. Analysis: Although GPT-4 is worse on counterfactual data than factual data, GPT-4 is better at answering bridge-type questions than comparison questions.

We also analyze the error types of the errors:

In factual data, since all knowledge of passages is from the real world. There are no contradictions. However, in counterfactual data, there are 5 contradiction errors which means the knowledge is contradicted to LLMs’ knowledge.

Accordingly, we added the experiment results and analysis in Appendix D, words in red.

Response weakness 3:

We made the following improvements in the revised manuscript: Figure and Table Revisions:

1. Figure 1: Increase the font size, correct the errors regarding the question-answer mismatch, and provide a clearer explanation in the caption.
2. Tables 3, 4, 7, and 8: Reorganize the presentation to ensure sequential referencing and easier cross-comparison of results. For example, we will consolidate key insights into a unified table or accompanying visualizations for better clarity. Clarifying Table 2:
3. We explicitly explain why the "Whole data property" column is not the sum of the others (e.g., due to factual questions sampled from other datasets). This will remove ambiguity and improve interpretability. We added the explanation of total data in section 3.1, paragraph “Dataset Analysis and Statistics”, words in red.
4. We reorganize Sections 4.2 and 5 to make the experimental results and analyses easier to follow. This includes adding transition sentences, summarizing key findings more explicitly, and aligning section content with the overarching narrative.

We sincerely thank the reviewer for your thoughtful feedback and detailed analysis of our submission. Below, we address the concerns raised in the review point by point:

Multi-Hop Grounding in CofCA:

Due to the space limit, the examples in Figures 1 and 2 are presented in single-passage QA, which is utilized to illustrate the difference between our evaluation framework and previous work. However, the CofCA questions are derived from multi-paragraphs (10 paragraphs, 38.42 sentences on average) of our dataset ensures that answering requires reasoning across multiple hops within the passage. We follow the passage settings of HotpotQA and 2WikiMultihopQA datasets. The counterfactual modifications, paraphrasing, and back-translation operations substantially alter the semantic structure of the passages. As a result, the questions often demand reasoning over distinctively transformed segments across paragraphs. Furthermore, the intermediate subquestion-answer pairs explicitly ensure that the multi-hop reasoning is both well-structured and fully grounded. We manually verified these pairs to validate their alignment with the generated questions and to avoid spurious correlations.

Revisions:

We added the explanation of data annotation in section 3.1, paragraph “QA-pair Annotation and Checking”, words in red. The reply is based on our understanding of “multiple pieces of evidence”. If you have further questions, we would be happy to answer them.