Multi-hop Question Answering under Temporal Knowledge Editing

Thanks for your valuable feedback. Our response for each question is as follows:

W1. Data Quality

Please refer to our response to reviewer-EvWU.

W2. Ablation Studies:

We performed two ablation experiments on MQUAKE:

After removing data augmentation, the M-ACC drops by up to 3.57%. We test TEMPLE-MQA with sub-question decomposition excluding the inference path. the M-ACC drops by up to 15.44%. It has a huge impact on performance.

W3. Scalability

We perform a quick empirical analysis on large KG (X represents the multiple of all the original edits):

These results show that Temple-MQA is scale-able. We will add these details to the final draft to clarify the computational complexity and scalability of our work.

W4. Limitations

Our work poses following limitations: * Temple-MQA requires pre-processing of TAG, which brings additional overhead. * Temple-MQA is very effective on the existing and our proposed benchmarks, but it requires writing prompts manually to adapt to different tasks. * Although retrieval of Temple-MQA doesn't need fine-tuning an encoder, it has a complex workflow.

Q1. Disambiguation:

Entity disambiguation is a core part of our model, as it helps improve retrieval performance by filtering entities that are not relevant for a given context. We report the results of our model with and without data augmentation in the table given above. The performance declines by up to 3.57%.

Q2. Strange results

The possible reason for the performance shortfall is that the TEMPLE-MQA uses a structured query prompt:

Inference step: <U.K., head of government, person>
Time: <2019,2022>
Answer: Boris Johnson

which leads to lower accuracy than sub-questions with natural language form adopted in MeLLo:

Question:Who was the head of government of U.K. from 2019 to 2022? 
Answer: Boris Johnson

One possible way to address this problem is using natural language form query prompts.

Thanks for your valuable feedback. Our response for your questions is as follows:

Response to Reasons

W1. See our response to Question 2,3.

W2. See our response to Question 6.

Response to Questions

Q1. Unstructured:

Unstructured format means sentences in natural language form. For example, "The president of the U.S. is Joe Biden." is an unstructured text format, while (U.S., president is, Joe Biden) is structured. In the future, we plan to extend it to support knowledge editing by complex unstructured paragraphs. E.g., "The Eiffel Tower is located in Paris. It was designed by french engineer Gustave Eiffel...", we can extract knowledge (Eiffel Tower, located in, Paris),(Eiffel Tower, design by, Gustave Eiffel),(Gustave Eiffel, nationality, French).

Q2. f_old and f_cur:

We add two examples to better explain it: f_old means historical knowledge, e.g. "The president of the U.S. was Donald Trump from 2016 to 2021." f_cur means current knowledge, e.g., "After 2021, the President of the U.S. is Joe Biden."

Q3. Appendix-A:

We will augment the details in Appendix-A. Also, we explain the contents of Appendix A as follows: - We add the key features of inference-path planning prompt to support high accuracy, such as amount and diversity of demos. - We give some running examples to make the workflow easier for the readers.

Q4. Data Augmentation

We explain it through a simple example: For entity 'United States of America'(its id is Q30), we first construct the following SPARQL statement:

SELECT ?alias WHERE {
wd:Q30 skos:altLabel ?alias.
FILTER(LANG(?alias) = "en").
}

Then we query the website "query.wikidata.org/" to get all the aliases (USA, United States, the US ...).

Q5. Typo:

We rephrase it as "alternate between plan and solve phases".

Q6. Metrics:

Owing to limited space, we provided details about the evaluation metrics in Appendix B.3.

Q7. Self-checking:

Self-checking implies using LLMs to check if the retrieved fact contradicts the generated answer. The reason why we do not use self-checking is that we have observed that many LLMs have been trained with RLHF, which makes them have a high tendency to reject external knowledge that contradicts their own knowledge. However, we admit that self-checking is a very good design that can effectively resist harmful edits.

Thanks for your valuable feedback. The response to each question is as follows:

W1. Datasets

We further elaborate on our data construction process and associated quality control, as follows:

• Sampling Fact Triplets. We first manually collect multiple relations containing time information from WikiData, e.g. "President is". Then, we use SPARQL to sample facts based on the relation and subject entity. For example, given a query (U.S., President is, ?), WikiData will return every president of the U.S. and their terms of office. Thus, We select common relations and entities to control quality.

• Generate Relation Path Template. In order to generate natural and coherent multi-hop questions, we look for high-quality relation combinations. E.g., (live in, the father is) is a bad combination because the object type of 'life in' (location) does not match the subject type of 'father is' (person); a location cannot have a father. Thus, we arrange and combine the collected relations into a relation path template and filter out unreasonable relation combinations.

• Generate Multi-hop Question Template: Then, we use GPT-4 to generate the multi-hop question template for each relation path template, e.g., (father is, company is) -> "What is the company of father of {}?". Replace "{}" into the start point entity to get the final questions. We also conduct manual revisions to ensure its quality.

• Augmented data is reliable: All the augmented facts are reliable and are sampled from Wikidata after 2022.

We will add these details in the final draft to further clarify the data curation pipeline of our work.

W2. Compare with other papers

There are some recent papers on temporal multi-hop QA datasets that are worth comparison to the differences.

• Thank you for pointing out this related work. We noticed that the paper you have mentioned only covers temporal MQA and may not be well-suited for temporal MQA under KE. Moreover, the paper was released one and a half months before the CoLM deadline. However, we will try to add its comparisons for inference path planning against Temple-MQA in the final draft.

Thanks for your valuable feedback. We address your concerns as follows.

W1. Implicit Time

For questions with hidden time, we consider the following examples:

• "Who was the U.S. president before Biden?",
• "Who was the previous president of the U.S.?"

For example 1, TEMPLE-MQA needs to determine the specific time when Biden is president. We can revise the ICL prompt to achieve it. For example,

Question: Who was the U.S. president before Biden?
Time: Biden is president after 2021, Thus time span is <2017,2020>.
Inference Path: <U.S.,president is,?>

However, we observe some datasets contain questions that are more ambiguous, as shown in examples 2. Answer to these questions is decided on the training periods of LLMs. For this, we need to give current time to LLM.

Question: Who was the previous president of the U.S.?
Time: Today is 2021, and Biden is current president, Thus time span is <2017,2020>.
Inference Path: <U.S.,president is,?>

The results on the AToke-SE and AToke-ME with ambiguous hidden time (like example 2) are shown below:

We can see that TEMPLE-MQA still performs better, especially on the HRS metric.

W2. Inference Path

For this, we add two experiments: * Ablation Study: remove inference path planning. * Explore the influence of the number of shots on the ICL prompt.

Here are our results

If we exclude the inference path, the M-ACC drops about 15.44%, 14.39%, 9.53% respectively. The M-ACC are drops about 10.43%, 7.77% and 3.51% in 1,2 and 4 shot setting. Thus more examples can help LLMs follow instructions better, also the quality and diversity of the examples is important.

We will add the above findings in the final version.