Harnessing the Wikipedia Graph for Effective Multi-Entity Question Answering

Clarity: The paper lacks clarity and purpose. Therefore, in its current form, it may not be useful to the community.

• There are several places where sufficient information is not provided to get a point through. For example, in section 2.3, the authors cite a couple of papers stating that those papers propose GraphQA benchmarks. The authors then simply go on to say that those benchmarks are insufficient without providing any specific reason. In section 4.1, the authors mention that they use named entity recognition and relation extraction models but fail to provide any details of the model or a reference to a pre-existing model they use for the purpose. Another example is section 5.2, which tries to describe the process of generating QA pairs to create the dataset. From the two paragraphs in the section (lines 306 to 320), I was unable to understand how they created the QA pairs.

Quality: The experiments are insufficient to establish the usefulness of the dataset as well as the real efficacy of the proposed model.

1. The core idead in this paper, i.e., using a relational table to structure multi-entity data, has been studied in work "Semantic Table Retrieval Using Keyword and Table Queries", "Web Table Extraction, Retrieval, and Augmentation: A Survey", "Novel Entity Discovery from Web Tables" and "Web Table Extraction, Retrieval and Augmentation".
2. The question and answer is based on the contents of the relational table. How to ensure the accuracy of the entity relations extracted by GPT-4? it has not been explained whether there is a more efficient way to build structured entity relations, and whether the additional time and space complexity introduced by the construction table.
3. The performance achieved is heavily dependent on GPT-4, and thus the availability is relatively limited. Therefore, it is recommended that the author utilize other open-source large models as tools to replace GPT-4 for comparison with existing methods, in order to verify the validity of the core idea.

1. While the overall idea of the SQA approach is straightforward and original, its description is rather vague.
   1. In Section 4.1, the paper describes the first step to parse the question into entities and relations. The paper mentions the use of GPT-4, NER, and RE. However, no details are given on how these are achieved.
   2. In Section 4.2, the paper describes the schema generation prompts in the appendix but does not state how Mistral-7B populates the table.
   3. In Section 4.3, there is no detail of how a question is converted to SQL.
2. The paper should discuss many existing multi-hop reasoning datasets to motivate another benchmark. For example, this article reviews seven multi-hop QA datasets (ref). In addition, the paper mentions a quality control step in Section 5.3 but does not provide any detail or evidence that it is done thoroughly.
3. The comparison in Table 3 shows that SQA is better than RAG. However, some important, missing baselines, such as StructQA or HOMLES, might be missing.
4. The paper could benefit from writing improvement. Several paragraphs do not convey important reasoning or evidence but rather an opinionated advantage of the approach, such as Section 4.4, Lines 103 - 107, and Lines 358 - 362. Some minor comments about the notations include notations not being used, unclear what $P$ and $t$ are, duplicate duplicates of $V$, and lack of definition of $P_v$.

1. There is a weak correlation between the problem and the solution. While the authors focus on multi-entity question answering, the method primarily addresses the simplification of long retrieval documents. In this sense, the proposed method does not need to focus on entities but can be applied in any scenario where the retrieval text is too long, but the authors do not experimentally validate it.
2. The significance of the benchmark remains unverified. In the main experiments, the authors evaluate only the base model, RAG, and Structure QA, without exploring alternative RAG methods [1] or other graph-based approaches[2,3]. We do not know if the benchmark is necessary for further research.
3. The effectiveness of the proposed method is not convincing. As noted in W1 and W2, SQA in the main experiment is only compared with basic methods, Specifically, the experiment employs the highly capable GPT-4 as the main model of SQA (refer to lines 212 and 252), resulting in a potentially imbalanced comparison with less advanced models such as GPT-3.5 and Llama. We hope that the authors can add more baseline[1,2,3].
4. The authors should consider conducting ablation experiments to verify the necessity of SQL in their approach. A comparative analysis between the SQL query-based table method and simpler techniques, such as keyword extraction and summarization, would clarify the relative advantages of SQL in this context.

[1] Wang H, Li R, Jiang H, et al. Blendfilter: Advancing retrieval-augmented large language models via query generation blending and knowledge filtering[J]. arXiv preprint arXiv:2402.11129, 2024. [2] Wu Y, Huang Y, Hu N, et al. CoTKR: Chain-of-Thought Enhanced Knowledge Rewriting for Complex Knowledge Graph Question Answering[J]. arXiv preprint arXiv:2409.19753, 2024. [3] Sun J, Xu C, Tang L, et al. Think-on-Graph: Deep and Responsible Reasoning of Large Language Model on Knowledge Graph[C]//The Twelfth International Conference on Learning Representations.

1. Many KBQA systems already handle multi-entity questions effectively, utilizing pre-existing data structures within the knowledge base to support relational queries. The paper does not convincingly establish why a dedicated approach, such as SQA, is necessary.
2. The idea of converting entity data into structured tables is well-established in database query systems, and approaches like NL2SQL have long enabled structured QA without the need for a specialized framework. Thus, SQA lacks originality and does not present a significant innovation in the context of established KBQA solutions.
3. The description of the SQA approach lacks depth in explaining how the schema is derived or validated, how the table generation process addresses entity disambiguation, and the specific steps of entity extraction. A more detailed breakdown of each process would help clarify the novelty and complexity of SQA.
4. The paper primarily discusses simple schema extraction for relatively straightforward queries. However, in real-world applications, queries can vary widely in complexity, requiring more sophisticated relational or semantic inference. The paper could be strengthened by addressing how SQA would handle more complex or ambiguous queries.