PhantomWiki: On-Demand Datasets for Reasoning and Retrieval Evaluation

We thank the reviewer for their positive feedback on the design of PhantomWiki and for acknowledging its novelty in providing a scalable and data leakage-resistant evaluation framework. We address your concerns in turn.

1. Using templated articles

Article content and questions lack realism. The articles are generated using templates, are very short and lack realism for real-world retrieval augmented generation applications.

We agree that the current articles and questions are intentionally stylized and minimalistic due to their templated construction. To explore how LLMs can be leveraged to improve the realism of PhantomWiki articles, we add new experiments using Llama-3.3-70B-Instruct to rephrase the articles. We prompt the model in two ways (see https://imgur.com/a/ffvugvg): the “short” prompt instructs the LLM to paraphrase the templated articles, while still retaining all factual information; the “long” prompt permits the LLM to expand on the articles, without contradicting existing facts. We experimented with multiple temperature and top-p settings to mitigate hallucinations while encouraging creative outputs (see e.g., https://imgur.com/a/wFvjAFP).

Additionally, we quantify the effect of using these rephrased articles by reporting F1 scores of Llama-3.3-70B-Instruct with ZeroShot and CoT prompting: see https://imgur.com/a/DQuvQIA. Remarkably, we find similar trends in performance regardless of whether we use LLM-generated or templated articles. Thus, templated articles provide four key benefits: (1) they are cheap—no GPU or API costs, (2) fast—no latency from querying LLMs, (3) 100% factually consistent and (4) they allow for larger universe sizes in limited context windows. We leave as future work how to ensure that question-answer pairs are consistent with rephrased articles without relying on human intervention.

2. Use case of PhantomWiki and evaluating LLMs fine-tuned on PhantomWiki data

If the dataset is focused on context free grammar, is it easy to learn after fine-tuning on the same distribution because CFG is well defined?

Is this dataset intended for any training or evaluation only?

These are great questions! To assess the viability of PhantomWiki for training language models, we add new fine-tuning experiments: see https://imgur.com/a/QHftpCM. Specifically, we perform full fine-tuning of Qwen2.5-0.5B-Instruct and parameter-efficient fine-tuning of Qwen2.5-3B-Instruct (via LoRA applied to all linear layers). We also experiment with two popular training algorithms: Group Relative Policy Optimization (GRPO) and supervised fine-tuning (SFT). For Qwen2.5-0.5B, we find that GRPO and SFT both improve F1 compared to prompting-based methods, likely due to improved ability to output the proper answer format. For Qwen2.5-3B, we find that GRPO improves F1 slightly, whereas SFT worsens F1, likely due to overfitting on the training samples. These experiments show that further advances beyond fine-tuning are needed to truly close the gap on PhantomWiki. Please see also our rebuttal to reviewer YauR for full experiment details.

3. Evaluating retrieval

the questions and articles are not appropriate for neural retrieval.

a question like "What is the job of the father of the uncle of the sister of {personA}?" requires iterative bridge entity resolution, making neural retrieval ineffective.

RAG prompting only retrieved 4 most relevant documents, which are likely insufficient for many multi-hop questions.

To address all three concerns, we include new multi-hop RAG baselines (namely, IRCoT and Self-Ask) using BM25 (top-k=5) as the retriever and Llama-3.3-70B-Instruct as the generator model: see https://imgur.com/a/OMjtf3t. We find that these baselines outperform the original ZeroShot-RAG and CoT-RAG baselines on low-difficulty questions, but similarly struggle on questions with high difficulty.

4. Ambiguous questions & Calculation of F1 score

PhantomWiki questions always have at least one correct answer and often have multiple correct answers, e.g., “Who is the friend of X?” if X has several friends, or “What is the hobby of the sister of X?” if X has multiple sisters. Furthermore, we explicitly instruct the model to return all possible answers (see prompts in App. C). We compare the LLMs’ predicted answer list against the ground-truth list to compute precision, recall, and F1 per question. Final F1 scores (e.g., in Table 2) are averages over all 500 questions in a PhantomWiki instance. We will include this clarification in Section 4.3.

5. Additional references

We will include your suggested references on synthetic data generation in our final manuscript.

We hope that these new experiments and clarifications have significantly strengthened our message and empirical results, and if so we would like to politely ask you to consider raising your score. Thank you!

We thank the reviewer for the thoughtful and encouraging assessment of PhantomWiki. We are pleased that you find PhantomWiki a high-quality benchmark for fine-grained evaluation of reasoning and retrieval. We are also delighted to hear that our work is well-documented, and presents solid experimentation and analysis. We address your concerns below and outline how we plan to revise the paper accordingly.

1. Multi-hop RAG baselines

This paper could be improved by incorporating more recent baselines. e.g. IRCoT

We add a new experiment with results of IRCoT from Trivedi et al. (2022) and Self-Ask from Press et al. (2022) in comparison to ZeroShot-RAG and CoT-RAG: see https://imgur.com/a/OMjtf3t. We would like to point out that IRCoT and Self-Ask continue to struggle on questions with high difficulty, measured by number of reasoning steps. We will include a more detailed comparison of multi-hop RAG to Agentic prompting techniques in our main paper.

2. Comparison to other existing benchmarks

While using a context-free grammar and Prolog to generate question-answer pairs is intriguing, the concept of generating questions beyond the pre-training set of LLMs is not novel.

We appreciate the reviewer’s observation regarding prior benchmarks that also generate dynamic or leakage-resistant question-answer pairs. However, we respectfully argue that PhantomWiki is fundamentally different in motivation, construction, and evaluative capabilities compared to these works:

• ToolQA [4] focuses on tool-use evaluation, involving curated tools and questions that explicitly require external tools (e.g., calculators, databases). While it minimizes data overlap, its scope is about measuring LLMs’ ability to use predefined tools. In contrast, PhantomWiki evaluates reasoning and retrieval in a self-contained world and no external tools are assumed. The choice of external tools is left to the LLM and the prompting method.
• REALTIME QA [5] and FreshQA [6] both target dynamic, real-world, up-to-date question answering, emphasizing temporal freshness and world knowledge. These benchmarks evaluate how well LLMs adapt to new information by querying real events or fast-changing facts. Our work takes the opposite approach: we intentionally avoid real-world knowledge to enable clean disentanglement of reasoning and retrieval.

The core novelty of PhantomWiki lies not just in generating unseen questions, but in generating entire fictional universes, complete with articles, knowledge graphs, and questions. As you also point out, our datasets are coherent, and allow for fine-grained control over question difficulty and retrieval complexity—these functionalities distinguish PhantomWiki from [4,5,6]. LLMs can thus be evaluated on-demand on their ability to navigate and reason over an unseen yet structured universe.

We appreciate the reviewer’s framing—you're right that our main contribution lies in the fine-grained controlled evaluation that fictional synthetic universes offer. We’ll make this positioning clearer in the revised manuscript and expand the related work section to reflect this comparison.

3. On Question Diversity and Domain Scope

The generated questions tend to be less diverse, primarily focusing on reasoning between personal relationships and objects.

We appreciate the reviewer’s point about the limited scope of the current question types, which primarily focus on relationships and attributes within a fictional universe. This design was intentional for the initial version of PhantomWiki, as it allowed us to carefully control the reasoning complexity and ensure data consistency and leakage resistance.

That said, we completely agree that increasing the diversity of question types—especially to include temporal, causal, and dynamic aspects of the universe—would significantly expand the benchmark’s utility and better reflect real-world reasoning demands.

Due to space constraints in this paper we are planning future work on extending PhantomWiki to support temporal components, such as characters aging, changing jobs, or forming new relationships over time. This will enable questions involving temporal reasoning, event sequences, and dynamic changes in the knowledge graph. These additions would naturally move PhantomWiki closer to the evolving and dynamic nature of benchmarks like REALTIME QA [5] and FreshQA [6], while still retaining the key benefits of synthetic control and contamination resistance.

We’ll highlight these future directions more clearly in the discussion section of the revised manuscript, and we appreciate the reviewer for raising this valuable point.

Thank you again for your detailed review, and please let us know if you have any more questions or concerns that could help improve the paper further!

Thank you for your thoughtful comments and suggestions, which we believe improve our manuscript significantly. We are pleased that you find PhantomWiki a promising alternative to the popular “needle-in-a-haystack” test. We address your comments below and will add results for suggested experiments to our manuscript.

1. Use of templated articles

Thank you for pointing this out. We have added an experiment where we use Llama-3.3-70B to rephrase our templated articles (see prompts: https://imgur.com/a/ffvugvg and example generations: https://imgur.com/a/wFvjAFP) and evaluated on these rephrased PhantomWiki corpora: see https://imgur.com/a/DQuvQIA. We find the drop in F1 scores versus question difficulty to be similar to our original experiments (Table 2 of the manuscript). On the other hand, templated articles are cheap, fast, and free of LLM hallucinations, while enabling compelling evaluations for LLM reasoning capabilities. Importantly, PhantomWiki is designed to be modular to incorporate LLM-generated articles, as future research in reducing LLM hallucination advances.

2. Evaluating LLMs finetuned on PhantomWiki data

Second, as the data was generated via templates, I am concerned it will be easy to game with additional training (for example by using the same templates to generate training data).

Additionally, as the paper proposed a new evaluation method for RAG, it will be helpful to evaluate the gains from fine-tuning on the synthetic distribution (see Claims and Evidence).

Thank you for suggesting this experiment. As the relationships, names, and attributes in each PhantomWiki universe are generated randomly on-demand, we believe that naive training on PhantomWiki data can only yield limited improvements. To support this claim, we have a new experiment where we generate 10 new PhantomWiki dataset instances (question depth 20 and universe size 50) amounting to 5K training question-answer pairs. We then perform full fine-tuning of Qwen2.5-0.5B-Instruct and parameter-efficient fine-tuning of Qwen2.5-3B-Instruct with LoRA on all linear layers.

For each base model, we employ two popular training algorithms. The first is Group Relative Policy Optimization (GRPO) from Shao et al. with an F1-score reward between 0 and 1. We use the CoT prompt template from App. C.4, a batch size of 32, and 8 generations per prompt to sample. The second is supervised fine-tuning (SFT) with answer(s) as the ground-truth label. We use the zeroshot prompt in App. C.2 and a batch size of 4. For all training experiments, we train for 3 epochs (or until convergence) using the AdamW optimizer with initial learning rate set to $5\times 10^{-6}$ for full fine-tuning and $10^{-4}$ for LoRA fine-tuning.

We then evaluated the fine-tuned models using the PhantomWiki instances of size $n=50$ from Table 2: see https://imgur.com/a/QHftpCM. For Qwen2.5-0.5B, we find that GRPO and SFT both improve F1 compared to prompting-based methods, likely due to improved ability to output the proper answer format. For Qwen2.5-3B, we find that GRPO improves F1 slightly, whereas SFT worsens F1, likely due to overfitting on the training samples. These experiments show that further advances beyond fine-tuning are needed to truly close the gap on PhantomWiki; we hope that PhantomWiki will serve as a valuable tool for future research on LLM reasoning and retrieval.

3. Additional multi-hop RAG baselines e.g. IRCoT and Self-Ask, discussion on synthetic data from KG

Following your suggestion, we have performed additional evaluation using IR-CoT and Self-Ask with BM25 as the retriever and Llama-3.3-70B-Instruct as the generator model: see https://imgur.com/a/OMjtf3t. Specifically, we used the implementations from FlashRAG (Jin et al. 2024) and re-wrote the few-shot examples where needed to match the formatting of PhantomWiki questions. We find that both IR-CoT and Self-Ask are competitive with ZeroShot-RAG and CoT-RAG. However, when decomposing model performance on question difficulty (like Figure 3), both IR-CoT and Self-Ask struggle like other prompting methods. In fact, this is a key contribution of PhantomWiki: quantitatively decomposing performance of prompting methods and LLMs on axes of reasoning and retrieval.

Notably, IR-CoT and Self-Ask alternate between reasoning and retrieval like agentic prompting (e.g., ReAct). A key difference: ReAct uses LLM-driven retrieval via tools, while multi-hop RAG methods delegate retrieval to an external retriever—decoupling reasoning and retrieval. We’ll highlight this nuance in the revised manuscript. We’ll also add a brief discussion about generating data from knowledge graphs.

Given these new experiments support the paper's message and strengthen our empirical results, we would like to politely ask you to consider increasing your score. Thank you again!

1. Question 1 / Evaluating language models fine-tuned on PhantomWiki data

Another avenue is to try to actually fine-tune models on multiple instances of PhantomWiki, and assess how this affects performances on fresh instances.

We add a new experiment and find that fine-tuning on PhantomWiki data helps performance on held-out instances, but only to a limited extent: see https://imgur.com/a/QHftpCM (please see also our rebuttal to reviewer YaUR for full experiment details). Both prompting and fine-tuning approaches struggle as question difficulty increases. Thank you for suggesting this experiment idea! We are excited to see how PhantomWiki inspires research in LLM reasoning.

2. Response to Claim 3 and Question 2

PhantomWiki itself is not subjected to any direct evaluation of fitness for a purpose, nor are its scores related to existing benchmarks for consistency and/or redundancy assessment, nor is the score variability between two instances of PhantomWiki assessed.

We appreciate your framing of “what would negative results look like?”, which we find especially helpful for clarifying our benchmark’s utility. Below, we outline how PhantomWiki provides actionable assessments and how we plan to revise the manuscript.

A poorly designed benchmark might fail to appropriately differentiate models and/or distinguish between reasoning paradigms. Our experiments suggest the opposite: PhantomWiki differentiates performance based on both model architecture and prompting techniques, and elucidates failure modes (e.g., hallucinated intermediate hops or context retrieval mismatches) that align with known model weaknesses.

Based on our evaluation results, PhantomWiki is not a good benchmark for reading comprehension, as evidenced by the near perfect F1 scores of all LLMs/prompting techniques for questions that require 1 reasoning step. Indeed, PhantomWiki is meant to complement more complex reading comprehension benchmarks like DROP and SQuAD. Nonetheless, it is a good benchmark for evaluating multi-step and multi-branch reasoning. Quantitatively, we can see this from the rapid drop in performance in Figure 3 as the number of reasoning steps increases.

One area where LLMs struggle is with complex relations. For example, Llama-3.3-70B-Instruct struggles with “Who is the great-grandchild of X?”, not because it doesn’t know what great-grandchild means, but rather because of a high step count (child of child of child) and a high branching factor (having to find the children of each grandchild). One actionable insight is to improve language models’ ability to keep track of not only multiple steps (this is corroborated by the Self-Ask paper, which highlights a compositionality gap), but also multiple branches. PhantomWiki thus sheds light on both of these challenges in LLM reasoning. We will include the discussion of failure modes of different models in the revision.

In terms of benchmark stability, we run all the evaluations with the same hyperparameters on the first three seeds of generated PhantomWiki instances and measure variability through standard deviation across runs. This provides evidence that PhantomWiki is robust to sampling variance and not overly sensitive to individual instance characteristics.

The use of a single template to generate all entries is very disputable (and part of my concerns in Claim 3), especially considering that this template basically amounts to concatenating sentences of the form "The X of Y is Z."

In a new experiment, we use Llama-3.3-70B-Instruct to rephrase our templated articles (see prompts: https://imgur.com/a/ffvugvg and example generations: https://imgur.com/a/wFvjAFP), which we then use for downstream evaluation: see https://imgur.com/a/DQuvQIA (please see also rebuttal w1Q5 for further details). We report similar performance trends as in Figure 3 from our manuscript, and in fact, using rephrased articles only makes PhantomWiki more challenging. Thus templated articles have the benefit of being cheaper, faster, and free of LLM hallucinations, while still providing valuable insight into LLM reasoning capabilities. Importantly, PhantomWiki is designed to be modular to incorporate LLM-generated articles, as future research in reducing LLM hallucination matures.

3. Additional references on long-context and friendship graphs

We greatly appreciate the reviewer’s suggestion of connecting to the long-context literature. We will revise the Related Works section to include them. We also plan to support more realistic friendship graphs, especially heavy-tailed degree distributions, in future versions of PhantomWiki.

4. Example of a small universe

Please see https://imgur.com/a/QLhC54R.

We hope the new experiments and discussion addressed your concerns and strengthened our results, and if so we would like to politely ask you to consider raising your score. Thank you again!