The World According to LLMs: How Geographic Origin Influences LLMs' Entity Deduction Capabilities

Thank you for your thorough feedback and interest in our work! We address your feedback below:

Feedback1: I am not sure the claim in lines 123-126 is warranted, that this dataset avoids contamination better than other 20Q datasets. For example, LeBron James or any other famous sports star (47.9% of the people entities) are definitely commonly studied.

Thank you for highlighting this concern. We agree that well-known figures in our dataset, such as LeBron James and other prominent individuals, are indeed likely to have been frequently encountered during model pre-training. Our claim of reduced contamination is relative to prior 20Q datasets, which often feature generic or extremely common categories (e.g., “car,” “fruit”) and sometimes have game-play data available online, thus leading to pretraining contamination.

While our approach cannot fully eliminate the risk of pre-training exposure, it does prioritize geographically specific and fine-grained entities to reduce overreliance on memorized facts. We will make the following revision to the claim thanks to your suggestion:

Old Text: Avoiding Contamination. Our dataset design avoids generic or trivia-style entities commonly featured in pretraining corpora and standard 20 Questions datasets. By selecting geographically specific entities, we reduce the risk of models succeeding through memorization of benchmark examples rather than through genuine deductive reasoning.

Revised Text: Avoiding Contamination. Rather than using generic, high-frequency entities (e.g., “car,” “fruit”) that are likely to be memorized during pretraining, we focus on geographically specific and fine-grained instances (e.g., “Audi,” “Alphonso mango”). This reduces the risk of models succeeding through rote recall and encourages evaluation of genuine deductive reasoning over entity-specific knowledge.

Feedback2: The corpus frequency analysis only considers the English-only Dolma, while your entities are multilingual. This makes that analysis less robust. You should have used a corpora like Fineweb-2 instead. RQ2 does not account for variants of the same entity. For example, "Jack Ma" could be "Ma Yun", "马云", "CEO Ma", etc.

We acknowledge that our frequency analysis currently does not account for variant representations of the same entity (e.g., “Jack Ma,” “Ma Yun,” “马云,” “CEO Ma”). Capturing these variants, especially across different languages and scripts, would provide a more robust estimate of an entity’s presence in pre-training data. Nevertheless, we will highlight this as an area for improvement and consider implementing more sophisticated entity normalization and matching strategies in future work.

Thank you very much for your thorough evaluation of our work. We appreciate the recognition of the novelty of our geographic fairness focus, the contribution of the Geo20Q+ dataset, and the depth of analysis enabled by our multi-turn evaluation framework. We address your concerns below:

Feedback1: Although the evaluation strategy is interesting, one could argue it is not strictly needed; standard bias evaluations may still suffice for some questions. The added value over simpler probing techniques should be better justified.

Thank you for raising this important point! Many static prompt-based evaluations only test single-turn interactions similar to training data, missing implicit process-level biases that emerge during multi-step reasoning. Our game-based approach reveals these biases, demonstrating that some reasoning asymmetries only surface through extended interactions. We will further clarify this added value in our introduction.

Feedback2: The paper does not mention any plans to release the Geo20Q+ dataset (maybe I missed it), which limits reproducibility and broader community utility.

We are indeed planning on publicly releasing the Geo20Q+ dataset upon publication of the paper for community use. Our apologies that we did not make that clear in the draft. As evidence of good faith, note that we had included the processed dataset in the supplementary materials for review.

Feedback3: Reproducibility concerns due to unclear temperature used during completion generation. While this may be less critical in the absence of a public dataset, it still undermines methodological clarity.

Thank you for this helpful feedback. Following Zhang et. al., 2024, we set the temperature to 0.8 for the guesser and 0.2 for the judge. We will add this detail in the camera-ready.

Yizhe Zhang, Jiarui Lu, and Navdeep Jaitly. 2024. Probing the Multi-turn Planning Capabilities of LLMs via 20 Question Games. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1495–1516, Bangkok, Thailand. Association for Computational Linguistics.

Feedback4: While the paper is promising and well written, a major issue lies in the core regression analysis. Table 3 reports multiple negative R^2 values which are not explained. Maybe these are adjusted values? The authors rely on these results to argue that entity popularity and frequency do not explain model performance disparities, yet the low or negative variance explained weakens this claim, especially without raw scatter plots or visualizations to support the interpretation.

Thank you for this helpful suggestion, and apologies for the overly concise presentation in the original submission. In Table 3, the reported values are simple correlations (squared), which provide both direction and magnitude. These values are equivalent to adjusted R², as there is only one independent variable (popularity or frequency). Negative R² values indicate a negative relationship between the dependent (number of turns taken to answer) and the independent variable (popularity/frequency), i.e. popular entities should be guessed in lesser number of turns. We will clarify this in the revision to avoid confusion. Regarding our claim, the low R² values and odds ratios close to 1 support our interpretation that popularity and frequency partially explain model performance, as stated in lines 258–259. Additionally, since we use only one independent variable in a linear regression, the R² values are equivalent to the square of Pearson correlation values.

With regards to your suggestion, we will add raw scatter plots and relevant visualizations in the final version to further support our analysis. Here is the scatter plot for success/ failure upon variation in popularity score; we observe that there is a limited effect of increasing popularity on Success: link to anonymized image

Feedback5: The use of odds ratios is statistically valid, but not very intuitive for readers. Including raw correlations or binned visual summaries would improve clarity and interpretability.

Thank you for this helpful suggestion. We agree that including raw correlations and binned visual summaries alongside odds ratios would improve clarity and reader intuition. We will incorporate these additional visualizations and descriptive statistics in the camera-ready to enhance interpretability.

Thanks for your feedback! We are glad that you found the paper well-written, the newly introduced dataset useful as a complementary evaluation resource, and the evaluation methodology extensive and rigorous. We address your key comments below:

Feedback1: I have concerns regarding the two-role game design. There may be instances where the judge provides an incorrect answer during the conversation -- either due to insufficient knowledge of the entity or because the guesser's question is ambiguous, yet the judge is still required to respond with a "yes" or "no." In such cases, the guesser might never arrive at the correct answer.

We agree that our interactive framework cannot eliminate all sources of judgment error (section 7 - limitations), however, we believe the following steps we took may address your concern:

• All entities in our evaluation are based on Wikipedia articles, a source that is extensively used in the pre-training corpora of all evaluated LLMs. This design choice mitigates the risk that the judge (or guesser) completely lacks knowledge about a given entity, as these entities are highly likely to be present in the models' knowledge base.
• Using the same model for both roles ensures that while there may be insufficient knowledge of the entity, both instances have the same knowledge cutoffs.
• To minimize errors due to insufficient knowledge, we filtered entities to ensure that all models agreed on a primary country attribution (see Section 3.3), which reduces cases where the judge lacks key knowledge of the entity.
• For ambiguous questions, the judge is permitted to reply with "Maybe" (Appendix A.3), which serves as a soft signal of uncertainty rather than misleading the guesser. On average, “Maybe” occurs infrequently, only in about 1.8% of the turns.

Feedback2: Although the authors conduct extensive experiments, they do not provide explanations for why LLMs display such disparities or biases. This is a crucial aspect of the paper. While the results suggest that entity popularity plays a minimal role, the authors are encouraged to propose some hypotheses, perhaps in the limitations or future work sections.

Our primary focus of the paper was to find a novel evaluation strategy to effectively reveal implicit geographic biases in models. As for why exactly they display such behavior, we did in fact test a particular hypothesis (i.e. RQ2 and Table 2) and demonstrated that these biases are partially explained by simple pretraining data coverage, revealing that these behaviours are driven by a multitude of complex factors.

We therefore fully agree with you that investigating these potential factors would provide important insights, and we will propose the following experiments as future work in the revised version:

• Experiment: Analyze model attention and reasoning patterns during deduction to determine whether certain activation paths and model regions are systematically favored at various decision points, using interpretability techniques, such as Patchscopes (Ghandeharioun et. al., 2024).
• Experiment: Systematically evaluate the impact of different tuning and alignment techniques (e.g., instruction tuning, reinforcement learning) on geographic bias by retraining models with regionally balanced data and observing shifts in deduction performance.

These are potential directions that we believe will help disentangle the contributions of data coverage, model architecture, and training dynamics to the observed biases. We also welcome any suggestions from you for future experiments that will help towards this goal.

Ghandeharioun, Asma, et al. "Patchscopes: A unifying framework for inspecting hidden representations of language models." arXiv preprint arXiv:2401.06102 (2024).

Q1: In lines 121–122, the authors mention that they retain only those entities for which all models agree on a primary country. How many entities were filtered out at this stage?

Thank you for raising this clarification. At the disambiguation stage, we used an iterative sampling process for both “Things” and “Notable people,” with minor differences in sourcing. For “Things,” we scraped country-specific entities from Wikipedia and only retained those with unanimous model agreement on primary country; otherwise, we resampled until each country had 50 valid entities. For “Notable people,” we drew from a global notability dataset, again requiring full agreement, resampling as needed. This approach ensured all included entities had consistent country assignments. Approximately 12% of candidates were excluded for ambiguity or disagreement, resulting in final totals of 8,375 “Things” and 24,049 “Notable people” (see Section 3.3). We will clarify the exact filtering statistics in the camera-ready.