Supposedly Equivalent Facts That Aren’t? Entity Frequency in Pre-training Induces Asymmetry in LLMs

We thank the reviewer for the thoughtful and constructive feedback. Below, we address each concern in detail.

“…only one aspect (frequency) is considered…”

We focus on frequency as a well-isolated, quantifiable factor hypothesised to have a causal link to equivalence asymmetry. We design controlled experiments to isolate its effect. Other aspects like correctness or conflict are important but require different setups. Our dataset is English-only, so multilingual issues are out of scope.

“...not clear how these findings would mitigate such an issue…”

While we do not propose a direct mitigation method, our findings provide a diagnostic lens for identifying entity frequency imbalances – even in models with undisclosed training data. Notably, many so-called open-source LLMs release only their model weights, not the underlying data. As a potential mitigation strategy, one could consider augmenting the training data with reversed subject-object pairs that preserve semantic meaning, thereby counteracting asymmetries. We will include a brief discussion of this direction in the revised version.

“...effects on different entity types…”

Our main goal is to isolate the effect of frequency, independent of entity and relation types. We design our experiments to demonstrate that the frequency effect generalizes across these categories. While not central to our analysis, our experiments also show that some relation types (e.g., bordersWith) are inherently more difficult than others (e.g., siblings).

“...entity ambiguities…”

We mitigate entity ambiguity by including multiple aliases (synonyms) associated with the same entity in Wikidata5M during inference (see Section 4.3).

“...one word instead of generating the thinking process…”

We constrain responses to one word to enable exact-match evaluation, avoiding reliance on LLM-as-judge metrics. This also improves efficiency, given the number of tokens needed to be generated.

“...no background knowledge about the two entities…”

We mitigate ambiguity by including several known synonyms for each entity (see our earlier response). This helps avoid vague cases like “Mary likes John.” Moreover, when entities appear together, e.g., Mary Lee and Tom Cruise, the context significantly reduces ambiguity. In many cases, the high-frequency entity effectively serves as "background knowledge" for the lower-frequency one.

“...How to address the entity linking (or disambiguation) problem…”

Please refer to our earlier response.

“...filtering process in line 167-168 is reasonable…”

The filtering in lines 167–168 is used solely to construct the probing dataset based on Wikidata5M and is unrelated to pre-training data. It ensures sufficient probing examples for statistically meaningful analysis.

We sincerely thank the reviewer for the detailed and thoughtful feedback. Below, we address each of the raised concerns or questions.

“...non-symmetric relations…” and Question 1

We focused on symmetric relations to enable tightly controlled experiments isolating subject-object frequency effects. We acknowledge that extending to non-symmetric relations is important. However, the experimental design is non-trivial, as it requires handling differing prior probabilities and semantics between a relation and its inverse, and probing while conditioning on these priors. For instance, "employs" and "employedBy" will incur two different logits and thus different priors. We consider this a valuable direction for future work.

“...proposal for mitigation…” and Question 2

As a potential mitigation strategy, one could consider augmenting training data with reversed subject-object pairs that preserve semantic meaning, thereby counteracting asymmetries. We will include a brief discussion of this direction in the revised version.

We sincerely thank the reviewer for the detailed and thoughtful feedback. Below, we address each of the raised concerns or questions.

“… the absolute numbers of triples in the three sets are different, though I couldn't find those numbers reported…”

The triple counts for each frequency category (H→L, L→H, H→H) are reported in the “Total” columns of the result tables. We will clarify this in the text to make it more apparent.

“...the motivation of the dataset construction…”

We would like to recall from Introduction (line 31-42) that our motivation is to explain why equivalence asymmetry arises by linking it to frequency patterns in the pretraining data, rather than merely stating the effect. To this end, we construct probing datasets with reliable frequency counts drawn directly from the pretraining corpus.

“...considerable amount of work on the frequency bias in LMs…”

We appreciate the reviewer’s recognition that our use of actual frequency counts from the pre-training data distinguishes our work. This grounding in pretraining statistics is a key contribution beyond prior studies on frequency bias.

We sincerely thank the reviewer for the detailed and thoughtful feedback. Below, we address each of the raised concerns and provide clarifications or additional results where appropriate.

“…4 relations in WikiData…”

We selected the four relations based on their distribution in Wikidata5M, ensuring sufficient sample size to support robust analysis. We agree that expanding to more relations would strengthen generality and plan to extend our probing datasets using other reliable sources in future work.

“...Experiments with larger LMs…”

We would like to first note that we already include results from the largest variant of the OLMo models (which directly associate with Dolma). To further address this point, we conducted additional experiments with LLaMA-3.1-70B-Instruct, a strong open-weight model. We found that the asymmetry persists in the most extreme freq range (0–1K), with one exception for bordersWith under the statement template. In less extreme freq ranges, the model either shows the same trend or no statistically significant difference, likely due to its increased capacity. Please see the result table below:

We will include these results in the revised version.

“...statements are exclusively true…”

We rely on curated knowledge bases, which contain verified facts. Due to the Open-world Assumption [1], constructing statements that are definitively false is non-trivial -- absence in the KB does not imply falsehood. Our setup focuses on isolating frequency and position effects without introducing noise from uncertain labels.

“...less confident in facts that are about uncommon entities…”

We would like to note that our setup includes both high- and low-frequency entities in the same sentence. Reversing subject and object yields the same fact with swapped positions. This isolates the effect of position, not entity presence, effectively ruling out a simple bias against uncommon entities.

“...treating settings like "high -> low (forward)" and "low -> high (backward)" as distinct settings…”

We retain the original Wikidata ordering to reflect how the relation is typically expressed in real-world usage. This preserves the natural subject-object structure as curated by human annotators and avoids introducing artificial symmetry through preprocessing.

[1] Galárraga et al. "AMIE: association rule mining under incomplete evidence in ontological knowledge bases." WWW 2013.