Wiki Entity Summarization Benchmark

W1. Line 175-176, how did you identify 'mentions' of Wikidata items in a Wikipedia page? If only relying on hyperlinks, recall could be low.

Answer: Our approach indeed identifies mentions of Wikidata items using Wikipedia hyperlinks in abstracts. We chose this method for its higher precision, as hyperlinks are manually curated, which is critical for generating high-quality ground-truth summaries for entity summarization. To address potential low mention counts on less popular pages, we provide two user-defined parameters: the initial set of input entities and the minimum number of hyperlinks in the abstract. This method is cost-efficient, precise, and easily accessible. The entire pipeline is publicly available, allowing researchers to customize components to suit their needs.

In short, our entity mentions pipeline works as follows:

1. Extract a graph of relevant entities and relationships from Wikipedia hyperlinks and Wikidata.
2. Compute the intersection of these graphs, marking the triples as the summary.
3. Expand the Wikidata graph using a 2-hop random walk.
4. Add available node and label metadata.

In cases with multiple edges between two entities, we use DistilBERT as a tiebreaker to select the most relevant label, choosing the one that best aligns with the context of the entire abstract.

technical comparison of your approach with https://doi.org/10.1016/j.ipm.2013.12.001 which also relies on Wikipedia abstract to create gold-standard entity summaries.

Answer: There are key distinctions between WikES and the approach by Xu et al. [1]. First, we use Wikidata, while Xu et al. rely on DBpedia. Second, their method involves complex NLP techniques for entity linking and relation extraction, whereas our graph-based approach leverages the editorial work in Wikidata and Wikipedia. As mentioned earlier, entity linking improves recall but can reduce precision. Thus, Xu et al.'s approach is orthogonal to ours. We will add this duscussion and citation in the introduction of the revised paper.

[1] Xu, D., Cheng, G., & Qu, Y. (2014). Preferences in Wikipedia abstracts: Empirical findings and implications for automatic entity summarization. Information Processing & Management, 50(2), 284-296.

W1 (2). Have you conducted experiments to evaluate the quality of your generated gold-standard summaries?

Answer: We did consider evaluating the quality of the generated summaries. However, since Wikipedia itself serves as the ground truth and is curated by a large community of users, conducting a user study would not likely provide a better consensus than what Wikipedia already offers. To complement our analysis, we will include examples of the extracted summaries in the supplementary material.

As an example, consider the Elvis Presley entity:

(Elvis Presley)-[military unit]-> (32nd Cavalry Regiment)
(Elvis Presley)-[genre]-> (blues)
...
(Jim Morrison)-[influenced by]-> (Elvis Presley)
(Elvis Country – I'm 10,000 Years Old)-[performer]-> (Elvis Presley)
(The King)-[main subject]-> (Elvis Presley)

The computed summary is

(Elvis Presley)-[genre]-> (rockabilly)
(Million Dollar Quartet)-[has part(s)]-> (Elvis Presley)
(Jailhouse Rock)-[cast member]-> (Elvis Presley)
...
(Viva Las Vegas)-[cast member]-> (Elvis Presley)
(Elvis Presley)-[genre]-> (rhythm and blues)
(Elvis Presley)-[record label]-> (Sun Records)
(Elvis Presley)-[genre]-> (pop music)

W1 (3). Do you consider or ignore literals?

Answer: We include literals as graph nodes during the random walk exploration.

W1/Q1: The summarization methods are limited to graph-based summarization techniques ... explore text generation or abstractive summarization models (such as GPT or T5) to create summaries from the knowledge graph or the Wikipedia text.

Thank you for the suggestion. While abstractive summarization models, such as GPT or T5, could serve as baselines, there are challenges in assessing their quality for our datasets. First, these models are trained on vast data, likely including Wikipedia, which introduces data leakage and makes evaluations unfair. Second, Large language models (LLMs) are prone to hallucinations, leading to inconsistencies in the evaluation pipeline compared to graph-based methods. Third, to our knowledge, there are no established methods for entity summarization using LLMs.

W2. The paper focuses on scalability but only evaluating the small version of their dataset.

Answer: Thank you for raising this point. The results in Table 2 are limited to the smallest WikES dataset due to the inefficiency of previous methods on larger datasets. Yet, this smaller dataset enables meaningful comparisons in a controlled setting.

To run experiments even on the smaller datasets, we made significant efforts to optimize and parallelize the implementation of prior methods. Adapting these methods to larger datasets, while technically feasible, would require substantial engineering effort beyond the scope of this work, which focuses on providing a robust pipeline for entity summarization.

W3: The authors could have considered a dynamic approach, where the hop count is adjustable based on the entity or relationship type.

Answer: Thank you for the suggestion. While a dynamic hop count approach could treat each entity differently, it may also introduce frequency-related biases similar to ESBM and produce excessively large, uncontrolled graphs that do not represent the local neighborhood effectively. We opted for a simpler, theory-grounded solution that delivers the expected results efficiently, as shown in Figure 3.

W4: The authors could provide some qualitative examples of the generated summaries of model like LinkSum in Appendix.

Answer: Below is a set of three examples of generated summaries using Linksum.

Top-10 summary for Victor Hugo:

Victor Hugo, place of death, Paris
Victor Hugo, place of burial, Panthéon
Victor Hugo, child, Léopoldine Hugo
Victor Hugo, spouse, Adèle Foucher
Victor Hugo, father ->  Joseph Léopold Sigisbert Hugo
Victor Hugo, father, Joseph Léopold Sigisbert Hugo
Victor Hugo, mother, Sophie Trébuchet
Victor Hugo, movement, French Romanticism
Victor Hugo, notable work, Ninety-three(novel by Victor Victor Hugo, instance of,  human
Victor Hugo, described by source, Library of the World's Best Literature

Top-5 summary for Barbra Streisand:

Barbra Streisand, award received, Academy Award for Best Actress
Barbra Streisand, record label, Columbia Records
Barbra Streisand, influenced by, Billie Holiday
Barbra Streisand, award received, Grammy Lifetime Achievement Award
Barbra Streisand, award received, AFI Life Achievement Award

Top 5 summary for Claude Debussy:

Claude Debussy, award received, Prix de Rome
Claude Debussy, educated at, Conservatoire de Paris
Pelléas et Mélisande, composer, Claude Debussy
Claude Debussy, student of, Antoine François Marmontel
Claude Debussy, place of birth, Saint-Germain-en-Laye

Thanks for the insightful questions. We have made an effort to clarify the points below.

Relying Wikipedia’s abstract to generate the ES datasets is cost-efficient and novel.

Answer: Thanks for appreciating our core contributions.

W1. entity summarization generated based on the abstract text rather than the triples of the entities in the knowledge graph.

Answer: Thank you for raising this point. While the Wikipedia abstract of an entity does semantically represent its summary, it also serves as the counterpart to a Wikidata entity. By carefully mapping each entity and its relationships to the corresponding Wikipedia page, we are able to extract the most representative information that Wikipedia contributors have synthesized in the abstract, ensuring alignment with the knowledge graph.

W2. The DistillBERT correctness of the final property is not evaluated

Evaluating DistilBERT's predictive correctness is challenging without ground truth. Yet, we emphasize that its role in our pipeline is limited to selecting a triple when multiple relationships exist between two nodes—a scenario that arises in only 20% of edges and rarely involves more than two triples.

We chose DistilBERT, a robust and scalable model, over manual annotation for this task. For example, given the abstract:

The album featured 'Blowin' in the Wind' and 'A Hard Rain's a-Gonna Fall,' which adapted the tunes and phrasing of older folk songs,

and the corresponding triples in Wikidata

(Bob Dylan)-[notable work]->(Blowin' in the Wind)
(Blowin' in the Wind)-[composer]->(Bob Dylan)
(Blowin' in the Wind)-[lyrics by]->(Bob Dylan)
(Blowin' in the Wind)-[performer]->(Bob Dylan)

DistilBERT selected the triple (Blowin' in the Wind)-[lyrics by]->(Bob Dylan) from several valid options, a choice we found reasonable but hard to verify.

To provide transparency, we included a set of examples comparing the triple selected by DistilBERT (column "picked") against all candidate triples between two node, demonstrating the model's general effectiveness in selecting meaningful triples.

In this case, a human annotator could reasonably select any of these triples. DistilBERT chose the third triple, (Blowin' in the Wind)-[lyrics by]->(Bob Dylan), which we consider a reasonable choice. Based on manual inspection of several examples, we found DistilBERT's selections to be generally appropriate.

To provide transparency, we have created a small set of examples, comparing the triple selected by DistilBERT (column "picked") against all candidate triples between two nodes. These examples demonstrate the model’s reasonable performance in selecting meaningful triples.

W3(1). Figure 3 only shows the F1 evaluation on WIkiProFem

Answer: Thank you for highlighting this omission. Due to space constraints, we included results for only one representative dataset, WikiProFem. However, we conducted experiments on all datasets, and the findings are consistent across them. The detailed results are now included in the supplementary material (Figures 4–11) for completeness.

W3(2). Table 2 shows that results of entity summarization methods on the smallest WIKES datasets.

Answer: Thank you for raising this point. The results in Table 2 are limited to the smallest WikES dataset due to the inefficiency of previous methods on larger datasets. Yet, this smaller dataset enables meaningful comparisons in a controlled setting.

To run experiments even on the smaller datasets, we made significant efforts to optimize and parallelize the implementation of prior methods. Adapting these methods to larger datasets, while technically feasible, would require substantial engineering effort beyond the scope of this work, which focuses on providing a robust pipeline for entity summarization.

We thank the reviewer for the comments. We acknowledge that most of the confusion is on the term benchmark. Our main contribution is on a pipeline for realistic dataset generation for entity summarization.

W1. Why the automated mapping from Wikipedia to Wikidata allows for better summarization over entities

Answer: We refer to the efficiency of generating summaries through automated mapping from Wikipedia to Wikidata, not the quality. Unlike manual annotation, which is costly and time-intensive, our approach scales effortlessly to thousands of entities with minimal cost, requiring only computational resources. We have clarified this in the revised manuscript.

W2.The motivation behind choosing these 4 algorithms for comparison is missing.

Answer: We selected top-performing models on ESBM, excluding those that failed to return results within two days or relied on unavailable external knowledge. For example, BAFREC [1] could not generate results on the small graph (13000) within two days, and MPSUM [2] did not terminate even after 15 days. INFO [3] was excluded because it heavily depends on external knowledge that is no longer accessible.

This ensures our evaluation is both practical and reproducible under reasonable computational constraints.

[1] Hermann Kroll et al. Bafrec: Balancing frequency and rarity for entity characterization in linked open data. 1st International Workshop on Entity REtrieval (EntRE), 2018.

[2] Dongjun Wei et al. Mpsum: entity summarization with predicate-based matching. EYRE@CIKM’2018, 2018.

[3] Gong Cheng et al. Generating characteristic summaries for entity descriptions. TKDE, 35(5): 4825–4835, 2023.

W3. A thorough comparison with all the benchmark datasets given in the related work is missing

Answer: We acknowledge that the term "benchmark" may have been overloaded in our discussion. Our primary contribution lies in the automatic generation of datasets for entity summarization, not in providing a comprehensive benchmarking framework. A full benchmarking analysis is left for future work. For comparisons with the existing ESBM benchmark, we refer to https://github.com/nju-websoft/ESBM/tree/master/v1.2. We have clarified our contributions in the revised manuscript.

W4. The main contribution on why this benchmark dataset is needed should be motivated more clearly.

Answer: We acknowledge that the motivation for our benchmark could have been more clearly emphasized. The only existing benchmark, ESBM—of which INFO is a part—includes just 175 manually annotated entities. ESBM has significant biases (see Figure 2) toward high-degree entities and relationship frequency, which allows a simple baseline that selects triples with the most frequent relationships to outperform some existing algorithms.

In contrast, WikES is a large, automatically generated, and realistic pipeline for generating datasets that accurately reflects the characteristics of the underlying Wikidata knowledge graph. As we showcase on four datasets in Figure 3, the WikES pipeline generates more representative and unbiased datasets, addressing the limitations of existing benchmarks and enabling more robust evaluation of entity summarization methods.

W5. The overall results in table two are very low, what would be the reason behind that?

Answer: Thank you for this observation. The low overall results reflect how previous algorithms are tailored to outperform ESBM, effectively overfitting to its specific characteristics. As shown in Figure 2, ESBM is biased toward entity and relationship frequency, and many of these algorithms exhibit similar biases. In contrast, WikES is built on different principles, revealing the limitations of these approaches and underscoring the need for further research in this critical area.