KoLA: Carefully Benchmarking World Knowledge of Large Language Models

Thank you for investing your time and expertise in reviewing our work. We are grateful for your recognition of our efforts in the design of framework and the evolving tests, and we are delighted to clarify the concerns and answer the questions you raised.

• The paper said that "Comparing the second-season results with the first season, the rankings of most open-source models have declined" but Table 2 and Table 3 seem to show results of four levels of the same season.

Due to space constraints, we placed the complete results of the first season in Appendix E.2. In the content, we primarily present the results of the second season (the latest season at the time of submission) and its comparison to the first season, which is reflected in the Rank column for each level.

• Why COPEN and 2wikiMultiQA are also considered as exclusive?

To ensure the reliability of our evaluation, we reached out to the authors of several works, including COPEN and 2WikiMutiQA, at the very beginning of our project. We obtained their unpublished held-out test sets for evaluation in KoLA. As explained in the caption of Table 1, “Exclusive tasks mean their test sets are newly developed or sponsored by the original authors and were not publicly disclosed.” Based on this criterion, we labeled these tasks as “Exclusive.”

• How the knowledge is designed in Figure 2 to apply self contrast metric please?

To more clearly demonstrate the evaluation process of the knowledge creation task, here we use the example in Figure 2 for a detailed explanation. The process can be divided into three main steps:

First, the model to be evaluated generates a completion $T$ using only the context $C$ (The Battle of Evesham marked the defeat of Simon de Montfort...), allowing it to freely create content. Second, the model uses both the context $C$ and event knowledge $K$ (i.e., the structured content in the middle part of the figure) as inputs, to generate the other completion $T_{k}$. Third, by contrasting the model's generated results $T$ and $T_{k}$ under these two conditions, we can calculate the rationality of the model's event knowledge creation, $\partial \left( T, T_{k} \right)$ (its ability to freely create subsequent event knowledge, e.g., Prince Edward’s Escape). Together with the other two contrast metrics, this comparison ultimately leads to the scoring of the knowledge creation level. We hope this explanation can further aid your understanding of the paper.

• One of the potential strength of the paper is to analyse the results on the fresh data of each season that the paper claims; however, we don't find such results in the current version.

We analyzed the results on evolving tasks compared with known tasks in Section 3 (the second point of “Design Analysis”), which shows that the known-evolving performance gap is closer at the higher levels, indicating that the high-level abilities are more generalizable. We totally agree that a thorough analysis on evolving data is essential. We plan to execute a retrospective diachronic analysis after completing the first phase (six seasons as planned) and hope to have intriguing findings.

• Meanwhile, the paper draws some conclusions (the ones related to knowledge) that don't seem to be part of the contributions of this particular paper.

We believe that the experimental conclusions of this paper can provide contributions in two key aspects. First, the analysis upon a Bloom’s cognitive taxonomy, a rather unique aspect of this paper, offers a new perspective in analyzing the knowledge capabilities of LLMs. Second, although some speculations about model performance, such as the alignment tax, have been widely discussed, our results provide new experimental support for these speculations. As other reviewers have mentioned, although some conclusions might not be novel, the evaluation results provided by this framework are valuable. Additionally, KoLA, as a long-term maintained benchmark, can be utilized to observe whether these issues show improvement in the future.

[Final Remark] Thank you again for reviewing our paper and for the pleased comments. We hope that our response and clarification have addressed your questions and concerns. We sincerely invite you to engage with us if you have more questions.

Thank you for dedicating your time and effort to review our work, and for acknowledging our overall design, experimental analysis, and writing. We are glad to answer the questions you've raised.

• Q1: The paper identified four knowledge capabilities. It is clear on knowledge memorization and knowledge creation. However, it is vague to distinguish knowledge understanding and knowledge application. Take knowledge understanding as an example, would reading comprehension a task of knowledge understanding or knowledge attention? Why knowledge understanding only has extraction tasks?

This is a valuable question, and we are pleased to discuss it with you.

In Bloom's Cognitive Taxonomy [1], Understanding can be described as ”Determining the meaning of instructional messages“, typically involving actions like “Explain, Summarize.” Applying is described as “Carrying out or using a procedure in a given situation”, generally including actions such as 'Implement, Execute’. Indeed, since the revision of Bloom's theory [2], there has been considerable debate about the demarcation between these two cognitive levels.

Despite ongoing discussions, one consensus principle to differentiate them is whether it involves information and abilities beyond the given content [3]. Consequently, we categorize tasks involving information extraction (which ensures that the output information is faithfully extracted from the input text) as Understanding. It is worth noting that the Knowledge Understanding (KU) level is not limited to information extraction; it also includes the ability of conceptual abstraction, as seen in our investigated task COPEN. Under this criterion, if a reading comprehension question requires only knowledge from the text, we would classify it as a KU task. Once it involves more complex background knowledge and reasoning, it might lean more towards a Knowledge Application (KA) task.

Ref:

[1] Krathwohl D R. A revision of Bloom's taxonomy: An overview[J]. Theory into practice, 2002, 41(4): 212-218.

[2] Forehand M. Bloom's taxonomy: Original and revised[J]. Emerging perspectives on learning, teaching, and technology, 2005, 8: 41-44.

[3] Wilson L O. Anderson and Krathwohl Bloom’s taxonomy revised understanding the new version of Bloom’s taxonomy[J]. The Second Principle, 2016, 1(1): 1-8.

• Q2: Benchmark on emerging corpora is a great idea and it is quite encouraging to see the authors promised to refresh the benchmark regularly. However, it is not clear how to maintain such benchmark in the long term. As the time goes by, some evolving benchmark is no longer most up-to-date, how to handle the benchmark data? How would this evolving benchmark interact with the known dataset?

As stated in Section 2 of our paper, we are committed to updating our evaluation for at least six seasons. After this, we will further extend tasks to introduce new versions of KoLA. The data for every new season will be up-to-date to ensure the test data is not leaked, maintaining the relative fairness of the evaluation. For the ended seasons, we do not plan to add them to the known set immediately. Instead, they will be publicly released after the completion of six seasons, and a retrospective analysis of all past seasons will be conducted. The primary workload for running a new season includes collecting and annotating new data, constructing a new list of models, and publishing evaluation results. To host these tasks, we have established a maintenance team and carefully controlled costs. This includes adding a large amount of automated pre-annotation (as described in Appendix B) and controlling the size of the evaluation set. As mentioned in our discussion with reviewer 1m8m, we expect the cost of every season will be under USD 2,000. Therefore, we are confident to fulfill the committed maintenance and we also welcome contributions from the community.

We appreciate your careful review and constructive points. Your assessment stating that our 'knowledge benchmark fills the blank of thoroughly evaluating world knowledge of current large language models' is highly encouraging to us. According to your comments, we have edited the manuscript as your suggestions as follows:

• The knowledge-wise strength of Rouge-L used in Eq. 3 doesn't look to be strong enough in capturing the knowledge association especially when T is a free-generation result, maybe replacing the measurement with another model (e.g. a entailment model) would be better? (w/ additional computational cost)

Employing models for automatic evaluation of knowledge aspects is indeed a valuable suggestion and represents a promising direction. In the design phase of KoLA, we considered such an approach. However, we ultimately chose model-free evaluation metrics to potentially prevent the evaluation model from favoring certain types of models. In the early experiments, we tried various metrics, including some model-based metrics like BertScore, BLEU, etc., and test whether their results align with human evaluations. Bases on these, we finally choose Rouge-L. We plan to continue exploring the integration of your suggested model-based evaluation methods in subsequent seasons. Thank you once again for your insightful recommendation!

• The caption of the figures should include necessary details to understand the them if the space allows (e.g. Figure 4).

We appreciate the reviewer's suggestion to elaborate on this detail. Although we mentioned this part in the first paragraph of Appendix B.4, we overlooked emphasizing this information in the caption of Figure 4. Following your reminder, we have enriched the captions of all the tables and figures in the paper, further enhancing the readability of the article.

[Final Remark] We thank you again for investing the time and effort to review our paper and for the helpful comments that helped us improve the submission. We hope that our responses will address your concerns on the evaluation metrics and caption details and sincerely invite you to engage with us if you have more questions.

Thank you for your valuable feedback. We are greatly delighted to note your recognition of the contributions our paper makes in terms of data, evaluation design, and results analysis. We are happy to address the questions you’ve raised.

• I would have liked to see the human evaluation results in the main part of the paper. It would strengthen the analysis.

Adding the human evaluation to the main text is indeed a good suggestion. However, due to the constraints of length, we attempted but failed to include the results of human evaluation and its background in the content. In our revised paper, we have further highlighted this part (in Section 3), with the aim of directing readers' attention to the corresponding appendix.

• Do authors discuss the costs involved with these studies for both seasons?

This is a valuable concern. To sustain the project over the long term, we have kept the budget for each season below USD 2,000. Generally, each season’s expenditure comprises two main parts: 1) the cost of data annotation, and 2) the expenses for model deployment and API calls.

The data annotation includes labeling knowledge triples and event arguments. As introduced in Appendix B, to reduce the difficulty and cost of annotation, we have pre-labeled the collected text automatically, allowing annotators to focus on the core tasks. Overall, each season requires approximately USD 660-700 for triple annotation and USD 400-420 for event annotation.

Due to the scale of the KoLA test sets, the costs for model deployment and API calls have been kept within an acceptable range. The deployment expenses, estimated from GPU usage time, are approximately USD 200-240, while the API incurs an additional cost of USD 150-180.

Overall, for the two seasons completed thus far, the costs have not exceeded USD 1,600 per season. With the anticipated increase in the number of models in future seasons, we believe that a budget of USD 2,000 per season should be sufficient to maintain the normal operation of the project.

[Final Remark] Thank you once again for recognizing our work and offering invaluable suggestions for improvement. Please feel invited to leave more comments in case you have additional questions.