ChroKnowledge: Unveiling Chronological Knowledge of Language Models in Multiple Domains

Thank you for your detailed feedback. Your insights help refine our paper and clarify key aspects.

---

"The paper mentions evaluating knowledge across multiple domains; however, it only introduces knowledge from the medical and legal domains, where changes of knowledge along the timeline are relatively subtle."

We appreciate this observation and would like to clarify several aspects regarding domain selection and coverage:

1. Diverse Domains:
   • Our benchmark includes general knowledge, which spans multiple temporal knowledge like personal history, and is inherently dynamic. It plays a critical role in evaluating knowledge with frequent temporal changes.
   • Additionally, we include time-invariant knowledge, which provides a baseline for evaluating how temporal cues affect recall across stable datasets.
2. Importance of Medical and Legal Domains:
   • Medical and legal domains, while subtle, are pivotal areas where temporal reasoning is crucial for practical applications.
   • For instance, even minor updates in biomedical guidelines or legal precedents can have significant implications, making their inclusion essential.

Planned Action:

• We updated Figure 1 to better emphasize the overall benchmark structures.
• We plan to expand the dataset to include more diverse and dynamic domains.

---

"The knowledge within these datasets may have already been learned by LLMs, primarily reflecting their memory capacity rather than their ability to handle time dependencies."

We agree that demonstrating temporal dependencies distinct from general memory recall is crucial. Our benchmark addresses this through the comparison of time-variant and time-invariant knowledge with more fine grained analysis inside time-variant; Dynamic and Static:

1. Role of Time-Invariant Knowledge:
   • The inclusion of time-invariant datasets helps isolate the effect of temporal cues. While these datasets remain stable regardless of temporal prompts, time-variant knowledge often fluctuates, indicating the influence of time-specific context on recall.
2. Dynamic and Static Comparisons:
   • By comparing dynamic and static subsets, we show how temporal cues impact LLM performance. For instance, dynamic datasets reveal where the model struggles to track time-sensitive updates, while static datasets highlight areas of stable recall.

Planned Action:

• Revised Figure 2, 3 and 4 shows more clear difference among temporal affected results, with complement view by time invariant results in the Appendix A.5.

---

"This paper essentially transforms knowledge into a temporal knowledge graph (TKG). However, there are established benchmarks in TKGs. The authors should provide comparisons with these benchmarks and clarify the specific improvements made."

We recognize the importance of situating our work in the context of existing TKG like HisRES [1]. While there are similarities, our approach differs in significant ways:

• TKG benchmarks frame temporal knowledge in a graph format using temporal snapshot to gather information at that specific time point.
• In contrast, ChroKnowBench evaluates knowledge through prompting-based methods, where temporal context is explicitly provided to test LLMs’ ability to align responses with specific time spans. So our approach is basically based on Knowledge-Centric Tracking to construct temporal knowledge.
• Our temporal prompts enable finer-grained temporal reasoning compared to TKGs, directly query models about specific time points.
• By testing the integration of multi-temporal knowledge through ChroKnowPrompt, we enable a dynamic assessment of LLMs’ ability to handle temporal dependencies beyond static graphs.
• We compare TKG datasets in Section 3 and Appendix A.3.3, summarized below.

Planned Action:

• A new subsection compared ChroKnowBench with prominent TKG benchmarks, clarifying the unique contributions of our approach.
• A table summarizing the methodological differences between ChroKnowBench and existing TKG datasets is included.

---

We hope these clarifications address your concerns and demonstrate the strengths of our framework. Thank you for your thoughtful and constructive review.

---

[1] Temporal knowledge graph reasoning based on evolutional representation learning

Thank you for your detailed and insightful review, which highlights key areas for improvement in our framework and figures.

---

"There is another possible explanation to the large amount of performance increase in the biomedical section of the dataset: domain-specific data are more static / less dynamic than general data, and your method on distinguishing dynamic and static knowledge needs to be updated to a more fine-grained extent."

We agree that a more fine-grained distinction between static and dynamic data would enhance the analysis and provide a clearer understanding of our results.

• Our current method classifies any entity with one temporal change as dynamic. We analyzed object changes in Appendix A.3.1, quantifying them for results in Figures 2, 3, and 4 (C).
• Less volatile biomedical data show fewer object changes, aligning with ChroKnowPrompt’s effects on handling unchanged objects.
• Furthermore, we conducted an ablation study to analyze performance gains based on object changeability in Section 7.2.
• ChroKnowPrompt performs better with less-frequent changes, as visualized in Figure 7 resulting in the increase of Known.

Planned Action:

• We included details in Section 5, 7 and Appendix A.3.1, analyzing the performance of ChroKnowPrompt across varying degrees of dynamicity.

---

"Figure 2 and 3 need extra work on clarity to deliver the message. Figure 4 and 5 also need improvement, especially with clearer textual hints and better integration with the main text."

We acknowledge the need to improve the figures for clarity and coherence.

1. Figures 2 and 3 → modified in revision Figure 2, 3, and 4:
   • Enhanced clarity with heatmap scores, spider plots, and object change quantifications. Results for the legal domain are now included (Figure 4).
2. Figure 4 → modified in revision Figure 5:
   • Revised with clearly targeting for ChroKnowPrompt.
3. Figure 5 → Figure 6, Complement with Result in Figure 7:
   • More textual explanation in Figure 6, in addition to Figure 7 visualizing ChroKnowPrompt’s impact: notable decrease in Partial Known, increase in Known, and the standout effectiveness in the biomedical domain.

---

"The rest of the data are of the same type in format and structure, but only legal is unstructured and of a different type."

We included legal data to address two key goals of our benchmark:

1. Diverse Formats:
   • The legal domain provides a unique challenge due to its unstructured nature, allowing us to test LLMs' ability to process non-standard formats.
   • As noted in (Wu et al., 2024b) [1], knowledge editing and recall are particularly challenging in unstructured domains, making this an essential inclusion.
2. Domain-Specific Challenges:
   • Legal knowledge often requires nuanced reasoning about temporal and context-dependent facts, making it a valuable addition to our benchmark.

Planned Action:

We revised results in Appendix to Figure 4 in revision for including analysis of legal domain, highlighting its unique role in evaluating LLM capabilities for unstructured knowledge.

---

"Is it possible that the quality of your collected data in different time points is a more influential factor?"

This is an insightful question. The quality and consistency of data across time points are indeed critical. We took several measures to ensure reliability:

1. ChroKnowledge Results:
   • For each time point, data were curated to minimize inconsistencies and ensure alignment with the benchmark's goals.
   • The use of temporal prompts highlights the relevance of knowledge specific to each time point, mitigating biases from unequal data quality.
   • This is reflected in the results of each (A) in Figure 2, 3, 4.
2. ChroKnowPrompt Results:
   • The observed performance improvements in ChroKnowPrompt further validate the importance of multi-temporal data collection, as models perform better when provided with context from multiple time points.
   • The results visualized with Figure 7.

---

"Do more fine-grained classification of static and dynamic data. Update Figure 2 and 3 for more clarity. Move some information from the appendix to Section 7."

We sincerely appreciate these suggestions and agree they will enhance the paper.

1. Figure Updates:
   • Figures 2 and 3 is redesigned for clarity, and separated with sub plots for fine-grained analysis.
   • Also, Figure 4 of legal domain originally in the Appendix is added and revised.
   • Figure for ChroKnowPrompt is complemented with Figure 7 results.
2. Appendix Integration:
   • Analysis of ChroKnowPrompt is added in Section 7 with corresponding Appendix.

---

We believe these clarifications and revisions address your concerns. Your feedback has been vital, and we are confident the revised version will better reflect our contributions.

---

[1] Updating Language Models with Unstructured Facts: Towards Practical Knowledge Editing

Dear Reviewer D8Co,

Thank you for your thorough review of our work. We have been eagerly awaiting your feedback on our previous response and would be grateful to hear your thoughts. Additionally, we would like to share more analysis that addresses your suggestion about the fine-grained classification of data.

---

"Do more fine-grained classification of static and dynamic data."

We performed an analysis of the skewness and cumulative characteristics of each domain’s dynamic dataset.

• Biomedical Domain
  • Our analysis shows that the biomedical domain exhibits the least skewness, reflecting a balanced distribution of object changes.
  • Its characteristic, a narrower range of object change frequencies compared to the general domain, aligns with the observed performance improvements noted in our previous response: ChroKnowPrompt benefits from handling less volatile dynamic datasets, as demonstrated in Figure 7.
• General Domain
  • The general domain displays a wider range of object changes, though it is more skewed compared to the biomedical domain.
  • As its range of object change is wider, this variability impacts performance, underscoring the challenge of addressing a domain with greater changeability in object.
• Legal Domain
  • The legal domain is the most skewed, with the majority of object changes concentrated in a single occurrence.
  • Despite the narrower range of object change frequencies compared to the biomedical domain, the performance in this domain remains lower.
  • This suggests that ChroKnowPrompt is more sensitive to the dataset’s format and struggles with unstructured data.

We have included these detailed findings in Figure 8 of Appendix A.3.1, providing comprehensive visualization of skewness and cumulative patterns.

Here, we summarize the statistics of dynamic dataset object changeability.

Although the number of object changes varies across domains, the general domain tends to be more dynamic than others, we believe this effectively represents domain specificity, which contributes to the results of ChroKnowPrompt.

We would greatly appreciate your feedback on whether these additional analyses address your concerns.

If you have any further suggestions for improvement, please feel free to share any additional thoughts or questions.

Thank you for your detailed review. Your comments highlight areas for clarification and improvement.

---

“For example, one drug may have multiple side effects, each of which was identified at different times. I am unsure how the proposed benchmark and evaluation strategy addresses this accumulative nature. For example, a model that always generates the old knowledge will be considered correct based on the definition in Table 1.”

We are wondering if we might not have clearly conveyed the intended scope of our benchmark and evaluation strategy. Our benchmark aims to evaluate whether a model can track the temporal adaptability of knowledge at a given time frame, rather than just accumulating all knowledge without temporal awareness.

1. How Accumulative Knowledge is Addressed:
   • The benchmark evaluates temporal knowledge as it evolves. For instance, if side effects of a drug are identified at different times, the model must provide the correct side effect relevant to the queried time frame.
   • Here, our methodology is focused on the knowledge first, then we tracked the change of that knowledge across temporal contexts. It is also one of the main difference from Temporal Knowledge Graph(TKG), which is in the Appendix A.3.3.
   • Generating only outdated knowledge would not be considered correct unless it aligns with the specific temporal context of the query. For example, if model correctly recall old knowledge in appropriate time context, it is correct as it is right in that context.
2. Clarification on Table 1:
   • The definitions in Table 1 ensure that knowledge is evaluated based on temporal alignment, explicitly preventing a model from being rewarded for simply repeating outdated information.

Planned Action:

We revised our text to more clearly define our concept of accumulative knowledge and emphasized how our approach ensures temporal correctness while addressing evolving knowledge. Additionally, we added more detail description compared with other similar datasets.

---

“It is very difficult to understand how their results support these statements in Section 5 (Figures 2, 3).”

We acknowledge that the presentation of our results could be improved to ensure clarity.

1. Issue with Interpretation:
   • The figures may be difficult to follow due to the way the axes are scaled and labeled.
   • The narrative explaining the connection between the results and the statements in Section 5 could also be more explicit.

Planned Action:

1. We totally modified Figures 2 and 3 for more appropriate explanation with better comparability.
2. A clearer explanation of how the results relate to each benchmark dimension (e.g., temporal state, domain) is also provided in Section 5.
3. Descriptive captions for figures are expanded to reduce reliance on the main text for interpretation.

---

“The proposed prompting strategy improves in the biomedical domain, whereas results in Section 5 and Figure 3 show little variability across times. Any intuitive explanations of why?”

We appreciate the observation and acknowledge the need for a more balanced explanation.

1. Clarifying Biomedical Results:
   • The biomedical domain exhibits relatively stable performance across dynamic and static datasets, as it is less affected by temporal variability.
   • While the CHROKNOW Prompt shows some improvement in this domain, its effectiveness appears more pronounced for unchanged objects. This may stem from the biomedical domain's low variability, which allows the model to focus on multi-temporal prompts without excessive interference.
2. Variability Across Times:
   • In domains with high temporal variability (e.g., general knowledge), the CHROKNOW Prompt reduces ambiguity for unchanged objects to an extent but faces notable limitations for changed objects. These limitations likely arise from the inherent complexity of dynamic changes, which require more sophisticated temporal reasoning approaches beyond the current prompt design.

By addressing these limitations, our CHROKNOW Prompt suggests as an exploratory step toward handling temporal knowledge though it is not a complete solution.

Planned Action:

We explicitly added more explanations in Section 7 with suitable results in Figure 7, emphasizing how temporal variability impacts the effectiveness of our prompting method, which is reducing Partial Known by increasing Known category(completely known in all time frame).

“The ChroKnowBench considers only changes on object while keeping subject and relation unchanged. How does it capture the change of relation between the subject and object?”

Our benchmark emphasizes object-level changes as the primary focus for measuring temporal knowledge dynamics. This decision reflects a balance between practical considerations and the methodological goals of ChroKnowBench. Below, we address this design choice in detail:

1. Current Design and its Scope:

   • ChroKnowBench evaluates the ability of models to identify and align temporal knowledge for specific object-level changes at given time points.
   • For example:
     • Query 1: "In 2001, Zidane was a player at Real Madrid"
     • Query 2: "In 2010, Zidane was a coach"
     • These are treated as distinct evaluations to isolate how well the model reasons about object-level transformations while keeping the subject and relation fixed.

   This design captures significant, interpretable changes in knowledge over time while maintaining a structured and scalable evaluation process.

2. Why Object-level Focus:

   • Scalability:
     • Tracking relational changes creates combinatorial complexity in subject-relation-object triples. Object-level focus ensures scalability and clarity.
   • Precision:
     • Object changes capture nuanced updates (e.g., roles, affiliations) better than relation changes, which are often binary (e.g., “is a player” → “is not a player”).
   • Flexibility:
     • Unlike Temporal Knowledge Graphs which may overlook fine-grained changes in individual knowledge, our object-centered approach avoids rigid relation constraints, enabling precise and interpretable evaluations.

Planned Action:

We expanded our discussion in Introduction, Section 3, Appendix A.3.2 and A.3.3 to explain the rationale behind our focus on temporal knowledge along with sufficient comparison.

---

“It is hard to interpret these results; suggest using the same y-axis range.”

We understand the concern and agree that consistent y-axis ranges would improve the interpretability of the figures.

Planned Action:

1. We separated same kind of simple line plot into three different kind of plots for its own purpose: heatmap for checking each model’s performance in temporal-wise analysis, spider plot for verifying effect of template in each type of dataset, and bar plot for object changes in dataset in the purpose of detail analysis. Each Figure 2, 3 and additional 4 represents each domain, which becomes more easy as all Figure has same form.
2. Annotations highlighting key trends and variations added to aid interpretation in Section 5 and corresponding analysis in the Appendix.

---

“What does this mean: models show reduced knowledge in multiple-choice question answering (MCQA) settings, highlighting the impact of formatting and prompting on temporal abilities?”

We apologize for the unclear description regarding the template-wise comparison in the temporal results section.

Summary of Findings:

• Before revision, Figure 6 revealed lower and fluctuating scores in dynamic datasets (e.g., legal domain) between 2015–2016 compared to static datasets, suggesting that formatting affects model performance, particularly in specific domains.
• To address contradictory trends in general and biomedical domains, we introduced an additional template: True/False (T/F).

Key Insights from Figures 2, 3, and 4:

1. Direct answer generation is the most challenging format across domains.
2. MCQA and T/F templates complement each other:
   • Biomedical domain: Consistent high performance with MCQA.
   • Legal domain: Significant gains with T/F format.

Planned Action:

We rewrote the section and modify figures to clarify the relationship between formatting, prompting, and temporal reasoning, providing insights from our experiments to illustrate the impact.

---

“What does this mean: models fail to track recent updates between 2022 and 2023; Static datasets remain consistent, reflecting the stable nature of scientific knowledge?”

We appreciate the opportunity to clarify these statements:

1. Dynamic Datasets (2022–2023):
   • In Figure 3, models struggle with knowledge recall between 2022–2023 due to the recency of this information and limited exposure during training.
2. Static Datasets:
   • These datasets, by design, focus on stable knowledge. Consistent performance in this category indicates that models can recall well-established facts, even without recent updates.

Planned Action:

We rephrased these statements in Section 5.1 to better distinguish dynamic and static datasets and their implications for model evaluation.

---

We hope these responses address your concerns. Your feedback has been essential in improving our work, and we look forward to refining it further. Thank you for your review.

We greatly appreciate your insightful feedback, which has been instrumental in improving our paper.

---

“A table summarizing average results for each benchmark dimension—Domain, Format, Temporal State, and Time Dependency—would help clarify performance. Currently, it’s unclear whether the performance limitations stem from temporal aspects or the models' domain-specific capabilities.”

We agree that summarizing results across dataset dimensions improves clarity. Current figures visualize trends but lack explicit summaries.

More clarified observations:

• Performance of general and biomedical domains decline over time; generation tasks show lowest score but MCQA and TF remain robust.
• The legal domain performs well on static datasets but struggles with dynamic datasets, indicating a challenge of temporal reasoning in unstructured format.

Planned Action:

1. We reorganized Figures 2 and 3 into three distinct visuals: a heatmap for temporal state performance, a spider plot for format-wise performance over time, and object change statistics.
2. Each subsection now includes concise explanations to highlight findings across Temporal State, Format, and Domain.

---

“How do the authors justify these outcomes as evidence of the prompting method’s efficiency?”

1. Dynamic vs. Static Knowledge

   Biomedical data, being less volatile, shows stable results across dynamic and static contexts.

2. Object Change Analysis:

   In Section 7, we analyzed CHROKNOW Prompt’s performance, finding it effective for unchanged objects but less so for changed ones, underscoring its potential and areas for improvement in handling dynamic datasets.

3. Why Biomedical Shows Higher Gains

   • In the general domain (high variability), gains are limited due to a larger proportion of changed objects.
   • In biomedical and legal domains, with lower variability, show greater gains in unchanged objects.

4. Demonstrating the Prompt's Efficiency:

   The CHROKNOW Prompt leverages inherent capabilities for temporal knowledge elicitation without relying on external mechanisms such as retrieval.

Planned Action:

We clarified these findings in Section 7, supported by Figure 7, and expanded on domain-specific observations.

---

“In Line 313, the authors state that ‘GPT-4 performs best in generation tasks.’ Do they provide any rationale for this observation?”

As you noted, GPT-4o-mini (might be mistaken as GPT-4) excels in generation tasks across general domain but is less dominant in other cases. While we don't know details in architecture, its performance in knowledge elicitation suggests substantial exposure to relevant formats during the training stage. Similarly, open-source models like SOLAR excels in MCQA tasks, whereas Gemini-1.5-pro demonstrates weaker performance in this area. Specific formats and data distributions they were exposed to may shape their strengths and weaknesses.

Planned Action:

Results are clarified through a new figure and accompanying descriptions.

---

“The paper would benefit from more detailed error analysis and examples. In which cases does the model fail to generate correct answers, and are there discernible patterns in these errors?”

We agree that deeper error analysis adds valuable insights.

Planned Action:

• Section 7.2 now includes error analysis from three perspectives: Object changeability, Chronological Spans, and Chat Prompting.
• Supplementary materials in the Appendix provide experimental details.

---

“What is the evaluation score for the ‘Generation’ experiments in Figures 2 and 3? How do you extract the answer from the model output? Is it an exact match?”

Evaluation was conducted using a fuzzy matching strategy, as detailed in Section 2.2 and Appendix A.2.2:

1. Model-generated outputs were parsed and compared with ground-truth object pools using fuzzy match criteria.
2. Scores were calculated as the ratio of matches to queries.

Planned Action:

While this is mentioned in Section 2.2, we made it more explicit in the Figures 2 and 3 for clarity.

---

“Is this with the 5-shot setting or with the zero-shot and Chain-of-Thought generation?”

We apologize for confusion in experimental details. As in Section 2.2 and the appendix:

1. We used a 4-shot setting as per MMLU strategy, not 5-shot or zero-shot.
2. Chain-of-Thought generation was excluded to focus on direct knowledge checking.

Planned Action:

We referenced more papers reflecting this points, in addition to third template: True/False.

---

“Are the results for the ‘Time Invariant’ data being reported?”

Yes. The results for "Time Invariant" data are in Appendix A.5 (Figure 9).

Planned Action:

Figure 1 and Section 5.1 have been revised to present these findings more prominently.

---

Thank you for your feedback, which has greatly enhanced our work. We believe these revisions address your comments effectively.