An Empirical Study on Reconstructing Scientific History to Forecast Future Trends

Dear Reviewer 8usN,

Thank you for your thorough review and valuable feedback. We address your concerns as follows:

Response to Weaknesses

w1. Missing References

We acknowledge the omission of references ([1-4]) and will incorporate them. While our framework focuses on temporal reasoning and causal forecasting, the cited works primarily cover combinatorial impacts and link prediction, which, though complementary, are not directly comparable. We will clarify their relevance to our approach.

w2. Comparison with More Baselines

We agree that additional baselines, including those from [1-4], would improve the rigor of our study. However, our framework combines historical synthesis with future prediction, making direct comparisons difficult. Reasons for non-comparison include:

We will clarify baseline selection in the final version.

w3. Lack of Ablation Study

Although we did not include ablation study results without temporal reasoning, early tests showed significant degradation in both historical synthesis and prediction accuracy without it. Temporal reasoning clearly improves performance, and we prioritized its inclusion in HorizonAI.

We will include these results in the final version to demonstrate the necessity of temporal reasoning.

w4. Limited Scope to LLM-Related Topics

The focus on LLM-related topics was intentional for the following reasons:

• Clear and Well-Defined Trajectories: Research on LLMs exhibits rapid development and clear trends, making it an ideal testbed for evaluating HorizonAI’s ability to forecast research directions.

• Clear Benchmark and High-Quality Data: A defined scope ensures the availability of high-quality data and allows us to perform robust evaluations, such as comparisons with AutoSurvey. HorizonAI, as a framework, is inherently domain-agnostic. Its key components, including PaperTKG for structured knowledge representation and LLMs for reasoning, are adaptable to any scientific domain with appropriate data sources.

• Future Generalizability: We are already planning experiments in other domains, such as bioinformatics and materials science. However, due to the complexity and costs involved, these experiments require extensive time and domain-specific validations. To address this in the long term, we are collaborating with industry partners to conduct real-world evaluations in these fields.

Response to Questions

Why Are the Topics Related to LLMs?

As mentioned above, the focus on LLM-related topics is deliberate to ensure rigorous evaluation with well-defined research trajectories.

We respectfully request that you reconsider your evaluation, as we believe HorizonAI offers significant advancements in forecasting research trends through its combination of temporal reasoning and LLM-based analysis.

Thank you once again for your insightful feedback.

Best regards,
Authors

Dear Reviewer AXnm,

Thank you for your comments，we will provide a more detailed explanation of our innovative contributions：

1. Novelty of HorizonAI: A Dual-System Framework for Temporal Knowledge Graph Construction and Reasoning

We respectfully disagree with the perception that our use of a knowledge graph is a straightforward historical summary. In fact, HorizonAI represents a significant innovation in the field by being the first system to fully automate the construction of temporal knowledge graphs (TKG) using LLMs, and to use these graphs for both historical completion and future forecasting with causal and temporal reasoning.

• Automated TKG Construction: HorizonAI leverages LLMs to automatically build and update temporal knowledge graphs (TKGs), a major departure from static or manually curated systems. This dynamic construction allows for continuous updates as new research data emerges, making it highly adaptable in fast-moving fields like LLM.
• Historical Completion and Future Predictions: Beyond generating historical summaries, HorizonAI uses its TKG to fill in gaps in historical research trajectories, predicting how research fields evolve. This temporal-causal reasoning provides deeper insights compared to traditional systems that only offer trend extrapolation.
• Dual-System Architecture: HorizonAI’s dual-system framework integrates both historical analysis and future predictions. This combination not only synthesizes past research but also uses predictive modeling to forecast the next phase of research, making it a powerful tool for identifying emerging trends.

2. Differences from Si et al. (2024)

We acknowledge that Si et al. (2024) introduce a novel idea generation system. However, there are fundamental differences in our research trajectory and goals:

• Si et al. (2024) focus primarily on the generation of novel research ideas, providing a framework to suggest new directions in research based on existing literature and trends.
• In contrast, HorizonAI is focused on understanding the evolution of research trajectories over time. Our framework not only identifies past research trends but also predicts future developments in research fields through temporal-causal reasoning. For example, Si et al. (2024) might propose novel areas like AI models for specific tasks, while HorizonAI will trace how these architectures evolve and predict future breakthroughs or developments within the field includes AI models for specific tasks.

3. Dynamic and Real-Time Updates in HorizonAI

One of the key differentiators of HorizonAI is its ability to perform dynamic, real-time updates as new research is published. This feature ensures that our system remains relevant and accurate in fields that are constantly evolving, such as LLM and other cutting-edge AI areas. Unlike Wang et al. (2024) and Si et al. (2024), which focus on static models or pre-existing datasets, HorizonAI continuously updates its temporal knowledge graphs, ensuring that the research trajectories it analyzes reflect the latest advancements in the field.

4. Task Versatility and Enhanced Performance

While Si et al. (2024) focus on generating new research ideas, and Wang et al. (2024) focus on summarizing research history, HorizonAI is designed to tackle both tasks with a broader and more versatile scope. In addition to generating historical summaries, HorizonAI provides robust predictions grounded in causal and temporal reasoning.

For example, in the LLM domain, HorizonAI can not only synthesize the development of transformer-based models but also predict how future LLM architectures might evolve, For example, expanding the Knowledge Base to prevent the compression of LLM model data from causing knowledge loss. This predictive power is a key strength that sets our system apart from Wang et al. (2024) and Si et al. (2024), who do not provide such forward-looking predictions grounded in temporal-causal logic.

5. Conclusion: Unique Contributions of HorizonAI

We believe that HorizonAI offers a unique and innovative approach to understanding and predicting the evolution of research. By combining LLM-driven automation with temporal knowledge graph (TKG) construction and causal reasoning, it provides deeper insights into research trajectories and future research trends. These contributions are distinct from the novel idea generation presented by Si et al. (2024), which primarily focuses on suggesting potential future research topics without providing a deeper understanding of their evolution.

We greatly appreciate your continued engagement and thoughtful feedback, and we look forward to revising the manuscript to better highlight these distinctions.

Best regards,
Authors

Dear Reviewer AXnm,

Thank you for your thoughtful review and feedback. We address your concerns as follows:

Response to Weaknesses

W1. Contribution Comparison with Wang et al. (2024) and Si et al. (2024)

We respectfully disagree with your assessment that our work’s use of a knowledge graph is a straightforward historical summary. The proposed PaperTKG is not a conventional knowledge graph; it incorporates causal and temporal logic into its structure, enabling a deeper understanding of research trajectories and their evolution over time. Specifically:

• Dynamic and Real-Time Updates: Unlike static systems, HorizonAI dynamically updates its historical synthesis and future projections as new data becomes available. This feature ensures its applicability in rapidly evolving fields, which Wang et al. (2024) and Si et al. (2024) do not address.
• Ablation Studies for Historical Quality: We conducted ablation experiments that demonstrate the importance of quality over quantity in historical synthesis. Our strategy ensures both comprehensive and high-quality historical summaries, which are validated against ResBench.
• Future Predictions with Temporal-Causal Logic: HorizonAI’s future predictions leverage temporal-causal reasoning within the PaperTKG, offering richer insights into research trends compared to simple trend extrapolation.

W2. Real-World Applications

We acknowledge the importance of clearly articulating the real-world impact of our framework. HorizonAI has several practical applications that benefit researchers, organizations, and policymakers:

• Efficiency in Research: By identifying relevant and high-impact papers, HorizonAI enables researchers to focus on the most pertinent work in their field, significantly improving research efficiency.
• Discovery of Hidden Connections: HorizonAI uncovers implicit relationships between research topics, providing new perspectives and directions for interdisciplinary studies.
• Guiding Research Funding and Policy: HorizonAI’s ability to predict future trends can guide funding agencies and policymakers in allocating resources to emerging areas with high potential impact.

W3. Evaluation Metric Reliability

Thank you for raising concerns about the reliability of our evaluation metrics. While it is true that some aspects rely on LLM-generated scores, we have implemented additional measures to ensure robustness:​

• Human Validation: In addition to LLM-based evaluations, we conducted human validation for predictive accuracy. This involved comparing the JSON-defined directions and steps generated by HorizonAI with target papers, using a semantic similarity threshold (e.g., over 50% overlap in key steps) to score predictions.​
• Alignment of Validation Strategies: The human validation process mirrors the LLM-based evaluation strategy, ensuring consistency and reducing potential biases in scoring.​ ​

W4. Explanation of Citation Overlap Metrics

We define:

• Citation Overlap: The percentage of papers identified by HorizonAI that overlap with citations in target papers.
• Key Citation Overlap: Focuses on foundational papers, which are more significant than general citation overlap.

W5. Comparison with Si et al. (2024)

While we acknowledge the relevance of Si et al. (2024) to our work, their task differs significantly from ours. Si et al. primarily focuses on novel idea generation, while HorizonAI integrates historical synthesis with future forecasting based on causal and temporal reasoning. This fundamental difference makes direct comparison challenging. However:

• Complementary Contributions: Si et al. (2024)’s work provides valuable insights into generating novel ideas, whereas HorizonAI offers a comprehensive framework for analyzing and predicting research trajectories. We will discuss these distinctions in the revised paper.
• Future Comparisons: Although Si et al.’s system was not publicly available ,when we were doing our research, we plan to implement similar experimental setups for comparison in future work.

W6. Experimental Details

We used GPT-4-o for consistency in all experiments. Future work will include additional comparisons with other models (e.g., PaLM, Claude). We will expand the experimental section to clarify the methodologies used.

Response to Questions

Q1. Novel Contribution Compared to Wang et al. (2024) and Si et al. (2024)

See response to W1.

Q2. Real-World Significance

See response to W2.

Q3. Explanation of Evaluation Metrics

See responses to W3, W4, and W6.

Q4. Comparison with Si et al. (2024)

See response to W5.

We believe HorizonAI offers significant advancements in forecasting research trends with its integration of temporal reasoning and LLM-based analysis. We respectfully request you reconsider your evaluation.

Thank you again for your valuable feedback.

Best regards,
Authors

Dear Reviewer YdPc, Thank you for your thorough review and valuable feedback. We address your concerns as follows:

Response to Weaknesses

W1. Dataset Size and Scope

We acknowledge your concerns regarding the dataset size and scope, and we would like to clarify the following:

• Comprehensive Dataset Design: While the visible dataset may seem small, it is both rigorous and highly effective. We publicly released 20 topics, each including 1 source papers, 10 target papers per topic, and a single comprehensive survey paper, amounting to 240 papers in total. Each survey paper represents the most comprehensive historical summary for its respective topic, carefully curated through multiple rounds of evaluation.
• Scalability Through PaperTKG: Beyond the visible dataset, the temporal knowledge graphs (PaperTKG) constructed by HorizonAI encompass an average of 15,000 papers per topic, dynamically linking historical research to future directions. This scalability ensures the framework’s robustness and adaptability to larger datasets.
• Expert-Validated Historical Nodes: The dataset includes critical historical nodes identified and annotated by domain experts, ensuring both accuracy and relevance. This combination of machine-based synthesis and expert validation significantly enhances the dataset's quality.

W2. Baseline Justification

We agree that AutoSurvey serves as a meaningful baseline for our task. However, we would like to elaborate on the relationship between our framework and AutoSurvey:

• Historical Summarization Beyond Surveys: Our task extends beyond simple historical summarization. HorizonAI not only synthesizes past research but also dynamically fills historical gaps, enabling more robust survey generation. This process includes strategies for completing topic-specific histories from a single paper—a capability that directly aligns with AutoSurvey’s objectives but also surpasses them in scope.
• Strong Baseline Choice: AutoSurvey relies on expert-curated datasets for historical evaluation, making it an ideal benchmark for assessing the completeness and quality of HorizonAI’s historical synthesis. Our framework demonstrates competitive performance compared to AutoSurvey while introducing a temporal-causal reasoning layer for future predictions.

We believe AutoSurvey is a suitable baseline for evaluating historical completeness, and the results validate HorizonAI’s efficacy. We will revise the manuscript to further clarify and justify this choice.

W3. Accuracy and Evaluation Framework

We appreciate your feedback on the evaluation framework and would like to clarify the following:

• Accuracy and Validation: Accuracy was weighted highly in our evaluation because it incorporates human validation alongside LLM-based scoring. This ensures the reliability and credibility of our results. For example, human evaluators assessed whether the generated directions aligned with the target papers’ steps and semantic content (e.g., over 50% overlap in direction-specific steps was considered a match).
• LLM Evaluation Strategy: Subsequent evaluation criteria (e.g., semantic similarity, temporal consistency) leverage the strengths of LLMs in content analysis. The weighting reflects a balance between human-validated metrics and the nuanced insights provided by LLMs.

W4. Relationship Between Historical Summarization and Future Prediction

We respectfully disagree with the notion that the tasks of historical summarization and future prediction are loosely connected. In HorizonAI, these tasks are inherently intertwined:

• Impact of Historical Completeness on Prediction: The effectiveness of future prediction depends on the quality and completeness of historical synthesis. Incomplete or inconsistent historical data leads to less reliable predictions, as demonstrated in our ablation experiments. This mirrors human cognition, where missing or fragmented memories hinder accurate reasoning about future events.
• Temporal-Causal Logic: The use of PaperTKG as a temporal-causal structure bridges the two tasks. By embedding both temporal and causal relationships, PaperTKG connects historical patterns to future trends, enabling HorizonAI to infer plausible research trajectories.
• Baseline Comparisons: To isolate the impact of historical synthesis, we used GPT-4-o as the baseline model for predictions without historical context. The results confirm that integrating historical knowledge significantly improves prediction quality, validating the synergy between the two tasks.

We respectfully request that you reconsider your evaluation, as we believe HorizonAI offers significant advancements in forecasting research trends through its combination of temporal reasoning and LLM-based analysis.

Thank you again for your thoughtful feedback and for providing us with the opportunity to improve our work.

Best regards,
Authors

Dear Reviewer bFb5,

Thank you for your thorough review and constructive feedback. We appreciate your comments, which have helped us clarify key aspects of our work. Below, we address your concerns in detail.

Response to Weaknesses

Weakness 1: Limited Scope to AI Topics

We agree that broader applicability is important for demonstrating HorizonAI's versatility. However, the focus on AI topics is intentional for the following reasons:

• Clear Benchmark and High-Quality Data: A defined scope ensures the availability of high-quality data and allows for robust evaluations, such as comparisons with AutoSurvey. HorizonAI is domain-agnostic, with key components like PaperTKG and LLMs adaptable to any scientific domain with appropriate data.
• Future Generalizability: We are planning experiments in other domains, such as bioinformatics and materials science. These require extensive time and domain-specific validation, but we are collaborating with industry partners to conduct real-world evaluations in these areas.

Thus, while the current work focuses on AI, HorizonAI’s architecture and methodology are designed for broader applicability.

Weakness 2: Limited Data Sources

We acknowledge the concern about reliance on arXiv. However:

• ArXiv’s Breadth: ArXiv is a comprehensive repository with broad coverage in fields like physics, mathematics, and engineering.
• Adaptability to Other Sources: HorizonAI can integrate data from sources like Semantic Scholar and Google Scholar. The modular design of PaperTKG ensures seamless incorporation of various data formats. While arXiv suffices for this work, the framework is easily adaptable to other sources.

Weakness 3: Lack of Expert Review

We appreciate the concern regarding the absence of expert reviews. Due to time and resource constraints, we were unable to include expert evaluations in this iteration. However:

• Mitigation with ResBench Calibration: We used ResBench, a benchmark calibrated with human-verified data, ensuring LLM scoring aligns with realistic, practical benchmarks.
• Transparent and Rigorous Evaluation: We selected AI topics with well-defined trajectories to cross-validate predictions against recent trends, ensuring reliability.
• Scoring Methodology: Our scoring methodology compares JSON-generated future directions to target subtitles, counting overlaps exceeding a 50% threshold. This ensures objective and consistent evaluation. We plan to incorporate expert reviews in future iterations.

Weakness 4: Dataset Size

We understand the concern about dataset size. However, our dataset is carefully curated and effective for the intended evaluations:

• Comprehensive Coverage: We released 20 topics, each with 1 source papers and 10 target papers, resulting in 200 target papers and 20 surveys. Each survey represents the most comprehensive historical summaries for its topic.

• Large-Scale Knowledge Graphs: While the visible dataset size may appear small, the underlying knowledge graphs built by HorizonAI are extensive, with each topic incorporating approximately 15,000 papers, totaling around 300,000 papers across all topics.

• Focus on Quality: We prioritized quality over quantity, selecting representative works for each topic to ensure accurate historical synthesis and future predictions.

Question: Improvements Over Pre-LLM Methods

We appreciate your request for clarification on how HorizonAI improves upon pre-LLM methods. Unfortunately, we have not found pre-LLM works addressing the same dual synthesis and prediction task comprehensively. If you have specific references, we would be grateful for the comparison and will include them in future revisions. That said, HorizonAI offers several distinct advantages:

• Unified Framework: Unlike traditional methods that focus on historical synthesis or prediction in isolation, HorizonAI integrates both tasks into a single framework.
• Dual-Component Design: By combining PaperTKG (structured knowledge graphs) with LLM reasoning, HorizonAI enables robust historical modeling and accurate forecasting, leveraging the strengths of both methodologies.
• Benchmark Innovation: ResBench provides a novel evaluation standard specifically for the dual tasks of historical completeness and predictive reliability.

We believe these innovations represent meaningful advancements over existing methods.

Conclusion

We hope this response clarifies the strengths of HorizonAI. By focusing on a defined scope, leveraging a rich dataset, and employing robust methodologies, we demonstrate the framework’s potential. HorizonAI is adaptable to other domains, and we are working to extend its applications to new fields.

We respectfully request that you reconsider your assessment, as we believe the issues raised can be addressed without detracting from the novelty and utility of our approach.

Thank you again for your valuable feedback.

Best regards,
Authors