Beyond the Reported Cutoff: Where Large Language Models Fall Short on Financial Knowledge

We thank the reviewer for engaging with our paper and recognizing its clarity and methodological transparency. While we appreciate the feedback, we respectfully would like to clarify several aspects of the assessment. Our work intentionally focuses on a well-scoped and underexplored problem—temporal and cross-sectional financial knowledge gaps in LLMs—and makes methodological contributions that we believe are of independent value. Below, we clarify our motivations, address points of misunderstanding, and highlight how our results offer actionable insights for both researchers and practitioners in the LLM-for-finance community.

“The paper's contribution is incremental in nature. It lacks a novel approach or an innovative analytical method, and does not provide a new empirical application of natural language processing techniques to finance.”

Response: We respectfully disagree with the characterization of our contributions as incremental. To our knowledge, this is the first study to systematically assess temporal and cross-sectional knowledge gaps in LLMs within the financial domain using a large-scale, multi-decade QA benchmark. Prior work has primarily focused on post-cutoff date issues or benchmark performance on generic financial tasks, whereas our framework evaluates LLMs retrospectively—across time and firm characteristics—introducing a new lens for auditing factual knowledge in pretrained models.

Our methodology includes:

• A 197K QA benchmark spanning 43 years and 17K+ companies;

• A cross-sectional regression framework analyzing biases across market cap, investor attention, and readability—factors previously unexplored in this context;

• A joint analysis of accuracy and hallucination behavior, revealing paradoxical overconfidence patterns in high-cap firms.

Rather than presenting a new application of NLP to finance, our contribution lies in auditing the epistemic boundaries of LLMs—an increasingly vital area of inquiry as these models are adopted in high-stakes settings like finance. We believe this diagnostic perspective is both novel and broadly relevant.

“Analyzing raw model behavior alone may offer limited insights... revenue recall is a narrow reflection of financial knowledge.”

Response: We agree that real-world systems often benefit from retrieval components, but our goal is to isolate the pretrained model's knowledge—a standard and necessary baseline in LLM evaluation. Many high-stakes applications (e.g., retail investing tools) still rely on raw LLM outputs without augmentation, making this analysis highly relevant. We chose revenue precisely because it is a canonical, time- and firm-sensitive variable with ground truth available, enabling rigorous, large-scale evaluation.

“The conclusions may lack depth… It is expected that models will have a knowledge cut-off and favor larger firms with more coverage.”

Response: We would like to clarify that our contribution lies in quantifying and verifying these assumptions systematically and at scale, which had not been done before. Crucially, we show that hallucination rates also rise for large-cap firms and recent years—contrary to expectations—suggesting overconfidence where LLMs appear most reliable. This tension between knowledge recall and hallucination has direct implications for downstream use as demonstrated in Appendix K. We believe surfacing and formalizing this nuanced behavior adds both depth and value to the community’s understanding.

“The target audience appears highly specialized within finance… unclear if the topic aligns with the broader COLM audience.”

Response: While our primary evaluation is in finance, we explicitly demonstrate the generality of our framework through a domain-transfer study on La Liga soccer statistics (Discussion, Section 5; Appendix L). There, we find similar temporal and prominence-based knowledge biases in model performance—supporting the broader applicability of our approach. Our methodology is domain-agnostic and can be used to assess LLM factual knowledge across any structured entity-time setting (e.g., law, medicine, history), which we believe aligns well with COLM.

We would also like to respectfully note that the motivations, novelty, and broader applicability of our framework are clearly outlined in the Abstract, Introduction and Discussion sections. We hope these clarifications help with a renewed assessment of the full scope and relevance of our contributions.

We thank the reviewer for their thoughtful assessment of our work and for recognizing the contributions we make, including the construction of a large-scale dataset, the study of retrograde and cross-sectional knowledge biases, and the insights into how the same factors that drive model accuracy also increase hallucination rates. We are encouraged by the acknowledgment that these findings are important and novel, and we appreciate the opportunity to address the constructive critiques raised.

Theory of LLM:

We respectfully clarify that our work is not intended as a theoretical or architectural study of LLMs. Rather, our goal is to empirically uncover structured biases in model behavior—specifically, how temporal and cross-sectional financial factors influence both model success and hallucination. As noted in the Discussion section (section 5), we believe establishing these empirical regularities is a critical prerequisite for any theoretical analysis. Given the lack of access to model internals or detailed pretraining corpora, we intentionally refrain from speculative causal claims.

Finance Domain Models

We appreciate this suggestion. As noted in Appendix F (lines 626–633), we did include FinMA-7B—a finance-specialized LLM—in our evaluation. However, the model frequently failed to produce usable outputs (e.g., returning empty strings or numbers without units), limiting its reliability for large-scale analysis. That said, we would be happy to move these results into the main paper in the camera-ready version if the reviewer believes it would improve clarity and completeness.

We also note in our Ethics section that many prominent finance-specific LLMs (e.g., BloombergGPT) are not publicly available for inference, making them difficult to evaluate at scale. We hope our framework will be a useful tool for assessing such models as they become more accessible.

We hope that the breadth of our empirical analysis, combined with the clarity of our framing makes a compelling case for acceptance.

We sincerely thank the reviewer for their thoughtful and constructive feedback. We are especially grateful for the recognition of our work’s contributions—both in terms of dataset novelty and the analysis of LLM behavior along temporal and cross-sectional dimensions. We also appreciate the reviewer’s insightful observations about the paradoxical trends we document, as well as the broader applicability of our evaluation framework. In what follows, we address the concerns raised, particularly around full dataset usage, scope, and the need for more discussion on the causal underpinnings of observed biases.

Limited evaluation for full dataset

Thank you for your understanding regarding the API cost constraints. As noted, we evaluated expensive models like GPT-4.5 and Gemini on a representative subset, while running full-sample evaluations for open or lower-cost models like Llama-8B, Llama-70B, GPT-4o-mini and GPT-4o. Encouragingly, the key trends—such as retrograde bias and cross-sectional disparities—are consistent across both subsets and full-sample evaluations, supporting the robustness of our findings.

Narrow focus:

We appreciate the suggestion to broaden the scope beyond revenue-based questions. While our core analysis focuses on revenue to enable clean evaluation of factual knowledge, we emphasize in Section 5 and Appendix L that our framework is generalizable across domains—we illustrate this by applying it to Spanish soccer league statistics. Additionally, in Appendix K, we explore multi-turn prompting for stock recommendations, showing how revenue knowledge gaps influence downstream financial decisions. These extensions underscore the broader relevance and applicability of our evaluation approach.

Causal explanation:

We agree that understanding the root causes of retrograde and paradoxical hallucination biases is an important direction. However, with the lack of access to pre-training data—even for open-weight models—causal analysis (e.g., ablations or retraining) is not feasible. That said, we hypothesize these trends stem from disparities in pre-training data availability: recent years and prominent companies are more widely discussed and thus likely overrepresented. We partially support this through correlations with SEC access logs and retail/institutional attention, but refrain from making causal claims without further evidence.

We hope these clarifications highlight the broader relevance and rigor of our work, and we thank the reviewer again for the thoughtful engagement.

We appreciate your thoughtful and constructive review. Your recognition of our exploration into the depth and bias of financial knowledge within LLMs—especially under a zero-shot setup across a large and diverse set of financial QA instances—is encouraging. We are particularly grateful for your acknowledgment of our systematic evaluation pipeline, extensive dataset design, and reproducibility practices. Below, we address your concerns regarding scope, prompting strategies, dataset coverage across models, and your question about the 2021–2022 dip.

Geographic Scope:

Thank you for pointing this out. Our study focuses on U.S. publicly traded companies, which includes many international firms such as Toyota from Japan (TM), Alibaba from China (BABA), Infosys from India (INFY) and many more. Additionally, many top US companies have most of their revenue coming from outside the US and are heavily influenced by global events. Moreover, as discussed in Section 5 and Appendix L, our evaluation framework is generalizable beyond finance—we also demonstrate its applicability in assessing knowledge biases in domains like Spanish soccer.

Prompting Strategies:

We use zero-shot prompting to avoid injecting answer-related bias (e.g., via revenue in one-shot examples). Manual checks (Appendix D) confirm that models understood the prompt well. Additionally, we explore more sophisticated prompting strategies in Appendix K.

Dataset Coverage Across Models:

Due to API cost constraints (e.g., $3,000 for full-sample GPT-4.5), we ran expensive models on a *representative subset* ($120). In contrast, we evaluated open or lower-cost models Llama-8B, Llama-70B, GPT-4o-mini and GPT-4o on the full dataset. Importantly, our key findings remain consistent across models.

Question about the 2021–2022 Dip:

We discuss this anomaly in lines 199–204 and agree it merits further study. Emerging work [1] explores why LLMs may misalign with their stated cutoffs. Our study complements this by going beyond cutoffs—probing retrograde and cross-sectional knowledge gaps. We view this as a growing area for future research.

[1] Cheng, Jeffrey, et al. "Dated data: Tracing knowledge cutoffs in large language models." arXiv preprint arXiv:2403.12958.

We appreciate your thoughtful engagement with our work. We hope our clarifications address your concerns and reinforce the broader significance and rigor of our study—both in methodology and in its implications across domains.