FinTruthQA: A Benchmark Dataset for Evaluating the Quality of Financial Information Disclosure

We appreciate your time and effort in reviewing our work and these insightful comments, which have helped us improve our manuscript. We provide a list of responses below: (W=Weaknesses; RW=Response to Questions)

W1: Although the dataset focuses on investor Q&A in the Chinese stock exchange context, the lack of testing across different regulatory and cultural contexts limits its generalizability. Without broader applicability, the impact of FinTruthQA as a global benchmark remains restricted.

RW1: We appreciate your insightful feedback and the opportunity to address this concern regarding the generalizability of FinTruthQA. We would like to highlight that the proposed FinTruthQA dataset is rooted in two novel platforms within China’s financial disclosure system, specifically the investor Q&A platforms provided by the Shanghai Stock Exchange (SSE) and Shenzhen Stock Exchange (SZSE). These platforms represent a unique and innovative regulatory approach, enabling direct communication between investors and listed companies. To our knowledge, similar platforms are not widely observed in other regulatory systems globally, making FinTruthQA a pioneering effort to study and analyze regulated corporate-investor interactions in this context.

While the platform itself is unique to the Chinese financial system, the methodologies and principles underpinning FinTruthQA are not confined to any single language or regulatory framework. The annotation framework, criteria, and pipeline we developed are inherently language-agnostic and adaptable. All the code and processes were designed for adaptability, enabling their easy application across various languages and cultural environments. The detailed description of the criteria, coupled with our inclusion of advanced machine learning and pre-trained models, provides a replicable framework for extending FinTruthQA to other regions and contexts. For example, the criteria employed (question identification, question relevance, answer readability, and answer relevance) are universally applicable to similar platforms worldwide. By demonstrating the framework’s efficacy in one regulatory context, we establish a proof of concept that can inspire similar efforts in other markets, thereby expanding its generalizability over time.

W2: The reliance on human annotation for subjective criteria (e.g., answer readability and relevance) introduces potential biases. While the authors conducted quality control, it is unclear how consistently these subjective labels would hold across diverse annotator groups or application settings.

RW2: We acknowledge your concerns regarding the potential subjectivity in our manual annotation process. To address this, we have established a rigorous annotation procedure, as detailed in Section B of our appendix. All annotators underwent multiple rounds of training and pre-annotation exercises, comprising five iterative sessions. During these, feedback was provided to refine annotators’ understanding and agreement levels on subjective criteria. To ensure high-quality annotation, only the top four (out of six) annotators, based on their inter-annotator agreement (IAA) scores from these rounds, were selected for the formal annotation phase. After several rounds of quality control, the IAA rate reached 85.52%, demonstrating a strong level of agreement among the annotators and the robustness and consistency of our annotation process. Comprehensive annotation guidelines were developed and included in the supplementary material.

That said, we understand the potential value of including traditional metrics for comparative purposes. We are willing to incorporate the Gunning Fog Index and similar readability baselines in a supplementary analysis for the final version of the paper, allowing for a broader comparison across methodologies.

W3: Label Distribution and Utility

RW3: We appreciate your insightful observations on the label distribution and its implications for model performance. Although Labels 2 and 3 appear less frequently, they provide critical insights into intermediate states of readability. This skew reflects the real-world data distribution in financial disclosures, where companies often aim to provide answers that are either very concise (Label 1) or very clear (Label 4). Removing or consolidating these labels would risk losing granularity in the quality assessment, which is especially important for developing nuanced NLP models. While this imbalance introduces challenges for classification models, it is representative of the actual data characteristics.

W4: Definition of Readability

RW4: We appreciate your detailed observation regarding the definition of readability in our paper. In our current framework, readability is evaluated based on ease of understanding, i.e. whether the content is clear, concise, and free of ambiguities. This choice was guided by the practical constraints and the intended applicability of the FinTruthQA benchmark in assessing financial Q&A disclosures. These dimensions align with the immediate goals of ensuring transparency and accessibility for retail investors. However, we acknowledge that readability encompasses broader attributes such as structure, clarity, style, and grammar. Our evaluation criterion implicitly considers these aspects through the scoring system (e.g., levels indicating the presence of ambiguities). We have enhanced our explanation in the revised manuscript to explicitly convey this comprehensive interpretation:

Answer readability: This criterion evaluates how easily an answer can be understood and whether it is well-written. It considers multiple dimensions, including structural coherence, clarity of information, stylistic appropriateness, and grammatical correctness, ensuring a comprehensive evaluation of the answer's quality.

References:

[1]Pitler E, Nenkova A. Revisiting readability: A unified framework for predicting text quality[C]//Proceedings of the 2008 conference on empirical methods in natural language processing. 2008: 186-195.

[2]Ghosh S, Sengupta S, Naskar S K, et al. FinRAD: Financial Readability Assessment Dataset-13,000+ Definitions of Financial Terms for Measuring Readability[C]//Proceedings of the 4th Financial Narrative Processing Workshop@ LREC2022. 2022: 1-9.

We appreciate your time and effort in reviewing our work and these insightful comments, which have helped us improve our manuscript. We provide a list of responses below: (W=Weaknesses; RW=Response to Questions)

W1: Title Discrepancy

RW1: Thank you for your thoughtful feedback regarding the title discrepancy in our paper. As outlined in the introduction of our paper, the Q&A interactions analyzed are sourced from platforms regulated by the China Securities Regulatory Commission, where companies are legally obligated to ensure the accuracy of their disclosures. Within this highly regulated framework, we define "truthfulness" not merely as factual accuracy but as a broader concept encompassing factual compliance, informational completeness, and relevance to investors' queries. Companies, while meeting their legal obligations, often provide responses that are indirect or minimally informative. This regulatory environment affects our method of analyzing responses, concentrating on aspects like readability and relevance to assess the quality of disclosure.

Recognizing that the original introduction may not have sufficiently linked the context to the dimensions of readability and relevance, we have revised the introduction to clarify this connection as follows: The regulatory framework governing these platforms ensures that listed firms are legally obligated to provide truthful and accurate disclosures. However, companies often respond with limited or indirect information, aiming to satisfy regulatory requirements without fully addressing investor concerns. This situation presents a distinct challenge in assessing the quality of disclosures, which extends beyond simple factual accuracy to include dimensions such as relevance, readability, and informational completeness. These aspects collectively shape the "truthfulness" of disclosures in this context.

In response to the reviewer’s suggestion to better align the title with the paper’s content, we propose to revise the title to: "FinDisQA: A Benchmark Dataset for Evaluating the Quality of Financial Information Disclosure". This title more accurately reflects the focus of our work on assessing the quality of disclosure using criteria such as readability and relevance, without narrowly emphasizing "truthfulness." We will implement this title revision upon acceptance of the paper.

W2: Baseline Methodologies for Readability

RW2: Thank you for your suggestion to include more established readability baselines, such as the Gunning Fog Index, in our evaluation. We acknowledge the value of traditional readability metrics in many contexts; however, our study, FinTruthQA, specifically focuses on evaluating financial Q&A data, where domain-specific nuances and contextual readability play a crucial role.

Traditional indices like the Gunning Fog Index rely heavily on surface-level linguistic features (e.g., sentence length, syllable count) which do not adequately capture the complexities of financial discourse. Studies[1] have shown that surface-level readability scores often do not correlate well with human-judged readability, particularly for domain-specific texts. FinRAD[2] also highlighted the limitations of traditional readability indices when applied to financial texts. For instance, financial answers often include numerical data, technical terms, and abbreviations that are inherently challenging to parse but essential for conveying accurate information. These aspects are not well-addressed by traditional readability metrics.

As emphasized in our paper, we adopted a fine-grained, domain-specific approach by annotating answer readability using human-labeled scores that account for clarity, coherence, and contextual appropriateness in financial communication (Sections 3.3 and 4.3.3). This methodology aligns more closely with recent trends observed in workshops such as TSAR, where modern, context-aware readability measures are favored over traditional indices. For example, the TSAR workshop proceedings highlight the use of neural and task-specific methods for assessing text simplification and readability rather than relying solely on formula-based measures. FinRAD[2] also used neural-based models to assess the readability of financial definitions.

Furthermore, our experiments demonstrate that advanced NLP techniques, such as FinBERT and RoBERTa-Extractive-QA, outperform traditional methods in readability evaluation tasks (Table 3). These models leverage pre-trained embeddings to effectively account for the semantic and structural intricacies of financial texts, providing a more reliable and nuanced assessment than formulaic indices could offer.

Dear Reviewer u6Wu,

Thank you for your valuable feedback. We have carefully addressed all your concerns and deeply appreciate your thoughtful suggestion to revise the manuscript.

We clarified the title alignment, opting for a revised title emphasizing disclosure quality rather than "truthfulness." For readability baselines, we prioritized domain-specific, context-aware metrics over traditional formulas, offering to include traditional metrics for comparative analysis in supplementary materials. We addressed concerns about label distribution, highlighting its reflection of real-world data while retaining granularity for nuanced NLP modeling. Additionally, we expanded the definition of readability, ensuring a comprehensive evaluation framework.

We hope these revisions comprehensively address your comments and are happy to discuss any additional concerns you may have.

Best regards.

I thank the authors for their response. I have increased my score for "Soundness" score based on their response.

I still have some reservations about the Label Distribution and Utility, as labels are ordinal and misclassification is not just 1 level above or below the original label. If other reviewers suggest acceptance, I would not object to their decision.

We sincerely thank the reviewer for their time, effort, and detailed, insightful comments. We acknowledge that the significant imbalance in the data distribution stems from real-world scenarios. In response to the reviewer's concern, we have expanded our discussion in the limitations section to include the following:

"Lastly, the question relevance and answer readability tasks suffer from a highly imbalanced dataset, which may limit the model’s ability to effectively learn from and evaluate rare instances. However, we chose to retain this task in the benchmark to reflect the real-world data distribution in financial information disclosure."

We believe that the decision to collapse the labels or retain the fine-grained labels should be left to the users, depending on their specific applications. While techniques such as sampling or boosting could help mitigate these issues and improve model performance, it is important to emphasize that model optimization is not the primary focus of our work.

The main contribution of this paper lies in the development of a novel and domain-specific open dataset with practical applications in the financial information disclosure field. This dataset provides a foundation for users to test advanced AI methods and improve performance. Our study establishes a comprehensive baseline, offering a clear and reliable point of comparison for future research.

We truly appreciate the reviewer’s responsiveness, which provides valuable suggestions for further exploration. We believe this dataset serves as a valuable resource for advancing timely and automated financial disclosure analysis, a critical area in the accounting research domain. We encourage users to build upon our dataset and adopt advanced methodologies to address the challenges highlighted.

We appreciate your time and effort in reviewing our work and these insightful comments, which have helped us improve our manuscript. We provide a list of responses below:

(Q=Question from Weaknesses and Details Of Ethics Concerns; RQ=Response to Questions)

The following questions are from Weaknesses:

Q1: The FinTruthQA dataset is exclusively from Chinese markets. In addition, the proposed dataset focuses only on the Q&A format, excluding other types of financial disclosure (like, long reports, and analytic reports).

RQ1: We appreciate your observation regarding the dataset's focus on Q&A data from the Chinese markets. The focus on the Chinese market is intentional and addresses a unique gap in financial research, given the unique and underexplored nature of investor Q&A platforms in this regulatory and cultural context. The emphasis on the Q&A format in FinTruthQA was to address the growing prevalence of investor interactive platforms that utilize this format for real-time communication. Unlike static financial disclosures such as annual reports, Q&A exchanges provide an interactive and dynamic medium, often reflecting investor concerns and corporate responsiveness in near real-time. By focusing on this novel format, FinTruthQA fills a significant gap in existing benchmarks, which primarily evaluate static disclosures like earnings calls or annual reports.

Q2: In question relevance, the label is imbalanced (99.51% positive), and limited number of samples for some categories. This may affect the validity and generalizability of the benchmark results.

RQ2: Thank you for your insightful feedback regarding the imbalanced label distribution and the limited sample size for some categories. While it is true that the dataset exhibits significant class imbalance, this imbalance reflects the actual conditions on investor interactive platforms, where most investor queries on platforms like SSE and SZSE pertain to relevant financial topics. By preserving this distribution, we ensure that the benchmark realistically evaluates the performance of models in real-world financial information disclosure contexts. Artificially modifying this distribution could lead to models optimized for a theoretical scenario, thereby reducing their practical applicability.

While the negative labels are limited, they cover diverse instances of irrelevant questions, ensuring a representative sample of the minority class. We intentionally preserved them in the dataset to enable models to learn to identify rare but critical instances of irrelevant questions.

Q3: Limited overall technical contribution to financial Q&A or general domain Q&A.

RQ3: We appreciate the reviewer’s comment and would like to clarify and highlight the unique contributions of our work.

The primary contribution of our work lies in the creation of the first novel and annotated dataset related to financial information disclosure, providing a valuable resource for the financial NLP field and contributing to improving financial information disclosure practices in real-world financial markets. As we emphasized in our paper, accurate and transparent financial information disclosure is critical in the fields of accounting and finance, ensuring market efficiency and fostering investor confidence. As a result, we collected data from various platforms and systematically annotated it to construct the first benchmark for evaluating the quality of financial information disclosure.

To ensure the quality of the dataset, we established detailed annotation guidelines and implemented strict quality control throughout the dataset construction process. Additionally, we introduced multiple models to assess their performance on this benchmark. The consistent performance across different models demonstrates the reliability of our approach, while the lower performance on certain tasks highlights the challenging nature of the benchmark. This indicates that our benchmark can serve a valuable resource to promote future research in the field.

Beyond direct application in financial and regulatory domains, our benchmark serves as a valuable resource for academic research in natural language processing, particularly in areas concerning the automated assessment of text quality. By establishing a benchmark, we provide a foundation for further research into improving the accuracy and applicability of NLP techniques across diverse fields, potentially leading to innovations that could benefit a wide range of industries where transparent communication is vital.

I have read the rebuttal and I will keep my score.

We appreciate your time and effort in reviewing our work and these insightful comments, which have helped us improve our manuscript. We provide a list of responses below:

(Q=Question from Weaknesses and Details Of Ethics Concerns; RQ=Response to Questions)

The following questions are from Weaknesses:

Q1: Limited Evaluation Settings

RQ1: We appreciate your feedback highlighting the limitations in scope. We have conducted evaluations on tasks such as question identification, question relevance, answer readability, and answer relevance. Tasks like question identification and question relevance achieved sufficient performance with traditional models (>96% accuracy, as shown in Table 2 of the manuscript), making detailed comparisons unnecessary for these tasks. Instead, we focused on more challenging tasks, such as answer readability and answer relevance, where performance differences between traditional ML-based models and PLM-based models were more pronounced.

Due to resource restrictions, we evaluated LLMs only in a zero-shot setting, prioritizing challenging tasks like answer relevance to demonstrate their generalization potential. We have already emphasized in Section 4.1 of the manuscript that PLM and ML models were evaluated with supervised fine-tuning, while LLMs were tested in a zero-shot setting. This background is included to ensure readers clearly understand the methodological context of the comparisons. As we have released the dataset fully open-accessed, users can easily conduct additional experiments on our dataset.

To address potential confusion, we have revised the limitation section:

Given the substantial computational cost of training, we limited LLM evaluations to a zero-shot setting for answer relevance. This decision balances feasibility with the need to showcase LLM capabilities. Future studies will incorporate fine-tuned evaluations of LLMs to provide a more consistent comparative framework.

Q2: Copyright and Data Usage Concerns

RQ2: We appreciate your feedback highlighting the importance of ensuring clarity regarding copyright compliance and data usage. The Q&A data used in this study was obtained from publicly accessible interactive platforms of the Shanghai Stock Exchange (SSE) and Shenzhen Stock Exchange (SZSE), which facilitate public disclosure of communications as part of regulatory requirements. It solely comprises communications between public companies and investors, containing no private or restricted content. As the topics focus on publicly listed companies and the information is required to be openly disclosed, we identified no restrictions on its usage. Our dataset has been developed exclusively for academic purposes, aiming to enhance the efficiency and effectiveness of financial communication through data-driven evaluations of financial information quality.

Q3: Exclusion of Financial LLMs

RQ3: We appreciate your insightful feedback regarding the financial domain-specific LLMs. As we discussed in Section 4.3.4 (under the "Suboptimal Performance of LLMs in Chinese Financial Contexts" section), we explored the use of Chinese financial-specific LLMs, including CFGPT2 and DISC-FinLLM, which represent state-of-the-art domain-specific financial LLMs tailored for Chinese texts. We decided not to incorporate financial-specific LLMs due to the suboptimal performance they demonstrated in our preliminary experiments. Despite their domain specialization, these models failed to generate accurate and reasonable predictions. For example, CFGPT2 frequently generated responses containing extensive redundant phrases (the input question was repeated multiple times), failing to provide substantive information or contextual judgment. Thus, we decided to exclude them from our evaluation.

We have emphasized the following statement in the revised manuscript:

Preliminary experiments with Chinese financial-specific LLMs demonstrated significant challenges in output quality. Outputs often exhibited excessive repetition and encoding errors. For instance, CFGPT2 redundantly repeated phrases such as “公司是否存在应披露而未披露的重大其他风险” ("Does the company have any significant undisclosed risks?") multiple times when the phrase was part of the input question. Similarly, encoding issues resulted in outputs that were partially unreadable, further limiting their practical utility.