Long BERT for bankruptcy prediction

Thanks to the reviewer for the feedback.

Here we are addressing the comments:

• The research paper recommended DocBERT, it might lead to confusion if is not read carefully. Besides the name is DocBERT, the authors propose the distillation of BERT into an LSTM model, that means that the final model used for prediction is an LSTM model. As is well known, LSTM models were replaced by BERT mainly because is unable to process in parallel and also not good in very long sequences (Of course are better than the RNN). This was reflected in our results, more than 5 LSTM time sequences, the performance drops. DocBERT uses a dataset that according to the provided dataset table, the maximum average number of words per document is bellow 400. Considering only 1.2 tokens per word, we are dealing with a maximum number of 480 tokens, even less than one BERT segment. Our dataset in average has 2.8K tokens per document and 2.7K words per document (English). This is the reason that a pure LSTM or BERT network does not work for processing long text.
• Also as explained in the paper, we compared with established techniques for bankruptcy prediction (that are mainly numbers) and the text itself is not better than that. Because of this reason our project main goal is to have an integrated model that is having text information, numerical information and other sources of information.
• GPT models like Llama 2 and GPT-3 are broadly used nowadays, but the main problem of these family of models (generative) is the hallucination. We are using also these models to process the text, but the amount of dirty answer that we receive is above 30%. Commercial models like GPT-4 and even the best performed versions of GPT3.5 can be costly for some companies that needs to process millions of documents. For this reason, we are focused on ways to have lighter classification models like BERT-Based models, faster and cheaper to implement. Nevertheless, we are including in the paper the results of using Llama 2 for bankruptcy prediction and the current limitations.
• The references provided are the ones that are focused on BERT, lately most of the research done was following the wave of GPT-based models, but those are not the models that we are pursuing because of the reason explained before. There is not much work related to bankruptcy prediction using long text sequences (or in other domains), and this is why we considered important to share our results and contributions.

Thanks to the reviewer for the feedback.

Here we are addressing the comments:

• We are clarifying that besides the research object of bankruptcy prediction we are also contributing to the architecture for dealing with long text using BERT models that can run even in a laptop with 1 GPU. We propose to use BERT+LSTM architectures for not so big documents (<=5 segments) and use BERT+concatenation for bigger documents (more than 5).
• With respect to the question 1, We were applying these architectures in other models where we need to feed by page (3 segments) and other long documents (8 segments). The accuracy for other classification tasks we had were around 97 % (not related to bankruptcy). We are planning to create another paper to focus on these different use cases that the architecture can allow us.
• With respect to the question 2, the model BERT itself despite the fact of handling multiple languages, those embeddings are not fully related between each other, it can be seemed like parallel models per language. We could easily prove by comparing the distance between the embeddings of one word in different languages. In the other hand the more complexity of the GPT models can handle this problem easier. You can provide a text with different languages and to expect the answer in a specific language. The main problem of this GPT for classification is that there are going to hallucinate and create answer that could be difficult to process unless you apply more layers of validation, and also the resources required are bigger. One of our open research questions is how to address BERT embeddings of different languages to work together.
• With respect to the question 3. Thanks for the contribution, we are starting to include Explainability and ranking the top tokens that can lead us to the bankruptcy prediction.

Thanks to the reviewer for the feedback.

Here we are addressing the comments:

• The test set was determined randomly by document. As the model propose to use many segments for each document, we can not do randomization of the segments by default, we need to do it by document to avoid losing the integrity in the document, because it will allow us to integrate in the next phase in a LSTM or concatenation model. As explained in the paper, the weakness of the paper of Mai et al, as other similar research, is that they are using the risk section of the SEC fillings, where are implicitly mentioning the level of risk of the company and an analysis, in that case they do not need to process the entire document, just the paragraphs that contains that information. In our case the dataset does not contain an explicit section of the risk analysis, and this is why we need to feed the entire annexes in order to look for risk information. We thank the proposition of selecting randomly 512 tokens (510) of each document to use as a base line to compare our model.
• Also, as we mentioned and explain, because of the high unbalanced data, we are just focusing on the precision of risk prediction as the performance measurement, because if we see the other metrics like recall or accuracy, the model is above 90%, that can lead us to bad conclusions. We thank the reviewer that we agree that information of the Figure 4 and Table 3 are the same and should be integrated. As explained in the previous paragraph the conclusion of the figure 4 is that the best performed model for not so long text sequences is to have an LSTM of 80 hidden layers with the proposed model m4 (applying dense layers after BERT and after LSTM). Bigger than that, the model starts to be too complex and to lose performance.
• As mentioned in the paper, the prediction of the bankruptcy is one of the goals, the other objective is to provide with two architectural options (LSTM or concatenation) for dealing with long text prediction. In our research project we are already using these models to do predictions by documents (other labels) and by pages, that requires more than 510 tokens. In consequence, the architecture can be broadly applied to any domain that requires the use of BERT models for long text prediction.
• About the question of shuffling, is because we shuffle the data at document level because in the preprocessing stage, we already have the documents divided into segments. To avoid the corruption of the integrity of the documents itself, we do the shuffling before the pre-processing phase.
• With respect to the number of segments kappa that depends on delta, we correct the equation 1, because as commented, the number of overlapping tokens affects the number of required segments. Is important because it allow the model to keep the previous context in each segment analysis. As suggested by the other reviewed we are going to use random selection of segments to make a base line to compare with.

Thanks for the feedback.

Here we are addressing the comments:

• One of the main goals of this paper is to see how good a bankruptcy prediction can be done with a model using only textual information, and it was mentioned in the abstract and conclusions, this is part of a bigger project that is going to integrate textual information, numerical information and other sources of information. We are going to put more emphasis in this in the paper to make it clearer.
• We have compared the results of our model with other methods like Logistic Regression, Random Forest and LightBM, that are also working with a very similar data. We thank the reviewer for providing the info about the FIM models but, we could not find a specific finance paper about a method / model FIM for bankruptcy prediction. We thank in advance is the paper's name can be provided in order to improve our work.
• To provide a structure of probabilities as a result of the model is a very interesting proposition, we have started working in doing this based on the classification result and three classes according to the degree of confidence, to still be able to use the current labeling.