HHD-Ethiopic: A Historical Handwritten Dataset for Ethiopic OCR with Baseline Models and Human-level Performance

Thank you for your comments.

[This paper is a dataset paper, without new technical methods]: Yes, the paper is a datasets paper and our goal is to create a historical handwritten dataset for a low-resourced script and provide benchmark results with SOTA OCR models. In addition we compare the human-machine performance based on evaluation results from several human annotators and the baseline models and we believe this contribution is a good fit for “dataset and benchmark" themes (https://iclr.cc/Conferences/2024/CallForPapers) of ICLR-2024.

[The academic challenges of this dataset are not representative]: The experimental evaluation results of this dataset, involving several SOTA OCR models and human performance, indicate limited recognition performance. Therefore, this dataset presents challenges and can be valuable for further investigation of new methods to address such challenges. If you have a particular aspect you are curious about, let us know.

[The impact of this new benchmark seems limited]: To the best of our knowledge, this is the first sizable historical handwritten dataset of the Ethiopic script, characterized by a unique syllabic writing system, low-resource availability, and complex orthographic diacritics. We believe that the human-level performance and results of the SOTA OCR models reported in this work can be utilized as benchmarks for various state-of-the-art OCR models.

Thank you for reviewing our work. Your input has helped us improve the paper. We've addressed your questions and comments below.

[full page are not provided to evaluate line detection+HTR]: line detection + recognition is one way of OCRing. in literature, the cascaded detection and recognition task is most common in scene text recognition. Our focus was to create segmented text-line images for the recognition task and the annotation was done for lines images. However, we will keep updating the dataset for detection and recognition tasks in the future.

[include reference paper done for Ethiopic]: we included the paper in our reference list.

[no reference or comparison with the already available Ethiopic_Eritrea HTR model in Transkribus] This model is released in the form of webApp, and cannot be retrained. Furthermore, the authors do not provide an API to easily upload samples and download results. Therefore, we limited ourselves to evaluating its performance manually by uploading a few with a samples manuscript image from our test dataset and comparing the results with those of our model. The Ethiopic_Eritrea HTR model from the Transkribus model archives a CER of~65.2% while our Hopt-attn-CTC model archives 17.82 for this sample specific test manuscript. Link for demo:https://github.com/ethopic/hhd-ethiopic-I/blob/main/Hopt_attn_CTC_vs_Transkribu.ipynb

[the models tested have never been evaluated on standard handwriting recognition bases. How do they compare with standard libraries such as Transkribus, pylaia and trOCR?] We have assessed eight methods, including SOTA OCR techniques validated on Latin, Chinese, and Amharic scripts as documented in the literature, under the same computational constraints and resources. Our focus is on models that are comparatively more lightweight than those requiring extensive computing resources or subscriptions. Additionally, we have incorporated the evaluation results of TrOCR (refer to Table 3 in the revised version). TrOCR, being a model with a large number of parameters, is fine-tuned for fewer epochs. This approach is not chosen because it requires more time for training and inference, leading us to prioritize smaller, more efficient networks. Similarly, the Ethiopic_Eritrea Handwritten Text Recognition (HTR) model in Transkribus, which you recommended and we have evaluated, is developed using PyLaia. It's important to note that models in Transkribus are not free and require a subscription for extensive HTR tasks. They also require a stable internet connection. Furthermore, the need to upload manuscripts for the Transkribus models raises ethical concerns.

[Tesseract model], Tesseract is an OCR framework designed for hundreds of scripts including Ethiopic. We evaluated the model proposed for Amharic on a sample test set. The results being so poor, this model cannot be considered state of the art. For this reason we did not include it as part of our baselines.

[inter-annotator agreement of annotators] As discussed in section 3.1 and Appendix B, each text-line image in the training set is annotated individually. In contrast, the test sets undergo expert review, and we evaluate the ground-truth text differences before and after the review in CER (see section 4.1.)

[details about the manuscripts] As discussed in appendix B, the text line images are extracted from manuscripts between the 18th and 20th century; collected from ENALA covering 7 different books of cultural and religious contents (~ 1300 pages), while the remaining is from the internet.

We appreciate your valuable feedback. Thank you.

[Correction on structure of table 2 and add a row for average performance]: we have corrected it (the middle horizontal line is moved four steps down, and inserted one additional row for average performance of each group. Furthermore, we have updated the corresponding values of CER and NED in the text.

[Mismatch form section 4.1, the Figure 5 and the conclusion inferred from it are also wrong]: After fixing the table structure in Section 4.2 (Table 2), the remaining inferences drawn from this table are now accurate.

[** According to the previous two reasons, I think the experiments of the paper are insufficient**]: Although we have previously shared the source code, trained models, and dataset for reproducibility, an “\hline” mistake in the LaTeX table code caused confusion in a section of the paper derived from this table. We have now corrected this error and the experiments are corrected.

[New approach for historical handwritten Ethiopic script] as mentioned in the introduction section, there is lack of historical handwritten dataset for Ethiopic, the main goal of this paper is to create the dataset and evaluate it with SOTA OCR models. In addition, we evaluate human-level performance as a benchmark.

[Why not include accuracy as a metric?] We are using standard metrics (e.g e.g https://arxiv.org/pdf/1909.07741.pdf) for sequence to sequence comparisons, where 3 types of errors occur at the character level: insertions, substitutions, and deletions. The 1-CER (CER=Character Error Rate) is as close as it gets to an accuracy (taking into account the 3 types of possible errors). At the sentence level, the accuracy is usually zero because there are usually a few character-level mistakes. Of course, if a language model or lexicon would be integrated in our system to mitigate some of the mistakes, the sentence-level error would go down, but this is no in the scope of this research.

Thank you for taking the time and effort to reviewing our paper. We address your concerns and questions as follows.

[Content rearrangement] we reorganized contents in appendix A, B moving a portion to section 1 and 2 of the main paper (updates are highlighted )

[Evaluate models proposed in baseline papers on other Ethiopic datasets] we evaluate our HHD-ethiopic dataset using models proposed for Ethiopic script by Dikubab et al. (2022) and Belay et al. (2019a) and present the result in Table 3. However, neither the source code nor pretrained models are provided by Abdurahman et al. (2021) and it is difficult to reproduce the results and use the model to evaluate our dataset. In addition, the dataset of Abduraman is a word level dataset extracted from just 1574 lines of modern Amharic handwritten texts, which follow different writing structures (Example in modern ethiopic writing space is a separator while both space and “two dots” are used as a separator in historical scripts–see Figure 11 in appendix section). To add more, we evaluated our dataset using a similar CNN-RNN-CTC architecture proposed by Abdurahman et al. (2021), although we haven't accessed the implementation details provided by Abdurahman (See Table 3 in the main paper).

[Question on human performance and details of the annotation]. We provided the details of the annotation process (in section 3.1 and appendix B, phase I&II) and human-level performance evaluation (in section 3.2 and appendix B, phase III). To make it more clear, below is the answer for each question:

How would the human performance be with access to reference material? Those individuals participated in the human-level performance have no access for references. The only individuals having access to the reference are people participating in Ground-truth annotation and their supervisors ( see section 3.1 and appendix B, phase I).

Do the annotators make the same errors or different ones? Annotators make different errors in the case of tests, as each text line is annotated by more than one person. However, in the training set, due to the cost associated with annotating each line by many annotators, each text-line image was annotated by an individual.

Do the model make the same mistakes as the human evaluation? Based on our observations during the experiment and the sample predictions shown in Figure 7, the mistakes differ. Both models and humans made substitution errors, but the model tends to have a higher number of insertions and deletions.

For the training set labeling, is there a measure of, for example, inter-annotator agreement? For the training set, since each text-line is annotated by an individual annotator we haven’t measured inter-annotator agreement for the training set. However we measure the agreement between the supervisors and the expert for the test set (see the revised version of the main paper, section 4.1 highlighted in yellow).

[The paper is interesting and the proposal of a new dataset valuable, but it could be more suited to other venues like DAS, ICFHR, ICDAR or maybe ICML], Thanks for the suggestion. We found that the "datasets and benchmarks" theme at ICLR 2024 aligns well with our work, and that's why we're interested in submitting to this venue.

[Why are CRNN, ASTER, SVTR, etc. models worse than the CTC ones despite being larger? In appendix C we learn that they were not trained to the end due to the lack of resources, but that makes the result confusing, if not misleading if they are understood as baselines]. As discussed in Appendix C.2, we conducted runs for all models with a limited wall time, less than 24 hours, regardless of their complexity. The initial CTC-based models, with smaller parameters, underwent more epochs, whereas models with larger parameters could not. Furthermore, we implemented the CTC-based model using optimized hyperparameter values through Bayesian optimization (refer to section 3.2 of the main paper), resulting in improved outcomes.

[In the first test set, did you make sure that the same writer or document does not appear in the training and test set?]. Text-line image samples in the IID test sets are randomly selected from the distribution using sklearn train-test split. For OOD, only 18th-century documents are chosen without mixing the same documents.

[Minor remarks] We corrected and highlighted the changes.