Chronicling Germany: An Annotated Historical Newspaper Dataset

We address questions and feedback below:

1. The authors claim that modern human readers will struggle reading the contents of Chronicling Germany however, only provided examples of font differences. As layout analysis is also the primary focus of this dataset, it will be nice if the authors could also focus more on the difficulties for modern readers to understand the layout differences.

Thanks for the pointer! The corresponding passage in our submission was indeed unclear on this point, we have updated it now. The issue is twofold: While Humans struggle with the text font, but don't have a problem with layout, computers struggle with font and layout. Newspapers set in Fraktur font are particularly hard for layout recognition, since their visual appearance differs from that of antiqua texts due to its higher rate of black pixels. This combination makes digitization complex.

2. The authors did not provide information regarding the image sizes or DPI. This could help researchers evaluate the usefulness of the dataset.

Historical Newspapers use various page formats, differing regionally as well as over time. The final image resolution depends both on the size of the original (analogly set) pages and the digitization process. Newspapers were digitzed by a variety of libraries, archives and external service providers using different types of scanners. Many newspapers were first scanned to microform in the 80s and 90s and later on digitized from these films. The web portals providing digitized newspapers disclose neither the original page size nor information on their digitization methods, making it impossible to calculate DPI.

We thank UUca for the suggestion. Overall, 83,8% of our dataset scans fall into a range between 4500 and 5499 width times 6500 and 7499 height pixels. The largest and smallest widths are 5375 and 1800, for the height we have a maximum of 7230 and a minimum of 2510. We have added this information to the introduction section of our paper.

3. The benchmark methods doesn’t seem very well justified. Have you explored how existing pipelines trained on modern newspaper perform on your dataset?

Font differences do not allow us to transfer results for models trained on modern newspapers. The same differences also complicate the weight transfer of weights trained on historic American data, i.e. from Dell et al. (2024). Most American datasets have no baseline annotations, which datasets of European origin typically do. Our annotations follow European standards for compatibility reasons. To the extent possible, we evaluated Dell for the layout and Kraken and Pero for the OCR.

4. There is no information about the legibility of the data

Yes, legibility is hard to measure. We could have asked our annotators to rate legibility, but this would have reduced the size of our data set since our resources are finite.

5. As the authors mentioned that comprehending the pages is not trivial for modern German readers, why is the reading order within scope of this dataset?

While the average modern German reader will struggle to read the fraktur font, this is not the case for our human annotators, who are domain experts. Having a (mostly) sensible reading order provided with the automated OCR facilitates the correction by our human annotators. Reading order could also be annotated manually. Currently, this is not a priority.

6. Follow up: how was the reading order automatically assigned and evaluated to have “satisfactory results”?

Reading Order includes the order of lines within a region, as well as the order of all text regions within a single page. (Reading order in general and with regard to multiple pages is a future research topic.) Besides sorting lines vertically within one text region, we use an algorithm that estimates the number of columns within a page. This is based on bounding box information of all regions within that page. All elements within one column will be sorted vertically, while different columns are processed from left to right. Furthermore, we consider large separators, that divide a page into different sections, which are completely independent of one another.

We have done our best to address your concerns below:

1. cite significant existing projects

We thank CCLu for the suggestion. The paper already lists the NZZ blackletter dataset from the impresso team. Unfortunately, we cited the Mannheim University library, which published an OCR-D ground truth update. To rectify the situation, we are now additionally citing [6], where the dataset appeared originally.

Thank you for bringing News Eye to our attention. We have created a new table, in the paper and below:

[1] http://dataset.primaresearch.org/www/assets/papers/ICDAR2015_Clausner_ENPDataset.pdf

[2] https://github.com/UB-Mannheim/reichsanzeiger-gt/

[3] https://zenodo.org/records/3333627

[4] https://github.com/UB-Mannheim/NZZ-black-letter-ground-truth

[5] https://zenodo.org/records/4943582

[6] Ströbel, Phillip and Clematide, Simon Improving OCR of Black Letter in Historical Newspapers: The Unreasonable Effectiveness of HTR Models on Low-Resolution Images

[7] NewsEye / READ AS Finnish Newspapers https://zenodo.org/records/5654858

[8] NewsEye / READ AS French Newspapers https://zenodo.org/records/5654841

[9] NewsEye / READ AS Austrian Newspapers https://zenodo.org/records/5654907

2. OCR-D in main text

Thank you for the pointer: We already mention the OCR-D guidelines in the text at the end of section three (lines 297-299); a citation is included. Because of the page limit, we are forced to keep the annotation guidelines in the appendix.

3. leverage high-performance OCR systems such as Pero-OCR

We respectfully disagree. Please take a look at section 4.3; we do compare to Pero. Table 6 lists transformer results using a fine-tuned Transformer as proposed by Kodym and Hradis ( https://arxiv.org/abs/2102.11838 ). The two are the original authors of the Pero-OCR system.

4. The paper’s reliance on an OCR system that requires baseline detection is questionable

Our dataset actually includes curved lines due to their low-quality paper, age and often indirect digitization. Following Kodym & Hradis (2021) [Pero] we apply a CNN-based text baseline detection system by adding line height and text block boundary predictions to the model output, allowing the system to extract more comprehensive layout information. This serves to create bounding boxes around each line, so that OCR can be performed on each line individually. For the transfomer model that follows Kodym & Hradis (2021), the baselines themselves are not processed further, they are merely used to establish boxes around text lines. However, our model based on Kiessling (2022) [Kraken] uses baselines. This approach yields better results.

We are currently exploring the pylaia model and will report back if we have satisfying results.

5. The use of a relatively older UNet model for layout detection is debatable

We have chosen the Unet model, because the segmentation approach captures our problem well and we do not need the higher flexibility of a detection model. Nevertheless, we evaluated the YOLOv8 (same as Dell) on our dataset and got the results below. In some of the classes the YOLOv8 model performs better than our current UNet. But as the UNet performs best on the paragraph class, which is for us the most important, we still believe the UNet is the better suited approach. We list F1-scores:

6. The full evaluation of the pipeline is limited

This paper is a prerequsite for article separation in the Fraktur letter domain. Article separation is another topic that is not within the scope of this work. Generally this paper is about a new dataset, it does not aim to make an algorithmic contribution. Consequently, we aim to demonstrate that our dataset allows automatic text extraction on unseen samples using Character Error Rate. Beyond Character Error Rate Table 9 now reports numbers for the entire pipeline.

7. The legend for Table 3 lacks clarity

Thank you for the suggestion, we have reworked table 4 (formerly table 3) taking your feedback into account.

Dear CCLu, thank you for suggesting Pylaia. We successfully trained the model on our Chronicling Germany Dataset. We report our results below:

It turns out that PyLaia performs slightly better than the LSTM and the Pero-Inspired transformer. Thank you for the suggestion. We will update the paper accordingly. This experiment strengthens the paper since it shows that our dataset allows multiple pipelines to converge. In all cases, convergence produces generalizing networks.

References

[1] Are Multidimensional Recurrent Layers Really Necessary for Handwritten Text Recognition? from Joan Puigcerver, published in the 14th IAPR International Conference on Document Analysis and Recognition (ICDAR 2017). (https://atr.pages.teklia.com/pylaia/usage/training/)

We address questions and feedback below:

1. Novelty and contribution.

In addition to stating sizes in the text, we have now added a table illustrating the size of the data set in comparison to related work. You are reviewing the largest dataset of historic newspaper pages set in fracture letters. For your convenience, we have also listed the table below.

[1] http://dataset.primaresearch.org/www/assets/papers/ICDAR2015_Clausner_ENPDataset.pdf

[2] https://github.com/UB-Mannheim/reichsanzeiger-gt/

[3] https://zenodo.org/records/3333627

[4] https://github.com/UB-Mannheim/NZZ-black-letter-ground-truth

[5] https://zenodo.org/records/4943582

[6] Ströbel, Phillip and Clematide, Simon Improving OCR of Black Letter in Historical Newspapers: The Unreasonable Effectiveness of HTR Models on Low-Resolution Images, https://www.zora.uzh.ch/id/eprint/177164/1/ Improving_OCR_of_Black_Letter_in_Historical_Newspapers_The_Unreasonable_Effecti.pdf

[7] NewsEye / READ AS training dataset from Finnish Newspapers (19th C.) https://zenodo.org/records/5654858

[8] NewsEye / READ AS training dataset from French Newspapers (19th, early 20th C.) https://zenodo.org/records/5654841

[9] NewsEye / READ AS training dataset from Austrian Newspapers (19th, early 20th C.) https://zenodo.org/records/5654907

2. Lack of quantitative comparison with existing datasets, which fails to show the superiority of the Chronicling Germany dataset.

In addition to the discussion on pages 2 and 3, we have added this information in tabular form (See Table 2 in the updated paper and above).

3. Lack of detailed assessment of the annotation quality, such as the review of annotation consistency.

Thank you for this question. We are an academic project with limited resources. Assessing annotation quality is expensive. To make more valuable annotations available, we opted to spend our domain experts' time creating and checking as many annotations as possible. To ensure high quality, we defined and communicated detailed guidelines as outlined in A.6 of the paper.

4. Section 5 does not provide detailed results of testing on OOD data, which is insufficient to reflect the generalizability of the pipeline.

Thank you for your input. We now provide out-of-distribution (ood) performance measurements in supplementary table nine.

5. The paper incorrectly used the ICLR 2024 template instead of the ICLR 2025 one

Thank you for pointing out this mistake. We have rectified the issue in the current version.

Questions

1. In comparison to other German historical newspaper datasets, how does the quantitative aspects of your dataset fare in terms of size, annotation quality, and diversity?

See point 1.

2. Have the authors explored employing more advanced baseline methods for your tasks?

We assume 'more advanced' refers to transformer-style architectures or a YOLO model. Table 6 lists results for a Transformer on the optical character recognition task. We added Table 8, which lists performance measurements of a YOLOv8 model trained on this dataset.

We address questions and feedback below:

1. The dataset is addressed to a narrow audience and not in the mainstream of the representation learning community nor the scope of the conference.

The ICLR 2025 call for papers includes datasets and benchmarks in the list of relevant topics. In this regard, we contribute a dataset for the study of a low-resource computer vision problem. We address a pressing need of history scholars. The mass digitization of historical newspapers allows historians and social scientists to address new questions and analyze old questions from a new perspective with big data. However, this potential has not yet been realized. Key reasons for current shortcomings are insufficient layout recognition and OCR (which is often correlated) of historical newspapers. Current layout detection pipelines frequently miss article-level or even column separators. This prevents precise information retrieval because combined search terms are applied to the page level and not the article level, thus generating hits for unrelated search terms (false positives). However, especially when combining large databases, precision is essential; for example, many advanced Natural Language Processing methods need article-level precision of OCR data. For a more detailed discussion of why machine learning is crucial for the study of history, please see Appendix A.4 of the paper.

2. Authors should identify other points that make this dataset interesting for a larger audience

Following (Zhang et al., 2024), we argue that low-resource problems should receive more attention in computer vision because studying such tasks has led to significant progress in natural language processing. See pages one and two for the full argument.

3. crossfold validation with dell American stories

This paper makes a datasets and benchmarks contribution. Consequently, its main focus is to establish the validity of the presented data using standard algorithms. We do not establish a novel pipeline. Crossfold validation with the American stories dataset would currently not be feasible since we employ baseline information in our pipeline, which are not annotated in the American stories dataset.

4. It would be good to have different datasets and different methods for the comparison.

In addition to discussing dataset sizes in section one, we have added a new table (Table 2) that lists our and related datasets. We compare different methods in Tables four, six and seven of the updated paper.

Dear reviewers, Thank you very much for your constructive feedback. If there is anything else we can do to address your concerns, please let us know. Your feedback has helped us improve this paper. If you feel the same way, please consider raising your score.