Thank you for your thoughtful feedback. We appreciate your diligence in reviewing our work.

Weaknesses

The lack of novelty in the model architecture's methodology is a weakness. It suggests that the approach may not bring any new or innovative ideas to the field. It would be better for an industrial track paper.

Please refer to our general response 1.

The reported results being limited to English data is a weakness. It is unclear if the model can support other languages, such as Chinese or Japanese. However, it would be interesting to evaluate the transfer learning ability of the model for layout analysis, which could potentially address this limitation.

Please refer to our general response 2.

Figure 5 and section 5.4 on repetitions during inference being difficult to follow. It indicates that the explanation or presentation of this aspect of the model may not be clear or well-explained, making it challenging for readers to grasp the concept.

Thank you for your feedback. We will rework this section to make it easier to follow.

Questions

What does “out-of-domain documents” refer to on page 8, and what is the objective of conducting experiments on repetition data?

“out-of-domain documents” are in our case academic documents found on the internet including scanned or digital books (for examples, see in Appendix B: Fig B.1 and Fig B.2). We realize this was not clear in the original text and updated it. Thank you for the pointer.

In Eq. X, does “a” or “b” denote a paragraph index?

Yes, in Eq. X, “a” or “b” denote a paragraph index.

Why were Table, Plain text, and Math treated as separate modalities for experimentation, and how does this differ from conventional methods?

In our study, we chose to categorize Tables, Math, and Plain Text as distinct text modalities, only for our evaluation process. This distinction was made to gain more detailed insights into the potentially varying results, acknowledging their inherent differences. For instance, the way Tables might yield different outcomes compared to Plain Text is noteworthy. It might be more apt to use the term "category" rather than "modality". Importantly, this differentiation was applied only in the evaluation phase, not during training. As such, the reference to 'conventional methods' isn't directly applicable here, since our approach didn't alter the training methodology, but rather focused on a more nuanced evaluation strategy.

Can the model be directly applied to visually rich document understanding tasks?

Please refer to our general response 3.

Would it be better to move the first paragraph in Section 3 Model from page 2 to the related work section?

Yes, that would be better, thank you for the suggestion!

Thank you for your thoughtful feedback. We appreciate your diligence in reviewing our work, however we disagree regarding the lack of novelty, please see our general response 1.

Thank you for the thorough review. We agree with your assessments and added more context regarding GROBID in the paper. In Section 5.3, we provide a clarification on the utilization of LaTeX OCR. That might not have been apparent solely from the table, so we included a reference there as well.

Are the scores presented in % for the editdistance metric? This metric is referred to as CER (Character Error Rate) in the document understanding community.

Thank you for the pointer. The scores are not presented in %. We will rename the metric in the paper.

Fig. 2 shows that LaTeX sources are recompiled

Recompiling LaTeX sources using SyncTeX could indeed streamline our data generation process, but it comes at the cost of significantly increased computational requirements. To mitigate this, we have opted for a more general approach by utilizing the original PDF files supplied by the authors. This choice allows us to accommodate various data sources, including cases such as PMC documents where no LaTeX source files are provided. Fig. 2 was only supposed to show the relationship between the tex and pdf files, not to imply that we are recompiling the LaTeX sources for each paper. The image caption does clarify the relationship: “d) the PDF file provided by the authors”

The label noise strategy is not clear

Thank you for the feedback. The strategy is as follows: With a chance of 10% we change one token in the ground truth text to another randomly selected token. The likelihood of changing two tokens simultaneously is 1%, calculated as (10%)^2. Similarly, the probability diminishes exponentially with each additional token substitution, such that the chance of changing three tokens is 0.1%, and so forth.

Text vs equations vs tables all have different semantics. Could you comment on which metrics are the most relevant for each text category?

In my opinion, the CER is the most important metric for all of the modalities, since we pre-process all equations and tables to reduce ambiguity. However, using a symbolic graph could help give a clearer signal. Thank you for the pointer, we will investigate it in the future.

The limitation of 4096 tokens seems low for a full page

On average a page has 1400 tokens. The 4k context size is more than enough for most of the pages. The only problem we noticed are with large tables, because they are token intensive since the GALACTICA tokenizer was not trained on LaTeX tables. We will leave long context exploration to future work.

post-processing strategy used to normalize section/subsection formatting

No, we don’t have a post-processing strategy for section titles.

Adaptability to other languages

Please refer to our general response 2.

What if repetitions appear inside the page and not at the end? (ex: a sentence is repeated twice, but after that the model continues reading until the end of the page). In this case your method will not work as it will just stop the generation. 1) does this case happen? 2) how would you deal with this issue?

If a sentence is repeated in the document, the model is able to do that as well and continue with the rest of the text. The repetition detection during inference will not stop the generation, because the logits variance does not drop because of it.

Do you handle references to other parts of the page All references (to e.g. citations, formulas, tables, figures) are replaced with the corresponding label in the LatexML pre-processing step. So you would need to pair them up again during post-processing. Do you handle references to the original image/document (e.g. select a word in the Markdown to highlight it in the PDF)? This feature could be used by users to check an equation if it does not make any sense, for example. To go beyond recognition, how could Nougat be adapted to a multitask setting with e.g. Visual Question Answering / Neural Machine Translation / Summarization?

This is a good idea, but we did not look into it yet. One could possibly use the attention maps of the vision transformer to highlight the most important image sections for a given output token. We agree, adapting Nougat to Visual Question Answering would be an interesting direction. Please refer to our general response 3.

Thank you for your thoughtful feedback.

In our OCR problem domain, the task is to predict the next token, and there is only one correct token to be predicted at each step. Given this nature of the task, we opted for greedy decoding as it ensures determinism in our predictions. Nucleus sampling introduces a level of randomness that is unnecessary for our specific use case, as there is a single correct token at each decoding step.

During the course of our experimentation, we did explore other sampling techniques such as top-k and top-p. However, we observed that these methods were also prone to introducing repetitions in the generated sequences. In the next version of our paper, we will report those additional observations.