Multilingual Contextualization of Large Language Models for Document-Level Machine Translation

We thank the reviewer for the constructive feedback.

Since DocBlocks is entirely constructed from publicly available datasets, the novelty and contribution of the dataset itself may be limited.

We respectfully disagree with the assertion that DocBlocks lacks novelty due to its reliance on publicly available corpora. While the raw documents are indeed obtained from public sources, the novelty lies in our rigorous, large-scale curation and filtering process and the purpose-driven design for long-context, multilingual document-level MT -- a resource that, to our knowledge, does not currently exist in comparable scope or quality.

Specifically, DocBlocks is not a simple aggregation of existing datasets. We systematically clean, deduplicate, and segment documents using tools and metrics such as Bicleaner, langid and CometKiwi, applying strict thresholds to filter out noisy or misaligned pairs—where the source and target texts are incorrectly matched (see Section 2.1). This process significantly improves data quality and coherence at the document level, addressing common issues in existing datasets. As recent work shows [1, 2], data quality now outweighs quantity in LLM training. Careful filtering, rather than broad collection, is essential for building effective models, as cleaner datasets can outperform much larger, noisier ones.

DocBlocks also encompasses a wide range of genres and discourse types—including news articles, TED talks, parliamentary proceedings, and novels—enabling broader domain generalization than most prior document-level MT corpora, which are often limited in scope or language coverage.

Furthermore, we apply structured document context transformation techniques, MRD2D and CAPT, to generate more diverse examples that encompass multiple translation paradigms. We show that these transformations, when used together, play a central role in improving downstream model performance.

Figure 2 presents statistics for the original datasets... but not for the final DocBlocks dataset.

We appreciate the reviewer’s feedback and will include comprehensive statistics such as post-filtering document counts, language pair distributions, and average sentence and document lengths in the final version of the paper. We will also publicly release DocBlocks to facilitate reproducibility and encourage further innovation. Additionally, we emphasize that the impact of our data curation process is quantitatively evaluated in the ablation studies presented in Table 6, which directly link filtering and the use of MRD2D and CAPT techniques to downstream model performance, demonstrating the effectiveness of our curation methodology.

The fine-tuning approach lacks novelty, as it relies on two existing training strategies: MRD2D and CAPT.

We appreciate the reviewer’s feedback. While it is true that we do not introduce a new fine-tuning algorithm (e.g., a novel loss function), we disagree that our approach lacks novelty. Our contribution lies in designing a practical and effective recipe for adapting sentence-level MT LLMs to document-level translation by integrating existing techniques in a novel and impactful way, and in empirically demonstrating the superiority of this approach over other popular strategies, such as agent-based methods.

Specifically:

• We show that instruction-tuned sentence-level LLMs can be effectively transformed into strong document-level models through targeted fine-tuning on DocBlocks, using relatively lightweight training. As noted in Line 246, our document-level models are trained on 4 H100 GPUs for approximately 120 hours.
• We combine MRD2D and CAPT with high-quality, context-rich instructions, leveraging DocBlocks to support multiple document translation paradigms. The effectiveness of this combination and its impact across both Doc2Doc and chunking-based decoding strategies have not been demonstrated in prior work. Furthermore, we show that applying these techniques together results in consistent improvements across all decoding methods (see Table 6), which, to our knowledge, has not been previously explored or reported.
• Our resulting models are competitive with, or outperform, significantly larger systems (e.g., GPT-4o) despite using fewer resources.

These contributions, supported by rigorous evaluation across multiple translation paradigms represent a meaningful step toward scalable, open-source, document-level machine translation and offer a practical framework for the community to adopt and extend.

We sincerely thank the reviewer for the detailed and constructive feedback.

The evaluation is mainly based on sentence-level metrics, or poor document-level metrics […] I would expect to see at least a small human evaluation.

We fully acknowledge that evaluating document-level MT remains an open and complex challenge, particularly due to the lack of universally accepted automatic metrics. To mitigate this limitation, we employed multiple document-level metrics as described in Section 3.1. These provide complementary perspectives on translation quality, particularly targeting coherence, consistency, and terminology preservation.

While we agree that these metrics are not perfect, we would respectfully argue that labeling them as “poor” may overlook their continued utility. For instance, BLEU, despite its known limitations, has served as the de facto standard for evaluating MT for decades, and its document-level variant (d-BLEU) still offers useful signals, especially when triangulated with metrics like COMET and LTCR. We view these as valuable proxies that help quantify improvements, particularly in the absence of comprehensive human evaluation.

Regarding human evaluation, we agree it could significantly enrich the analysis. However, given the multilingual nature of our work, the length of test documents (e.g., TED Talks), and the fine-grained nature of discourse-level quality, a robust and reliable human evaluation would require professional translators across many languages, which was infeasible within our current resource constraints. Moreover, the subjectivity inherent in human assessments, particularly for phenomena such as literary style preservation or discourse coherence, complicates the design of a universal evaluation framework [1,2,3]. We expect future work to overcome these challenges in the upcoming years through more document-level human annotations and agreed evaluation protocols in WMT campaigns, with subsequent use of these data to improve automatic evaluation.

The creation of a new dataset is overstated [...] it's actually a compilation. Clear statistics are missing. [...] Comparison with original datasets is needed.

We appreciate the reviewer’s feedback and agree that the framing of our dataset's contribution can be made clearer. While DocBlocks is indeed a compilation of existing corpora, please note that it is not a simple aggregation. As described in Section 2.1, we applied a rigorous cleaning and curation pipeline, including:

• Filtering using Bicleaner and CometKiwi,
• Document-level alignment and deduplication,
• Language identification filtering.

These steps significantly improve the quality of the data compared to the raw sources—this is key in obtaining high-quality and practically useful datasets.

This is substantiated by the comparison that we actually do with the original, unfiltered datasets in Table 6, where models fine-tuned on the unfiltered DocBlocks data perform much worse than those using the curated version. While we already compare performance using the unfiltered dataset in Table 6, we have not included per-source comparisons, as our aim is to construct a high-quality multilingual dataset rather than optimize for any single corpus.

Regarding dataset statistics, please see Figure 2, which presents counts of documents, sentences, and average lengths across domains. However, we agree that providing additional post-cleaning statistics (e.g., final document counts per language pair) would enhance clarity and completeness. We will incorporate these details into a revised version of the paper and also release the DocBlocks dataset publicly to support further research in multilingual document-level machine translation. We hope this clarifies the concerns.

It is not clearly stated which languages you work on [...] Font in figure 2 is unreadable [...] I would also expect results per language pair [...]

Thank you for the feedback. We would like to clarify that all relevant dataset characteristics, including the full list of language pairs, are already presented in Figure 2. However, we acknowledge that the small font size may hinder readability, which could have contributed to the impression that this information was missing. We will update Figure 2 with larger, more readable fonts in the final version of the paper to ensure clarity. Additionally, we agree that providing results per language pair would offer valuable insight. We will include a detailed breakdown in the appendix of the final version.

We thank the reviewer for the constructive feedback and their positive assessment of our work, particularly the recognition of the value of our DocBlock dataset, the thoroughness of our experimental analysis, and the clarity of our writing.

The primary concern lies in the reliability of the evaluation presented in the paper. In the main results (Table 2), some metrics appear contradictory. For instance, in the last row, Qwen2.5-7B-Instruct performs significantly worse than DocMT-Qwen2.5-7B-Instruct on BLEU, yet it surpasses it on COMET and ties on d-COMET. [...]

We appreciate the reviewer’s careful observation. The discrepancy noted is a well-documented and ongoing challenge in MT evaluation. While BLEU focuses on n-gram matching across the document, COMET captures semantic adequacy, which often leads to different preferences, easily observed in MBR decoding [1]. As mentioned in the paper (Lines 195–197), developing more robust and reliable document-level metrics remains an open problem and important direction for future work.

Currently, there is no single dominant metric for document-level MT. Much like sentence-level MT a few years ago, the field is transitioning from lexical metrics like BLEU toward neural metrics such as COMET, which better correlate with human judgments. In this evolving landscape, we report a diverse set of metrics—d-BLEU, COMET, d-COMET, and LTCR—to provide a balanced and transparent evaluation. While d-BLEU remains widely used and interpretable, COMET-based metrics offer complementary insights into discourse and semantic quality. Occasional contradictions between metrics—such as the one in Table 2—are expected and reflect the inherent complexity of evaluating long-form translation [2]. This is why, instead of relying on a single metric, we opt to report a comprehensive suite of metrics at both the sentence and document levels.

Ultimately, our contribution is not to claim superiority on any single metric, but to demonstrate a general and effective approach for adapting sentence-level MT models into document-level systems. The consistent gains over all base models validate the robustness and versatility of our method, even if sporadically an individual metric might counter the trend.

[...] Incorporating human evaluations would strengthen the reliability and comprehensiveness of the results.

We agree that human evaluation is valuable; however, performing meaningful human evaluation for document-level MT is currently very challenging for two reasons. First, there is no standardized evaluation framework that adequately addresses the ambiguity and subjectivity involved in assessing long-form translations. For example, in IWSLT2017, where each document is a full TED talk, evaluating an entire talk's translation requires considering coherence, style, discourse, and fidelity across dozens of paragraphs, making consistent judgments highly challenging. Second, the cost of such evaluations is quite significant: it would require expert annotators to read and compare entire documents in multiple language pairs, making it impractical at scale. While we recognize its value, we believe that advancing automated document-level metrics (like d-COMET and LTCR) remains the more scalable and actionable direction for now.

The description here is somewhat vague. Could the authors clarify what is meant by ‘continuing fine-tuning on document-level data’?

We appreciate the request for clarification. “Continuing fine-tuning” refers to a second-stage supervised fine-tuning process applied to models that have already been instruction-tuned—either on sentence-level MT data or other general-purpose data. In this stage, we continue the fine-tuning on document-level translation tasks using the curated DocBlocks dataset. This includes instruction-based examples in multiple formats: contextual blocks, full-document pairs, and sentence-level pairs. The goal is to adapt the models to longer contexts while preserving their sentence-level performance.

We will clarify this terminology more explicitly in the revised version.

We thank the reviewer for their insightful feedback and for recognizing the contributions of our work, particularly the introduction of the DocBlocks dataset and the effectiveness of our fine-tuning strategy.

The distinction between results of language pairs is not exploited in the analysis of the results. [...] a family-wise analysis could have offered deeper insights [...]

Thank you for suggesting a family-wise analysis. Our aim in the current version was to highlight the broad multilingual effectiveness and generalizability of our approach across diverse language pairs. We agree, however, that a more fine-grained analysis may offer valuable insights into model performance across specific language families. In the final version, we will include an extended family-wise analysis using our existing evaluation setup with IWSLT2017 and GuoFeng. We believe this addition could help us identify the strengths and weaknesses of our models more precisely.

The absence of the Llama model is a notable omission. It provides support in almost 8 languages and only misses a few language pairs used in this paper.

We appreciate the observation and understand the interest in including a LLaMA model, given its popularity. However, our model selection was primarily driven by the goal of leveraging LLMs with state-of-the-art translation capabilities, rather than starting from general-purpose backbones like LLaMA, which would require extensive adaptation for machine translation. Our experiments include TowerInstruct-Mistral-7B [1], a model explicitly optimized for translation tasks using Mistral-7B as the backbone. Notably, it outperforms both TowerInstruct-7B and TowerInstruct-13B, LLM-based translation systems built on LLaMA. We will clarify these choices in the final version of the paper.

Why was only GPT-4o and Qwen selected as baseline models for translation output? [...]

We selected GPT-4o and Qwen2.5-72B-Instruct as baselines for the following reasons: GPT-4o is currently the most capable publicly accessible commercial LLM for document-level translation tasks [2,3], while Qwen2.5-72B-Instruct is among the top-performing open-source multilingual LLMs. Both models support extended context windows (128K tokens for GPT-4o and 32K tokens for Qwen2.5-72B-Instruct), which are essential for document-level machine translation. This selection enables us to evaluate performance across both open-source and closed-source paradigms.

[...] And is there a lack of discussion on the selection criterion for other models?

We selected Tower, EuroLLM, and Qwen2.5 (7B) as training models based on the following considerations. All three are strong sentence-level MT LLMs and provide a solid foundation for exploring extensions to document-level translation. Tower, built on Mistral, originally supported long contexts but lost its capability due to extensive continued pretraining [4] and instruction tuning in the Tower recipe, which primarily focuses on sentence-level translation tasks. EuroLLM offers broad multilingual coverage but is limited to a 4K token context window, making it a natural candidate for extension, as it was never exposed to long-context tasks. Qwen2.5 supports long-context inputs, but its 7B parameter version currently falls short in delivering high-quality document-level MT. We believe this selection offers a balanced and informative basis for studying how to generalize diverse MT systems to the document level.

We hope that our answer clarifies your main concerns. We are happy to address any other question you may have.

[1] https://arxiv.org/abs/2402.17733

[2] https://arxiv.org/pdf/2412.11732

[3] https://arxiv.org/abs/2502.12404

[4] https://arxiv.org/abs/2503.19206