DelTA: An Online Document-Level Translation Agent Based on Multi-Level Memory

Q1. Claims about undertranslation issues and translation quality decline (L177-178) should be supported with citations to relevant work.

Thanks for your reminder. We have added two relative citations here in our paper.

Q2. Comparison with long-context LLMs and commercial document translation tools.

We compare DelTA with Gemini-1.5-Flash, Claude-3.5-Haiku, and the translation tool of DeepL on IWSLT2017. The results are shown in the following tables:

We observe that DelTA outperforms Gemini, Claude, and DeepL in the consistency metric, highlighting its strength in maintaining translation consistency. However, DelTA lags slightly behind Gemini and DeepL in the quality metric, which we attribute to inherent limitations in bilingual capability. It is worth noting that the long-context capabilities of these models come with significantly higher GPU memory consumption and increased inference latency, which lead to higher deployment costs. Our work aims to enhance the long-document translation performance of models with smaller context windows and lower deployment costs, such as the open-sourced models.

Q3. Will the code be released?

The codes and data are released in the ANONYMOUS repository: https://anonymous.4open.science/r/DelTA_Agent-7716. We have added this link in our abstract. Thanks for your feedback!

Q4. More detailed efficiency analysis.

We apologize for any confusion caused. When we state in the paper that our system is memory-efficient, we mean that DelTA consumes significantly less overall GPU memory compared to the Doc2Doc method. The Doc2Doc approach utilizes previous texts and their translations as chat histories to leverage the LLM’s context capabilities. However, as the document length increases, this method leads to a rapid escalation in memory usage. In contrast, DelTA’s memory consumption grows at a much slower rate, primarily due to the summary extension mechanism. This makes DelTA a more practical and scalable framework compared to the Doc2Doc approach.

Q5. How does DelTA handle context-dependent translations?

We conduct a comparison between the Sentence method and DelTA on the mentioned context-dependent translation test set called CTXPRO. Due to time constraints, we only take the first 1000 instances of the “mini.gender.opensubtitles” dataset as the test set and test our systems in the English-German direction. The results are shown in the following table:

We can observe that DelTA achieves higher generative accuracy compared to the sentence-level method, indicating the strong performance of our framework on the context-dependent translation task.

We have added this experiment to Section 6 of our paper. Thanks for your suggestion!

Q6. Statistical significance tests.

The p-values of t-tests between DelTA and Sentence/Context methods are demonstrated in the following table:

Note that our framework achieves more consistency improvements on language pairs that do not share common alphabets, because proper nouns just need to be copied into the target sentences during translation for other language pairs. The p-values of LTCR-1 in En <=> Zh directions are less than 0.05.

We have added these results in Section 5.2 of our paper. Thanks for your feedback!

Thanks for adjusting your scores.

We compare the wall time taken by DelTA with that of the baseline approach that retains the translation of each sentence as a historical chat context. The evaluation employs Qwen2-7B-Instruct and Qwen2-72B-Instruct as the backbone models. A document consisting of 100 sentences, sampled from the IWSLT2017 dataset, is translated from English to Chinese to measure both the total wall time and the average wall time per sentence. We use 1x NVIDIA A800 80G GPU for Qwen2-7B-Instruct, and 2x NVIDIA A800 80G GPUs for Qwen2-72B-Instruct. The results are shown in the following table:

As observed, when the 7B model is used as the backbone, where GPU memory usage is relatively low, DelTA incurs a longer inference time per sentence compared to the baseline method. In contrast, when the 72B model is employed, the wall time of the baseline method exceeds that of DelTA, mainly due to the exhaustion of memory and the over-length of context. These results further demonstrate that our framework offers greater ease of deployment compared to the baseline, particularly when utilizing large-scale models.

Furthermore, as part of our future work, we plan to explore case-by-case adjustments to the inference trajectory, dynamically determining whether specific components should be invoked based on the characteristics of individual sentences. Additionally, we aim to investigate chunk-by-chunk document translation to enhance the parallelism of the inference process. We believe these strategies will further improve the efficiency of our framework.

Q7. Why does consistency drop significantly for the En-Xx 11-20 sentence distance bucket?

We believe this phenomenon is attributed to the inherent distribution of the data, as all systems show a drop in consistency rates within this bucket, while such behavior is not observed in other translation directions.

Q8. Why use IWSLT2017 and Guofeng instead of standard WMT test sets?

There is a misunderstanding. The Guofeng dataset is indeed the official data set of the WMT23 literary translation task which is mainly composed of web novels, and we conduct our experiments on the Test 2 dataset (see https://www2.statmt.org/wmt23/literary-translation-task.html for more details). IWSLT2017 is another widely used document-level translation dataset, characterizing long TED speeches. We believe that these datasets with long documents are challenging benchmarks for document-level translation, which our DelTA framework is still capable of.

Q4. The contribution of each auxiliary memory module is unclear.

We apologize for the inadequacy of the ablation experiments due to space limitations. We report the ablation results for each module individually in the table below. The backbone model is GPT-3.5-Turbo, and the test set is IWSLT 2017 En-Zh (the same as the settings in the paper).

We can observe that

1. Short-Term Memory enhances translation quality but negatively impacts consistency due to disruptions caused by the short-span context it introduces (Row "Short-Term Memory (Context)").
2. Similarly, Long-Term Memory improves translation quality but also leads to a decline in consistency (Row "Short-Term Memory"). However, these two modules complement each other, as their integration leverages information from different spans, resulting in further quality enhancements while mitigating consistency issues (Row "Long- & Short-Term Memory").
3. Proper Noun Records contribute to both translation quality and consistency, as improved accuracy in proper noun translation directly enhances overall quality (Row "Proper Noun Record").
4. Bilingual Summary also positively impacts translation quality by introducing the document's main idea, which guides the generation of target sentences more effectively (Row "Summary").

Each module can bring its own performance improvement, but no single module can outperform the combination of them all. We believe this is because the architecture of DelTA can introduce auxiliary information from different granularities and scales to better improve translation quality and consistency, which is the synergy brought by our framework.

We have added the more detailed ablation study results in Appendix E in our paper. Thanks for your opinion!

Q5. Why isn't the summary for chunk m conditioned on the summary for chunk m-1?

In our implementation, we generate a summary for each chunk and iteratively merge it with the overall summary of the preceding text. This approach effectively conditions the new summary on its predecessor. However, we still conduct an additional experiment where the summary for chunk m is generated based on the summary of chunk m−1. Subsequently, this chunk-level summary is merged into the overall summary. The results of this experiment are presented in the following table:

This approach brings no significant improvements, but we are considering generating the new overall summary directly on the condition of the old one in our future works, omitting the generation of chunk summaries. We appreciate your insightful suggestion.

Q6. The effect of adjusting the numbers of sentences to use as short- and long-term memory. The model would probably need more context at later stages?

We conduct ablation studies on the window sizes of Short-Term and Long-Term Memory. The results are demonstrated in the following tables:

Expanding the Short-Term Memory window results in a slight improvement in consistency but significantly increases computational costs. This is because the translation prompt for each sentence is extended by two additional sentence pairs. Considering the trade-off between computational cost, translation quality, and consistency, we opted not to further increase the window size.

Further increasing the Long-Term Memory window offers no significant benefits. Inspired by your suggestion that longer context windows might be more beneficial in the later stages of document translation, we experimented with a dynamic window size that grows by one sentence pair every eight sentences translated. However, this approach also failed to deliver performance improvements. We believe that the key information from more distant contexts is effectively captured by the Summary module. The iterative process of generating and merging summaries can itself be viewed as an alternative mechanism for expanding the context window.

This part of the experiments has also been added to Appendix E in our paper.

Q1. The Proper Noun Consistency metric is not sufficient to show the improvement of DelTA in terms of consistency compared to baselines.

Proper noun translation consistency (also referred to as lexical or terminology translation consistency) is a well-recognized challenge in document-level translation[1,2]. For language pairs that do not share the same alphabet set, translating proper nouns across a document cannot simply rely on copying strings. The primary difficulty lies in maintaining a consistent translation for a proper noun throughout the entire document, which poses a significant challenge for LLMs with limited context capabilities. Moreover, as detailed in Section 6, we also conducted evaluations on pronoun and context-dependent translations, tasks that require more implicit contextual understanding to achieve accurate results. We believe these findings effectively demonstrate DelTA's improved ability to model discourse structures in documents during translation. In our future work, we plan to expand our study to more advanced consistency tasks in document-level translation, such as logical consistency. We appreciate your valuable feedback.

[1] Lyu X, Li J, Gong Z, et al. Encouraging lexical translation consistency for document-level neural machine translation[C]//Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing. 2021: 3265-3277.

[2] Karpinska M, Iyyer M. Large language models effectively leverage document-level context for literary translation, but critical errors persist[J]. arXiv preprint arXiv:2304.03245, 2023.

Q2. Significance tests are missing.

The p-values of t-tests between DelTA and Sentence/Context methods are demonstrated in the following table:

Note that our framework achieves more consistency improvements on language pairs that do not share common alphabets, because proper nouns just need to be copied into the target sentences during translation for other language pairs. The p-values of LTCR-1 in En <=> Zh directions are less than 0.05.

We have included these results in Section 5.2 of our paper. Thanks for your valuable feedback!

Q3. The performance gap between DelTA and context is small. If the Context model were enhanced with the Proper None Record module, would its performance on the proper noun consistency metric be improved?

We conduct an extra comparative test between DelTA and the context method with Proper Noun Records attached. The results are shown in the Lines “DelTA” and “Short+Record” in the table in Q4. We can observe that there are fairly large performance gap between DelTA and the short+record method. This indicates that the different modules in the DelTA framework synergistically improve translation quality and consistency by extracting key information at multiple granularities and scales.

Q1. The involved automatic metrics fail to capture discourse-related linguistic dimensions.

It is widely acknowledged that evaluating document-level translation using a unified automated metric poses significant challenges. Meanwhile, as outlined in Section 6, we conducted additional evaluations focusing on pronoun and context-dependent translation accuracy, tasks that require a deeper understanding of implicit contextual information to produce coherent results. We hope these findings highlight DelTA's enhanced capability to model discourse structures in documents to some extent. Due to time constraints, we will consider adding human preference evaluations in our paper's revised version. Thanks for your insightful opinion!

Q2. Consider using the WMT24 shared task benchmark test sets.

Thank you for your valuable advice! We have already evaluated our system on Guofeng, the official test set for literary documents in WMT23, to demonstrate our progress. We look forward to further testing DelTA on additional language pairs and enhancing our framework based on these test sets as they are released.

Q1. As authors point out in limitations, the efficiency of the method is relatively low.

To overcome these limitations, we propose several potential improvements. First, introducing special boundary tags to explicitly mark sentence boundaries can enable sentence-level alignment within generated paragraphs, allowing LLMs to process multiple sentences simultaneously and reduce invocation frequency. Second, using more precise alignment tools and scripts to extract proper nouns and their translations, instead of relying on LLMs, can improve the efficiency of DelTA. Additionally, components like the Long-Term Retriever could be implemented with a dense retriever rather than depending on LLMs to identify related sentences. In the summary component, streamlining the process by merging sentences within the window directly into a single-step summary can further reduce runtime. Last but not least, training an inference trajectory designer to decide, on a case-by-case basis, whether a specific component should be invoked--a focus of recent research on intelligent agents--can add an additional layer of efficiency. These proposed optimizations will serve as key areas for future work.

Q2. How well does this scale to extremely low-resource languages?

We evaluate how well DelTA scales to low-resource languages. Considering that the spaCy NLP tool used for evaluation does not support many low-resource languages, we select Lithuanian => English as the language pair to test. We randomly sample a document (with 556 sentences) from the Europarl v9 training set as our evaluation set and employ GPT-3.5-Turbo as the backbone model. The results are shown in the following table, indicating that DelTA also performs well on the low-resource language pair:

We have added this part in Appendix F in our paper. Thanks for your advice!