SimulPL: Aligning Human Preferences in Simultaneous Machine Translation

Thank you for your positive feedback on our work, which is truly encouraging! We also appreciate your valuable suggestions. The following are our detailed responses.

Q1: Regarding the sharing of experimental materials.

A1: Thank you for your consideration of our work. We are organizing our code, datasets, and other experimental materials. We plan to share these materials with the community after our paper is published.

Q2: Regarding multiple translation options.

A2: Thank you for your interesting suggestion. Offering users the choice between latency-prioritized and quality-prioritized translations would indeed better accommodate the diverse needs of different users. In practical applications, we can run multiple SimulPL models in parallel with different hyper-parameter values of $n$, which would allow us to generate translations under varying latency conditions and fulfill the scenario you proposed. We will explore this point further in our future work.

Q3: Discussion on latency evaluation.

A3: Your comments on latency evaluation are insightful. Existing evaluation metrics based on average latency indeed fails to fully reflect the worst-case latency, and the RMS latency also provides useful reference. We appreciate your understanding of our use of common metrics, as it allows for easier and fairer comparison with previous work. We will explore more appropriate evaluation metrics for latency in our future work.

Q4: Collaboration with professional interpreters for further research.

A4: Thank you for your suggestion. In our study, both the manual evaluation of the training data and the manual revision of the test set were performed by professional simultaneous interpreters, which greatly helped us to improve the dataset's quality. As you suggested, we will consider collecting data from institutions specializing in interpreting education in our future work.

Q5: The sharing of the benchmark and GitHub repo.

A5: Yes, we plan to release our benchmark. We are organizing our code and data and will provide the GitHub link in the preprint of our paper.

Thank you again for your recognition of our work and your insightful suggestions! We truly appreciate your deep understanding and unique perspective on the SiMT task. We are committed to incorporating your suggestions into our future work.

Q5: Additional analysis experiments on more language pairs.

A5: First, we conduct experiments on translation quality (Section 5.2) and preference evaluation (Section 5.3) with three language pairs: Zh$\rightarrow$En, En$\rightarrow$Zh, and De$\rightarrow$En. Results on SacreBLEU and COMET show that SimulPL can achieve better translation quality. Both human evaluation and automatic multi-aspect evaluation indicate that SimulPL aligns better with SiMT human preferences. We believe these experiments can sufficiently validate the effectiveness of SimulPL.

Second, as for the analysis experiments, such as ablation studies (Section 5.4), impact of $\alpha$ (Section 5.5), and generalization to other preference optimiazation methods (Section 5.6), we believe our experimental findings can apply to other language pairs as SimulPL is language-agnostic.

Third, we further validate these experiments on De$\rightarrow$En task. Specifically, the results of ablations studies are as follows:

The results analyzing the impact of $\alpha$ are as follows:

The results demonstrating SimulPL's generalization to other preference optimization methods are as follows:

Through these experiments, we further validate our findings presented in the paper: (1) Both SimulDPO and MSFT improve model performance, with a more pronounced effect at low latency levels. (2) Although the effect is less pronounced in the De$\rightarrow$En task compared to the Zh$\rightarrow$En task, SimulPL's performance is still influenced by $\alpha$. (3) SimulPL also generalizes well to other preference alignment methods on the De$\rightarrow$En task.

These results confirm the reliability of our analyses. We will include these experimental results in our revised paper and further supplement the experiments on the En$\rightarrow$Zh task.

Q3: Generalizability of SimulPL.

A3: Firstly, our work focuses on aligning human preferences in SiMT task, rather than issue of general preference alignment in LLMs. We propose SimulPL to address the limitations of existing general preference learning methods in the SiMT scenarios. We believe that solving preference alignment for specific tasks is equally important.

Secondly, SimulPL still maintains distinct generalizability. For instance, the SimulPL framework can be extended to simultaneous inference tasks (Chen et al. [1]), which represents a promising direction for reducing latency of LLM responses. We will provide further clarification on this point further in the revised version of our paper. We will also explore the preference alignment of simultanous inference tasks in our future work.

Reference

【1】Chen, Chuangtao, et al. "LiveMind: Low-latency Large Language Models with Simultaneous Inference." arXiv preprint arXiv:2406.14319 (2024).

Q4: Threshold of confidence in the inference process.

A4: Thank you for your suggestion regarding the threshold $\gamma$ for confidence $c_t$. We have conducted additional experiments analyzing the impact of different threshold $\gamma$. Specifically, we assess the performenace of SimulPL on $\gamma=0.1,0.3,0.5,0.7,0.9$ with different $n$ during inference. Due to time constraints, we have only performed experiments on the Zh$\rightarrow$En task. The results are as follows:

When $\gamma$ is set to a small value ($\gamma=0.1$), the model is allowed to output tokens with low confidence, which results in a decline in translation quality, especially in low latency levels (LAAL < 4). When $\gamma$ is set to a higher value ($\gamma=0.9$), the model imposes stricter constraints on token quality, leading to unnecessary delays. We plan to further explore the impact of $\gamma$ in our future work.

Thank you for your valuable comments. We fully understand your concerns and questions. The following are our detailed responses.

Q1: Validation of our constructed data quality.

A1: Firstly, we conduct both human evaluation and automatic multi-aspect evaluation for GPT-generated training set. In the human evaluation, we employ professional simultaneous interpreters to assess the translations generated by GPT-4/4o. The results in Figure 2 indicate that the GPT-generated translations aligns better with human preferences compared to the original references. In the multi-aspect evaluation, we assess the translations with Ref-free COMET (Table 1) as well as our defined NIR, DD, and SLR metrics (Table 3 in Appendix A.2). These results demonstrate that the GPT-generated translations outperform the original references in terms of Translation Quality Preference, Monotonicity Preference, Key Point Preference, and Simplicity Preference. We will emphasize these experimental findings in the revised version of our paper.

Secondly, when constructing the test set, we ask professional simultaneous interpreters to manually revise the GPT-generated drafts to produce the final annotations, which better align with human preferences. This is mentioned in lines 221-222 on Page 5 of our paper.

Thirdly, to further validate the quality of our data and address your concerns, we add a comparative analysis of GPT-generated translations and those manually revised by interpreters on the test set. Please refer to our General Response: Further Evaluation of GPT-generated Data.

We will include our new analysis experiments in the revised version of our paper, hoping that these could alleviate your concerns.

Q2: Our novel contributions & Distinctions from related work.

A2: Our Novel Contributions: Our contribution is not limited to SimulDPO. Instead, we propose a comprehensive preference learning framework (SimulPL) tailored to the SiMT task, which includes:

1. Categorization of SiMT human preferences: We categorize human preferences in SiMT scenarios into five aspects: translation quality, monotonicity, key points, simplicity, and latency. This bridges the gap in the study of human preferences in SiMT task (Section 4.1).
2. Efficient preference dataset construction with human preference prompts: By leveraging our categorized human preferences, we can create human preference prompts, which enable the efficient construction of dataset that aligns with human preferences using LLMs like GPT (Section 4.2).
3. MSFT and SimulDPO in the training process: Existing preference optimization methods neither consider latency preferences in SiMT nor optimize the read/write policy. In contrast, SimulPL addresses these issues. In MSFT phase (Section 4.3), SimulPL jointly learns the translation ability and read/write policy for initial preference alignment. In SimulDPO phase (Section 4.4), SimulPL can not only incorporate latency preferences into the training objectives but also integrate the read/write policy into the optimization process.

Therefore, we believe your concerns regarding the novelty of our work are unnecessary. We have outlined these contributions in our Introduction and Conclusion, and we will further emphasize our contributions in the revised version of the paper.

Distinctions from Related Work: Firstly, the objectives and methods of Park etal.[1] are entirely opposite to our proposed SimulPL. In Park etal. [1], the authors aim to prevent models from generating too long responses and use a regularization term of "$-\alpha |y|$" to achieve this. In contrast, for the SiMT task, audiences prefer translations with low latency, which requires the SiMT model to translate as much content as possible based on the already received source prefix. To achieve this, we choose "$+\alpha |y|$" as an additional constraint (Equation 6 in our paper). It is important to note that our goal is not to optimize for the length itself, but rather to optimize for latency preferences. We choose "$+\alpha |y|$" because it is equivalent to optimizing for latency preferences, as proven in Appendix B.1.

Secondly, we use $|\mathrm{y}| = \sum_{t=1}^{|\mathrm{y}|+1} c_t$ to make $|\mathrm{y}|$ differentiable, allowing gradient signals to be directly propagated to the parameters through backpropagation, as shown in line 290 and Equation 8 of our paper. In contrast, Park etal. [1] treats the "$-\alpha |y|$" as a margin without further processing.

Thank you for the article you recommended. It is a valuable and interesting work. We will cite the article you recommended and clarify these distinctions in the revised version of our paper.

Reference

[1] Park, Ryan, et al. "Disentangling length from quality in direct preference optimization." arXiv preprint arXiv:2403.19159 (2024).

Finally, thank you once again for your insightful suggestions! We also appreciate the paper you recommended in Q2, which we truly believe is a valuable contribution. We sincerely hope that our responses have addressed your concerns. If so, we would be grateful if you could consider recognizing our efforts and increasing your score. Should you have any further questions, please do not hesitate to reach out. We look forward to further discussion with you!

We sincerely appreciate your recognition of our work and the increase in our score. It is an honor for us to address your concerns, and we promise to incorporate your invaluable insights into the final revised version of our paper.

Thank you for your valuable comments! The following are our detailed responses.

Q1: Highlighting our contributions.

A1: Our contribution is that we propose SimulPL, a comprehensive preference learning framework tailored to the SiMT task, which includes:

1. Categorization of SiMT human preferences: We categorize human preferences in SiMT scenarios into five aspects: translation quality, monotonicity, key points, simplicity, and latency. This bridges the gap in the study of human preferences in SiMT task (Section 4.1).
2. Efficient preference dataset construction with human preference prompts: By leveraging our categorized human preferences, we can create human preference prompts, which enable the efficient construction of dataset that aligns with human preferences using LLMs like GPT (Section 4.2).
3. MSFT and SimulDPO in the training process: Existing preference optimization methods neither consider latency preferences in SiMT nor optimize the read/write policy. In contrast, SimulPL addresses these issues. In MSFT phase (Section 4.3), SimulPL jointly learns the translation ability and read/write policy for initial preference alignment. In SimulDPO phase (Section 4.4), SimulPL can not only incorporate latency preferences into the training objectives but also integrate the read/write policy into the optimization process.

We outline our contributions in the Introduction and Conclusion. We will further emphasize these points in the revised version of our paper.

Q2: Validation of our constructed data quality.

A2: Firstly, we conduct both human evaluation and automatic multi-aspect evaluation for GPT-generated training set. In the human evaluation, we employ professional simultaneous interpreters to assess the translations generated by GPT-4/4o. The results in Figure 2 indicate that the GPT-generated translations aligns better with human preferences compared to the original references. In the multi-aspect evaluation, we assess the translations with Ref-free COMET (Table 1) as well as our defined NIR, DD, and SLR metrics (Table 3 in Appendix A.2). These results demonstrate that the GPT-generated translations outperform the original references in terms of Translation Quality Preference, Monotonicity Preference, Key Point Preference, and Simplicity Preference. We will emphasize these experimental findings in the revised version of our paper.

Secondly, when constructing the test set, we ask professional simultaneous interpreters to manually revise the GPT-generated drafts to produce the final annotations, which better align with human preferences. This is mentioned in lines 221-222 on Page 5 of our paper.

Thirdly, to further validate the quality of our data and address your concerns, we add a comparative analysis of GPT-generated translations and those manually revised by interpreters on the test set. Please refer to our General Response: Further Evaluation of GPT-generated Data.

We will include our new analysis experiments in the revised version of our paper, hoping that these could alleviate your concerns.

Q3: Explanation of the uppercase $\mathrm{X}$ and lowercase $\mathrm{x}$ in Equations 6 and 7.

A3: In Equations 6 and 7, $\mathrm{X}$ represents the complete source sentence, while $\mathrm{x}$ denotes the received source prefix. This distinction is noted near Equation 3 (around lines 236-240) and Equation 6 (around line 279). We will further clarify this point in our revised paper.

Q4: Explanation of the ablation studies.

A4: The numerical results of the ablation studies in our paper are presented as follows. The "$\Delta$ BLEU" means the improvement in SacreBLEU score of SimulPL compared to MSFT.

The results show that SimulPL outperforms MSFT at all latency levels, with a maximum increase of 1.02 BLEU and an average increase of 0.83 BLEU, which is a notable improvement. Furthermore, based on analyses from existing work [1], the gains from incorporating SimulDPO into SimulPL are also reasonable. Therefore, we believe that SimulPL indeed outperforms MSFT, and concerns about its effectiveness are unnecessary.

Reference

[1] Saeidi, Amir, Shivanshu Verma, and Chitta Baral. "Insights into Alignment: Evaluating DPO and its Variants Across Multiple Tasks." arXiv preprint arXiv:2404.14723 (2024).

Once again, thank you for your valuable feedback! We hope our responses have addressed your concerns and questions. If so, we would greatly appreciate it if you could consider acknowledging our efforts and kindly increasing your score. Should you have any further questions, please feel free to reach out. We look forward to further discussions with you!

Q3: The requirement for recalculating in LLMs.

A3: As you pointed out, existing LLM-based methods [1] [2] [3] often require recalculating the hidden states after the insertion point when new source tokens are input. However, previous work [3] has shown that this recalculation does not significantly affect real-world performance. Furthermore, techniques such as SimulMask [4] can also be applied to existing LLM-based SiMT models to mitigate the overhead of recomputing hidden states.

Additionally, our work focuses on improving the SiMT human preferences alignment. Therefore, we adopt a simpler LLM-based SiMT model, which does not impact the applicability of SimulPL to LLM-based SiMT models that do not require repeated hidden state calculations.

Thank you for your insightful question. We will explore this issue further in our future work.

Reference

[1] Wang, Minghan, et al. "Simultaneous machine translation with large language models." arXiv preprint arXiv:2309.06706 (2023).

[2] Koshkin, Roman, Katsuhito Sudoh, and Satoshi Nakamura. "Transllama: Llm-based simultaneous translation system." arXiv preprint arXiv:2402.04636 (2024).

[3] Guo, Shoutao, et al. "SiLLM: Large Language Models for Simultaneous Machine Translation." arXiv preprint arXiv:2402.13036 (2024).

[4] Raffel, Matthew, Victor Agostinelli, and Lizhong Chen. "Simultaneous Masking, Not Prompting Optimization: A Paradigm Shift in Fine-tuning LLMs for Simultaneous Translation." arXiv preprint arXiv:2405.10443 (2024).

Q4: Feasibility of constructing a reward model.

A4: Thank you for your insightful suggestion. The idea of constructing a reward model sounds promising and could potentially offer higher performance ceilings. We will explore this direction further in future work.

Once again, we deeply appreciate your recognition of our work and your valuable suggestions! We are committed to incorporating your suggestions into our future work.

Thank you for recognizing our work and providing valuable feedback—it is truly encouraging! The following are our detailed responses.

Q1: Regarding data construction.

A1: Training Cost: The cost of annotating all the data was approximately \$960. Given the recent price reduction of GPT-4o from \\$5 to \$2.5, as well as the introduction of GPT-4o mini, this cost can be further reduced.

Data Quality: First, our test set was manually revised by professional interpreters. This is mentioned in lines 221-222 on Page 5 of our paper. Therefore, the evaluation results based on these test sets strongly support the effectiveness of SimulPL in aligning with human preferences rather than GPT-generated preferences.

Second, as you suggested, we compare the GPT-generated translations with the manually revised translations. Please refer to our General Response: Further Evaluation of GPT-generated Data for more details.

Q2: The impact of the confidence threshold.

A2: Thank you for your advice. As you suggested, we validate the impact of the confidence threshold on SimulPL. Specifically, we assess the performenace of SimulPL on $\gamma=0.1,0.3,0.5,0.7,0.9$ with different $n$ during inference. Due to time constraints, we only perform the experiments on Zh$\rightarrow$En task. The results are as follows:

When $\gamma$ is set to a small value ($\gamma=0.1$), the model is allowed to output tokens with low confidence, which results in a decline in translation quality, especially in low latency levels (LAAL < 4). When $\gamma$ is set to a higher value ($\gamma=0.9$), the model imposes stricter constraints on token quality, introducing unnecessary delays. We plan to further explore the impact of $\gamma$ in our future work.