Non-Autoregressive Machine Translation as Constrained HMM

We sincerely appreciate your thorough and thoughtful review and your recognition of the significance of our work. It's truly encouraging to know that our efforts to view DAT as a variant of HMM and address the label bias issue have been understood and valued.

About the Lower Triangular Mask in DAT

We understand that you attempt to treat DAT as a conventional HMM, where the transition probability between any two states can be greater than zero, instead of a left-to-right HMM where we only allow left-to-right transitions.

To illustrate the importance of the lower triangular mask (LTM) for optimizting the objective function in DAT, let's start with a toy example. This example is similar to but slightly more complex than the one in Figure 3(a) of our paper. Assume we have 6 positions (or latent states, 0-5) and 4 target tokens [BOS, y1, y2, EOS]. We force the start and end of the target sequence to be the first and the last positions, i.e., 0 and 5. Thus, position 0 and 5 are responsible for predicting the special BOS and EOS tokens.

Removing the mask confuses training

With the LTM, the latent paths that can yield the target sequence [BOS, y1, y2, EOS] are [0,1,2,5], [0,1,3,5], [0,1,4,5], [0,2,3,5], [0,2,4,5], [0,3,4,5]. However, without the LTM, we would have additional latent paths, namely, [0,2,1,5], [0,3,1,5], [0,4,1,5], [0,3,2,5], [0,4,2,5],[0,4,3,5]. Note that an additional path like [0,2,1,5] corresponds to tokens [BOS, y1, y2, EOS], respectively.

According to the Eq. (5) in the original DAT paper [1], only one latent path will stand out during the training process for each training instance, and this standing-out path can be any valid path. For example, with LTM, [0,1,2,5] might be the most probable path for the target sentence [BOS, the, world, EOS] while [0,1,3,5] is the most likely one for [BOS, the, earth, EOS]. Without LTM, [0,1,2,5] might still be chosen for [BOS, the, world, EOS], but [0,2,1,5], which has a right-to-left transition 2->1, might be selected for [BOS, the, earth, EOS]. Thus, when we keep the LTM, the model is more likely to share some learned patterns (e.g., placing "the" at position 1) between different training examples. Nevertheless, when we remove the LTM, half of the training instances will likely to have a winning left-to-right latent path whereas another half choose right-to-left. This is because the DAT model parameters are randomly initialized at the start of training. Consequenttly, this random L2R-R2L choices will make it hard for the model to share learned patterns among the training examples. Hence, the model cannot compress the training data effectively and struggles to converge.

Removing the mask creates loops during inference

Another reason for keeping the LTM is that we can avoid loops in the decoding path, which is important for generating a complete sequence. For example, if we remove the LTM, and we happen to have the transition probability P(1->2)=0.99 and P(2->1)=0.99 during greedy inference, arriving at position 1 would make position 2 the most likely next position. However, once at position 2, the most likely candidate becomes position 1, which leads to a loop. We will never reach the EOS token (the last position) and simply generate a repetitive and meaningless sequence.

This LTM is important for both training and inference, and we sincerely thank you for asking this question so that more readers can understand.

About Glancing Training in other HMMs

This is an interesting question! Even though we are not experts in the speech domain, we'd like to share our understanding of glancing training in HMM-DAT.

Ideally, the DAT decoder inputs should be only the learnable positional embeddings plus the unk token embeddings, but with glancing training, we replace some unk tokens with the ground truth target tokens, which provides richer training signals in the forward pass as well as rich gradient info in the backward pass, and thus leads to better convergence. By richer, we refer to the fact that different training examples will provide different glancing input tokens with glancing training. Without glancing, the inputs are always the same, consisting only of the unk token plus positional embedding. If you are using a non-autoregressive decoder in speech recognition, you can similarly adopt the glancing training strategy. We are happy to discuss more on this topic if you don't mind providing more background information or related papers.

Thank you once again for your insightful review. We hope our response has addressed your queries satisfactorily. We look forward to further discussions and exchanges our views.

[1]: Directed Acyclic Transformer for Non-Autoregressive Machine Translation (Huang Fei et al., ICML 2022)

Thank you for your insightful review and constructive feedback. We appreciate your time and effort, which has helped us improve our paper.

Response to Question 1

We attempted to integrate two strategies, but we did not see an improvement in performance compared to using each approach separately. We believe this is due to the following reasons:

• In AW-HMM, as discussed in Section 5.2 (see Figure 4), there exists a delicate balance between mitigating label bias and preserving the expressiveness offered by a larger window size in AW-HMM.
• In the context of Bi-HMM, L2R and R2L HMMs attempt to counteract the label bias by sharing all the parameters except for the transition matrix (which contains only 0.5M paramters). Consequently, the expressive capabilities of both L2R and R2L HMMs are somewhat restrained.
• If we impose the window size constraint of AW-HMM on both L2R and R2L HMMs, the expressiveness of both HMMs is further reduced, which consequently did not lead to improved performance.

We appreciate this insightful question and have incorporated a detailed discussion on this phenomenon in our revised manuscript (see the first paragraph in Section 6).

Response to Question 2

The performance improvements our methods can bring to Directed Acyclic Transformer (DAT) across different translation directions are closely correlated with the performance gap between DAT and the Autoregressive (AR) Transformer. As confirmed both theoretically and empirically in [2], Non-Autoregressive (NAR) models are not as expressive as AR models. Hence, we can consider the performance of the AR Transformer as a theoretical upper bound for all NAR models. We have tabulated the best NAR model performances for all translation directions from the Table 1 of our paper, and compare them with the AR Transformer.

As can be seen, the DAT model can perform similarly to AR Transformer on En-De and Zh-En, with only a 0.4 and 0.2 BLEU gap. However, for De-En and En-Zh, the BLEU score differences are 1.0 and 1.1, respectively. Given that our AW-HMM/Bi-HMM are also NAR models and DAT is already pretty close to the upper bound (AR Transformer) on En-De and Zh-En, the performance gains improvements appear to be relatively modest.

This point has been clarified and added to our revised paper (see the last paragraph in Section 5.1). We sincerely thank you for bringing up this important issue.

We hope that our response has satisfactorily addressed your questions. If there are any additional questions or points of clarification you require, we respectfully invite you to share them.

References

[1]: Directed Acyclic Transformer for Non-Autoregressive Machine Translation (Huang Fei et al., ICML 2022)

[2]: An EM Approach to Non-autoregressive Conditional Sequence Generation (Zhiqing Sun, Yiming Yang, ICML 2020)

In my view, the first two weaknesses you mentioned are based on a bias against the NAT field rather than inherent issues with the paper itself. When evaluating a paper, we should focus more on the content of the paper itself, rather than imposing the problems of the entire field on a single paper. Moreover, I believe the issues with BLEU are not intentionally caused by the authors, as previous works have used BLEU for EN-DE and DE-EN, so it is acceptable to use BLEU for fairness. The authors also reported SACREBLEU results for WMT 17 EN-ZH.

I think this paper has a deep understanding of DAT (in the view of HMM). Beyond the acceleration advantages of NAT this model is also a variant different from the existing autoregressive models. I believe the academic community must encourage such exploration, otherwise, neural machine translation (NMT) will never replace SMT. To my knowledge, the performance of early NMT models (e.g., RNN Search) was not significantly better than SMT.

Thank you for taking time to review our paper.

Firstly, we would like to clarify that we aim to improve the best NAR model from a novel theorectical perspective, not to push the translation frontier.

The primary focus of our study is on the extension of the understanding of the Directed Acyclic Transformer (DAT) as a left-to-right Hidden Markov Model (HMM) and to address the inherent label bias problem with our AW-HMM/Bi-HMM approaches. Our objective was to delve deeper into the theoretical aspects of this model and propose improvements in the same, rather than advancing the translation landscape. We believe our findings are original and can add new knowledge to our community.

The concerns you raise about the broader Non-Autoregressive Translation (NAT) community, while relevant to the field as a whole, fall outside the scope of this particular study. We must highlight that even after the publication of DAT, many research papers continue to use WMT'14, WMT'16, WMT'17 translation tasks as the testbed for NAR models, e.g., [2],[3],[4],[5],[6],[7],[8],[9],[10]. Even the authors of [11], whom you cited, used WMT'13, 14, 16 for their evaluations, as their goal was also not to push the translation frontier.

As Reviewer z6kj accurately highlighted, the potential limitations existing within the broader NAT community should not overshadow the specific contributions made in individual studies, such as ours, which aims to enhance the state-of-the-art NAR architecture from a novel HMM perspective.

About BLEU and SacreBLEU

We choose to report BLEU for EN-DE and SacreBLEU for ZH-EN for the sake of a consistent and fair comparison with previous work, as it was hard to accurately reproduce all the previous baselines and report their SacreBLEU scores.

About GPU speedup numbers

Comparing a non-optimized NAR model with an optimized AR model would be unfair. The AR model field has various optimization techniques for acceleration, while the NAR field, being at its early stage, lacks specially-designed speedup techniques. However, it is crucial to note that the time complexity of NAR/AR model is $O(N)/O(N^2)$ [12], respectively. Theoretically, as indicated in [12], NAR models can surpass AR models in terms of decoding speed under any test conditions, provided that they are properly optimized. We look forward to such advancements in NAR accerleration, but again, it falls outside the scope of this paper.

About Parameter Count

We list our parameter count below:

The table indicates that our AW-HMM/Bi-HMM and DAT models only require a 3%-5% increase in parameter count compared to the baseline models. This suggests that our performance improvements are due primarily to our algorithm, rather than an increase in parameters.

This information has been added to our revised paper (see Appendix D).

In response to your minor queries:

Response to [minor 1]

Our models/methods are not language-specific, and should thus be applicable to other language pairs. We chose En-De and En-Zh to maintain consistency with previous work, as in [1].

Response to [minor 2]

Regarding the window size trend experiment depicted in Figure 4, we confined our investigation to the WMT'14 En-De dataset. This decision was made due to the significant computational cost associated with examining 8 different window sizes (ranging from 1 to 8) for each of the 4 translation directions (En-De, De-En, Zh-En, En-Zh).

However, please note that we have conducted experiments with window sizes of 7 and 8 for all language pairs, the results of which are documented in Appendix B.

In relation to the tuning of the window size $\beta$ hyperparameter, we followed standard hyperparameter tuning practices. The hyperparameter was tuned on the development set, a process that is mentioned in the caption of Figure 4.

Response to Questions

Our training hyperparamters follow previous work [1]. For the hyperparmeters in our AW-HMM and Bi-HMM, we tune them on the dev set.

We welcome further questions regarding the DAT-as-an-HMM perspective and our Bi-/AW-HMM models and would be glad to elaborate on these points.

[1]: Directed Acyclic Transformer for Non-Autoregressive Machine Translation (Huang Fei et al., ICML 2022)

[2]: Rephrasing the Reference for Non-autoregressive Machine Translation (Chenze Shao et al., AAAI 2023)

[3]: Selective Knowledge Distillation for Non-Autoregressive Neural Machine Translation (Min Liu et al, AAAI 2023)

[4]: Diff-Glat: Diffusion Glancing Transformer for Parallel Sequence to Sequence Learning (Lihua Qian et al., Dec 2022)

[5]: DePA: Improving Non-autoregressive Machine Translation with Dependency-Aware Decoder (Jiaao Zhan et al., IWSLT 2023)

[6]: Deep Equilibrium Non-Autoregressive Sequence Learning (Zaixiang Zheng et al., ACL 2023)

[7]: Viterbi Decoding of Directed Acyclic Transformer for Non-Autoregressive Machine Translation (Chenze Shao et al., EMNLP 2022)

[8]: Non-autoregressive Streaming Transformer for Simultaneous Translation (Zhengrui Ma et al., Oct 2023)

[9]: DINOISER: Diffused Conditional Sequence Learning by Manipulating Noises (Jiasheng Ye et al., 2023)

[10]: AMOM: Adaptive Masking over Masking for Conditional Masked Language Model (Yisheng Xiao et al., AAAI 2023)

[11]: Non-Autoregressive Neural Machine Translation: A Call for Clarity (Schmidt et al., EMNLP 2022)

[12]: Deep Encoder, Shallow Decoder: Reevaluating Non-Autoregressive Machine Translation (Jungo Kasai et al., ICLR 2021)