ExLM: Rethinking the Impact of $\texttt{[MASK]}$ Tokens in Masked Language Models

We appreciate the insightful suggestions from Reviewer j1MR. In the following sections, we will address all your concerns. These discussions will also be incorporated into the final camera-ready version of the paper. Any further comments are welcome!

Q1: What is the authors' main contribution in analyzing unreal tokens?

Thank you for the question. The repeated MLM experiments are primarily designed to analyze the key factors that influence the performance of MLM models and the underlying reasons. Toward this goal, this section presents the following two main contributions:

1. We demonstrate that the issue of unreal tokens, which has received significant attention in previous works, is not the primary factor affecting MLM performance.
2. We show that corrupted semantics and the resulting multimodality phenomenon are more critical in influencing the performance of MLMs.

In summary, this analytical section offers a new perspective for understanding MLMs by highlighting the previously overlooked importance of corrupted semantics. It also suggests that future developments of MLMs should focus more on addressing the problem of multimodality.

Q2: The connection between Section 3 and Section 4 is weak.

Thank you for the suggestion. We have revised Sections 3 and 4 to better highlight their connection. These changes will be included in the final camera-ready version. Specifically:

1. The last paragraph of Section 3.3 has been updated to emphasize multimodality as a key factor affecting MLM performance. An effective model should thus be able to mitigate the impact of semantic multimodality.
2. Section 4.1 has been reorganized to align with Section 3’s analysis. To address multimodality, ExLM introduces two mechanisms:
   • Intra-token Multimodality: Each missing token may have a diverse set of plausible candidates. Thus, we propose States Expansion to build a larger semantic space, enabling richer token predictions.
   • Inter-token Multimodality: The meaning of a token is intricately dependent on the semantics of its surrounding tokens. Thus, we introduce a Dependency Capture mechanism using a transition matrix to model semantic dependencies across tokens.

Q3: How does ExLM model multiple possible semantics?

We introduce a state expansion mechanism in ExLM, where each [MASK] token is associated with multiple expanded states, each representing a different possible semantic choice for that token.

Q4: The baseline approaches seem too weak as they were proposed a few years ago.

Thank you for the valuable suggestion. We have added comparisons with a more recent baseline model, 3ML_{self} [Liao et al., 2022], in Table 3. The results show that our ExLM model outperforms this stronger baseline, as detailed below:

In addition, we further evaluated the ExLM model on the SuperGLUE benchmark [Wang et al., 2019]. The following table shows that ExLM also achieves significant improvements over several strong baselines, further confirming the effectiveness of our approach:

Q5: The proposed method, ExLM, incorporates 2D RoPE, which further enhances model performance, making it difficult to assess the true effectiveness of the approach.

Thank you for pointing this out. In our experiments, all compared MLM baselines, including Vanilla MLM and Vanilla MLM++, adopt the same backbone architecture as ExLM, which uses RoPE as the default positional encoding. Therefore, the performance comparisons are made on a fair basis and can indeed assess the true effectiveness of the proposed method.

References:

[Hou et al., 2022] Hou, Le, et al. "Token Dropping for Efficient BERT Pretraining." ACL 2022.
[Zhong et al., 2023] Zhong, Qihuang, et al. "Revisiting Token Dropping Strategy in Efficient BERT Pretraining." ACL 2023.
[Wang et al., 2019] Wang, Alex, et al. "SuperGLUE: A stickier benchmark for general-purpose language understanding systems." NeurIPS 2019.
[Liao et al., 2022] Liao, Baohao, et al. "Mask More and Mask Later: Efficient Pre-training of Masked Language Models by Disentangling the [MASK] Token." EMNLP 2022.
[Ke et al., 2021] Ke, Guolin, Di He, and Tie-Yan Liu. "Rethinking Positional Encoding in Language Pre-training." ICLR 2021.

We express our gratitude to Reviewer 3GXq for the suggestions. In the following, we address all your concerns regarding the evaluation of NLU tasks, additional experiments on more tasks, the efficiency analysis of ExLM, and the details of repeated MLM experiments. These discussions will also be incorporated into the final camera-ready version of the paper. We hope our responses help clarify the paper, and we welcome any further comments or suggestions!

Q1: These dev sets are potentially biased to the official test sets, and the results are unstable.

A: Thank you for your insightful comment. We have evaluated the stability of the ExLM model on the GLUE benchmark by reporting the standard deviation of performance across multiple runs. The results, summarized in the table below, demonstrate that ExLM exhibits good stability overall:

Standard Deviation on the GLUE Benchmark:

Furthermore, to further verify the effectiveness of ExLM, we also evaluated its performance on the SuperGLUE benchmark [Wang et al., 2019]. As shown in the table below, ExLM significantly outperforms several strong baseline models, further supporting the validity of our proposed approach:

Q2: At least one more dataset result (e.g. QQP, RTE) is needed to fully justify the claims.

A: Thank you for the suggestion. In fact, the repeated MLM results on QQP and RTE have already been included in Appendix F. As stated in Section 3.2 of the main text: “We also provide the results of this experiment on more tasks in Appendix F.”

Q3: The computation efficiency of ExLM compared to MLM and other MLM variants, e.g., time cost, memory cost.

A: Thank you for your comment. As mentioned in Section 5.3 of the main text, “Efficiency and entropy analysis of EXLM are provided in Appendix Q”. Specifically, we present a detailed discussion of ExLM's training time cost in Appendix Q.2, including a comparison with MLM models. Moreover, memory cost analysis is provided in Appendix I.4, which is also referenced in Section 4.3 of the main text: “The efficiency analysis is provided in Appendix I.”

Q4: The details of repeated MLM experiments are not clear. I am not sure whether they pre-train the MLM (e.g., on Wiki) and then fine-tune on MNLI, or just fine-tune on MNLI.

A: Thank you for your comment. In each group of repeated MLM experiments, we pre-trained separate MLM models using different values of k and p, and then fine-tuned and evaluated them on downstream tasks using the same settings. As stated in Appendix B.1 (Pre-training Configuration): “In the Repeated MLM experiment, we train a series of MLMs with different p and k parameters.”

References:

[Devlin et al., 2019] Devlin, Jacob, et al. "Bert: Pre-training of deep bidirectional transformers for language understanding." NAACL 2019.

[Hou et al., 2022] Hou, Le, et al. "Token Dropping for Efficient BERT Pretraining." ACL 2022.

[Zhong et al., 2023] Zhong, Qihuang, et al. "Revisiting Token Dropping Strategy in Efficient BERT Pretraining." ACL 2023.

[Wang et al., 2019] Wang, Alex, et al. "SuperGLUE: A stickier benchmark for general-purpose language understanding systems." NeurIPS 2019.

Thank you for your valuable suggestions. In the following, we provide detailed responses to all your concerns regarding the missing reference, model details, writing organization, and implementation issues. These discussions will also be incorporated into the final camera-ready version of our paper. We sincerely welcome any further feedback you may have.

Q1: Missing reference about the noising techniques for ASR error robustness

A: Thank you for your suggestion. We have now added the missing citation to the draft of our paper, and we will also ensure that it is properly included in the final camera-ready version.

Q2: The explanation of algorithm details should be in the main text

A: Thank you for the feedback. We have reorganized the structure of the paper accordingly. In earlier versions, some important methodological details were placed in the appendix due to space limitations. However, since the camera-ready version allows for an additional page, we have moved more details about ExLM—such as the dynamic programming procedure and the forward-backward algorithm—into the main body of the paper.

Q3: The relation between Section 3 and Section 4 is not clear

A: Thank you for the insightful comment. We have revised the writing of Section 4.1 to more clearly highlight the connection between Section 3 and Section 4. Specifically, the analysis in Section 3 demonstrates that multimodality is a key factor impacting the performance of MLMs. To enhance ExLM’s capacity for handling multimodality, we propose two mechanisms tailored to address two distinct levels of multimodality:

1. Intra-token Multimodality: The possible candidates for each masked token can be highly diverse. To address this, we introduce a States Expansion mechanism to construct a larger semantic space, allowing the model to learn richer and more diverse semantic information.
2. Inter-token Multimodality: The meaning of a token is intricately related to the meanings of surrounding tokens. To capture this, we incorporate a Dependency Capture mechanism, where a transition matrix is used to explicitly model semantic dependencies between different states.

This design clearly establishes the relevance of the findings in Section 3 to the architectural choices in Section 4. These improvements have been added to the camera-ready version.

Q4: The impact of corrupted semantics in MLM is obvious and well-managed, so it’s unclear why it’s framed as a significant issue

A: Thank you for raising this important point. We agree that "corrupted semantics" are intentionally introduced as a form of noise in the MLM denoising process, and such noise is a legitimate component of the training strategy. Indeed, our goal is not to eliminate this noise—on the contrary, we explicitly mention in the paper that a certain level of corrupted semantics can benefit training.

However, the central issue we aim to address is the unintended side effects this noise may have on model behavior. Specifically, the purpose of the denoising process is to train the model to recover clean data from noisy inputs. While noise facilitates learning, it also modifies the model's input distribution, which may interfere with the model’s ability to learn accurate representations.

Therefore, a well-designed denoising process should maintain a reasonable noise level while minimizing the adverse effects of noise on model behavior. Our study identifies one such undesirable effect—multimodality—arising from corrupted semantics. We aim to mitigate the impact of multimodality without eliminating the beneficial aspects of noise, which is the motivation behind our focus on corrupted semantics in this work.

Q5: Why is the index of the last state chosen for the final loss? What if the last state is never reached?

A: Thank you for the question. The reason we choose the index of the last state (i.e., [EOS]) for the final loss is that we enforce a structural constraint: every valid decoding path in the DAG must terminate at the last state. This design eliminates the need to enumerate all possible ending states when summing the probabilities of valid paths, thereby reducing computational complexity.

Moreover, the DAG in ExLM is constructed in a way that guarantees the reachability of the last state from all other nodes. Thus, the last state is always reachable, and the validity of the loss computation is ensured. We will include these details in the camera-ready version.

Q6: How is ExLM modeled without the transition matrix? Is it like CTC?

A: Yes, your understanding is exactly correct. When the transition matrix is not used, ExLM is trained in a manner similar to CTC (Connectionist Temporal Classification), where the model considers all possible alignments between the expanded states and the target tokens. More implementation details on this setup will be added to the camera-ready version of the paper.

Thank you for your thoughtful and constructive feedback. We sincerely appreciate your valuable suggestions. Below, we address all your concerns regarding the SMILES tasks, the standard deviation of model performance, and additional NLU tasks. The relevant results will be incorporated into the final camera-ready version of our paper. We hope our responses help clarify the work, and we welcome any further comments.

Q1: Why do they take SMILES as the evaluation task?

A: Thank you for the question. The reason for selecting SMILES as the evaluation task is that masked language modeling (MLM) on SMILES also encounters issues similar to those observed in natural language modeling. We conducted Repeated MLM experiments on the BACE task (a SMILES classification task), and the results are shown in the table below. These results exhibit a trend consistent with Repeated MLM experiments on natural language data, indicating that semantic corruption significantly affects SMILES modeling:

These findings can be attributed to the presence of strong semantic dependencies in SMILES representations. For instance, forming valid molecular functional groups often requires precise coordination among multiple atoms, which conventional MLM fails to capture due to its inability to model such semantic associations effectively.

Q2: The standard deviation of the results in Table 1 and Table 3

A: Thank you for the helpful suggestion. We have computed the standard deviations of ExLM’s performance on both the GLUE and MoleculeNet benchmarks. The results are summarized in the tables below. Overall, ExLM demonstrates good stability across these benchmarks:

Standard deviation on MoleculeNet benchmark:

Standard deviation on GLUE benchmark:

Q3: There should be more results on NLU tasks to demonstrate the effectiveness of ExLM

A: Thank you for the excellent suggestion. We further evaluated the performance of ExLM on the SuperGLUE benchmark [Wang et al., 2019]. As shown in the table below, ExLM outperforms several baseline models by a notable margin, which demonstrates the effectiveness of our method:

We hope these additional results provide more clarity and demonstrate the strong performance of our proposed model. Please let us know if further information are needed!

References:

[Devlin et al., 2019] Devlin, Jacob, et al. "Bert: Pre-training of deep bidirectional transformers for language understanding." NAACL 2019.

[Hou et al., 2022] Hou, Le, et al. "Token Dropping for Efficient BERT Pretraining." ACL 2022.

[Zhong et al., 2023] Zhong, Qihuang, et al. "Revisiting Token Dropping Strategy in Efficient BERT Pretraining." ACL 2023.

[Wang et al., 2019] Wang, Alex, et al. "Superglue: A stickier benchmark for general-purpose language understanding systems." Advances in neural information processing systems 32 (2019).