MEXMA: Token-level objectives improve sentence representations

We thank the reviewer for their comments and suggestions. It resulted in several additions to the paper, that we believe improved its quality.

Fair comparisons:

We appreciate the recommendation to enhance the fairness of the comparisons. Although we did not include it in the paper, as our primary focus was on presenting the optimal scores for each model, we did explore your suggestions and incorporated them into our analysis.

For LaBSE, we did reimplement their training objective using XLM-R and our training data. However, since the xsim++ results were significantly worse than the ones we obtained with the public LaBSE model, we decided not to include them.You can find those results in the table below under “LaBSE with XLM-R”. Given there is no official public implementation of LaBSE, and the large gap in xsim++ compared to the public model, we opted to always compare it to the public model.

For fairer comparison with SONAR, we trained MEXMA using the NLLB encoder as backbone. This is easier than training SONAR with XLM-R as backbone, since we would need a translation decoder for XLM-R, or initialize it randomly, which would be less fair to compare to SONAR that starts from NLLB, a very strong translation model. The results are worse than MEXMA and SONAR across all tasks. This result is likely due to the fact that NLLB was pre-trained with a translation objective, and was never trained to do masked language modeling, like XLM-R.

We report here the results obtained with LaBSE/XLM-R and MEXMA/NLLB in the following table:

We thank the reviewer for these suggestions. These results have been added to the paper under appendix B.4.

Importance of token-level gradients:

The token-level gradient in MEXMA is both important and subtle. We try to clarify this concept in the following and will update the paper with a clearer explanation. Our experiments are designed to demonstrate the importance of how the encoder is updated, whether from token-level and/or sentence-level gradients, on overall performance. Specifically, when we apply only cross-entropy (CE) or combine CE with alignment while stopping the gradient flow at the token level, the encoder is updated solely through gradients derived from the sentence representation. Our goal is to illustrate that allowing gradients to flow from each masked token provides the encoder with a richer training signal, thereby enhancing its quality. The ablation study, which involves stopping the gradients, is intended to highlight this effect. We wish to underscore not merely the presence of a token-level loss, but the distinction between updating the encoder exclusively via sentence-level gradients versus incorporating gradients from all tokens. Our findings indicate that the inclusion of these additional gradients significantly boosts performance.

Does KoLeo loss apply to all CLSE?

The KoLeo loss is particularly relevant in MEXMA because we have a non-contrastive alignment loss, which tends to create a space that is more collapsed than a contrastive objective, similarly to DINOv2 (cite). We can say this according to a basic measure of the variance across components. Approaches like SONAR and MEXMA would benefit from the KoLeo loss. However, a contrastive approach like LaBSE would not, given that it already has "repulsing forces" that increase the spread over the space. Therefore, anisotropy is a more significant issue in MEXMA (and other non-contrastive approaches), making the KoLeo loss a valuable addition. Thank you for pointing out this was not clear enough, we will improve its clarity in the paper.

Monolingual cross-entropy, instead of cross-lingual

Based on our experiments the cross-lingual objective is crucial for the performance of the model. Without it the tokens are not as aligned across languages, and the sentence representations do not encode as much information. The end result is that this worsens performance. However, thank you for your suggestion, we have added a comparison of monolingual and cross-lingual unmasking in appendix B.2 of the paper, where we show that monolingual performs much worse.

We thank the reviewer for their comments and suggestions, we believe they increased the quality of our paper. We added fairer comparisons with current SOTA in appendix B.4, and added an ablation of the importance of the cross-linguality in appendix B.2.

Thank you for going through our paper thoroughly and in detail. We are glad that you appreciated our work and found it valuable.

Section 5.6 is not clear what we gain by knowing about the lowest entropy in MEXMA. The attention sink is more interesting for the paper and was placed in the appendix.

We are grateful for your recognition of the importance of Section 5.6. Our primary objective was to delve into the behavior of the models to elucidate why they yield varying results in downstream tasks. By examining both the content of the tokens (Section 5.5) and the manner in which these tokens are integrated to form sentence representations (Section 5.6), we gain a comprehensive understanding. Section 5.6, in particular, reveals that MEXMA's focuses on specific tokens, rather than attending uniformly to all tokens, underscoring its ability to highlight the most critical tokens for sentence representation. While the attention sink phenomenon is indeed intriguing, to preserve the focus of the paper, we decided not to involve too much of the main manuscript on it. However, as per your valid suggestion, we have referenced various relevant papers for readers that might be interested.

L457 we should specify the size of NLLB model

We apologize for the oversight and thank the reviewer for pointing this out. We had mentioned the size of the SONAR encoder in Table 7 (766M parameters), which led us to believe that specifying the size of NLLB was unnecessary (1.3B parameters). We appreciate your attention to detail and have added this information in the paper. Regarding the representations used for the comparison, we only use the output of the encoder to create the sentence representations, we do not use the decoder. We have also added this information in the paper to improve clarity.

Thank you for your comments and suggestions for improvement, we have incorporated them into our paper.

Thank you for the thorough comments and suggestions, we believe that they resulted in the addition of relevant experiments, related work and strengthened our paper.

Novelty:

We appreciate the mention of those relevant papers. The first mentioned paper https://openreview.net/forum?id=r1xCMyBtPS shows that aligned word representations result in better sentence performance across languages, which is very relevant for our work indeed, as shown by DAP that we cite in our paper. We now cite those two papers, thanks for the suggestion. The objective of the mentioned papers is, however, different from ours, since they aim at aligning word representations, while our objective is to create general sentence representations that encode the relevant information of the sentences and are aligned across languages.

However, compared to existing Cross-lingual sentence encoder (CLSE) approaches, combining both MLM and alignment or not, our approach differs from previous existing approaches in the following key novel aspects:

• Previous approaches did not use unmasking as a means to create a better sentence representation as we do, with heavy token unmasking given the sentence representation, enforcing the sentence to encompass all the relevant information in the input. Previous approaches also don’t mix tokens in one language and sentence representations in another. We show this is crucial for the performance of MEXMA, and added an ablation in appendix B.2 that shows the importance of cross-linguality for performance.
• Another relevant point is that most previous approaches updated the encoder only with a gradient from the sentence representation, meanwhile we update the encoder with gradients from both the sentence and the individual tokens. We show in sections 5.1 and 5.2 that this change is crucial to achieve our new SOTA results, increasing performance across all tasks. Specifically, it improved the xsim++ results in 1.7% (taken from table 5 in the paper).
• Additionally, most approaches are contrastive which require large batch sizes and/or good hard negatives to work well. In MEXMA (and SONAR), this is not needed because of the usage of a non-contrastive alignment loss.

Massive quality supervised data:

We acknowledge the reviewer's concern regarding the dependency on large datasets in SOTA approaches. It is indeed a common characteristic among current SOTA methods, such as SONAR, LaBSE, and DAP, to require substantial amounts of translation data. Many of these approaches, namely LaBSE, utilize datasets that are either difficult to obtain or process. In contrast, our approach leverages publicly available data, specifically a subset of the data used to train SONAR. The dataset we employ is described in the NLLB paper [1], which released high-quality data for 200 languages, including many low-resource languages. The availability and quality of the NLLB data facilitate the extension of our work to additional low-resource languages or the augmentation of existing language datasets with relative ease. We believe that the reliance on large datasets will continue to be a significant factor for SOTA approaches in the foreseeable future. Our approach, by utilizing accessible and high-quality data, aims to address this challenge and contribute to the ongoing development of multilingual models. This strategy not only supports the scalability of our method but also underscores its potential applicability to a broader range of languages.

Multimodality:

Extending the work to multiple modalities is out of the scope of this paper, given that our objective was to create multilingual sentence representations from text. If we were to include multiple modalities we would have to consider different encoders, losses and strategies, namely CLIP-like. However the MEXMA architecture would be suitable to replace a CLIP-like approach, i.e. an approach where a contrastive loss is applied between a text and image encoder.

Analysis:

• Concatenating tokens instead of using the sentence representation as context
  1. If we concatenate the tokens of A and B, in a similar manner to XLM-R, e.g. [CLS] t_a_1, t_a_2, … , t_a_n [SEP] t_b_1, t_b_2, … , t_b_n [SEP], we risk not ensuring that the sentence representation captures all relevant information from the input. In this setup, the masked tokens would be contextualized by all other tokens in another language, rather than focusing on the sentence representation, as is done in MEXMA. While DAP employs a method akin to your suggestion and performs quite effectively, our results across several tasks indicate that our approach more effectively enforces the sentence representation to encode the necessary input information.
• Language specific results
  1. We add the results in appendix E.1.

Thank you for your comments on the paper, and for pointing out some relevant papers. We added them in the related work section.

The first paper “Wei et al., 2021. On learning universal representations across languages”, as Mexma, combines masked language modeling with an alignment loss. However there are several key differences:

• (Wei et al., 2021) uses a contrastive loss to align sentences. Although it is the most common alignment loss in Cross Lingual Sentence Encoders (CLSE), using it effectively requires large batch sizes or careful hard negatives [1,3,7]. The usage of non-contrastive losses makes training easier [3,4,5,8], since they do not have those restrictions, however they are prone to collapse (producing the same output regardless of the input). Using a Masked Language Modeling (MLM) loss on the tokens prevents this collapse, so we use a non-contrastive loss, making it less reliant on large batch sizes or hard negatives and easier to train. Although (Wei et al., 2021) also has a MLM loss, they still rely on the contrastive loss to create the sentence alignment.
• In (Wei et al., 2021) the sentence representations that are used for alignment are based on masked inputs, which creates a mismatch between training and inference, since at inference time tokens are not masked. This mismatch degrades inference time performance, as shown in ELECTRA [6]. In MEXMA the alignment between sentences is done between two sentence representations coming from an unmasked input, which eliminates this mismatch. Table 11 in our paper shows the improvements we get from using the symmetrical architecture without mismatch, compared to an alignment with masked representations. We get an improvement of 0.72% xsim++ going from aligning a masked to a non-masked sentence representation (which should already be better than aligning two masked representations) to a non-masked to non-masked alignment.
• The unmasking task in (Wei et al., 2021) is used to keep the token quality similar to the one in the backbone pre-training. However, in MEXMA the unmasking task is used as a way to improve the quality of the sentence representation, by using the sentence in a different language as the context for token unmasking. The improvement brought by this change is now presented in appendix B.2 for completeness.
• (Wei et al., 2021) uses a contrastive loss to approximate the sentence representation to the token representations (and vice versa). Although this does promote token alignment across languages, it also imposes a hard restriction on the sentence representation. In MEXMA, we instead use the unmasking head based on the sentence representation in another language to align the tokens across languages by transitivity, without introducing hard constraints in the sentence representation. This allows the sentence representation to take the best values necessary for the alignment and MLM losses.

More generally, the key difference is that the concept in MEXMA is using the sentence representation in one language to allow the unmasking of tokens in another language, which forces the sentence representation to encode all of the information in the input, and enable representations to be more language independent, and aligned. This concept is not present in either of the mentioned approaches, and is key to our approach.

Furthermore, a direct comparison is challenging due to the absence of a public model checkpoint or open-source code. However, we can still provide a comparison based on the results reported for the Tatoeba mining task in XTREME, alongside the results from LaBSE. LaBSE reports an accuracy of 83.7%, while (Wei et al., 2021) reports 69.1%. Considering that our approach has consistently outperformed LaBSE across all tested mining tasks, and noting the significant difference between the results of (Wei et al. , 2021) and LaBSE, it is reasonable to anticipate that our method also surpasses (Wei et al., 2021). To facilitate replication and future comparisons, we will open source our training code and release a model checkpoint.

The second mentioned paper (Fan et al., 2022) is also very interesting and related to our work since it combines the usage of masked language modeling with an alignment loss. We thank this suggestion and have added it in our related work. Some key differences:

• The approach uses information specific to the sentiment analysis task, which is very interesting, although it is orthogonal to our approach, since our goal is to create task agnostic representations, and keep our method generic.
• Their model creates representations that excel at the task of sentiment analysis, but introduces constraints that might worsen the performance in other unrelated tasks. We show in our paper that MEXMA performs well on a wide array of tasks.