Thank you for your insightful comments. We have edited the paper to fix the typos and acknowledge the work you suggested. We have also updated the paper to cover two of the points you mentioned as below:

[Minor]: the work could be stronger if this could also be extended to larger models (e.g., >1-5B) and the discussion of applicability on more architectures (enc-only, enc-dec and dec-only) could be more details.

Thank you for the suggestion, we have conducted more experiments with different sizes of the MT model used for constrained decoding in Codec and update the results in the Appendix C.3 of the paper.

[Minor]: the paper could also be improved with more comparison against zero-shot results from larger multilingual LLMs.

In our original submission, we fine-tune mDebertaV3-base model on English data (FT_En) as a zero-shot baseline for the Cross-lingual NER task. Inspired by your comments, we have added two multilingual pre-trained models, which are more competitive for African languages, for the zero-shot baseline: AfroXLMR-large [1], a model that is adapted to 17 African languages, and Glot500 [2], a model which is pre-trained on 500+ languages including African languages. Below are results of models on the test set of MasakhaNER2.0. We have updated these results in the Appendix C.1 of the paper.

[1] Alabi et al. Adapting pre-trained language models to African languages via multilingual adaptive fine-tuning. COLING 2022.

[2] ImaniGooghari et al. “Glot500: Scaling multilingual corpora and language models to 500 languages”. ACL 2023.

Thank you for your insightful feedback. Please find below our response to each point of your comments and some updates in the paper based on your suggestions.

The approach relies on an external MT system... It would be nice if there can be an analysis on the influence of translation quality.

Thank you for your suggestion, we have conducted additional analysis about this topic and updated the paper with the new analysis in Appendix C.3. The update about this analysis is as follows. There are two places where MT systems are used in Codec: one is for generating the translation template, and the other one is for decoding.

• We analyzed the impact of using different sizes of NLLB (600M, 1.3B, and 3.3B) to generate the translation templates, while fixed on NLLB-600M as the MT system for constrained decoding. We evaluate the methods in the translate-test setting on the MasakhaNER2.0 dataset, over 18 languages. Overall, the average F1 scores of methods using NLLB-600M, NLLB-1.3B and NLLB-3.3B as the template-translator are 68.6, 70.6 and 70.4 respectively. We observe the performance improves the most when changing from NLLB-600M to 1.3B, where NLLB’s translation quality also improves the most [1].
• Given the same translation templates (i.e., using NLLB-3.3B as the template-generator), we explored using Codec with the three model sizes of NLLB as the constrained decoding module. The average F1 score of methods using NLLB-600M, NLLB-1.3B, and NLLB-3.3B are 70.4, 70.1, and 70.1, respectively.

[1] NLLB Team. No Language Left Behind: Scaling Human-Centered Machine Translation. https://arxiv.org/abs/2207.04672

In some cases, the proposed method does not perform well, for example, the NER results are worse for Chichewa and Kiswahili, and the EAE results seem close to the baselines. It would be much better if there can be more analysis on why these happen to provide some guidance on how to select the label-projection methods for a new language.

For the case of Chichewa and Kiswahili, we observe that, for a large portion of the data, the entities in the target language are exactly the same as their English form, therefore, the multilingual model fine-tuned only on English data does extremely well in these two languages , and outperforms all label projection methods. We have added a discussion about this in the Experiments section.

I’m wondering whether it would still be effective to replace the searching algorithm with some simpler alternatives, such as greedy pruning...

We have implemented a simpler search algorithm, we dub it as Constrained-Space Beam Search (CSBS). In particular, this algorithm behaves similarly to the vanilla beam search, but we constrain the search space of it to contain only valid hypotheses - following the translation template and having the correct number of markers. Similar to Codec, CSBS also returns up to 5 best hypotheses (equal to the beam size when the beam size is smaller than 5), and a re-ranking step is performed to choose the best hypothesis. Below is the performance of CSBS with different beam sizes and Codec on the MasakhaNER2.0 test set. The methods are evaluated in the translate-test setting (see below). We have updated these results in the Appendix E of the paper. Overall, a higher beam size will increase the performance of CSBS, however, even with a beam size of 16, CSBS’s average F1 score still falls behind Codec with a big gap.

Thank you for your insightful feedback. Please find below our response to each point of your comments and some updates in the paper based on your suggestions.

The author say that inserting markers would degrade the translation quality. However, although they provide a translation template to guide the model during translation, the proposed method still relies on markers, which is not completely solving this issue.”

There seems to be some confusion here, our proposed method, Codec, does not include markers during the translation phase; so, the final translated text quality from Codec will be the same as any MT system, which is also the same as the word alignment-based label projection approaches. Only after this translation phase (i.e., after we generated this translation template), the markers are inserted through a second constrained decoding phase, in which the translation is constrained to be the same as the original translation template, while markers are being inserted. We have updated the Introduction of the paper to make this point clearer.

This assumption largely relies the translation model’s ability to handle markers... I suggest the authors to report the results of different translators to show the stability of the proposed method.”

Thank you for your suggestion. We have added experiments with two more different MT systems (M2M-100 and MBart), besides NLLB.

• For the NER task, we explore using M2M-100 instead of NLLB for decoding. M2M-100 is chosen for NER as it supports 6 African languages in the MasakhaNER2.0 dataset. In addition, for this task, we also experiment with different model sizes of NLLB (i.e., 600M, 1.3B, and 3.3B) as the decoding module for Codec. See the table below (under “Codec” column) for results under the translate-test setting. We have also included the results of these analyses in Appendix C of the paper.

(‘-’: these low-resource languages are not supported by the M2M-100 MT system)

• For the EAE task, we have conducted additional experiments using MBart50-large (a competitive MT system for Chinese and Arabic) for both translation template generation and constrained decoding in Codec, as well as other comparison baselines. See table below for results. We have updated the paper (Appendix D.2) to include these results.

Thank you for your insightful feedback. Please find below our response to each point of your comments and some updates in the paper based on your suggestions.

In response to your two main concerns:

The main issue with the work is 1) the lack of recognition of others ... 2) lack of more comprehensive comparisons to more baselines. The main baseline is definitely the EasyProject method and I agree with that, but I feel that not enough care has been provided to optimize the word alignment-based baselines ...

For 1), thank you for pointing this out. There is indeed a long history of research on cross-lingual label projection. We should cite as many papers as possible as we can fit in the page limit. We have updated our Related Work section to include all the papers you mentioned.

For 2), thank you for your suggestion. Our paper focuses on improving the marker-based approach for label projection (especially low-resource languages), thus EasyProject, the best marker-based approach is used as the most important baseline for comparison as you mentioned. In terms of comparison to the word alignment-based approaches, in our original submission, we followed the prior work [1] in choosing Awesome-align (one of the state-of-the-art word alignment models) with multilingual BERT as the baseline. In response to your comments, we have added two more word-alignment baselines that are optimized for African languages, namely Awesome-align with AfroXLMR-large [2], a model that is adapted to 17 African languages, and Awesome-align with Glot500 [3], a model which is pre-trained on 500+ languages including African languages. These are novel and very competitive word alignment-based baselines for the MasakhaNER2.0 dataset.

The table below compares the fine-tuning of English data baseline (FT_En), and Awesome-align with multilingual BERT (result in the main paper), Awesome-align with two aforementioned pre-trained models, and our proposed Codec in the translate-test setting, the results are reported on the test set of MasakhaNER2.0 dataset. To save space, in this table, we also include the experiments of using Codec with different MT models (NLLB and M2M-100) for decoding, which is to respond to your later comments. We have updated the paper (Appendix C.1) to include these results.

(‘-’ marks the low-resource languages that are not supported by the M2M MT system)

[1] Chen et al. Frustratingly easy label projection for cross-lingual transfer. Findings ACL 2023.

[2] Alabi et al. Adapting pre-trained language models to African languages via multilingual adaptive fine-tuning. COLING 2022.

[3] ImaniGooghari et al. “Glot500: Scaling multilingual corpora and language models to 500 languages”. ACL 2023.

In response to your other comments:

The number of evaluation tasks is slightly underwhelming and the paper should extend the scope of tasks ...

Our proposed work primarily targets low-resource languages and span-level NLP tasks, where human-annotated datasets (that are directly annotated on these languages, other than annotating the translated texts from existing English datasets) are in scarcity. MasakhaNER2.0, which covers 18 African languages we can experiment on, is one of the best benchmarks that is available. Besides the NER task, we also did experiments on the ACE 2005 dataset for the EAE task, which is a commonly used benchmark in the label projection literature. We politely argue that having conducted experiments in 20 languages and two tasks (with 3 different MT systems, and 4 competitive baselines -- see our updates) are sufficient to support our claimed contributions, while agreeing with you that it is always nice to include even more experiments if possible.