Q1: The dataset is small, risking overfitting and unreliable evaluation.

A1: Thank you for this question. Our data scale is sufficient for training and validating an adapter‑based multilingual CLIP fine‑tuning model. Specifically, on MSCOCO we sample 5,000 instances per language across ten target languages, yielding 50,000 training samples in total, as for XM3600, the total number of training sample is 28,800, which are adequate for robust evaluation. In addition, CiPCL essentially acts as a data augmentation method (injecting key terms into the original sentence), which increases diversity and is less prone to overfitting. Moreover, as shown in Table 1, we also provide an explicit overfitting check. The performance gap between the training and test sets is consistently ≤ 1.5%, indicating no memorization effects. Meanwhile, we adopt the adapter model to fine-tuning MCLIP, which is a parameter-efficient fine-tuning technique. Adapter does not update the original pre-trained parameters but instead adds LoRa residual layers, thus reducing the risk of overfitting. Therefore, our evaluation is reliable and overfitting is minimal. We will discuss this more clearly in the final version.

Table 1: Performance on training set and test set of MSCOCO$_{36}$

Q2 Token similarity and key term extraction may misinterpret polysemy or ignore multi-word semantics.

A2: Thank you for this question. In TaPCL, we measure cross‑lingual similarity using the language–level overall token overlap. Therefore, several polysemous cases do not dominate the decision. The role of token overlap in cross‑lingual transfer has been emphasized in prior work [1–3], and our empirical results in Figure 1 corroborate this. For CiPCL, as shown in Table 7 in the appendix, we compared multi‑key‑term and single‑key‑term prompts, where multi‑term prompts bring no significant gains while increasing training time, so we adopt the top‑1 scheme, which maintains effectiveness while reducing overhead. Moreover, we believe that a single key term can still serve as a strong semantic anchor (e.g., artificial → “人工” (Chinese) or “artificielle” (French)). Therefore, our similarity metric and term‑selection strategy are reasonable and effective in practice. We will discuss this point more clearly in the final version.

[1] Telmo Pires, Eva Schlinger, and Dan Garrette. "How Multilingual is Multilingual BERT?" ACL 2019.

[2] Vaidehi Patil, Partha Talukdar, and Sunita Sarawagi. "Overlap-based Vocabulary Generation Improves Cross-lingual Transfer Among Related Languages." ACL 2022.

[3] Tomasz Limisiewicz, Jiří Balhar, and David Mareček. "Tokenization Impacts Multilingual Language Modeling: Assessing Vocabulary Allocation and Overlap Across Languages." ACL Findings 2023.

Q3: Are batch sizes consistent across experiments?

A3: Thank you for this question. To ensure a fair comparison, all experiments, including all the baselines and our variants, use the same batch size of 128. Similar results can also be observed based on other batch sizes (e.g., 64 and 256). We will discuss this more clearly in the final version.

Q4: Why are initial losses in Figures 4 and 5 inconsistent?

A4: Thank you for this question. The initial loss difference is caused by the sampling strategies differ across methods; thus, the first mini‑batch contains different samples, leading to different starting losses in Figures 4 and 5, but their initial accuracy is the same. By comparing the early‑stage slopes of the loss curves across methods, we can assess convergence speed. From Figures 4 and 5, our approach shows a steeper initial slope, indicating higher convergence efficiency. Moreover, we will add accuracy‑vs‑iteration curves to further validate the effectiveness of the proposed approach in the final version. We will discuss this point more clearly in the final version.

Q5: Generalization ability for other tasks.

A5: Thank you for this question. Our study focuses on the fine‑tuning mechanisms of multilingual CLIP. As multilingual cross‑modal retrieval is the primary application scenario for multilingual CLIP, we organize our main experiments on this task. To further assess the scalability of the proposed approach, Appendix A.3 reports results on the multilingual natural language inference task. We find that, on this task as well, our method achieves substantial training‑efficiency gains without loss in accuracy. These results verify the scalability of our approach. We will discuss this more clearly in the final version. We will discuss this more clearly in the final version.

Dear Reviewer,

 We sincerely appreciate your time and effort in reviewing our manuscript and offering valuable suggestions. Since the discussion DDL comes closer, we would like to confirm whether our responses have effectively addressed your concern. If you require further clarification, please do not hesitate to contact us. We are more than willing to continue the discussion with you. 

Warm regards,

Authors

Q1 Why train a single multilingual model instead of multiple independent monolingual models?

A1: Thank you for this question. Compared with monolingual models, a single multilingual model offers advantages in training efficiency, inference efficiency, and operational simplicity, and therefore has broad real‑world applicability. Specifically, a multilingual model leverages cross‑lingual similarity to enable knowledge transfer across languages, which reduces training cost and improves performance on low‑resource languages. In multimodal retrieval, queries are naturally multilingual or even mixed‑language inputs (e.g., social‑media search). Additional language‑identification and routing stages would be required when deploying separate monolingual models, which struggle with mixed‑language inputs and introduce extra latency and failure modes. Moreover, language‑specific traffic fluctuates over time; maintaining many monolingual models complicates capacity planning and load balancing. A single multilingual model avoids these issues while providing consistent behavior across languages, making it the more practical choice in many real‑world scenes. We will more clearly clarify this in the final version.

Q2 Can the proposed methods be applied to other tasks?

A2: Thank you for this question. Our study focuses on the fine‑tuning mechanisms of multilingual CLIP. As multilingual cross‑modal retrieval is the primary application scenario for multilingual CLIP, we organize our main experiments on this task. To further assess the scalability of the proposed approach, Appendix A.3 reports results on the multilingual natural language inference task. We find that, on this task as well, our method achieves substantial training‑efficiency gains without loss in accuracy. These results verify the scalability of our approach. We will discuss this more clearly in the final version.

Q3 The practical impact of improving training efficiency.

A3: Thank you for this question. Our method reduces training time by 30%–50% while achieving the same accuracy as standard multilingual‑CLIP fine‑tuning methods (PCL‑base). This gain is proportional rather than a fixed offset. In real‑world settings, fine‑tuning datasets are often very large—for example, the full WIT dataset contains about 170M training samples, and a single fine‑tuning run with PCL‑base on 4 GPUs takes about 600 hours. Under the same setup, our approach could significantly cut training time by 180 to 300 hours. Moreover, in real‑world scenes, models are updated frequently as new data and new businesses emerge; thus the training‑efficiency improvements delivered by our approach have concrete practical value. We will discuss this point more clearly in the final version.

Q1: Reproducibility & statistical reliability.

A1: Thank you for this question. There may be a misunderstanding on our main focus. As stated in the Introduction, this work primarily targets the fine‑tuning efficiency of multilingual CLIP models rather than maximizing performance. As stated in the Introduction, directly incorporating parallel corpora for fine-tuning will cause substantial computational overhead, hindering the flexible deployment of MCLIP models for downstream applications; thus, it is necessary to improve fine-tuning efficiency. PCL‑base can be regarded as an upper bound on multilingual performance as it uses the full parallel corpus; therefore, our goal is to improve training efficiency while achieving comparable performance with PCL‑base. Regarding random seeds, all experiments in the paper used a fixed, randomly chosen seed (42). To more reliably validate our results, we additionally ran experiments with three seeds (i.e., 43, 44, and 45); the results are reported in Table 1 and Table2. The outcomes are consistent across different seeds, showing performance comparable to PCL‑base along with >30% improvements in training efficiency, which confirms the robustness of our method and the reliability of our conclusions. We will discuss this more clearly in the final version.

Table 1: Performace on MSCOCO$_{36}$ with different seeds

Table 2: Performace on XM3600 with different seeds

Q2: Why weak Otk correlations slow convergence but leave final accuracy essentially unchanged?

A2: Thank you for this question. As stated in our rebuttal to the above question, our objective is to improve fine‑tuning efficiency, rather than maximize accuracy. When parallel corpora are used, different sampling strategies will, given sufficient iterations, converge towards the same upper‑bound accuracy (comparable to PCL‑base); the difference is how quickly they reach it. Sampling based on other similarity measures leverages source‑language transferability less effectively and thus has lower training efficiency than Otk. Moreover, as shown in Table 3, when training is stopped after the same iteration number, the variants using other sampling methods underperform Otk. We will discuss this in more detail in the final version.

Table 3: Ablation study of TaPCL under the same number of iterations

Q3: No mention of the broader impact statement, which is clearly a part of the author guidelines.

A3: Thank you for this question. As required by the author guidelines, we explained in the checklist why this work does not include a broader impact statement. This work does not present any dataset or benchmark. It proposes strategies to enhance the efficiency of parallel corpus-based fine-tuning for downstream tasks. Therefore, there is no potential societal harm associated with this work. We will more clearly clarify this in the final version.

Q1: Are the improvements from token similarity, or from uneven base‑model language quality?

A1: Thank you for the question. In this work, we selected 9 target languages with comparable zero‑shot performance to more clearly isolate the effect of token similarity on transferability. This design effectively removes the confounding influence of base‑model pretraining quality on transfer. The zero‑shot results for these 9 target languages are reported in Table 1, which shows that the base model has similar zero-shot capability on those target languages. Moreover, our conclusions also hold for low‑resource languages: following your suggestion, we evaluated the Source-only method (fine‑tuned only on the source language) on several low‑resource languages from [1], with results in Table 2. We observe that token overlap still has a significant effect on performance gains—languages with higher token overlap achieve larger improvements. We will make this more explicit in the final version.

[1] Pratik Joshi, Sebastin Santy, et al. "The state and fate of linguistic diversity and inclusion in the NLP world." ACL 2020.

Table 1: Zeroshot performance on 9 target languages

Table 2: Performance of low digital resource languages

Q2: When to use TaPCL vs. CiPCL?

A2: Thank you for the question. TaPCL and CiPCL can be chosen as needed. Empirically, TaPCL achieves slightly higher accuracy than CiPCL (+0.24% on MSCOCO, +0.11% on XM3600), but requires more fine‑tuning time (+46% on MSCOCO, +39% on XM3600). We therefore recommend selecting between them by trading off accuracy and training time for the target use case. We will make this guidance explicit in the final version.

Q3: Scalability and dependence on computing token similarities and translation model.

A3: Thank you for the question. As target-language training samples are hard to collect, the target-language training data in MSCOCO and XM3600 are obtained by translating source‑language datasets. Under this setup, the use of a translation model aligns with the existing pipeline. As for token similarity computation, since it only uses input-layer token embeddings and is performed once as a pre-training step, computing token similarities for the entire dataset takes only a few seconds and thus does not affect scalability. Therefore, these two designs do not limit the scalability of our approach. We will discuss this more clearly in the final version.

Q4: Key‑term selection in Eq. 8 may be simplistic.

A4: Thank you for the question. Since the essence of retrieval is to match images and texts by computing the similarity between their cls token features, considering the attention weights from other tokens to the cls token alone can effectively reflect the importance of different tokens during keyword selection. When additionally considering the memory stored in the Key layer of the feed-forward network (FFN) for identifying input patterns [1], the performance improvement is limited. Specifically, according to the guidance in [1] for knowledge retrieval, the key most relevant to the cls token can be regarded as the one that best identifies its pattern. We compute the relevance between other tokens and this key, and use this value together with the attention weights to guide keyword selection. However, the experimental performance did not improve as expected and instead showed a slight decline, as shown in Table 3. We will provide a more detailed discussion of this in the final version.

[1] Mor Geva, Roei Schuster, Jonathan Berant, and Omer Levy. "Transformer feed-forward layers are key-value memories." EMNLP 2021.

Table 3: Performance of CiPCL with additional factor