MERIT: Multilingual Semantic Retrieval with Interleaved Multi-Condition Query

We sincerely thank Reviewer 6FnV for the thoughtful and constructive feedback. We are especially encouraged by your recognition of the real-world relevance of our problem setting, the rigor and scale of the MERIT dataset, and the effectiveness of our CORAL framework. Below, we address your comments point by point:

---

Q1: "The product data is sourced from a private, 'internal dataset'. While the authors have anonymized the data, the lack of information about the original source (e.g., a single e-commerce platform vs. multiple) makes it difficult to reason about potential inherent biases in the product distribution, metadata structure, or visual style."

A1: Thank you for your question. Our original data is sourced from a single e-commerce platform. We have provided distribution statistics in Appendix Section D and data examples in Appendix Section E. Due to the double-blind reviewing process, we cannot disclose additional data information at this time. However, we have committed to providing more comprehensive data details and fully open-sourcing our data and code (both annotation and training components).

---

Q2: "The focused modality (text, image) and scenario (product search) are not accurately reflected in the title (semantic retrieval is too broad)."

A2: Thank you for your suggestion. We will consider revising the title of our paper in future iterations, as you know, modifications are not permitted during the rebuttal phase. We will consider incorporating keywords related to modality and scenario.

---

Q3: "Some experiment settings/details are not clear."

• Q3.1: "In Figure 2, the query condition 'same selling method' is ambiguous. Could you clarify this example?"

• Q3.2: "How does model performance vary with the number of conditions in a query (e.g., two vs. three vs. four)? The main results do not show this breakdown."

• Q3.3: "For the out-of-distribution results in Figure 6(b), could you clarify the setting? Was the Qwen2.5-VL trained on a subset of MERIT? "

• Q3.4: "For clarity, consider making the introduction of the CORAL framework (Line 241) a new subsection separate from 'Preliminaries'."

• Q3.5: "In Eq. 5, the CORAL loss function includes two hyperparameters, λ1 and λ2, to balance the reconstruction losses. Could you elaborate on how these weights were selected and their sensitivity?"

A3: Thank you for your question! We apologize for any unclear details in the manuscript. Here, we will explain them point by point.

• A 3.1: We inadvertently reversed the plotting of product1 and product2 in the figure. "Same selling method" refers to the bundled selling approach where 3 items are sold together as a package. We have revised this mirror in the revised manuscript.

• A 3.2: We thank you for this valuable suggestion. Due to time constraints, we have supplemented Table C in the supplementary material with results showing the performance of several key models as a function of the number of query conditions. As demonstrated in Table C below, we observe that the performance of nearly all models degrades with increasing condition count. Notably, under the most complex scenario with 4 conditions, even the best-performing model achieves only 36.84 R@5, highlighting the inherent difficulty posed by the combinatorial complexity of multi-condition queries.

Table C. Impact of query condition count on model performance on MERIT

• A 3.3: In Figure 6(b), we evaluate three types of OOD scenarios. You can find the specific experimental details and more comprehensive result tables (please refer to Tables 7, 8, and 9) in Appendix Section E.4. In brief, "OOD" refers to removing the same language, class, or attribute from the training set during testing, while "mixed" refers to training on the complete dataset.

• A 3.4: Thank you for your feedback. We have committed to incorporating your suggestions in the revised version of the manuscript.

• A 3.5: We conduct ablation studies on the mask ratio, temperature (τ), loss weights (λ), and the decoder, as shown in Table D. Overall, CORAL demonstrates robust training dynamics and exhibits low sensitivity to variations in temperature and the number of decoder heads.

Regarding the selection of loss weights, we first establish through ablation experiments in Table 3 that joint reconstruction of language and vision modalities is more effective than reconstructing individual modalities. Therefore, in the ablation experiments presented in Table D below, we maintain equal weights for both modalities. Our findings indicate that smaller values of λ have minimal impact on the results. However, overly high loss weights can lead the model to overemphasize the reconstruction objective, consequently degrading retrieval performance.

Table D. Further ablation study for CORAL on MERIT using Qwen2.5-VL-3B

---

Last but not least, we would like to sincerely thank Reviewer 6FnV again for the valuable time and constructive feedback provided during this review.

We sincerely thank Reviewer WJjZ for the thoughtful and constructive feedback. We are encouraged that you found our benchmark MERIT to be a novel and significant contribution, especially in highlighting multilingual and multimodal product retrieval, and appreciated the effectiveness and breadth of CORAL’s empirical evaluations. Below, we address each concern point by point.

---

Q1: "What precisely do the authors mean by 'interleaved' and why do existing benchmarks fail to meet this property. I would be willing to raise my evaluation score if the authors can provide a clear definition of interleaved and explain why this is a novel property of their dataset."

A1: We provide a point-by-point clarification of what constitutes interleaved:

• (1.1) Definition: In MLLMs, "interleaved" refers to the model's capability to simultaneously process alternating sequences of images and text.

  • This capability enables more flexible understanding and generation of multimodal content, analogous to human cognitive processes [Dong et al., 2024].
  • The typical sequence structure follows: [Text₁] → [Image₁] → [Text₂] → [Image₂] → ...
  • Post-2024 MLLMs have predominantly adopted interleaved sequence processing, as this better reflects real-world usage patterns where users frequently present queries containing multiple distributed images.

• (1.2) Our Contribution: Existing retrieval datasets exclusively consider single images or singular retrieval conditions, namely using an image to retrieve text, or using an image-text pair to retrieve an image-text pair. Our work presents the first interleaved retrieval dataset. As demonstrated in Figure 2 in the manuscript, our query format follows: [Text₁] → [Product Image₁] → [Product Title₁] → [Product Image₂] → [Product Title₂] → ..., representing a fundamental departure from previous dataset formulations.

• (1.3) Practical Significance: Interleaved data enables the construction of multi-condition, cross-modal retrieval tasks, allowing for more precise interpretation of user intent. Moreover, the interleaved format aligns with the input structure of MLLMs, which uses interleaved input data[Alayrac et al., 2022], enhancing their semantic representation capabilities and thereby advancing progress in multimodal learning [Zhu et al., 2024] .

We have clarified the definition in the revised manuscript and better emphasized its novelty.

References:

• [Dong et al., 2024] "Internlm-xcomposer2: Mastering free-form text-image composition and comprehension in vision-language large model." arXiv:2401.16420, 2024.
• [Zhu et al., 2024] "Multimodal c4: An open, billion-scale corpus of images interleaved with text." NeurIPS, 2023.
• [Alayrac et al., 2022] "Flamingo: A visual language model for few-shot learning." NeurIPS, 2022.

---

Q2: "How were the products sourced for the MERIT dataset? What precisely is the role of GPT-4o in the dataset construction process? How do the authors control for potential errors from using GPT-4o in the dataset construction process? I would be willing to raise my evaluation score if the authors can provide answers to these questions and, in general, provide more details on how the dataset was sourced (including perhaps a file with data examples)."

A2: We apologize for any unclear aspects in our paper and will address your concerns point by point.

• (2.1) As described in Appendix Section A.1, our data is sourced from our internal dataset. Specifically, we curated popular products across six Southeast Asian countries. Due to the double-blind review process, we cannot provide further details at this time. However, we commit to releasing more comprehensive details once the paper is accepted.

• (2.2) Overall, GPT was involved in attribute selection, product annotation, query generation, and filtering processes, as indicated by the robot icons in Figure 2 in the manuscript. We take annotation accuracy very seriously. Following each GPT annotation step, we implemented GPT self-verification procedures. For critical steps (attribute identification and query generation), we employed manual filtering (10,000 labor hours in total) to ensure quality. We provide an 11-page detailed description of our annotation process in Appendix Section A, which thoroughly explains the usage of GPT in our pipeline.

• (2.3) Regarding detailed data information, we provide distribution statistics in Appendix Section D and data examples in Appendix Section E. Due to rebuttal limitations preventing file uploads and the rule stating "All the texts you post (rebuttal, discussion) should not contain any links to external pages", we cannot upload the requested folders containing data examples and sources. However, we commit to fully open-sourcing our data and code (both annotation and training) following the completion of the review process.

We acknowledge these unclear aspects you have identified and commit to addressing them comprehensively in the revised version of our paper.

---

Q3: "How were the expert annotators recruited for this project, and how were they compensated? What is the license of the MERIT dataset?"

A3: Thanks for your question.

• (3.1) Our annotators are professional in-house annotation staff with extensive labeling experience and bachelor's degrees or higher. The specific compensation is determined by our institution's policy, with an estimated total exceeding $10,000 USD. We ensure that annotators receive fair compensation, maintain appropriate workloads, and prioritize their physical and mental well-being.

• (3.2) Quality control was strictly enforced during annotation. All annotators underwent training and a qualification phase (including trial annotations) before formal participation. Additionally, we employed dedicated quality-check specialists to conduct periodic random inspections.

• (3.3) Our licensing follows CC BY-NC 4.0, which we have explicitly specified in the revised version of the manuscript.

---

Q4: "Are there any works in the multimodal contrastive learning literature that are related to CORAL? What distinguishes CORAL from these prior works? I would be willing to raise my evaluation score if the authors could include a deeper discussion of related work around the CORAL model (particularly in contrastive learning). Currently, I feel that this important positioning of CORAL in relation to the literature is lacking in the paper."

A4: We thank you for your insightful question and will address each point systematically:

• (4.1) While we acknowledge that the literature on multimodal contrastive learning is indeed extensive, our method specifically targets MLLM-based retrieval approaches, focusing on training algorithms for transitioning from MLLMs to multimodal retrieval models. Therefore, our related work section concentrates on discussing MLLMs and Multimodal Retrieval Models as most relevant to our contribution.

• (4.2) Through our experimental analysis, we identified key limitations in existing MLLM-based retrieval methods: over-reliance on global semantic information while neglecting conditional query elements, failing to extract specific attributes, and misinterpreting visual content. To address these challenges, we propose CORAL, which introduces embedding reconstruction on top of conventional contrastive learning to preserve fine-grained conditional elements rather than solely supervising the [EOS] token, as detailed in lines 100-110 of our manuscript.

• (4.3) Moreover, to the best of our knowledge, we are the first to introduce multimodal embedding reconstruction in MLLM-based retrieval. The results on MERIT and 12 standard benchmarks also demonstrate the effectiveness of our approach.

We greatly appreciate your valuable feedback and have committed to incorporating more comprehensive discussions of contrastive learning-related work in the revised version of our manuscript.

---

Last but not least, we would like to sincerely thank Reviewer WJjZ again for the valuable time and constructive feedback provided during this review.

We sincerely thank Reviewer MHMi for the thoughtful and constructive feedback. We are glad that you found our benchmark timely and realistic, CORAL conceptually simple yet effective, and our experiments strong and comprehensive. Below, we address your questions and concerns in detail.

---

Q1: "The section of 'Masked Embedding Reconstruction' is poorly written and needs to be reorganized..."

A1: Thanks a lot for your suggestion. We have addressed the mentioned typos and improved the elaboration in the manuscript. We commit to making detailed revisions to these unclear aspects to improve readability. Additionally, we commit to open-sourcing the code and our trained models for public use.

---

Q2: "Lack the analysis of reconstruction mask ratio δ, choice of decoders, and temperature."

Q3: "How sensitive is CORAL to the 50 % visual/text masking ratio? Could adaptive masking yield further gains?"

A2 & A3: Thank you for your comment. We conduct ablation studies on the mask ratio, temperature (τ), loss weights (λ), and the decoder, as shown in Table D below:

Table D. Further ablation study for CORAL on MERIT using Qwen2.5-VL-3B

Key findings:

• Overall, CORAL demonstrates robust training dynamics and exhibits low sensitivity to variations in temperature and the number of decoder heads.
• However, we observe that excessively high mask ratios can be detrimental to performance. This phenomenon likely stems from the model's diminished capacity to leverage existing knowledge for embedding reconstruction [Xiao et al., 2020]. Specifically, when the mask ratio is increased to 0.75, we observe a substantial 10.1% decrease in MRR compared to the 0.5 setting.
• Furthermore, overly high loss weights can lead the model to overemphasize the reconstruction objective, consequently degrading retrieval performance.

Reference:

• [Xiao et al., 2020] "RetroMAE: Pre-Training Retrieval-oriented Language Models Via Masked Auto-Encoder." EMNLP, 2022.

---

Q4: "The metric relies solely on recall/MRR. User-centric metrics (CTR, dwell time) or ranking quality are absent."

A4: Thank you for your question. We address this concern from the following aspects:

• (4.1) First, it is important to clarify that our task falls within the domain of semantic retrieval based on multi-modal content comprehension rather than recommendation systems. Therefore, the standard evaluation metrics are recall and MRR, and our related work also does not involve user-centric metrics such as CTR.
• (4.2) Nevertheless, we have supplemented our evaluation with human preference experiments. Our human preference study is conducted in a multiple-choice comparison format. Specifically, we randomly sampled 100 items from MERIT, with each model providing their top-1 (Choice@1) or top-3 (Choice@3) most similar items as options. Five annotators were asked to select the best option (when the best item was retrieved by multiple models, we credited the model with the highest similarity score during recall). We report the average selection rate. As shown in Table B, CORAL demonstrates superior performance in the human preference study, particularly achieving 58.9% selection rate in the Choice@3 setting.

Table B. Human preference study on MERIT

---

Q5: "How many GPU-hours did full CORAL training require for the 7B model, and what is the inference latency compared to baseline contrastive tuning?"

A5: Thank you for asking.

• (5.1) We first clarify that our paper presents results for the 3B model rather than the 7B model. Under full-parameter training conditions using 8×H100 GPUs with batch size 64 for the LLM backbone, one epoch requires approximately 5 hours for the 3B model and 6.5 hours for the 7B model.
• (5.2) Since our approach only introduces an additional decoder during training compared to the baseline, our method incurs no additional inference latency relative to the baseline model.

---

Q6: "Have you measured language- or category-specific biases in the retrieval results beyond the small sample in Fig. 7? Any mitigation plans?"

A6: Thank you for your insightful question. We measure the biases as follows:

• (6.1) In Figure 5(a), we evaluate language bias and observe that models trained with multilingual data do not exhibit significant language bias.
• (6.2) As demonstrated in Table 7 of the supplemental material, Qwen2.5-VL exhibits biased zero-shot performance across different languages and classes, though the magnitude of bias remains relatively modest. This bias likely stems from the distributional characteristics of the pretraining data.
• (6.3) Comparing Table 7 and Table 9 reveals that jointly training across multiple classes and languages improves performance on individual classes and languages. This suggests that increasing data diversity can effectively mitigate such biases.

Table 7. Out-of-distribution test results for Qwen2.5-VL trained on in-domain data

Table 9. Out-of-distribution test results for Qwen2.5-VL trained on the whole train set

---

Q7: "The dataset focuses on consumer goods; tasks like scene or document retrieval are out of scope."

A7: Thanks for your comment.

• (7.1) Interleaved multi-condition retrieval represents a novel and inherently challenging task that has not been previously explored in the literature due to the complexity of data collection. Our proposed MERIT constitutes the first dataset specifically designed for this task formulation, as collecting multi-condition retrieval data for document retrieval presents significant practical difficulties.
• (7.2) Meanwhile, product retrieval serves a representative and tractable domain for multi-condition retrieval scenarios and holds substantial real-world significance, which motivated our domain selection.
• (7.3) We acknowledge that extending interleaved multi-condition retrieval to additional domains beyond product search represents an important direction for future work, which we defer to subsequent investigations.

---

Q8: "CORAL needs access to image embeddings during decoding; models without token-level vision hooks may not benefit."

A8: We appreciate your insightful question.

• (8.1) MLLM-based models demonstrate superior performance in addressing semantic retrieval challenges [Jiang et al., 2024], and CORAL is specifically designed as a methodology for MLLM-based retrievers.
• (8.2) Current mainstream MLLMs are equipped with token-level vision hooks, including but not limited to GPT, InternVL, Qwen, Phi-V, LLaVa, and other prominent models in this domain. Thus, our method can benefit most MLLM-based retrievers.

Reference:

• [Jiang et al., 2024] "VLM2Vec: Training Vision-Language Models for Massive Multimodal Embedding Tasks." ICLR, 2025.

---

Last but not least, we would like to sincerely thank Reviewer MHMi again for the valuable time and constructive feedback provided during this review.

We sincerely thank Reviewer YCpi for the thoughtful and constructive feedback. We appreciate your recognition of our novel MERIT dataset and the CORAL framework for improving multi-condition semantic retrieval. Below, we address your concerns point by point.

---

Q1: "While the dataset is multilingual and sizable, it seems to be domain/business specific (e-commerce in Southeast Asia), limiting its generalizability as a standard benchmark for semantic retrieval."

A1: Thank you for this important point. We respond as follows:

• (1.1) Interleaved multi-condition retrieval represents a novel and inherently challenging task that has not been previously explored in literature due to the complexity of data collection. Our proposed MERIT constitutes the first dataset specifically designed for this task formulation. Product retrieval represents a canonical multi-condition retrieval scenario; thus, we chose to approach this problem from the product retrieval domain.

• (1.2) Southeast Asia constitutes an active multilingual and culturally diverse market, whose products exhibit representative characteristics.

• (1.3) We acknowledge that extending interleaved multi-condition retrieval to additional domains beyond product search represents an important direction for future work, which we defer to subsequent investigations. Additionally, based on your suggestion, we have further refined the elaboration in the manuscript.

---

Q2: "The authors demonstrated that existing methods overall have poor performance on MERIT dataset in Table 2. The metrics however, might be biased since some models are finetuned on the e-commerce scenario more than others, like GME-QwenVL2 vs pretrained Qwen2.5-VL..."

A2: Thank you for your observation. We clarify as follows:

• (2.1) The purpose of Table 2 is to measure the zero-shot capability of existing models on the interleaved multi-condition semantic retrieval task. Most existing models are trained on proprietary datasets with inherent distributional differences, making it impossible to guarantee complete consistency across training data.

• (2.2) For our method, based on Qwen2.5-VL, we conducted comprehensive ablation studies as shown in Table 3, where we ensure that the fine-tuning data is completely consistent. As shown in Table 3, our method achieves a 45.9% improvement in R@1 over traditional contrastive learning methods, which we believe provides a fair and convincing validation of our approach’s effectiveness.

If you have any remaining concerns about our experimental setup, please let us know, and we would be happy to discuss them further.

---

Q3: "The authors benchmarked and evaluated CORAL/MERIT as a symbiotic duo… it would be nice to isolate and demonstrate CORAL’s effect on other benchmarks as well."

A3: We appreciate this suggestion. We have now updated results on 4 new standard benchmarks and provided more comprehensive comparisons on the 9 standard benchmarks shown in Figure 9, following the same experimental setting as depicted in Figure 9, as presented in the Table A below:

Table A. Comparisons with other methods across 12 standard retrieval tasks

Experimental results demonstrate that our method (Qwen2-VL-2B+CORAL) achieves consistent improvements across these 12 standard retrieval tasks, with particularly notable performance on VisDial, where our approach exhibits a 181% enhancement over the contrastive learning baseline. Additionally, our method shows advantages compared to other foundational models, such as CLIP and E5-V, with especially significant improvements on CIRR, where we achieve a 275% enhancement over OpenCLIP.

We commit to updating our revised manuscript with the new experimental results and detailed experimental configurations. We thank you once again for your valuable feedback.

---

Q4: "The core diff of CORAL is the introduction of BERT-like reconstruction losses... This seems similar to CLIP. I'm wondering why this new task helped."

A4: Thank you for your insightful question. We provide the following detailed responses:

• (4.1) Recognizing that neglecting specific conditional elements in queries constitutes a primary source of error as highlighted in Section 3.4, it naturally follows that reconstructing these elements could prove beneficial. Moreover, reconstruction can enhance task complexity to improve data utilization efficiency [Xiao et al., 2020]. Therefore, we propose CORAL to enhance MLLM-based retriever performance in addressing interleaved multi-condition semantic retrieval tasks through the integration of visual reconstruction during the fine-tuning process of the MLLM-to-retrieval model adaptation.

• (4.2) Simultaneously, we emphasize that our reconstruction approach is symmetric. When performing visual reconstruction, we maintain text visibility with the intention that textual information can assist in visual recovery, thereby enhancing cross-modal understanding capabilities. As demonstrated in the ablation study in Table 3, reconstructing only vision or language modalities in isolation yields suboptimal performance.

---

Reference:

• [Xiao et al., 2020] "RetroMAE: Pre-Training Retrieval-oriented Language Models Via Masked Auto-Encoder." EMNLP, 2022.

---

Last but not least, we would like to sincerely thank Reviewer YCpi again for the valuable time and constructive feedback provided during this review.