Model's Efficiency

Thank you for your valuable feedback. We evaluate our approach against existing baselines on three metrics: training time (50 epochs), convergence time to optimal performance, and average inference time per query, as summarized below. All time comparisons are performed on the same device. Two key insights can be drawn from the results:

(1) While the inclusion of the proposed modules slightly increases inference time, this overhead is acceptable given the significant performance improvement (as the CCL loss is not computed during inference).

(2) Adding TSA, CCL, and MIO components significantly accelerates convergence to optimal performance during training. This is attributed to the strong guidance provided by TSA and CCL in the early stages of training, facilitating efficient fine-tuning and deployment in new scenarios.

Performance Across Domains and Extended Interactions

To validate the effectiveness and generalization capability of the proposed model in more generalized settings, we conduct performance comparisons across three scenarios:

(1) CIRR. Please refer to the Global Author Rebuttal for additional experiments, we evaluate performance on the single-turn composed retrieval dataset CIRR, which involves general objects and scenes.
(2) CIRR-MT. To assess multi-turn composed retrieval in the general domain, we adapt the CIRR dataset using our proposed Modification Generation Framework (MGF) to create CIRR-MT. This dataset contains 30,000 transactions (larger than existing multi-turn datasets) with varying turn numbers (3-turn, 6-turn, and 8-turn). The results are summarized in the table below.

(3) Product-MT. To evaluate performance in general e-commerce scenarios, we adapt the large-scale e-commerce dataset M5Product [f], which includes diverse categories such as furniture, toys, etc. Using MGF, we construct the Product-MT test set with the same multi-turn configurations (3-turn, 6-turn, and 8-turn) as CIRR-MT.

From these additional experiments, we conclude that our approach consistently achieves superior retrieval performance in both generalized domains and longer interaction sequences, demonstrating its robustness and effectiveness.

[f] Dong et al. M5Product: Self-harmonized Contrastive Learning for E-commercial Multi-modal Pretraining, CVPR 2022.

Performance on Larger Data Scale

To evaluate our approach on larger-scale data and analyze its memory efficiency, the constructed Product-MT test set includes three turn settings, each containing 131,710 transactions, resulting in a total of 395,130 transactions—16 times larger than the FashionMT test set in the original paper.

Following the same settings as Table 5 in the original work, we evaluate the SOTA method SPRC under different module settings to measure mean memory cost and mean recall.

As shown in the table, our approach achieves optimal performance with relatively low memory overhead.

Evaluation on Existing Datasets

Thank you for your valuable comments. Please refer to the Global Author Rebuttal for additional experiments. We have included comparison results across the following three categories: 1. Multi-turn: MT FashionIQ and MT Shoes. 2. Single-turn: FashionIQ, CIRR and CIRCO. The consistently optimal performance across these existing datasets demonstrates the effectiveness and generalization of our proposed MAI in multimodal information aggregation for diverse retrieval settings.

Variable Turn Numbers

Thanks to our proposed Modification Generation Framework (MGF), we efficiently construct multi-turn retrieval datasets with arbitrary turn numbers for any image domain. Specifically, we extend our dataset, FashionMT, to include 3-turn, 6-turn, and 8-turn settings, ensuring an equal number of transactions for each configuration. The results below demonstrate that our approach consistently outperforms across different turn counts.

Additionally, we plan to extend our dataset to version 2, ALL-MT, leveraging MGF to scale to tens of millions of transactions with variable turn counts, encompassing both e-commerce and general domains, aiming to further drive progress in the multi-turn retrieval field.

General Domain

Please refer to the Variable Turn Numbers and General Domain section in Global Author Rebuttal, where we use MGF to adapt the general-domain CIRR dataset for multi-turn retrieval, resulting in the CIRR-MT dataset. Our approach achieves leading results on this dataset, demonstrating its robustness across different domains.

Statistics for Modified Text

We categorize modified text based on the following two criteria to compare performance.
(1) Retrospective-based setting. Table 2 in the main paper categorizes performance into two retrospective-based modified text scenarios: Rollback and Combination. Additional statistics and corresponding mean recall for these two settings are provided below.

(2) Product category. Following the evaluation protocol on the FashionIQ dataset, we provide a comparison of performance across 12 major product categories in the table below.

More Visualization

Additional visualizations have been included in Section 7.4 of the Appendix, addressing three key aspects.

(1) Dataset Examples. We provide examples from the proposed FashionMT dataset under both the Combination and Rollback settings to offer better insights into the new dataset. Additionally, to validate the quality of the dataset, Section 7.3 in the Appendix includes a Quality Control analysis comparing our proposed dataset with existing multi-turn datasets in terms of accuracy, granularity, and coverage.

(2) Dataset Statistics Visualizations. Visualizations related to dataset statistics are added, including the proportion of the two settings, major category distributions for each setting, text length comparisons, and a dataset scale comparison with existing datasets.

(3) Modality Gap. To validate the theoretical insights presented in Section 4.3, we include t-SNE visualizations of the query-side and target-side representations from the final turn, both with and without the proposed MAI approach, to demonstrate the reduction in modality gap.

Dataset Availability

We will release the FashionMT dataset along with the dataset construction framework MGF to facilitate the progress of multi-turn composed retrieval scenarios.

False Negative Samples

Thank you for your insightful question. We argue that the presence of false negative samples in existing single-turn datasets primarily arises from two factors: (1) ignoring the possibility of multiple similar targets, and (2) overly simplistic modified text descriptions. To address these issues, we propose the following solutions.

(1) Target Similarity Filtering. Based on the caption generated from image $I$, we utilize a pre-trained CLIP model to measure the similarity between the caption text embedding and image embeddings. We filter out images other than $I$ that exceed a set similarity threshold. If the number of filtered images exceeds 20, it indicates a high risk of false negatives for this target, and we discard image $I$ to avoid potential issues.

(2) Fine-grained Attribute Modified Text. By introducing fine-grained attributes, our generated modified texts are more precise and detailed, with an average length approximately 2.4 times longer than existing single-turn datasets like FashionIQ. This enhanced specificity allows for more accurate target localization.

Given the significance of this issue for our task, we also conduct a quality evaluation in Appendix Section 7.3 (Table 10) using metrics such as Accuracy, Granularity, and Coverage, highlighting notable improvements over existing benchmarks.

ZS-CIR Methods

While ZS-CIR methods are tailored for zero-shot tasks, their multimodal semantic fusion paradigm serves as a strong baseline due to its ability to unify representations in the text space, which can be advantageous in text-rich multi-turn interaction scenarios.

(1) Training. The reference image is mapped into tokens via a mapping network, concatenated with the modified text, and then processed by a text encoder. Multi-turn features are aggregated using the FashionNTM method and aligned with target image features. Following the training configurations of these ZS-CIR methods, we train only the mapping network for Pic2word and Context-I2W, while for Image2sentence, both the image encoder and mapping network are optimized.

(2) Testing. All modules are frozen, and similar to the training process, we obtain aggregated multi-turn features, which are then compared with image features from the gallery to calculate similarity for retrieval. This training strategy yields better performance compared to direct zero-shot testing, with some methods achieving competitive results.

Performance on Single-turn Datasets

Please refer to the Global Author Rebuttal for additional experiments on single-turn datasets, FashionIQ and CIRR. Our approach also achieves SOTA results compared to existing methods, validating its effectiveness in the specific case where the number of turns is 1.

Clustering Details

In the MIO module, the number of cluster centers (n_cluster) is set to 32, balancing performance and efficiency, as shown in the table below.

Unlike the GMM algorithm, which relies on an iterative process (typically Expectation-Maximization) to estimate optimal parameters such as means, covariances, and mixture weights, our approach eliminates the need for iterative updates. As a non-parametric, instance-based approach, it directly utilizes the training data to compute distances between query points and stored data during inference, avoiding iterative optimization or parameter fitting.

Additionally, we analyze its impact on training (50 epochs) and inference time per query, comparing with two parameter-free baselines. The results show that MIO introduces minimal overhead while selectively retaining critical information, enhancing performance and reducing memory usage.

Backbone

BLIP-2 with the Flan-t5-xxl language model is the image captioning model we use for caption extraction. To ensure a fair comparison with the existing SOTA method SPRC, we initialize the Q-Former parameters using the same blip2_pretrain_vitL model as SPRC in all current experiments, including the base model in Table 5. This detail has been added to the Implementation Details section.

Typos

Thank you for pointing out the typos and minor issues. We have thoroughly reviewed the entire paper and corrected the non-standard punctuation marks, including the ones you mentioned.

Dataset Availability and Key Characteristics

Thank you for your valuable comments. We will release our FashionMT dataset along with the dataset construction framework MGF to support the advancement of multi-turn composed retrieval scenarios.
Additionally, we provide more details on the key characteristics of the dataset in the table below. For consistent performance comparison across methods, our FashionMT dataset currently fixes the number of turns per transaction to 3.

Complexity of MIO

The complexity of the proposed MIO module is analyzed from two perspectives: algorithmic complexity and experimental response time. We also replace DPC-kNN with two representative parameter-free algorithms, DBSCAN and Gaussian Mixture Models (GMM).

(1) Algorithmic Complexity Analysis. $N$ represents the number of tokens (e.g. 64), and $k$ is the number of selected tokens (e.g. 32). $B$ is the batch size (e.g. 2048), and $C$ is the token dimension (e.g. 256).

• MIO Module Complexity:
  • Distance Computation: $\mathcal{O}(B \times N \times C)$
  • Density Estimation: $\mathcal{O}(B \times N \times C)$
  • Token Pruning: $\mathcal{O}(B \times k \times C)$
  • Total: $\mathcal{O}(B \times (N + k) \times C) \approx \mathcal{O}(B \times N \times C)$
• DBSCAN Complexity: $\mathcal{O}(B \times N \times C)$
• GMM Complexity per iteration: $\mathcal{O}(B \times (N \times k + k^2) \times C)$

In summary, the proposed MIO module achieves computational efficiency by (1) Eliminating the quadratic scaling and iterative procedures typical of GMM, offering advantages with small $N$ and moderate $k$. (2) Employing learnable tokens for multi-turn embeddings with a reduced dimension of 256, which is lower than the original embedding dimension, leading to reduced time and memory overhead.

(2) Experimental Complexity Analysis. The table below presents the average response time per sample and the corresponding recall values on FashionMT's test set. Our proposed MIO demonstrates advantages in runtime efficiency and performance.

Ablation of Components in CCL Loss

Table 6 presents ablation experiments to further validate the effectiveness of our proposed two-stage fusion process. When w/o CCL (i.e., without CCL), the results reflect the performance of only the TSA module. Due to the TSA module's effectiveness in multimodal semantic aggregation, its performance surpasses methods like FashionERN. The addition of the CCL constraint further enhances performance. All compared methods utilize the same ViT-L backbone to ensure a fair comparison.
For better readability, we have also reorganized Table 6 as follows.

Evaluation on Existing Datasets

Please refer to the Global Author Rebuttal for additional experiments. Through performance comparisons on existing multi-turn composed retrieval datasets, MT FashionIQ and MT Shoes, as well as single-turn datasets, our proposed MAI consistently achieves optimal results, demonstrating both its effectiveness and generalizability.