FlexiReID: Adaptive Mixture of Expert for Multi-Modal Person Re-Identification

In the related work section, the author could include a review of the literature on feature fusion.

A1: Thank you for the suggestion. We appreciate your insight and will include a review of relevant literature on feature fusion in the related work section in the revised version.

The authors could conduct additional ablation studies to independently analyze the impact of applying the adaptive MoE mechanism to the text and image modalities on performance.

A2: Thank you for your suggestion. As requested, we have added the corresponding ablation experiments, with the results shown in the table below. In Row No.1, the adaptive MoE mechanism is applied on the image modality side, while in Row No.2, it is applied on the text modality side. As shown, applying the adaptive MoE to the image modality yields a greater performance improvement. This is because the image branch contains three modalities, making it more suitable for the adaptive MoE mechanism.

Talbe1: Ablation study on the impact of applying adaptive moe to different encoders

In the feature fusion module, have you considered directly fusing two modalities without using learnable features as replacements when a modality is missing?

A3: Thank you for your question. In fact, as shown in Row No.4 of the ablation study in Table 3 of the paper, we conducted an experiment without using learnable features to replace the missing modalities. The comparison shows that using learnable features to fill in missing modalities is beneficial for performance improvement.

In the ablation experiment section, how is feature fusion performed when the feature fusion module is not used?

A4: Thank you for your question. In the ablation study, when the feature fusion module is not used, we concatenate the features and pass them through a Transformer module to extract global features for fusion. This corresponds to the fusion method in Row No.1 of Table 1 in our response to reviewer NWLb. Additionally, the same table includes ablation experiments comparing different fusion strategies, which clearly demonstrate the performance advantages of our CMQF method.

The ablation experiment section lacks a comparison between the adaptive routing mechanism and the Top-K routing mechanism. It would be helpful to include this comparison.

A1: In fact, we have already compared a method using the Top-K mechanism in our ablation study, specifically in Row No.1 of Table 3. Additionally, we have supplemented our work with comparative experiments of CMQF combined with other routing mechanisms(see Table 3 in our response to reviewer YSVq). These ablation studies on routing strategies further validate the effectiveness of our proposed adaptive routing.

The paper could explore different feature fusion methods, test their performance, and compare them with the CMQF feature fusion method proposed in the paper to highlight the advantages of the proposed approach.

A2: Following your suggestion, we introduced three fusion strategies for comparison with our CMQF, as shown in the table below. The three strategies are: Concatenation—concatenating features from different modalities and feeding them into a Transformer module for fusion; Summation—summing features from different modalities and then using a Transformer for fusion; and Hierarchical Fusion—passing each modality through its own Transformer module first, followed by a shared Transformer for final fusion. It can be seen from Table 1 below that our CMQF achieves the best performance among all methods.

Table1: Comparative experiment of feature fusion methods

Thanks for your thoughtful feedback. We'll address each of your concerns in detail.

No. The paper misses model comparison experiments for multi-modal retrieval on benchmark datasets.

In Table 1, except for T-R task, there are few comparison results for other task, which cannot demonstrate the superiority of the proposed method. Similarly, in Table 2, there are no comparison results for six tasks. And the compared methods for IR-R task are not SOTA, the proposed method demonstrates relatively average performance on the dataset.

Based on the experimental results presented in the paper, the performance of the proposed method is relatively modest.

A1: There are few models for multi-modal retrieval, with UNIReID being one of the representative works. Our work is the first to explore flexible compositional retrieval, so no existing methods are directly comparable for the many-to-one setting. If you have any recommended works (preferably with open-source code，due to time constraints), we’d be glad to include them. Nevertheless, We provide additional comparisons with S-R and IR-R methods in Table 1 below, and report UNIReID’s performance on three tasks in Table 2 below. It is important to note that we did not employ any task-specific module designs or training strategies for individual retrieval tasks, which may limit performance in single-task scenarios. However, compared to traditional single-modality cross-modal retrieval, FlexiReID supports a wider range of retrieval modes and demonstrates better generalization capabilities.

Table1: Supplementary experiments on CUHK-PEDES

Table2: Supplementary experiments on SYSU-MM01

It is suggested to compare CMQF with other routing mechanism.

A2: We added comparisons between CMQF and other routing mechanisms (Table 3). Our adaptive routing shows better average performance than the alternatives.

Table3: Comparison experiment of routing mechanisms

The paper focuses on unified cross-modal person re-identification framework, could you explain the motivation behind this idea? Given that RGB -RGB is the most common task, why not integrate it into a more unified framework?

A3: Our unified framework aims to handle diverse real-world inputs beyond fixed modality pairs (e.g., Text-RGB, IR-RGB). In practice, users may provide multiple modalities (e.g., text, sketch, IR), which existing models struggle to integrate. FlexiReID supports seven modality combinations, improving retrieval performance and robustness. Following your suggestion, we also added RGB-to-RGB retrieval, with results on Market-1501 and MSMT17 shown in Table 2 of our response to reviewer dFYK.

Could you assess the impact of synthetic data on model performance through comparative experiments? And it would be more convincing to evaluate the approach on other real-life datasets.

A4: We use generative models to fill missing modalities, enabling flexible retrieval despite incomplete dataset coverage. To evaluate the impact of synthetic data, we tested on real-world datasets including SYSU-MM01 (Table 2 in the article), RegDB (Table 4 below), and PKU-Sketch (Table 1 in our response to reviewer dFYK). The results confirm FlexiReID’s practical effectiveness as a unified, adaptable framework with strong real-world deployment potential.

Talbe4: Experiments on RegDB(IR-R)

Could you also analyze the computational complexity and inference speed of the proposed method?

A5: We analyzed the computational complexity and inference speed of FlexiReID. With a frozen CLIP encoder and lightweight AEA-MoE and CMQF modules, it runs at 19 GFLOPs and 14 ms/query on a single NVIDIA 3090 GPU—comparable to UNIReID (17 GFLOPs, 11 ms/query), which supports only three fixed modality combinations. In contrast, FlexiReID supports seven flexible combinations with similar efficiency, making it more adaptable and deployable.

I am curious about the comparative experiment. This paper only provides the comparison with SOTA methods on Text-to-RGB task. However, there exists previous methods for other dual modalies retrieval. But this paper do not provide the comparison results.

A1: Thank you for the valuable question. Currently, there are still few works focusing on multi-modal retrieval. Among them, UNI-ReID is one of the representative methods that support retrieval across multiple modalities (text, sketch, and their combination). Our work is the first to systematically explore the novel query paradigm of flexible compositional retrieval, and we are the first to propose the many-to-one retrieval setting. Therefore, there are no existing methods that can be directly compared with ours under this setting. If the reviewer has any recommended works (preferably with open-source code，due to time constraints), we would be glad to include them in our discussion.

Nonetheless, we have conducted comprehensive comparisons with existing methods on several dual-modality retrieval tasks. For example, we compared with recent Sketch-to-RGB methods on the PKU-Sketch dataset (see Table 1 in our response to reviewer dFYK), and with recent Infrared-to-RGB methods on the RegDB dataset (see Table 4 in our response to reviewer YSVq). In addition, we have included the latest methods for the S→R and IR→R tasks in Table 1 of the paper (see Table 1 in our response to reviewer YSVq). These results demonstrate that FlexiReID achieves competitive performance across multiple tasks, further validating its advantages as a unified and flexible framework for multimodal retrieval.

The author used generative models (StyleGAN3, InfraGAN, GPT-4) to extend the modality of sketches, infrared images, and text descriptions. Although this approach is effective, it may result in significant differences between the generated data and the actual collected data.

A2: Thank you for the valuable question. It is true that using generative models such as StyleGAN3, InfraGAN, and GPT-4 to synthesize modalities like sketches, infrared images, and textual descriptions may introduce certain differences compared to real-world collected data. However, this approach is particularly meaningful at the current stage, as publicly available person re-identification datasets spanning three or more modalities are extremely limited. The inclusion of synthetic modalities greatly enriches the diversity of data modality, facilitates the construction of a unified multimodal dataset, and provides essential support for training our proposed flexible compositional retrieval framework.

To assess the practical effectiveness of our method, we have also conducted evaluations on several real-modality datasets. For instance, as shown in Table 2 of the article, SYSU-MM01 is a real NIR-to-RGB retrieval dataset. Additionally, we evaluated our method on the PKU-Sketch dataset, which contains real sketch modality data (see Table 1 in our response to reviewer dFYK), and on RegDB, a dataset with real infrared images (see Table 4 in our response to reviewer YSVq). These experiments further validate the effectiveness and applicability of FlexiReID in real-world scenarios.

We believe FlexiReID serves as a solid and flexible foundation, which can be further fine-tuned using real-world data to enhance its adaptability. While some discrepancies exist between synthetic and real data, this reflects a necessary stage in the research process. Our work offers new insights and methodologies for multimodal person re-identification and lays a foundation for future practical deployment.

Thanks for your careful and valuable comments. We will explain your concerns point by point.

Table1 illustrates FlexiReID achieves promising accuracy in the T+S+IR→R situations. However these sketches are generated from RGB images, while in real-world scenarios, sketches is a front-facing portrait sketched from memory, which leads to a substantial discrepancy between the sketches depicted in the paper and the official version.

The paper focuses on the multi-modal person re-ID, but these multi-modalities are generated via AI tools which is not align with real-world scenarios. Thus, there remains a gap compared to practical applications.

A1: In response to the two similar concerns you raised, we provide the following explanation. It is indeed true that sketches and other multimodal data generated using intelligent tools may differ from those in real-world scenarios. However, this approach still holds significant value. Currently, publicly available person re-identification datasets that span more than two modalities are scarce. Leveraging such generative methods can substantially enrich the modality diversity of datasets, facilitating the construction of unified multimodal datasets and providing more comprehensive support for model training.Moreover, we have also conducted experiments on real-world test sets. For instance, as shown in Table 2, SYSU-MM01 is a real near-infrared (NIR) to RGB retrieval dataset. The inclusion of generated modalities led to notable performance improvements. In addition, we evaluated our method on the PKU-Sketch dataset, which contains real sketch modality data (as shown in the table1 below), and on the RegDB dataset, which includes real infrared modality data (see Table 4 in our response to reviewer YSVq), further validating the effectiveness of our approach in real-world applications.

We believe that FlexiReID can serve as a foundational framework, which can be further fine-tuned with data from real-world scenarios to enhance the model's adaptability to practical environments. Therefore, although some modality discrepancies exist at this stage, this reflects a common characteristic of research in its developmental phase. Our work introduces new perspectives and methodologies for the field of multimodal person re-identification and lays a solid foundation for future research in real-world applications.

Table 1: Experiments on PKU-Sketch

In Line433, the authors claims "No.2 employs the traditional MOE method, while No.3 utilizes AEA-MOE." which seems not consistent with Table 3?

A2: Thank you for your correction. There was indeed a labeling error in the manuscript. In fact, No.0 corresponds to the zero-shot CLIP backbone baseline, No.1 adopts the conventional MoE approach based on the Top-K mechanism, while No.2 and No.3 employ the proposed AEA-MoE method. We will make the necessary corrections in the revised version.

The authors propose a flexible multimodal re-ID framework, but why do the authors ignore RGB-to-RGB retrieve, which is the main stream of re-ID?

A3: Thank you for your constructive suggestion. Following your advice, we have included RGB-to-RGB retrieval in our evaluation. We assessed the performance of FlexiReID on the MSMT17 and Market-1501 dataset, and the corresponding results are presented in the table2 below:

Table 2: Experiments on Market-1501 and MSMT17

In Figure2, why do the authors apply SDM loss to features of the same images, in which the model cannot retrieve persons across various situations?

A4: I may have misunderstood your question. Are you asking why the SDM loss is applied to images of pedestrians with the same pose in the same scene during training? In fact, during the data processing stage of training, we construct modality pairs using images of the same identity captured in different scenes and with different poses, then compute the SDM loss based on these pairs. During testing, the model is also capable of retrieving images of the same pedestrian taken in different scenes from the query. We will refine the illustration in Figure 2 to eliminate this ambiguity. If you have any further questions, we look forward to continued discussions.