Multi-Modal Object Re-identification via Sparse Mixture-of-Experts

Q1: While the experiments are comprehensive, the analysis lacks depth. For instance, Table 3 presents results for M(TIR) without providing a corresponding discussion in which the performance of MFRNet is lower than TOPReID.

A1: Thank you for your suggestion. TOP-ReID benefits from specific training tailored for handling modality-missing scenarios, whereas our method does not include dedicated training for such cases. During training, we utilize all modalities. For testing, when a modality is missing, we supplement it using Equation 6 from the paper. For example, if RGB is missing, its features are generated based on NIR and TIR as follows: $I_R=w_R^{N}(I_N) \times g(I_N) + w_R^{T}(I_T) \times g(I_T)$. Here, $g(I_N)$ represents the features for RGB generated from NIR, $g(I_T)$ denotes the features for RGB generated from TIR, $w_R^{N}(I_N)$ and $w_R^{T}(I_T)$ are the respective weights.) To ensure model generalizability, we do not employ dedicated training methods for handling missing modalities. Notably, performing specific missing modality training could yield even better results. As shown in Table A, training our model to address the missing TIR led to a 7.7% improvement in mAP and a 7.0% increase in R-1 compared to TOP-ReID.

Table A: Experimental results for missing TIR modality.

Q2: This paper could be better by considering published literature in the cross-modality ReID community such as [1, 2]. [1] Zhang, Yukang, and Hanzi Wang. "Diverse embedding expansion network and low-light cross-modality benchmark for visible-infrared person re-identification." Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2023. [2] Park, Hyunjong, et al. "Learning by aligning: Visible-infrared person re-identification using cross-modal correspondences." Proceedings of the IEEE/CVF international conference on computer vision. 2021.

A2: Thanks for your suggestion. We have included these papers in Sec. 2 to make our work more comprehensive. If you have any other recommended works that are related to this paper, you can provide them in the discussion. We are willing to add them in.

Q3: Besides the above mentions, I have one more question: While the authors' claim is convincing, the results in Table 7 do not provide sufficient evidence that the Feature Fusion Module (FFM) requires more experts. It would be beneficial to include a comparison with the case where the number of experts in FFM is set to 1 to better validate this claim.

A3: Thank you for your suggestion. We conducted additional experiments as shown in Table B. The results indicate that the optimal performance, with an average accuracy of 87.5%, is achieved when the number of experts in the FFM is set to 3.

Table B: Performance analysis under different expert numbers for FFM.

Q4: In Equations (5) and (9), $W_1$ and $W_9$ originate from different modules and should be explicitly distinguished to avoid confusion.

A4: Thank you for your suggestion. We have added markers in the top-right corner to distinguish between different components.

Q5: The notation for Mean and Std in Equations (12) and (13) is inconsistent in font style. Please ensure uniform formatting for clarity and consistency.

A5: Thank you for your suggestion. We have corrected them following your comments.

Q1: Specifically, in Section 3.1, the description of the FFM states that transformations for all three modalities occur simultaneously. Yet, Figure 2 only illustrates interactions involving the RGB image, which may not fully represent the fusion process.

A1: Thank you for your suggestion. This transformation process applies equally to all three modalities. While the figure specifically illustrates the NIR+RGB->TIR interaction, we have conducted the same interaction for NIR and RGB. We will revise the caption of Figure 2 in light of your feedback to ensure the process is described clearly.

Q2: The necessary references have been discussed, but the related work section is somewhat lengthy. It would be better to condense it.

A2: Thank you for your suggestion. We will accordingly reduce the content of the related work section.

Q3: Given that prior works such as TOP-ReID [1], EDITOR [2], and RSCNet [3] both use an ImageNet-based ViT as the backbone, while MFRNet adopts a CLIP-based ViT, I am curious about MFRNet’s performance when using an ImageNet-based ViT.

A3: Following your suggestions, we conducted experiments on our server to evaluate TOP-ReID and our method using ImageNet-based ViT and CLIP-based ViT. As shown in Table A, the results demonstrate that both methods perform better with CLIP-based ViT compared to ImageNet-based ViT. Specifically, with ImageNet-based ViT, our method outperforms TOP-ReID by approximately 2.2% in mAP and 3.7% in R1. With CLIP-based ViT, our method surpasses TOP-ReID by about 9.7% in mAP and 9.6% in R1. These findings underscore the superior performance of our method over TOP-ReID under identical settings and backbones.

Table A: Performance comparison on ImageNet-based and CLIP-based ViT.

Q4: Some details of the testing phase are not entirely clear. In Table 3, I would like to know how the FFM work in the absence of certain modalities.

A4: Following your suggestion, we will refine this section in future versions to enhance clarity. For the missing modality, as illustrated in Equation 6 (lines 190-199), the other two modalities are utilized for prediction. For example, if RGB is missing, its features are generated based on NIR and TIR as follows: $I_R=w_R^{N}(I_N) \times g(I_N) + w_R^{T}(I_T) \times g(I_T)$. Here, $g(I_N)$ denotes the features for RGB generated from NIR, $g(I_T)$ represents the features for RGB generated from TIR, $w_R^{N}(I_N)$ and $w_R^{T}(I_T)$ are the respective weights.

Q5: Some captions lack some details. What do ‘RE’ and ‘GE’ mean in Figure 2?

A5: We sincerely apologize for any inconvenience caused. 'GE' refers to the generation experts in FFR, while 'RE' represents the representation experts in FRM. Based on your suggestion, we will revise Figure 2 accordingly.

Q6: Eq.12 and Eq.13 are not aligned.

A6: Thank you for your suggestion. We have correctted it in these days.

Response to Reviewer PMp6:

Q1: From Table 4, we can observe that the FRM has decreased the 'Params' and 'FLOPs'. Generally, the MoE structure should keep similar or slightly higher 'Params' and 'FLOPs' with the baseline during the inference.

A1: Thank you for your suggestion. As demonstrated in Equation 8, the FRM module leverages RepAdapter to build MoE representations. Compared to a traditional MLP, RepAdapter reduces computational costs by incorporating two convolutional layers. If replaced with the original MLP structure, as indicated in Table A, the parameters and FLOPs would increase by 3.7M (from 53.5 to 57.2) and 1.4G (from 20.7 to 22.1), respectively.

Table A: Comparison of FRM and original MLP.

Q2: The discussion of 'Params' and 'FLOPs' only contains this method itself. I understand that it can be similar efficiency to the baseline method. But it would be better if the author could show a comparison with recent methods.

A2: Following your suggestion, we compared the computational costs with three recent works. As shown in Table B, our method not only demonstrates significant improvements in metrics such as mAP and R-1 but also achieves the lowest Params and FLOPs, with values of 57.1M and 22.1G, respectively.

Table B: Comparison of computational cost with recent methods.

Q3: Table 7 may not enough to show that 3 is the optimal selection.

A3: Thank you for your suggestion. We conducted additional experiments as shown in Table C. The results indicate that the optimal performance, with an average accuracy of 87.5%, is achieved when the number of experts in the FFM is set to 3.

Table C: Performance analysis under different expert numbers for FFM.

Q4: FFM doesn't seem to conflict with VIT. It seems can be inserted in the network, but it is not discussed at all.

A4: Following your suggestion, we further validated the approach by inserting FFM into the 3rd, 6th, and 9th layers of the network. As shown in Table D, applying FFM before the ViT leverages the pixel-by-pixel alignment of multimodal image data more effectively, resulting in optimal model performance.

Table D: Performance analysis under different locations for FFM.

Q5: There are still several typos in the paper.

A5: Thank you for your suggestion. We will correct these typos in the updated version of the paper.

Q1: While the MFRNet presents a novel perspective within multi-modal ReID by introducing a sparse Mixture-of-Experts (MoE) framework, its overall novelty remains moderate. It is more like a representing of an effective combination of existing techniques.

A1: Our novelty primarily lies in two aspects: generation and representation. For FFM generation, unlike previous methods that rely on coarse-grained feature interactions, we creatively harness the pixel-level consistency of multimodal data (pixel-wise alignment) to achieve fine-grained multimodal interaction. For FRM representation, our approach avoids the independent encoding of each modality. Instead, we introduce multimodal feature representation experts based on the MoE architecture, achieving a balance between modality-specific and shared features while maintaining minimal computational cost.

Q2: Figure 2 is slightly cluttered, and the author seems to have forgotten to describe GE and RE.

A2: We sincerely apologize for any inconvenience caused. GE refers to the generation experts in FFR, while RE represents the representation experts in FRM. Based on your suggestion, we will revise Figure 2 accordingly.

Q3: The visualization section appears to focus solely on the Feature Representation Module, while the visualization of the Feature Fusion Module is also essential for a comprehensive analysis.

A3: Thank you for your suggestion. In response, we have also visualized FFM. However, due to the constraints of the rebuttal format, we are unable to include the image directly. Nonetheless, it will be incorporated in future versions of the paper.

Q4: Can the authors provide a more detailed comparison of computations with different methods?

A4: Following your suggestion, we compared the computational costs with three recent works. As shown in Table B, our method not only demonstrates significant improvements in metrics such as mAP and R-1 but also achieves the lowest Params and FLOPs, with values of 57.1M and 22.1G, respectively.

Table A: Comparison of computational cost with recent methods. The best results are shown in bold.

Q5: The author mentioned that ‘TOP-ReID has a specific training phase for the modality missing condition’ while MFRNet does not. Will MFRNet be better than TOP-ReID in the 'M (TIR)' protocol with the same specific training phase?

A5: Following your suggestion, we conducted additional experiments targeting the 'M (TIR)' protocol. As shown in Table B, training our model to address the missing of TIR resulted in a 7.7% improvement in mAP and a 7.0% increase in R-1 compared to TOP-ReID.

Table B: Experimental results for missing TIR modality.

Q6: Why can the FRM reduce computational complexity?

A6: Thank you for your suggestion. As demonstrated in Equation 8, the FRM module leverages RepAdapter to build MoE representations. Compared to a traditional MLP, RepAdapter reduces computational costs by incorporating two convolutional layers. If replaced with the original MLP structure, as indicated in Table C, the parameters and FLOPs would increase by 3.7M (from 53.5 to 57.2) and 1.4G (from 20.7 to 22.1), respectively.

Table C: Comparison of FRM and original MLP.

Q7: There are several typos in the paper.

A7: Thank you for your suggestion. We will correct the typos in the updated version of the paper.