Multi-Modal Interactive Agent Layer for Few-Shot Universal Cross-Domain Retrieval and Beyond

R-1: Weakness1 & Question1

We appreciate the reviewer's question. Our motivations for introducing modality-coupling are as follows:

1. The primary goal of vision-language models is to align image and text modalities within a shared feature space. In CLIP, this alignment is achieved through dual encoders jointly trained on 400M image-text pairs during pre-training.
2. When adapting to downstream tasks （typically the few-shot classification problem） or new domains, maintaining this cross-modal alignment remains crucial. Early approaches like CoOp [1] fine-tune only the text encoder, which may disrupt the alignment between the image and text encoders under the few-shot setting. Therefore, MaPLe [2] and MMA [3] propose to fine-tune both the text and image encoders such that their optimal alignment can be achieved on the downstream domains.
3. Role of Modality Coupling: As demonstrated in MaPLe and MMA, explicitly modeling the interaction between image and text branches (modality coupling) during fine-tuning improves representational coherence. Such coupling ensures that updates to one modality are reflected in the other, thereby promoting more consistent cross-modal alignment.
4. As demonstrated in both the MaPLe and MMA papers, as well as in our own work, modality coupling proves effective in enhancing performance for both few-shot classification and FS-UCDR. It is therefore natural for us to explore how modality coupling can be incorporated into the third category of PEFT methods—namely, partially fine-tuned methods. This motivation led to the development of MAIL, which achieves state-of-the-art performance on both few-shot classification and FS-UCDR tasks.

In summary, we argue that any fine-tuning method intended to adapt a foundational multi-modal model like CLIP should preserve the alignment established during pre-training. Introducing modality coupling is a principled extension of this objective, supported by empirical evidence from works like MaPLe and MMA. If the reviewer still has concerns about the motivation, we will try to explain from a different perspective (the regulation effect) in R-3.

R-2: Weakness2 & Question2

We apologize for any misunderstanding caused by our writing. We would like to offer the following clarifications:

1. CLIP-based PEFT methods have demonstrated remarkable success in few-shot classification scenarios with limited training data, primarily due to CLIP's strong generalization capabilities. In fact, the challenge of generalizing under limited supervision is largely mitigated by the powerful CLIP backbone, while subsequent PEFT methods—including our proposed MAIL—serve to further enhance CLIP’s adaptability and performance.
2. Current CLIP-based PEFT methods mainly test on the few-shot classification scenario. We identify the Universal Cross-Domain Retrieval (UCDR) setting, which provides a more challenging and informative benchmark for evaluating the generalization ability of PEFT frameworks. Specifically, UCDR requires models to generalize to both unseen domains and unseen classes during testing, making it a more rigorous standard for assessing cross-domain and cross-category generalization. This observation motivated us to explore the potential of CLIP-based PEFT methods in addressing the FS-UCDR setting.
3. We empirically demonstrate that modality-coupled methods are good designs for FS-UCDR, and MAIL is a general-purpose modality-coupled PEFT framework proposed in our work, it is not designed exclusively for FS-UCDR. Rather, our goal is to provide a flexible framework that can be adapted to a range of few-shot tasks.
4. Conceptually, we could have written a standalone paper focused solely on few-shot classification with MAIL, similar to works like MaPLe and MMA. However, we chose to broaden the scope to offer a more comprehensive study, which we believe not only benefits the PEFT community but also benefits the UCDR community—particularly under low-resource conditions.
5. Moreover, one can design MAIL variants specifically tailored to FS-UCDR. For example, introducing domain-specific agent layers may be beneficial, given that UCDR inherently involves multiple domains.

R-3: Question3

• If the reviewer is expecting a theoretical or mathematical framework, we sincerely apologize that we are currently unable to provide one. However, as this work is submitted to the Vision-Language Application track, we believe that thorough experimental validation and strong empirical results should be given greater emphasis in this context.

  1. In fact, methods such as MaPLe [2] and MMA [3] also pursue the alignment of parameter updates. The key difference lies in the scope of alignment: they focus on aligning the updates of newly introduced parameters (e.g., prompts or adapters), whereas our MAIL aims to align updates of internal parameters (i.e., target layers) within the pre-trained backbone.

  2. It is worth noting that the original MaPLe and MMA papers do not provide a formal theoretical justification for why aligning parameter updates improves performance. Instead, they argue that such alignment leads to more sufficient and complete adaptation of vision-language models—an intuition rooted in the inherently multi-modal nature of VLMs. Our work builds on this intuition by extending it to a different fine-tuning regime.

• Here we would like to the rationale behind aligning parameter updates in an intuitive way.

  When fine-tuning CLIP, a key challenge is to prevent overfitting. As shown in our Equation (11), one of the most common and straightforward approaches is to apply an regulation loss separately on the outputs of the two modality-specific heads:

  $$L_{reg}^v=f(V, V^F), L_{reg}^t=f(T, T^F)$$

  where $V$ and $T$ denote the visual and textual outputs of the frozen CLIP, while $V^F$ and $T^F$ denote the fine-tuned outputs given the same input. $f(\cdot)$ denotes the regulation loss, typically the $L_2$ loss.

  • We argue that aligning parameter updates can be also viewed as a form of regularization. Such coupling ensures that updates to one modality are reflected in the other, thereby encouraging more consistent cross-modal alignment.

  • This explains why modality-coupled methods perform well in the FS-UCDR setting. In FS-UCDR, both the domains and classes encountered during testing are unseen during training, which requires the model to possess strong generalization capabilities.

  • This also helps to interpret the results in Table 4 from our main paper. While our method achieves comparable accuracy to DeKg (ICLR’25) on the base classes, it yields a significant improvement on the new classes. The “new” setting refers to test classes that are not present during training, thereby placing higher demands on the model’s generalization ability—precisely where our approach excels.

R-4: Weakness 3

We sincerely thank the reviewer for pointing out this issue. We will address it carefully in the revised version. We also apologize for any confusion that may have arisen during your review.

Once again, we greatly appreciate the reviewer's insightful comments. We hope our responses could address your concerns. If you have any additional questions or suggestions, please let us know.

[1] Kaiyang Zhou, Jingkang Yang, et al. Learning to prompt for vision-language models. IJCV, 2022.

[2] Muhammad Uzair Khattak, Hanoona Rasheed, et al. Maple: Multi-modal prompt learning. CVPR, 2023.

[3] Lingxiao Yang, Ru-Yuan Zhang, et al. Mma: Multi-modal adapter for vision-language models. CVPR, 2024.

R-1: Weakness-1

We respectfully disagree with the claim that the performance is not significantly different from MMA under the same number of tunable parameters.

• First, as shown in Table 1, our method achieves consistent improvements on the 2-shot setting of DomainNet. Specifically, it achieves a 1.74% absolute gain in average mAP and a 2.69% gain in precision compared to MMA. Such margins are non-trivial in few-shot settings, where performance improvements are generally hard to obtain due to limited supervision.
• More importantly, on the widely adopted and more challenging few-shot classification benchmark, our method outperforms MMA by a 1.23% gain in average harmonic accuracy, as shown in Table 4. With the inclusion of the proposed regularization term, this gap further increases to 1.89%, averaged over 11 diverse datasets. This demonstrates the robustness and generalization ability of our approach across domains and tasks.
• Here, we provide a simpler example by listing the performance of recent methods from top-tier conferences on few-shot classification. The performance gain of MAIL over DeKg is already quite notable—especially when compared to the typical improvements between methods from 2023 to 2024. The gap becomes even more striking when compared to MMA.

Methods	Base	New	HM
CLIP [ICML'21]	69.34	74.22	71.70
CoCoOp [CVPR'22]	80.47	71.69	75.83
MaPLe [CVPR'23]	82.28	75.14	78.55
PromptSRC [ICCV'23]	84.26	76.10	79.97
MMA [CVPR'24]	83.20	76.80	79.87
TCP [CVPR'24]	84.13	75.36	79.51
DeKg [ICLR'25]	84.96	76.38	80.44
MAIL [Ours]	85.19	77.39	81.10

While our method achieves comparable accuracy to DeKg (ICLR’25) on the base classes, it yields a significant improvement on the new classes. The “new” setting refers to test classes that are not present during training, thereby placing higher demands on the model’s generalization ability—precisely where our MAIL excels.

Therefore, we believe the improvements are both statistically meaningful and practically valuable.

R-2: Weakness-2 & 3

We thank the reviewer for the constructive feedback. We acknowledge that the y-axis not starting at zero may lead to visual misinterpretation. While our intention was to emphasize subtle performance differences, we agree that this could be misleading. In future versions, we will ensure to follow best practices for visualizations by starting the y-axis from zero and clearly indicating the axis range.

Regarding Figure 1, our MAIL not only uses fewer learnable parameters than MaPLe and MMA but also achieves higher mAP. Although the training time is longer, the reduced parameter count results in fewer model weights to store, which is advantageous for real-world deployment. Importantly, we did not intentionally use the lower parameter count to obscure the longer training time—these two aspects are not equivalent, and suggesting so would be misleading. Interestingly, Reviewer-1 (Reviewer hW62) mentioned a related work, SVFT [3], an improvement over LoRA, which also employs fewer trainable parameters but requires longer training time.

R-3: Weakness-4 & Question-1

As discussed in the paper, training time and memory usage are the main limitations of our method. From our perspective, it is not always feasible to design an algorithm that simultaneously achieves lower training time, faster inference, and higher accuracy. We hope the reviewer understands this trade-off and the challenges it presents.

Here we illustrate the reasons why our MAIL is still valuable though having more training time.

1. In the context of vision-language models (VLMs), studying modality coupling is inherently valuable. Its importance becomes even more evident in few-shot scenarios, where it has already demonstrated strong effectiveness. We believe the reviewer has recognized this point.
2. MAIL fills an important research gap—namely, the lack of modality coupling exploration into the third type of PEFT methods, i.e., partially fine-tuned approaches. This contribution is meaningful, as it can serve as a foundation and reference for future research. We do agree that training efficiency is important, and we also agree that how to reduce the training time can be more challenging, but we feel that our work represents an important starting point for further deeper analysis. This paper is not the end of the story, but rather a contribution to the broader line of research on modality coupling within the partially fine-tuned PEFT community.
3. While MAIL may involve longer training time, the more important is, our MAIL has achieved remarkable performance, as responded in R-1. It's a worth trade-off.
4. What's more, compared to UCDR, FS-UCDR (which is also first proposed in this paper) already achieves a substantial reduction in training time. Therefore, we argue thats increase introduced by MAIL would not become a bottleneck.

R4: Question-2

Thank you for the reviewer’s insightful comment. We would first like to clarify why MaPLe, despite having the largest number of parameters, exhibits the highest efficiency. Then, we explain why MAIL is slower than MMA.

1. Regarding the efficiency of MaPLe [1]: The additional parameters in MaPLe come from extending the input sequence length rather than introducing new computational operations. These extended tokens are processed within the Transformer through matrix multiplications, which are highly parallelizable. As a result, the increase in computational workload is efficiently handled by modern hardware. In contrast, both MMA and our proposed MAIL introduce additional operations within each Transformer block. These operations are inherently sequential relative to the standard forward pass and cannot be easily parallelized. Therefore, both MMA and MAIL experience longer training times.
2. Why MAIL is slower than MMA: MAIL performs its operations at four local positions within each Transformer block, whereas MMA applies its operation at a single global position. This design in MAIL leads to more frequent sequential computations, resulting in longer training time compared to MMA.

Finally, we kindly ask the reviewer not to assess our method based solely on efficiency. As noted in the appendix (and also explicitly stated in the Limitations section of the main paper), we acknowledge the shortcomings of our approach, namely the training time and memory usage. However, given that the few-shot tuning paradigm for vision-language models is generally efficient and time-saving, we believe training time is unlikely to pose a significant bottleneck in practical applications.

[1] Muhammad Uzair Khattak, Hanoona Rasheed, et al. Maple: Multi-modal prompt learning. CVPR, 2023.

[2] Lingxiao Yang, Ru-Yuan Zhang, et al. Mma: Multi-modal adapter for vision-language models. CVPR, 2024.

[3] Lingam, Vijay Chandra, et al. SVFT: Parameter-efficient fine-tuning with singular vectors. NeurIPS, 2024.

R-1: Weakness1

We appreciate the reviewer’s insightful comment. While it is true that CLIP’s pre-trained features can be suboptimal in highly specialized domains such as medical imaging or remote sensing, this limitation is not specific to MAIL, but rather a general challenge faced by all CLIP-based methods.

Importantly, MAIL is designed as a plug-and-play framework that builds on top of CLIP and enhances its adaptability. Moreover, it can be easily integrated with other foundation models that are pre-trained on medical or remote sensing images, potentially leading to even better performance in such specialized domains.

R-2: Weakness2

Thank you for bringing the paper SVFT [1] to our attention. We agree that it is relevant to our work, and we will be happy to cite and include it in the discussion of our paper. Attempting to generalize our MAIL to the sparse matrix (instead of the diagonal matrix) is an interesting idea for future work.

To further acknowledge SVFT's contribution, we conducted experiments using two variants of SVFT, namely SVFT$^P$ and SVFT$^R_{d=12}$. Below, we report their FS-UCDR (2-shot) evaluation results (%) on DomainNet.

Method	Sketch				Quickdraw				Painting
	Unseen		Mixed		Unseen		Mixed		Unseen		Mixed
	map	prec	map	prec	map	prec	map	prec	map	prec	map	prec
ProS* [CVPR'24]	64.67	60.01	58.43	54.63	28.42	25.44	23.18	21.27	75.16	69.55	71.20	66.12
CLIP [ICML'21]	42.20	35.28	36.62	29.79	7.44	5.61	6.00	3.17	61.68	55.07	56.53	50.14
LORA [ICLR'22]	54.85	49.23	48.71	43.33	22.16	18.10	17.73	14.21	71.46	65.10	66.81	60.81
SVFT$^P$[NeurIPS'24]	57.72	51.94	50.04	47.01	22.32	18.88	18.12	15.71	69.99	63.11	65.82	60.00
SVFT$^R_{d=12}$[NeurIPS'24]	60.74	53.41	53.82	51.90	24.98	20.39	19.40	17.63	73.95	67.37	68.44	63.33
MAIL [Ours]	65.76	61.57	59.05	55.25	29.41	26.95	22.83	21.26	76.05	70.85	71.12	66.44
Method	Infograph				Clipart				Average
	Unseen		Mixed		Unseen		Mixed		Unseen		Mixed
	map	prec	map	prec	map	prec	map	prec	map	prec	map	prec
ProS* [CVPR'24]	57.98	54.42	52.19	49.56	76.48	71.86	72.28	68.15	60.52	56.26	55.46	51.95
CLIP [ICML'21]	50.08	44.74	43.75	38.91	60.37	51.30	56.08	46.91	44.35	38.40	39.80	33.78
LORA [ICLR'22]	58.01	53.84	51.86	48.20	70.52	64.11	66.00	59.63	55.40	50.08	50.22	45.24
SVFT$^P$[NeurIPS'24]	58.47	52.55	52.13	48.65	71.77	65.94	67.33	60.57	56.05	50.48	50.69	46.39
SVFT$^R_{d=12}$[NeurIPS'24]	57.96	52.04	51.30	47.26	72.12	66.56	69.02	62.25	57.95	51.95	52.40	48.74
MAIL [Ours]	60.11	57.40	53.34	50.95	78.94	74.80	73.91	70.14	62.05	58.31	56.05	52.80

As shown in the table, SVFT performs worse than our proposed MAIL. Interestingly, while SVFT$^R_{d=12}$ achieves performance comparable to LoRA in the original SVFT paper, it exhibits a more significant improvement over LoRA under the FS-UCDR setting.

So far, LoRA-based methods have seen limited application in the context of CLIP. As shown in our Table 4 （from the main paper）, the current state-of-the-art methods for few-shot classification are all PEFT approaches that are not based on LoRA. This may be because LoRA has been primarily explored in the domain of large language models, while its adaptation to CLIP remains under-explored. As a result, directly applying LoRA to CLIP without further optimization may lead to suboptimal performance.

[1] Lingam, Vijay Chandra, et al. SVFT: Parameter-efficient fine-tuning with singular vectors. Advances in Neural Information Processing Systems, 2024.