Gray-Box Fine-Tuning for Single Backbone Domain Experts

Dear reviewer KBkw, thanks for your detailed review and recognition of our thorough experiments. We address your key points below:

1. Comparisons with Other Gray-Box Methods:

   Thank you for your suggestion to compare our methods with MaPLe. Following your request, we conducted that comparison. Below we attach qualitative comparisons on different benchmarks. Our results show that our LGA, which operates at the same information exposure level as MaPLe, consistently outperforms it. These results are now included in the revised manuscript (e.g., Tables 3 and 11). For Co-CoOp, unfortunately, their design is not suitable for the retrieval task (as we describe in lines 193-197, main paper), due to the requirement of both modalities (e.g., text, image) to be encoded jointly rather than separately (e.g., as in CLIP). It requires online computation of the embeddings, conditioned on the query.

2. Clarification of Last Layers FT:

   Although Last Layers FT only exposes the final layer weights, it does involve direct access to model layers, which places it in the white-box category according to our definition. In contrast, our gray-box definition strictly avoids any exposure of model weights while permitting different exit points for gradient propagation (as described in lines 72–75). We have clarified this distinction further in the revised manuscript, with updates highlighted in blue for improved clarity.

Comparison examples on a few benchmarks:

Our results demonstrate that our LGA, operating at the same level of information exposure as MaPLe, consistently outperforms it across several benchmarks. Notably, even our more restrictive DGA approach, which does not rely on intermediate points for gradient propagation, achieves superior results compared to MaPLe in certain cases.

COCO:

Building:

Road:

MSR-VTT:

Stanford Cars:

We hope these responses address your concerns and are happy to discuss further if needed. Thank you for your thoughtful feedback.

Dear reviewer mA3M, Thank you for your positive comments on our work. We appreciate your recognition of the practical focus of our study and its generality across tasks. Below, we address your concerns:

1. Evidence Supporting Reduced Model Exposure Clarification of Motivation:

The motivation for reducing model exposure is grounded in the practical and theoretical risks associated with sharing foundational model weights and architectures. As cited in our manuscript (lines 49–58), previous works highlight how unauthorized access to model internals can lead to several critical issues, such as:

• Unauthorized use of proprietary models (OpenAI, 2023),
• Recovery of sensitive training data embedded in model weights (Haim et al., 2022),
• IP violations, including the reconstruction of original unpublished model weights through adaptations like LoRA (Horwitz et al., 2024).

While we do not claim that our method is completely invulnerable, it does offer a safeguard that counters the assumptions exploited by these prior works. We further expand on these concerns in lines 204–217, detailing risks related to model theft and misuse.

2. Computation Cost Savings:

Efficiency Analysis: Building on findings from prior research (Pope et al., 2023; Lester et al., 2021), which explore service utilization at scale (as discussed in lines 39–43), we highlight that managing multiple specialized models for different tasks or domains results in inherently lower resource utilization compared to employing a single, versatile model capable of handling diverse tasks. This distinction is particularly significant in large-scale industrial batch operations. To further demonstrate this advantage, we have added a simple illustrative figure (Section D, Fig. 4) that highlights the benefits of maintaining a unified model over managing multiple, separate models.

Specifically, our input/output adapters act as pre/post-processors for task-specific data and can be computed independently on separate machines, enabling improved load balancing and efficient resource utilization.

Experimental Validation:

To substantiate these claims, we conducted inference experiments comparing two setups:

1. A single backbone combined with 10 pairs of DGA adapters (for 10 different tasks or domains).
2. Ten separate backbones without using our DGA framework. In each setup, we utilize CLIP encoders to encode 10 sampled sets of 100 pairs of images (224x224) and their captions, resulting in a total of 1,000 paired samples and their corresponding feature vectors.

The results demonstrate significant computational and memory efficiency with our approach:

• Our framework required 22.760 GFLOPs for 1000 samples, compared to 203.223 GFLOPs for the separate backbone setup.
• Similarly, GPU memory usage was reduced to 1.462 GB, as opposed to 14.54 GB in the alternative setup.

These results highlight the resource efficiency and scalability of our framework in managing diverse tasks or domains. We have included this analysis and the corresponding figure in our revised version, as we greatly value your feedback and its contribution to improving the clarity and depth of our paper.

We hope these additions address your concerns and provide further evidence for the broad applicability of our approach. We remain open to any further questions or suggestions you may have.

Dear reviewer ZYrt, we thank you for your thoughtful review and for acknowledging the significance of our work in balancing model adaptability with privacy. We appreciate your feedback, and we address your main points below:

1. Distinction Between Our Method and Federated Learning:

We understand the need to differentiate our Gray-box approach from federated learning (FL), particularly as terms like “gray/white/black boxes” appear in both contexts. While both prioritize privacy, our framework focuses on model privacy whereas FL addresses data privacy. Our Gray-box framework seals the foundational model’s structure and weights, focusing on secure task adaptation. Federated learning is a distributed machine learning approach that allows models to be trained on devices without accessing the clients' training data. In this paper, we focus on model privacy rather than data privacy. We added this distinction for clarity, in the revised manuscript.

2. Additional Baseline Comparisons:

We have conducted additional comparisons as suggested. Our results demonstrate that our LGA method, operating at the same level of information exposure as MaPLe, consistently outperforms it. These findings have been incorporated into the revised manuscript, with details provided in Tables 3 and 11. Below, we include a few examples of these comparisons for reference.

We sincerely appreciate the reviewer’s suggestion to include comparisons with the cited methodologies. Here, we address the specific works referenced by the reviewer:

1. Liu et al., “Language Models as Black-Box Optimizers for Vision-Language Models” (CVPR 2024),

   Wang et al., “Connecting the Dots: Collaborative Fine-Tuning for Black-Box Vision-Language Models” (ICML 2024): These papers suggest a black-box prompt optimization for VLLM, and are limited to text manipulation or text-to-text mapping, without altering the other modality. More specifically, they optimize textual prompts, for 16-shot classification tasks. Therefore, they are incapable of performing tasks such as in Video or Sketch retrieval as we demonstrate in our paper, that are based on (or limited to) the visual domain. We further added this discussion to Appendix D.

2. Lu et al., “ZooPFL: Exploring Black-Box Foundation Models for Personalized Federated Learning” (arXiv 2023):

   While the ZooPFL approach is interesting, their code is not publicly available, which limits our ability to directly evaluate its applicability to our tasks. However, ZooPFL addresses personalization challenges in federated learning with black-box models through distributed training across multiple clients, and it is not fully clear to us how their method could be adapted to our specific context. Additionally, the number of trainable parameters employed in their distributed training setup is significantly higher than the parameter-efficient methods we explore.

Given these differences in scope and methodology, we believe that a direct comparison between our framework and these works falls outside the scope of our study. We hope this explanation clarifies our selection of comparative baselines and highlights the contributions of our work.

Comparison examples on a few benchmarks:

Our results demonstrate that our LGA, operating at the same level of information exposure as MaPLe, consistently outperforms it across several benchmarks. Notably, even our more restrictive DGA approach, which does not rely on intermediate points for gradient propagation, achieves superior results compared to MaPLe in certain cases.

Due to lack of space, please see results in our comment for reviewer KBkw or in our revisited manuscript.

We hope these responses address your concerns and are happy to discuss further.

Dear reviewer Zs3x, thanks for your valuable feedback and for recognizing the significance of our work. We appreciate your thorough review and address your main points below:

1. Generalization to Various Foundation Models and Tasks

   We thank the reviewer for this comment. While experiments in the main paper focused on representation learning we strongly believe that our approach generalizes to other backbones and tasks. Following this question we conducted additional experiments on image captioning and general language understanding using different backbones. Specifically, we apply our LGA approach on (1) a VLLM (BLIP-2, 3.7B) for Image Captioning and (2) an LLM (DeBERTa-v3-base) for GLUE benchmark (on the MRPC dataset). Results can be found in the table below. The results demonstrate that our approach performs effectively in new contexts, reinforcing its applicability to a variety of foundational models and tasks.

Image Captioning, VLLM BLIP2 (3.7B):

General Language Understanding Evaluation, on MRPC dataset with LLM Deberta-v3-base:

2. Method design is straightforward

   We acknowledge the straightforward nature of our method, but see this simplicity as a key strength - it ensures ease of implementation, reduces the complexity of integration, and facilitates broader adoption across different applications. However, we want to emphasize that a key aspect of our novelty lies in the context of our gray-box fine-tuning framework, which is specifically designed to address scenarios with restricted access to model internals, balancing performance, security, and adaptability.

We hope these points address your concerns and strengthen the evidence for the broad applicability of our approach. We are open for more questions.