Understanding Model Reprogramming for CLIP via Decoupling Visual Prompts

W (Weakness); R (Response)

W1: Applicability to other VLMs and other tasks

R1: Thank you for your advice! While we follow previous VR studies on classification, the idea of DVP-PRM can extend to other tasks, because CLIP learns general-purpose visual and text representations, suitable for tasks like segmentation and detection, and DVP does not fundamentally change the CLIP-based prediction pipeline.

• In segmentation, DVP can motivate methods operating at the pixel/patch level with decoupled prompts specific to different semantic elements (boundaries, textures), while using PRM to generate dense prediction maps.
• In detection, DVP can integrate into a two-stage detector, where decoupled prompts first generate region proposals focusing on objectness and scale variations, then region-specific PRM refines classification and localization.

We appreciate that these extensions are promising directions beyond the scope of this paper. In this paper we focus on extending single visual prompt capabilities and provide interpretability for reprogramming. Experiments on these applications can be explored in future work.

W2: Training complexity and low-data regimes

R2: Thank you for the feedback.

Regarding training complexity, we acknowledge that DVP-cse and DVP-cls could increase trainable parameters compared to a single visual prompt.

While we argue that the increased complexity is not that significant, we further introduce DVPlite, a variant specifically designed to address this potential efficiency concern. DVPlite decouples the padding-based visual prompt into four separate components (top, bottom, left, and right) trained independently, while keeping the total number of trainable parameters equivalent to the baseline (AttrVR).

As shown in the table below (will also be included in our revision), DVPlite maintains the same parameter count (0.04M) as AttrVR while achieving improved average performance (79.8% vs. 78.5%), demonstrating the benefits of our decoupled framework can be achieved without necessarily increasing parameter complexity.

Regarding how probabilistic reweighting matrix (PRM) performs in low-data regimes, we clarify that all experiments presented in this paper were conducted in a challenging 16-shot setting. The strong performances across downstream datasets under this condition show PRM's effectiveness even when training data availability is already limited. Moreover, as detailed in the paper (L232), PRM incorporates a top-$k$ mechanism that leverages information aggregated from the $k$ most relevant descriptions when calculating the reweighting factors, mitigating potential data scarcity issues and ensuring the stability of probability estimations.

W3: Not comparing with prompt tuning methods

R3: Thank you for raising this point. We acknowledged CoOp/CoCoOp are important methods in prompt tuning for VLMs but did not compare with text prompt tuning methods (e.g., CoCoOp) for the following reasons:

• Our work focuses on visual reprogramming (VR), where modifications are made directly to the input image space, without altering model parameters or internal text representations. In contrast, CoOp/CoCoOp perform text prompt tuning, learning continuous embeddings that interact with CLIP's text encoder, optimizing language prompt within the model's representation space. In other words, VR and prompt tuning work at different levels.

• Given this difference in mechanism and locus of adaptation (input images vs. internal text embeddings), we consider them to address different research questions regarding model adaptation, which is why a direct comparison was not included in the initial submission.

We will clarify this in the related works of the revision.

W4: Regrading the best performance and external models

R4: Thank you for your concern. Although DVP-cse generally outperforms DVP-cls, the standalone DVP-cls still shows significant advantages over the baseline DVP. For example, it leads by 3%, 2.5%, and 2.1% on UCF101, Cars, and Aircraft, respectively. This demonstrates that the proposed DVP-PRM framework still holds clear advantages without external models.

---

Thank you again for your time!

T (Theoretical Claims); E (Experimental Designs Or Analyses); W (Weaknesses); R (Response)

T/W1: Issues of mathematical symbols

R-T/W1: Thanks for the feedback on clarity and notation! We acknowledge these concerns, will supplement a notation table and smooth out the presentation in the next version.

Transpose in Equation 2: You are correct. $M_y$ is defined as a column vector, and the RHS in Equation 2 should have required a transpose to ensure dimensional consistency. We will add the transpose symbol to correct this.

For the relationship of $c,m$ and $v$. We apologize for the confusion. $c$ indeed represents the number of classes. As defined in the text, $m$ is the number of descriptions per class (L152, first column), and $v$ is the total number of visual prompts (L160, second column). Importantly, $v$ is a hyperparameter chosen independently of $c$ and $m$. We will clarify this in the next version.

E: Not providing the source code

R-E: We kindly note that we have already included an anonymous code link in the abstract section of our submission. Please feel free to check it out.

W2: More results of baseline methods with our module

R-W2: We will first explain the difference between the method 'w/o CSE' and the baseline method AttrVR in Table 1. AttrVR follows the pipeline in the original paper, using a 'knn module' proposed by the authors and two types of attribute descriptions. However, the 'w/o CSE' in the ablation study is equivalent to the scenario where DVP-cls contains only one visual prompt, meaning it only utilizes the probabilistic reweighting matrix (PRM) module proposed in this paper and relies solely on the reweighting the descriptive attributes from AttrVR. In summary, "w/o CSE" is primarily used to explore the results of reweighting descriptions without decoupling.

Next, following your suggestion, we present the performance gains (+%) achieved by integrating our module with other baseline methods. As shown in the table below:

It can be observed that our DVP-PRM module consistently achieves significant improvements across all baseline methods.

W3: Why does each class use all descriptions instead of only the descriptions generated by themselves?

R-W3: Thank you for raising your concern. Your understanding is correct, the text descriptions are initially generated for specific classes based on causes. Crucially, during the integration step detailed in Equation 5, each class $y_q$ only considers the description generated specifically for it. This is ensured by the summation condition $a' \in \mathcal A (y_q)$ in Equation 5, which restricts the aggregation to the set of descriptions $\mathcal{A}(y_q)$ associated solely with class $y_q$. The computation thus aligns precisely with the intuition that class-specific descriptions should be used for each class.

While Fig.3 visualizes the PRM matrix (which is sparse in fact, with most values being zero), the core class-specific computation is defined by Equation 5. A detailed explanation of this computation can be found in Appendix B.2. We hope this clarification resolves your concern.

---

Thank you for your time! If you have any further questions, please feel free to ask—we’d be happy to provide more clarification. If our response has resolved your concerns, we hope you might reconsider your rating.

M (Methods And Evaluation Criteria); E (Experimental Designs Or Analyses); R (Response)

M1/E1. Increased computation resource caused by DVP

R-M1/E1: Thank you for your suggestion! We have included the parameter numbers for various methods in the following table and will incorporate it into the main text in the next version.

Indeed, using DVP-cse or DVP-cls does increase the number of visual prompts, which will multiply the trainable parameter numbers.

However, in response to your feedback, we proposed DVPlite--a decoupling method that keeps the parameter numbers the same as a single visual prompt--to test the effectiveness of our DVP-PRM framework. DVPlite decomposes the original padding-based visual reprogramming pattern into four separate components (top, bottom, left, and right), training them as independent visual prompts.

As shown in the table above, DVPlite achieves performance comparable to DVP-cse or DVP-cls without increasing the parameter numbers, demonstrating its efficiency compared to the baseline method AttrVR.

M2. The feasibility of using a fully connected layer

R-M2: Thank you for your question! Replacing the Bayes rule computation of probabilistic reweighting matrix (PRM) with a single-layer neural network is theoretically feasible but presents two major issues:

• Significant increase in parameter count – Assuming each class has $m$ descriptions and the total number of classes is $c$, the additional trainable parameters would be $mc^2$. For large-category tasks like SUN397, this results in two orders of magnitude more parameters than a visual prompt.

• Inability to Preserve PRM’s Sparsity – As shown in Equation 5, PRM is a sparse matrix, meaning each class is only associated with its own descriptions, with zero weights for others. Using a fully connected layer to approximate this sparse matrix would likely yield suboptimal results.

For these reasons, we compute PRM using the Bayes rule instead of a trainable linear layer.

E2. Different results of the baseline method

R-E2: Thank you for your question. For the baseline results, we directly used the results provided in the paper of AttrVR. The difference in the average value is because AttrVR uses 12 datasets, while our study focuses on 11 finer-grained downstream classification tasks to better figure out the functionality of each visual prompt, leading to a different average accuracy.

---

Thank you again for your time!

F (Feedback); R (Response)

F1. Pros and cons of visual reprogramming (VR)

R1: Key difference: VR applies transformations only to input and/or output, while other PEFTs adapt within internal architectures, leading to 3 pros of VR:

• VR untouches any parameters and architecture, is applicable to any models, crucial for cases requiring entire model preservation to avoid catastrophic forgetting, copyright issues, or working with black-box APIs; in contrast, other PEFTs are architecture-dependent, e.g. VPT was sepcifically developed for ViT.
• In VR, #trainable params depends only on input data dims, leading to lower training overhead compared with other PEFTs (e.g. LoRA, VPT) whose parameters scale with the pretrained model.
• VR is orthogonal to other PEFTs and can be combined with them.

So, VR has a unique position within PEFT the spectrum, offering complementary values to other methods.

F2. Motivation experiment regarding the 'height' of flowers

R2: We replaced the 'height' with 'color' descriptions (i.e., 'Artichoke petals exhibit a mix of soft purple, lavender, and green tones') and still observed similar results, confirming the reliability of our motivation. In the revision, we will refine results in Fig. 2 by replacing 'height's with 'color's.

F3. Proof of Lemma 4.2

R3: Thanks for pointing it out. We can analyze the hypothesis spaces induced by VR and DVP methods, which is the standard way to compare the potential performance of models with different capacities. Due to the character limit, we provide a proof sketch and will present the completed proof in the next version.

1. Define the hypothesis spaces representable by VR $\mathcal{H}\_{\rm VR}$ parameterized by $\delta$ and DVP $\mathcal{H}\_{\rm DVP}$ parameterized by $\Theta$, mapping inputs to outputs under fixed $\omega\_{\rm fix}$.
2. $\mathcal{H}\_{\rm VR} \subseteq \mathcal{H}\_{\rm DVP}$ as DVP has a larger parameter space and it specializes $\delta\_i$ for description partitions $\mathcal{A}\_i$, offering greater flexibility.
3. We conclude that optimization over the richer space allows DVP to find a function $f$ within $\mathcal{H}\_{\rm DVP}$ that fits training data $\mathcal{D}$ at least as well as the best function found within $\mathcal{H}\_{\rm VR}$.

Thus, the minimum empirical risk $\hat{R}^{\rm dvp}\_{\mathcal{D}}$ searched over $\mathcal{H}\_{\rm DVP}$ must be less than or equal $\hat{R}^{\rm vr}\_{\mathcal{D}}$.

F4. More experiments

R4: We include more PEFTs (VPT, Adaptor) and mark the number of trainable parameters as reqeusted. Due to space limitations, we report the average accuracy across all 11 datasets. DVPlite is a new DVP variant with reduced parameters by decoupling the padding pattern of visual reprogramming (top, bottom, left, right) while maintaining the same number of parameters as the baseline.

Regarding the reweighting matrix (PRM): we did not apply it to baselines, as PRM is one of the contributions of this study. In the ablation study, we have evaluated the standalone effect of PRM to isolate its utility.

We thus observe two key merits of our method:

• Works with all architectures (RN, ViT-based), but VPT is incompatible with RN-based models.
• Improve accuracy without necessarily increasing parameters.

F5. Application in other tasks/multi-modal LLMs

R5: While we follow previous VR studies on classification, the idea of DVP-PRM can extend to other tasks because CLIP learns general-purpose visual and text representations, suitable for tasks like segmentation and detection, and DVP does not fundamentally change the CLIP prediction pipeline.

• In segmentation, DVP can motivate methods operating at the pixel/patch level with decoupled prompts specific to different semantic elements (boundaries, textures), while using PRM to generate dense prediction maps.
• In detection, DVP can integrate into a two-stage detector, where decoupled prompts first generate region proposals focusing on objectness and scale variations, then region-specific PRM refines classification and localization.

Besides, the workflow of DVP is similar on MLLM and CLIP, with the only difference being the use of instructions for downstream tasks.

We appreciate these extensions are promising directions beyond the scope of this paper. In this paper, we focuse on extending single visual prompt capabilities and provide interpretability for reprogramming. Experiments on these applications can be explored in future work.

---

Thank you for your time! If our response has resolved your concerns, we would appreciate a higher score.