Bayesian-guided Label Mapping for Visual Reprogramming

W1: Thank you for your question. This paper falls in the scope of Visual Reprogramming (VR). Thus, the experimental setting of this paper aligns with those used in previous VR studies, particularly ILM [1], to ensure comparability and consistency in the field. This setting serves as an established benchmark for evaluating VR techniques.

Regarding when fine-tuning the last layer is not feasible, we consider the following cases:

1. copyright and legal constraints: modifying some pretrained models may violate licensing agreements or intellectual property rights.
2. prevent catastrophic forgetting: keeping the pretrained model intact helps to maintain its general knowledge across tasks
3. black-box optimization: in addition, our method is useful when only the predicted logits of input are available. It does not need access to the model's internal parameters which fine-tuning clearly struggles to handle.

W2: Thank you for your question. Due to space limitations, lines 253-266 of the original text are not fully described. We will move this section to the appendix in the next version and include detailed information.

W3: We appreciate your feedback regarding the presentation of Sec. 4 and algorithm tables. We acknowledge your point about the placement of the algorithm tables. Due to space constraints, we had initially placed them in the Appendix (lines 455 and 462). However, we recognize the importance of making these algorithms more readily accessible to readers. We plan to move the main algorithm table to Sec. 4 as you suggested in the final version.

We understand your concern regarding the equations in Sec. 4. While the mathematical formulations are important for a rigorous presentation, we will smooth out the section with more practical insights. We aim to strike a balance between mathematical precision and practical applicability.

W4: Thank you for your question. The use of Padding or Watermarking depends on the specific downstream tasks, and we believe the selection may partly depend on the relationship between the pretrained model's input size and the downstream task's image dimensions. For instance, let’s consider the results of BLM+ using ResNet-18 as the pre-trained model:

Our observations:

• watermarking performs better when there is a significant size disparity: for tasks with much smaller images (32x32, e.g., GTSRB, SVHN, CIFAR10, CIFAR100) compared to the pre-trained model's input (224x224), watermarking often outperforms padding. It prevents introducing too many parameters around the image and significantly downscaling it.
• padding may perform better with larger downstream images in some cases: for tasks using larger images (128x128, e.g., Flowers102, EuroSAT), padding tends to perform better. It maintains the original image integrity by avoiding pixel value alterations.

In general, the choice impacts how we adapt pretrained model to new tasks. Therefore, the goal for input VR is to maximize the transfer of learned features while accommodating the new task's visual characteristics - the optimal choice may depend on factors beyond just image size and deserves future explorations.

W5: Thank you! Indeed, the accuracy typically improves with larger training sets. Our focus on smaller $n$ values serves as a purpose in evaluating the robustness of our proposed methods, BLM/BLM+, compared to the baseline ILM under limited data availability conditions. Our rationale for this evaluation is:

1. Practical implications: VR was proposed for adapting pretrained model to data-limited tasks. In this sense, obtaining large-scale labeled downstream task data can be challenging. By demonstrating robust performance with smaller $n$ (25%, 50%, and 75% of the original size), we highlight the practical advantages of BLM/BLM+ in data-limited scenarios, which better aligns with real-world applications.
2. Overfitting risk: Smaller training sets inherently carry a higher risk of overfitting. By testing our methods under these conditions, we can better assess their generalization capabilities compared to the baseline ILM. Thus, these experiments were conducted to validate the effectiveness and reliability of BLM/BLM+ across various data regimes. While increasing $n$ may not be that meaningful, our evaluation aims to demonstrate that BLM/BLM+ maintains competitive performance even with limited data, thus providing a more comprehensive evaluation of the practical utility.

[1] Chen et al. Understanding and improving visual prompting: A label-mapping perspective. In CVPR, 2023.

W1: Thanks! The gap between left-hand side (LHS) and right-hand side (RHS) comes from the relationship between Maximum Likelihood Estimation (LHS) and Empirical Risk Minimization (RHS) in statistical learning theory.

• LHS: represents the true objective of VR is to maximize the conditional probability of downstream label given downstream input image.
• RHS: provides a practical and empirical approximation using a finite training set and a chosen loss function.

This formulation aligns with fundamental principles in learning theory:

• We use empirical risk as a proxy for the expected risk.
• A loss function is chosen to approximate the negative log-likelihood.
• This approximation connects to generalization bounds. The gap between the expected and empirical risks can be bounded using techniques like VC-dimension.
• The law of large numbers ensures convergence of empirical to expected risk as the training set size increases.

However, we agree that using RHS directly is more straightforward. We plan to incorporate your suggestion in the next version for easier understanding.

W2: Thanks for the suggestion, we will include () in the next version.

W3: Thank you for your question. In fact, step 3 (applying $\omega$) is performed after step 2 (calculating $\omega$, and the blue dotted line indicates the flow of $\omega$). However, after calculating $\omega$, the logits output from step 1 will also be used to obtain the final prediction result in step 3 (as shown by the purple solid line). We will revise Fig. 2 to more clearly illustrate this process in the next version of the paper.

W4: Thanks for pointing it out. The omission of $x$ in Eq. 10 was intentional because:

• Our analysis in Sec 4.2 (and Appendix Sec. C) focuses on output label mapping (LM) and specifically compares different LM methods, provided that the pretrained model $f_{\rm pre}$, input $x$, and input transformations $f_{\rm in}$ are the same across different LM methods (details in Appendix Sec. C.1).
• The expected accuracy calculation is based on conditional probabilities where $x$ is implicitly part of the condition. As all LM methods operate on the same $x$, we can safely omit it, allowing us to concisely highlight the differences of LM themselves.

We will also add a note in the main text to clarify this omission, emphasizing our focus on comparing LM methods and the scope of this analysis.

W5: Thank you for your question. We will clarify it in the next version. $y_r^{\rm S}$ means a class that is more relevant to $y_{i}^{\rm T}$ (line 256), while $y_{\bar r}^{\rm S}$ is a class that is less relevant to $y_{i}^{\rm T}$ (line 257). Here we assume two classes, $y_{r}^{\rm S}$ and $y_{\bar r}^{\rm S}$, in the output space of the pre-trained model, which satisfy $p(Y^{\rm T}=y_i^{\rm T}|Y^{\rm S}=y_r^{\rm S}, X^{\rm T})>p(Y^{\rm T}=y_i^{\rm T}|Y^{\rm S}=y_{\bar r}^{\rm S}, X^{\rm T})$.

W6: Thank you for your question. Please refer to our reply to Common Question 1.

W1/Q1: Thank you very much for your suggestion. In the next version, we will expand our literature review to include relevant transfer learning concepts, particularly focusing on how Visual Reprogramming relates to and differs from traditional transfer learning methods. To better position this paper, we'll also discuss the spectrum of transfer learning techniques, connect and compare Visual Reprogramming with other parameter fine-tuning methods, highlighting its merits in scenarios where pretrained model preservation is crucial.

W2/Q2/W3/Q3: Thanks for the question. To make our reply relevant, we will discuss the transferability in the context of VR.

Transferability in VR Drawing on theoretical foundations from [1], the transferability of a pretrained model to a downstream task can be bounded by: \mathcal{L}^{\rm{T}} \leq \mathcal{L}^{\rm{S}} + \mathcal{W}(\mu^\rm{S}, \mu^\rm{T}) + \epsilon where $\mathcal{L}^{\rm{T}}$ and $\mathcal{L}^{\rm{S}}$ denotes error for downstream and pretrained tasks. \mathcal{W}(\mu^\rm{S}, \mu^\rm{T}) is the Wasserstein distance between the logit distributions of pretrained input \mu^\rm{S} = f_{\rm pre} (x^{\rm S}) and the reprogrammed downstream input \mu^\rm{T} = (f_{\rm pre} \circ f_{\text{in}}) (x^{\rm T}), $\epsilon$ is a small constant. We therefore know that the performance on downstream tasks can be related to the pretrained task performance and the alignment between pretrained task and downstream task.

Model selection This bound suggests some insights on selecting models:

• Pretrained model with higher capacity may lead to lower $\mathcal{L}^{\rm{S}}$,
• Pretrained feature is relevant to the downstream task; or input VR and the output LM that effectively transform downstream input and output (both potentially contribute to lower \mathcal{W}(\mu^\rm{S}, \mu^\rm{T}) distance, indicating better alignment)

For example, a model with large capacity pretrained on general object recognition (e.g., ResNet on ImageNet) may transfer better to another recognition task than a model pretrained on a highly specialized dataset.

Transferring between dissimilar domains (e.g., car-animal) The feasibility depends on how is the Wasserstein distance minimized - this is theoretically possible even for seemingly unrelated domains, but the practical difficulty varies because of the challenge of measuring domain similarity and feature relevance.

When to use a pretrained model Currently, there lacks theoretical tools to accurately measure the above Wasserstein distance BEFORE training, which makes it difficult to directly tell when and whether a pretrained model can be used. Therefore, we look forward to future work that focuses on effective ad-hoc estimation of such distance and techniques that minimize it through optimized VR and LM methods.

In short, while VR offers a flexible way of transfer learning, the choice of pretrained model and its adaptability to a specific downstream task depends on both model capacity and the ability to align the feature distributions.

W4/Q4: Thank you for your questions. For details on training time, please refer to Common Question 1. Regarding the fine-tuning results, since we follow the settings outlined in [2], we will directly quote these results from [2] in the next version of our paper.

W5: Thank you for your suggestion. Due to space limitations, we have placed the algorithm tables in the appendix (lines 455 and 462). In the next version, we will prioritize incorporating them into the main text to enhance readability and understanding for our readers.

W6: Thank you for your suggestion. We have added a detailed explanation in Common Questions 2 and introduced a simple example to illustrate. We will include this revision in the final version.

[1] Yang et al. Voice2Series: Reprogramming Acoustic Models for Time Series Classification. in ICML 2021.

[2] Chen et al. Understanding and improving visual prompting: A label-mapping perspective. In CVPR, 2023.

W1: Thank you. We have added a detailed explanation in Common Question 2 and provided a simple example to illustrate the conditional probabilities.

W2: Thanks. We want to clarify how our analysis can be extended to multi-class cases.

Expected Accuracy Definition The formula (Eq. 10) evaluates the expected probability of correctly mapping each $y^{\rm S} \in \mathcal{Y}^{\rm S}$ to the corresponding $y^{\rm T} \in \mathcal{Y}^{\rm T}$, remaining valid even for multi-class settings. It is agnostic to the number of classes in both $\mathcal{Y}^{\rm S}$ and $\mathcal{Y}^{\rm T}$.

Mapping Function Revision Recall the definition of PLM (Definition C.1) and DLM (Definition C.2) are

• PLM: $\mathrm{Acc}(f_{\rm plm}) = \sum_{y^{\rm T} \in \mathcal{Y}^{\rm T}} p(y^{\rm T}) \cdot \sum_{y^{\rm S} \in \mathcal{Y}^{\rm S}} p(y^{\rm S}) \cdot \omega_{y^{\rm S}, y^{\rm T}}$
• DLM: $\mathrm{Acc}(f_{\rm dlm}) = \sum_{y^{\rm T} \in \mathcal{Y}^{\rm T}} p(y^{\rm T}) \cdot \sum_{y^{\rm S} \in \mathcal{Y}^{\rm S}} p(y^{\rm S}) \cdot \delta_{y^{\rm S}, g(y^{\rm S})}$

For multi-class cases, we need to revise the mapping function from $f_{\rm lm}(y^{\rm S}) \in \lbrace y^{\rm T}, 1 - y^{\rm T} \rbrace$ in binary label spaces to align with $\mathcal{Y}^{\rm S} = \lbrace1, 2, ..., k_{\rm S} \rbrace$, $\mathcal{Y}^{\rm T} = \lbrace1, 2, ..., k_{\rm T} \rbrace$. Thus, we need to expand previously used identity/flip mapping rule $g(y^{\rm S})$ to cover a broader range of possible mappings.

Proof Sketch PLM can achieve at least the accuracy of the optimal DLM by constructing $\omega_{y^{\rm S}, y^{\rm T}}$ to match the optimal deterministic mapping rule $g^*(y^{\rm S})$, which ensures $\sum_{y^{\rm S} \in \mathcal{Y}^{\rm S}} p(y^{\rm S}) \cdot \omega_{y^{\rm S}, y^{\rm T}} \geq \sum_{y^{\rm S} \in \mathcal{Y}^{\rm S}} p(y^{\rm S}) \cdot \mathbb{I}[g^*(y^{\rm S}) = y^{\rm T}], \forall y^{\rm T} \in \mathcal{Y}^{\rm T}$. Intuitively, this inequality holds, especially when $g^*(y^{\rm S}) \neq y^{\rm T}$, $\mathbb{I}[g^*(y^{\rm S}) = y^{\rm T}]=0$, while $\omega_{y^{\rm S}, y^{\rm T}}$ can be greater than 0 in such cases, showing better flexibility.

Due to the character limit, we leave the complete proof to future work. We hope this clarification addresses your concern.

W3: Thank you. In Appendix Sec. E, we have acknowledged that optimal values may vary across datasets (Fig. 8-9). Our initial use of universal hyperparameters was intended to show that BLM/BLM+'s performance gains are not sensitive to hyperparameters.

Following your advice, we quickly run additional experiments using a 70%/30% train/validation split of the original training set to find optimal hyperparameters for each dataset, shown as

Observe that dataset-specific tuning indeed yields better performance compared to shared hyperparameters, suggesting that optimal hyperparameters tailored to each dataset are desired. We plan to include these in the revision.

W4: Thanks for your question. This observation stems from the statistical nature of our BLM+ algorithm, which computes label mappings based on the co-occurrence of predicted pretrained labels and ground truth downstream labels throughout the VR learning iterations (Fig. 4 visualizes the top-3 predicted pretrained label at each iteration).

In the later stages of VR learning, we observe that for marigold images $X^\mathrm{T}$, the conditional probabilities shifted: $p(Y^{\rm T}={\tt Marigold}|Y^{\rm S}={\tt Pineapple}, X^{\rm T})>p(Y^{\rm T}={\tt Marigold}|Y^{\rm S}={\tt Teddy}, X^{\rm T})$

$p(Y^{\rm T}={\tt Marigold}|Y^{\rm S}={\tt Pineapple}, X^{\rm T})>p(Y^{\rm T}={\tt Marigold}|Y^{\rm S}={\tt Broccoli}, X^{\rm T})$

This indicates that the feature of marigold images, after processing by input VR and the pretrained model, share more similarities with pineapple than with teddy bears or broccoli. Thus, pineapple replaced these other labels among the top-k predicted pretrained labels.

Visually, this replacement is intuitive as well: The colors of a pineapple — primarily yellow, orange, and gold — are similar to those of Marigold. Teddy bear and Airedale are predominantly brown, while guacamole and broccoli are mainly green. Additionally, the shape of a pineapple, which is typically oval, resembles that of Marigold.

W5: Thank you for your suggestion. While space limitations led us to initially place the algorithm tables in the appendix (lines 455 and 462), we recognize the importance of making this information more accessible. In the revision, we will prioritize incorporating them into the main text. We will also work on enriching and smoothing out the method presentation to make it more engaging.

W6: Yes, BLM+ without Top-K and Bayes is the same as BLM without Bayes. Thanks for the reminder, we will explain it clearly.

W7: Thank you for your questions. As we follow the settings in [1], the fine-tuning results can be directly quoted from [1], and we will include this in the next version of our paper. Regarding training time, please refer to Common Question 1 for a detailed answer. Regarding the standard to determine when VR is needed, we believe that (1) when issues such as copyright or avoiding catastrophic forgetting exist, the pre-trained model needs to be kept unchanged; (2) when the resources for training downstream tasks are limited, VR can be used.

[1] Chen et al. Understanding and improving visual prompting: A label-mapping perspective. In CVPR, 2023.