Q1. Lack of novelty.

Let us recap the main novelties of this paper.

1. We proposed a novel task called Black-Box Forgetting, which aims to achieve selective class forgetting under the assumption that the parameters and gradients of the model are inaccessible.

2. Aiming at improving derivative-free prompt tuning for Black-Box Forgetting, we proposed a novel context parametrization method called Latent Context Sharing (LCS) that explicitly models shared and unique components over multiple latent contexts.

To our knowledge, there is no literature addressing Black-Box Forgetting, nor has the idea of explicitly modeling the shared and unique components of learnable contexts in prompt tuning been explored. Moreover, in all the experiments reported in our paper, we compared our LCS with BBT and showed that ours was superior to BBT. We would be happy to discuss these novelties in more detail if the reviewer could provide specific examples of publications that contradict these novelties.

Q2. Loss function not novel.

Note that we are not claiming novelty in the loss function. We argue that our novelty lies in Latent Context Sharing (LCS), a novel parametrization of the learnable latent contexts in prompt tuning for Black-Box Forgetting.

Q3. Baselines are black-box tuning methods, not forgetting methods.

As far as we know, there is no existing method for Black-Box Forgetting, i.e., there is no existing forgetting / unlearning method that can be directly compared. Therefore, we compared our method with the existing black-box tuning methods, i.e., BBT and CBBT, under the same loss function. We would appreciate it if the reviewer could provide any specific existing methods that are directly comparable or existing loss functions to be applied so that we can have clearer discussions.

Q4. Datasets limited to three natural image datasets.

Due to the nature of the pre-trained CLIP that are naturally biased toward natural images, the performance of zero-shot CLIP has been reported to be significantly degraded in non-natural image domains (e.g., [a,b]). Since our task is selective forgetting of classes that can successfully be recognized by the pre-trained model, it is difficult to test its effectiveness on non-natural image datasets.

On the other hand, we agree that conducting experiments on more diverse datasets is beneficial, so we conducted additional experiments using ImageNet30. The results can be found in Table G in the attached PDF. Our method is superior to the other methods in terms of $H$, which supports the effectiveness of our method further.

[a] Shu et al., Test-Time Prompt Tuning for Zero-Shot Generalization in Vision-Language Models, in Proc. NeurIPS, 2022.

[b] Gondal et al., Domain Aligned CLIP for Few-shot Classification, in Proc. WACV, 2024.

Q4. Typo in $L_{uniform}$.

We will fix it in the final version. Thanks!

Q5. CUB-200-2011 in Table 1, white-box CoOp only improves $Acc_\text{mem}$ by 0.2%, and the proposed method even hurts $Acc_\text{mem}$.

We respectfully disagree with the reviewer's premise. Our problem is to achieve forgetting only specified classes while maintaining the accuracy of the other classes to be memorized, i.e., to improve $Err_\text{for}$ while maintaining $Acc_\text{mem}$. Achieving both of these is more challenging than merely improving $Acc_\text{mem}$ alone.

To verify our argument here, we report an additional quick analysis in Table H in the attached PDF. "CoOp (White-Box w/ only memorization)" shows the results when white-box tuning by CoOp is performed to minimize the cross-entropy loss over only the classes to be memorized, i.e., forgetting is not performed. We can see a significant improvement in $Acc_\text{mem}$, as expected by the reviewer. This result proves that, even in a white-box setting, achieving both improving $Acc_\text{mem}$ and $Err_\text{for}$ is more difficult than merely improving $Acc_\text{mem}$ alone. Since the black-box setting is generally more challenging than the white-box setting, it is not at all surprising that our method leads to a slight degradation in $Acc_\text{mem}$.

I appreciate the authors' comprehensive responses! Most of my concerns are well-addressed.

About the novelty and contribution, I think if the authors want to claim the novelty of this work on the problem setup itself, i.e., Black-Box Forgetting, this problem should be challenging so that naive application of the existing method is struggled with.

For example, similar work on the Black-Box optimization setup [Oh et al. 2023] clarifies the problems of naive extension of the white-box methods [Bahng et al. 2022] and addresses those problems with their new method. However, in Table 2. of the submitted draft, BBT with a different configuration of CMA-ES, which is a naive extension of the existing method, already achieves very strong performance. This raises doubt about the difficulty of the considered problem setup, and given that, I still lean towards the negative side because the proposed method, LCS, lacks the necessity.

Could the authors give further comment and opinion on this?

---

• Oh et al. 2023, BlackVIP: Black-Box Visual Prompting for Robust Transfer Learning
• Bahng et al. 2022, Exploring Visual Prompts for Adapting Large-Scale Models

The authors' statement on the novelty of problem setup is somewhat convincing, and I want to express my huge gratitude for the extensive responses. However, I would regretfully say that I want to keep my score.

This black-box forgetting problem has not been explored yet, and the authors address it with their new method. The proposed method, latent context sharing (LCS), is designed to reduce the number of learnable parameters for better black-box optimization. I still do not see the necessity of this proposal, and there is a missing link in the connectivity between LCS and the black-box forgetting problem.

To be specific, the LCS method is not tailored for "forgetting," and it can be regarded as a method for improving parameter efficiency that will be applied to other PEFT scenarios. The weak connection between the LCS method and the problem setup is the main reason that I kept my score towards rejection.

I am so sorry for my decision-invariance attitude, but I want to adhere to my rating.

These long discussions might cause your mind severe damage, and I am regretful for keeping my statement due to the unsolved concern.

I can not agree with the authors' claim that "LCS is designed for forgetting." Figure 5 is just an empirical result, where the authors provide post-experiment interpretation based on the results.

To be specific, it is only a consequential interpretation, and the design of the methodology cannot be justified as it is for forgetting. That is why I think the connection between the problem definition and the derived methodology is weak, and I will try to stick to this position.

However, since I found some good implications from this study and other reviewers are looking positively, I will not be upset even if this paper is accepted.

Sorry again,

Reviewer SKod

Q1. Why must we forget some classes?

We assume that the reviewer already acknowledged the benefit of preventing information leakage through forgetting. In addition to this, we here would like to emphasize the potential benefits of exploring selective forgetting.

1. Toward addressing the "Right to be Forgotten": If a service provider is asked to remove information so that their model cannot recognize certain information, it might need to comply with the request. This can be accomplished by training the model from scratch by removing samples of that class from the training data. However, retraining a large-scale model consumes an enormous amount of energy, which should be avoided. Selective forgetting may provide an efficient solution to this problem.
2. Toward efficient large-scale pre-trained models: Improving the efficiency of large-scale pre-trained models is of current interest to many researchers. Various attempts have been made such as model compression and architecture optimization (e.g., https://sites.google.com/view/elvm/program). Meanwhile, as the "scaling law" indicates, the reasonable size of a model correlates with the amount of knowledge it memorizes. Therefore, if the number of classes (amount of knowledge) that can be recognized by the model is limited, the model can be scaled down accordingly, thereby improving the efficiency of the model. This contributes to expanding the applicability of large-scale pre-trained models.
3. Toward better control over text-to-image generation: While diffusion-based text-to-image generation can generate diverse types of high-quality images, controlling the content of images remains a challenge. Recent research has focused on "forgetting" visual concepts in order to avoid generating undesirable content [a-c]. These methods forget certain classes by directly fine-tuning the diffusion model, but tuning the diffusion model itself is often costly. In contrast, given that typical generative models use a text encoder of a pre-trained VLM (e.g., CLIP) for conditioning, our method may provide an efficient approach to class forgetting by fine-tuning only the prompts of the text encoder.

We believe that our method will open new directions for these important problems of interest to the ML community, even if these are not immediately feasible with this paper alone. We would like to add the discussion as a broad impact of this work in the final version.

[a] Heng et al., Selective Amnesia: A Continual Learning Approach to Forgetting in Deep Generative Models, In Proc. NeurIPS, 2023.

[b] Lu et al., Mace: Mass Concept Erasure in Diffusion Models, In Proc. CVPR, 2024.

[c] Zhang et al., Forget-me-not: Learning to Forget in Text-to-Image Diffusion Models, In Proc. CVPR, 2024.

Q2. Forgetting vs. Learning from scratch.

As requested by the reviewer, we show the accuracy of ResNet-18 and ViT-B/16 trained from scratch over only the classes to be memorized in Table D in the attached PDF. As can be seen, both models cause severe overfitting to the training data. That is, while these models achieve reasonable accuracy on the training data, they exhibit severely poor performance on the test data. We also tested both models when initialized with ImageNet pretrained weights. While the results improve somewhat for ResNet-18, these are still far behind our forgetting-based method. The reason for this is that, following the common protocol in context optimization [a], we conducted our experiments in few-shot scenarios as explained in Line 201, which overwhelmingly lacks the number of training samples to learn the weights for even ResNet-18 and Vit-B/16.

[a] Zhou et al., Learning to Prompt for Vision-Language Models, IJCV, 2022.

Q3. Is the method applicable to detection / segmentation?

It would be possible by combining with some detectors / segmentors. For example, recent approaches in object detection rely on a two-stage framework [a,b], which first uses region proposal network to get object proposals and then applies zero-shot CLIP to each proposal to identify the object class. Replacing the zero-shot CLIP with one tuned by our method would prevent detecting classes that have been forgotten.

[a] Zhao, et al. Exploiting unlabeled data with vision and language models for object detection, in Proc. ECCV, 2022. [b] Zhao, et al. Taming Self-Training for Open-Vocabulary Object Detection, in Proc. CVPR. 2024.

Q4. Ours vs. Solely Fine-tune CLIP on memorized classes?

We tested the performance of Solely Fine-tune CLIP on memorized classes on four datasets including ImageNet30. The results are shown in Table E in the attached PDF. As we can see, ours and Solely Fine-tune CLIP are comparable in $ACC_\text{mem}$ for the three datasets, except for CUB-200-2011. That is, even if we perform forgetting, it does not significantly impair accuracy on the memorized classes.

Q5. In the white-box setting, one can update more model parameters beyond just the input prompts.

Yes, but it does not work. We tested "CoOp (White-box) + Parameter Update", i.e., updating not only the learnable contexts in the prompt but also the model parameters. The results are shown in Table F in the attached PDF. We see that simultaneously updating the model parameters does not improve performance, but rather hurts it. This is not surprising, as it is known that straightforward fine-tuning of the zero-shot CLIP does not improve performance [a].

[a] Wortsman et al., Robust Fine-Tuning of Zero-Shot Models, in Proc. CVPR, 2022.

Q6. The proposed method still requires access to the embeddings and output logits, I do not consider this a reason for rejection.

Thank you for this fair consideration of the limitation described in our original paper (Sec. 6). In fact, existing black-box tuning methods such as BBT and CBBT assume the same setup, which we followed. On the other hand, the consideration of a complete black-boxing approach is an interesting challenge that we would like to pursue.

Q1. Could the same result not be achieved using just the class name?

Good suggestion! We tried to tune the latent contexts by only using the class names (i.e., class embeddings). Specifically, let $z_c$ and $z$ denote the class embeddings before and after prompt tuning for the class to be forgotten, respectively. Only $z$ is trainable. We aim to tune $z$ by minimizing the following negative NT-Xent loss, $\log \frac{\exp(z_c ^\top z / \tau)}{\sum_i \exp(z_i ^\top z / \tau)}$, where $z_i$ is the class embedding for the $i$-th class to be kept memorized. This loss requires $z$ to be orthogonal to $z_c$ as well as be similar to the embeddings of the other classes $\\{z_i\\}$. The results are reported in Table B in the attached PDF. We found that Ours is much better than the approach using only the class embeddings (C-Emb.), which proves that tuning with only the class embeddings does not provide satisfactory performance.

Q2. 512-D should not be high-dimensional.

Derivative-free optimization based on multi-point search, such as CMA-ES, is directly affected by the curse of dimensionality, so the number of dimensions exceeding, typically 10-D, is often considered high-dimensional [a]. This is why the existing black-box tuning method, BBT, optimizes low-dimensional (around 10-D) latent contexts instead of the raw 512-D embeddings.

[a] Hansen et al., Reducing the Time Complexity of the Derandomized Evolution Strategy with Covariance Matrix Adaptation (CMA-ES). Evolutionary Computation, Vol. 11, No. 1, pp. 1-18, 2003.

Q3. Given the pretrained model, extracting information of the model should not be challenging.

Following the same setup as in previous black-box tuning (e.g., BBT and CBBT), we assume that no pre-trained model is given and can only access to the input/output of the pre-trained model (e.g., via an API). In such a case, we cannot know the internal information of the model and cannot extract information about the model architecture or its parameters.

Q4. Machine Unlearning vs. Black-Box Forgetting.

These two are closely related but different. Machine Unlearning typically aims to remove the influence of specified training samples on the training model, whereas Black-Box Forgetting aims to prevent the recognition of specified classes. Furthermore, we in this paper address the black-box setting, which has not yet been well-traveled in machine unlearning.

Q5. Experiments on more diverse datasets, such as ImageNet-1K?

Although we were not able to complete the experiment with ImageNet-1K in this short rebuttal period, we instead report the results of additional experiments with its subset, ImageNet30 [a]. The results shown in Table G in the attached PDF demonstrate that our method outperforms all the methods in $H$. We will add the results on ImageNet-1K in the final version.

[a] Hendrycks et al., Using Self-Supervised Learning Can Improve Model Robustness and Uncertainty, in Proc. NeurIPS, 2019.

Q6. Why is $Acc_\text{mem}$ lower than the baselines?

This is because $Err_\text{for}$ and $Acc_\text{mem}$ are in a trade-off relationship, with $Acc_\text{mem}$ tending to decrease as $Err_\text{for}$ is increased. This is presumably because features between classes are not completely disentangled in the feature space, so forgetting one class may negatively affect other classes (just as dog and cat share come common features). To provide justification for this, we report the results of using a loss more prioritized (weighted) for $Acc_\text{mem}$ in our method in Table C as "Ours ($Acc$ prio.)" in the attached PDF. We can see that Ours ($Acc$ prio.) outperforms all the other methods in $Acc_\text{mem}$, with sacrificing $Err_\text{for}$. Notably, both Ours and Ours ($Acc$ prio.) outperform BBT and CBBT in $H$, indicating that our method achieves a better trade-off than BBT and CBBT.

Q7. Can the proposed approach be applied in a continual learning setup?

Thank you for your interesting suggestion! While application of our method to continual learning is not in the scope of this paper, theoretically we think it is possible. Our method is based on a context optimization as CoOp. So it can readily be coupled with some of standard continual learning methods, such as distillation-based [a].

[a] Li et al., Learning without Forgetting, IEEE TPAMI, Vol., 40, No.12, pp. 2935-2947, 2017.

Q1. CMA-ES is covered quite shortly.

We will expand the description of CMA-ES in the first paragraph of Line 105 in Sec. 3 as follows:

We employ CMA-ES, a widely used evolutionary algorithm for black-box optimization in continuous, because a textual prompt to be optimized is a continuous variable. CMA-ES is a multi-point search algorithm based on a multivariate normal distribution and proceeds the search by iterating (i) sampling candidate solutions, (ii) evaluating the loss values of the candidates, (iii) weighting the candidates based on the loss values, and (iv) updating the mean and covariance matrix of the distribution by using the weighted candidates. Due to the nature of multi-point search, the performance of CMA-ES degrades in high-dimensional problems, typically ten or more dimensions [a,b]. While several extensions have been proposed, e.g., [a,b], these methods require knowledge of independence among variables, which is not always known. In this paper, we propose a customized extension of CMA-ES to Black-Box Forgetting.

[a] Ros and Hansen, A Simple Modification in CMA-ES Achieving Linear Time and Space Complexity, in Proc. PPSN, 2008.

[b] Akimoto and Hansen, Projection-based Restricted Covariance Matrix Adaptation for High Dimension. in Proc. GECCO, 2016.

Q2. LCS could be explained a bit more elegantly.

To facilitate intuitive understanding of our LCS, we add at the beginning of the LCS section (Line 139) the inspiration behind our LCS and the justification for the composition of ULC and SLC.

Fig. 2c shows the overview of LCS. The key idea is to assume shared parameters among different latent contexts. This inspiration comes from successful word embedding methods; most word embedding methods are trained on the assumption that locally co-occurring words have semantic correlations between them (e.g., [a-c]). This inspires the idea of explicitly modeling semantic correlations between words in a prompt as shared components.

[a] Mikolov, Efficient Estimation of Word Representations in Vector Space, arXiv pre-print 1301.3781, 2013.

[b] Pennington, et al., GloVe: Global Vectors for Word Representation, in Proc. EMNLP, 2014.

[c] Devlin et al., BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding, in Proc. NAACL-HLT, 2019.

Q3. Sec. 4.3 could be reorganized.

As suggested, we will move 4.3.1 and 4.3.5 to the appendix to put priority to the analyses on the effectiveness of the parametrization of our LCS. Thanks!

Q4. Some nomenclature and texts in figures and equations could be improved.

As suggested, we will revise the numbering of Eq. 2 and the name of the loss (i.e., change "CE" in Fig. 1 to "uniform") in the final version. The text in Fig. 1 will be enlarged. The superscript of $z$ is to distinguish shared components from unique components, so is essential for our LCS.

Q5. Some more contexts on machine unlearning?

We will add a review of machine unlearning in the Related Work section in the final version:

Machine unlearning aims to remove an arbitrary sample from a pre-trained model, i.e., obtaining a model that is identical to the one trained from scratch without that sample [a-g]. Many methods have been proposed, for example, to construct a forgettable model by transforming the learning algorithm into a sum of the training samples [a], to achieve forgetting by linear approximation of a nonlinear model [b], and to update the model to be closer to / farther from the original model in the retain / forget samples [e]. Methods specific to certain learning algorithms such as LDA [f] and SVM [g] have also been explored. Machine unlearning and Black-Box Forgetting are closely related but different; Machine unlearning aims to remove the influence of specified training samples on the training model, whereas Black-Box Forgetting aims to prevent the recognition of specified classes. Forgetting specified classes has attracted much attention recently in various contexts [h-l]. We in this paper address the black-box setting, which has not yet been explored.

[a] Cao and Yang, Towards Making Systems Forget with Machine Unlearning. In Proc.IEEE Symp. Security and Privacy, 2015.

[b] Golatkar et al., Mixed-Privacy Forgetting in Deep Networks, In Proc. CVPR, 2021.

[c] Sekhari et al., Remember What You Want to Forget: Algorithms for Machine Unlearning, In Proc. NeurIPS, 2021.

[d] Bourtoule et al., Machine Unlearning, In Proc. IEEE Symp. Security and Privacy, 2021.

[e] Kurmanji et al., Towards Unbounded Machine Unlearning, In Proc. NeurIPS, 2024.

[f] Guo et al., Certified Data Removal from Machine Learning Models, In Proc. ICML, 2020.

[g] Chen et al., A Novel Online Incremental and Decremental Learning Algorithm based on Variable Support Vector Machine, Cluster Computing, 2019.

[h] Heng et al., Selective Amnesia: A Continual Learning Approach to Forgetting in Deep Generative Models, In Proc. NeurIPS, 2023.

[i] Lu et al., Mace: Mass Concept Erasure in Diffusion Models, In Proc. CVPR, 2024.

[j] Zhang et al., Forget-me-not: Learning to Forget in Text-to-Image Diffusion Models, In Proc. CVPR, 2024.

[k] Shibata et al., Learning with Selective Forgetting, in Proc. IJCAI, 2021.

[l] Ye et al., Learning with Recoverable Forgetting, in Proc. ECCV, 2022.

Q6: Hyperparameters tuned on validation sets or test sets?

All the hyperparameters were tuned on validation sets. We will clarify this in the final version.

Q7: Classes to be forgotten determined at random or intentionally?

These were determined randomly for fairness. We will clarify this in the final version.

Q8: No error bars reported.

Good suggestion! All the results in our paper are averages of three runs with different random seeds. Check out Table A in the attached PDF, which is Table 1 of our submission with the standard deviations added. Ours clearly outperforms the compared methods. We will add standard deviations to all the results in the final version.

following on (a) and (d): yes, that is what I meant. I also agree that it is a larger question than what the paper covers currently. I appreciate the results provided and the thorough explanation with the new results. I agree that the insights from the discussion are indeed of interest for further research!

Overall, my questions have been answered. Thanks to the authors for their thorough responses. Some clarifications within the discussion would be needed for the final version, but I have no further questions. I think the questions from the other reviewers and the author responses to them are also satisfactory. Some of them highlight some of the limitations of the proposed setting and also should be included in a final version. The only comment from a reviewer that I also consider important to note, is how simple the datasets used are (Q4 from SKod), and agree with the observation. The results on ImageNet-30 by the authors are good step to solve that issue, although I would be cautious of results obtained during a rebuttal (not due to suspicion of malice, but just due to the fast implementation). However, I am satisfied with the experiments provided. Therefore, I would be in favour of accepting the paper.