Approximate Domain Unlearning for Vision-Language Models

We sincerely thank Reviewer tNbB for their thoughtful and constructive review. We are especially grateful that the reviewer recognized all the main contributions of our work, including the introduction of Approximate Domain Unlearning (ADU) as a novel problem setting, the effectiveness of our proposed solution, and the clarity of our presentation. We believe that, as the first study to explore ADU, our paper can meaningfully contribute to the community. We also appreciate that the reviewer's comments were framed not as criticisms of fundamental flaws, but rather as constructive suggestions for further development. We view these points as valuable opportunities to deepen and expand our work, and we address them point-by-point below.

Q1. Domain labels not always fully available.

As we discuss in the Limitations section of our paper, we acknowledge that domain labels may not always be available for all training samples in real-world scenarios. However, it is important to note that most standard formulations in domain adaptation and domain generalization (e.g., [a–c]) assume full access to domain labels during training. Therefore, our setting aligns with this widely accepted convention and should not be considered an unrealistic simplification.

Meanwhile, to evaluate the performance of our method under partial domain label availability, we consider a setting in which a portion of the training samples (ranging from 30% to 70%) lack domain annotations. To assess the effectiveness of combining domain estimation with our method in such cases, we adopt a simple pseudo-labeling approach: a domain classifier is trained using only the samples with known domain labels and then used to assign pseudo labels to the unlabeled samples.

As shown in Table C-1, without domain estimation, the $H$-score drops from 75.89% to 49.97% under 70% missing domain labels. In contrast, with domain estimation, the $H$-score remains substantially higher at 61.21% even under the same missing rate. These results support our claim that our approach can be extended to more realistic cases where only partial domain annotations are available.

Table C-1. Performance on Office-Home with Partial Domain Labels.

Q2. Choice of kernel and batch formation for MMD?

All results reported in the paper use the RBF (Gaussian) kernel. To assess sensitivity to kernel choice, we additionally tested four different kernels on Office-Home in Table C-2. Among them, the linear, Laplacian, and RBF kernels demonstrated generally stable performance, with only modest differences in accuracy. When validation-based kernel selection is feasible, we recommend choosing the best among these three; otherwise, the linear kernel serves as a conservative and reliable default.

Regarding batch formation, we use a batch size of 128 and randomly select 128 samples per batch from the entire training data.

Table C-2. Impact of Kernel Choice.

Q3. Concern about limited representation change in early layers due to model freezing.

Many prior studies have proven that fully fine-tuning CLIP often leads to catastrophic forgetting, especially in few-shot regimes [e, f]. Prompt tuning with frozen model parameters has thus become a standard alternative, which we follow. As shown in Table C-3, updating model parameters in our task indeed causes severe catastrophic forgetting, reinforcing this concern.

Despite freezing, our method affects both low- and high-level representations. Fig. 7 shows stable unlearning performance across various prompt insertion depths, including early layers. This suggests that the learned prompts induce meaningful changes throughout the network, addressing the concern about limited adaptability in early stages.

Table C-3. Impact of Updating Model Parameters on Office-Home.

Q4. Robustness to Partial Domain-Class Overlap.

We conducted additional experiments on Office-Home where samples within three random classes (specifically, mug, radio, and hammer) were excluded from only "art" and "clipart" domains, creating a setting with partial domain-class overlap. As shown in Table C-4, even under such imbalance, our method maintains competitive $H$-scores. These results suggest that our method can handle realistic class-domain imbalances without critical loss in effectiveness.

Table C-4. Robstness to Partial Domain-Class Overlap.

Q5. Robustness to ambiguous domain definitions.

We agree that domains may overlap semantically or visually, making clear boundaries difficult. To examine this, we report per-domain accuracy on Office-Home in Tables C-5. Our method demonstrates selective forgetting and memorization even between highly similar domains, indicating effective disentanglement despite domain ambiguity.

For robustness to label noise, our response to Q1 may offer a partial answer, as it addresses the case where domain labels are only partially available — a setting that can also be interpreted as involving random label noise on unlabeled samples (Table C-1).

Table C-5. Per-domain Accuracy on Office-Home.

Q6. The method can forget domain knowledge without hurting class knowledge?

Yes, this is precisely the goal of ADU. The task is to forget domain-specific information (e.g., "sketch") while preserving class-relevant, domain-agnostic features. As shown in Table 1, our method significantly improves the forgetting score ($For$) on the target domain, while maintaining the memorization score ($Mem$) on the rest. These results demonstrate that the model can forget domain-specific representations of a class (e.g., "sketch" dog) without degrading its classification accuracy on the same class in other domains (e.g., "real" dog).

Q7. How $Mem$ and $For$ scores evolve over training?

As shown in Table C-6, we see that $For$ increases rapidly while $Mem$ drops in the early stages, suggesting a conflict between the two. However, both socres improve steadily after this phase, indicating that the model gradually learns to balance forgetting and retention. These results suggest a two-stage process: initial domain disentanglement followed by controlled forgetting and memorization.

Table C-6. $Mem$ and $For$ over Training Iterations on Office-Home.

Q8. The method work with plain vision encoders?

Thank you for this valuable suggestion! Our primary focus is on pre-trained VLMs, particularly CLIP, where domain-specific representations are highly entangled. While our DDL and InstaPG are in principle applicable to plain vision encoders, the absence of public pre-trained vision-only models with domain generalization capabilities comparable to CLIP makes immediate evaluation difficult. We are interested in exploring this point in the final version.

---

References

[a] Wu et al., Test-Time Domain Adaptation by Learning Domain-Aware Batch Normalization, AAAI 2024.

[b] Cho et al., PromptStyler: Prompt-driven Style Generation for Source-free Domain Generalization, ICCV 2023.

[c] Jhoo and Heo, Collaborative Learning with Disentangled Features for Zero-shot Domain Adaptation, ICCV 2021.

[d] Mitsuzumi et al., Generalized Domain Adaptation, CVPR 2021.

[e] Zhou et al., Learning to Prompt for Vision-Language Models, IJCV 2022.

[f] Kumar et al., Fine-tuning Can Distort Pretrained Features and Underperform Out-of-Distribution, ICLR 2022.

We sincerely thank Reviewer gH1G for their thoughtful and constructive review. We are especially grateful that the reviewer clearly acknowledged all the main contributions of our paper, including the novelty of the problem setting, the effectiveness of our proposed solution, and the comprehensiveness of our experiments. We believe that our paper makes the first attempt at approximate domain unlearning in Vision-Language Models and can meaningfully contribute to the community by addressing key challenges of this emerging task. Below, we provide point-by-point responses to all questions and concerns raised. All the comments are constructive and help strengthen our work. We also appreciate that none of the points raised indicate fundamental flaws in our core ideas or methodology.

Q1-1. 8 training samples per domain quite small.

Tuning of VLMs is typically performed under a few-shot protocol [a-d], and recent work on unlearning for VLMs [e] also adopts this standard setting. Accordingly, we follow the same protocol. As shown in Appendix A.1, we report results for varying numbers of samples (from 1 to 32), where our method consistently and significantly outperforms the baseline across all cases.

Q1-2. Results on more complex domain distributions or larger datasets?

We additionally evaluated our method on DomainNet, an established multi-domain benchmark dataset that comprises 177K test samples over six domains and 345 classes, making it one of the complex and large-scale datasets of its kind. The results are shown in Table A-1. Ours is clearly better than the baseline (the best competitor compared with our method in our main paper), which emphasizes the strong effectiveness of our method.

Table A-1. Results on DomainNet.

Q2. Discussion on ambiguous or similar domains, with a note on failure cases.

To the best of our knowledge, there is currently no established multi-domain benchmark datasets having ambiguous domain boundaries. Instead, we report per-domain accuracy on Office-Home, Mini DomainNet, and DomainNet, those which include domains with substantial visual similarity (e.g., "sketch" vs. "quickdraw").

The results are shown in Tables A-2 to A-4 below. Overall, they demonstrate that our method can selectively suppress classification accuracy on the forgotten domain while maintaining performance on the others, even those with similar visual properties. For instance, in Mini DomainNet, forgetting "sketch" reduces its accuracy significantly from 72.54% to 20.64%, whereas similar domains such as "clipart" and "painting" experience only minor drops (within 1%). These results indicate that our method provides fine-grained control even in the presence of overlapping domain features.

Regarding a failure case, we found that unlearning can be more difficult for certain domains. For example, when forgetting "real" in Office-Home, our method decreases the accuracy from 81.29% to 55.48%, which, although significantly lower than the original zero-shot CLIP, still retains moderate recognition ability. One possible reason is that CLIP is extensively pre-trained on image-text pairs dominated by real-world photos, making the "real" domain more strongly encoded and harder to forget. Nonetheless, as shown in Table 1 of the main paper, our method consistently achieves strong forgetting performance on average across datasets. These results suggest that although our approach is generally effective, slight variations may occur depending on the VLM’s pretraining bias; as a potential remedy, one may consider incorporating techniques for mitigating dataset bias.

Table A-2. Per-domain Accuracy on Office-Home.

Table A-3. Per-domain Accuracy on Mini DomainNet.

Table A-4. Per-domain Accuracy on DomainNet.

Q3. Computational complexity?

Table A-5 below summarizes GPU memory usage and training time with NVIDIA RTX A4000 GPU on Office-Home. While our method incurs only slightly higher computational cost than lightweight baselines such as CoOp and LP++, it remains on par with advanced CLIP fine-tuning methods such as MaPLe and CLIPFit. These results prove that our method is sufficiently efficient for practical use. Notably, adding InstaPG increases memory usage by only 1 GB and training time by just 1 minute, indicating minimal overhead.

Table A-5. Computational Complexity.

Q4. Why not report domain classification accuracy?

According to the reviewer's suggestion, we reported domain classification accuracy before and after unlearning on Office-Home in Table A-6. Accuracy improved significantly from 25.80% before unlearning to 79.43% after unlearning when using our full method. Furthermore, when our two key components, i.e., DDL and InstaPG, are removed, the domain classification accuracy drops substantially to 31.06%, indicating that our method plays a critical role in achieving effective domain separation. We believe these results provide strong additional evidence of the effectiveness of our method, particularly in facilitating domain unlearning. We are grateful to the reviewer for directing our attention to this important perspective!

Table A-6. Domain Classification Accuracy When "Art" is Forgotten.

---

References

[a] Zhou et al., Learning to Prompt for Vision-Language Models, IJCV 2022.

[b] Khattak et al., MaPLe: Multi-modal Prompt Learning, CVPR 2023.

[c] Li et al., Vision-Language Model Fine-Tuning via Simple Parameter-Efficient Modification, EMNLP 2024.

[d] Huang et al., LP++: A Surprisingly Strong Linear Probe for Few-Shot CLIP, CVPR 2024.

[e] Kuwana et al., Black-box Forgetting, NeurIPS 2024.

We sincerely thank Reviewer ZXGU for their thoughtful and insightful review. We are especially grateful that the reviewer clearly recognized all the key contributions of our work, including the novelty of the Approximate Domain Unlearning task, the effectiveness of our proposed methods (DDL and InstaPG), and the clarity and comprehensiveness of our experiments. We believe this paper makes the first concrete attempt at Approximate Domain Unlearning for Vision-Language Models, and we are confident that our contributions will meaningfully benefit the community by highlighting important challenges and proposing practical solutions in this emerging area. Below, we provide detailed responses to each comment and question. We greatly appreciate that all feedback was constructive and aimed at further strengthening the contribution, without pointing out any fundamental flaws.

Q1. Domain labels not always fully available.

As we discuss in the Limitations section of our paper, we acknowledge that domain labels may not always be available for all training samples in real-world scenarios. However, it is important to note that most standard formulations in domain adaptation and domain generalization (e.g., [a–c]) assume full access to domain labels during training. Therefore, our setting aligns with this widely accepted convention and should not be considered an unrealistic simplification.

Meanwhile, to evaluate the performance of our method under partial domain label availability, we consider a setting in which a portion of the training samples (ranging from 30% to 70%) lack domain annotations. To assess the effectiveness of combining domain estimation with our method in such cases, we adopt a simple pseudo-labeling approach: a domain classifier is trained using only the samples with known domain labels and then used to assign pseudo labels to the unlabeled samples.

As shown in Table B-1, without domain estimation, the $H$-score drops from 75.89% to 49.97% under 70% missing domain labels. In contrast, with domain estimation, the $H$-score remains substantially higher at 61.21% even under the same missing rate. These results support our claim that our approach can be extended to more realistic cases where only partial domain annotations are available.

Table B-1. Performance on Office-Home with Partial Domain Labels.

Q2. Robustness of DDL to domain imbalance?

We agree that domain imbalance can arise in practical scenarios, where some domains may contain fewer samples than others (e.g., abundant "real" images vs. sparse "art" or "clipart" images in autonomous driving). To evaluate the robustness of our DDL under such imbalance, we conducted experiments on Office-Home by reducing the number of training samples from selected domains (i.e., "art" and "clipart"), while keeping the other domains fixed. Our default setting uses 8 shots, and we test the performance when this is reduced to 4 or 1 shot(s).

As shown in Table B-2, our method maintains stable performance under domain imbalance. Even with just one sample from each of the "art" and "clipart" domains, the model achieves a $For$ of 73.85 and a $Mem$ of 64.31 for $|D_{for}| = 3$. These results indicate that our method can still perform effective domain disentangling and selective forgetting even in severely imbalanced scenarios.

We appreciate this suggestion, as it offers an additional valuable perspective for further demonstrating the robustness and practical utility of our approach!

Table B-2. Robustness to Domain Imbalance.

Q3. Performance with other VLMs?

Although evaluating on diverse pre-trained VLMs would be ideal, such efforts are limited by the scarcity of publicly available CLIP-like pre-trained models. This would be a reason that most existing studies on CLIP fine-tuning or unlearning (e.g., [f–j]) focus solely on CLIP.

Nevertheless, we report the performance of our method applied to another VLM, SigLIP [e], in Table B-3 using ImageNet. As shown, our method outperforms the baseline (the best competitor compared with our method in our main paper), indicating its strong effectiveness even if not coupled with CLIP. These results demonstrates that our approach is not specific to CLIP and can be extended to other VLMs.

Table B-3. Performance with SigLIP on ImageNet.

Q4. Computation complexity?

Table B-4 shows computation time of our method with and without InstaPG using NVIDIA RTX A4000 GPU. The results indicate that adding InstaPG introduces a very minor overhead, namely an additional 50 seconds for training and only 0.2 seconds for inference.

Table B-4. Training and Inference Times.

---

References

[a] Wu et al., Test-Time Domain Adaptation by Learning Domain-Aware Batch Normalization, AAAI 2024.

[b] Cho et al., PromptStyler: Prompt-driven Style Generation for Source-free Domain Generalization, ICCV 2023.

[c] Jhoo and Heo, Collaborative Learning with Disentangled Features for Zero-shot Domain Adaptation, ICCV 2021.

[d] Mitsuzumi et al., Generalized Domain Adaptation, CVPR 2021.

[e] Zhai et al., Sigmoid Loss for Language Image Pre-Training, ICCV 2023.

[f] Zhou et al., Learning to Prompt for Vision-Language Models, IJCV 2022.

[g] Khattak et al., MaPLe: Multi-modal Prompt Learning, CVPR 2023.

[h] Li et al., Vision-Language Model Fine-Tuning via Simple Parameter-Efficient Modification, EMNLP 2024.

[i] Huang et al., LP++: A Surprisingly Strong Linear Probe for Few-Shot CLIP, CVPR 2024.

[j] Kuwana et al., Black-box Forgetting, NeurIPS 2024.

We sincerely thank Reviewer ubkk for their thoughtful and constructive review. We are especially grateful that the reviewer acknowledged all the main contributions of our paper, including the novelty of the problem setting and the effectiveness of our proposed solution. We believe this paper represents the first attempt at Approximate Domain Unlearning (ADU) in Vision-Language Models (VLMs) and offers a meaningful contribution by tackling key challenges in this emerging area. Below, we provide detailed, point-by-point responses to all questions and concerns raised. We appreciate the constructive feedback, which we believe improve our work.

Q1. Dependence on Domain Labels.

As we discuss in the Limitations section of our paper, we acknowledge that domain labels may not always be available for all training samples in real-world scenarios. However, it is important to note that most standard formulations in domain adaptation and domain generalization (e.g., [a–c]) assume full access to domain labels during training. Therefore, our setting aligns with this widely accepted convention and should not be considered an unrealistic simplification.

Meanwhile, to evaluate the performance of our method under partial domain label availability, we consider a setting in which a portion of the training samples (ranging from 30% to 70%) lack domain annotations. To assess the effectiveness of combining domain estimation with our method in such cases, we adopt a simple pseudo-labeling approach: a domain classifier is trained using only the samples with known domain labels and then used to assign pseudo labels to the unlabeled samples.

As shown in Table D-1, without domain estimation, the $H$-score drops from 75.89% to 49.97% under 70% missing domain labels. In contrast, with domain estimation, the $H$-score remains substantially higher at 61.21% even under the same missing rate. These results support our claim that our approach can be extended to more realistic cases where only partial domain annotations are available.

Table D-1. Performance on Office-Home with Partial Domain Labels.

Q2: Limited Real-world Validation.

To the best of our knowledge, no public autonomous driving dataset contains annotated roadside advertisements and multi-domain versions of objects like vehicles, making direct empirical evaluation infeasible at this time.

Instead, the best possible evaluation we can currently perform is a proxy experiment that partially simulates such scenarios using an existing multi-domain dataset, Mini DomainNet. Specifically, we focus on seven vehicle-related classes (bus, car, motorbike, pickup truck, police car, school bus, and tractor), and evaluate our method when retaining only the "real" domain while forgetting the other illustration domains ("clipart," "painting," and "sketch"). As shown in Table D-2, our method effectively preserves recognition of real vehicles while substantially suppressing classification on illustrated ones. These results suggest that our approach holds promise for practical applications such as autonomous driving or car counting.

We acknowledge that full-scale validation on real-world deployment tasks remains an important direction for future work. Nonetheless, we believe our findings serve as an informative first step toward domain-specific unlearning under realistic constraints.

Table D-2. Per-domain Accuracy on Vehicle-Related Classes in Mini DomainNet.

Q3. Unclear Computational Efficiency.

Table D-3 below summarizes GPU memory usage and training time with NVIDIA RTX A4000 GPU on Office-Home. While our method incurs only slightly higher computational cost than lightweight baselines such as CoOp and LP++, it remains on par with advanced CLIP fine-tuning methods such as MaPLe and CLIPFit. These results prove that our method is sufficiently efficient for practical use. Notably, adding InstaPG increases memory usage by only 1 GB and training time by just 1 minute, indicating minimal overhead.

Table D-3. Computational Complexity.

Q4. DDL reduce accuracy on memorized domains?

Partially true. As shown in Table D-4, the per-domain accuracy on Office-Home illustrates that DDL can have varying effects across domains. For example, when "art" is forgotten, the accuracy for the retained domain "real" slightly drops from 81.29% to 75.91%, whereas the accuracy for "product" improves from 80.43% to 88.72%. These results indicate that the effect of DDL depends on inter-domain correlations and task structure, leading to either small gains or small losses in retained domains. Note that such degradations are modest and do not lead to critical failure. Overall, DDL serves as an effective mechanism for suppressing domain-specific information while maintaining a favorable balance between forgetting and retention.

Table D-4. Per-domain Accuracy on Office-Home.

---

References

[a] Wu et al., Test-Time Domain Adaptation by Learning Domain-Aware Batch Normalization, AAAI 2024.

[b] Cho et al., PromptStyler: Prompt-driven Style Generation for Source-free Domain Generalization, ICCV 2023.

[c] Jhoo and Heo, Collaborative Learning with Disentangled Features for Zero-shot Domain Adaptation, ICCV 2021.

[d] Mitsuzumi et al., Generalized Domain Adaptation, CVPR 2021.