NegMerge: Sign-Consensual Weight Merging for Machine Unlearning

We thank the reviewer for the constructive questions. We have done our best to address the concerns raised, as detailed below.

Weakness 1: Why Model Merging for Machine Unlearning?

This is because models fine-tuned with different hyperparameters tend to specialize in either unlearning or retaining (as shown in Figure 1). Therefore, while searching for effective hyperparameters is a prerequisite for approaching optimality, we argue that merging them provides a way to combine their strengths and resolve the trade-off that a single model cannot, which also holds in unlearning. As shown in Figure 1(a), no single model achieves high performance on both objectives.

We believe model merging is effective for unlearning because it provides access to multiple optimization trajectories, which allows us to identify parameters that are related to the forget set (e.g., via sign consensus). This information is not available from a single model. By selectively negating these parameters while preserving others, we can more precisely remove unwanted information without degrading performance on the retain set. As shown in Table 1, the merged model consistently outperforms all single models. We will clarify this point in the final paper.

Weakness 2: Mitigating Hyperparameter Sensitivity via Merging

Due to the sensitivity issue, obtaining a model with strong unlearning performance inevitably requires a hyperparameter search process. However, rather than selecting a single best model from this search, we mitigate the sensitivity by merging all fine-tuned models obtained from diverse hyperparameter configurations. This strategy is inspired by the model soups framework (Wortsman et al., 2022), which shows that averaging the weights of independently fine-tuned models can improve performance and robustness. The key idea is that different hyperparameter choices introduce independent variations, and merging helps cancel out noise from any single run. As a result, the final model is less sensitive to specific hyperparameter choices, while still capturing the shared structure needed for effective unlearning. We will highlight this point in the final paper.

Weakness 3: Clarifying Metrics in Table 2 (Acc Dr, Acc Df, MIA, etc.)

Each metric (Acc Dr, Acc Df, Acc Dtest, MIA) is favorable when its value is equal to that of the Retrain model (“≃”), indicating that the ideal Unlearn model needs to behave similarly to the Retrain model. This notation will be clarified in the caption of Table 2.

Weakness 4: The “Conflict” Ablation in Table 3

The weaker performance of the "Conflict" method suggests that using conflicting-sign parameters is not effective. As shown in Section 4.3, it degrades unlearning performance. In contrast, the "Consensus" method, which uses only sign-consistent elements, performs best, indicating that consistent signs better capture changes tied to the forget set, while conflicting signs likely reflect noise from different training setups.

We thank the reviewer for recognizing our method’s originality. Below, we address concerns on LLM experiments and related work.

Weakness 1: Lack of LLM experiments

We agree that unlearning in LLMs is an important direction. We believe our method can generalize to transformer-based LLMs given its demonstrated effectiveness on transformer architectures, although the current study focuses on image classification tasks. We appreciate the suggestion and will include LLM experiments and discussion on our method’s applicability to LLMs in the final version of the paper.

Weakness 2: Additional references and deeper intuition (CCA, task localization)

1. Connection to Canonical Correlation Analysis (CCA)

As the reviewer suggested, CCA-based methods [1,2] have been used to analyze similarity between layers of neural networks. Indeed, it would be interesting to adapt such a viewpoint to identify which layers are most effective for unlearning. Our work, however, starts from the assumption that certain parameters—rather than entire layers—are more central to the given task. We thus focus on parameter-level unlearning, where we isolate specific parameters that are consistently implicated in the task across multiple models. We agree that layer-level unlearning via CCA is a promising direction, and will add discussion.

2. Relation to Task Localization / Merging Methods

We appreciate the additional pointers to task localization methods [3,4,5]. In our Related Work section, we cite some approaches (e.g., TIES-Merging, AdaMerging, MagMax) that also address how to combine or localize tasks. However, these methods focus largely on task addition, attempting to merge models trained on different tasks to maximize the differences that define each new task. For instance, the Tall Mask method [3] seeks “taller” vectors for merging.

In contrast, our goal is to identify parameters shared across models trained on similar tasks. Rather than negating the most diverse features, we look for the parameters that share a consistent effect (sign) across models. These shared parameters most effectively capture the task vectors that are attributed to the forget set and are therefore the prime candidates for unlearning. We will further clarify this difference in the revised paper.

Suggestions 1: Lack of retain set (Dr) performance in Table 4

We have added the full Table 4 results (please see Reviewer aohJ, Weakness 4) and will clarify further in the final version.

Suggestions 2: “Avg gap” metric not clearly defined

Following SalUn’s evaluation protocol, we report the Avg. Gap, which is the average difference between the Unlearn and Retrain models across Dr, Df, Dtest, and MIA. We will add further clarification in the final version.

Suggestions 3: Sign consensus analysis

We conducted several analyses using ViT-B/32 and ViT-L/14 on the Cars dataset with 30 models. First, we examined the block-wise consensus ratio. As shown in the tables below, the consensus ratio increases in deeper blocks, in both architectures. This suggests that more parameters are being negated as the blocks go deeper. Here, we grouped blocks into three ranges and reported the average consensus ratio per range. Each block corresponds to one transformer block.

ViT-B/32:

ViT-L/14:

Additionally, we compared the Stem layer (i.e., the patch embedding), MLP layers (c_fc and c_proj weights/biases), and Attention layers (query, key, value, and output projection weights/biases). The standard deviation refers to the variability across different layers within each category (MLP or attention). Surprisingly, the Attention layers consistently showed 100% sign consensus in both architectures. We hypothesize that this is because the query-key-value mechanism plays a fundamental role in capturing contextual relationships, which remain stable across different fine-tuning configurations.

ViT-B/32:

ViT-L/14:

We will include these findings, which we believe enhance the paper’s depth and quality. We sincerely thank the reviewer for the helpful suggestion that led to this.

Suggestions 4: NTK reference

We will ensure that our next revision will correct it.

We appreciate the reviewer’s positive comments on the clarity of our paper and for recognizing the potential of our method as an alternative to parameter-based unlearning approaches. The reviewer has expressed concerns regarding the generalizability of our work and questions regarding instance-wise unlearning. We hope the following feedback addresses these concerns.

Weakness 1: Limited evaluation: class unlearning, more datasets, and different architectures

We provide additional results for class-wise unlearning and 50% random data forgetting on CIFAR-10 using ResNet18, using only the forget set. As shown below, our method consistently surpasses baseline methods. Please refer to our response to Reviewer aohJ (Weakness 3) for results on the 50% random forgetting setting.

Class-wise Unlearning Results:

Additionally, we present new results on 10% random data forgetting conducted on TinyImageNet and Swin-T. With this, we now evaluate instance unlearning across three datasets—CIFAR-10, TinyImageNet, and CUB—using three architectures: ResNet-18, VGG-16, and Swin-T. Our approach consistently outperforms baseline methods across various datasets and architectures, including TinyImageNet and Swin-T. Please find our new results below.

On the TinyImageNet dataset:

Using Swin-T:

Weakness 2: Is instance unlearning truly happening?

In the standard classifier unlearning scenario, the Retrain model is trained from scratch using only the retain set and excluding the forget set, and has been regarded as the (ground-truth-like) upper bound of performance, as none of the forget set is used during training. The goal is for the Unlearn model to closely match the Retrain model's performance on each of Dr, Df, Dtest, and MIA individually, which has been the standard evaluation protocol in this scenario. In instance-level unlearning, classification accuracy on the forget set often remains similar, as observed in the corresponding table. This is presumably due to the generalization effect: even though the Retrain model is not trained on the forget set, it can still perform well on it by leveraging class-level information learned from the retain set.

We appreciate the reviewer’s positive feedback on the clarity and intuitiveness of our method. We hope to address the concerns regarding generalizability and scalability in our responses below.

Weakness 1: No theoretical claims were provided

We provide a theoretical claim for our method. We would like to note that an informal version was presented in Appendix E.

Theorem. Consensus-based merging of multiple task vectors yields a robust and effective unlearning direction by inducing sparsity and guiding the model toward a low-loss region.

Lemma 1. As the number of task vectors $\tau_k$ increases, the merged vector $\tau_{merged}$ becomes sparser.

Proof. Consensus merging keeps each element only if all task vectors agree in sign, acting like an AND operation. With more vectors, agreement decreases, increasing sparsity.

Lemma 2. If $\theta\_{ft}$ is centered around $\theta\_{ori}$, then $\theta^*\_{unlearn} = \theta\_{ori} - \tau\_{merged}$ lies closer to $\theta\_{ori}$.

Proof. Given $\tau\_{merged} = \bar{\theta}_{ft} - \theta\_{ori}$,

we have \theta^*\_{unlearn} = \theta\_{ori} - (\bar{\theta\}_{ft} - \theta\_{ori}).

If $\bar{\theta}_{ft} \approx \theta\_{ori}$, then: $\theta^*\_{unlearn} \approx \theta\_{ori}$.

Lemmas 1 and 2 imply that consensus merging improves robustness (via sparsity) and effectiveness (by staying near $\theta_{ori}$ under linear mode connectivity), yielding a reliable unlearning direction.

While our theoretical analysis may offer a reasonably plausible justification, its effectiveness was most clearly demonstrated through strong empirical results across diverse datasets and architectures. We consider a more rigorous theoretical analysis as future work.

Weakness 2: Inability to conduct experiments on ViT-L/14

ViT-L/14 results for Task Arithmetic were actually included in Table 1. For Linear Task Arithmetic, we ran a scaled-down experiment on ViT-L/14 using six models on the Cars dataset to complete it within the rebuttal period. As shown in the table below, ours outperforms the baselines, achieving the lowest forget accuracy (Df ↓) while maintaining comparable retain accuracy (Dr).

Weakness 3: Evidence is not enough (50% random forgetting and class-wise unlearning)

We extend Table 2 with additional experimental results under 50% random data forgetting and class-wise unlearning, using only the forget set. These experiments were conducted on the CIFAR-10 dataset with the ResNet18 architecture. As shown below, our method outperforms baselines in unlearning performance. For class-wise unlearning and 10% forgetting on TinyImageNet and Swin-T, please see our response to Reviewer zcBE (Weakness 1).

50% Random Data Forgetting Results:

Weakness 4: Lack of retain set (Dr) performance in Table 4

In Section 4.2, we mentioned that the retain set (Dr) accuracies for all methods remain around 60%, following the implementation details of Ilharco et al. (2022) for the CLIP unlearning scenario. Below is the extended table (Table 4) including both Df and Dr results. We will provide further clarification in the final version.

Suggestions 1: Minor typos

We appreciate the reviewer’s notice. We will carefully proofread again before the final submission.

Questions 1: Is the proposed method efficient when the pool size is large?

Although our method scales with the number of fine-tuned models, it does not incur additional overhead, since all methods—including single-task arithmetic—share the same underlying model pool. Comparable cost increases also arise in single-task arithmetic due to its broader hyperparameter search space. This is discussed in detail in Section 3.3, Analyses on Computational Cost.