Response to Rev. wwLa

We thank Reviewer for constructive feedback.

1. Ensemble is a general approach that can be applied to other pre-trained models (e.g., LLM).

Thank you. Below is the table showing performance on a multimodal LLM, LLaVA‑1.5‑7B (with CLIP ViT‑L/14 as the vision encoder). We additionally validate our method on the multimodal CLEAR benchmark (Dontsov et al., 2024), which involves fictional author profiles with paired face images and captions. As shown in the table below, we report forgetting and retention accuracy on the VQA task. Our approach achieves the best forgetting while maintaining comparable retention performance.

TABLE

Conceptually, our task‑vector framework extends to other multimodal settings, including open‑vocabulary CLIP tasks and image‑text retrieval. Adapting to those scenarios would require defining task vectors on the corresponding objective (e.g., fine‑tuning CLIP’s multimodal head to capture specific image‑caption associations) and applying our unlearning procedure in the same manner. We will explore these directions in future work.

2. Improving other task arithmetic operations besides unlearning.

While we focus predominantly on unlearning in this work, below we demonstrate incremental learning scheme. The reason why we focus on unlearning is that we model $\boldsymbol\theta\_1,\ldots,\boldsymbol\theta\_Q$ as variations in task vectors for a given unlearning dataset. Therefore, we assume that:

• ${\boldsymbol\theta}\_i$ results only in sparse changes: $\sum\_{j=1}^{86M} {\mathbb 1}( |\theta\_{j,0}-\theta\_{j,i}|>\epsilon)\ll m$ for small $\epsilon$ (86M is the number of parameters for ViT-B).
• maximum diameter between any two task vectors $\eta=\lVert\boldsymbol\theta_i-\boldsymbol\theta_j\rVert_2\quad\forall i\neq j$ is small.

These assumptions are required for modeling with the Dirichlet distribution as per Resp. 1 to Rev. uLHj. For $\epsilon=1e-4$ (CLIP ViT-B/16), we checked that only 4.6% parameters changed (for unlearning) so as long as these assumptions are met (they are easily met for unlearning), we can also consider multi-task learning based on task vector simplices by adding the sum of the task vectors of 8 tasks (datasets): Cars , DTD, SUN397, EuroSAT, GTSRB, MNIST, SVHN, and RESISC45 following Task Arithmetic (Ilharco et al., ICLR 2023). We report the average absolute accuracy (%) w.r.t. CLIP ViT-B/16 in the table below.

TABLE

3. Line 151, the Hessian term may contain subscript.

Absolutely. Thank you.

4. Equation 2, I would suggest using integration, rather than summations.

Absolutely.

5. It is also helpful to specify how to compute the first-order term.

Absolutely, for the standard expansion we have:

$\frac{1}{|\Delta\_\theta|}\sum\_{\boldsymbol\tau\in\Delta\_\theta}\boldsymbol{\tau}=\frac{1}{Q}\sum\_{i=1}^Q\boldsymbol{\tau}\_i,$

while for Corollary 1 we have :

$\frac{1}{|\Delta\_\theta|}\sum\_{(w,\boldsymbol\tau)\in\Delta\_w\times \Delta\_\theta}\\!\\!\\!w\boldsymbol{\tau}=\frac{1}{\sum_{j=1}^Q w\_j}\sum\_{i=1}^Q w\_i\boldsymbol{\tau}\_i.$

6. Parentheses in Eq. 10/11.

Duly noted.

7. Increase $Q$ in Figure 3.

Absolutely. In Figure 3, we provide the comparison between function-level ensembles from task vectors and our task simplex w.r.t. the number of task vectors Q for CLIP ViT-B/32. In the following table, we further include the performance for $Q=40, 50$. We can notice the convergence of unlearning performance..

TABLE

8. In Table 3, does the ensemble method still keep the retain set performance?

Yes, within 95% of the original performance.

9. Discuss limitations of the proposed methods (e.g., additional costs of computing/storing Hessian).

10. Regarding line 604, it would be better to open the code.

Absolutely.

11. The randomness of the ensemble is from the subscript i: should we have a full covariance?

Thank you. This is very interesting question. We assumed each $\mu_i$ for $i=1,\ldots,C$ is independent for simplicity and $f_i$ produces likelihood of the $i$-th class. As some of $Q$ task vectors may be noisy for some classes, we assumed limiting that variance alone a bit may be sufficient - and improvements validate that. However, reducing covariance (off-diagonal terms) could indeed help also decorrelate the model. However, that would require some serious rethinking of the Taylor expansion in Eq. (2) to produce the outer product as a closed form solution because of the integration operator.

12. The Bienaymé formula.

For a fixed sample ${\bf x}$, one may think of $f(\cdot)$ as a transformation of random variable (task vector sampled from the Dirichlet distribution (simplex)) into another random variable living in the class space (surface of a simplex) which follows some resulting from this transformation class distribution. The Bienaymé formula tells us what happens with variance of each class (treated independently) as the number of ensembled functions grows. That number depends on $Q$ task vectors. However, $\rho$ is required to be low. The opposite means each $Q$ functions are not independent - say in extreme case we have $Q$ identical functions, clearly they cannot reduce the variance because they all are identical. For that reason we investigated the idea of learning $w_i$ or even moving $\boldsymbol\tau_i$ in a small radius to help reduce the variance. We agree as well that reducing cross-terms in covariance could strengthen this effect.

Response to Rev. vtZ9

We thank Reviewer for constructive feedback.

1. Providing more intuitive explanations.

Absolutely. The main idea is to devise a distribution easy to construct and sample give $86M$ parameter space (ViT-B). This can be easily achieved by an ensemble of functions based on Taylor expansion which leverages 0th, 1st and 2nd order statistics of these task vectors together. Taylor expansion is devised at $\boldsymbol\theta\_0$.

• We will illustrate such a Taylor expansion and resulting const, linear and quadratic terms. The linear term depends on the mean of task vectors, and the quadratic term on specific second order statistics which we will illustrate. Kindly notice rebuttal does not permit figures (alas).
• We will also illustrate the ensemble in terms of the mean and variance for which we can adjust outliers using weights $w_i$.

1. Divide "Proposed Method” into multiple meaningful subsections.

Absolutely. In general, we have the core "Problem Formulation":

• Problem definition with Closed-form Aggregation of Functions from Task Simplex,

followed by "Extensions" and "Theoretical Analyses":

• Advanced Aggregation Scheme.
• Distillation from Ensemble.
• Computing and Controlling Variance of Ensemble.

2. Sentence with "the state-of-the-art approach by 3.3% on ViT-Base/32-based CLIP" is not directly supported by Table 1.

Absolutely. We had a typo due to changes in the paper. We meant that compared to function ensemble [8] we achieve 7.55%, 7.72% and 6.34% improvement in unlearning on ViT-Base/32, ViT-Base/16 and ViT-Large/14, respectively.

3. "Table 1 shows..." repeats Table 1 twice.

Thank you. We have revised accordingly.

Response to Rev. uLHj

We thank Rev. for constructive feedback.

1. Validate task vectors form a convex simplex.

Thank you. This is a complex question.

Kindly note that we do not claim task vectors follow any specific distribution. Task vectors have $m=86M$ dim. (CLIP ViT-B-16). For low number of vectors ($Q=30\ll 86M$), there exists no technique that can reliably estimate the distribution to tell its kind due to the curse of dimensionality - one needs $Q\gg 86M$ to reliably estimate. Our $Q=30$. $\let\b\boldsymbol\let\th\theta\let\al\alpha\let\la\lambda\let\ta\tau\let\da\downarrow\let\ua\uparrow$ $\newcommand{\fg}{Forget (\da)}\newcommand{\rt}{Retain (\ua)}\newcommand{\tz}{{\b\th}\_0}\newcommand{\ti}{{\b\th}\_i}\newcommand{\tj}{{\b\th}\_j}$

As rebuttal does not allow figures, we will add t-SNE in revision. But we chose Dirichlet (educated choice) as under large $m=68M$ dim., we have:

• simplicity to set/estimate PDF (for Dirichlet, our task vectors set PDF)
• ability to sample from it (our Taylor derivation does that)
• compact support (not exceeding observed min-max values of individual params.) (1a)
• ability to exploit 0th, 1st, 2nd order moments (simplex mean, implicit cov.) in modeling - Dirichlet lets us use of low-order moments in Taylor expansion (1c)

1a. Intuitive reasoning.

Notice that:

• Few-epochs fine-tuning $\tz$ on unlearning dataset toward $\ti$ yields sparse change: $\sum\_{j=1}^{86M} {\mathbb 1}( |\th\_{j,0}-\th\_{j,i}|>\epsilon)\ll m$ for small $\epsilon$. [A] confirms sparsity of $\b\ta$. For $\epsilon=1e-4$ (CLIP ViT-B/16), we checked that only 4.6% parameters changed.
  
  [A]. Efficient Model Editing with Task Vector Bases: A Theoretical Framework and Scalable Approach, ICLR'25.
  
  Thus, we assume that interpolating between such sparse differences in $\ti$ and $\tj$ captures a feasible parameter subset pertinent to the unlearning dataset, and realizes several magnitudes of "active" parameters.

• As counterexample, let task vectors be Normally distributed: ${\b\th}'\sim\mathcal{N}(\b{\mu},\b{\sigma}^2)$ (off-diagonal is 0; cannot estimate $86M\times 86M$ dim. covariance) but $\b{\mu}=\frac{1}{Q}\sum\_{i=1}^Q\ti$ and $\b{\sigma}^2=\frac{1}{Q-1}\sum\_{i=1}^Q (\ti-\b{\mu})^2$ and so we obtain ${\b\th}'=\b{\mu}+\b{\sigma}\odot\bf{v}$ where ${\bf v}\sim \mathcal{N}(\bf{0},\bf{1})$.
  
  Table below (CLIP ViT-B-16): function-level ensemble $\frac{1}{S}\sum_{i=1}^S f({\bf x};{\b\th}'\_i)$ under the Normal dist. ($S$ task vectors ${\b\th}'$ sampled from it):

  TABLE

  Distribution	$\fg$	$\rt$
  Normal ($S=100$)	16.80	64.32
  Normal ($S=300$)	15.98	64.54
  Dirichlet (Ours)	12.17	64.93

  The Norm. dist. performs worse than our Dirichlet model as PDF arms decay slowly toward $\infty$ whereas Dirichlet dist. has compact support: it produces finite values of filter parameters constrained by simplex.

1b. Theory/analysis.

We provide a rigorous bound: interpolated task vector model deviates from the expected convex combination by at most $L\la\eta$.

The bound is small if:

• task vectors differ by a small diameter $\eta=\max_{i\neq j}\lVert{\b\ta}\_i, {\b\ta}\_j\rVert_2$
• $\la$ is small
• $g$ is smooth (low Lipschitz const. $L$)

These conditions are met in our paper (low $\eta$ due to sparse change between task vectors, low $\la<1$).

Theorem: Interpolation Performance Bounds.

Let $g:\th\to\mathbb{R}$ be the unlearning function. Let $g$ be $L$-Lipschitz continuous: $\lVert g(\ti)-g(\tj)\rVert_2\leq L\lVert\ti-\tj\rVert_2\quad\forall i\neq j$

Given task vectors $\{\b\ta_1,\ldots,\b\ta_Q\}$ and any convex combination $\b\ta=\sum\_{i=1}^Q\al_i\b\ta_i$ where $\sum_{i=1}^Q\al_i=1, \al_i\geq 0$, we have:

$\bigg\lVert g(\tz-\la\b\ta)-\sum_{i=1}^Q\al_i g(\tz-\la\b\ta_i)\bigg\rVert_2\leq L\la\eta$

Theorem's Importance: any interpolated choice for $(\al_1,\ldots,\al_Q)$ deviates from expected convex combination by less/equal $L\la\eta$. We can provide Proof in discussions (rebuttal space limits).

1c. Analyzing Taylor orders (how do non-linear task dep. affect framework).

• Vector task aggregation model [A] has the same 1st order Taylor expansion as our method (it equals also to NTK): $\underbrace{f\bigg({\bf x};\tz-\frac{\la}{Q}\sum_{i=1}^Q\b{\ta}\_i\bigg)}\_{\text{approach [A]}}\approx f({\bf x};\tz)-\la\bigg\langle J\_f(\tz), \underbrace{\frac{1}{Q}\sum\_{i=1}^Q\b{\ta}\_i}\_{\b\mu\_\ta}\bigg\rangle= f({\bf x}; \tz)-\la\bigg\langle J\_f(\tz,\underbrace{\frac{1}{|\Delta\_\th|}\sum\_{\b\ta\in\Delta\_\th}\b{\ta}}\_{\b\mu\_\ta}\bigg\rangle\approx\underbrace{\frac{1}{|\Delta\_\th|}\sum\_{\b\ta\in\Delta\_\th} f\bigg({\bf x}; \tz-\la\b{\ta}\_i\bigg)}\_{\text{our method}}$ As 0th & 1st order moments for our technique are identical with [A] & [B], integration over simplex is not an issue under this expansion by virtue of methods [A] & [B].

  [B]. Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time, ICML'22

• Now we compare 2nd order expansions:

  • (approach [A]): $\qquad\qquad\qquad\\; f\big({\bf x};\tz-\frac{\la}{Q}\sum_{i=1}^Q\b{\ta}\_i\big)\approx f({\bf x};\tz)-\la\langle J\_f(\tz), \b\mu\_\ta\rangle+\la^2 \b\mu\_\ta^T H\_f(\tz)\b\mu\_\ta$

  • (basic function ensemble [B]): $\frac{1}{Q}\sum\_{i=1}^Qf({\bf x};\tz-\la\b{\ta}\_i\ )\approx f({\bf x};\tz)-\la\langle J\_f(\tz), \b\mu\_\ta\rangle+\la^2 \frac{1}{Q}\sum\_{i=1}^Q\b\ta\_i^T H\_f(\tz)\b\ta\_i$

  • (our Simplex ensemble): $\\;\\;\\;\frac{1}{|\Delta\_\th|}\sum\_{\b\ta\in\Delta\_\th} f({\bf x};\tz-\la\b{\ta}\_i)\approx f({\bf x};\tz)-\la\langle J\_f(\tz), \b\mu\_\ta\rangle+\underbrace{\rho\la^2\b\mu\_\ta^T H\_f(\tz)\b\mu\_\ta+\zeta\la^2\frac{1}{Q}\sum\_{i=1}^Q\b\ta\_i^T H\_f(\tz)\b\ta\_i}\_{\text{interpolation between [A] and basic func. ensemble}}$ for $\rho=Q/(Q+1)$ and $\zeta=1/(Q+1)$.
    
    As 0th and 1st order terms are identical, and our Simplex ensemble interpolates between two known approaches [A] ("Vector Uniform Merge" in our paper) and [B] ("Function Ensemble" in our paper) for 2nd order expansion (convex functions), which is why it is OK to integrate functions over task vectors uniformly drawn from the Dirichlet dist. (our simplex ensemble).
    
    Use of concentration parameter $\al$ of Dirichlet leads to $\rho=\al Q/(\al Q+1)$ and $\zeta=1/(\al Q+1)$ which lets smooth interpolation between [A] & [B].

    TABLE (CLIP ViT-B-16)

    $\al$	$\fg$	$\rt$
    0.1	14.65	64.49
    0.5	12.83	64.26
    0.8	12.05	64.68
    1.2	12.48	64.85
    1.5	12.70	64.92
    2.0	13.29	64.81
    1.0	12.17	64.93

    If we use Eq. (6) (paper), weight $w_i$ per vertex customizes interpolation (help reduce var.)

1d. Parameter augmentation.

As averaging all task vectors gives viable unlearning (model [A]), and ensembling functions on individual task vectors is viable unlearning (model [B]), interpolation between the mean $\frac{1}{Q}\sum_{i=1}^Q\b{\ta}\_i$ and individual $\b{\ta}\_j$ is "parameter augmentation". Let $p({\bf x}; \Omega)=\frac{1}{|\Omega|}\int\_{\b{\ta}\in\Omega} f\big({\bf x};\tz-\b{\ta}\big)d\b{\ta}.$ Based on 1b, if $f$ changes fast in $\Omega_{fast}$ & slow in $\Omega_{slow}$ (same neighborhood sizes) then intuitively entropy of prediction $Ent(p({\bf x}, \Omega_{slow}))$ is lower than $Ent(p({\bf x}; \Omega_{fast}))$ as smooth changes contribute coherently to vote - chaotic changes are incoherent leading to cancellation of class peaks: $Ent(vec(1/C))$ is max. So our method leverages smoothness of $f$ w.r.t. augmentation. See also [C]:

[C]. Averaging Weights Leads to Wider Optima and Better Generalization, UAI'19

1e. Do task vectors form simplex?

They do not have to so long they are close to simplex. We take $Q=10$ out of 30 task vectors fine-tuned on SUN397 and build simplex. We count sketch dimensions to $Q=d+1=11$. Remaining 20 task vectors are in $0.9(\eta/2)$ radius of the simplex while task vectors of Cars exceed $1.6(\eta/2)$ distance (cluster around their own simplex).

2a. Cost: Many fine-tunings.

We fine-tune over mere 6-35 epochs (not 1000). We can reduce $Q=30$ to 10 for speed or fine-tune in parallel.

See sequential fine‑tuning time/unlearning inference time (our paper's setup):

TABLE (CLIP ViT-B-16)

Below we reduce epochs range (6-35) by 1/2 or 1/3rd:

2b. Lang. or cross-modal task.

We apply our method on multi-modal CLEAR benchmark (Dontsov et al., 2024: fictional author profiles with face images-captions pairs). See the avg. accuracy (VQA task) (LLaVA‑1.5‑7B, CLIP ViT‑L/14 as vision encoder). Task vectors are derived by fine‑tuning on the corresponding target data.

TABLE

3.Constraints on Taylor.

Kindly note $0\leq\la\leq 1$: $\la^2$ in 2nd order term decays quadratically; 1st order term $\la$ is linear. In practice, $\la$ is chosen by cross-validation. For low $\la^2$ our method reverts to 1st order expansion (as [A] & [B]). Below we vary $\la$ (CLIP ViT-B/16) on SUN397. Bold: $\la$ best in cross-val.

TABLE

Response to Rev. 5xSJ

We thank Reviewer for constructive feedback.

1. Computational overhead is front-loaded (fine-tuning 30 models per task).

Thank you. Kindly note we can vary $Q=10,15,30$, etc. Moreover, we can vary number of fine-tuning epochs (currently in range 6-35) to 1/2 or 1/3rd (e.g., range 2-12) which is significantly lower than model pre-training/training from scratch. Finally, obtaining $Q$ tasks can be parallelized.

Tables below show results w.r.t. $Q$ and the number of fine tuning epochs, respectively.

TABLE (CLIP ViT-B-16)

1b. One must manage updating the base model.

Only fine-tuning from $\boldsymbol\theta\_0$ to $\boldsymbol\theta\_i$ task vectors. However, unlearning itself is performed by mere Taylor expansion -- unless distillation is desired to "compact" the model.

2. Could you clarify how Equation (3) was implemented?

Eq. (3) uses so-called Hessian-Vevtor Product (HVP) which requires the use of vmap, jacrev and hvp from torch.func and torch.autograd.functional. HVP never computed the Hessian matrix but directly and efficiently obtains $\bf{v}=(\bf{H}\cdot\boldsymbol\tau)$. Vector $\bf{v}$ then can be vector-multiplied with $\boldsymbol\tau$. HVP takes 2-4x the Jacobinan computation which makes is extremely efficient. In our case, one HVP takes approx. 3 seconds.

3. How important are the second-order terms versus first-order?

Resp. 1 to Rev. uLHj shows that under 1st order Taylor expansion, our approach degrades to the 1st order Taylor expansion of [A] ("Vector Uniform Merge" in our paper) and [B] ("Function Ensemble" in our paper)**, and to the NTK model:

[A]. Efficient Model Editing with Task Vector Bases: A Theoretical Framework and Scalable Approach, ICLR'25. [B]. Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time, ICML'22

However, the 1st order Taylor expansion is not the same as running [A] (think full expansion). Table below compares results (CLIP ViT-B/16) between the 1st and 2nd order solutions:

TABLE

4. Would the method scale to ViT Huge with billions of parameters ?

Yes, in a way it is always cheaper to run few epochs of fine-tuning than retrain entire ViT-H. Indeed, $Q$ task vectors can be obtained in parallel. To save memory, ViT-H could be equipped with LORAs, i.e., each $\boldsymbol\tau_i={\bf AB}^T$ for some tall (low rank) matrices ${\bf A}$ and ${\bf B}$. For ViT‑H with LoRA adaptation (rank‑16), fine‑tuning reduces to ~12h, while inference remains ~14 minutes, since LoRA adds negligible overhead to the forward pass.

TABLE

5. Does the advanced aggregator (Theorem 2) become too slow?

Thank you. No, the main cost is in obtaining $Q$ task vectors. The overall Taylor approximation takes seconds not hours.

Generally, Theorem 2 requires $2C$ runs of jacrev and $2QC$ hvp runs compared to the basic approach that uses $QC$ hvp runs. Variance minimization does not require extra computations as jacrev and hvp are pre-computed and shared with the basic method. Storing $30 (Q)\times 86M (params)\times 8 (double)$ requires 19GB memory which can be kept on the GPU until we move to process the next class in $1,\ldots,C$. Where needed, for larger $Q$ or ViT-H (632M), vectors can be stored in CPU RAM and moved around, they can be computed on several GPUs (e.g., one hvp per GPU), or LORA can be used to limit the size of parameters.

Below is table showing computation time excluding task vector fine-tuning for several variants of our method:

TABLE (1st, 2nd order, Theorem 2, variance tuning) (CLIP ViT-B/16)

6. Removing knowledge is positive for privacy. Could it be misused in any way?

Thank you. Absolutely, as any AI tool, there exists always a possibility of misuse. We will make it clearer in the Broader Impact and Limitations section. Indeed, a rouge actor with access to the model and unlearning data could intentionally “unlearn” important facts from a model. Indeed, our approach provides mere functionality but this is not a model that can monitor fairness of unlearning tasks.

7. Might repeated unlearning degrade a model in unforeseen ways?

Yes, this is a very interesting question. Unlearning could degrade the ability of $\boldsymbol\theta\_0$ to deal with some class labels which have strong semantic correlation with classes we unlearn. However, as unlearning is merely obtained by task arithmetic, we can always store $\boldsymbol\theta\_0$ and consecutive sets of task vectors to be able to unroll changes.

Below is the retain set performance (forget set accuracy can be found in Table 3 in the main text) on the retaining accuracy under incremental unlearning using CLIP ViT-Base/32:

TABLE

Note that the pre‑trained model achieves 63.3% accuracy on the retain set; our incremental unlearning preserves roughly 90% of this performance on the test set and 95% on validation set.

8. Open-vocabulary CLIP tasks or retrieval tasks.

To demonstrate cross‑modal applicability, we additionally validate our method on the multimodal CLEAR benchmark (Dontsov et al., 2024), which involves fictional author profiles with paired face images and captions. As shown in the table below, we report forgetting and retention accuracy on the VQA task using LLaVA‑1.5‑7B (with CLIP ViT‑L/14 as the vision encoder). Our approach achieves the best forgetting while maintaining comparable retention performance.

TABLE

Conceptually, our task‑vector framework extends to other multimodal settings, including open‑vocabulary CLIP tasks and image‑text retrieval. Adapting to those scenarios would require defining task vectors on the corresponding objective (e.g., fine‑tuning CLIP’s multimodal head to capture specific image‑caption associations) and applying our unlearning procedure in the same manner. We will explore these directions in future work.

9. Task specificity: forgetting is measured in terms of classification accuracy.

Thank you. Certain image-text pair associations in CLIP would require obtaining task vectors on lager number of image-text unlearning pairs by just fine-tuning with the CLIP loss. Then $f$ could be replaced with the feature output of CLIP and the rest follows. However, Taylor expansion can operate on feature spaces too.

Kindly see Resp. 8 above where we work with VQA instead of classification.