MTL-UE: Learning to Learn Nothing for Multi-Task Learning

Q1. Scenarios involving fine-tuning (using a pretrained feature encoder) and advanced augmentations.

A1. Thanks for the suggestions. We conducted experiments on the proposed scenarios. The table presents results of training MTL models on UTKFace with ImageNet-pretrained-encoder fine-tuning or UEraser [1] as advanced augmentations. Fine-tuning showed no impact on any method, while strong augmentations affected all, with our attacks still outperforming baselines.

Q2. Line-by-line description of Algorithm 1.

A2. Thanks for your suggestions! We’ll add a detailed description in the updated version.

Q3. Examples of partial task protection, where only one task is protected.

A3. Following Table 8, we show single-task protection results. We can see that for STL models, unprotected tasks match clean data accuracy, while protected tasks perform like those trained to protect all tasks. Results on MTL models degraded performance, likely due to shared encoder learning both benign and spurious features.

Q4. Tables 11 and 12 are missing references in Sections B.2 and B.3, which should be addressed.

A4. Thanks for your suggestions. We will fix it in the updated version.

[1] Learning the unlearnable: Adversarial augmentations suppress unlearnable example attacks. ICCVW 2023

Q1. How would these methods perform if ViT-B were used as the backbone for the surrogate models?

A1. In addition to the results in Table 6, we conducted experiments using ViT-B as the backbone for the surrogate models on the CelebA dataset. The results in the table below show that this choice improves performance when the victim models also use ViT-B but can negatively impact performance when the victim models are CNN-based. This suggests a trade-off in selecting CNNs or ViTs for the surrogate models, depending on the target victim model's backbone.

Q2. Among the 40 tasks in CelebA, are there specific tasks that are significantly easier or harder to degrade?

A2. We compute the accuracy drops for all 40 tasks when training STL models on MTL-UE compared to models trained on clean data. In MTL-UE-EM, tasks 38, 35, and 24 experience the largest drops (-58.74, -58.03, -44.39), while tasks 23, 15, and 39 are least affected (-0.48, -0.70, -1.00). In MTL-UE-TAP, tasks 25, 1, and 40 degrade the most (-80.22, -67.02, -61.90), whereas tasks 36, 38, and 27 show minimal impact (+0.02, 0.00, -0.32). In MTL-UE-SEP, tasks 15, 37, and 2 suffer the highest degradation (-61.94, -49.69, -43.03), while tasks 26, 27, and 14 are least affected (+0.09, +0.02, 0.00). These results suggest that the susceptibility of tasks to degradation varies significantly across different base UE strategies. Tasks experiencing the largest drops may rely on more vulnerable feature representations, making them easier to disrupt. In contrast, tasks with minimal impact likely depend on more robust or less perturbed features.

Q3. More analysis on the influence of the number of tasks on the UEs performance.

A3. Thank you for the suggestion! We provide a detailed analysis of how the number of tasks affects UE performance. As shown in Figure 5, when training MTL models, increasing the number of tasks to 10 causes only a slight performance drop (minor accuracy increase). From 10 to 40 tasks, MTL-UE remains stable, maintaining ~60% accuracy for both MTL-UE-TAP and MTL-UE-SEP. This stability is likely due to the shared encoder in MTL models (for both the UE generation and victim model training), which facilitates learning spurious features across all tasks, preventing performance degradation.

However, when training STL models with varying task numbers, MTL-UE performance gradually declines as the number of tasks increases from 10 to 20 and 20 to 40. This may result from a mismatch between the UE generation and victim model training, as the former uses MTL surrogate models while the latter adopts STL models. We will include this analysis in the updated version.

Q4. A direct runtime comparison with baselines.

A4. Thanks for your suggestions. The table below provides a direct runtime comparison for generating perturbations across all competing methods in terms of hours. The generation process was conducted on a single RTX 4090 using the CelebA dataset (with ViT as the backbone of the surrogate MTL model) and the NYUv2 dataset. Notably, LSP and AR, which use predefined patterns, require almost no time for perturbation generation. The results indicate that our MTL-UE has a comparable computational cost to the base UE methods and is sometimes more efficient.

Q1. Insufficient coverage of existing literature.

A1. Thank you for the advice! We’ll add recent works on data poisoning in the related work. As we focus on UE, we add a comparison between UE and other poisoning attacks (the table below) in the updated paper to broaden the literature coverage and emphasize the need for UE.

Q2. Challenges of new issues are not significant across datasets.

A2. Following original settings, we report accuracy (%) for CelebA and UTKFace, while for ChestX-ray14 the metric is AUC-ROC (see Sec. 5.1). The drop in AUC-ROC from 0.7577 to near 0.5 is significant.

Q3. Questioning of method validity.

A3. MTL-UE works well with surrogate-dependent UE methods, improving performance across most datasets. The exception is ChestX-ray14 with TAP and SEP, where the clean model performs not well (AUC-ROC=0.75), limiting the success of adversarial attacks and, consequently, the effectiveness of adversarial-attack-based UE methods like TAP and SEP. However, MTL-UE still improves on EM, boosts TAP and SEP in MTL, and matches STL results for TAP and SEP.

Q4. Insufficient technical novelty.

A4. MTL-UE innovates in UE for MTL by solving model misalignment with shared task embeddings, boosting STL performance. It optimizes at the distribution level using class-wise embeddings and allows flexible task protection without retraining. Theoretically, we introduce embedding regularizations (Intra-ER & Inter-ER) to maximize intra-task separation and promote inter-task independence, enhancing the learning of spurious features. See A5 for more details on the technical novelty to solve challenges. We’ll refine the theoretical analysis and revise the paper to clarify these.

Q5. The applicability of MLT tasks is unclear.

A5. MTL is vital in real-world applications like autonomous vehicles, which handle multiple tasks simultaneously. Research has advanced multi-task datasets and MTL models. While UE has been studied for STL, we are the first to tackle unauthorized training on multi-task data and protect data privacy. To bridge the gap, we adapt UE methods from STL to MTL and highlight key challenges:

• Model Alignment Issue: GPU constraints limit UE generation to use one MTL model instead of multiple STL models (e.g., 40 for CelebA), causing misalignment when training STL victim models. In MTL-UE, task-wise embeddings align UE within each task, reducing performance degradation when training STL models compared to MTL ones. In CelebA, MTL-UE-EM shows minor MTL gains but major STL improvements.
• Lack of Distribution-Level Optimization: Prior UE methods optimize samples independently, ignoring distribution-level effects. In MTL settings like CelebA (40 tasks), this issue worsens. MTL-UE uses class-wise embeddings to poison distributions per task, reducing intra-class variance and reinforcing spurious correlations, enhancing attack effectiveness.
• Re-optimize for different protected task sets: Prior sample-wise UE methods require re-optimizing perturbations for each combination of tasks to protect. MTL-UE, once optimized on all tasks, allows flexible task selection to protect without re-optimizing (Sec. 5.3).

We'll add these in the paper.

Q6. The question of the plausibility of attack scenarios.

A6. Unlearning in MTL protects sensitive multi-task data from unauthorized training due to privacy concerns. In medical AI, MTL models use X-rays, MRIs, and patient histories for diagnosis, where unauthorized training risks privacy and ethics violations. Facial recognition models (e.g., CelebA) can enable surveillance and profiling. Social media analysis (e.g., Weibo-20) enables large-scale surveillance. Smart surveillance models (e.g., DukeMTMC) risk privacy infringement. MTL-UE prevents unauthorized use by introducing unlearnable perturbations, degrading MTL and STL model performance, thus enhancing data privacy and intellectual property protection. We'll add these in the paper.

Q7. Evaluation of computational complexity.

A7. Please refer to A4 for reviewer AfLP.

Q1. Further clarify the experimental results. What does STL mean in Figure 2, and why do the baseline methods perform poorly, even on STL?

A1. The pipeline for UE has two stages:

• Stage 1: UE generation process (Section 4.2).
• Stage 2: UE performance evaluation, where generated UEs train victim models, and these models are tested on clean data.

Stage 1 focuses on generating UE to protect all tasks simultaneously using a surrogate MTL model. Stage 2 evaluates in two ways: a) Train one MTL model for all tasks. b) Train individual STL models for each task, and averaging the evaluation results across all tasks.

Figure 2 shows UE performance vs. the task number. For each x-axis point, only the first $K$ tasks are selected for both stages, and UE are generated to protect them. Stage 2 evaluates a) MTL (left) and b) STL (right). Other tables use full task sets with the above stages for a complete UE assessment. All baseline methods designed for STL, perform poorly in Stage 2 for both MTL and STL, as Stage 1 generates UE for MTL models, not STL settings. Figure 2 also shows that when $K$ is small, these baselines perform well but degrade as $K$ increases due to greater misalignment between Stage 1 and Stage 2. We will clarify these in the paper.

---

Q2. Can original sample-wise UE be modified to work under the MTL scenario? Except for class-wise noises making the optimization easier, what is the potential obstacle that makes sample-wise UE not work?

A2. To reduce the high intra-class variance in optimization-based methods, we add additional loss terms. As these methods use a surrogate MTL model during optimization, we define $L_{std}=[L_{std}^1,\ldots,L_{std}^D]$, and $L_{std}^d$ is defined in line 201 of the paper. We use two loss terms, $L_{std}^{mean}$ and $L_{std}^{max}$: the mean and maximum of $L_{std}$. In the perturbation optimization (Eqs. (1) & (2)), we add $\lambda_1 L_{std}^{mean}+\lambda_2 L_{std}^{max}$ to the original loss. If the original optimization is to maximize, we apply a negative sign.

We experiment on CelebA with a batch size of 1024. The table below shows MTL model results with various hyperparameters. As $L_{std}^{mean}\approx5$ and $L_{std}^{max}\approx100$, these choices are reasonable. Despite different settings, the new losses don't outperform the original one. $L_{std}^{max}$ largely degrades performance, and $L_{std}^{mean}$ has a smaller, but still negative, effect.

Challenges in Adopting These Loss Terms

• Batch Size Limitation: With 160k images in CelebA and a 1k batch size, variance estimates are unreliable, and increasing batch size is infeasible due to GPU limits.

• Surrogate Model Misalignment: TAP and SEP use a surrogate MTL model trained on clean data, misaligning with the victim model trained on UE. EM partially addresses this by training on UE data, but weak early-stage perturbations lead to suboptimal results.

• Conflict with Original Loss: Minimizing $L_{std}^{mean}$ and $L_{std}^{max}$ conflicts with the original loss. $L_{std}^{max}$ degrades UE effectiveness, and $L_{std}^{mean}$ with small $\lambda_1$ has minimal impact.

• Computational Overhead: These losses increase computation by $\times28$.

Potential Obstacles for Existing Sample-Wise UE in MTL

• Model Alignment Issue: Due to GPU limits, UE generation uses one MTL model instead of multiple STL models (e.g., 40 for CelebA), causing misalignment with STL victim models. In MTL-UE, a surrogate MTL model is used, and images with the same label $y^k$ for the $k$-th task share the embedding $e_{y^k}^k$. Thus, MTL-UE suffers less degradation when training STL victim models than MTL models. CelebA results show MTL-UE-EM improves slightly in MTL but shows gains in STL, highlighting its alignment effectiveness.
• Lack of Distribution-Level Optimization: Perturbations mislead the victim model by mapping $x_i+\delta_i$ to labels, poisoning the data distribution.
  • Previous UE methods individually optimize samples, missing distribution-level effects.
  • EM trains on batched poisoned data, implicitly considering distribution, is better than TAP and SEP.
  • In MTL settings like CelebA (40 tasks), lack of distribution optimization is problematic.
  • MTL-UE creates poisoned distributions with class-wise embeddings, reducing intra-class variance and effectively misleading victim models.
• Re-optimizing for different protected task sets: Prior sample-wise UE methods require re-optimizing perturbations for each combination of tasks to protect. MTL-UE, once optimized on all tasks (Sec. 5.3), allows flexible task selection to protect without re-optimizing.