Why Fine-Tuning Struggles with Forgetting in Machine Unlearning? Theoretical Insights and a Remedial Approach

We sincerely thank the Reviewer J9SB for the valuable time and positive comments on our work! We hope our following answers could address your concerns.

Response to Weakness 1: Thanks for the comment! Overparameterized linear models are widely adopted as a foundational framework for studying learning problems (e.g., transfer learning [1], continual learning [2], In-context learning [3,4]) and can be extended to more general settings such as neural tangent kernel (NTK) analysis. While the setting of overparameterized linear regression is indeed simplistic, it provides a valuable starting point for analyzing training dynamics, capturing the trajectory of learning rather than just upper or lower bounds. In this work, we start with the overparameterized linear regression model and hope to extend the analysis to more general cases such as multi-layer neural network in the future.

Response to Weakness 2: Thank you for your comment! Our primary motivation stems from the empirical observation that while fine-tuning can maintain model utility on the remaining data, it often struggles to effectively forget targeted data. To provide theoretical insights into this phenomenon, we begin with a simpler case where the dataset is completely separable.

For example, consider a dataset containing two categories: bananas and cars. Bananas have a distinct feature like "elongated shape," and cars have a unique feature like "mirrors." These features are entirely distinct. We then extend this setup to include overlapping features, such as color, where both bananas and cars might share a feature like "yellow."

We recognize that most practical cases involve overlapping features. However, presenting only the overlapping case would relegate the distinct case to a special case, disrupting the presentation flow. Our goal is to demonstrate that even in the extreme case of distinct features, fine-tuning still fails to unlearn. Therefore, starting with a simpler, distinct feature setup allows us to build a foundation to explain why fine-tuning fails to unlearn. From this base, we advance the analysis to handle more complex, overlapping cases.

Response to Weakness 3: Yes, the scenario you mention aligns with our findings in Theorem 4.1, which demonstrates that discarding overlapping features leads to an increase in the retaining loss. To address this, in the next section, we propose using both retain loss and unlearn loss as objective functions, allowing the model to effectively balance retaining overlapping features while unlearning targeted information.

Response to Question 1: As mentioned in our previous response, we discuss the case of nonoverlapping features as a starting point to build intuition about the phenomenon of fine-tuning failing to unlearn. While this case may be less common in standard ML, it provides a simplified setup that allows us to derive theoretical insights and establish a foundation for analyzing more complex overlapping scenarios.

Our current work focuses on overparameterized linear regression as a framework for machine unlearning. Extending this analysis to strongly convex settings or a two-layer linear network would indeed require different techniques, as closed-form solutions are not readily available. We agree that such extensions are valuable and will consider them as part of future work. Thank you for the suggestion!

References:

[1] Wu, Jingfeng, et al. "The power and limitation of pretraining-finetuning for linear regression under covariate shift." Advances in Neural Information Processing Systems 35 (2022): 33041-33053.

[2] Ding, Meng, et al. "Understanding Forgetting in Continual Learning with Linear Regression." arXiv preprint arXiv:2405.17583 (2024).

[3] Wu, Jingfeng, et al. "How Many Pretraining Tasks Are Needed for In-Context Learning of Linear Regression?." arXiv preprint arXiv:2310.08391 (2023).

[4] Chen, Xingwu, Lei Zhao, and Difan Zou. "How transformers utilize multi-head attention in in-context learning? a case study on sparse linear regression." arXiv preprint arXiv:2408.04532 (2024).

We thank the Reviewer NAAU for the detailed review. We hope our following answers can address your concerns.

Response to Weakness 1: Thanks for the point! Overparameterized linear models are widely adopted as a foundational framework for studying learning problems (e.g., transfer learning [1], continual learning [2], In-context learning [3,4]) and can be extended to more general settings such as neural tangent kernel (NTK) analysis. While the setting of overparameterized linear regression is indeed simplistic, it provides a valuable starting point for analyzing training dynamics, capturing the trajectory of learning rather than just upper or lower bounds. As for the focus on regression versus classification, although we use linear regression to analyze the fine-tuning process, our assumptions about the data structure are based on a classification problem, which is why our experiments focus on class-wise forgetting. In this work, we start with the overparameterized linear regression model and hope to extend the analysis to more general cases such as multi-layer neural network in the future.

Response to Weakness 2: Thank you for pointing out this! We acknowledge that there is limited distinction between ICE-FT approach and CE-FT, primarily differing in the choice of the parameter $\alpha$. However, based on the analysis in Sections 3 and 4, we aim to understand the empirical phenomenon of why naive fine-tuning fails to forget. Therefore, Theorem 4.1 provides the rationale for selecting the parameter $\alpha \in(0,1]$ for the unlearning loss (the design of ICE-FT), whereas in [5], the constraint $\alpha \in(0,1]$ is applied to the retain loss. While our objective formulation may appear similar to existing work, our novelty/contribution goes beyond the design of the regularizer. In response to this weakness, we have rewritten Section 5 to modify our previous statement. Please check it. Thank you for the comment!

Response to Weakness 3: We appreciate the Reviewer’s comment and agree that our experiments do not cover more complex unlearning settings or comparisons with a wide range of SOTA methods. However, the primary focus of our paper is to understand the empirical phenomenon of why naive fine-tuning fails to forget, as analyzed in Sections 3 and 4. The experiments are designed to validate the conclusions derived from our theoretical framework rather than to benchmark against numerous unlearning methods.

Response to Weakness 4: Thank you for pointing this out! We have corrected this notation in our revised version.

Response to Question 1: Distinct features refer to features unique to either forgetting data or retaining data, while overlapping features are shared between the two. For example, consider a dataset containing two categories: bananas and cars. Bananas have a distinct feature like an "elongated shape," and cars have a unique feature like "mirrors." These features are entirely distinct. We then extend this setup to include overlapping features, such as color, where both bananas and cars might share a feature like "yellow."

Response to Question 2: Thank you for your question! In the machine unlearning community, various evaluation metrics and settings exist, including relearn time, as mentioned by the Reviewer. Relearn time, as defined in [6], measures the time needed to achieve a margin of $\alpha \\%$ around the original accuracy in a zero-shot unlearning setting, where $\alpha \%$ reflects the accuracy range of the original model on the forget classes. In Table 2, we provide the runtime efficiency of our methods, which is similar to relearn time defined in [7] but differs slightly as we report the entire training process rather than just a few epochs needed to achieve the margin.

References:

[1] Wu, Jingfeng, et al. "The power and limitation of pretraining-finetuning for linear regression under covariate shift." Advances in Neural Information Processing Systems 35 (2022): 33041-33053.

[2] Ding, Meng, et al. "Understanding Forgetting in Continual Learning with Linear Regression." arXiv preprint arXiv:2405.17583 (2024).

[3] Wu, Jingfeng, et al. "How Many Pretraining Tasks Are Needed for In-Context Learning of Linear Regression?." arXiv preprint arXiv:2310.08391 (2023).

[4] Chen, Xingwu, Lei Zhao, and Difan Zou. "How transformers utilize multi-head attention in in-context learning? a case study on sparse linear regression." arXiv preprint arXiv:2408.04532 (2024).

[5] Fan, Chongyu, et al. "Salun: Empowering machine unlearning via gradient-based weight saliency in both image classification and generation." arXiv preprint arXiv:2310.12508 (2023).

[6] Chundawat, Vikram S., et al. "Zero-shot machine unlearning." IEEE Transactions on Information Forensics and Security 18 (2023): 2345-2354.

[7] A. Golatkar, A. Achille, and S. Soatto, “Eternal sunshine of the spotlessnet: Selective forgetting in deep networks,”.

We thank the Reviewer h2Yv for the thorough review and the insightful comments. We hope our following answers can address your concerns.

Response to Weakness 1: Our primary motivation arises from the empirical observation that fine-tuning can maintain model utility on remaining data but struggles to effectively forget targeted data. Sections 3 and 4 provide theoretical insights to explain and address this issue. Section 5 redesigned a discriminative regularizer aligned with the principle deduced in Section 4: regularization should prioritize remaining accuracy over unlearning accuracy. In response to this weakness, we have rewritten Section 5 to emphasize the redesign rationale for the regularizer and its implications, rather than introducing it. Please check it. Thank you for the comment!

Response to Weakness 2: We agree that there is limited distinction between our proposed regularization approach and existing objective functions, primarily differing in the choice of the loss function and the parameter $\alpha$. However, the main novelty of our paper lies in providing the first theoretical framework to understand the empirical phenomenon of why naive fine-tuning fails to forget. Based on the analysis in Sections 3 and 4, we aim to address this phenomenon by designing suitable objectives. Additionally, Theorem 4.1 provides the rationale for selecting the parameter $\alpha \in(0,1]$ for the unlearning loss, whereas in [1], the constraint $\alpha \in(0,1]$ is applied to the retain loss. While our objective formulation may appear similar to existing work, our novelty/contribution goes beyond the design of regularize. In response to this weakness, we have rewritten Section 5 to modify our previous statement. Please check it. Thank you for the comment!

Response to Weakness 3: We agree with the Reviewer that KL-FT and (I)CE-FT require access to the forget set, unlike Vanilla FT. We appreciate this observation and welcome the opportunity to discuss this point in the context of current machine unlearning settings. Beyond methods requiring access to the forget set, there is also the zero-shot unlearning setting [2], which achieves unlearning without access to either the forget or remaining datasets. Thus, we view this distinction as representing different settings rather than a weakness of our approach. Regarding a more comprehensive evaluation, as noted in our response to Weakness 2, the primary novelty of our paper is to provide theoretical analysis and insights into the unlearning process via fine-tuning methods. Thank you for the valuable feedback!

Response to Minor comments 1: Thank you for pointing this out! The legend in Figure 1 should be RL/UL instead of RA/UA. We corrected this in the revised version.

Response to Minor comments 2: Thanks for the comment! Our goal is to make the fine-tuned $\mathbf{w}_t$ model as close as possible to the golden model $\mathbf{w}_g$. From the equation on line 289, we can observe the difference between these two solutions, which naturally leads to the following discussion.

Response to Minor comments 3: Thanks for pointing out this! Yes, $d_o$ refers to $d_{\text {lap }}$ and we corrected the notations in the revised paper.

Response to Minor comments 4: Thank you for the comment! The function $\\mathcal{L}\_{\mathrm{KL}}(\\cdot)$ in Equation (6) corresponds to the KL divergence between the output of the forgotten model and the one-hot encoded random labels. Specifically, $\\mathcal{L}\_{\\mathrm{KL}}=\\sum\_{i=1}^n \\mathrm{KL} (\\mathbf{w}\_t^{\top}\\mathbf{x}\_i\\| Y\_i^{\prime })$, where $\mathrm{KL}\left(p_i \| q_i\right)=\sum_{j=1}^{|\text{Class}|} p_i(j) \log \left(\frac{p_i(j)}{q_i (j)}\right)$. Regarding the baseline of random label or KL-based loss functions, our primary focus is on providing theoretical analysis to explain the observed empirical phenomenon. Thus, we focus our experiments on naive fine-tuning and the most closely related methods to validate our conclusions. Thank you for your feedback!

Response to Question 1 and Question 2: Although we use linear regression to analyze the fine-tuning process, our assumptions about the data structure are based on a classification problem, which is why our experiments focus on class-wise forgetting.

Response to Question 3: In the table 2, we select the corresponding best $\alpha \in (0,1]$ for each methods ((I)CE-FT, KL-FT) to report the results.

I thank the authors for the updated manuscript and the point-by-point responses. Based on reading the revised manuscript, I continue to struggle with some of the core conceptual aspects of the paper:

It is a bit counterintuitive that the new "regularization" is "prioritizing retaining accuracy over unlearning accuracy" in fine-tuning based unlearning. To the best of my understanding, this is something that already happens in fine-tuning as fine-tuning literally just optimizes for the retaining accuracy with no regards to the unlearning accuracy (as also defined in (2) except for the $\mathbf{w}_o$ term in the loss $|| \mathbf{w} - \mathbf{w}_0 ||^2$). So fine-tuning is already only prioritizing "retaining accuracy" in the most extreme sense, and still is unsuccessful in unlearning (as shown in section 3).

Thus, the main message I take away from the (re)reading this paper is the following:

• M1: (section 3) Fine-tuning (as defined in (2)) is not successful in unlearning as there is a significant gap between the golden model $\mathbf{w}_g$ and the fine-tuned model $\mathbf{w}_t$ (although there are some caveats; see Note A and Note B).
• M2: (section 4) In the overlapping setup, it is better to leave the overlapping feature weights (instead of resetting them) as this provides the best tradeoff between unlearning loss and retaining loss among all two ways to "regularize" (reset) the weights for fine-tuning. However, in the (pratically usual) case where $d_{\text{lap}} = d$, this regularization is a no-op as there is nothing to be done before fine-tuning, and we are back to basic fine-tuning (except that we are solving a specific instance of fine-tuning defined in (2)). So effectively, the best regularization is to just do fine-tuning.
• M3: (section 5) This message is reiterated with the experiments where the authors show that putting more weight on the retaining accuracy is better for unlearning/utility tradeoff. However, again, note that putting all the weight on the retain accuracy is the usual fine-tuning based unlearning.

Note A: The $d_{\text{lap}} = d$ case is not clear. Given $\mathbf{w}\_\* = \mathbf{w}\_\*^{\text{lap}}$ when $d_{\text{lap}} = d$, let the original problem (1) solution be $\mathbf{w}\_o = \mathbf{w}\_\*$. Since $y\_r = X\_r^\top \mathbf{w}\_\*$, $\mathbf{w}\_t = \mathbf{w}\_\*$ with $L(\mathbf{w}\_t, D\_r) = 0$ since this value would minimize $||\mathbf{w}\_o - \mathbf{w} ||^2$. Also $L(\mathbf{w}\_t, D\_f) = 0$ so no forgetting happens. However, $\mathbf{w}\_g = \mathbf{w}\_\*^{\text{lap}} = \mathbf{w}\_\* = \mathbf{w}\_t$. So $L(\mathbf{w}\_g, D\_r) = 0$ but also $L(\mathbf{w}\_g, D\_f) = L(\mathbf{w}\_*, D\_f) = L(\mathbf{w}\_t, D\_f) = 0$, so the golden unlearned model also has low unlearning loss and no gap exists between the golden and the fine-tuned model. So it is not clear when and where the mentioned gap would exist.

Note B: It seems to me that the results in section 3 regarding the gap between $L(\mathbf{w}\_t, D\_f)$ and $L(\mathbf{w}\_g, D\_f)$ stems from the formulation of the objective $\arg\min || \mathbf{w} - \mathbf{w}\_o ||^2$ for fine-tuning in (2) involving the original training solution $\mathbf{w}\_o$. This makes sense given that the fine-tuning process initiates from $\mathbf{w}\_o$. However, this formulation seems critical to have the gap between fine-tuning and golden model. Since the golden model minimizes the norm $|| \mathbf{w} ||^2$ in (3), it drops the disentangled $\mathbf{w}\_\*^f$ part of the weights. In contrast, the fine-tuning solution in (2) is limited to be a minimal distance solution with the $|| \mathbf{w} - \mathbf{w}\_o ||^2$ term, thereby forcing the $\mathbf{w}\_t$ to keep the $\mathbf{w}\_\*^f$ component of the weights. To me, this is not an inherent behaviour of fine-tuning based unlearning, but rather an artifact of the fine-tuning problem formulation. Instead if the fine-tuning problem (2) was formulated as following where we again look for the lowest-norm solution while being close to the initial $\mathbf{w}\_o$, it is not clear if the results would continue to hold

$\arg \min_{\mathbf{w}} || \mathbf{w} ||^2 \text{ s. t. } \text{(i)} y_t = X_t^\top \mathbf{w}, \text{(ii)} || \mathbf{w} - \mathbf{w}_o ||^2 \leq \epsilon.$

In this case, the preservation of the $\mathbf{w}\_\*^f$ component in the $\mathbf{w}\_t$ is no longer ensured, and the unlearning loss gap between $L(\mathbf{w}\_t, D\_f)$ and $L(\mathbf{w}\_g, D\_f)$ would probably depend on $\epsilon$ in constraint (ii), which corresponds to how thorough we allow the fine-tuning (using the retain loss) to be. If $\epsilon$ is large enough that $|| \mathbf{w}_o - \mathbf{w}_g ||^2 \leq \epsilon$, then fine-tuning should find $\mathbf{w}\_g$.

Minor: Given a ground-truth distribution $p$ and a predicted distribution $q$ (which we are optimizing over), is it not the case that minimizing the cross-entropy loss $\min_q CE(p | q) = \min_q -\sum_i p(i) \log q(i)$ is equivalent to minimizing the KL-divergence

$

\min_q KL(p|q) & = \min_q \sum_i p(i) \log \frac{p(i)}{q(i)} \newline & = \min_q \sum_i \left[ p(i) \log p(i) - p(i) \log q(i) \right] \newline & \equiv \min_q - \sum_i p(i) \log q(i)

$

What am I missing here if they are equivalent optimization problems? Is the actual difference that we use different ground-truth distributions $p$ for CE vs KL?

We sincerely thank the reviewer for the thoughtful feedback and for the opportunity to clarify some core aspects of our paper! We address each of the points raised below to enhance the clarity and understanding of our work.

Response to Counterintuitive Nature of "RA over UA":

We understand that it might seem counterintuitive that the regularization involves "prioritizing retaining accuracy over unlearning accuracy" in fine-tuning-based unlearning. In standard fine-tuning methods, the optimization focuses solely on retaining accuracy (RA) without explicitly considering unlearning accuracy (UA). This lack of an unlearning objective is precisely why naive fine-tuning often fails to effectively unlearn. Our method revisited a regularization term that explicitly balances RA and UA. By incorporating both objectives into the fine-tuning process, we can improve unlearning performance while maintaining acceptable performance on the retained data. This regularization is not inherent in standard fine-tuning, which does not address UA at all. Therefore, the discussion on regularizer differs from naive fine-tuning by directly incorporating unlearning into the optimization objective.

Response to M1: Gap Between Fine-Tuned Model and Golden Model:

In Section 3, we analyze scenarios where fine-tuning, as defined in equation (2), does not achieve the same unlearning performance as the golden model. We acknowledge that when the overlapping dimension $d_{\text {lap }}=d$, the situation requires further examination.

Addressing Note A:

When $d = d_{lap}$, the solutions for each problem should be $\\mathbf{w}\_o^{} = \\mathbf{P}\\mathbf{w}\_*^{}$, $\\mathbf{w}\_g^{} = \\mathbf{P}\_r\mathbf{w}\_*^{}$, $\\mathbf{w}\_t^{} = \\mathbf{P}\\mathbf{w}\_*^{}$. Therefore, the loss for Note A should be $L(\\mathbf{w}\_g^{}, D\_f) = \\frac{1}{n\_f}\\|\\mathbf{X}\_f (\\mathbf{P}\_r - \\mathbf{I})\\mathbf{w}\_*^{}\\|^2 \\neq L(\\mathbf{w}\_t^{}, D\_f) = 0.$ There is still a gap between fine-tuned model $\mathbf{w}_t$ and $\mathbf{w}_g$. This demonstrates that even when all features overlap, the fine-tuned model does not match the golden model's unlearning performance.

Addressing Note B:

The reviewer suggests that the gap arises due to the specific formulation of the fine-tuning objective, particularly the $\\|\mathbf{w}-\mathbf{w}_o\\|^2$ term in equation (2). In the following, we show that when considering the fine-tuning process as $$ \arg \min_{\mathbf{w}} \|\mathbf{w}\|^2 \text { s. t. (i) } \mathbf{y}_t=\mathbf{X}_t^{\top} w, \text { (ii) }\|\mathbf{w}-\mathbf{w}_o\|^2 \leq \epsilon

We thank the reviewer once again for the thoughtful responses and insightful comments.

Response to Main Concern:

We understand and appreciate the reviewer's primary concern regarding the differing conclusions under different conditions, specifically:

1. When the fine-tuning dataset $D_t$ is similar to $D_r$, and the inequality constraint becomes inactive.
2. When the inequality constraint is active, and $\epsilon$ is sufficiently large.

For the first scenario, if the inequality constraint is inactive, the fine-tuned model $\mathbf{w}_t$ can achieve perfect unlearning/retention loss, same as the golden model $\mathbf{w}_g$. However, in what situations does the inequality become inactive? If this occurs, the problem becomes disconnected from the pretrained model $\mathbf{w}_o$, meaning the fine-tuned model cannot truly be regarded as a fine-tuned model since it does not utilize the pretrained model's information.

For the second scenario, if $\epsilon$ is large, it suggests that the pretrained model does not significantly contribute to the fine-tuned model. Consequently, both the remaining loss and unlearning loss deviate from the golden model and produce suboptimal results.

Regarding Section 4, the elimination of forgetting data features assumes that the structure of the data/model is already known, as noted in Minor 1. This assumption is indeed restrictive in practice. To address this, we propose adding another unlearning loss function to the naive fine-tuning approach, which aligns with the overarching idea of eliminating features associated with the forgetting data.

We are grateful for the reviewer's insightful comments and will incorporate these discussions into our revised version. However, we still believe the current problem setup is the most suitable way to describe the unlearning process compared to the scenarios discussed above.

Response to Minor 1:

We agree with the reviewer that accessing nicely disentangled features is not always feasible. For analytical purposes, we started with a simplified case but are keen to extend this framework to deeper analyses with more generalized assumptions and models. There is substantial existing work on feature learning that describes similar challenges, such as [1], and we aspire to extend our work along these lines.

Reference: Allen-Zhu, Zeyuan, and Yuanzhi Li. "Towards understanding ensemble, knowledge distillation and self-distillation in deep learning." arXiv preprint arXiv:2012.09816 (2020).

Response to Minor 2:

Thank you for the reviewer's feedback. During our experiments, we observed that the choice of function (CE or KL) leads to only minor differences in the outcome. The more significant factor lies in selecting the appropriate parameter. That's why we follow your suggestion to rewrite the Section 5 part. Thanks again for your suggestion!

We are grateful to Reviewer bHDY for the detailed review and constructive feedback. We hope address your concerns accordingly.

Response to Weakness1:

• (1) Setting of Regularization: You raised an important point regarding the scale of the two terms in the loss (Eq. 6). Our design intentionally restricts $\alpha \in (0,1]$ to prioritize retaining accuracy over unlearning accuracy, which differs from the approach in [1], where unlearning accuracy is considered more critical. We understand this distinction may appear to be an arbitrary design choice; however, it aligns with the goals of our previous conclusion: we should upperbound the unlearning accuracy instead of remaining accuracy in [1].
• (2) Experimental Setting of $\alpha$: In our experiment, we explore the sensitivity of the regularization parameter within the range $[0,0.1, \ldots, 0.9]$. Yes, ICE-FT is exactly the same as CE-FT when $\alpha \in[1, \infty]$. However, as shown in Figure $4b$, increasing $\alpha$ did not improve the performance of unlearning accuracy, and the retain accuracy remained unchanged.
• (3) Difference Between ICE and CE Under $\alpha$: The optimal $\alpha$ for CE-FT should ideally be greater than 1; however, the analysis in [1] restricted the range of $\alpha$, making 1 the best value under this limitation. Furthermore, in Figure $4b$, ICE and CE converge and are identical when $\alpha=1$. To highlight the differences between the two methods, Figure $4 b$ only presents the results for $\alpha$ in the range $[0.1,..., 0.9]$.

Response to Weakness2:

• (1) Why is only one fine-tuning baseline presented? Our primary motivation stems from the empirical observation that while fine-tuning can maintain the utility of the model on the remaining data, it struggles to effectively forget the targeted data. We aim to provide theoretical analysis to explain this phenomenon. At this stage, we focus on fine-tuning-based experiments to validate our initial analysis. In the future, we plan to extend our work by including additional analysis and experiments with other unlearning methods, such as gradient ascent.
• (2) Additional experimental details: We have included details about tuning and training settings, along with more experimental results, in Appendix A.2. Specifically, we provide information on the unlearning setup, datasets, model architectures, and the loss values for different $\alpha$ values in ICE-FT and CE-FT.
• (3) Improving experimental presentation: Thank you for your suggestions on improving the experimental presentation! We have addressed this by: (a) Extending Figure 3-style visualizations to all datasets. (b) Including UA vs. RA curves for CIFAR-10 in Figure 4, with the Retain curve (blue line) and Forget curve (orange line) highlighted. We are working on further extending these visualizations to other datasets. (c) Could you clarify your concerns about Table 2’s organization and the incorrect legend colors in Figure 4b? This feedback would help us improve these aspects. Thank you!

Response to W3: We use the matrices $\\mathbf{F}$ and $\\mathbf{R}$ to illustrate the structure of the data matrix. Specifically, the entire data matrix can be represented as $\\mathbf{X}=[\\mathbf{X}\_r, \\mathbf{X}\_f]$, where $\\mathbf{X}\_r^{\\top}=[\\mathbf{R}^{\top}, \\mathbf{0}]$ and $\\mathbf{X}\_f^{\\top}=[\\mathbf{0}, \\mathbf{F}^{\top}]$. Consequently, based on the problem setup (lines 141-157), we can deduce the existence of $\\mathbf{w}\_*^f$ and $\\mathbf{w}\_*^r$ such that $\\mathbf{w}\_*^{} =\\mathbf{w}\_*^r + \\mathbf{w}\_*^f$, $\\mathbf{y}^f=\\mathbf{X}\_f^{\top} \\mathbf{w}\_*^f$ and $\\mathbf{y}^r=\\mathbf{X}\_r^{\top} \\mathbf{w}\_*^r .$ The dimensions $d\_f$ and $d\_r$ should be consistent with the partitioning of the feature space. The term "feature overlapping" in our work refers to scenarios where some features are shared between $\\mathbf{X}\_r$ and $\\mathbf{X}\_f$.

Thank you for your response.

W1 / W2.2

I still have reservations about this point, along with the fact that it remains unclear how the $\alpha$ value is set in your experiment. I couldn't find in Appendix A.2 how the $\alpha$ value is set?

W2.3

For Table 2, the best entry is usually bolded. As for Figure 4b, I apologize for the imprecise comment—I can’t recall what I meant and don’t see any issue with it now.

W3

Thank you for that clarification. My comment was more about whether there are missing constraints on $d_f$ or $d_r$. Don’t you need to ensure that $d_f > n_f$ and that $d_r > n - n_f$ as well?