Dynamic Model Editing to Rectify Unreliable Behavior in Neural Networks

We sincerely thank the reviewer for their thoughtful feedback and for highlighting areas for improvement. Below, we address the specific concerns raised.

The problem that is addressed is narrow … claims to explore domain adaptation …

We would like to clarify that our work is not about exploring domain adaptation, but rather correcting model unreliabilities such as caused by spurious correlation and neural Trojan. We will make this even more clear in the final paper.

The primary focus of our paper is on addressing specific types of model unreliabilities, and we have chosen to frame our language around this focus. We will update the paper accordingly to ensure this distinction is clearer.

… challenges are not very clearly described and are never shown empirically …

Motivation and Scope. Our work focuses on correcting unreliable model behavior by repurposing rank-one model editing for this task, rather than directly addressing the inherent challenges of applying rank-one model editing to domain adaptation. The identified challenges serve to illustrate critical limitations of rank-one editing in domain adaptation, which we sidestep in targeting a distinct task. The task shift makes it extremely hard to fairly compare performance across tasks, which does not make for convincing empirical evidence. Thus, we provide theoretical proofs (in App. A.1) to substantiate the identified challenges.

More on Empirical Evidence. While directly evaluating the impact of challenges within domain adaptation is beyond the scope of our work, our results demonstrate the advantages of our method within model unreliability correction. Specifically, our approach effectively corrects model behavior with minimal samples while retaining high overall performance. These benefits contrast with applications of rank-one editing in domain adaptation, which often require numerous samples and lead to more significant overall performance degradation. These outcomes underscore the importance of sidestepping the identified challenges.

Improving Clarity. To enhance clarity, we will further summarize the challenges in a more easily understandable way in our revision. For challenge 1, in domain adaptation, the model learns associative mappings that exclude $k^*$, as $k^*$ represents features that remain unaligned across domains. However, in reliability correction, $k^*$ corresponds to features aligned with the model's learned domain, necessitating its inclusion in $C$.

Repeated editing has been explored in the past …

We thank the reviewer’s valuable suggestions and for bringing up the related work [1]. While repeated editing has been explored in [1], we believe our approach introduces key novelties. In contrast to [1], which focuses on the capacity of a fixed layer to be repeatedly edited, our method introduces dynamic layer identification, allowing the model to select layers for editing based on susceptibility. This dynamic selection extends the concept of repeated editing from a fixed layer to the entire model, thus providing greater flexibility and adaptability for further exploring the potential of the edited model. To the best of our knowledge, our work is the first to propose model editing specifically for correcting unreliable behaviors, as opposed to incorporating new associations in [1]. We will incorporate a discussion of this in the revision.

Line 043 & 077

We appreciate the reviewer’s detailed reading. However, we believe there may have been a misinterpretation. In lines 043-046, we clearly state that rank-one model editing is well-suited for correcting a model’s unreliable behavior, not specifically for domain adaptation. The phrasing in our manuscript is precise on this point, but we will review the text to ensure that it is unequivocally clear and minimizes any potential for misreading. For line 077, to maintain an objective tone, we will revise the wording to allow the evaluation results to stand on their own merit.

Questions

1. In Equation (1), $f_l(k^*;W’)$ represents the mapping of a feature key $k^*$ by a layer $f_l$ with weights $W’$. We acknowledge the omission of the layer index $l$ and will correct this typo in the revision.
2. For Lemma2, the notation $x^* \rightarrow k^*$ refers to the mapping of an input $x^*$ to its corresponding feature map $k^*$ through the intermediate layers of the neural network. Additionally, $\mathcal{X}$ represents the input set of samples used to train the network. We will revise the explanation in the manuscript to make these definitions clearer.

We appreciate the reviewer’s feedback and provide our detailed response below.

… varing the proportion of backdoor data and comparing results across different configurations.

We would like to clarify that the detailed experimental setup, including the poisoning rate, is already provided in Appx 3.1 and referenced in Line 331 of the manuscript. Specifically, we poison $0.1$% of training samples $x$ with label $y\neq y^*$ to embed the backdoor trigger on ImageNet. For CIFAR-10, we set the poisoning rate of $1$%. These poisoning rates are consistent with standard configurations in backdoor attack research, ensuring a realistic balance between attack stealth and effectiveness. We believe these configurations sufficiently demonstrate the robustness of our proposed method, and varying the attack proportions is unlikely to impact the conclusions of our results.

… experiments in other domains …

We appreciate the reviewer’s suggestion to further demonstrate the method's applicability. In our submission, we intentionally focused on image datasets to establish the core effectiveness of the method. However, we agree that highlighting the scalability and versatility of our approach is valuable. To address this, we will clarify in the Introduction that the current scope of this paper is limited to image-based experiments. We will also emphasize that our method is inherently modality-agnostic, as the underlying procedures are generic and can be readily extended to other data types, such as sequences or graphs. This generalizability stems from the design of our approach, which does not rely on image-specific assumptions. We recognize this as an important avenue for future research and will include a statement to this effect in the revision.

… analysis of the time complexity …

We thank the reviewer’s suggestions regarding time complexity analysis. Below, we detail the time complexity of our method and highlight its efficiency. Let $L$ denote the number of layers in the model, and $p$ the number of key-value pairs used in the editing process. The time complexity of our approach is primarily determined by two key components:

1. Attribution computation: This step involves a forward and backward pass through the model, scaling as $O(A_L)$, where $A_L$ is the costs of the forward and backward passes across all layers.
2. Rank-one editing: The cost of inverting the static matrix $C \in \mathbb{R}^{p \times p}$ is $O(p^3)$. This inversion is computationally lightweight in our method since we use only a few cleansed samples, resulting in small $p$.

Given that the editing process iterates until convergence, let $T$ denote the number of iterations required. The total time complexity of our algorithm is $O(T\cdot (A_L+p^3))$. Our algorithm’s efficiency arises from two factors:

1. Small $p$: Since $p^3$ depends on the number of key-value pairs (determined by the cleansed samples), this term remains small due to our use of minimal samples.
2. Few iterations T: In practice, $T$ is smaller than the number of layer $L$ in the model, ensuring the overall time complexity remains low.

In comparison to existing model editing techniques that rely on larger cleansed samples and model retraining, our approach achieves competitive computational efficiency, which is evident as the total complexity remains bounded by $O(T\cdot (A_L+p^3)) \leq O(L\cdot (A_L+p^3))$.

… when the original training dataset or clean samples are unavailable …

As a model defender, clean samples are always accessible in practice, particularly when there is some knowledge about the domain or application. Additionally, the presence of corrupted samples can typically be inferred through existing anomaly detection or unreliable behavior detection techniques, which, while not the direct focus of our method, are well-established in related research.

Even in scenarios where the original training dataset is unavailable, our method offers strong capacity for practical use due to the following advantages:

• Minimal Cleansed Data Requirement. Our method demonstrates effectiveness with minimal data, requiring as few as a single pair of clean and corrupted samples to perform effective model editing.
• Image-Level Editing. Unlike methods that demand precise identification of unreliable patterns, our approach operates at the image level, eliminating the need for precise localization of corruption within samples.

Additionally, we provide further discussion on the identification of clean and corrupted samples in Appendix 8, offering insights into handling such scenarios effectively.

The definition of "UNRELIABLE BEHAVIOR" is unclear. It is quite confusing why the proposed rank-1 approach can alleviate the challenges. There is no evidence supporting the claim that rank-one model editing can "correct the model’s unreliable behavior on corrupt or spurious inputs." Additionally, the most recent reference for the study of UNRELIABLE BEHAVIOR dates back to 2019.

The key value optimization presented in Eq. (1) is a well-established formula developed by others. Therefore, the authors' contribution should primarily focus on Eq. (2). This clearly requires a gradient disagreement for estimating the function disagreement and necessitates a closed form for f.

I am uncertain how this operates within neural networks. Essentially, it is not straightforward to define a feature mapping function for each layer. Relying solely on the loss function in the final iteration may lead to issues. For Eq.~(2), updating $x$ requires a trustworthy direction and appropriate step sizes. It is unclear why $\hat x$ is updated using $x$ as a given teacher, as this could result in significant problems if the update is low-confidence.

Overall, the presentation raises a significant number of concerns. It would be beneficial to reorganize the work, as it should be more self-explanatory. The current statement comes across as a self-promotional piece of art, although there may be useful ideas.

We thank the reviewer for their recognition and constructive suggestions. We provide our response to your comments below.

Clarity ...

We do want to note that Reviewer RTf7 explicitly notes that the paper is “well written” and mentions “this is an enjoyable paper to read”.

On sidesteping challenges. To clarify how our method sidesteps the challenges identified, we provide a rigorous discussion: For a susceptible model, the training process integrates both clean samples and their corresponding corrupted counterparts. This integration ensures: $C=KK^T, K=[k_1, k_2, …, k^*], V=[v_1, v_2, …, v^*]$. This eliminates the residual $r$ such that $C^{-1}k^*\in span(K)$. Thus, the unchanged key-value associations preserve model performance, sidestepping Challenge 1. By incorporating {$x,\tilde{x}$} $\in \mathcal{X}$ in training, the model ensures $||k^*-f(x^*;W_D)||->0$ as $x^*\in \mathcal{X}$, when $|\mathcal{X}|>>0$ is not available. It mitigates the need for extensive data preparation, sidestepping Challenge 2. Please note that Paragraph 207-215 is a part of Sec 4.2 which focuses on handling the challenges highlighted in Sec 4.1. The challenges in 4.1 are for the ‘domain adaption’ problem for which rank-one editing has been used previously (while facing those challenges). We propose the first use of rank-one editing for handling backdoor and spurious correlation. We do not let those challenges limit our method by sidestepping them. Section 4 is carefully organized to provide this clear picture.

On providing empirical evidence. Our work focuses on repurposing rank-one model editing to rectify unreliable model behavior. Our objective is not about addressing the challenges of rank-one editinging domain adaptation. We deal with a different problem of unreliable model behavior. The challenges we identify in domain adaptation identify the limitations of rank-one editing, which our method sidesteps for our task. A fair experimental comparison across different tasks is nontrivial. To address concerns regarding evidence, we emphasize that our theoretical discussion validates the identified challenges. Nonetheless, our results demonstrate the advantages of our method in correcting model unreliability. Specifically, our approach achieves effective behavior correction with minimal samples while maintaining high overall performance, contrasting with domain adaptation applications that typically require more samples and suffer greater performance degradation. We will further clarify this in the revision.

Novelty over ROME … Question 1

We thank the reviewer for recognizing our contributions. Below, we clarify the additional differences, and articulate the unique contributions of our approach:

Objective focus. Unlike ROME and related works that focus primarily on domain adaptation, our method is aimed at correcting model unreliability, specifically addressing issues such as spurious correlations and neural Trojans. This distinct focus enables us to sidestep challenges typically encountered in model editing and facilitates effective correction of undesired behaviors.

Layer localization technique. We introduce a novel attribution-based layer localization technique to identify layers contributing to unreliable behavior. In contrast to methods editing with a fixed layer (e.g., the penultimate layer in Santurkar et al.), our technique allows for flexible layer selection across the model. This not only enhances editing effectiveness but also broadens the model’s editing capacity from a single layer to the entire model.

Efficiency with Minimal Data. Our approach achieves remarkable data efficiency, requiring as few as a single cleansed sample while retaining the model overall performance. In contrast, ROME often demands extensive data preparation and leads to greater performance degradation.

tldr for AC: I still have reservations about this paper’s scope, novelty, and presentation. However, the idea of editing models to improve reliability has long been of interest to the community, and this seemingly positive result could spur further research. I’d advise the AC to accept this paper if they find the overall application of editing (to fix some backdoors and spurious correlations in image classifiers) sufficiently compelling to outweigh my listed reservations. I will be maintaining my score though.

---

longer:

Discussion of rebuttals I seem to share the same thoughts as reviewer 1hwe – while the rebuttal experiment assuaged their concerns, it did not mine, as I don’t understand why the strongest version of DRO was not used. This leads to questionable results, where DRO oddly offers almost no gain. Thus, my concerns that the proposed method wouldn’t work for most spurious correlation cases linger, especially given the need for manual cleansing and that Waterbirds itself is still rather simple. If I could, I’d raise my score by 0.5 to acknowledge the effort; since reviewer 1hwe already increased their score by 1, I will keep mine as is.

I also found the response to reviewer RTF7’s critique on the simplicity of the blend attack unsatisfactory. The authors argue the blend attack is strong, but I’d argue it is easier to thwart via simple detection, and also probably easier to edit out than other attacks. As mentioned, the current work assumes knowledge of the backdoor or spurious correlation to edit -- this is also a noteworthy limitation.

Overall pros and cons

Pros:

• Good application of editing (fixing models with backdoors or spurious correlations)
• Seems to work in the selected settings, including a real world spurious feature for skin cancer detection.
• Seemingly new editing layer localization technique

Cons:

• Generally, it feels like the paper is oversold
  • Novelty over ROME still feels slim to me, even after the authors’ rebuttal.
  • The ‘rectifying unreliable behavior’ in actuality corresponds only to fixing (i) backdoors/trojaned models and (ii) spurious correlations, with the spurious correlations implemented in a way that is extremely similar to the trojans.
  • Also, the ‘models’ in question are just simple image classifiers.
• The key novelty – the layer localization technique – is not adequately studied / compared to baselines. The ‘dynamic’ part of the method seems to only marginally improve performance.

Feedback going forward

• I would reorganize this paper to focus on the novel elements. I did not find the theoretical discussion particularly insightful, and instead would have appreciated a deeper dive on the layer localization technique: which attribution technique is best? Is performance stable over different archs? Any explanation as to why different layers are selected, based on the trigger or dataset?
• I’d expand experiments to test different kinds of triggers and settings. You can probably extend beyond image classifiers easily as well.
• I’d avoid upselling the paper – mitigating backdoors is already interesting enough! A title change could be appropriate as well.

---

Final comment: I thank the authors for their work on the paper and rebuttal. I think they are working on an important problem and I encourage them to see this paper through. This paper is definitely ready for additional feedback from Workshops, and when expanded, I believe will be impactful in its full form. I wish the authors the best of luck.

We sincerely thank the reviewer for their feedback. We emphasize that Reviewer 1hwe was satisfied with our response, requesting no further clarification. Hence, our response did not go any further than satisfying the reviewer. It is a bit unfair to give a lower score based on the concerns that have already been addressed. Appreciating Nh8i’s reservations, we provide further clarification and hope that the reviewer re-evaluates our work based on the added information.

For Rebuttals

1. Comparison with Group GRO.

We would like to clarify that our method addresses a different challenge compared to Group DRO. Group DRO is a training strategy specifically designed for distributionally robust optimization. Spurious correlations in datasets such as Waterbirds and CelebA are often a result of imbalanced data, which Group DRO mitigates in model training. In contrast, our approach focuses on post-training correction. This distinction is important as our method aims to rectify the behavior of established models, and its effectiveness also depends on the model's ability to recognize the true features.

To address the concern about the performance of Group DRO in our experiments, we conducted additional tests where we applied Group DRO with stronger L2 regularization and group adjustments. As shown in Table 10, our method further enhances a model trained with Group DRO, achieving higher performance with a significantly reduced computational cost.

Table 10. Experiments on Waterbirds.

In Table 11, we also conducted experiments on the CelebA dataset to demonstrate the generalizability of our method. Using ResNet-34, we corrected spurious correlations related to gender and improved performance while keeping computational costs low. These experiments show that our method can achieve competitive results.

Table 11. Experiments on CelebA.

2. Concerns on the simplicity of Blend attack to detect and defense.

To address the concern about the visible trigger patterns, we have conducted additional experiments using the ISSBA method (Li et al., ICCV’ 2021), an invisible backdoor attack characterized by imperceptible perturbations as triggers. These invisible triggers are more challenging to detect.

In Tables 13 and 14, We evaluated ResNet-18 models trained on CIFAR-10 and GTSRB datasets. The results demonstrate that our proposed method is not limited to visible triggers and performs effectively even under invisible attack scenarios. Importantly, for ISSBA, our approach does not require knowledge of the exact trigger pattern but only clean samples. These results also align with the rationale for our methodology as detailed in Appendix 3.2.

Table 13. Performance comparison of defending against the ISSBA backdoor attack on CIFAR-10.

Table 14. Performance comparison of defending against the ISSBA backdoor attack on GTSRB.

3. Prior knowledge of backdoor or spurious correlation.

We would like to clarify that our work significantly mitigates the reliance on detailed knowledge of backdoors or spurious correlations. Specifically, we achieve reduced reliance on cleansed samples requiring as few as one pair of samples, and eliminate the need for precise trigger information by enabling image-level corrections. In contrast, existing post-hoc techniques for correcting model behavior often demand a full dataset assessment or precise trigger information. We believe our approach strikes an excellent trade-off between minimal knowledge requirements and robust performance compared to related methods. For a post-training method like ours, it is impractical to require performance on par with training-based methods or to expect it to operate with no prior knowledge as the model attacker.

For Cons

1. Oversold.

We respect the reviewer’s perspective on the novelty. However, we have clearly defined the scope of "unreliable behavior" in the paper, with a detailed discussion in the related works section. Regarding spurious correlations, in addition to the ISIC dataset, we now include experiments on both Waterbirds and CelebA. We believe that referring to spurious correlations and backdoors as forms of unreliable behavior is both valid and consistent with the objectives outlined in our paper.

2. The improvement of the layer localization technique.

We must highlight that the improvement brought by the layer localization technique is significant.

The layer localization technique is central to our method and is motivated by a critical observation: editing different layers yields distinctive performance outcomes. This is demonstrated in Figures 2 and 8, where we identify a serious limitation in existing methods that focus on editing only the last layer, often leading to suboptimal results or performance degradation.

To ensure fairness, we avoid direct comparisons with methods constrained by this limitation in the main text. However, for completeness, we supplement these comparisons in Tables 6 and 7 (Appendix A.6), which clearly demonstrate the effectiveness of layer localization in achieving robust performance. For ease of reviewing, we briefly conclude the main results in Tables 15 and 16.

Table 15. Attack Success Rate comparison of defending against the backdoor attack on Trojaned models trained with the Phoenix logo on CIFAR-10 and ImageNet.

Table 16. Errorously increased accuracy for spurious features on models trained with the Phoenix logo on CIFAR-10 and ImageNet.

In addition to the aforementioned experiments, we have designed a variant of the Blend attack that fixes the penultimate layer to manipulate the model's prediction using triggers. The results, presented in Table 17, demonstrate that the default setup used in existing model editing work is vulnerable to this simply modified attack. In contrast, our layer localization method effectively mitigates this vulnerability, showcasing its advantages in enhancing model robustness.

Table 17. Comparison of attack success rate on models trained under a Blend attack with the fixed penultimate layer on CIFAR-10 and ImageNet.

For Going Forward

We thank the reviewer for the valuable suggestions to improve our work. We would like to clarify that many of the suggestions have already been addressed or are incorporated in the current version. The concerns, particularly regarding attribution methods and the mitigation of backdoor Trojans, are important and have been considered with respect to readers from diverse backgrounds.

1. Alternative attribution methods.

The selection of the path attribution technique is based on its ability to measure changes between samples while satisfying the Completeness axiom, ensuring reliability in identifying critical layers. Other methods, including gradient-based or perturbation-based techniques, do not meet these requirements for measuring attribution change. Thus, experiments for testing different attribution methods are not feasible in this context.

To show the stability of our layer localization technique across architectures, we have provided results in Table 9 (Appendix A.8), which demonstrate consistent performance across different model architectures.

2. To test different kinds of triggers and settings.

We have already tested different trigger patterns, as shown in Tables 1, 6, and 7. Additionally, in Tables 2 and 3, we analyze the generalizability of our method on models Trojaned with triggers of varying visibility and locations. We have also included experiments on invisible trigger patterns, offering a comprehensive discussion on trigger variations and settings.

3. A title change.

We thank the reviewer for recognizing our contribution. Whereas we maintain that the title reflects the content well, we are open to adjusting it. The reviewer can suggest an appropriate title.

We thank the reviewer for the thoughtful feedback. We have provided our response below and hope it addresses your concerns.

… the reliance on the identification of unreliabilities …

While our method relies on the identification of unreliable behaviors, this reliance is reasonably practical. In fact, as compared to existing methods, it is more practical. Specifically, our method offers the following advantages:

• Reduced reliance on cleansed samples. Our approach requires only a single pair of corrupted and cleansed samples to achieve effective correction. This makes it adaptable to scenarios where extensive cleansed datasets are unavailable, enabling robust model editing even in resource-constrained settings.
• No need for precise trigger information. Unlike methods that depend on exact pinpointing of backdoor triggers or spurious features, our method operates at the image level. This bypasses the need for pixel-level precision, reducing the complexity and effort associated with image cleansing while maintaining strong correction performance.

Enabling model editing with only coarse detection of anomaly, our method remains practical. In Appx. 8, we discuss the generalization capabilities of our method, and we will emphasize additional insights and future research directions to enhance its applicability in broader contexts.

… for highly complex networks …

Our method is extendable to complex networks. The concern regarding the applicability of our method to models with intricate computational graphs aligns with the motivation of our attribution-based layer localization approach. Attribution methods are designed to trace output changes back to specific architectures within the model, enabling us to identify the layers or modules most responsible for unreliable behavior. This identification mitigates the need for exhaustive modifications, making our approach suitable for complex architectures.

… dealing with stronger poisonous attacks …

We appreciate the reviewer’s feedback and agree that clarifying the severity of the attack would strengthen our argument. Unlike newer attack methods that focus on stealth, such as those using minimal perturbations to evade detection, the blend attack embeds triggers directly into the input, maximizing the attack's impact on the model's predictions.

The blend attack’s capacity to strongly bias the model’s output toward the target class underscores its severity as a threat. Given this, our method’s demonstrated robustness against such attacks provides strong evidence of its efficacy. Moreover, we believe that our approach would generalize effectively to newer or more aggressive attack methods, as they typically balance between stealth and potency in ways the blend attack already represents.

To address the reviewer’s concern, we will include additional content in the revision to explain our rationale for selecting the blend attack and to compare its severity with other types of attacks.