Thanks for the constructive and insightful comments. We carefully revised the manuscripts based on all reviewers' comments. Please see the revised manuscript at https://anonymous.4open.science/r/Gradient_rewiring_editing-E16E/GRE_NeurIPS24.pdf.

[W1: Presentation on GRE and GRE+]

Ans: hank you for the constructive comment. We respectfully disagree with the assertion that GRE isn't consistently ranked as the second-best method behind GRE+. We believe that treating GRE and GRE+ as different methods in our writing is reasonable.

GRE achieves second-best performance in both sequential editing and batch editing, while demonstrating competitive performance in independent editing. In the independent editing task, all methods can achieve a high success rate since it is relatively simple to correct one target sample. However, GRE still secures second-best performance in several instances, such as on the ogbn-arxiv and ogbn-products datasets in GCN. More importantly, GRE excels in more complex editing scenarios, such as sequential editing and batch editing. For sequential editing, GRE achieves second-best performance across various datasets, as shown in Figures 3 and 6. Similarly, for batch editing, GRE also ranks second-best, as evidenced by the results in Tables 6 and 7.

In summary, GRE shows significant improvement over baselines in sequential and batch editing. From a technical perspective, GRE provides a closed-form solution for gradient rewiring, while GRE+ requires a numerical QP solver. Therefore, we treat GRE and GRE+ separately due to the significant performance improvements offered by GRE and its distinct closed-form solution.

[W2: Model editing exists for CNNs and NLP tasks but not specifically the Transformer architecture]

Ans: To avoid confusion, we have revised the statement of Observation 1 in Section 4.2. Specifically, we removed the part about transformers and generally mentioned that all editors can effectively rectify model predictions for independent editing in the graph domain.

[W3: No analysis/study of the cost/complexity of the model editing methods]

Ans: For comparing time complexity and memory consumption, we measured the wall-clock editing time in milliseconds and GPU peak memory in megabytes, reporting average values over 50 independent edits in Appendix D.5. Due to the limited rebuttal length, we only show time complexity and memory consumption for GCN.

Our results show that the proposed method is scalable in terms of memory consumption and has manageable editing time overhead. For example, in the GraphSAGE architecture on the Flickr dataset, GRE+ (5) results in only a 13.5% increase in peak memory compared to GD. In terms of wall-clock editing time, the most time-consuming version, GRE+ (5), shows an insignificant overhead, with only a 6.31% increase on the ogbn-products dataset in the GraphSAGE architecture. These observations demonstrate the scalability of GRE+ for large datasets. It is noteworthy that model editing is usually efficient and fast, making slightly slower editing affordable while improving editing effectiveness.

[W4: Add citations and other nitpicks.]

Ans: We have added citations for several statements and revised the format and writing issues accordingly.

[Q1: Fig 3 where are results on ENN?]

Ans: Thank you for your constructive comment. Model editing consists of two stages: the pre-training stage, where a well-trained model is obtained, and the editing stage, where undesirable behavior is corrected. The GD, GRE, and GRE+ methods perform editing during the editing stage using the same pre-trained model. For these methods, a higher test accuracy drawdown implies a lower test accuracy after editing. In contrast, the key goal of the ENN baseline is to improve the model during the pre-training stage. The objective at this stage is to obtain a model that can be easily edited using gradient descent while maintaining good task performance. Since the test accuracy of the pre-edited models varies, it is infeasible to compare these methods in terms of test drawdown. To fairly compare all methods, we have added a comparison of the test accuracy performance after editing for sequential editing in Appendix D.6. It is observed that the accuracy of GRE+ is higher than all baselines.

Thanks for the constructive and insightful comments. We carefully revised the manuscripts based on all reviewers' comments. Please see the revised manuscript at https://anonymous.4open.science/r/Gradient_rewiring_editing-E16E/GRE_NeurIPS24.pdf.

[W1: Add comparison in terms of editing time and memory.]

Ans: For comparing time complexity and memory consumption, we measured the wall-clock editing time in milliseconds and GPU peak memory in megabytes, reporting average values over 50 independent edits in Appendix D.5.

Our results show that the proposed method is scalable in terms of memory consumption and has manageable editing time overhead. For example, in the GraphSAGE architecture on the Flickr dataset, GRE+ (5) results in only a 13.5% increase in peak memory compared to GD. In terms of wall-clock editing time, the most time-consuming version, GRE+ (5), shows an insignificant overhead, with only a 6.31% increase on the ogbn-products dataset in the GraphSAGE architecture. These observations demonstrate the scalability of GRE+ for large datasets. It is noteworthy that model editing is usually efficient and fast, making slightly slower editing affordable while improving editing effectiveness.

[W2: Tackle lots of writing/grammar issues]

Ans: Thanks for the careful review. We have revised the manuscripts significantly according to your comments.

[W3: Format issues on Table and Figures]

Ans: We have revised the caption and uniform decimal precision in Tables 1 and 2, increased the font of the text for both Figures 1 and 2, and revised Figure 4.

I thank the authors for their rebuttal. I believe that the authors have significantly improved the writing for their revision. The new time and memory comparison also strengthens the paper.

Still, I would like to point out that the writing could still be improved:

• Line 298: in this experiment, we scrutinize the sensitivity of our proposed method - in my opinion, "scrutinize" is an odd term in this context. While not wrong, I believe that "analyze" would have been a better choice.

• Line 21: I think the first sentence is still an odd way to present GNNs; "integrating" doesn't really fix the main issue. I would suggest that the authors revamp their first introduction paragraph entirely.

• Line 28: "In the ideal scenario, the promising property of tackling such errors would be threefold: ..." - something along the lines of "An ideal method that could tackle such errors would need to have the following properties: ..." would be, in my opinion, much clearer.

• There are still some decimal precision issues in the tables - for instance, in Tab. 1 the SR column contains both $1.0$ and $0.98$, Tab. 2 Column Flickr-ACC; 2-GCN-GD $11.0$ for the std but GCN-GRE $1.50$.

Again, this is not a comprehensive list. I would recommend that the authors do some more passes through the manuscript for the next revision.

Nevertheless, the writing has been significantly improved, and the method seems effective and practical - I am leaning towards acceptance right now, and have updated my score from a reject (3) to a borderline accept (5), with the presentation score going from poor (1) to fair (2).

I will continue watching the rebuttal discussions, and might modify the score further depending on other comments.

Thanks for the constructive and insightful comments. We carefully revised the manuscripts based on all reviewers' comments. Please see the revised manuscript at https://anonymous.4open.science/r/Gradient_rewiring_editing-E16E/GRE_NeurIPS24.pdf.

[W1: This general approach can be tested on various tasks.]

Ans: Thank you for your insightful observation. We agree that while our experiments were conducted on graph datasets, the proposed gradient rewiring method is not inherently specific to graphs, the gradient rewiring method is particularly suitable in the graph domain due to the small model size. Specifically, graph models are typically a few layers and thus are smaller in model size compared to models (e.g., Transformers) used in NLP and CV tasks. This results in lower computational and storage costs for gradients, making our strategy particularly suitable for the graph domain. Additionally, it is more challenging to edit nodes in a graph due to the inherent propagation process within neighborhoods. Such propagation may lead to significant gradient discrepancies within the graph domain.

In this work, we primarily focus on gradient rewiring within the graph domain, exploring multiple editing manners (i.e., independent editing, sequential editing, and batch editing). We acknowledge the potential applicability of our gradient rewiring strategy to other domains and leave its exploration for future work in Appendix B.

[W2: There are noticeable formatting errors and typos in the paper.]

Ans: We have carefully revised the manuscript. Please check the revised manuscript for the details.

[Q1: Provide a more detailed explanation of the experimental process and results in Figure 1.]

Ans: Thank you for the insightful comment. Gradient similarity does not necessarily imply consistent test and training loss. Loss consistency primarily depends on the model parameters rather than the first-order gradient. Therefore, there are cumulative effects of gradient inconsistency; significant initial gradient discrepancies can lead to substantial differences in model parameters, resulting in inconsistent training and test loss. We have added a more detailed explanation in Figure 1.

[Q2a: Conceptually compare their method with curriculum learning]

Ans: Thanks for the insightful comment. We have added the discussion with curriculum learning in Appendix~F. Here is the discussion:

Curriculum learning and model editing are two distinct approaches in the field of machine learning. Curriculum learning is an approach where the network is trained in a structured manner, starting with simpler tasks and gradually introducing more complex ones. This method aims to improve the learning process by mimicking how humans learn. Model editing is a fast and efficient approach to patch the well-trained model prediction for several failed test cases. Although both are multi-stage training stages, there are several key differences: (1) Goals: Curriculum Learning aims to improve the overall learning process by structuring the training data in a way that mimics human learning. In contrast, model editing aims to make targeted adjustments to a pre-trained model to correct undesirable behaviors. (2) Approach: curriculum learning mainly focuses on the sequence and complexity of the training data. Model editing typically modifies the model's parameters or architecture to correct undesirable behavior goals. (3) Additional information in the multi-stage process. Model editing requires failure feedback for well-trained models as the target samples to patch, e.g., test failure cases after production is launched. In other words, such feedback can only be obtained after model pertaining. In curriculum learning, all information is given in multi-stage training. In summary, curriculum learning focuses on structuring the training process to improve overall learning, while model editing focuses on making targeted adjustments to a pre-trained model to correct specific behaviors. Both approaches can be complementary and used together to achieve better model performance.

[Q2b: More experimental results on batch editing.]

Ans: We have added experimental results on batch editing in Appendix D.4 (Tables 6 and 7). We observe that, compared to independent editing, batch editing is more challenging as all batch samples need to be patched simultaneously. Additionally, GRE and GRE+ both demonstrate significant performance improvements compared to GD and ENN on various model architectures and datasets.

Thanks for the constructive and insightful comments. We carefully revised the manuscripts based on all reviewers' comments. Please see the revised manuscript at https://anonymous.4open.science/r/Gradient_rewiring_editing-E16E/GRE_NeurIPS24.pdf.

[W1: The significance of editing graph neural networks remains unclear]

Ans: Model editing is crucial for ensuring that machine learning models remain reliable and effective post-deployment. This practice addresses high-profile failure cases and misbehaviors that emerge after a model's initial training, often brought to attention through user feedback.

In general, there are editing-worthy scenarios, such as high-profile failures and prioritizing critical mistakes. For example, users may try to prompt large language models (LLMs) to misbehave (e.g., giving criminal advice, system prompt, etc.) to induce "high-profile failures." In the context of self-driving cars, misclassifying a child as a cat poses far greater risks than misclassifying a cat as a dog. Although graph applications typically don't have this kind of direct interaction with users and are not as intuitive, they certainly have high-stake scenarios, such as patient readmission [1] and flood prediction [2], which warrant the study of model editing.

In short, while model editing has been widely explored in computer vision (CV) and natural language processing (NLP) tasks, it has rarely captured attention in the graph learning community. It is indispensable to investigate graph model editing for high-stake graph applications.

[1] Predicting patient readmission risk from medical text via knowledge graph enhanced multiview graph convolution. SIGIR 2021 [2] Kazadi, Arnold N., et al. "Flood prediction with graph neural networks." Climate Change AI. Climate Change AI (2022).

[W2: Please discuss gradient modification literature in related work]

Ans: We have added a discussion on the literature regarding gradient modification in continual learning and meta-learning. Please see more details in Appendix~E.

[W3: More experimental details such as data spliting]

Ans: For all datasets, we first randomly split the data into train, validation, and test sets. Specifically, we ensure that each class has 20 samples in the training set and 30 samples in the validation set. The remaining samples are used for the test set. The target node is randomly selected multiple times from the validation set where the well-trained model makes incorrect predictions.

[W4: The time complexity is not provided.]

Ans: For comparing time complexity and memory consumption, we measured the wall-clock editing time in milliseconds and GPU peak memory in megabytes, reporting average values over 50 independent edits in Appendix D.5. Due to the limited rebuttal length, we only show time complexity and memory consumption for GCN.

Our results show that the proposed method is scalable in terms of memory consumption and has manageable editing time overhead. For example, in the GraphSAGE architecture on the Flickr dataset, GRE+ (5) results in only a 13.5% increase in peak memory compared to GD. In terms of wall-clock editing time, the most time-consuming version, GRE+ (5), shows an insignificant overhead, with only a 6.31% increase on the ogbn-products dataset in the GraphSAGE architecture. These observations demonstrate the scalability of GRE+ for large datasets. It is noteworthy that model editing is usually efficient and fast, making slightly slower editing affordable while improving editing effectiveness.

[W5: Some minor issues (e.g., a superscript linking to nothing in the caption of Figure 1).]

Ans: We have carefully revised the manuscript. Please check the revised manuscript for the details.