Discovering Knowledge-Critical Subnetworks in Neural Language Models

We would like to thank reviewer VC2F for their helpful feedback. We are grateful that the reviewer finds our analysis vast and our paper well-written and accessible. The reviewer also stated that our work has significant implications for future work in model interpretability. We address their questions on limited metrics and whether knowledge erasure is prompt dependent in the general message to all reviewers. We wrote the requested changes and clarifications in the paper in purple.

The effectiveness of knowledge removal: Due to limited metrics, it is not clear whether knowledge removal is effective. Perplexity on verbalized triplet prompts alone is not enough.

As mentioned in the general message to all reviewers, the reviewers wondered whether the rank of the gold tail entity can be used as an alternative metric. We also wondered about this and we report the rank differences between the remaining model and the original pretrained model in Appendix D, Table 10 (originally Table 11). These results align with the suppression and maintenance criteria described in Section 4.1. We added a forward pointer to these results in the main body in Section 5 - “Success Metrics” paragraph.

The effectiveness of knowledge removal: Could the perplexity drop be specific to the verbalization template?

We provide performance on paraphrases in Appendix H (originally Appendix G) for three WordNet hypernym TargetKGs. We have found that the average perplexity differences are high on paraphrases of TargetKG, and the perplexity differences for ControlKG either are near 0 or are negative. We added a forward pointer to these results in our main body in Section 6.3.

The effectiveness of knowledge preservation: Besides removing knowledge in TargetKG, the pruned model could sacrifice some other knowledge that is not in ControlKG and ControlLM. To make sure that the maintenance criterion is not overfitted to ControlKG and ControlLM the authors should use a separate corpus for ControlKG and ControlLM.

As stated at the end of Section 5, paragraph “Datasets”, all results on ControlKG and ControlLM are on a held-out test set.

Lack of analysis on sparsity: 99% sparsity (~1M parameter for GPT2-small) for 10-20 triplets in TargetKG does not seem very sparse. I would personally expect a much higher sparsity (e.g., 99.99%) for a knowledge-critical subnetwork of 10-20 triplets, as a pre-trained language model typically stores a vast amount of knowledge.

The reviewer has an interesting intuition on the size of knowledge-critical subnetworks, which we also share. An efficient and modular language model would ideally dedicate only a small amount of parameters to remove information on 10-20 triplets. However, it is not clear whether the amount of parameters needed to encode information scales with the amount of knowledge. In particular, the model could robustly encode information in a repeated manner across several parameters, since it has been originally pretrained with dropout. Moreover, this size could also be dependent on the knowledge type. For example, with limited analysis, we found that subnetworks for the representation KG tend to be larger than the location and communication KGs, although representation has fewer triplets than the rest. A hypothesis could be that more common or fundamental knowledge that affects other types of knowledge requires more parameters to be removed.

We also want to emphasize that the sparsity was calculated over the masked regions rather than the whole subnetwork. Therefore, the majority of the subnetworks were a bit smaller than 1M parameters.

We would like to thank reviewer njyU for their feedback. Notably, the reviewer found our approach as a novel way to discover knowledge-critical subnetworks. The reviewer considers our approach a potential future application in model editing and finetuning. We wrote the requested changes and clarifications in the paper in purple.

Can you provide more details about the filtering processes applied to the sampled connected KGs? How did these processes ensure the quality and balance of the sampled graphs?

We describe the filtering processes applied to the sampled connected KGs in Appendix A. In summary, once a small connected KG is sampled, we apply two filtering processes. The first one enforces many-to-one relationships in the $K_T$ graph to avoid head entities with multiple tails. This ensures that the dataset is not dominated by certain head entities and balances the graph. The second filtering process reduces the tail-entity imbalance to avoid over-fitting to a small set of tokens. For this, we count the frequency of the tail tokens in the sampled graph and keep at most a quartile amount of triplets with shared tail entities.

How did you validate the effectiveness of the discovered subnetworks in suppressing the expression of target knowledge triplets?

We validate the effectiveness of the discovered subnetwork by:

1. Measuring the effect on the TargetKG, a validation ControlKG, and a validation LModeling when the subnetwork is removed from the original model
2. Comparing with a randomly masked baseline to see if the increase in the perplexity can be matched with an equally sparse removed subnetwork
3. Doing an ablation study to test whether all of the terms are required to get the most effective suppressing effect
4. Verifying whether, for a downstream task, a forgotten test set knowledge can be transferred through finetuning – we find that removing the subnetwork indeed hinders this ability
5. Establishing whether worse scoring paraphrases of the triplet on average increase in perplexity for TargetKG and stay similar for ControlKG.

We would like to thank reviewer ikUx for their thoughtful feedback. We appreciate that the authors find the problem very timely and our method as a sensible way to approach it. We also address the reviewer’s question on limited metrics, cross-topic subnetwork analysis, and robustness to knowledge rephrasing in the general message to all reviewers. We wrote the requested changes and clarifications in the paper in purple.

The authors are showing the sparsity level over the masked parameters. Therefore 98% sparsity does not mean 98% of parameters in LLMs are not used. It would be clearer if the authors displayed the real sparsity levels.

Indeed, 98% sparsity means the sparsity of the removed subnetwork mask over the masked layers (in this case, the upper-half layers of the model). To make this more intuitive, we suggest reporting the density of the removed subnetwork across all parameters in the network, including those that are not masked. As it would affect the phrasing in many parts of the paper, we suggest making this revision for camera-ready.

Perplexity ranges can be very big. Would it be more sensible to use the rank of the target entity token?

As mentioned in the general message to all reviewers, the reviewers wondered whether the rank of the gold tail entity can be used as an alternative metric. We also wondered about this, and so in the submission, we report the rank differences between the remaining model and the original pretrained model in Appendix D, Table 10 (originally Table 11). These results align with the suppression and maintenance criteria described in Section 4.1. We added a forward pointer to these results in the main body in Section 5 - “Success Metrics” paragraph.

We chose to display perplexity differences in the main body instead of rank as they better capture model certainty on the gold-tail token. For example, there could be cases where the remaining model has a larger rank than the original model leading to a large difference, but the probability differences between the two can be very small, and therefore not enough information about model certainty is reflected. Particularly, in our setting, where we optimize for a small difference between the TargetKG tail-token prediction distribution and a uniform distribution over all tokens in the vocabulary, we are bound to have multitudes of tokens being considered equally probable. This makes the rank metric less interpretable than perplexity. However, we agree that both metrics provide complementary views on this question, which is why we reported rank in the appendix initially.

Lack of ablation on sparsity levels – Does varying the sparsity level from 98% to 20% or 60% change the conclusion?

The reviewer brings an interesting perspective from the point of view of compression. Differentiable-weight masking is popular as a compression technique (Sanh et al 2020, Zhao et al 2020). Varying the sparsity level would require implementing a different type of regularization which could be possible. However, we instead designed an aggressive sparsity regularization as our goal in finding a knowledge-critical subnetwork is to find the minimum compression ratio to achieve semantic suppression of the knowledge. As a result, while an interesting research direction on its own, we think this analysis may not align with the core inquiry of our paper.

We would like to thank reviewer cTTe for their insightful feedback. We thank the reviewer for emphasizing that very little is known about how pretrained language models represent knowledge. We also value their acknowledgement of the possible future applicability of our work in factual error correction. In addition to this comment, we also address the reviewer’s questions on cross-topic analysis of subnetwork structures and whether knowledge erasure is prompt-dependent in the general message to all reviewers. We wrote the requested changes and clarifications in the paper in purple.

The experiments showed that the subnetworks typically consisted of between 1% and 2% of the network parameters. But which groups of parameters were they?

To address the question, we created new sections Section 6.2 and Appendix K called “Structural Analysis”. While a more detailed analysis can be found in the revised paper, we summarize here our findings across three WordNet TargetKGs and three seeds:

• Depth-wise: We have found that consistently the subnetwork is most dense around the first and final masked layers (in the case of GPT2-small, these would be layers 7 and 12).
• Layer-type wise:
  • For layer 7 and layer 12, we found that knowledge-critical subnetworks are most dense in the output layer of attention modules.
  • However, in middle layers, the most dense layer types are the Feed-Forward networks, in particular, the first linear layer in each Feed-Forward network.
• Contiguity/Locality/Distributedness:
  • We have not found any complete columns or rows of weights that were dense in the critical subnetwork. This means that there are no input or output neuron features that get completely removed when the critical subnetwork is removed. Therefore the masked region may not be working to zero-out the knowledge by turning specific features off, which would counter the prevailing view that neuron-level changes are necessary for mechanistic interventions (Dai et al. 2022, Meng et al. 2022).
  • However, in Figure 9, in the case of three WordNet TargetKGs and three seeds, regardless of the KG and seed configuration, we found that the 10th head in layer 7, and the 1st and 9th head in layer 12 are significantly less sparse. Therefore while the subnetworks are not overlapping according to the Intersection-over-Union metric, they tend to be dense in similar areas.

Was there any other topical structure associated with these subnetworks?

As mentioned in the general message to all reviewers, to address this inquiry, we created a new section in Appendix J called “Knowledge-Based Analysis”. We did the same experiments as Appendix I (originally H), where we calculated the Intersection-over-Union of the subnetworks across different layers and layer types for the same seed but different KGs (Fig. 7) and composed subnetworks for the same seed and different KGs (Table 18).

We had similar findings as Appendix I, where the Intersection-over-Union was overall low but most significant at the attention sublayer of the final transformer block. We also observed similar results for composing subnetworks. Therefore, it may be possible to combine subnetworks across seeds or KGs naively, however, they may not guarantee the maintenance criteria to the same extent.

Could a motivated individual recover the erased knowledge using clever prompting or a fine-tuning process? (i.e. the knowledge is still in the network)?

We provide performance on paraphrases in Appendix H (originally Appendix G) for three WordNet hypernym TargetKGs. We have found that the average perplexity differences are high on paraphrases of TargetKG, and the perplexity differences for ControlKG either are near 0 or are negative. We added a forward pointer to these results in our main body in Section 6.3, as it is also a robustness analysis.

Moreover, we show in Section 6.4 that removing a knowledge-critical subnetwork harms the ability to transfer the knowledge to a downstream task. We have changed Tables 5 and 15 to clearly show accuracy differences from the original model (rather than the actual accuracy result for each baseline).

If there were many explicit instances of the knowledge in the training set used to finetune the model, it’s possible that fine-tuning the model after removing the knowledge-critical subnetwork could re-form a representation to recover the knowledge. However, we also note that in the case of blackbox systems, users often have no access to finetune the underlying model, so removing subnetworks may represent a promising method for dampening the model’s knowledge of particular types of sequences.