Localizing Memorization in SSL Vision Encoders

W1: Novelty vs. extension of [1]

The paper targets an orthogonal research question to [1]. While [1] is concerned with the question of quantifying memorization and identifying memorized samples, we answer the question of where inside the SSL encoders is the information stored. While doing so, we make multiple original contributions as summarized in W3 for Reviewer Reviewer BVAM.

W2a: Claims on being first metric for localizing memorization in SSL

We wanted to frame our work as being the first metrics particularly designed for localizing memorization and the first that allow localization down to the level of individual units. To better express this framing according to the reviewer’s insightful remark, we altered the following sentences (change bolded):

• Introduction: "We propose LayerMem and UnitMem, two practical metrics that allow to localize memorization in SSL encoders on a per-layer basis and, for the first time, down to the granularity of individual units.”
• Conclusion: "We propose [REMOVED] practical metrics for localizing memorization within SSL encoders on a per-layer basis, and, for the first time, down to the individual-unit level.”

Additionally, we added in line 107 "The same trend was also reported for SSL [36]" to give credit to Meehan et al. [36] for their contribution to the field (see their Appendix A7).

W2b: Downplaying claim on memorization in ViTs

We also changed this claim in the paper (changes in bold):

• Introduction: “Yet, with our methods to measure memorization, we are able to show that the same trend holds in vision transformers that was previously reported for language transformers, namely that memorization happens in the fully-connected layers.”
• Line 217: “The localization of memorization in transformers was primarily investigated in the language domain [REMOVED] and the methods for analysis and the findings are not easily transferable. Yet, with our methods to localize memorization, we [REMOVED] show that the same trend holds in vision transformers that was previously reported for language transformers, namely that memorization happens in the fully-connected layers.”

W2c: Empirical comparison to prior work

The two existing works on memorization in SSL [36,47] (SSLMem and SSL Deja Vu) consider an orthogonal research question, namely quantifying memorization and identifying memorized samples. We, in contrast, focus on localizing memorized content. An empirical comparison does not seem sensible.

Yet, we can compare the trends reported by the ablation from Meehan et al. [36] (their Appendix A7: Deja Vu memorization for 4 different layers of VICReg) to ours. They report that Deja Vu Memorization increases monotonically and occurs mainly in the last two layers. A similar trend for accumulated memorization is observed by our LayerMem, while our Delta LayerMem shows that the actual increase is not monotonic.

W3a: Difference in memorized samples between two different encoders

This interesting research question is orthogonal to our work where we identify where in the encoder training data is memorized.

W3b: Difference in memorized samples across different layers

We performed a new analysis in Figure 2 and Table 3 (attached PDF). The overlap within the 100 most memorized samples between adjacent layers is usually high but decreases the further the layers are separated. Our statistical analysis to compare the similarity of the orderings within different layers’ most memorized samples using the Kendall’s rank correlation coefficient shows that while for closer layers, we manage to reject the null hypothesis (“no correlation”) with high statistical confidence (low p value) which is not the case for further away layers.

W4: Hyperparameters and variation in memorized samples.

We performed an additional experiment where we trained encoders f and g independently with a different random seed (yielding f’ and g’). We compared the overlap in most memorized samples between encoder f (from the paper) and f’. The results (Table 4, attached PDF) show that overlap is overall high (min. 69% in layer 2) and increases in the later layers (max. 90%, final layer).

W5: Weight decay

We performed the experiment suggested by the reviewer and trained ResNet9 using SimCLR on CIFAR10 with three different levels of weight decay. Our results in Figure 1 (attached PDF) show that stronger weight decay yields lower memorization, yet also decreases linear probing accuracy.

Q1: BatchNorm

We measured the cosine similarity between the weights and biases of the BatchNorm layers for two encoders (trained on CIFAR10 and STL10). The results in Table 5 (attached PDF) show a high per-layer cosine similarity (average over all layers=0.823). This suggests that the statistics are not far off, hence, no adjustment is required. We hypothesize that the similarity stems from the fact that the data distributions are similar and that we normalize both input datasets according to the ImageNet normalization parameters.

Q2: Compute times

Following our standard setup from Table 1 and Figure 1 from the main paper, we report the following timing on an Nvidia RTX4090 (24G), 64GB RAM (for ResNet9 trained 600/200 epochs):

Limitation 1: Need of two models

This limitation only holds for LayerMem. UnitMem operates directly on the given encoder without the need for a second model. We added the limitation of LayerMem to Appendix F.

Limitation 2: Lack of sample-level analysis

This is an orthogonal research question to this work. We do not aim to study memorization from the sample but from the encoder perspective, asking the question where inside the encoder training data is stored.

I would like to thank the authors for addressing my concerns. The rebuttal and new experiments provide interesting insights and will make the paper stronger. Therefore, I increase score.

We thank the reviewer for their insightful comments.

W1: It is not clear why the defined metric is a good measure of memorization, especially for comparing memorization across different layers. The magnitude of the distance between representations of different augmentations is highly dependent on the activation distribution of the layers and the distance metric used. While the normalization proposed in equations 2 and 4 attempts to address this, I am still unsure if it is sufficient.

To avoid that our memorization measure overfits to a particular set of augmentations whose activations might indeed vary throughout layers, we measure our LayerMem as an expectation over different pairs of diverse augmentations. Following SSLMem, in our experiments, we calculate LayerMem as an average over 10 different randomly chosen pairs of augmentations from the full augmentation set used during training.

The only pitfall that could arise would be that layers with higher output dimensionality get a higher LayerMem (simply because there is more mass in the activation). To verify that this is not the case, we report for ResNet50 trained with SimCLR on CIFAR10 the LayerMem and output dimensionality for the last layers of each block. In the Table below, we see that the dimensionality of the representations after the 1st block and the 4th block are the same. However, the memorization score is much higher for the layer after the 4th block highlighting that our LayerMem is independent of the output dimensionality.

W2: To justify the validity of the LayerMem memorization metric, they argue that effective memorization in SSLs leads to strong downstream performance. They also point out that replacing the layers with the highest memorization scores, as determined by the metric, results in a more performance drop, indicating the metric's reliability. However, I believe this justification alone is insufficient. Are there other pieces of evidence supporting the validity of this memorization metric?

The finding that effective memorization leads to strong downstream performance has been proven for supervised learning by [23] and for SSL by [47]. We build on these prior findings and show that LayerMem can help us to select layers for fine-tuning to achieve better downstream performance. Does the reviewer have any other suggestions on how we could provide additional evidence? We are happy to implement them.

Q1: Layer Replacement

We agree that swapping affects subsequent layers. If we replace layer N, the input to layer N+1 and all subsequent layers is altered. Following this logic of a cascading effect, replacing the earliest layers should result in the highest impact, simply because most layers are affected by the change. However, this is not what we experimentally observe (e.g., in Table 22). Instead we see that swapping out the most memorizing layer(s) causes the largest change in the downstream performance.

What is the interesting insight from the results is that the replacement of different layers has a different impact on the test accuracy of STL10. In particular, we observe that replacing most memorized CIFAR10 encoder layers with their STL10 equivalent boosts the STL10 performance significantly more than replacing, for example, the least memorized or random layers. This indicates that the layers with the highest memorization have the highest impact on the downstream performance.

Q2: Distance Metric

Following SSLMem, we used the $\ell_2$ metric for the main paper. To address the reviewer’s questions, we performed an additional experiment reporting LayerMem with different distance metrics ($\ell_1$ distance, cosine similarity, angular distance). Our results in the table below (see also Table 1 in the attached PDF) highlight that 1) the memorization scores are very similar, independent of the choice of the distance metric, and 2) the most memorizing layers according to LayerMem and Delta Layermem are the same over all metrics. This suggests that our findings are independent of the choice of distance metric.

W1a: In depth analysis and practical insights

We thank the reviewer for their suggestion. We went through the paper and identified multiple sections for adding practical insights. We summarized them in the first reply to W1 for Reviewer BVAM.

If reviewer see some other parts in the paper where we could add more analyses (as well as intuitions and potential implications), we’d be grateful for the suggestions.

W1b: Applying findings in real-world settings

In addition to Section 4.2, we also provide a further real-world application of our metrics by showing that UnitMem can effectively be used to inform compression schemes. In particular, it seems to be most helpful to prune the least memorizing units (vs. random units), see Table 5. On the contrary, the most memorizing units should be preserved.

W2a: For example, according to Equation 1, LayerMem seems to depend on the neural network’s properties. If neural networks are sufficiently equivariant with respect to data augmentation, then SSL and SSLMem should be zero.

Could the reviewer further clarify what they mean by SSL (and SSLMem) should be zero? We do agree that a perfectly (over)fitted SSL-trained encoder f can have a zero representation alignment, i.e. the representations of two different augmentations of the same input sample x are identical. However, SSLMem and therefore also our LayerMem rely on an additional reference model g and report the difference in representation alignment between f and g.

In case g has not seen x but still achieves perfect representation alignment means that the distribution of other data is sufficient to perfectly fit x, which also means that f does not need to memorize x. Then indeed LayerMem will be zero, correctly indicating no memorization. However, if the representation alignment for g is worse than for f, their difference will be non-zero, hence LayerMem will also be non-zero.

W2b: Also, these metrics might depend on the norms of intermediate representations.

To address the reviewer’s concern, we performed additional measurements on the outputs of each convolution block in ResNet50 trained with SimCLR on CIFAR10. In particular, we measured the alignment loss ($\ell_2$ distance between the representations of two different augmentations) on the encoder f and g used for LayerMem. This is the signal that the metric operates on (not directly on the representations), and hence, its magnitude (norm) would influence the metric.

In the Table below, we observe that while the alignment loss first increases and then drops significantly, LayerMem increases continuously, suggesting that LayerMem does not depend on the norm of the representation alignment. The magnitude of the norm could also be influenced by the dimensionality of the representation. Therefore, we also report output dimensionality. We observe that the dimensionality of the representations after the 1st block and the 4th block are the same. However, the memorization score is much higher for the layer after the 4th block highlighting that our LayerMem is also independent of the output dimensionality.

W3a: There is room for improvement in the writing. The paper claims that “memorization increases but not monotonically,” yet I observed that LayerMem increases monotonically and even uniformly [...].

The quote used in the review “memorization increases but not monotonically” is not taken from our paper. Our sole statement about monotonicity in layer memorization is (in line 197) “Delta LayerMem indicates that the memorization increases in all the layers but is not monotonic.” We agree that LayerMem would indicate a monotonic increase (Table 1). As we note in the paper (line 195) “However, our Delta LayerMem presents the memorization from a more accurate perspective, where we discard the accumulated memorization from previous layers, including the residual connections.” Looking at the Delta LayerMem column in Table 1, we see that values go from 0.029->0.002->0.027…. which is not a monotonic increase. We changed the above statement to “Delta LayerMem indicates that the memorization increases overall with deeper layers but it is not monotonic.”

W3b: Some interesting results, e.g., Lines 247 and 387, are only included in the Appendix.

We appreciate the reviewer’s suggestion on which results should be additionally featured in the main body of the paper and would use the additional page in case of acceptance for including the “Verification of Layer-Based Memorization” from Appendix C.6 and the “Additional Verification of UnitMem” experiments from Appendix C.1.

W3c: The paper attempts to correlate atypical datasets and memorization, but the evidence does not strongly support the claims.

Relating atypical examples with memorization is not a contribution from our work. This relation has been established, e.g. by [23] and [47]. We only state their findings.

Overall, considering the paper as one of the first attempts to investigate memorization in a self-supervised learning setting, I don’t find major weaknesses. Despite some limitations, I lean toward acceptance since I believe such attempts should be encouraged.

We thank the reviewer for their encouraging feedback and are happy to provide further insights and clarifications that would strengthen the reviewer’s confidence for an acceptance.

W1: Adding insights and explanations

We thank the reviewer for their suggestion. We went through the paper and identified the following sections for adding explanations and insightful remarks within the additional page that could be added to the paper in case of acceptance:

• Section 4, definition of LayerMem: We will add further insights on how the use of the reference encoder g makes LayerMem independent of the internal representation dimensionality of the encoder. Therefore, we point to our experiments showing that LayerMem is independent of the internal representation dimensionality, since we always set two encoders’ internal layers with same dimensionality in relation.
• Section 4.1, line 195: We will elaborate more why metrics, like LayerMem that measure memorization in the forward pass measure an accumulated effect of memorization. This is happening since the output of the previous layer influences the current one. This motivates the study of other metrics, such as Delta LayerMem, which consider layers more independently and can lead to more fine-grained insights on memorization.
• Section 4.1, line 219: We will relate the observation that “fully-connected layers memorize more than attention layers” to the number of parameters in the layers. The fully connected layers have more storage capacity, which could, potentially have an impact on privacy-preserving architectural choices in neural networks.
• Section 5.1, line 314: We will explain more on the memorization patterns between different datasets and how they relate to the distribution of atypical examples in the dataset.
• Section 5.1, line 327: We will add insights on how our finding–that highest memorizing units memorize the highest memorized data points from prior work–suggests that there is a particular set of neurons responsible for memorization, and that the memorized data is not distributed over all neurons within the network.
• Following the suggestion by Reviewer 2UD6, we also added additional insights into the consistency of data memorized in different layers (see response to the reviewer and attached PDF). If reviewer see some other parts in the paper where we could add more explanations (as well as intuitions and potential implications), we’d be grateful for the suggestions.

W2: Presentation of results

We appreciate the reviewer’s suggestion. We wanted the tables to convey the full picture of results. We hope that the added insights (W1) will provide guidance to the reader on how to interpret the trends. Additionally, we performed additional experiments during the rebuttal (see general response, attached PDF, and responses to the other reviewers) and we tried to include more Figures (1 and 2) that would appeal to the readers with visual preference.

W3: Novelty

While our LayerMem builds heavily on SSLMem, it is only a part of our innovations. In summary, the paper makes the following original contributions:

1. We introduce UnitMem as the first practical method to localize memorization in SSL encoders down to the unit level (Section 5).
2. We derive Delta LayerMem from LayerMem (which is indeed based on SSLMem), inspired by our insight that LayerMem does measure accumulated memorization rather than individual layers’ memorization.
3. We perform thorough studies on localizing memorization in layers and units and leverage these insights to 1) inform the design on more effective fine-tuning approaches (Table 4) and 2) identify possible future pruning strategies to reduce both memorization and model complexity (Table 5).

Q1a: Memorization pattern over different datasets

While the distributions of UnitMem scores (Figure 1a) look similar, the main difference is in the number of highly memorizing units (those that exhibit high values of UnitMem): the SVHN dataset, which is visually less complex than the CIFAR10 or STL10 dataset, has the lowest number of highly memorizing units. Additionally, all encoders considered in this work are trained with self-supervised learning objectives, yielding a similar memorization pattern.

Q1b: Memorization pattern over different augmentations

Note that the different augmentation sets presented in the submission are only used at the inference time only during the calculation of UnitMem (see Equation 5), not during training. To study how different augmentations during training affect UnitMem, we performed additional experiments using different augmentation strengths during training. The results, reported in Table 2 in the attached PDF highlight that stronger training augmentations lead to higher UnitMem. We hypothesize that this effect is caused by the model having to incorporate more information on the training samples to make up for the scarser input information (through the stronger augmentation).

Q2: Practical Benefits

Our results suggest that UnitMem can also inform memorization-based compressing schemes for neural networks. As shown in Table 5, pruning the least memorized units preserves better downstream performance than just removing random units (while removing memorizing units harms downstream performance most). We further hypothesize that UnitMem could yield a signal to strengthen white-box membership attacks against SSL encoders and that the knowledge of which neuron memorizes which data point from training could be used as a form of model watermark.