Understanding Reconstruction Attacks with the Neural Tangent Kernel and Dataset Distillation

We thank the reviewer for their thoughtful critiques regarding the theory aspect of this paper, as well as recognizing the sound empirical findings. We would like to clarify the questions raised by the reviewer in this rebuttal, and these changes will make their way into the paper in future revisions.

Theorem 1

We will reword the theorem as follows:

Theorem 1 If $L_\textrm{reconstruction} \to 0$ (from Eq. 7), as $w\to \infty$, then $X_R \to X_T$ in probability for training data, $X_T$ on the unit hypersphere.

The proof follows closely with the outline provided in the text as is, but to clarify some bits which are possibly unclear, we rewrite the proof steps here:

$

L_{recon} = &\quad \Big|\Delta \theta - \sum_{\mathclap{\alpha_j x_j \in \alpha^R, X_R}} \alpha_j \nabla_{\theta_f} f_{\theta_f} (x_j)\Big|^2_2 =\Big|\sum_{\mathclap{\alpha_i, x_i \in \alpha^T, X_{T}}}{\alpha_i \nabla_{\theta_0}f(x_i)} - \sum_{\mathclap{\alpha_j x_j \in \alpha^R, X_R}} \alpha_j \nabla_{\theta_f} f_{\theta_f} (x_j)\Big|^2_2 \ &= \Big|\sum_{\mathclap{\alpha_i, x_i \in \alpha^T, X_{T}}}{\alpha_i k_{\theta_0}(x_i, \cdot)} - \sum_{\mathclap{\alpha_j x_j \in \alpha^R, X_R}} \alpha_j k_{\theta_f}(x_j, \cdot)\Big|^2_2 &\to \Big|\sum_{\mathclap{\alpha_i, x_i \in \alpha^T, X_{T}}}{\alpha_i k_{NTK}(x_i, \cdot)} - \sum_{\mathclap{\alpha_j x_j \in \alpha^R, X_R}} \alpha_j k_{NTK}(x_j, \cdot)\Big|^2_2 \mathbf{i.p.}

$

Define

$

P_T = \sum_{\alpha_i, x_i \in \alpha^T, X_{T}}{\alpha_i\delta(x_i)}, P_R = \sum_{\alpha_j, x_j \in \alpha^R, X_{R}}{\alpha_j\delta(x_j)}

$

Which are both signed measures over $\Omega = S^{d}$.

Then we can rewrite

$

\sum_{\mathclap{\alpha_i, x_i \in \alpha^T, X_{T}}}{\alpha_i k_{NTK}(x_i, \cdot)}

$ as

$

\int_{\Omega} k_{NTK}(x, \cdot) dP_T(x)

$, and likewise for$P_R$. Then, we have

$

L_{recon} \to \Big|\int_{\Omega} k_{NTK}(x, \cdot) dP_T(x) - \int_{\Omega} k_{NTK}(x, \cdot) dP_R(x)\Big|^2_2 = MMD_{k_{NTK}}(P_R, P_T)

$

This is the maximum mean discrepancy described in [1]. From [2], we have that $k_{NTK}$ being universal over $S^{d}$ [3] implies that i $MMD_{k_{NTK}}(P_R, P_T) = 0$ implies that $P_R = P_T$, i.e. $X_R = X_T$. $L_{recon} \to 0$, as assumed, so $MMD_{k_{NTK}}(P_R, P_T) \to 0$, in probability. As $k_{NTK}$, and the MMD are continuous in their inputs, this implies that $X_R \to X_T$ in probability as well.

We hope that this clarifies the proof. We acknowledge that the description in the text is rather space constrained and in future revisions we will have the more informal statement in the main text, and defer the proof to the appendix, as suggested by the reviewer.

[1]Arthur Gretton, Karsten M. Borgwardt, Malte J. Rasch, Bernhard Sch ̈olkopf, and Alexander Smola. A kernel two-sample test.

[2] Jacot, A., Gabriel, F., & Hongler, C. (2018). Neural Tangent Kernel: Convergence and Generalization in Neural Networks.

[3] Bharath K. Sriperumbudur, Kenji Fukumizu, and Gert R.G. Lanckriet. Universality, characteristic kernels and rkhs embedding of measures.

[4] Timothy Nguyen, Zhourong Chen, and Jaehoon Lee. Dataset meta-learning from kernel ridge- regression.

We would like to thank the reviewer for acknowledging the contributions of this paper in linking the training dynamics of neural networks with the memorization phenomenon, and in recommending its acceptance. We would like to address the questions raised by the reviewer here:

Data Dimension Indeed the efficacy of our reconstruction attack depends on the dimension of the data. Previous work has shown that the quality of the NTK approximation suffers with high data dimension [1,2], and as our method relies on the NTK approximation to prove theorem 1, we expect our attack to be less effective with larger data dimension. Indeed, in appendix J.2. (page 26) and figure 18 and 19 (page 25), we perform the same attack on a higher resolution dataset Tiny-Imagenet (resolution 64x64x3). We see that reconstruction quality suffers, but we are still able to reconstruct images with high quality. Combining the approach provided in this paper with prior work on understanding quality of NTK approximation [1,2] could lead to a more robust definition of how networks “encode” their data points, which is the subject of future work.

Image Initialization In all our experiments we initialize reconstruction images with random noise, ensuring that we have no a priori knowledge about the dataset. We will clarify this in future revision. We observe that our method is not sensitive to the initialization scheme in practice, although we acknowledge that the reconstruction loss is provably non-convex, but in practice this does not seem to be a major concern.

[1] Bombari, S., Amani, M. H., & Mondelli, M. (2023). Memorization and Optimization in Deep Neural Networks with Minimum Over-parameterization.

[2] Adlam, B., & Pennington, J. (2020). The Neural Tangent Kernel in High Dimensions: Triple Descent and a Multi-Scale Theory of Generalization.

We were wondering if the reviewer has had a chance to consider our rebuttal, and if there were still any remaining concerns which we can address before the discussion period ends.

We would like to thank the reviewer for acknowledging our contributions and recommending our acceptance. We would like to take this space to address some of their concerns.

Various presentation points We apologize for the poor readability of some of the figures. Indeed we were very constrained in terms of space, and consequently had to reduce figure sizes, as well as move certain results to the appendix, which is why the results on convolutional architectures were moved there. We will address these concerns, along with the other minor concerns pointed out by the reviewer in future revisions.

Definition of outliers We define an outlier to be a datapoint which is harmful/uninformative to the training procedure, has a strange/unusual visual appearance or is far away from other training data in terms of training dynamics. Moreover, there is no agreed-upon definition in the literature. We show that easily reconstructed images fit these three properties as follows:

1. Harmful to the training procedure/uninformative - see fig. 6, in which we show that removing these data points do not adversely affect training
2. Strange visual appearance - see figures 25-40 in the appendix, where we see that the more easily reconstructed images tend to have more “unusual” features (i.e. solid-colored backgrounds for CIFAR-10, faint lines for MNIST)
3. Difficulty to fit - see the discuss of alpha values We do not believe that L2 distance in the euclidean space is a good definition for an “outlier” data point when working with neural networks, as it does not take into account data structure, or the training procedure. We believe that the qualities that we looked at in terms of defining an outlier are suitable properties and reflect a more nuanced view which takes into account training dynamics and dataset properties. We are happy to discuss this further.

Self Supervised Learning and output structure We note that our attack works with vector outputs, as seen by the results on multiclass classification. Extended our work to self-supervised learning may be more difficult as self-supervised learning typically occurs outside of the NTK regime, i.e. there is a large amount of representation learning, which is not modeled by the NTK. However, we believe that our work fills a vital first-step in understanding reconstruction attacks in a more simplified regime.

Non-MSE losses Our method should work on non-MSE losses such as cross-entropy. We include a discussion in appendix G.2. The main requirement of our attack to work is that the change in network parameters lies in the span of the datapoint gradients, and that these gradients do not change significantly over the course of training. This is fulfilled by wide neural networks under any loss (see [1], appendix B for more details). We chose to focus on MSE loss in this work as it allows a closed for solution for the $\alpha$ values, simplifying analysis, but the analysis still applies. In the proof for Theorem 1, we do not explicitly require MSE loss, so this proof still holds for non-MSE losses in the kernel regime.

Weight Decay We do not assume weight decay, but including weight decay would not affect the theoretical results in our paper. Like in the answer to the previous question, weight decay does not affect whether the weight changes remain in the span of the dataset gradients, and thus our analysis still holds. Note that we can have a similar effect to weight decay with early stopping, and we carefully ablate the effects of early stopping in appendix G.1.

We hope that these points address the reviewers main concerns and we would be happy to discuss further with them.

[1] Lee, J., Xiao, L., Schoenholz, S. S., Bahri, Y., Novak, R., Sohl-Dickstein, J., & Pennington, J. (2020). Wide neural networks of any depth evolve as linear models under gradient descent